Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) Report Number : ICRR0020961 1. Project Data Project ID Project Name P106684 AR PROSAP2 2nd Prov Agric Dev Country Practice Area(Lead) Argentina Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-75970 30-Mar-2015 453,000,000.00 Bank Approval Date Closing Date (Actual) 25-Sep-2008 15-Mar-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 300,000,000.00 0.00 Revised Commitment 300,000,000.00 0.00 Actual 299,992,549.45 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as articulated in the Project Appraisal Document (PAD, p. 5) was to: "increase the productivity and sales volume of small and medium-size producers." The Project Development Objective (PDO) as articulated in the Loan Agreement (LA, p. 5) was to: "increase the productivity and sales volume of small and medium-size producers benefited under the Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) Project." The PDO statement in the PAD and the LA were almost identical. Therefore, IEG adopts the LA formulation as the statement of the PDO. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project included three components. 1.Support to Pre-Investment Activities (appraisal cost: US$11.00 million, actual cost: US$14.80 million). This component would provide support to: (i) meeting provincial eligibility requirements through the implementation of minimum functional institutional development activities and the preparation of a Provincial Agricultural Development Strategy in each participating province; (ii) identifying and preparing sub-project profiles and, once selected by the Investment Programming Committee, full subproject proposals; (iii) designing, on a pilot basis, Regional Development Initiatives to increase the competitiveness of selected micro-regions based on the creation or strengthening of Development Consortia which would develop and implement Regional Improved Competition and Development Plans; and (iv) promoting, on a pilot basis, the establishment of public-private innovation networks to exchange knowledge regarding agricultural practices for purposes of improving the competitiveness and market access of small and medium size farmers, as well as for facilitating the link between knowledge institutions and farmers. 2. Investment Sub-projects for Competitiveness (appraisal cost: US$392.70 million, actual cost: US$371.90 million). This component would finance the cost of agricultural competitiveness and/or agricultural productive investments subprojects whose profiles have been approved by the Investment Programming Committee, based on their agreement with the project objective and provincial priorities, and whose detailed preparation reports were cleared by the Technical Evaluation Committee. Sub-project typologies to be supported include: (i) public infrastructure and services (with complementary private on- farm and off-farm investments); (ii) institutional development; and (iii) pilot instruments, including Regional Development Initiatives and Innovation Transfer Initiatives. 3. Project Management (appraisal cost: US$19.40 million, actual cost: US$33.20 million). This component would finance works, goods, and equipment; consultant and non-consultant services; training; monitoring and evaluation and operating costs related to overall project coordination and management. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The project was estimated to cost US$423.80 million. Actual cost according to the ICR (Annex 1) was US$420.60 million. Financing. The project was financed through a Specific Investment Loan (SIL) worth US$300.00 million. According to the ICR Annex 1) the loan was fully disbursed. Borrower Contribution. Counterpart financing from the Borrower was expected to be US$57.30 million, in addition to US$66.5 million in private investments. Actual amounts disbursed according to the ICR (Annex 1) were US$86.60 million and US$34.00 million from the Borrower and local farmer organizations, respectively. Dates. The project was expected to close on March 30, 2015. It closed almost two years later on March 15, 2017. It was restructured three times, all level 2. The first restructuring was on November 16, 2012, when the amount disbursed was US$100.48 million, because the project triggered two additional safeguard policies: OP/BP 4.04 (Natural Habitats) and OP/BP 4.37 (Safety of Dams) due to the nature of six new sub-projects proposed by the Borrower. The second restructuring was on September 8, 2014, when the amount disbursed was US$150.17 million, in order to extend the closing date by 18 months to 09/30/2016; reallocate Loan funds between categories; and increase the Bank’s percentage financing from 80% to up to 100% for Category 1(b) Works under Provincial sub-projects. The third restructuring was on August 3, 2016, when the amount disbursed was US$254.56 million, in order to extend the closing date by about five and half months to March 15, 2017 to allow for the completion of works under Provincial and National subprojects, and finalization of key studies; and change percentage of eligible expenditures to up to 100%" for Categories 2(a), 2(b), 2(c). The Mid-Term Review was conducted on November 20, 2012, compared to a planned date on March 15, 2012. 3. Relevance of Objectives & Design a. Relevance of Objectives In Argentina, agriculture contributes around 60% of total goods for exports, 9% of GDP, and 22% of the value added to the goods sectors. It generates about 12% of national employment and over 20% when including transport and commerce services related to food and agriculture. Further, about 35% of the country's population live in predominantly agricultural areas or reside in cities linked to agro-industrial activities. To enhance the competitive conditions of small and medium-size producers, three areas need to be addressed: rural infrastructure, private investment, and generation and adoption of agricultural and marketing innovations (PAD, pages 2 and 3). At project appraisal objectives were in line with priorities of the Government for the agriculture sector and Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) its investment strategy for the Regional Economies. The strategy emphasized enhancing competitiveness conditions through: (i) strengthening the management and efficiency of irrigation systems; (ii) improving roads, electricity, and other rural infrastructure; (iii) transferring technology; (iv) establishing product sanitary and quality systems to meet international market standards; and (v) promoting agribusiness through collective actions in value chains. Objectives were also in line with two of the three pillars (sustained growth with equity and social inclusion), as outlined in the Bank's 2006-08 Argentina Country Assistance Strategy (CAS). At project completion, objectives remain substantially relevant to Government’s sector goals reflected in its Agri-food and Agribusiness Strategic Plan (Programa Estratégico Agroalimentario e Agro-industrial Participativo e Social 2010-2020), which the Argentine Government has extended to 2030 with revised targets. This strategy aimed to move the Argentine agriculture model from primary production to value- addition through supporting small and medium-sized producers, increasing productivity through investments in rural infrastructure, technical assistance, use of appropriate/innovative technologies including climate risk mitigation, and increasing market access. Objectives were also in line with pillar one of the Bank’s FY2015- 2018 Country Partnership Strategy (CPS) which sought to maintain Argentina’s position as a globally important food producer by addressing the needs of small and medium-sized farmers, an ongoing national development challenge. Pillar one of the CPS aimed to create sustainable employment creation in farms and firms to unlock long-term productivity growth and job creation. The CPS also emphasized raising the productivity of small and medium-sized farms in low income regions (CPS, Results area 3). The PDO was clear and focused, but lacked a clear connection to higher level objectives. Rating Substantial b. Relevance of Design The Results Framework reflected no clear connections between project inputs, outputs and expected outcomes. To achieve the stated objective, the project design featured market-oriented, demand-driven mechanisms to increase the availability and quality of productive assets, create human capital, strengthen institutional capacities, and improve the access of small and medium-size producers to markets. The Project design also sought to establish linkages with commercial actors to stimulate rural investment opportunities, while, at the same time, create conditions where technical and institutional services raise rural capacity for sustainable development. These activities were relevant and directly linked to the PDO. However, design suffered from notable shortcomings including: unrealistic design features due to inadequate analysis of critically important institutional elements particularly for the water infrastructure at the provincial level, and inaccurate targeting of large-scale provincial investments as the process did not factor in the political economy of national-provincial relationships and the superior efficiency of certain provinces. Design favored large irrigation sub-projects rather than a more balanced approach that would attend to producers’ constraints at the farm level, and it featured a multi-sector approach without including a coordination mechanism that would have involved non-agricultural ministries in the consultation, decision-making or implementation process. Design also attempted to improve linkages between production and markets, but efforts were limited to the physical facilitation of the production and marketing process through electrification and rural roads. Finally, target beneficiaries were not clearly defined because existing Argentine poverty and Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) household surveys did not cover rural areas. Poor targeting combined with inadequate cost-benefit analysis resulted in benefit inequetable distribution implications entailing expensive public infrastructure with relatively low numbers of direct beneficiaries. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective PDO: to increase the productivity and sales volume of small and medium-size producers benefited under the Project. The PDO as stated includes two sub-objectives: (a) to increase the productivity of small and medium-size producers benefited under the Project. (b) to increase the sales volume of small and medium-size producers benefited under the Project. Rationale Outputs • 19 investment plans (target: 15) were prepared and adopted in each province receiving financing from the project. • 3,274 ha (target 673 ha) were established in new productive areas as a result of improved infrastructure (output based on 6 out of 21 sub-projects). • Average percentage increase in the efficiency of water conduction of the rehabilitated irrigation systems reached 26% compared to a target of 65%. • Average percentage incremental increase in the yields of crops in zones corresponding to irrigation projects reached 13.7% (the range was between 7% and 25% depending on the crop) compared to a target of 20%. • Average incremental increase in the transit of vehicles in road improvement sub-projects reached 70% compared to a target of 2%. The PAD target was surprisingly very low. • Maintaining Argentina’s sanitary status in relation to Transmittable Spongiform Encephalitis (TSE); by project completion Argentina status was maintained according to the World Animal Organization (OIE) as a country with negligible risk. The project demonstrated via histopathological and biochemical analysis of animals/humans that TSE was not present in the national territory; the project also improved controls on the production and importation of risk materials; evaluated the national ovine genotype for prevalence of Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) genotypes susceptible to PrP; and provided comprehensive information on diseases with nervous system symptoms in the national territory. Outcome • This sub-objective was assessed based on the increase in the productivity of labor and the increase in the productivity of land. The evidence provided in the ICR were drawn from the impact and ex-post economic evaluations of irrigation, rural roads, rural electrification and seeds sub-projects. Labor productivity was measured as the units of gross income generated for each unit of man days of labor. Results from four project-financed irrigation sub-projects showed that the average increase in labor productivity was 8% compared to a target of 10%. Results for land productivity showed an average increase of 9.5% compared to target of 12%. Also, average yields of grapes and diverse vegetable crops for six sub-projects (irrigation, electrification, and seeds development) increased 13.8%, compared to a target of 20%, and ranged from 7% to 25%, depending on the crop. Beneficiary producers in project areas were each cropping an average of two additional hectares in 2016 compared to 2007. • While the above reported results seem encouraging, there were major concerns about the reported evidence. The project suffered from two notable shortcomings, highlighted by the ICR (para 3.2.8), that cast doubt about the reported evidence. First, there were concerns about the representation of the targeted beneficiaries which according to the ICR reached 79,000 compared to a target of 68,000 producer beneficiaries. However, 34,000 (43%) of these represented just the Fire Prevention sub-project. While 14,000 represented youth who benefitted from the Young Entrepreneurs Program, most received project-supported training and business plan guidance only. The national sub-projects (Innovation Transfer Initiative and Regional Development Initiative) had 12,500 beneficiaries, but the 12 Innovation Transfer Initiatives were not widely disseminated; and three of the 15 Regional Development Initiatives were implemented (one partially). Seed financing for business development benefitted only 400 beneficiaries. • The ICR (para 3.5.1) noted that the project financed infrastructure sub-projects valued at some US$69.0 million in two provinces in the Pampas region which, according to the PAD, was not among the regional economies that were to be targeted by the project. Second, the project was expected to cover about 675,000 ha or around 10 ha per beneficiary. However, fewer producer beneficiaries implies a significantly smaller area benefiting from project interventions. Further, beneficiary producers were cropping on average, only two additional hectares by 2016. This also implies that the land and labor productivity indicators and the sales volume (production volume) indicators, even if achieved, were at smaller scale than expected. • In a further communication during the preparation of this Review, the project team explained that the project was undermined by the lack of a functional M&E system including the absence of relevant baselines. This made it impossible to accurately assess the significance of project outcomes at completion, despite achieving some physical targets. • Based on the afore mentioned discussion, the efficacy of achievement of this sub-objective is rated modest because the project fell short on two outcome indicators, and the numbers of direct producer Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) beneficiaries and area covered by the project fell short of PAD expectations. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective (b) to increase the sales volume of small and medium-size producers benefited under the Project. Rationale Outputs • The outputs mentioned under sub-objective (a) pertain to this sub-objective as well. Outcome • The outcome of this sub-objective was to be assessed through increments in sales volume. However, due to methodological and measurement issues, including time constraints and lack of a baseline, it was not possible to measure changes in sales volume. Alternatively, production volume was used as a proxy measurement. Results from five irrigation sub-projects showed positive impacts, with increases in production volume ranging from 4% to 51%. The aggregate gross production increase with- project vs without-project was 9.4% compared to an end target of 20%; and, average variation across all 15 crops associated with 5 irrigation sub-projects in 3 provinces was 23%. However, as indicated under sub- objective a, concerns about the quality of evidence presented in the ICR and there being fewer producer beneficiaries than the 68,000 expected at appraisal, implied that the increment in production volume was at smaller scale than expected. Therefore, the efficacy of achievement of sub-objective b is rated modest. Rating Modest PHREVDELTBL PHREVISEDTBL 5. Efficiency Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) Economic and Financial Efficiency ex ante • The overall Economic Internal Rate of Return (EIRR) of the project was estimated at 17.4% at appraisal. This estimate did not consider some factors that were difficult to quantify including the social benefits derived from the improved availability of electricity and road transportation, increased drinking water availability, and safer food availability. • Seven sub-projects proposals (rehabilitation/modernization of irrigation systems, rural roads; and sanitary and quality control) were chosen as a sample to illustrate the cost/benefit flows. Financial results and incremental benefits were based on farm models of those sub-project proposals. • Analytical parameters included a 12% annual discount rate and a 20-year evaluation period. Financial prices were converted into economic prices, subtracting the Value-added Tax (VAT), which was 21% for investment goods and 10.5% for production inputs. • Assumptions based on farm models: incremental net benefits for the three model types were estimated on the basis of the farm models; all transfers, including taxes and subsidies, were excluded from the analysis, assuming that market prices reflect economic opportunity cost (as is the practice for economic analysis in Argentina); so no adjustments were made to prices in the models. • There was no ex-ante analysis of national components or sub-projects for enhancing competitiveness, such as: on-farm investment grants; youth entrepreneurial support; regional development initiatives; and technology innovation initiatives. ex post • The aggregate ex-post financial and economic IRRs were 10% and 13%, respectively. This was lower than the EIRR appraisal estimate of 17%. The analysis used the same analytical parameters as used at appraisal. • Ex-post financial and economic analyses were conducted for the provincial sub-projects implemented and operated for at least one production cycle. There was also an ex-post financial analysis for: (i) a small sample of on-farm investment grants (ANR, including those of young entrepreneurs); and, (ii) technology innovation initiatives already implemented and with observed results. • Provincial sub-projects: Nine provincial sub-projects were assessed. These represented 48% of the overall budget for provincial investment sub-projects. The overall ex-post financial and economic NPVs of the assessed provincial sub-projects were US$(587,187) and US$22.8 million, respectively, involving 6,176 direct beneficiary families with 66,971 hectares. The overall ex-post financial and economic IRRs were 12% and 15%, respectively. • Matching grants: A random sample of 24 (out of 608) on-farm investment grants (ANR) was assessed. The overall financial IRR was estimated at: 20% over 5 years; and 32% over 10 years. • Innovation Technology Initiatives (ITI): Three (out of 12) ITI were assessed. The overall financial IRR was estimated at 18% over 5 years, and 35% over 10 years. The beneficiary population involved 1,900 families. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) Administrative and Institutional Efficiency The project suffered from implementation delays. Despite a two-year extension, only six out of 10 irrigation sub-projects were completed in the final year--the remaining four (one national, three provincial) were still incomplete at ICR finalization. Implementation efficiency was negatively impacted by a slower flow of funds, import restrictions; and challenging and problematic procurement procedures. While Government staff turnover was an internal personnel management issue beyond the scope/control of the Bank team, repeated turnover of managers and technical personnel on both the Bank and Provincial Government teams was disruptive. Finally, at 171% of the appraisal estimate, the project management costs were significantly higher than anticipated. Efficiency is rated modest due to a lower than expected economic and financial returns on provincial sub- projects, lower overall ERR compared to the appraisal estimate and weaknesses of administrative efficiency. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  17.40 Not Applicable 0 ICR Estimate  13.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives was rated substantial while relevance of design was rated modest. Efficacy of the first sub-objective was rated modest because the project fell short on two outcome indicators, and the numbers of direct producer beneficiaries and area covered by the project fell short of PAD expectations. Efficacy of the second sub-objective was rated modest due to concerns on the quality of evidence presented in the ICR, in particular, fewer producer beneficiaries implied that the increment in production volume was at smaller scale than expected. Efficiency was rated modest due to lower than expected economic and financial returns on the provincial sub-projects, lower overall ERR compared to the appraisal estimate and weaknesses on administrative efficiency. With substantial relevance of objectives, modest relevance of design, modest efficacy for each of the two sub- objectives and modest efficiency, the overall development outcome rating is moderately unsatisfactory. Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating Institutional risk. Argentina has no national water law as water is under provincial jurisdiction. This puts the project financed irrigation infrastructure under provincial jurisdiction with regards to operation and maintenance. While some provinces had established O&M arrangements as the cases of Mendoza, Salta, San Juan provinces, other provinces like Catamarca had lower institutional capacity. Sustainability in provinces with lower capacity is dependent on continued investment in institution-building, social capital formation and civic involvement. On the other hand, national projects such as the National Service for Food and Agriculture Sanitation and Quality and Transmittable Spongiform Encephalitis (TSE) are likely to be sustained due to their high political and economic priority. Also, sustainability of the Non-Refundable Contributions (ANR) is likely because activities were successful and beneficiaries were legally required to pay for O&M. However, there are concerns on the sustainability of the Young Rural Entrepreneurs Program because it lacked an institutional anchor, and most of the 400 investments were very small. Also, sustainability of the associations established under the pilot Regional Development Initiative/Micro-regional Development Initiatives is questionable as no clear strategy was developed to address availability of new financing and ensure greater private sector engagement. Finally, sustainability of the Innovation Transfer Initiative will depend on the commitment of relevant public institutions. Environmental/Climate risk. Climatic events seriously damaged sub-projects in Cordoba, Catamarca and Salta provinces. This prompted the Unit for Agriculture and Rural Change (UCAR) to increasingly and systematically direct efforts to include climate risk provisions/projections into all aspects of sub-project formulation. Given the uncertainties outlined above in relation to several project investments, the Risk to Development Outcome is rated substantial. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry •The Second Provincial Agricultural Development Project (PROSAP II) built on the strengths of PROSAP I and continued to serve as the investment arm of the GOA’S strategy for agricultural development in the Regional Economies. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) • Project design benefitted from lessons and experience of PROSAP I in three main areas: resource allocation; capacity building; and private sector incentives. Notable lessons reflected in project design included: the need to design activities in accordance with the technical, institutional, and financial capacity of each province; and sub-projects need to carefully consider potential implementation risks and allocate additional capacity-building support as needed, a transparent mechanism to identify sub-project proposals, and to ensure private sector participation through including appropriate incentives. • Design suffered from notable shortcomings. First, it overlooked the political economy of national- provincial relationships and the superior efficiency of certain provinces which contributed to uneven implementation progress. Second, design was skewed in favor of large sub-projects rather than a more balanced approach that addressed constraints at the farm level. Third, water management in the country was not fully assessed and the implications of divergent provincial water laws were not taken into account. Fourth, design lacked a coordination mechanism between different ministries despite the project being a multi-sector operation. Fifth, design aimed to target small and medium-size producers, but the PAD did not characterize them by region, productive typology, assets or other variables. This created targeting problems during implementation. Finally, design did not feature enough activities to facilitate commercial links between the supply and demand sides of value chains; beyond electrification and rural roads. • Fourteen risks were identified at the appraisal stage. Three were rated high, six were moderate, three were significant and two were low. Overall Project Risk was assessed as Moderate. While risk analysis was sound and included relevant mitigation measures, some risks were overlooked. For example, provincial capacity was only viewed as a risk in the context of procurement despite that the participating provinces had not implemented with the Bank large-scale public infrastructure that required complex engineering, contracting, administration and competent supervision skills. Also, given the types of investments supported by the project, the risks associated with operational sequencing, implementation timeframe, and the likelihood that impact would be achieved by closing, were all not considered. • M&E suffered from implementation weaknesses (see section 10 b). • As noted above design suffered from major shortcomings that undermined Quality at Entry and subsequent implementation. Therefore, QAE is rated moderately unsatisfactory Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank conducted about 16 supervision missions an average of twice yearly after project effectiveness. The project was headed by four different TTLs throughout the implementation period, which also saw multiple rotations of Bank specialists. The project team did not include an irrigation specialist. In a further communication, the project team explained that the team relied on an FAO irrigation specialist who was skilled and knowledgeable of local conditions. Supervision was flexible and responded to borrower demands. However, performance was mixed. Supervision focused on facilitating execution of provincial sub-projects as they represented the majority portion of the loan. According to the ICR (para 5.2.1.) safeguards received intensive supervision, but the introduction of new safeguards requirements disrupted sub-project procurement/execution. While financial management was strong, procurement activities were negatively impacted by the repeated turnover of the Bank specialist. Although sub-project implementation was slow, the Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) ratings of the Implementation Progress (IP) and Development Objectives (DO) were consistently rated satisfactory, including in the MTR, up to mid-2015. These optimistic ratings are questionable given the lack of data to substantiate achievement. A notable shortcoming on behalf of supervision was the failure to address M&E weaknesses earlier during implementation. The supervision team did not address the weaknesses in the RF despite that the project was restructured three times. Despite the team's efforts in the final two years of the project, the lack of evidence undermined the final assessment of the project's outcomes. Overall Bank performance suffered from significant shortcomings with regards to Quality at Entry and supervision. Therefore, overall Bank Performance is rated moderately unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance National Government. The National Government created the Unit for Agriculture and Rural Change (UCAR) as a front-line sector institution responsible for provincial agricultural development. UCAR had a stable leadership throughout implementation which was helpful to the project. However, in earlier implementation period of PROSAP II counterpart funding was neither adequate nor timely. According to the ICR (para 5.2.2.) "Bank loan funds were used to finance local counterpart, a practice which persisted for several years despite Bank pressure to desist." Inter-sector coordination on the roads and energy sub- projects could have benefitted from more attention from the Government. Finally, sub-projects suffered from delays due to prolonged time needed to secure ministerial approval. Works price adjustments were challenging given the inflationary environment and distortions in the National Institute of Statistics and Censuses (INDEC) index. This created difficulties with contractors and slowed sub-project execution. Provincial Governments. Provincial Governments showed steady progress in their capacity to manage large-scale infrastructure works and understanding of Bank fiduciary requirements. The provinces also developed a better understanding of the internal structure and functions of the Provincial Execution Unit/Central Project Execution Unit. This facilitated better management and reduced the isolation which suppressed achievement at earlier stages. However, provincial institutions experienced frequent and disruptive structural change and a high rate of professional turnover in the provincial sub-project units. This negatively affected sub-project processing and implementation. Finally, provincial financial management and procurement capacity needed time and training/mentoring to improve. Government Performance Rating Moderately Satisfactory Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) b. Implementing Agency Performance According to the PAD (p. 70) the overall project coordination and administration would be the responsibility of the Secretariat of Agriculture, Livestock, Fisheries, and Food (SAGPyA) within the Ministry of Economy and Production. This would be through the Project Implementation Unit (PIU) that would be directly in charge of FM tasks. In 2010, the project implementation was transferred to the newly-established Unit for Agriculture and Rural Change (UCAR) within the Ministry of Agriculture, Livestock and Fisheries (MAGyP) where a central project Unit for Execution and Coordination (UEC) was responsible for the Bank loan. This change made the Bank loan one among other loans in a larger portfolio. This might have contributed to less attention to the project's M&E. The UEC interacted formally with the provinces through Provincial Execution Units (UEP) - responsible for sub-project coordination and supervision - Financial Administration Units and Sub-project Implementation Units (SIU). These institutional arrangements lacked a mechanism to coordinate with other relevant sector ministries, e.g., roads and energy. UCAR facilitated execution of sub-projects and strengthened the capacity of provincial institutions through training. According to the ICR (para 5.2.4.) "UCAR’s Environmental and Social Unit and provincial teams made a valuable contribution to the implementation and monitoring of the social safeguards plans." However, prioritization and selection of sub-projects by UCAR was "opaque despite established criteria." A notable shortcoming on behalf of UCAR was the mismanagement of the project's M&E (see section 10b for further details) combined with the problematic procurement management (see section 11b for further details). Implementing agency performance is rated moderately unsatisfactory due to a mixed performance that was overshadowed by significant shortcomings in M&E and procurement. Overall rating of Borrower performance is moderately unsatisfactory. This rating is in accordance with the ICR Review Guidelines when ratings are split then the overall Borrower performance rating is determined by the overall outcome rating which is in this case, Moderately Unsatisfactory. Implementing Agency Performance Rating Moderately Unsatisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design • According to the PAD (p. 15) the PIU through its M&E Unit would be responsible for monitoring the day-to- day activities of the overall Project and the sub-projects, in close coordination with the SIUs. In 2010, the project was managed under UCAR which also handled M&E activities. • The PDO was to be assessed through three outcome indicators that were relevant and measurable, but targets were on the low side (ICR, p. 9, para 2.3.2.) with no clear baseline data. • The Results Framework included sixteen intermediate outcome indicators to assess different project activities. Most of these were relevant and measurable, but lacked baseline data. However, there were no Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) indicators to assess institutional strengthening activities. • Overall M&E design was sound, but was undermined by the implementation of large costly infrastructure sub-projects whose causal sequence and results did not fit the RF as designed (ICR, para 2.3.2.). With these large sub-projects the RF provided inadequate coverage for both thematic and physical levels. b. M&E Implementation M&E implementation was poor except for the final year. UCAR attributed this weakness to the need of high cost of field surveys covering the dispersed beneficiary population and respective control groups for all sub- projects; and the lack of dedicated resources to cover the planned Independent Evaluation System (IES) operational costs. M&E implementation did not receive enough attention from Bank supervision which focused mainly on the implementation of the sub-projects. It was evident at the MTR stage that UCAR had limited preliminary data and baselines to demonstrate the project's achievements. Furthermore, sub-projects faced implementation delays and were completed later than expected where they "lacked the maturity to show or causally generate meaningful results (ICR, para 2.3.3.)." Also, the M&E unit under UCAR was burdened by overseeing a large portfolio of projects and coordinating M&E activities among different provinces. This situation negatively impacted M&E implementation and should have been rectified at an earlier stage of implementation by the UCAR and the Bank. In the final year of the project the RF was informally revised to reflect achievements on the ground. Assessment of outcomes relied on a mixed strategy that included a compilation of project activities and existing, partial evaluations; and preparation of case study/impact evaluations (by sub-project) with causal attribution using a difference in difference (DID) analysis. The PDO Indicator "volume of sales" was substituted by a proxy equivalent "volume of production". c. M&E Utilization Project data was not used to inform supervision because basic monitoring data aligned to the RF were incomplete and unreliable until quite late in the project. The ICR was informed by a recent series of evaluation and economic/financial studies that were part of UCAR's evaluation plan. Overall, utilization was poor and final project assessment was undermined by poor M&E implementation and lack of baselines. Overall, M&E design was sound, but implementation was poor, and consequently utilization was limited. Therefore, M&E is rated modest. M&E Quality Rating Modest 11. Other Issues a. Safeguards • The project was rated as a Category B project. It triggered the following safeguard policies: Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) Environmental Assessment (OP 4.01), Pest Management (OP 4.09), Involuntary Resettlement (OP 4.12), Indigenous Peoples (OP 4.10) and Forests (OP 4.36). • The sub-projects financed by the project mostly involve the rehabilitation and improvement of existing works, therefore, negative incremental environmental impacts were expected to be low. It was not possible to conduct specific Environmental Assessments (EA) prior to Appraisal due to the programmatic nature of the project. The Project had therefore developed an Environmental Management Framework (EMF). The overall responsibility for ensuring compliance with the EMF rests with the Social- Environmental Unit located in the PIU (later UCAR). • On November 16, 2012, the project was restructured as two additional safeguards were triggered: OP/BP 4.04 Natural Habitats and OP/BP 4.37 Safety of Dams - due to proposed sub-projects which: intervened in areas connected to natural and critical habitats; and entailed irrigation systems connected to existing dams. • According to the ICR (para 2.4.1.) "the Bank’s monitoring of environmental safeguards was continuous and systematic from 2011 to closing." and "each sub-project needed its own environmental and social evaluation which was then built into all aspects of subproject formulation including technical assistance." • Environmental safeguards: • Natural Habitats. The proposed Livestock and Commercial Development sub-project was assessed to negatively affect highland wetlands (mallines), the main source of pasture for cattle ranching. Measures introduced to mitigate negative impacts included the exclusive use of native species, water resources management and fencing to manage grazing land. Also, three sub-projects in San Juan (Irrigation Area Canal Cespedes-Sarmiento; Rural Roads for Agricultural Development; and, Rural Electrification for Livestock Development) were identified in/around a critical Ramsar site in the south: Lagunas de Huanacache, Desaguadero and Bebedero, a site jointly administered/protected by San Juan, Mendoza and San Luis provinces. The Provincial Government’s action plan fostered greater awareness and participation of users of the Ramsar site; improved the access of institutional and social actors to water data; and, trained users in agricultural best practices. • Pest Management (OP 4.09). The concept of a Pest Management Plan was not incorporated in the ESMP until 2012, followed by a diagnostic study, identification of the legal framework and measures for incorporating this safeguard into the project. Four sub-projects (Catamarca, Mendoza, Neuquén and Entre Rios) activated this safeguard. Mitigation measures were inserted into capacity-building and productive processes through technology transfer activities and institutional strengthening. Compliance and performance did not encounter major issues and were assessed as satisfactory. • Forests (OP/BP 4.36). Consideration of native forests came to be treated more uniformly in the sub- projects after 2011 when 18 provinces adopted laws for the Territorial Ordering of Native Forests as required under the National Native Forests Minimum Budget Law of 2007 which zoned such forests in three categories based on high, medium or low conservation value, the latter open to transformation with prior preparation of an environmental impact evaluation. Bank and UCAR environmental specialists consulted local authorities, then began to systematically apply specific requirements in relevant subprojects and strengthen authorities’ capacity to understand and apply measures to prevent, mitigate Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) and/or compensate for impacts. Technical assistance and training were provided to four subprojects (livestock, irrigation and rural electrification) in Neuquén, Salta, Entre Rios and San Juan Provinces. Compliance and performance were satisfactory. • Safety of Dams (OP/BP 4.37). Two proposed sub-projects (Modernization of Irrigation Systems – San Patricio del Chanar related to the El Chanar Dam in Neuquén Province, and Reconstruction and Improvement of Irrigation System Los Altos in Catamarca, associated with the Sumampa Dam reservoir) were determined to need strengthening of dam safety mechanisms. This triggered OP/BP 4.37 Safety of Dams. The Argentine Dam Safety National Authority provided an opinion on the dams’ condition, functionality, risks and needs. It concluded that El Chanar Dam had adequate safety, maintenance and emergency plans consistent with the requirements of OP/BP 4.37 and that no other activities or studies were needed. The irrigation works proceeded as planned. In the case of Sumampa, measures to improve dam safety procedures, maintenance and environmental conservation were included in the Environmental Impact Assessment (EIA) and EMP, and the Argentine Dam Safety National Authority signed a cooperation agreement with the Province of Catamarca for regular supervision of dam safety. The related action plan resulted in hydrological and sedimentation models for the sustainable long-term management of the dam and the irrigation system, and improved environmental conditions, O&M and emergency planning. • Social safeguards: • Involuntary Resettlement (OP 4.12). The Bank’s monitoring of social safeguards was continuous and systematic from 2011 to closing. UCAR’s Environmental and Social Unit and its provincial teams made a valuable contribution to the implementation and monitoring of social safeguards plans, and increased the number of professionals dedicated to social aspects of the project. However, retroactive application of new Bank social safeguards requirements introduced in 2012 delayed implementation of sub- projects. The ICR reported that seven provinces were still implementing 340 unfinished Affected Assets Plans (AAP) under OP 4.12, based on action plans agreed with the Bank. • Indigenous Peoples (OP 4.10). This safeguard was applied to three sub-projects: Livestock and Commercial Development and the Rural Electrification II (Neuquén); and in San Juan, the Rural Electrification for Livestock Development. Initial implementation of the Indigenous Peoples’ Plan (IPP) was not systematic, there was no pre-established set of processes and participation of indigenous communities was weak even though IPP provisions began to be systematically applied during sub-project formulation. Overall, progress in embedding indigenous rights in sub-projects was slow, but important measures were institutionalized over time. b. Fiduciary Compliance Financial Management (FM). FM performance was mixed, reflecting the substantial risk for flow of funds and/or capacity issues, both central and provincial. Overall, the project experienced slow flow of funds. FM rating was downgraded to moderately unsatisfactory when temporary use of Bank funds to finance the local counterpart share of expenditures, an issue detected in mid-2010 but persisted until 2013. The Borrower was required by the Bank to deposit in the project operating account its local counterpart share prior to the payment Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) of co-financing expenses, and the Loan Agreement was amended to permit the Bank’s financing of certain categories of expenditure at 100%. The Bank encouraged provinces to accelerate their submission of expenditures to avoid the immobilization of Bank resources. Also, the Bank approved a second increase to 100% for eligible expenditures under Category 1(b) works for provincial sub-projects, but the problem persisted. The problem was finally resolved when UCAR exerted immense and consistent effort to follow up with the provinces. The 2016 audit was qualified because auditors were not able to access records of disbursements under the PROSAP-FAO16 agreement totaling US$6.3 million. The Bank review noted that qualified opinions were also issued in previous years for the same situation. There were also some internal control weaknesses. According to the ICR (para 2.4.6.) "the Bank noted that none of the auditors’ qualifications raised accountability issues." UCAR was asked to inform the Bank within 30 days of measures to resolve the issues raised. The ICR was completed before the final audit. Procurement. Internal controls were weak and, this was compounded by uneven monitoring and updating of the Procurement Plan to reflect progress. UCAR's Procurement Unit was initially slow on the newly-installed Procurement Plan Execution System. Procurement also suffered from weaknesses in the management, consolidation and systematization of information related to execution of the Procurement Plan, especially in consultant management. Procurement was also hindered by repeated turnover of the Bank procurement specialist with resulting differences of approach, focus, and relationship with UCAR, and slow Bank responses on no objections. Procurement performance improved when the Bank agreed with UCAR on collaborative resolution of issues and implementing several action plans over time. c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Unsatisfactory Unsatisfactory Uncertainties in relation to Risk to Development several project investments Modest Substantial Outcome and environmental/climate risk. Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Borrower Performance Moderately Moderately --- Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) Unsatisfactory Unsatisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR included six lessons. The following three Lessons are emphasized with some adaptation of language: • Operations mixing different scales and levels of intervention need clear justification and conceptualization, realistic estimates of costs and benefits including their distribution, and special attention to the Theory of Change and Results Framework. Establishing M&E systems and practices starts at effectiveness and requires close Bank supervision and collaboration with the coordination unit. Early, high quality baseline development is of paramount importance. Formal restructuring of an unsuitable RF should not necessarily wait for the Mid-term Review. The chain of objectives, the activities capturing those goals and precisely which farmers are being targeted need to be embedded in a comprehensive set of appropriate, measurable indicators. Large infrastructure investments need an individual Theory of Change connected to the master Results Framework (RF). Classes of indicators are needed to measure technology transfer and institutional strengthening as a function of a system’s global efficiency. • To ensure proper targeting, the identity and characteristics of intended beneficiaries and the distribution of benefits should be analyzed, understood and documented in the PAD. Even in demand- driven projects, it is essential that the targeted beneficiaries be defined, quantified and described based on official, current household and geographic data, and that targeting be carefully designed and operationalized. Where national or regional databases are thin/non-existent or outdated, projects should consider financing their preparation. Also, projects can generate attractive rates of return, but the distribution of benefits needs close scrutiny. Future operations need to consider putting a cap on per beneficiary investment cost across investment instruments (which was done for the ANR), with variations based on the definition/segmentation of the targeted population of producers. • Institutional capacity building at sub-national levels needs customization to the unique institutional, legal and administrative situations in each case. This implies significant ex ante analysis and decisions about project expectations and design rooted in a realistic assessment of the legal and administrative context and incentives. Projects should avoid overly-standardized training and technical assistance of uneven relevance/utility, adopt both demand and supply-driven approaches and aim for durable change. Also, the constitution of water user consortia is not always appropriate or desirable and should not be an absolute requirement. Project design should not confuse institutional frameworks as fixed by law, and design decisions about institution-strengthening; the state’s legal and customary role needs to be understood. Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR PROSAP2 2nd Prov Agric Dev (P106684) 14. Assessment Recommended? Yes Please explain Many of the sub-projects were completed late in the implementation period. It is plausible to assume that additional operational and productive cycles are likely to boost outcomes longer-term due to operational maturity and additional sub-projects consolidating/coming onstream. An assessment would allow verification of results on the ground and gauging the actual impact of project activities. 15. Comments on Quality of ICR The ICR is well written. It provided thorough coverage of project activities and candidly reported on shortcomings. The ICR provided a good discussion on outcomes despite weaknesses in M&E and limited evidence. It also included sound justification for most of the assigned ratings. Annexes were detailed and included a wealth of information on the different aspects of the project. However, the ICR could have included explicit statements for compliance with the Bank's safeguard policies. a. Quality of ICR Rating Substantial Page 19 of 19