Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) Report Number: ICRR0022194 1. Project Data Project ID Project Name P120198 AR Norte Grande Road Infrastructure Country Practice Area(Lead) Argentina Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79910 30-Jun-2016 359,338,440.21 Bank Approval Date Closing Date (Actual) 20-Dec-2010 31-Dec-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 400,000,000.00 0.00 Revised Commitment 369,774,566.00 0.00 Actual 358,833,575.94 0.00 Prepared by Reviewed by ICR Review Coordinator Group Kavita Mathur Peter Nigel Freeman Victoria Alexeeva IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (page 5) was "to reduce transport costs for users of provincial roads along selected corridors of the provinces of the Norte Grande region, through the improvement of the quality of those roads and the introduction of improved road asset management tools and methods". Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) There is one minor difference in the wording of the PDO between the Loan Agreement and the Project Appraisal Document (PAD para 12), the Loan Agreement refers to the “Norte Grande region”, while the PAD calls it the “Norte Grande provinces”. The geographical scope is the same. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project comprised two components (cost at appraisal includes physical and price contingencies, and taxes): Component 1: Rehabilitation, Upgrading and Reconstruction of Provincial Roads (cost at appraisal US$465 million (IBRD financing US$365 million), actual cost US$ 345.1 million). This component planned to finance the rehabilitation, upgrading and reconstruction of selected roads (“sub- projects”) under the jurisdiction of Participating Provinces in the Norte Grande region. The civil works included base reconstruction, resurfacing, paving and/or asphalt concrete overlaying (as the case may be), as well as associated works such as shoulder, sidewalks, drainage, masonry, signage, lighting and road safety improvements. Component 2: Institutional Development and Project Management (cost at appraisal US$34 million (IBRD financing US$34 million), actual cost US$13.9 million). (a) Institutional Development: This sub-component was to finance measures to strengthen the institutional capacity of the Provincial Road Directorate (DPV) to improve their investment planning and management capabilities through: (a) the acquisition of road data collection equipment, computers and software, traffic equipment, and related training; (b) building or updating of DPV websites, (c) the setting up of GIS-based transport observatories; (d) preparation of multi-annual road maintenance plans and annual performance reports; (e) preparation of revised design standards documents; (f) technical assistance to review institutional framework, management and financing of provincial roads, current maintenance practices, and recommendations for their improvement; (g) provision of non-consulting services for data collection; and (h) technical assistance and training to provincial staff on social, environmental, and fiduciary aspects of project implementation. The Project would also fund a case-by-case tailor-made strengthening program for each participating DPV to support the preparation and implementation of comprehensive and efficient road management strategies and multi-year road maintenance programs. At a minimum, the goal was that at the end of the Project, 5 provinces would have the following: (i) a Road Asset Management Systems in place (with road inventory, condition and traffic data collected and entered into the system; as well as the use of economic decision models like Highway Development and Management Model - 4 (HDM-4) and Roads Economic Decision Model (RED); (ii) preparation of strategic multi-annual road investment plans; and (iii) preparation of road safety plans. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) [The HDM models simulate lifecycle conditions and costs and provide economic decision criteria for road design and maintenance alternatives. This is the conventional approach for comparing the costs and benefits of road subprojects with the “do-something” project scenario of road rehabilitation, reconstruction or paving works compared against a continuation of the status-quo current maintenance regime “without project scenario”. RED performs an economic evaluation of road investments and maintenance options customized to the characteristics of low-volume roads.] (b) Project Management and Supervision: This sub-component would support and strengthen the operational, social and environmental supervision capacities of the Executing Unit within Coordinating Unit for Programs and Projects with External Financing (UCPyPFE) and Participating Provinces to carry out their functions and responsibilities under the Project (including Project audits) through, inter alia, the provision of tailored technical assistance and training in the preparation of potential road and institutional strengthening investments, including the preparation of related feasibility studies, project designs, economic impact studies, social and environmental impact assessments, and sector strategic studies. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The actual total project cost was US$424.7 million, substantially lower than the appraisal estimate of US$500.0 million. This was due to: (a) lower disbursements under component 2 (Institutional Development and Project Management), about 40% was disbursed; and (b) adverse macro-economic situation in Argentina, which resulted in government budget constraints in 2014-2015 and subsequently in 2018-2019 (see explanation below under financing). Financing. At appraisal, the IBRD commitment was US$400 million and US$30.2 million was cancelled at the time of the fourth restructuring in June 2019, making the loan amount US$369.8 million. According to the ICR, US$12.6 million was cancelled from Component 1 and US$ 17.6 million was cancelled from Component 2. The ICR explains (para 63) that the counterpart funding was not made available on time, as a result of government budget constraints in 2014 and 2015. This created delays in the ongoing civil works and the awarding of new contracts as payments could not be maintained. The project experienced similar restrictions in 2018 and 2019 when the budget allocations did not meet the project financing needs, in particular for Component 2. Following this, the project implementing agency redefined its priorities and needs for budget resources, which slowed down decision-making and negatively affected the implementation of all activities under Component 2 and led to cancellation of funds. The actual loan disbursed was US$358.8 million. The difference between the revised loan amount and actual disbursement was due to exchange rate fluctuations between the US dollar and the Argentinian pesos. Co-financing. There was no co-financing. Borrower contribution: The actual borrower contribution was US$65.9 million, substantially lower than the appraisal commitment of US$100.0 million. The project team explained that the project was implemented during periods of difficult macroeconomic circumstances in Argentina. As a result of government budget constraints in 2014 and 2015, counterpart funding was not made available on time. The project experienced similar restrictions in 2018 and 2019 when the budget allocations did not meet the project financing needs, in particular for Component 2. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) Restructurings. The project was restructured four times: i. September 1, 2014, the results framework was revised after the mid-term review. ii. May 19, 2016, the closing date was extended from June 30, 2016 to May 31, 2018. iii. April 12, 2016, the closing date was further extended to June 30, 2019. iv. June 27, 2019, the results framework was revised, closing date was extended by six months to December 31, 2019, and the undisbursed amount US$ 30.2 million was cancelled. Dates. The project was approved on December 20, 2010 and became effective on December 23, 2011. The mid-term review was carried out in March 2017. The project closed on December 31, 2019, after a delay of 3.5 years after the original closing date of June 30, 2016. The closing date extension approved through the second restructuring in 2016 was needed because, following the change in government in 2015, implementation faced major delays, and the project stopped disbursing. The new government made institutional changes that led to a transfer of the implementation responsibility for the project from UCPyPFE to National Road Directorate (DNV). This led to implementation delays due to the change of signatories and the need to properly staff the DNV team. The other two extensions were provided through (i) the third restructuring approved by the Regional Vice President (RVP) on April 12, 2018 providing a 13-month extension from May 31, 2018 to June 30, 2019; and (ii) the fourth restructuring approved by the RVP on June 27, 2019 that included a six-month closing date extension until December 31, 2019. This was a result of the transfer of the implementation responsibility for the project from UCPyPFE to DNV (as mentioned above) and the PIU became fully functional in April 2018. The other reason was the budget constraints which happened again in 2018 and 2019 when the budget allocations did not meet the project financing needs. The project-implementing agency had to redefine its priorities and needs for budget resources, which slowed down decision-making and negatively affected the implementation of all activities under Component 2 and led to cancellation of funds. 3. Relevance of Objectives Rationale Country Context The Norte Grande region was the poorest region in Argentina and was one of its most marginalized and underdeveloped region in the country. The region included nine provinces covering one third of the national territory and comprised about 7.5 million people or 21% of the population. About 48% of the population was poor and 24.6% lived in extreme poverty. The region produced only 10% of GDP and 8% of the country’s total exports. Transport Context The Provincial Road Directorates (DPVs) in the Norte Grande region lacked capacity for strategically administering the road network and effectively addressing road safety issues. The Norte Grande region had a significantly lower ratio of paved provincial roads per 1,000 inhabitants, around 1.15 compared to 2.36 in Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) other parts of the country. At appraisal, the provincial roads were in exceptionally poor condition, with an average International Roughness index (IRI) of 8.6. The expansion of regional production and exports had exerted strains on the logistics and transport infrastructure. These became evident through increased congestion and reduced transport efficiency, translating into higher transportation costs for firms that were remote from the main agricultural commodity processing centers and major export destinations. A World Bank report in 2006 found that the average logistic costs for companies in the Norte Grande region were 50% higher than those for the other regions in the country. Alignment with Borrower and Bank Strategies The project objective was aligned with the Government of Argentina’s (GoA) Strategic Plan for Territorial Development 2010-2016, which set out two main objectives for its investment agenda: (i) consolidating market forces for the country’s inclusion in the global economy and (ii) mitigating isolation asymmetries in the development of disadvantaged regions and communities to ensure more equitable and sustainable growth for the country, particularly by bridging infrastructure deficits in poorer provinces. The PDOs was fully aligned with the Country Partnership Framework (CPF) for Argentina for the period FY15 -18 and contributed to the results area "reduction in the cost of freight transport in selected corridors of Northern Argentina by 20 percent (expressed in cost per ton per km)” under the first objective "Fostering Private Investment and Strengthening its Enabling Environment". The PDO was substantially relevant to the current Country Partnership Framework (CPF) for Argentina for the period FY19-22, which mentions telecommunications and transport/logistics as key areas of support through improvements to business regulations and enabling services (CPF para 10). The Logistics’ Performance Index (LPI) shows a declining performance by Argentina on infrastructure, and the 2019 Global Competitiveness Report shows Argentina was 92 out of 141 countries in quality of road infrastructure. The PDO contributed to the second objective "Foster stronger market institutions, productivity-led growth and increased exports" by improving transport/logistics performance, which is essential to fully realize the potential for commodity exports (i.e. grain, extractives) (CPF para 56). The CPF also stated that “closing basic infrastructure service gaps will demand new institutional delivery models including investing in connective infrastructure and strengthening local capacity will be key for convergence of living standards and for linking populations to economic opportunities (ICR page 13). The PDO was also relevant to the current CPF’s Focus Area 2, “Addressing Key Institutional Constraints for Better Governance and Service Delivery” and its Objective 4, “Strengthening transparency and reducing corruption” and indicator 4.2 "Implement anti-corruption risk-based warning system in the transport sector". The introduction of improved road asset management tools and methods would introduce a data and evidence-based decision-making process in the road sector, which would provide greater transparency regarding civil works costs, budgeting and execution. There was a clear alignment between the PDO and the country and the Bank strategies. However, the objective was pitched at a level that did not adequately reflect a sufficient solution to the development problem (i.e., the transport bottleneck in the region). The development objective "to reduce transport costs for users of provincial roads along selected corridors of the Norte Grande provinces" would contribute to the high level goal of improving the integration and competitiveness of the Norte Grande Region. However, the PDO did not capture "improved accessibility" for the poor population living in the provinces to increase Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) integration with the rest of the Argentinian economy. The team during implementation monitored the improved accessibility. The relevance is rated substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Reduce transport costs for users of provincial roads along selected corridors of the Norte Grande provinces, through the improvement of the quality of those roads and the introduction of improved road asset management tools and methods. Rationale Theory of Change The project’s theory of change linked the inputs such as rehabilitating and upgrading of provincial roads, and institutional activities for the provincial roads department, to outputs such as (a) improved roads quality; (b) development of road asset management systems in targeted provinces; (c) preparation of strategic multi- annual road investment plans; and (d) preparation of road safety plans. All these outputs were expected to result in the outcome relating to reduced transport costs along the selected corridors of the Norte Grande provinces. The long-term outcomes were increased competitiveness of the Norte Grande Provinces, improved integration of the Norte Grande regions to the wider Argentinian economy, and improved governance in the roads sector. The key assumptions underlying the theory of change were (a) the provinces had the capacity to prepare the proposals for the project roads and had the technical capacity to supervise the implementation of civil works; (b) the provinces were interested in utilizing the institutional strengthening support provided under the project; and (c) the availability of road maintenance funding. However, the assumption (b) did not materialize as during implementation the inherent weak capacity of the provinces made it difficult for them to decide on priorities and fully utilize the institutional strengthening activities provided by the project. Outputs relating to the improvements in the quality of roads.  The project rehabilitated 277 km of roads in the provinces of Corrientes, Tucumán, Santiago del Estero, and Chaco and upgraded 141 km in the provinces of Chaco, Tucumán and Catamarca (a total of 418 km, against the original target of 495 km and revised target of 400 km). Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198)  The gravel roads to indigenous communities near the provincial roads were upgraded to paved standards: RP 5 (31 km) from Santiago del Estero; RP 3 (5 km) from Chaco and RP 18 (7 km) from Santiago del Estero (no targets were set at appraisal). Outcomes With the rehabilitation and upgrading carried out by the project resources, the condition of the provincial roads improved. At project close, the international roughness index (IRI) ranged between 1.3 and 2.15, which indicates “good road condition”. For the rehabilitated roads, the IRI fell from a weighted average baseline of 5.86 to 1.67, exceeding the revised target of 2.6. For the upgraded roads (which were generally dirt roads before the project), the average roughness improved from 8.86 to 1.94, exceeding the revised target of 2.3. The transport cost was substantially reduced. For light vehicles, transport costs were reduced by 29%, exceeding the target of 26%; for omni-buses by 27.6%, slightly lower than the target of 31%; and for heavy vehicles by 29.3%, achieving the target of 29%. For the Norte Grande Region, the project improved the share of rural population with access to all weather roads (measured as persons living 2 km from a route all year) from 24% to 34%, but falling short of the target of 39%. For the five provinces, the share of rural population with access to all weather roads increased from 11% to 31% in Chaco, 10% to 41% in Tucuman, and 50% to 77% in Catamarca (no targets were set at appraisal) (ICR, para 16). The road user satisfaction surveys showed satisfaction rates between 77% and 97%, considerably above the target of 51%. The surveys focused on accessibility to health centers and educational facilities, production output perceptions, and accessibility during the year (ICR, para 33). Outputs and outcomes relating to the introduction of improved road asset management tools and methods. Under the sub-objective “introduction of improved road asset management tools and methods”, the project’s goal was that by project closure the five provinces would have: (i) a Road Asset Management Systems in place and (ii) strategic multi-annual road investment plans; and (iii) road safety plans. There were significant delays in the launch of the institutional strengthening component activities and the ICR notes that most of these activities ended up being implemented during 2018 and 2019. Delays stemmed from (a) the governmental transition following the 2015 elections; (b) insufficient interest and commitment from the provinces; and (c) the inability of UCPyPFE to collaborate more actively with the provinces on technical matters related to road asset management, since it lacked technical knowledge to be able to guide the provinces in institutional strengthening. It was only in 2015 that the first diagnostics for Chaco, Tucuman and Santiago del Estero were completed, allowing the identification of institutional strengthening activities under the project. The other two diagnostics were not completed (ICR para 63). Project implementation took nine years (compared with the originally projected 5.5 years), to accommodate the late start of the institutional strengthening activities and to allow the completion of all civil works. The intermediate outcome indicator “Annual performance reports indicating results achieved by the Provincial Road Directorate's (DPV) participating in the institutional development component prepared and publicly disseminated” was dropped because at MTR it was realized that this indicator would not be achieved given the lack of progress in the implementation of the institutional strengthening activities. This indicator was linked Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) to one of the three basic framework measures (“preparation of strategic multi-annual road investment plans, with yearly updates”) that were targeted to be taken in five Participating Provinces. (a) As targeted, road database management systems were put in place in all 5 provinces (Chaco, Jujuy, Salta, Tucumán, Santiago del Estero). At project close, all five provinces had a functioning road database for their entire network which measured the key road characteristics (roughness, deflections, traffic, vehicle structure) and presented it on a GIS platform, which meant it had a geo-location reference. This allowed the road authorities to have a precise knowledge about the condition of the road network at each location and to develop their maintenance and investment plans according to the conditions, thereby enabling better targeted interventions and better efficiency of spending. This has enhanced the capacity of DPVs to analyze overall network investment and maintenance needs and to develop multi-annual plans. To further enhance the capacity of DPVs, the project financed the software for cost analysis to help DPVs track and compare costs (for materials and work hours) between different contracts (ICR para 37). This allowed them to carry out more precise planning, better trace contract costs, and increase procurement transparency. The project assisted in the development of the standards, manuals and technical specifications for road works that can be applied by all provinces in Argentina. To test material for quality control, the project funded laboratories in all nine provinces. The project funded a Heavy Vehicle Simulator to understand future road deterioration under different vehicle loads. The simulator is managed by the National Road Directorate (DNV) but available for use by all provincial DPVs. (b) The project supported the preparation of Multi-annual Road Maintenance Plans for 5 provinces (Chaco, Jujuy, Salta, Tucumán, Santiago del Estero) as targeted. Chaco was the only province that succeeded in preparing on time the Multi-annual Road Maintenance Plan 2014- 2018 and at project close was preparing a new multi-annual plan for the period until 2024. All other provinces prepared their plans in April 2020. The project team explained that the Norte Grande provinces which participated in the project are among the poorest provinces and suffer from capacity constraints, making it difficult for them to decide on priorities and coordinate between each other in institutional strengthening activities. Chaco was ambitious and other provinces learnt from Chaco’s success. The implementation of Chaco’s plan resulted in an increase in the share of provincial network roads in good condition from 30% in 2014 to 63% in 2018 and a 7% decrease in road maintenance’s share in its budget . This points to the use of data-based planning through the Multi-Annual Plan, which resulted in a better average quality of the road network and therefore requires simpler and less expensive maintenance. (c) The project financed Road Safety Plans in the targeted provinces of Salta, Santiago del Estero, Tucumán and Jujuy. Each plan included: (i) a description of the situation of the targeted province in road safety terms, comparing it to the national level; (ii) an in-depth assessment of a pilot section identified by each DPV; and (iii) the proposal of a series of short-term much needed interventions for each of the pilot sections, along with conceptual-phase documentation. The team explained that where possible, road safety interventions were integrated in the civil works financed under the project. Rating Substantial Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project’s physical investments in the selected provincial roads in the Norte Grande region reduced transport costs for users. By introducing improved road asset management tools, skills, and methods in the participating provinces, the project laid an essential foundation for reducing transport costs over the medium to long term through more cost-effective investment programming and maintenance. However, there are some challenges. The delayed implementation of the institutional strengthening activities prevented the DPVs from having enough time during project implementation to experience the use of the new systems, carry out operational adjustments to ensure their practicality, and fully benefit from project support. There is a substantial risk that the institutional strengthening outputs will not be mainstreamed in the DPVs (ICR para 95) because no formal budget arrangements were made for their subsequent adoption. Overall Efficacy Rating Substantial 5. Efficiency Economic Analysis At appraisal, an economic analysis was undertaken for roads sub-projects in the provinces of Chaco, Tucuman and Corrientes. This covered a total of 212.85 km of roads, of which 57% was for rehabilitation and reconstruction works and 43% for road paving works. The analysis used three scenarios and the benefits included reduction in vehicle operating costs and travel time for road users, as well as the structural soundness of the roads for a longer period of time: Scenario 1, used the HDM-4 model, which simulate lifecycle conditions and costs and provide economic decision criteria for road design and maintenance alternatives (traditional cost benefit analysis). The Economic Rate of Return (ERR) was 21% and the Net Present Value (NPV) was US$56.7 million. Scenario 2, used Scenario 1 plus incorporated land appreciation benefits. The ERR was 28.7% and the NPV was US$87.9 million. Scenario 3, used Scenario 2 plus other exogenous benefits (tourism, and education principally) for the Provincial Road (RP) No.3 in Chaco. The ERR was 29% and the NPV was US$90.1 million. A sensitivity analysis was also conducted taking into account higher construction investment costs and reduced benefits due to lower than assumed traffic. The ERRs were: 24.2% if costs increased costs by 20%; 23.2% if benefits reduced by 20%; and 19.3% for combined scenario of costs increasing by 20% and benefits were reduced by 20%. An additional economic analysis was undertaken to measure the impact of road infrastructure enhancement on regional and local economies. The multiplier effect of the road investments to local production sectors and local Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) imports and exports was 1.42. The three roads identified at appraisal were also expected to have positive impact on regional employment: around 8000 employment opportunities, out of which approximately 5,600 were direct and 2,300 were indirect. At completion, the ERR was 20%, slightly lower than the appraisal estimate of 21%; the NPV was US$136.5 million. The team explained that this was due to the following: (a) at appraisal, only three routes were included in the analysis as technical details were available only for these sections, while the ex-post included more kilometers; (b) the increase in the cost of road works; and (c) the annual traffic growth rate was more than double the initial estimate. Administrative Efficiency The project implementation period was extended by 3.5 years. The ICR reports (para 48) that in the case of the road construction, the delays did not have a considerable negative impact. However, for the institutional strengthening component, the delays prevented the Provincial Roads Departments (except for Chaco) -- from utilizing the new asset management tools during the project’s life span (see section 4 above). Therefore, provinces could not identify additional activities to be included in the project and thus further increase the value for money from the project. The ICR (para 74) mentions that additional environmental assessments and measures were carried out for works only after work contracts had already been in implementation. This happened when modifications were made to rehabilitation/upgrading works, and was often related to sections where works intersected with water bodies or involved the use of water resources (see additional details in section 10a). This led to cost and time increases for the civil works and affected completion of Route 94 in Corrientes, Route 304 in Tucuman and Route 46 in Catamarca. On balance, the project efficiency is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  21.00  Not Applicable 0 ICR Estimate  20.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) 6. Outcome The relevance of objectives was substantial. The project’s physical investments in the selected provincial roads in the Norte Grande region reduced transport costs for users. By introducing improved road asset management tools, skills, and methods in the participating Provinces, the project laid an essential foundation for reducing transport costs over the medium to long term through more cost-effective investment programming and maintenance. The project efficiency was modest due to substantial delays and cost overruns. The overall outcome is rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The main risk to development outcome are:  Financial Risk (inadequate funding for maintenance). Although, the provinces provided the Federal Government a commitment guarantee for routine maintenance through the Participation, Transfer and Maintenance Agreements (PTA), and by project closure this commitment had been met, the risk for continued maintenance is assessed moderate as DPVs road budget transfers from the federal government in 2018 were just enough to cover the minimum necessary routine maintenance of the paved road network in the provinces (ICR para 94).  Institutional Risk. The delayed implementation of the institutional strengthening activities prevented the DPVs from having enough time during project implementation to experience the use of the new systems, carry out operational adjustments to ensure their practicality, and fully benefit from project support. There is a substantial risk that the institutional strengthening outputs will not be mainstreamed in the DPVs (ICR para 95) because no formal budget arrangements were made for their subsequent adoption. 8. Assessment of Bank Performance a. Quality-at-Entry The project design was based on the lessons from the then ongoing Provincial Road Infrastructure Project that experienced significant delays due to cumbersome approval processes at the provincial level due to a subsidiary loan agreement; so this project adopted a centralized loan structure. During preparation, the project team consulted with local communities and NGOs in Chaco and Santiago del Estero to understand the needs of the communities and adjusted road designs based on their Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) feedback. The ICR notes (para 59) that this was particularly evident in the preparation of the Indigenous Peoples Plan. The risk identification and mitigation measures were not adequate for the institutional aspects. The project identified the risk of insufficient implementation capacity at the provincial level regarding environment and social management. This risk was to be mitigated through training early on during the implementation period. Despite this, the risk materialized for some provinces and had to be managed by intensive supervision (see section 8b). The project design assumed that the provinces were interested in utilizing the institutional strengthening support provided under the project but this was not the case. A detailed menu of the institutional strengthening activities was based on the diagnostic work carried out by the Bank and parallel work carried out by the Inter-American Development Bank (IADB). However, this required collaboration across several institutions and proved complex to implement. Notwithstanding the project team explained that the program implementation depended on the ambition and capacity of different DPVs in each province. The Norte Grande provinces that participated in the project were among the poorest provinces and suffered from capacity constraints, making it difficult for them to decide on priorities and coordinate between each other on institutional strengthening. Another risk that was not identified was the weak technical capacity of the UCPyPFE to guide the provinces in selecting the institutional strengthening activities. The project's M&E design could have included a more comprehensive set of intermediate indicators to monitor the gradual progress on institutional strengthening. The quality of entry is rated moderately satisfactory due to weaknesses in risk identification and mitigation that adversely affected project implementation and sustainability of the institutional strengthening outcomes. The project design did not provide for adequate monitoring of the institutional component and the budget for this component was over-dimensioned given the capacity of the Provincial Road Directorates. The ICR (para 61) states that while the financing allocated to Component 2 might had been commensurate with the wide institutional strengthening needs of the Participating Provinces, experience from earlier projects or the then ongoing Provincial Roads Project should have signaled that the expectations were unrealistic. Also, there were no budgetary mechanisms to ensure the O&M for the new systems financed under the institutional development component. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Note: this section in the ICR is missing. The team found that the section was missing from the version that was sent to the Board. The Review is based on the text, which was originally part of the ICR reviewed and cleared in the process. The Bank team included the relevant expertise and had continuity, as the Bank Task Team Leader based in the country did not change during the project and maintained regular supervision and collaboration with the client. During supervision, the project team initiated collaborations with other Bank projects and utilized Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) these to expand the knowledge of the DPVs and DNV, especially in the aspects of environment management. The project had a positive impact on gender, specifically on women from the indigenous group Qom who live in Chaco province. Learning from the community consultations, the road design in Chaco included specific investments to build two community artisan centers along Route 3 to be managed by the Qom women, offering a space for them to meet, become economically active, and receive training. The centers are being used to promote women’s artisan products, allowing the women a means to earn an income and take advantage of the new opportunities created by the improved connectivity and increased transit and tourist visits. The ICR reports (para 91) that the Bank team started many initiatives and partnerships with other Bank teams and outside organizations on road management, environmental sustainability, and women’s empowerment. The ICR reports (para 74) that Bank specialists adequately addressed all mitigation measures relating to biodiversity, natural habitat protection, and forests. However, there were cases where provinces missed conducting reassessments of environmental impacts. This happened when modifications were made to rehabilitation/upgrading works, and was often related to sections where works intersected with water bodies or involved the use of water resources. The provinces did not identify environmentally acceptable technical solutions in advance. The Bank specialists identified these instances and assisted the provinces in updating the Environment Management Plans and in implementing the mitigation measures. Project implementation lasted 9 years against the originally projected 5.5 years. The project should have been restructured when it became apparent that there was insufficient counterpart funds for component 1 and insufficient commitment for the institutional reform. Overall, the quality of supervision is moderately satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design To measure the PDO "to reduce transport costs through the improvement of the quality of roads", the project M&E design included adequate outcome indicator to measure the reduction in transport costs as well as intermediate indicators such as roughness index to measure roads conditions and user’s perception surveys to measure the quality of service. Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) It also included output indicators such as (a) number of km of provincial (non-rural) roads rehabilitated, upgraded or reconstructed, (b) annual average daily traffic (AADT) on targeted roads, and (c) share of rural population that has access to an all-weather road in the Norte Grande region (core indicator). The M&E design for the institutional component included a PDO indicator "number of provinces in Norte Grande with strategic multi-annual road maintenance plans and road database management systems in place". The project's M&E design could have benefited from a more comprehensive set of intermediate indicators to monitor the gradual progress on institutional strengthening. b. M&E Implementation The ICR reports (para 69) that the data was regularly collected during implementation. The M&E was fine-tuned at the mid-term review to reflect more closely construction costs. As a result, the target for output indicator "roads rehabilitated" was reduced from 495 km to 400 km due to unit cost escalations. Baselines were also revised since at appraisal they were set based on a sample of the first group of identified projects (which included Chaco Route 3, Tucuman Route 304 and Corrientes Route 94). By the mid-term review more precise data was available as Chaco Route 3 and Santiago del Estero Route 5 had been completed, while Chaco Route 9, Tucuman Route 304 and Santiago del Estero Route 18 were undergoing works. The intermediate outcome indicator “Annual performance reports indicating results achieved by the Provincial Road Directorate's (DPV) participating in the institutional development component prepared and publicly disseminated” (yes/no) was dropped because it was realized at mid-term review that this indicator would not be achieved given the lack of progress in implementation of the institutional strengthening activities. This indicator was linked to one of the three basic framework measures (“preparation of strategic multi-annual road investment plans, with yearly updates”) that were targeted to be taken in five Participating Provinces. Instead, the project began to track one of the other basic framework measure -- “preparation of road safety plans by Participating Provinces” – against four provinces. Therefore, the indicator was replaced by a new indicator, “Road Safety Plans developed in targeted provinces” (baseline: 0; target: 4) (ICR para 18). c. M&E Utilization The ICR reports (para 71) that the project monitoring reports helped showcase to the Provincial Road Directorate (DPV) and the national authorities the usefulness of outcome-based results monitoring in road transport. The methodologies used for the data collection (road condition surveys, HDM-4 calculations, traffic surveys) were all of relevance to the DPVs for their operations, and thus the M&E implementation provided a hands-on experience to the DPVs. The data collection also helped in the institutional strengthening, as it reinforced the need to approach road management from an efficiency point of view (i.e., reduction in time and cost) rather than just output (i.e., length of km rehabilitated). The data collection and analysis methodologies are now an integral part of the DPV's road asset management systems, as reported by the ICR. Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) M&E Quality Rating Substantial 10. Other Issues a. Safeguards At appraisal The project was classified as Category B (partial assessment) and triggered the following safeguard policies: Environmental Assessment (OP/BP 4.01), Natural Habitats (OP/BP 4.04), Forests (OP/BP 4.36), Pest Management (OP 4.09), Physical Cultural Resources (OP/BP 4.11), Indigenous Peoples (OP/BP 4.10), and Involuntary Resettlement (OP/BP 4.12). The PAD (para 45) noted that the primary environmental impacts of road rehabilitation, upgrading and reconstruction were associated with noise, vibration, dust, and traffic disruption during the construction phase and haulage and final disposal of materials from the construction sites. Aspects such as impacts on natural habitats, native forests, drainage constrains and cultural properties, as well as social issues were considered as part of the Environmental Impact Assessment (EIA) process and specific measures were included in the Environmental Management Plan (EMP) (PAD para 46). The environmental assessments and management plans were prepared by the Provincial Roads Departments (DPVs) and disclosed by UCPyPFE for the three pre-selected roads (RP3 in Chaco, RP94 in Corrientes and RP304 in Tucuman) in April, 2010. Public consultations were carried out between March and August 2010 (PAD para 48). A framework for environmental and social assessment (ESMF) was completed. It provided guidelines and methodologies to assess future sub-projects or changes in the sub- project. The Bank’s Indigenous Peoples safeguards was triggered for Provincial Road No. 3 in the Chaco Province. For sub-projects to be identified during implementation, an Indigenous Peoples Planning Framework (IPPF) was developed as part of the ESMF. Although, roads rehabilitation and upgrading works were to be carried out mainly within the existing right-of- way of well-established roads, major resettlement was not expected. Still a Resettlement Policy Framework (RPF) was developed as part of the ESMF. At completion The ICR reports (para 74) that Bank specialists adequately addressed all mitigation measures relating to biodiversity, natural habitat protection, and forests. However, there were cases where provinces missed conducting reassessments of environmental impacts. This happened when modifications were made to rehabilitation/upgrading works, and were often related to sections where works intersected with water bodies or involved the use of water resources. The provinces did not identify environmentally acceptable technical solutions in advance. The Bank specialists identified these instances and assisted the provinces in updating the EMPs and in implementing the mitigation measures. The project introduced Norte Grande Provinces to new methods and tools to identify natural habitats and to mitigate negative impacts on fauna connectivity. For example, during the construction of the Chaco Route Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) 3, a very detailed natural habitat and landscape analysis was conducted in association with a local university, DPV, the GEF Rural Corridors and Biodiversity project (P114294), and was executed by the Argentina’s National Parks Administration. This resulted in the construction of underpasses and overpasses for fauna. The project also adopted design solutions that provided uninterrupted animal passage. A pilot activity to monitor deforestation near Chaco Route 9 was carried out in collaboration with the European Satellite Agency. The ICR reports that OP4.10 (Indigenous Peoples) and OP4.12 (Involuntary Resettlement) were updated during project life as appropriate (para 77). The Indigenous Peoples Plan (IPP) was implemented fully and on time, as all the planned access road improvements under the IPP were completed and the planned community trainings were carried out. The ICR states that the project had a positive impact on women from the indigenous group Qom who live in Chaco province. It ensured citizen engagement in design and integrated women’s empowerment actions in civil works. The implementation experience of the Involuntary Resettlement safeguards is not discussed in the ICR. The team clarified that during implementation there was no resettlement as all investments were on the existing right-of-way. b. Fiduciary Compliance Procurement The ICR reports (para 80) that the procurement was well managed and the procurement reviews did not find any major issues. At appraisal, the procurement risk was assessed as high and all civil works contracts were subjected to prior reviews. During the first procurement rounds, bid prices were high. The Bank team launched an analytical work to review road construction costs in Argentina using the World Bank-developed ROCKS methodology (“Road Works Contracts, Competition, and Efficiency: Adding Transparency to the Road Sector in Argentina”). This helped the implementing agency UCPyPFE to obtain the average and range of unit costs by the type of works. The project also provided training to the UCPyPFE and DNV staff on works cost analysis. This led to the development of a works cost system which is being used by the DNV and the DPVs. The procurement of goods and services for the institutional strengthening component were delayed due to some internal administrative processes in DNV that required all procurement to be carried out following both the national procurement guidelines for DNV as well as the Bank procurement guidelines, thus resulting in a longer procurement timeframe. Financial Management The ICR reports (para 84) that the financial management (FM) was adequate. The FM rating was downgraded to moderately satisfactory in May 2015 because the Bank funds advanced to the Designated Account (DA) had been used to finance the local share of project expenditures due to delays in counterpart funding. Argentina was facing budget constraints in 2014-15 and PIU planned to correct this issue once counterpart funding became available. The financial management team was quick to identify this problem Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) and developed a time-bound action plan with PIU to address this problem. It was upgraded to satisfactory after one year, but the last years of implementation were moderately satisfactory. The ICR reports (para 85) that project complied with the audit requirement except for the last year of project implementation, which was overdue by five months. Annual audit reports were unqualified throughout the life of the project, except for the 2014 report, which qualified the Project Financial and the Statement of Expenses statements because the project used Bank funds to finance counterpart contributions. c. Unintended impacts (Positive or Negative) The collaboration with indigenous communities in Chaco and Santiago del Estero provided opportunities for DPVs to carry out a design preparation process which was informed by the needs of these local communities. For example, in Santiago del Estero this led to the introduction of additional rural road access points to the provincial route, bus stops and safer access to schools, as well as awareness raising activities for the communities about safety in transport, including the wearing of helmets and other aspects of motorcycle safety. In Chaco this led to the production of bilingual road signs showing indigenous communities' historical sites, cultural–ethnic sites of interest, and fauna and flora maps (ICR para 56). d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory For weaknesses in risk identification and mitigation Bank Performance Satisfactory Moderately Satisfactory measures and design of the institutional component as well as shortcomings in supervision. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Lessons drawn from the ICR, adapted by IEG:  Project implementation units in projects that support decentralized levels of government need to include an institutional coordination mechanism to ensure active Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) participation. The implementation of institutional activities suffered during project implementation. A coordinating mechanism (which would ensure regular collaboration with the provincial governments), with a formal role in the implementation structure, could have been a platform for technical discussions and initiatives on institutional strengthening and contributed to a quicker absorption of institutional strengthening resources.  Civil works contracts in local currency need to include mechanisms to protect the public interest and contractors in periods of high inflation or currency devaluation. The currency devaluations affected the retention funds maintained by the investor and were a disincentive for contractors to repair deficiencies during the guarantee period. Similar risk arose regarding the performance guarantees provided by contractors which lose their value in periods of devaluation (averaging a 30% devaluation throughout the loan’s period; the guarantees’ value in local currency fell around 70% in a 2 year contract).  Rural roads projects can significantly contribute to improving women’s empowerment and access to economic opportunities by promoting their participation in income generating activities, strengthening their leadership and life skills. The project experience shows that by promoting women's active participation in the design and implementation of infrastructure projects, these projects can be instrumental in broadening women’s networks through their participation in community and women associations and strengthening their agency. 13. Assessment Recommended? No 14. Comments on Quality of ICR The quality of evidence is satisfactory and the analysis is linked to evidence. The lessons are based on project experience. The discussion on the project's impact on indigenous communities and the preservation of natural habitats is noteworthy. However, there are a few shortcomings: (a) for the institutional development support, the ICR could have been more clear on what activities were not completed and a more in-depth discussion of factors that led to the development and implementation of the Multi-annual Road Maintenance Plans in Chaco and the delays in other four provinces; (b) it did not report on involuntary resettlement safeguards (the team clarified that during implementation there was no resettlement as all investments were on the existing right of way); (c) the main text is 26 pages, longer than the OPCS recommended length of 15 pages; and (d) the quality of supervision section in missing due to a technical issue. a. Quality of ICR Rating Substantial Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Norte Grande Road Infrastructure (P120198) Page 19 of 19