Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) Report Number : ICRR0020600 1. Project Data Project ID Project Name P082026 PY Road Maintenance Country Practice Area(Lead) Paraguay Transport & ICT L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-74060 30-Jun-2012 105,410,000.00 Bank Approval Date Closing Date (Actual) 24-Aug-2006 30-Jun-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 74,000,000.00 0.00 Revised Commitment 74,000,000.00 0.00 Actual 72,000,280.61 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Ridley Nelson Christopher David Nelson IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 5) was: "To establish a sustainable road management strategy that ensures the upgrading and maintenance of the national road network, through, inter alia, the strengthening of the institutional capacity of the Ministry of Public Works and Communications (MOPC), the promotion of efficient planning and adequate resource allocation for the rehabilitation and maintenance of the road network, and the improvement of access of excluded rural communities to the primary paved road network in selected Departments." Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) The PDO as stated in the Project Appraisal Document (PAD, page iv) was: "To establish a sustainable road management strategy that ensures the upgrading and maintenance of the road network through a rational and balanced use of scarce resources." The PAD (page 5) also states that the project's specific objectives were: (i) developing the institutional capacities of the MOPC for strategic planning and management of the road network: (ii) arresting the deterioration of Paraguay's priority road network, composed of international and regional road corridors, and maintaining its level of service as perceived by road users: (iii) improving access for the poorest and most excluded rural communities to the primary paved road network in the Departments of San Pedro, Caaguazú and Caazapá. This assessment is based on the PDO as stated in the Loan Agreement. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components There were three components. Strengthening Strategic Planning and Road Management.(Appraisal estimate US$6.51 million. actual cost at closure US$7.42 million). This component aimed at developing the institutional capacities of the Ministry of Public Works and Communication (MOPC) for managing the road network. Activities included: (i) Strengthening their capacity for strategic planning through improving their legal framework and road classification system, setting up a strategic planning unit, implementing a traffic counting system, strengthening their procurement unit and training their staff: (ii) strengthening their capacity for preparing and executing budget through a monitoring, automated works certification and payment systems: and, (iii) Strengthening their capacity for developing road improvement and maintenance contracts, updating the pavement management and road inventory system and implementing a communication strategy on road maintenance. Improvement and Maintenance of the Paved Road Network. (Appraisal estimate US$43.35 million. actual cost at closure US$73.47 million). This component aimed at arresting the deterioration of the priority road network composed of international and regional corridors through increased use of private sector participation in road maintenance activities through performance-based contracting. Activities in this component had two subcomponents: (i) training the MOPC staff for preparing and monitoring performance- based contracts and rehabilitation works on 968 Kilometers (Km) of the national paved road network on four of the seven identified road segments in the Eastern Region using Performance-based Road Maintenance Contracts (GMANS in Spanish). and, (ii) preparing a Toll system Optimization Plan for improving the availability of financing for maintenance activities on the paved road network and investments to implement the plan on routes under performance-based contracts. Improvement and Maintenance of the Unpaved Road Network. (Appraisal estimate US$42.90 million. actual cost at closure US$26.34 million). This component aimed at the rehabilitation and conservation of the unpaved road network that connect to the national road network and secondary roads connecting rural Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) communities and providing access to the most excluded rural communities in three departments (San Pedro, Caaguazú and Caazapá). Activities in this component included: (i) Rehabilitation and maintenance of the unpaved road network (including specific works such as bridges and drainage): (ii) Capacity building for the MOPC district staff and for the Regional and the local governments on roads maintenance. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Appraisal estimate (including baseline cost, costs associated with physical and price contingencies and the Front-end IBRD fee) was US$105.42 million. Actual cost at closure was US$107.23 million, about 2% higher than the appraisal estimate. Actual cost of component one and two activities was 14% and 69% higher than expected at appraisal and the cost of component three activities was 61% of the appraisal estimate. Cost of component one and two activities was higher than expected at appraisal and the cost of component two activities was lower than estimated due to the reduced scope of this activity. The increase in cost of component one and three activities was covered through reallocation of IBRD loan proceeds from the "unallocated" category of civil works. Project Financing. The project was financed by an IBRD loan of US$74.00 million. Amount disbursed at closure was 97% of the loan at US$72.00 million.There was parallel financing for complementary road rehabilitation activities on the remaining three identified segments in the Eastern Region from the Inter- American Development Bank (IDB) and for activities associated with additional Road Master Plans in the departments from the Mercosur Structural Funds (FOSEM). Borrower Contribution. Appraisal estimate US$31.42 million. Their contribution at closure was 22% more than planned at US$38.51 million. Dates. The project was restructured five times. The first restructuring on June 2012 extended the project closing date by 20 months from June 6th, 2012 to February 28, 2014 for completing project activities which had been delayed due to a slower than expected start-up. The second restructuring on March 2013 reallocated IBRD loan proceeds from the "unallocated" category to civil works. The following changes were made through the third restructuring in January 22, 2014: (i) The project closing date was extended by 16 months from February 28, 2014 to June 30, 2015 for completing ongoing civil works activities which had been subject to delays due to administration changes associated with change in government. and, (ii) The results framework was revised, the Project Development Indicators were modified and new indicators were added for monitoring project performance. The fourth restructuring on June 2015 extended the project closing date by six months from June 30, 2015 to December 31, 2015 in view of the delays associated with completing civil works activities and finalizing social safeguards measures. The fifth restructuring on December 24th 2015, extended the project closing date by six months for finalizing activities associated with construction of communities centers in the three departments as part of the Indigenous Peoples Plan (IPP). The project closed four years behind schedule on June 30, 2016. 3. Relevance of Objectives & Design Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) a. Relevance of Objectives As a landlocked country, Paraguay's external trade depended on the road networks - the main transport mode - for connecting to ports in neighboring countries and export markets in the Mercosur region. In the years before appraisal, although Paraguay's total road networks was comparable to equivalent countries and adequately covered the entire country, the amount of paved roads compared unfavorably with the rest of the region: Paraguay had 0.77 kilometers (Kms)/1000 inhabitants as compared to 3 km/1000 inhabitants in - Argentina and 4 km/1000 inhabitants in Uruguay. Regarding the quality of the paved road network, only 14% was reported to be in good condition, while 56% and 30% were reported to be in either fair or poor condition respectively. The unpaved road network was poor, in view of the little amount invested in maintaining unpaved roads in the preceding years. As a result, many communities did not have all weather access and this contributed to their social exclusion. Alongside this, the Ministry of Public Works and Communication (MOPC) lacked a clear sector policy oriented towards maintenance, suffered from significant institutional weakness, especially those related to planning and operated in an inefficient manner due to human capacity constraints and poorly adopted internal policies. At appraisal, the Project Development Objective (PDOs) was consistent with the government's road sector strategy. The strategy set road maintenance as a priority and set budgetary targets for each road network type. The PDO was consistent with the recently approved National Development Plan (NDP) for the period 2014-2030 which sought to address issues regarding sustainable economic growth and poverty reduction over the medium term. The PDOs were aligned with the Bank Strategy for Paraguay. At appraisal, the PDOs were consistent with the Bank's Country Assistance Strategy (CAS) for the 2004-2008 period. The CAS objectives were closely linked to the Administrations' strategy. Of the four objectives identified in the CAS, the PDOs were consistent with three objectives: (i) the promotion of sustained growth particularly in rural areas where poverty was concentrated: (ii) social inclusion, to improve the coverage and efficiency of basic social services to help Paraguay meet the Millennium Development Goals (MDGs): and (iii) improved governance and transparency in public administration. The PDOs were consistent with the Bank's Country Partnership Framework (CPF) for the 2009-2013 period. The Bank's current CPF for the 2015-2018 period aimed at supporting the National Development Plan in regard to poverty reduction, especially in rural areas. The three pillars of this CPF are: poverty reduction: inclusive economic growth: and, international connectivity. Rating High b. Relevance of Design The statement of the PDO was clear and the causal links between project activities, their outputs and outcomes were logical. Component one activities aimed at the institutional strengthening of the Ministry of Public Works and Communication (MOPW), such as through developing their capacity for strategic planning, preparing and executing budget and developing road maintenance contracts were intended to improve their capacity for managing the road network. Component two activities aimed at rehabilitation and maintenance of the priority road network through performance-based contracting and implementing a Toll Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) System plan were intended to arrest the deterioration of the priority roads network. Component three activities aimed at maintenance of the unpaved road network that connected to national and secondary roads were intended to improve access of the poorest and excluded rural communities in the selected departments. The combination of these activities aimed at contributing to the PDO of establishing a sustainable road management strategy.The intended outcomes by aiding in social inclusion aimed at contributing to the higher level objectives of supporting economic growth and poverty reduction. There were no indicators specifically aimed at tracking resource mobilization for road sector expenditures. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To establish a sustainable road management strategy that ensures the upgrading and maintenance of the national road network, through the strengthening of the institutional capacity of the Ministry of Public Works and Communications (MOPC). Rationale Outputs. • Five laws as compared to the target of two were passed during the project execution phase. These included - the Transit and Road Safety Law in 2014, the Road Classification Law in 2016, the creation of the Road Planning Directorate (DPV) in the Ministry of Public Works and Communication (MOPC) in 2007, the toll revenue ministerial direction and the ministerial directive to create the transparency department (DTPC) in February 2007 (ICR, page 36). • The Road Strategic Planning Unit (DPV) was created and operational at project closure. The staff of DPV were trained as targeted (ICR, page 36). • A results-based budgeting system for the road sector was completed as targeted (ICR, Datasheet, Intermediate Outcome Indicator Number Three). • Communications plan on road maintenance was implemented as targeted and information was available to the public due to the Government and transparency improvement plan (ICR, Datasheet, Intermediate Outcome Indicator Number Four). • Traffic counting system was operational since October 2014 as targeted (ICR, page 36). • A road inventory system was carried out and was in the process of being completed at project closure (ICR, page 37). Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) Outcomes. • Budget expenditure for all programs carried out by the MOPC showed an increasing trend with 89% of the budget executed as compared to 68% at the baseline. This exceeded the target of 75% (ICR, Datasheet, Key Outcome Indicator Two). • The road strategy was elaborated and published as targeted. The five year plan was prepared in 2007 and was implemented at project closure. (ICR, Datasheet, Key Outcome Indicator Four). Given the preparation of the road strategy and implementation of the road plan and increasing trend of budget expenditure, it is clear that the activities significantly contributed to the institutional strengthening of the Ministry and thereby to the PDO of establishing a sustainable road management strategy for the upgrading and maintenance of the national road network. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To establish a sustainable road management strategy that ensures the upgrading and maintenance of the national road network, through the promotion of efficient planning and adequate resource allocation for the rehabilitation and maintenance of the road network. Rationale Outputs. • In addition to the outputs described above which were also relevant for this objective, the following outputs were produced. • A training program for the Ministry of Public Works and Communication (MOPC) staff on preparing and monitoring Performance-based Road Maintenance Contracts (GMANS) was completed as targeted (ICR, page 38). The project financed three out of the four contracts envisaged at appraisal. • 623 kilometers (Km) of paved road network were under GMANS at project closure. This was short of the revised target of 642 km (ICR, Datasheet, Intermediate Indicator Number Five). • Master plan for roads toll collection were completed and modern integral toll collection systems were built and equipped on 11 stations as per the revised target (ICR, Datasheet, Indicator Number Six). • The road monitoring system to measure compliance with the Road Strategy was established as targeted (ICR, Datasheet, Indicator Number Seven). Outcomes. • At project closure, 93% of the roads complied with the maintenance standards as compared to the Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) revised target of 90% (ICR, Datasheet, PDO Indicator No Five). • Activities associated with Performance based Road Maintenance System and the toll system can be expected to contribute to the sustainable road management strategy. However, in assessing the objective, we factor in one of the key elements of the objective which was to achieve sustainable maintenance. In the absence of a secure or committed basis for funding for road maintenance, it is not clear if there would be adequate resource allocation for funding the maintenance of roads rehabilitated under the auspices of this project. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 3 Objective To establish a sustainable road management strategy that ensures the upgrading and maintenance of the national road network through improving access of excluded rural communities to the primary paved road network in selected Departments. Rationale Outputs. • 144 km of the unpaved road network was improved. This exceeded the revised target of 140 km (ICR, Datasheet, Intermediate Indicator Number Eight). The ICR (page 24) notes that the original target of 338 km had to be reduced as the cost of the GMANs contracts was higher than anticipated due to a combination of factors including, delays in the contracting and execution of works, underestimation of drainage requirements and impact of issues connected to poor soil quantity and proneness to flooding. • An Indigenous People's Plan (IPP) was prepared and implemented with three multi-use community centers as targeted. • Six micro-enterprises were in place at project closure for maintenance of unpaved roads. This was short of the target of ten (ICR, Datasheet, Intermediate Indicator Number 10). • The number of Public Transport Vehicles in San Pedro, Caazapá and Caaguazú road segments increased from 34 vehicles per km at the baseline to 42 km at project closure as per the revised target (ICR, Datasheet, Intermediate Indicator Number 13). Outcomes. • 34,100 rural community beneficiaries (defined as the number of beneficiaries within 10 Kilometers (km) had improved access to all weather roads in the three departments at project closure as targeted (ICR, Datasheet, Key Outcome Indicator Number One). There was no baseline for this indicator (ICR, page 21). Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) • The Vehicle Operating Cost (VOC) - a proxy for the condition and quality of the road network - on the three selected departments on the poorest and most excluded rural communities to the primary paved road network - declined from 18,149 PYG in San Pedro, 17,886 in Gaazapa and 18,324 in Gaaguazu in Guarani per vehicle -kilometer (km) at the baseline to PYG 11,483, 11,283 and 11,360 in Guarani per vehicle km respectively at project closure. This was still somewhat higher than the revised target of 10682 in San Pedro, 10497 in Caazapá and 10569 in Caaguza (ICR, Datasheet, Indicator Number 11). • 77% of the beneficiaries based on a sample size of 637 road users in the three departments reported the roads to be in good condition at project closure. This exceeded the revised target of 50%.(ICR, Datasheet, Key Outcome Indicator Number Three). Given the number of excluded rural communities who had access to primary paved road network in the three departments, it is reasonable to conclude that the project made a significant contribution to the PDO of establishing a sustainable road management strategy through improving access of excluded rural communities. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Efficiency Economic Analysis. An economic analysis was conducted at appraisal and at closure using the same methodology for the three components: (i) rehabilitation of paved roads: (ii) improvement of unpaved roads: and, (iii) improvement of bridges. These components accounted for 81% of the cost at appraisal and 93% of the actual project cost. The benefits were assumed to come from savings in vehicle operating costs, travel time savings and savings in road maintenance costs. For paved roads, net benefits were evaluated using the Highway Development and Management Model (HDM-4) and for improvement of unpaved roads, net benefits were evaluated using the Roads Economic Decision Model (RED), which is tailored for economic evaluation of unpaved roads. The ex post Net Present Value (NPV) for rehabilitation of paved roads at 12% discount rate was US$33.82 million as compared to the ex ante NPV of US$23.40 million. The ex post Economic Internal Rate of Return (EIRR) was marginally lower at 42% as compared to the ex ante EIRR of 43%. The ex post NPV for improvement of unpaved roads at 12% discount rate was US$9.10 million as compared to the ex ante NPV of US$14.20 million. The ex post EIRR for unpaved roads was 15% as compared to the ex ante EIRR of 32%. The average ex post EIRR for the project was 21% as compared to the average ex ante EIRR of 33%. The lower ex post EIRR was due to a combination of factors including, higher than estimated capital costs as many road sections required more rehabilitation than had been expected and delays in implementation of works which resulted in the postponement of benefits to be derived from the road improvements. These factors were however partly offset by stronger than forecast traffic growth. Administrative and Operational Issues. There were a number of significant weakness that contributed to Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) the 230% increase in the planned project period, with the project closing over four years behind schedule. These were due to a combination of factors including, delays in project effectiveness with the project declared effective only in January 2008 although targeted for January 2007 due to the length of time taken to secure the necessary approvals and legal authority for the government to commit to the loan, cost overruns associated with Performance-based Roads Maintenance contracts (GMANS) as the contracts were based on initial basic designs and the rate of deterioration of the pavement proved to be greater than anticipated, delays in counterpart funding in the initial years of the project, procurement and implementation delays due to the weak capacity of the implementing agency exacerbated by impact of changes in government administration. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 81.00 Appraisal  33.00 Not Applicable 93.00 ICR Estimate  21.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objective with respect to government strategy and the Bank strategy are rated High. Relevance of design is rated as Substantial as the causal links between project activities, their outputs and their outcomes were logical. The contribution of two of the activities to the objectives - strengthening the institutional capacity of the ministry and improving access to excluded rural communities to the PDO of sustainable road management strategy - was rated Substantial. The contribution of the activity - to promote efficient planning and adequate resource allocation for the rehabilitation and maintenance of the road network - to the PDO was rated as only Modest because it is not clear if there is a secure and committed basis for funding maintenance activities on the rehabilitated roads. Efficiency was rated as Modest in view of the significant administrative and operational issues that contributed to the 230% lengthened project period with the project closing four years behind schedule. a. Outcome Rating Moderately Satisfactory Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) 7. Rationale for Risk to Development Outcome Rating Although the performance based contracting as done in the case of this project has highlighted the importance of road maintenance activities for preserving the road assets, at the macro level it is not clear, if adequate resources would be provided for funding periodic and routine road maintenance activities. This is particularly so given that there is no dedicated funding flows for road maintenance other than revenue from tolls for funding. It is also not clear if funding for continued operation of the three indigenous training centers constructed as part of the indigenous People's Plans (IPP's) would be available, given that the maintenance of these centers falls outside the purview of the road sector (ICR, page 18). a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared based on two main lessons from a prior Bank-financed project (Paraguay: Eighth Highway Project). (i) Unlike the prior project, which focused only on construction and rehabilitation of several highways rather than maintenance of the road network, this project aimed at a more balanced program for road rehabilitation and maintenance. and, (ii) The experience with the prior project which was implemented by the Directorate of Highways (DOH) of the Ministry of Public Works and communication (MOPC) had shown that DOH's capacity for executing the project was weak. This project was to be implemented by a dedicated Project Preparation and Management Unit (UPGP). Given that the government's main objective was recovery of confidence in state institutions through among other things a sustained fight against corruption, at preparation an improved Governance Action Plan was designed in collaboration with the government and the Bank's Department of Institutional Integrity, following global practices in the field. The team provided financial resources in the preparation phase, inter alia through the proceeds of a Japan Policies and Human Resources Development (PHRD) Grant. Appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in section 11). However, there were shortcomings at Quality-at-Entry. There were cost overruns with the costs of Performance-based Road Maintenance Contracts (GMANS) on paved roads higher than anticipated at appraisal, as the rate of deterioration of the pavement was greater than anticipated at appraisal. This increase in cost contributed to the reduction in scope of project activities associated with improvement of unpaved roads. The preparation team overestimated the absorptive capacity of the implementing agency, given that the project envisioned new forms of contracts such as performance-based contracts which were being introduced for the first time in the country context. The weak capacity in conjunction with delays in counterpart funding in the initial years of the project and administration changes during the course of the project, contributed to the large time overruns. There were drawbacks in M&E design (discussed in section 10a). Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Supervision missions were held on average twice a year. with 24 Implementation Supervision Reports (ISRs) filed over 11 years. Despite an extensive number of supervision missions in the early years of the projects, they were not effective in addressing delays in effectiveness or accelerating disbursements in the first three years. Given the delays in the initial years of the project, the supervision team developed an action plan in September 2011 to improve project performance through a specific focus on action steps in four areas such as capacity improvement of the Ministry of Public Works and Communication (MORC), acceleration of the Performance-based Road Maintenance Contracts (GMANS), acceleration of the unpaved road improvements and increasing IBRD disbursements and this improved implementation. Most project activities were largely completed at project closure. Bank supervision ensured fiduciary and safeguards compliance (discussed in section 11). Although indicators were revised, there were no indicators included for specifically tracking resource mobilization for road sector expenditure. There were shortcomings in monitoring of the project till the restructuring of the project which happened about six years after the declaration of effectiveness (discussed in section 10b). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The Government Commitment provided support at preparation. However, there were delays is the preparation of the Government and Transparency Improvement Plan (IGAP) and delays associated with the length of time taken for getting the necessary approvals and legal authority for the government to commit to the loan and this contributed to delays associated with project effectiveness. Although the project became effective in 2008, the government did not provide all the budgetary resources required for the project to be implemented in 2008 and 2009. These shortfalls contributed to delays in the appointment of key staff in the Ministry of Public Works and Communications (MOPC) and implementation of various project activities. This, in combination with changes in government administration following the elections at the end of 2008 which entailed key changes in ministerial and key official appointments, contributed to the delays associated with institutional strengthening activities. In the latter years of the project, the government collaborated with the supervision team to address the causes of delays. By project closure, their contribution by ways of counterpart funding was more than planned at appraisal. Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The Project Preparation and Construction Unit (UPGP) in the Ministry of Public Works and Communications (MOPW) was in charge of implementing the project. There was compliance with fiduciary and safeguards compliance (discussed in section 11). At project closure, the agency completed all major contract works including performance based contract, which were introduced in Paraguay for the first time under the project. There were delays associated with procurement activities on the part of the implementing agency. These delays were more due to to internal regulations rather than capacity constraints and contributed to the delays associated with the commencement of the Performance-based Road Maintenance Contracts (GMANS). Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design Of the six key outcome indicators - three of them, the percentage of the Ministry of Public Works and Communications (MOPC) portfolio executed through results-based management practices, creation of the ministry's Road Strategic Planning Unit and implementation of the road strategy, and level of user satisfaction were appropriate for monitoring project performance. The remaining three indicators - reduction in vehicle operating costs, increase in public passenger transport and reduction in transportation times, were relevant to measuring the expected impact on the project on the disadvantaged (connecting the excluded rural communities with the primary paved road network in the three departments). Several indicators (such as percentage of the Ministry of Public Works and Communication's (MOPCs) portfolio that were executed following results-based management practice, increased level of road user satisfaction, reduction in vehicle Operating Cost and reduction in average transportation times) had neither a baseline nor targets. There were no indicators specifically aimed at tracking resource mobilization for road sector expenditures. For investment components, implementation progress was to be measured through data collected by the Directorate Of Highways (DOH). A planned economic evaluation that was never done could have been useful for objective three associated with improving access of excluded rural communities. Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) b. M&E Implementation Given that some of the original PDO indicators were not relevant, the results framework and the indicators were revised under the January 2014 restructuring. However, no indicators were added aimed specifically at tracking resource mobilization for road sector expenditures. Given that the restructuring took place about six years after the declaration of effectiveness, monitoring of the project up to that point time was not effective, as no information was being collected in regard to the monitoring of the PDO. Although a social economic evaluation was envisioned, the Ministry of MOPC did not have the capacity for developing such an evaluation. (ICR, page 16). c. M&E Utilization The monitoring reports were focused on implementation issues and the indicators were used for monitoring performance of key and intermediate outcome indicators. M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was classified as a Category B project. Other than environmental assessment (OP/BP 4.01) two social safeguards were triggered: Involuntary Resettlement (OP/BP 4.12) and Indigenous Peoples (OP/BP 4.12). The PAD (page 96) notes than an environmental assessment of the project was conducted at appraisal. The assessment concluded that the negative impact of the project was expected to be minimal, given that most of the road rehabilitation and maintenance activities were to be on already existing roads. Although no resettlement of homes and businesses was expected since no new highway construction was envisaged, along the paved highways, there were significant number of structures and a few homes, built on the highway right-of-way, often directly on the highway shoulder to attract passing motorists. A Social Participation Framework (EPS) was set up at appraisal to guarantee a broad participation of stakeholders, creating awareness for road maintenance activities to preserve the road network and increase transparency and accountability during implementation. A Resettlement Framework was conducted at appraisal (PAD, page 110). An indigenous People's Plan (IPP) was prepared at appraisal in collaboration with representatives of the indigenous communities, indigenous associations and non-governmental organizations working with indigenous peoples in departments where the project was to be implemented (in the departments of Caazapá, Caaguazú and San Pedro. (PAD, page 102). The ICR (page 18) reports that with regard to involuntary resettlement, the ICR (page 17) there were delays associated with illegal occupation of the right of way and some land acquisition associated with the construction of the new toll stations and with the unpaved road component. This generated significant dissension among and between the served communities contributing to delays in completion of works. With regard to the implementation of the Indigenous People's Plans (PPIs) there were significant delays Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) associated with the construction of the three indigenous training centers in the three departments and these were completed by mid 2016 (ICR, page 17). b. Fiduciary Compliance Financial Management. The PAD (page 20) reports that an assessment was made at appraisal of the financial management arrangements of the implementing agency. The assessment concluded that the action plan proposed by the implementing agency would meet the Bank's requirements and that the implementing agency had sufficient capacity for addressing financial management issues. The ICR (page 18) reports that despite some delays in submission of reports and some shortcomings in justifying some eligible expenditures in the initial years of the project, these were rectified and the recent audit reports were deemed to be satisfactory (ICR, page 18). Procurement. The PAD (page 20) reports that an in-depth assessment of the Ministry of Public works and Communications (MOPC) capacity to address procurement issues and a detailed action plan was prepared at appraisal to address procurement issues. The ICR (page 18) notes that there were some delays in contracting of works and consulting services due to procedural issues in the initial years. With regard to Performance-based Road Maintenance Contracts (GMANS) for which no standard documents existed at the outset of the project, there were some capacity issues associated with lack of knowledge and experience of relevant Bank policies and procedures. These were rectified through training and the procurement management was deemed to be satisfactory in the final years of the project (ICR, page 18). c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory There is Substantial risk given Risk to Development that there is no secure of Modest Substantial Outcome commited basis of funding road maintenance activities. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) Moderately Moderately Bank Performance --- Satisfactory Satisfactory Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The following were the main lessons from the experience of implementing this project, with some adaptation of language. (1) Performance Based Contracting (PBC) can provide the critical underpinning for the strategy to improve and sustain road maintenance. The experience of this project, which introduced PBC for the first time in Paraguay demonstrated that such contracting can be successfully introduced in low capacity environments with proper planning and addressing of constraints. (2) Sectoral governance and transparency programs can play an important role in strengthening of road planning and management. The initiatives taken through the Government and Transparency Improvement Plan (IGAP) impacted on road management and proved effective in monitoring contracts. (3) Where maintenance funding is critical for sustainability, the project design should incorporate indicators associated with tracking resource mobilization for road sector expenditures. The project in this case looked at measures for improving efficiency and effectiveness of sector institutions to improve funding for road maintenance activities, but specifically included no measures for addressing issues pertaining to the persistent underfunding of maintenance needs by the government. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is concise and well-written. It candidly discusses the problems that were encountered at preparation (such as government delays associated with counterpart funding) that contributed to delays in project effectiveness in the early years of the project and changes in administration which contributed to delays during the implementation phase. It also candidly discusses the issues associated with M&E design. The ICR draws useful lessons from the experience of implementing the project. The ICR provides little by way of details of Bank supervision. It would have been useful to include an English translation of the Borrower's ICR. Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PY Road Maintenance (P082026) a. Quality of ICR Rating Substantial Page 16 of 16