Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) Report Number: ICRR0022713 1. Project Data Project ID Project Name P129401 Guiyang Rural Roads Project Country Practice Area(Lead) China Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-83370 30-Jun-2019 142,369,820.42 Bank Approval Date Closing Date (Actual) 06-Mar-2014 31-Dec-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 150,000,000.00 0.00 Revised Commitment 142,369,820.42 0.00 Actual 142,369,820.42 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Elisabeth Goller Victoria Alexeeva IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 5) and in the Project Appraisal Document (PAD, page 3) was: "To provide improved transport accessibility in selected areas of Guiyang in a sustainable manner ". Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components There were four components (PAD, pages 4 -5). 1. Upgrading/Improving County and Township Roads. The estimated cost at appraisal was US$92.6 million. The Bank task team clarified that this did not include the costs for project preparation and management, land acquisition, resettlement costs, and contingencies. 50% of the cost of this component was to be Bank-funded and the balance by the recipient. The actual cost was US$68.4 million, also excluding the costs for project preparation and management, land acquisition and resettlement costs. This component aimed to provide all-season roads in the selected rural areas of the Guiyang municipality. Activities in this component included: (i) upgrading a county road from Class IV to Class II road; and (ii) improving unclassified county and township roads to class IV roads. At appraisal, it was planned that about 570 kilometer (km) of rural roads would be upgraded/improved under the project (ICR, page confirmed by the project team), which would be selected and implemented by five batches on a rolling basis. At appraisal, the design identified a first batch of about 250 km of roads. With the Level 2 restructuring (discussed in section 2e), (i) two road sections under this Batch (15.1 km out of about 250 km) were removed from the project, due to implementation delays and contract disputes; (ii) some originally planned roads were no longer within the jurisdiction of Guiyang municipality after a new Gui'an district was established as a new development zone in Guizhou Province (two roads of n25.9 km); and (iii) while the project originally planned to rehabilitate five batches, the scope was reduced to three batches, due to implementation delays. With these changes, the total km of roads that were to be rehabilitated under the project was 520 km (as compared to the original target of 570). 2. Rehabilitation of County and Township Roads. The estimated cost at appraisal was US$56.5 million. The Bank task team clarified that this did not include the costs for project preparation and management, land acquisition, resettlement and contingency costs. 65% of the cost of this component was to be financed through Bank funding and the balance by the recipient. The actual cost was US$85.9 million (also excluding the costs for project preparation and management, land acquisition and resettlement costs. This component aimed to rehabilitate 67.5 km of the deteriorated county and township roads with improved safety and drainage features. At appraisal, the project envisioned rehabilitating 67.5 km of the county and township roads, as compared to the costlier option of reconstructing roads. The ICR (paragraph 14) notes that during implementation, some project roads were used to transport equipment and materials for the nearby highway and railway constructions. This caused substantial damages to the project roads. These roads needed to be reconstructed with works on the sub-base structure. This necessitated additional design work and increased costs. 3. Rural Road Maintenance Pilot. The estimated cost at appraisal was US$12.0 million. This component was to be financed entirely through Bank financing. The actual cost was US$0.7 million. This component aimed at enabling the Guiyang Municipal Bureau to improve its Rural Roads Maintenance Program. This program had three parts: (i) developing the road maintenance pilot aimed at providing Technical Assistance Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) (TA) for analyzing options addressing five specific issues (maintenance practices, overloading control, monitoring and surveying, funding for road maintenance, and options for pavement maintenance); (ii) implementing the pilot; and (iii) improving the Guiyang Rural Roads Management Information System (GRRMIS) (a computer-based rural road information and maintenance decision support system for data on road conditions). The GRRMIS activity was replaced with the Rural Roads Monitoring and Management System (RRMMS), a computer based rural road information and maintenance decision support system. The RRMMS was integrated with the Guiyang Digital Municipal Management Information System (GDMMIS) - a digitalized public asset and public services database - developed by the Guiyang Municipal Government, with its own budget and outside the project. According to the clarifications provided by the team, this system, integrated with the GDMMIS, has the same functionality as the GRRMIS, but it is more efficient and sustainable because it can be maintained without external consultants. Because of this, Bank financing for this activity was reduced rather substantially (from US$12.0 million at appraisal to US$0.9 million), and the funds were reallocated for reconstructing additional roads. Technical Assistance (TA). The estimated cost at appraisal was US$1.0 million. This component was entirely through Bank financing. The actual cost was US$0.5 million. This component planned to finance two TA programs for the longer-term sustainability of the rural roads program. Activities in this component included: (i) developing a rural road network plan; (ii) financing training and study tours. The rural road network plan activity was dropped as an important element of the plan - the sharing of the rural road network Geographic Information Map (GIM) with third parties - was restricted by local regulations. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The estimated cost at appraisal was US$250.0 million (inclusive of physical and price contingencies). The actual cost was US$222.9 million. Project financing. The project was financed by an IBRD loan of US$150.0 million. US$142.4 million was disbursed. Borrower contribution. The borrower contribution was estimated at US$100.0 million at appraisal. Their actual contribution was US$80.5 million. Dates. The project was approved on March 6, 2014, became effective on August 19, 2014, and was scheduled to close on June 30, 2019. The project closed a year and half behind schedule on December 31, 2020. Other changes. The following changes were made through the Level 2 restructuring on June 14, 2019.  The closing date was extended to December 31, 2020, due to the initial delays for factors such as, disruptions caused by the construction activities of an expressway and a high-speed rail adjacent to project roads, design changes, and counterpart funding issues in some counties and districts.  Loan proceeds were reallocated among project components and disbursement categories.  The Bank financing was increased to 100 percent for components one and two (civil works) activities. Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401)  The results framework was modified and some activities (including the road maintenance pilot activity and technical assistance for analyzing options for addressing road sector issues (discussed in section 2d) were dropped. 3. Relevance of Objectives Rationale Country and sector context. Although the Chinese economy grew at an average rate of over eight percent before appraisal in 2014, the growth was uneven with wide disparities between urban and rural areas. Guizhou, a mountainous and landlocked province in Southwestern China, is the poorest province in China. A third of the population in the province (4.32 million) lived in Guiyang municipality, and a third of the municipality's population (4.32 million) lived in rural areas. The rural road network in Guizhou had deteriorated due to usage beyond the expected service life, overloading, inadequate road maintenance, and overall weak capacity for maintaining the roads. This hampered the rural population's access to final destinations. This project addressed two interrelated challenges facing Guizhou. First, it aimed to improve the transport accessibility of Guizhou's rural population (with accessibility defined for the purpose of this review, as ease of access to final destinations such as markets, educational and health centers, and work opportunities). Second, it aimed at sustained access of the rural population to final destinations. Government and Province strategy. China's 12th Five Year Plan (FYP) for 2011 to 2015 articulated the need for improving transport accessibility in the lagging regions. At the provincial level, Guiyang municipality's Rural Roads component of the 12th FYP aimed to provide all the counties, township and administrative levels, with asphalt or cement concrete roads, improve the capacity for disaster mitigation and road safety, and improve service delivery through upgrading/ rehabilitating county and township roads. The ICR provides no information on the current Government and Provincial strategies. Bank strategy. The PDO was consistent with the strategic theme two of the Country Partnership Strategy (CPS) for 2013-2016. This theme highlighted the need for "promoting inclusive growth". The PDO was well- aligned with the Bank's Country Partnership Framework (CPF) for 2020-2025. The third area of engagement of this framework underscored the need for sharing the benefits of growth, by increasing access to quality and social services in rural areas. Prior Bank experience. This was the second Bank-financed project in Guiyang municipality. The first Guiyang Transport Project was completed in December 2013. Unlike the prior project, which included both urban and rural roads, this project focused on rural roads, to overcome the challenges associated with a project having urban and rural components. The design of this project included a pilot for rural roads maintenance planning, which could be replicated across the whole province. Even if part of the PDO is not challenging for a country like China, the project does not simply repeat the previous project. The PDO is in line with the current Bank strategy for China and it addressed a development challenge facing Guizhou, of improving transport accessibility in selected areas of Guiyang in a sustainable manner. Although the project did not simply repeat the previous project, part of the PDO of improving access in rural Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) areas did not focus on development outcomes and the key outcome indicators were closer to the output than the outcome level. Therefore, this review assesses the relevance of the PDO as substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To provide improved transport accessibility in selected areas of Guiyang in a sustainable manner. Rationale Theory of Change. The causal links between project activities, outputs and outcomes were logical and the intended outcomes were monitorable. Upgrading county roads from Class IV to Class II roads, improving unclassified county roads to Class IV roads, and rehabilitating/reconstructing county and township roads with improved safety and drainage features, were aimed at increasing the percentage of rural population with access to all-weather roads, and thereby their access to final destinations. Developing and implementing the maintenance pilot and Guiyang Municipality's Rural Road Management Information System, and technical assistance activities on the rural network plan, were aimed at contributing to the longer-term sustainability of the rural road network. The key outcome indicators - the number of beneficiaries benefitting from all weather- access roads, and the travel time savings on the rehabilitated roads were measurable, time-bound and attributable to the project activities, but they were closer to the output than outcome level. The theory of change explicitly assumes that the government would provide timely and sufficient counterpart funding. Outputs. (ICR, pages 5 - 6 and 18 - 19).  520 km of rural roads were either upgraded/improved/rehabilitated or reconstructed under the project, as per the revised target (short of the original target of 570 km). This included: (i) 121.8 km of roads that were either upgraded to class IV standards with adjusted alignment, asphalt concrete pavements, and safety and drainage facilities: and (ii) 398.2 km of roads were either rehabilitated/ reconstructed with new pavement and strengthened road furniture. The ICR (paragraph 23) notes that these roads before the project were frequently impassable during rainy seasons.  In addition to the roads mentioned above, about 30 km of the branch/feeder roads were improved upon requests from the local government and people, to extend the benefits to communities. These roads connected the project roads to nearby business/social centers and villages.  A RRMMS was developed as envisaged at project restructuring. It was integrated with the GDMMIS. This system uses independent monitors to gather data on road conditions, and enables the authorities to determine the need for road maintenance on selected road segments.  A study on the rural road maintenance pilot design was carried out. However, the pilot program to test and verify methods of rural road maintenance that could improve cost efficiency and efficacy of the Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) rural road maintenance program, was not implemented as targeted because after its design, there was only one year left until the original closing date.  Two overseas study tours were conducted (one to Vietnam in 2017 on quality contract and community-based maintenance schemes, and one to the United Kingdom in 2019 on road construction and maintenance).  157 people of the relevant rural road agencies were trained, exceeding the target of 125.  The other road maintenance activities such as providing Technical Assistance for analyzing options addressing five specific issues (such as overloading controls, monitoring and surveying, funding for road maintenance. and options for road maintenance) were not completed as targeted. Outcomes (ICR, pages 5 - 6 and 17 - 18). The project activities were expected to decrease travel time on the project roads and increase the number of people with improved accessibility to final destinations.  The roads rehabilitated or reconstructed under the project provided all-weather access to populations and enabled vehicles to travel at 30 - 50 km/hour, compared to 10 - 30 km/hour on the original roads. The travel time on the project roads was reduced by at least 35%, exceeding the target of 30%.  According to the annual surveys by the Guiyang Project Management Office (GPMO), 817,027 people (including transporters and residents nearby the project roads) benefitted from improved accessibility to final destinations. This exceeded the target of 778,027 people. The ICR however provides no details on the methodology followed in administering the surveys and how the intended beneficiaries for the surveys were identified.  Under the pilot operation of the RRMMS, the condition of the road networks was monitored using independent monitors on 259 km of rural roads in Xifeng county. 90% of feedback from independent monitors of the RRMMS was addressed by the municipality through road maintenance activities, exceeding the target of 80% (this indicator was defined as the percentage of responses to requests for routine road maintenance from independent monitors, that were addressed within ten business days). According to the information provided subsequently by the Bank team, the monitors collected and uploaded road condition in the system, such as through pictures and location of damages, by using mobile phone services. Based on the evaluation of the damages, the rural road management unit has been carrying out the necessary maintenance. (The Bank team also mentioned to IEG that Guiyang is being allocated road maintenance allowances on a regular basis to maintain the damaged roads (RMB 15,000 per km for county roads, and RMB 5,000 for rural roads). The team clarified that these allowances are the minimum subsidies provided by the Provincial Government, and that these allowances are in addition to the municipal budget for routine road maintenance and other relevant activities in funding road maintenance. For example, if the data collected by the RRMMIS revealed that road damages were caused by a natural disaster, the road maintenance works could be funded by a separate budget system for disaster recovery, and that if the damage requires capital intensive work beyond routine maintenance, appropriate budget can be used. The team also clarified that the Transport Commission is now able to recognize patterns of road damages due to heavier traffic that what they were designed for. Such information accumulated over time, will help in identifying road sections that need to be designed to higher standards. With the exception of the original target regarding the km of roads to be intervened, all defined targets were met. The ICR provides reasonable evidence that the project improved transport accessibility in selected areas of Guiyang. The information subsequently provided by the Bank team indicates that appropriate measures Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) are being used for the sustainability of the roads rehabilitated under this project (through the capacity to timely assess road conditions, decide on appropriate treatments, and provisions for maintenance of rural roads rehabilitated under this project). Therefore, IEG considers the achievement of the PDO as substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The ICR provides reasonable evidence that the project improved transport accessibility in selected areas of Guiyang. The clarifications provided by the Bank team provides reasonable evidence that the accessibility gains obtained under the project will be sustained. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis. A cost-benefit analysis was conducted for activities associated with upgrading/reconstructing/rehabilitating county and township roads. These activities accounted for 60% of the appraisal estimate, and 69% of the actual project cost. The project benefits were assumed to come from: (1) lower Vehicle Operating Costs (VOCs) due to better road conditions; and (2) savings in passenger travel time due to faster travelling speed on the rehabilitated roads. The ex post Economic Internal Rate of Return (EIRR) was 17.3%, as compared to the ex ante EIRR of 22.6%. Although the ex post EIRR was higher than the discount rate of 12%, it was lower than the ex ante EIRR, due to the higher road works cost. The ex post EIRR was subjected to sensitivity analysis to test different scenarios of costs and benefits. The results of the analysis indicated the project to be economically viable under the tested scenarios. Administrative and operational issues. Implementation was subject to delays in the initial years due to a combination of factors, including disruptions caused by the construction activities of an expressway and a high- speed rail adjacent to project roads, design changes and counterpart funding issues in some counties and districts. The cost estimates for Batch 1 roads had to be adjusted upward during the early stage of implementation, due to insufficient survey work during preparation, and price escalations in materials and labor costs. The delays in the latter years were beyond the control of the project such as, unfavorable weather conditions and restrictions on movement of people due to the COVID-19 pandemic. These delays contributed to reducing the project scope. (Only 520 km of roads as compared to the original target of 570 km were carried improved and rehabilitated). Some institutional strengthening and technical assistance activities were dropped. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) The actual total project cost was lower than the appraisal estimate (10.8 percent less), which reflects the reduced project scope together with cost savings for certain activities, such as construction supervision. Given the lower EIRR than expected at appraisal and the administrative and operational shortcomings during implementation, efficiency is assessed as modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 60.00 Appraisal  22.60  Not Applicable 69.00 ICR Estimate  17.30  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the PDO to the Bank and the Government strategies is assessed as substantial. Efficacy of the single objective - To provide improved transport accessibility in selected areas of Guiyang in a sustainable manner - is assessed overall as substantial. Efficiency is assessed as modest, given that the ex post EIRR was lower than the ex ante EIRR, the project had implementation delays and it financed a reduced scope of activities. Taking these ratings into account, outcome is assessed as moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Government commitment to funding road maintenance activities. The ICR (paragraph 68) notes that local governments in Guiyang municipality routinely experience shortage of funds. Most funds for rural road development are used for construction and rehabilitation, leaving road maintenance activities largely underfunded. The ICR (paragraph 68) notes that while this project was implemented, counterpart funding from some countries and districts was not readily available for funding of road maintenance. The Bank team clarified to IEG that this happened during project implementation and provided reasonable evidence that the roads rehabilitated under the project are likely to be sustained (see efficacy section) There is also a risk that Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) a long lockdown due to the COVID-19 pandemic could delay regular road maintenance activities and hence cause road damage, but this risk is not very likely to happen. Institutional risk. Although the Rural Roads Monitoring and Management System (RRMMS) system was designed to assist the Guiyang Municipal Transport Bureau (GMTB) to monitor road conditions and support decision-making on road maintenance practices, the pilot for testing road maintenance was not implemented. The other institutional risks include risks due to lack of prioritization, unclear responsibilities for maintenance, and inadequate capacity to plan and use advanced road asset management system. There is substantial risk that the long-term sustainability of the project could be undermined without significant institutional strengthening of the GMTB. 8. Assessment of Bank Performance a. Quality-at-Entry This project was prepared based on the experience of the prior Bank-financed project (Guiyang Transport Project). Lessons from the project incorporated at design included: (i) having a single implementing agency as opposed to two agencies; (ii) focusing on rural roads to overcome the challenges when a project has urban and rural components; (iii) tools for rural road maintenance planning; and (v) limiting the geographical scope of the project to one municipality (PAD, paragraphs 30 -36). The implementation arrangements were appropriate, with the dedicated office that was managing the ongoing transport project - the Guiyang Project Management Office (GPMO) - responsible for day-to-day project management (PAD, paragraphs 37 and 38). Several risks were identified at appraisal, including substantial risks associated with the weak capacity of the implementation agency and risks associated with counterpart funding. Mitigation measures incorporated at design included technical assistance and training for the staff of the GPMO, Guiyang Rural Road Construction Office (GRRCO) and the participating counties, and a guarantee commitment letter from the Guiyang Municipal Government to reduce delays associated with counterpart funding. With mitigation measures, the overall project risk was rated moderate at appraisal (PAD, paragraph 9). The mitigation measures adopted at design for addressing the risks associated with counterpart funding proved to be initially inadequate, as insufficient counterpart funding was an issue in some counties and districts. However, this issue was overcome through strict enforcement of the mitigation mechanism (see section 12 below). The arrangements made at appraisal for safeguards and fiduciary compliance were appropriate (discussed in section 9 and 10). There were shortcomings in M&E design (discussed on section 9). Quality-at-Entry Rating Satisfactory Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) b. Quality of supervision Supervision missions were held twice a year. These missions included inspection of the road work sites. In the wake of the local travel restrictions due to the COVID-19 pandemic, the Bank's task team carried out a virtual implementation support mission in April 2020 using available video and audio communication tools. The ICR (paragraph 44) notes that Guizhou Province remained a low risk throughout the pandemic, and civil works and other activities resumed during summer of 2020. This enabled the last Bank supervision mission in November 2020 to assess the completion of the remaining activities. During implementation, the supervision team provided advice on the design and implementation of the institutional strengthening programs. The supervision team appropriately handled the change in the scope of the road maintenance information database and decision support activities. This aided in integrating the RRMMS developed under the project with the municipal information system GDMMIS that covered the broad scope of public assets and services, and helping in ensuring sustainability of outcomes. The support provided by the team aided in safeguards and fiduciary compliance (discussed in section 10). The ICR provides no details on the continuity of Bank task team leadership during the project lifetime. The Mid-Term Review (MTR) was held as scheduled on April 17, 2017. The review noted the delays in the early years of the project and made recommendations for course correction. The Bank supervision team did, however, not correct the shortcomings in the M&E design through indicators for assessing the sustainability of the accessibility gains. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The Guiyang Municipal Transport Bureau (GMTB) was responsible for providing the results data. GMTB had in place the resources and procedures for providing the data with the systems developed under the prior Bank-financed project (PAD, paragraph 45). The results framework was logical and the M&E framework was clear. The key outcome indicators - the number of beneficiaries benefitting from all- weather access roads, and the travel time savings on the rehabilitated roads were measurable, time bound and attributable to the project activities. However, there were shortcomings in the M&E design: the project lacked adequate indicators to measure the sustainability aspect of the PDO. The PDO indicator measuring the length of roads served by the maintenance pilot activity was an output indicator. It does not provide robust evidence on the impact of this pilot on the sustainability of the project roads. The indicator that measured addressing the feedback from independent monitors in terms of maintenance was also weak because it does not show if it actually contributed to maintaining the project roads and how sustainable it is (that is, whether it is meant to last Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) beyond the lifetime of the project). A more appropriate indicator would have been to monitor the maintenance pilot activities that were expanded at a larger scale or the budget allocated in the municipality for road maintenance activities during the lifetime of the project. The project could have provided more details in terms of measurement methodologies. For instance, it is not clear if travel time savings were measured based on the actual time savings of the beneficiaries, or the theoretical savings based on the difference between the ex ante and ex-post travel design speed of the road multiplied by its length. The methodology for measurement was also not clear for the indicator associated with the road length impacted by the maintenance pilot activity. Further, the ICR does not provide details on the methodology followed in administering the beneficiary surveys. It only states that the people living near the roads rehabilitated under the project benefitted from project activities (implicitly assuming that everybody who lived near the road benefitted from project activities). A more appropriate way to capture the beneficiaries would have been to identify who actually were able to use the rehabilitated or improved road. This could have been done on a sample basis, or the project could have indicated an indicator to compare the time taken for the beneficiaries final destination in the intervened areas vis-a-vis the time taken on a representative sample of the non project-intervened areas. There is another area for improvement in future projects. All baselines had a value of zero. This does not provide any indication about the "without the project" situation of the counterfactual. For instance, knowing the number of people with access to an all-weather road at appraisal and using the information as baseline would have provided a better understanding of the overall contribution of the project to enhance access in the province. Similarly, the currently actual travel time as baselines would have helped to better appreciate the extent of the time savings obtained by the project. b. M&E Implementation When the project was restructured, the results indicators were not appropriately revised to reflect the actual needs. For example, no appropriate indicators were added to monitor the sustainability aspect of the PDO. The ICR (paragraph 51) notes that the project implementation data collected every quarter were processed and incorporated in the semi-annual project progress reports. The ICR (paragraph 52) notes that external monitoring was conducted by an independent consulting entity c. M&E Utilization The monitoring reports were used to monitor progress and to take actions to guide implementation (such as adjusting loan allocation and extend the project closing date). In sum, monitoring and evaluation is assessed as modest, given the lack of adequate indicators in the original design to measure that the accessibility gains will be sustainable, which were not rectified during implementation. Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as a Category B (partial assessment) project under the World Bank safeguard policies. Three safeguard policies were triggered at appraisal: Environmental Assessment (OP/BP 4.01): Indigenous Peoples (OP/BP 4.10): and Involuntary Resettlement (OP/BP 4.12). No other safeguard policies were triggered during implementation. Environmental Assessment. The PAD (paragraph 74) notes that the main environmental impacts of the project could include soil erosion, dust and noise, and social disturbance during construction. Due to the rolling selection process employed in the project, an Environmental Code of Practices (ECOP) was prepared for the overall project at appraisal. The ICR (paragraph 58) notes that the overall environmental performance of the project was satisfactory. Minor issues identified on some individual sub-projects, relating to improper waste management on-site and inadequate safety signage. These were rectified. The ICR notes that monitoring of environmental performance during implementation was through an external environmental monitoring consultant. Involuntary Resettlement. A Resettlement Policy Framework (RPF) was prepared and publicly-disclosed to address resettlement issues (PAD, paragraph 68). The PAD (paragraph 69) notes that a Grievance Redress Mechanism was established at appraisal to address three types of grievances: (i) complaints about the compensation standards and payment process: (ii) requests for support on livelihood restoration: and (iii) inconvenience of road transportation during the construction period. There was compliance with safeguards on involuntary resettlement. 1, 757.71 mu of land were acquired and 37,313.33 m2 of houses were demolished. This affected 3,323 households economically and 148 households physically. The actual resettlement cost was Renminbi (RMB) 161.23, 25%, more than the estimates of RMB 129.31 million. For the land acquired, cash compensation and livelihood restoration measures were implemented as set out in the RAP. Where house demolition was involved, the households were provided with options of cash compensation, property swap and replacement of houses. All affected households selected the option of cash compensation. A third party was involved to appraise the market value of the houses. The households affected by house demolition were also provided with relocation subsidies. Indigenous Peoples. The PAD (paragraph 65) noted that several ethnic minority groups resided in the project area (including Miao and Buyi communities). A Social Assessment was carried out during project preparation, with a focus on ethnic minority groups. An Ethnic Minority Development Plan (EMDP) was developed and publicly-disclosed at appraisal to address issues related to minority groups. The ICR does not report of issues with indigenous peoples. The ICR (page 30) notes that dedicated measures were taken to ensure that ethnic minorities benefitted from the project, such as identifying roads Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) in areas where they lived, extensive consultation during design with ethnic minority groups, and providing jobs to ethnic people. b. Fiduciary Compliance Financial Management. A financial management assessment of the GPMO was conducted at appraisal. The financial risk included the risk that counterpart funding may not be available in a timely fashion. The mitigation measures included an arrangement whereby the participating governments were required to place the share of counterpart funding in a designated account to be managed by the implementing agency. With mitigation measures, financial management risk was rated as moderate at appraisal (PAD, paragraph 31). The mitigation measure pertaining to the counterpart funding was not adequate, as there were issues of inadequate counterpart funding in some counties and districts during implementation. The ICR (paragraph 61) notes that the project accounting and financial report preparation and submission were in line with the regulations issued by the Ministry of Finance, which were consistent with the requirements in the Loan and Project Agreements. The audits were unqualified. Procurement. A procurement assessment GRRCO was conducted at appraisal. The GRRCO staff had executed the prior Bank-financed project. The major procurement risk identified at appraisal, was the risk associated with GRRCO's unfamiliarity with specific Bank procedures. The mitigation measures incorporated at design, included employing a qualified procurement agent with experience in Bank procurement policies, and training and capacity building of the GRRCO staff (PAD, paragraphs 62 and 63). The ICR (paragraph 62) notes that the procurement of the project was carried out in accordance with Bank Guidelines. The ICR (paragraph 63) notes that while most of the contracts were implemented according to the agreed schedules, a few contracts were seriously delayed and experienced disputes around contract variations. GPMO took actions to terminate the non-performing contracts and rebid four contracts for the remaining works. The rebidding processes were satisfactorily implemented. The ICR does not report any case of mis procurement. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) Bank Performance Satisfactory Satisfactory There were shortcomings in Quality of M&E Substantial Modest M&E design and in M&E implementation Quality of ICR --- Substantial 12. Lessons The ICR draws the following two lessons from the experience of implementing this project, with some adaptation of language. 1. An adaptive design and implementation scheme can be helpful in preparing rural roads projects. The adaptive approach used in this project consisted of identifying and preparing for only one of the five batches of roads at appraisal. The subsequent batches were to be identified during project implementation based on a set of pre-defined criteria. This approach aided in speeding up project preparation time and improved the readiness for implementation. 2. A counterpart funding assurance mechanism can be an effective tool to avoid a funding shortage and resultant implementation delays, but it needs to be enforced. The mechanism employed in this project required the participating local governments to submit a fund guarantee letter together with their annual project proposals and deposit their funding upfront in a designated project account. After weak enforcement of this mechanism in the initial years, which led to delays in some civil works, the mechanism was effective, and the availability of counterpart funding functioned as a criteria for identifying subsequent batches. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written. The theory of change presented in the ICR clearly expounds the logical causal links between the project activities, their outputs and the intended outcomes. The ICR presents a candid description of the issues surrounding inadequate counterpart funding in some counties and districts despite measures taken at appraisal to get a guarantee from the municipalities, and how these were eventually resolved due to the counterpart funding mechanism used in this project. The ICR draws reasonably good lessons from the specific experience of implementing this project, and the lessons drawn are linked to the narrative. The ICR is concise and tightly presented, with the main body of the text adhering to the usually recommended length of about 15 pages. There were some shortcomings. The quality of analysis particularly with respect to the indicators for monitoring project performance is rather weak, and the ICR would have benefitted for better interrogation of the evidence. The ICR provides very little information on sustainability of outcomes and the arrangements that were made for Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Guiyang Rural Roads Project (P129401) providing maintenance of the rural roads rehabilitated under this project. This information was however provided subsequently by the Bank team. The ICR could have provided more information on supervision, such as continuity of leadership, the efforts made by the team to encourage the timely implementation of the road maintenance-related activities, and whether the supervision team was based in the field or at headquarters. a. Quality of ICR Rating Substantial Page 15 of 15