Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) Report Number: ICRR0024193 1. Project Data Project ID Project Name P155522 BRRP Country Practice Area(Lead) India Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-59380 31-Dec-2022 147,972,220.25 Bank Approval Date Closing Date (Actual) 21-Dec-2016 31-Dec-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 235,000,000.00 0.00 Revised Commitment 153,000,000.00 0.00 Actual 147,972,220.25 0.00 Prepared by Reviewed by ICR Review Coordinator Group Marc H. Juhel Fernando Manibog Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The PDO was to improve rural road connectivity in project districts and enhance management of rural roads in Bihar. For this evaluation, the PDO will be broken down into its two sub-objectives: i. PDO 1: improve rural road connectivity in project districts, and Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) ii. PDO 2: enhance management of rural roads in Bihar. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components At appraisal the Project consisted of two components: a. Component 1: Rural Roads Improvement (Allocation at approval: US$320 million; Actual cost at completion: US$211 million). Construction and improvement of the state core network to improve the road connectivity of target habitations under MMGSY (Mukhya Mantri Gram Sampark Yojana), the GoB funded program to provide road connectivity to habitations with population in the 250‐499 range in Project districts. It included: • Sub-Component 1.1: Civil works for rural roads. Civil works for about 2,500 km of rural roads, including standalone bridges. The road improvements would incorporate road safety measures and environmentally optimized and climate resilient road designs. • Sub-Component 1.2: Design, implementation, and management support. Supervision, quality monitoring, and preparation of cost-effective and climate resilient designs. • Sub-Component 1.3: Pilots on innovative bridge construction, retrofitting road safety and climate resilience measures in the existing rural road network (identified under 2.1 and 2.3). b. Component 2: Asset Management and Institutional Effectiveness (Allocation at approval: US$15 million; Actual cost at completion: US$3 million). Support the Rural Works Department (RWD) to implement the Road Sector Modernization Plan (RSMP) through the following: • Sub-Component 2.1: Asset management. Implementation of the Asset Management Plan (AMP) and development of an Asset Management System (AMS); preparing plans for capital works and maintenance and development of a Geographic Information System (GIS) based road inventory and condition database; enhancing and expanding the use of maintenance contracts for 20 percent of the core network; climate vulnerability assessment for roads and bridges and development of mitigation plans; and introduction of improved, low-cost, innovative, environment friendly road and bridge designs to withstand flood related risks. • Sub-Component 2.2: Institutional effectiveness. Implementation of the existing Human Resource Professional Development strategy of RWD, which would include training of RWD staff & contractors; procurement of equipment for RWD labs, offices, and training facilities; digitalizing RWD business Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) processes; development of the long-term rural roads strategy; updating technical guidelines and manuals; and identifying measures to make the rural transport service in Bihar more effective. • Sub-Component 2.3: Road safety management. Training RWD staff, the local police and transport departments, designing road safety measures, undertaking road safety audits of priority roads, raising awareness among contractor staff, and development of the road safety module in AMS. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The actual total project cost was US$213,572,220. The project had five restructurings in its 7-year implementation period with a total approved disbursement for the project at closing of US$147,972,220. The Project was restructured five times, as summarized below: a. Restructuring 1: September 22, 2020 – Cancellation of US$50 million from International Development Association (IDA) financing as part of the overall COVID-19 response strategy to make funds available immediately for pandemic response measures. The remaining funds were expected to be adequate to complete the remaining activities, given the exchange rate gains from the depreciation of the Indian Rupee against the United States Dollar and savings in the cost of civil works. b. Restructuring 2: March 21, 2022 – Cancellation of US$27 million from IDA financing and revisions to the project scope and indicators due to slow progress and low disbursements. The Project faced more than a three-year delay in implementing activities under Component 2 and reported little progress in activities under Sub-Component 1.3. The pandemic, and flash floods which occurred in several districts also contributed to the slow progress in Project activities. The restructuring was carried out to address these issues following the mid-term review (MTR). The scope of Sub-Component 1.1 was reduced from 2,500 km to 2,000 km, as it was estimated that only about 2,000 km of roads could be completed by the original credit closing, and Sub-Component 1.3 was dropped due to lack of sufficient time in the Project to complete the original scope. The activity to develop an AMS was dropped at the request of GoB that the AMS was no longer required as a state-wide rural road maintenance policy was introduced. Consequently, the target “Improved asset management ‐ number of districts that have developed and are using asset management system” was dropped. The removal of the key outcome target of introducing an AMS, which was the only substantial change made to Project outcomes, was replaced by a new target to pilot an Artificial Intelligence (AI) system to monitor performance on at least 100 km of project roads. The corresponding new indicator was defined as “Pilot Artificial Intelligence (AI) based system in districts within the state (number)” with a target of 1. These changes did not have a substantial impact on the original Theory of Change. c. Restructuring 3: December 21, 2022 – Extension of the credit closing date by six months from December 31, 2022, to June 30, 2023, to enable the completion of the remaining works. d. Restructuring 4: June 27, 2023 – Extension of the credit closing date by six months from June 30, 2023, to December 31, 2023, to enable the completion of the remaining works and identification of potential savings for cancellation under the Project. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) e. Restructuring 5: December 26, 2023 – Cancellation of US$5 million of savings from IDA financing. Financing. The World Bank financed US$147,972,220 with IDA credits. Borrower contribution. The Government of India, through the Bihar Rural Works Department, financed US$65,600,000. Dates. The project was approved on December 21, 2016, and became effective on August 31, 2017. The Mid-Term Review took place on January 29, 2021. The original closing date was December 31, 2022, and the actual closing date was December 31, 2023. A split rating is applied in the evaluation of project efficacy due to the reduction in scope and PDO targets related to the road connectivity and asset management under Restructuring 2. 3. Relevance of Objectives Rationale Context at appraisal. At appraisal, Bihar was one of the faster growing low-income states in India, with 34 percent of its population living below the poverty line compared to a national average of 22 percent. About 90 percent of its population of 104 million was rural and about 76 percent of the population in north Bihar was subject to the recurring threat of floods. Bihar’s potential for growth in agriculture and agro‐based industries, tourism, handicrafts, and cottage industries was underutilized due to inadequate road infrastructure and market linkages. About 60 percent of Bihar’s rural road network was unpaved and had several deficiencies, such as missing links, dilapidated bridges or absence of bridges, inadequate geometry, poor drainage, weak pavements, and missing road safety measures. The network had also suffered severe damages due to floods, lack of maintenance, inadequate quality of initial construction, and overloading in some parts of the state. Inefficient sector management was also identified as a key challenge. Relevance to Government Strategies. At appraisal, the Project was aligned with the objective to support the Government of Bihar (GoB) in achieving the Sustainable Development Goals (SDGs): rural roads are proven poverty reducers (Goal 1: Ending Poverty); enable flow of agriculture inputs and farm produce (Goal 2: Food Security and Sustainable Agriculture); allow quicker access to health care and education facilities, specifically for women/girls (Goal 3: Health and well‐being, and Goal 4: Inclusive, Equitable and Quality Education); facilitate mobility of labor and increase employment opportunities (Goal 8: sustainable and inclusive economic growth and employment); and connect remote and inaccessible areas with economic opportunities (Goal 10: Reduce Inequality). As part of GoB’s priority of “Road connectivity to each habitation”, about 60,000 km of rural roads (50 percent of the total rural road network in the state) were constructed in the preceding decade leading to 2016, providing road access to 51 percent of its habitations. Several country and state level road development programs were in place to rehabilitate the remaining road network and provide access to all habitations in the state. The key programs were: (i) Pradhan Mantri Gram Sadak Yojana (PMGSY), the Government of India (GOI) funded program to provide all‐weather road access to all habitations with a population above 500; (ii) Mukhya Mantri Gram Sampark Yojana (MMGSY), the GoB funded program to provide road connectivity to habitations with population in the 250‐499 range, Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) which were not covered under PMGSY; and (iii) Grameen Tola Sampark Nischaya Yojana (GTSNY), to connect habitations with a population below 250. The GoB’s rural roads program was organized around three key priorities: (i) Priority One: All‐weather-road access to the remaining 49 percent habitations, (ii) Priority Two: Preserving existing road assets and upgrading them to acceptable standards, and (iii) Priority Three: Improved institutional effectiveness. At Project completion, the PDO is still aligned with the development priorities of the Government of India (GoI). GoI’s Strategy for New India@75 notes that increasing the coverage and quality of roads and highways is critical to enhance connectivity and internal and external trade. It identifies increasing connectivity by expanding the road network, improving road maintenance and safety, skill development in the road sector, and increasing emphasis on research and development as key priorities for the road sector. Furthermore, MMGSY, which the Project supported, was launched in 2013 by GoB with the aim of providing all-weather connectivity to rural habitations with population more than 250. This program was built around the three key priorities above, which involved not only infrastructure improvement, but also improving institutional efficiencies in the road sector. Relevance to Bank Strategies. At appraisal, the Project was fully aligned with the World Bank Group’s goal of reducing poverty and promoting shared prosperity, and with India’s Country Partnership Strategy 2013‐2017 priority of supporting low‐income states and its three themes: integration, transformation, and inclusion. The project would better integrate the rural population, businesses and industries in Bihar’s remote and poorer districts with the national and state economy through better transport connectivity, as well as through improved integration with the strategic transport corridors passing through Bihar. Improved road access would have a transformational impact on rural poverty through its effect on improved agricultural productivity, higher non‐farm employment opportunities and increased rural wages. It would facilitate better inclusion of the poorer and marginalized communities in the growth process through better access to markets, jobs, growth opportunities, and services. The World Bank had been engaged with PMGSY since its inception, through dialogue, technical assistance (TA), and a series of lending operations. At completion, the PDO is still aligned with the priorities identified in the World Bank’s India Country Partnership Framework (CPF) for FY18 – FY222. The PDO directly contributes to the achievement of Objectives 1.1 and 2.3 under Focus Areas 1 and 2 of the CPF: Objective 1.1 – promote more resource- efficient, inclusive, and diversified growth in the rural sector; and Objective 2.3 – improve connectivity and logistics. Objective 2.3 specifically highlights the Bank’s commitment to support climate and disaster resilient roads. The PDO is also consistent with the Bank’s engagement approach of strengthening public sector institutions through increasing their capability and strengthening core government systems. This objective was pitched at an appropriate level of ambition and is clearly aligned with the Government of India and Bihar's priorities of improving rural connectivity and institutional efficiencies in the road sector, as well as with the World Bank’s strategies. Based on the above rationale, the relevance of the PDOs is rated High. Rating Relevance TBL Rating High Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective PDO 1 - Improve rural road connectivity in project districts - Original Project Rationale The Theory of Change for PDO1 was that the activities to (i) build 2,500 km of rural roads, including standalone bridges, (ii) provide design, implementation and management support, and (iii) implement pilots on innovative bridge construction, retrofitting road safety and climate resilience measures in existing rural road network, would have the following outputs: (i) 2,500 km of rural roads constructed, (ii) cost-effective and climate resilient road design adopted, and (iii) selected bridges constructed using innovative methods, selected roads retrofitted with road safety and climate resilience measures. In terms of outcome, these outputs were to result in increased all-weather road connectivity to habitations. The Theory of Change for PDO2 was that the activities to (i) implement an asset management program through development of an Asset Management System (AMS), (ii) implement maintenance contracts for 20% of the core network, carry out a climate vulnerability assessment, and develop low-cost, innovative environmentally friendly road and bridge designs, (iii) improve institutional effectiveness through implementation of the Human Resources strategy, digitalizing, long-term rural roads strategy, updating technical guidelines, and identifying measures to improve rural transport, and (iv) improve road safety management through training, designing road safety measures, undertaking road safety audits, development of road safety module in AMS, would have the following outputs: (i) AMS operationalized, GIS-based road inventory and condition database developed, maintenance contracts adopted for 20% of the core network, network level climate vulnerability assessment undertaken, (ii) RWD staff trained on latest industry practices, RWD business processes fully digitalized, technical guidelines developed, and (iii) RWD staff trained on road safety practices, and road safety audits conducted on priority roads. In terms of outcome, these outputs were to result in (i) improved road assets management, (ii) improved road safety, (iii) improved effectiveness of project expenditures by using cost-effective measures in road design, and (iv) improved RWD capacity to manage the road network. For both PDOs taken together, the longer-term outcomes would then include (i) better integration of the rural population, business and industries in Bihar’s remote and poorer districts with the national and state economy through better transport connectivity, as well as through improved integration with the strategic transport corridors passing through Bihar, (ii) reduced rural poverty through improved access for better agricultural productivity, higher non-farm employment opportunities, and increased rural wages, (iii) better inclusion of the poorer and marginalized communities through better access to markets, jobs, growth opportunities and services, and (iv) efficient and resilient management of the road network. The critical assumptions were: (i) political commitment to implement institutional reforms under Component 2, and (ii) availability of adequate resources for infrastructure development. Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) Based on this logical sequence, it is indeed plausible that the effective implementation of outputs will lead to the achievement of the desired outcomes, and potentially of the anticipated longer-term outcomes further down the road. Outputs The Project rehabilitated 2200kms of rural roads in Bihar falling short of the target of 2500kms. Project roads delivered with satisfactory quality certified through independent quality reviews was 85% and met the target. All project contracts included provisions for five-year routine maintenance to ensure uninterrupted connectivity. On completion of the five-year routine maintenance period, these roads will then be maintained under the GoB funded performance-based maintenance contracts as per GoB’s Rural Roads Maintenance Policy. Outcomes The project improved the connectivity of 61,476 habitations compared to the target of 62,000. The project directly benefited 838,240 people, falling short of the target of 1,200,000 people. The ICR further provides additional outcomes but without supporting evidence. It notes that public consultations revealed that the roads now provide local farmers improved direct access to markets, which helps them to bypass intermediaries and thereby reduce costs. The improved quality of the road pavements also ensures uninterrupted access to critical social services, such as health care and schools, which was earlier restricted during the rainy season as the roads became either muddy or were damaged. The improved roads have increased school attendance of students (of female students in particular) in some areas and have improved access to healthcare facilities due to the availability of different transport modes (e.g., ambulances, private vehicles). Also, the practice of adding five-year routine maintenance provisions to road rehabilitation contracts, which is supported by a state level policy, will help the long-term sustainability of the road network, as well as reduce the fiscal impact of road rehabilitation costs to the state in the long run. However, it remains worth underscoring that a more systematic approach is required for the prioritization of roads for maintenance, based on road condition and other technical criteria, as opposed to a blanket time- based maintenance policy. Achievement of PDO indicators. The Project substantially achieved PDO indicator 1.1 (Increased road connectivity to habitations). However, the Project did not achieve the original targets for the length of roads and the number of beneficiaries. Considering the above, the achievement of PDO 1 is rated as Substantial with moderate shortcomings. Rating Substantial OBJECTIVE 1 REVISION 1 Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) Revised Objective PDO 1 - Improve rural road connectivity in project districts - Restructured Project Revised Rationale The original objective remained the same, only the project scope was reduced. The rationale is unchanged. Achievement of PDO indicators. The project improved the connectivity of 61,476 habitations exceeding the target of 61,200. The project directly benefited 838,240 people, exceeding the revised target of 7,44,499 people. Since the project exceeded the revised targets, the achievement of PDO 1 is therefore rated “High”. Revised Rating High OBJECTIVE 2 Objective PDO 2 - Enhance management of roads in Bihar. Original Project Rationale Theory of Change - The combined theory of change is described under Objective 1. Outputs 415 RWD staff were trained on road management compared to the original target of 400. However, the ICR notes that the structured approach to strengthening institutional capacities through the adoption of the Human Resources Professional Development Strategy did not materialize as envisioned by the Project, which could weaken the impact of these training activities in the long run. A network level climate vulnerability assessment was carried out under the Project for 500 km of rural roads (falling short of the target of 2000kms) in 10 flood prone districts to identify the climate vulnerability of each road section based on the exposure, sensitivity, and adaptive capacity of each road to the projected climate hazards for the period 2020-2039. The recommendations of the assessment are yet to be implemented but are being considered by RWD. A Rural Road Safety Action Plan was developed identifying interventions required to improve road safety. The project also developed Road Safety Audit Checklists for different stages of road works in line with the Rural Road Safety Manual 2016 and the Manual on Road Safety Audit and provided training to DPR consultants on incorporating road safety measures in DPRs. GIS based road and bridge inventory and condition database (including videography) established in project districts - Target 50%. This indicator was not achieved and dropped. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) Length of roads subject to planned maintenance contracting - the actual was 45,000kms compared to target of 10,000 kms. Road Safety improvement scheme related audits undertaken on priority roads (Km). The actual achievement was only 204 kms compared to a target of 2000kms (10% achieved) Outcomes Number of districts that have developed and are using asset management system - The original target was 10 and this was dropped. It was replaced by an indicator on Pilot AI based system in districts within the state The project introduced two innovative road paving methods to reduce costs: (i) use of waste plastic in flexible pavements; and (ii) use of panel concrete in rigid pavements. The innovative road paving methods, was used in 52% percent of the roads improved under the Project (exceeding the target of 50%), and saved around US$4.8 million (INR40.34 crores or approximately about 2 percent of the total cost) in road rehabilitation costs. The ICR notes that this is now also being used in the MMGSY program. Achievement of PDO indicators. The Project met the targets of introducing cost efficiencies in road improvement through innovative practices, and the number of RWD staff receiving training exceeded the original target. The network level vulnerability assessment was substantially underachieved. The Project did not implement the AMS, which was a key institutional improvement envisaged under the Project. Considering the above, the achievement of PDO 2 is rated “Modest”. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective PDO 2 - Enhance management of roads in Bihar. Restructured Project Revised Rationale The original objective remained the same, only the project scope was reduced. The rationale is unchanged. Outputs A network level climate vulnerability assessment was carried out under the Project for 500 km of rural roads (meeting the target of 500kms) in 10 flood prone districts. Even though the project achieved the reduced targets for completing network level vulnerability assessments, the corresponding recommendations are yet to be implemented. Outcome The AMP developed under the Project helped RWD identify the levels of service to be maintained for each road asset. Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) The original indicator on number of districts that have developed and are using asset management system (target of 10) was dropped and replaced by an indicator on Pilot AI based system in districts within the state. Compared to the original Project, the AMS was dropped due to the state’s maintenance policy and the introduction of the AI tool which was expected to support RWD’s asset management practices. While the indicator was achieved, it was funded by the state and cannot be attributed to the project. The tool enables RWD to capture and digitally record road conditions and take measures to rectify defects. However, the tool falls short of a comprehensive AMS since it only captures road condition data and does not consider other important criteria required for asset management such as traffic and crash data, connectivity, road structures etc. Also, while the state’s maintenance policy is actually being implemented, it only prescribes a basis for budget allocations for maintenance on a time-based methodology without considering a broader range of road conditions and key technical criteria to systematically prioritize road investments. Thus, the Project’s expected outcomes on this topic were not fully realized. The achievement of PDO 2 is therefore rated “Modest”. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale For the Original Project, the achievements of PDO 1 is rated Substantial with moderate shortcomings and of PDO2 is rated weak Modest. Consequently, the efficacy of the Original Project is rated Modest. Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale For the Restructured Project, the achievements of PDO 1 is rated High and of PDO2 is rated Modest. Consequently, the efficacy of the Restructured Project is rated Substantial with moderate shortcomings. Overall Efficacy Revision 1 Rating Substantial Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) 5. Efficiency Economic efficiency At appraisal, economic analysis was carried out for the project with the coverage of 2,500 km with flexible pavement improvement option within the roads. The primary benefits considered for the economic analysis included: (a) reduction in vehicle operating costs (VOCs) and travel time savings for vehicle users and (b) reduction in carbon emissions from vehicles using the improved roads. Based on estimates of economic costs and benefits over a period of 20 years, including 4 years of implementation, the economic internal rate of return (EIRR) of the project was 15.1 percent. At completion, economic evaluation was carried out for completed upgrading of gravel roads to paved roads with necessary cross-drainage works covering 2,200 km length. The results of the economic analysis conducted are based on: (i) final completion costs, (ii) implementation period; and (iii) observed traffic growth rate during the implementation period. The ex-post economic analysis found that the Project remained economically viable at completion, with an Economic Internal Rate of Return (EIRR) slightly above the rate at appraisal, the rate at completion being 15.9 percent. The Net Present Value (NPV) discounted at 6 percent is positive for the project. Reduction in per km cost of road improvements was a key driver of efficiency of the Project. As discussed in the efficacy section the low-cost paving methods introduced in the Project led to substantial cost savings. This also enabled the Project to improve around 200km of additional roads beyond the expected target of 2000km at Restructuring 2, despite the cancellation of US$27 million from the credit. Around US$5 million of funds remained unutilized at the end of the Project and were duly cancelled. Cost Effectiveness, Administrative and Operational Efficiency Project activities took 12 months longer to complete than the originally expected timeline, despite the reductions in scope. The Project closing date was extended twice to enable the completion of civil works that were delayed as a result of the COVID-19 pandemic, flash floods, and the procurement delays during early years of implementation. Initiation of activities under Component 2 was delayed by over three years, due to capacity constraints, staff turnovers and prioritization of civil works. However, the end-of-project EIRR of 15.9 percent indicates that the completed project is still economically viable with decreased construction cost, despite lower traffic growth and project coverage, and longer implementation period. Even though the Project faced some implementation delays, considering the economic benefits created and the substantial cost efficiencies generated, efficiency of the Project is rated Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) 100.00 Appraisal  15.10  Not Applicable 100.00 ICR Estimate  15.90  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's relevance of objectives is rated High. The efficacy of the original Project is rated Modest, whereas the efficacy of the restructured Project is rated Substantial. The project’s efficiency is rated Substantial. Based on these findings, the outcome of the original Project is rated Moderately Unsatisfactory, while the outcome of the restructured Project is rated Moderately Satisfactory. The overall outcome rating is Moderately Unsatisfactory based on the split evaluation below.   Original Revised Relevance of PDO High Efficacy of PDO Modest Substantial PDO 1 - Improved rural road connectivity Substantial High PDO 2 - Enhanced management of roads Modest Modest Efficiency Substantial 1 Outcome ratings Moderately Unsatisfactory Moderately Satisfactory 2 Numerical value of the outcome ratings* 3 4 3 Disbursement (US$ million) 101.26 46.64 4 Share of disbursement (Percent) 68.5% 31.5% 5 Weighted value of the outcome rating (2*4) 2.06 1.26 Moderately Unsatisfactory 6 Final Outcome rating (2.06+1.26=3.32 rounding to 3) a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome It appears possible that the development outcomes of the Project be sustained, although there are some significant risks. On the positive side, GoB’s commitment to carrying forward the key infrastructure and operational improvements supported by the Project is demonstrated by the following: (i) the plan to expand the AI maintenance system to the entire network, potentially with more features on asset management; (ii) the introduction of the Bihar Rural Roads Maintenance Policy in 2018 mandating performance-based maintenance for roads completing five-year routine maintenance; (iii) the state policy on Road Safety, which mandates road safety audits for every road section that is longer than five kms; and (iv) the adoption of innovative, low cost pavement methods in the state’s MMGSY program. However, on the negative side, the Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) fact remains that at this point the AI maintenance system remains much less comprehensive than the full- fledged AMS the Project initially considered, with no clear visibility on future developments. Another key risk to the sustainability of these outcomes stems from the low institutional capacity in the road sector agencies in terms of staffing and skills. Further strengthening the skills and the staffing in these agencies will be critical to sustain these outcomes and further enhance road sector management in the state. 8. Assessment of Bank Performance a. Quality-at-Entry The Project was strategically relevant and government commitment was secured at entry. The Project built on the ongoing activities in the state under the GoB’s broader MMGSY program and existing sector development plans, with an aim to bring in value from the Bank’s experience in the sector in India and globally as well as in the country. The development objectives were strategically aligned with the sector requirements and the government’s development plans. The key plans for asset management, human resource development and road safety improvement were developed and approved by GoB during preparation. Risks were adequately identified; however, the mitigation measures were not sufficient. The Project identified technical design and institutional capacity as substantial risks. Although mitigation measures were put in place to address staffing gaps through the engagement of a Project Management Consultant (PMC) and technical skills through capacity enhancements provided by local and international experts, they were not adequate. The capacity of the construction industry was adequately assessed as being less developed. The Project would have benefitted from including targeted capacity enhancement activities in the design and with better contract packaging (i.e., larger package sizes) to make contract management more efficient. Capacity enhancement of implementing agencies was given adequate consideration during preparation, including workshops and trainings by international experts on innovative technologies and on the use of local materials in rural road construction, support from consultants on the rehabilitation of bridges, and study tours to Japan and UK on climate resilience and disaster risk management in the road sector. A procurement capacity assessment was carried out and the required trainings were provided at appraisal. Also, an institutional assessment was carried out to inform the activities under Component 2 of the Project. The study identified actions required to improve business practices of the implementing agency in areas such as planning and project preparation, funding, procurement, and project monitoring. It also assessed the Online Management, Monitoring and Accounting System (OMMAS) used by the Bihar Rural Roads Development Agency (BRRDA) and provided recommendations to further improve the system. Owing to the fact that the limited adequacy of the institutional risk mitigation measures proved critical when delivering Component 2 of the Project, the Quality at Entry is rated Moderately Satisfactory. Quality-at-Entry Rating Moderately Satisfactory Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) b. Quality of supervision Implementation support was provided through regular missions, support from technical consultants, and workshops/trainings. The Bank supported project implementation through eight missions, complemented by technical visits and virtual discussions. The Bank provided substantial support to improve the quality of the DPRs for the road packages. Procurement support was provided to familiarize the implementing agency with the Bank’s procurement procedures and handholding support was provided to the implementing agencies on safeguards management. Furthermore, efforts were made by the Bank to introduce and mainstream the concepts and good practices of asset management in the road sector and to retain the original activities around asset management to ensure the achievement of the expected development outcome. Overall, implementation and disbursement progress was closely monitored, issues were candidly discussed and agreed actions were closely followed up. Close monitoring of the achievement of results and disbursements appropriately informed the cancellations and restructurings. The Bank team consisted of a mix of staff and consultants based in the country office, in Washington, and in the region. This enabled close follow-up and quick response to implementation challenges. However, the Bank could have been more proactive in addressing the long implementation delays in Component 2. The long delay in Component 2 implementation largely stemmed from inadequate capacity and a lack of strong commitment/ownership from the implementing agency which resulted in prioritization of civil works. While increased efforts by the Bank to provide more support through technical consultants and close follow-ups helped expedite implementation in the latter part of the Project, the Bank could have been more proactive in providing more support to the implementing agency at an earlier stage or to restructure the Project earlier to change the scope as appropriate. Overall, the quality of supervision is rated as Moderately Satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The retrofitted Theory of Change developed at the second restructuring clearly identifies the Project’s causal chain to achieve expected outcomes and the indicators are mostly relevant to the scope with some gaps. The indicators introduced in the original project are well defined, the baselines were clearly identified, and the targets were clearly defined. The methodologies for monitoring and data collection were simple and utilized the existing Management Information System (MIS) of the RWD. Responsibilities for reporting on progress towards achieving indicator targets were clearly identified. However, the indicators displayed the following weaknesses: (i) measurability: the PDO indicator “Roads in good and fair condition as a share of total classified roads” was difficult to measure due to the difficulties in collecting data given the different Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) institutions responsible for managing roads in Bihar and challenged in defining the applicable geographical area; (ii) attributability: the above PDO indicator and the intermediate indicator “Length of roads subject to planned maintenance contracting” were defined to consider roads in the entire state, while the project was only implemented in selected road sections; and (iii) assessing Project impact: the M&E design included indicators that adequately captures achievement of PDO 1, but the Project would have benefited by including a beneficiary satisfaction indicator to assess the impact of the improved roads based on beneficiary feedback. The original and post-restructuring PDO indicators to assess PDO 2 were output oriented and did not adequately capture impacts envisioned by PDO 2: (i) the original PDO indicator “Improved asset management” measured the number of districts that had developed and were using an asset management system, while the PDO indicator introduced at the second restructuring “Pilot AI based system in districts within the state” assessed the number of districts, and not their impact; and (ii) the PDO indicator “Improved effectiveness of project expenditures” assessed the percentage of Project roads designed using cost‐effective measures, but not cost savings. b. M&E Implementation Progress in achieving indicator targets was reported through quarterly progress reports submitted by RWD and progress reviews during missions. The Management Information System of the RWD was utilized for reporting on progress, as envisioned; however, quarterly progress reports were not provided regularly. An excel sheet-based monitoring mechanism was used to monitor progress of civil works and disbursements. The RWD submitted the Mid-Term Review (MTR) report with a detailed review of the progress of Component 1 and status of activities under Component 2. There were some deficiencies in the reporting of a few indicators. The intermediate indicator “Road Safety improvement scheme related audits undertaken on priority roads” was reported on state roads where safety audits were carried out, rather than on project roads. The intermediate indicator “Improved quality of built Infrastructure ‐ Project roads delivered with satisfactory quality certified through independent quality reviews” was measured through the state’s own quality monitoring system, rather than through independent reviews. However, a separate technical audit was later carried out to assess the quality of road works under the first batch of contracts. c. M&E Utilization The M&E data were utilized to inform Project progress monitoring, and in determining the changes needed to the scope, as part of the various restructurings. These included the savings in the credit to be addressed through cancellations, extensions to the credit closing date to complete the project and changes to the scope of Component 2.1 on asset management. Based on the above discussion of M&E Design, Implementation and Utilization, which delineated the shortcomings in some of the outcome indicators and reporting deficiencies, the overall quality of M&E is rated Modest. M&E Quality Rating Modest Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) 10. Other Issues a. Safeguards Environmental and Social Safeguards. The Project was classified as Category B and triggered the following Bank safeguard policies: Environmental Assessment (OP/BP 4.01); Physical Cultural Resources (OP/BP 4.11); Indigenous People (OP/BP 4.10); and Involuntary Resettlement (OP/BP 4.12). Safeguards management was guided by the Environmental Management Framework (EMF), Environmental Codes of Practices (ECoPs), Social Management Framework (SMF), and a Vulnerability Framework (VF), which were prepared and disclosed by the Project. The Project reported several shortcomings in environmental management during the first half of implementation: (i) delays in reporting environmental screening data for the roads; (ii) inadequate monitoring and reporting of implementation of environmental management aspects; (iii) capacity constraints in managing environmental safeguards. These issues were addressed in the latter half of the Project and the management and compliance of environmental safeguards improved with the mobilization of an environmental safeguards specialist to the Project. The Project did not involve any land acquisition or involuntary resettlement. Social risks and adverse impacts were largely temporary, site-specific, and manageable through the instruments developed during preparation. However, safeguards management remained weak during the first half of the Project due to the lack of adequately skilled and experienced staff. The hiring of a firm to support E&S activities helped improve safeguards management in the latter half of the Project. The Project reported instances of OP 4.12 non-compliance in two sub-projects; road alignments (culverts and small bridges) in these two sub-projects inadvertently passed through small linear parcels of private land. These were addressed by the Project under the guidance of the Bank, through consultations with villagers and subsequent changes in alignments to avoid encroachment on private lands. The Project also faced issues in reporting on grievances in the early stages, which were later resolved following the introduction of Janta Darbars. b. Fiduciary Compliance The Project complied with the Bank’s Financial Management (FM) guidelines and procedures. The FM performance ratings were mostly Moderately Satisfactory during project implementation. Annual project budgets were adequately and timely provided by GoB and there were no substantial delays in the flow of funds. Bihar Rural Road Development Agency (BRRDA), through which RWD implemented the Project, used a computerized accounting software to account for project expenditures and submitted Interim Financial reports (IFRs) regularly. BRRDA engaged a consultant under the project to operationalize a newer version of the accounting software in all district PIUs to efficiently manage FM functions, including timely consolidation of accounts and preparation of year-end entity financial statements. This activity could not be completed during the project period, and BRRDA has agreed to utilize GoB funds to complete this activity after project closure. There were delays in conducting internal audits of the project due to delays in the appointment of internal auditors. The Project external audits were submitted to the Bank within the due date of December 31 every year. The audit opinions were ‘unmodified’ and no significant accountability or internal control issues were noted by the auditor. The external audit report for the final year of the Project Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) (i.e., FY23-24 for the period April 01, 2023, to December 31, 2023) is required to be submitted by BRRDA to the Bank by December 31, 2024. Procurement under the Project consisted mainly of civil works, with a few consultancies and a small number of goods/non-consultancy services contracts. A Procurement Plan was created in STEP and was updated regularly. Bank procedures were duly followed, and Bank documents were used for all procurements. At times there were delays in procurement and there were some cases of rebidding; however, almost all planned procurements were completed during the first three years of the Project. There was a delay in the completion of most civil work contracts due to the COVID-19 pandemic; however, all packages were completed during the extended project period. There were some delays in updating STEP because of the numerous implementing offices in the state. Overall, the Project managed procurements in line with Bank systems and procedures. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons Strengthening institutions requires long term engagement and strong government commitment, and thus, a programmatic approach may be better suited for such projects. The lack of strong commitment from the implementing agencies along with changing political priorities can also hinder the process; this often leads to prioritizing infrastructure improvement, which are more straightforward, over institutional improvements that can be time consuming to implement and often requires strong ownerships and champions to execute. A programmatic approach, which goes beyond a typical project lifecycle, could help maintain continuous consultations and engagement with the government agencies to sensitize them to the benefits of institutional improvements and to build strong commitment over time to implement them. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) Strong and flexible Bank support is required where the implementing agency and the sector has significant capacity gaps. The flexibility in the level of Bank support in response to emerging capacity shortfalls proves important for the successful implementation of Project activities. In instances where significant capacity gaps are identified in the implementing agencies, such support can be built into the Project design at the beginning and options such as Hands-on Expanded Implementation Support (HEIS) can be explored for both procurement and technical support. Innovative road rehabilitation methods can bring benefits, but their sustainability need to be monitored over time. In particular, they need to be supported by a comprehensive Assets Management System to ensure that local climatic conditions are properly factored in lifecycle costs, including routine and periodic maintenance. Building trust through close consultations with communities can improve efficiencies in project implementation. As demonstrated by the experiences from the Janata Durbars initiative, it is clear that regular consultations with project communities play a key role in fostering trust towards the project and the implementing agencies. Engaging community members in project activities, promptly addressing their feedback and concerns, and incorporating their suggestions are essential strategies that not only build trust but also facilitate smoother and more efficient project implementation. Such participatory approaches can be integrated into the design of future projects to enhance their effectiveness and community acceptance. The adequacy and appropriateness of technology should be assessed against the specific contexts and environments in which it is used. A key lesson from the Project’s experience with the use of an app to record grievances is that when technology is used, particularly involving communities with poor literacy and inadequate access to technological devices, it is important to ensure that (i) the technology used is tailored to address the specific requirements of the users, (ii) users are aware and educated in how to use the technological solution, and (iii) the appropriateness of the use of technology is assessed in the specific context. As was evident from the Project’s experience with Janta Darbars which promoted more person-person interaction, at times, more human-oriented solutions can create better results compared to technological solutions. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is largely in line with the ICR guidelines. It is well-written and internally consistent. It contains detailed descriptions of the project context, implementation, and achievements. The theory of change is logically presented, and the document is clearly results-oriented. The ICR adequately reports on the achievement of the targets and provides additional information on project impacts. Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BRRP (P155522) In particular, the ICR is candid about the shortcomings of both the Borrower and the Bank during the first years of implementation, which led to the reduction in project scope for both the infrastructure and the institutional development components, and ultimately resulted in a mid-range outcome rating. The ICR provides adequate information on the project’s fiduciary compliance and performance. The lessons are important, based on evidence, and include valuable forward-looking pointers. a. Quality of ICR Rating Substantial Page 19 of 19