Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) Report Number: ICRR0022926 1. Project Data Project ID Project Name P127156 SECOND ROAD & SAFETY IMPROVEMENT PROJ. Country Practice Area(Lead) Ukraine Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-81950 31-Dec-2016 450,000,000.00 Bank Approval Date Closing Date (Actual) 20-Sep-2012 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 450,000,000.00 0.00 Revised Commitment 450,000,000.00 0.00 Actual 450,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Peter Nigel Freeman Victoria Alexeeva IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement dated October 11, 2012 (Schedule 1, page 5) and in the Project Appraisal Document (PAD, page 7): " An improved condition and quality of sections of the M - 03 road and increased road safety on roads " Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) As road condition and quality are interwoven with improved road safety, they could not be separated in this project. This review is hence is based on the single objective stated above. *The achievement of the project development objectives in Ukraine is assessed following the same methodological approach. This applies to all projects closed by June 30, 2021. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 16-Jul-2015 c. Will a split evaluation be undertaken? Yes d. Components There were three components (PAD, pages 9 - 10). 1. Road Rehabilitation. The estimated cost at appraisal (including Value Added Taxes (VAT) US$476.4 million. The actual cost was US$441.9 million. This component aimed to finance the rehabilitation of about 108 kilometers (km) of the Kyiv - Kharkiv - Dovzhansky road. The activities included: (i) rehabilitation of 30 km of an existing four lane section, widening 58 km of roads from two to four lanes (58 km) and constructing 20 km of three bypasses. The total length of roads to be rehabilitated was reduced from the original target of 108 km to 86.4 km through the first project restructuring. The total length of roads to be rehabilitated was reduced from 86.4 km to 57.3 km through the second project restructuring (discussed below). 2. Road Safety Improvements. The estimated cost at appraisal (including VAT) was US$60.0 million. The actual cost was US$2.0 million. The significant difference between the estimated cost at appraisal and actual cost was due to the transferring of many of the activities in this component to an ongoing project after the second project restructuring. Activities in this component were; (i) safety improvements along 60 km of high-risk road corridors; and (ii) introducing improved road safety features on the existing M -03 sections in urbanized areas. The scope of this activity was reduced from 60 km to 40.8 km through the project restructuring on July 16, 2015. 3. Institutional Support and Strengthening. The estimated cost at appraisal (including VAT) was US$3.6 million. The actual cost was US$1.9 million. Activities in this component included; (i) an assessment of 12,000 km of Ukraine's network of main highways roads for safety; (ii) developing a Strategic Program for Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) Road Sector Development until 2030; and (iii) capacity building of the Ukrainian State Institute for Design of Roads (Ukriprodor) and other Ukrainian institutions for designing and building safer roads. The activity of assessment of Ukraine's network of main highway roads for road safety was dropped and replaced with technical advisory services during the project restructuring. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The estimated cost at appraisal (including physical and price contingencies) was US$540.0 million. The actual cost was US$450.0 million. Project financing. An IBRD loan of US$450.0 million financed the project. The loan was fully disbursed. There was parallel financing of US$90.0 million for complementary activities on the government's road Infrastructure Program from the European Bank for Reconstruction and Development (EBRD), the European Investment Bank (EIB) and the European Union (EU). Borrower contribution. The recipient contribution was estimated at US$90.0 million. However, there had been no recipient contribution by project closure. Dates. The project was approved on September 20, 2012 and became effective on December 24, 2012. The project was scheduled to close on December 31, 2016. However, the project closed four and half years behind schedule on June 30, 2021. Other changes. There were five Level 2 restructurings during the project lifetime. The Bank supported the following changes made through the first restructuring on July 16, 2015.  At Government request, the Bank agreed to finance the land acquisitions for the project. As per the original financing plan, the government was to finance the land acquisitions. The government was unable to do this due to the Ukrainian political crisis in 2015 (the conflict between Russia and Ukraine that began in 2014, centering on the status of the Ukrainian regions of Crimea and Donbas). The crisis affected the government's capacity to finance the planned counterpart contributions. Therefore, the Government requested the Bank to finance land acquisitions and up to 100 percent of the project activities VAT. With this, the recipient contribution was reduced from US$90.0 million at appraisal to US$28.1 million.  As noted above, the project scope was reduced for all three components. With the second restructuring on October 28, 2016, the closing date was extended by two years from December 31, 2016, to December 31, 2018, due to land acquisition and procurement delays for activities associated with constructing bypasses. With the third restructuring on June 28, 2018, the closing date was extended by one and half years from December 31, 2018, to June 30, 2020, due to the delays in constructing the three bypasses and road safety improvements. The Government supported the following changes through the fourth restructuring on November 2019. project. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156)  As noted above, there was change in project scope.  Due to the implementation delays (due to multiple contract terminations), there was insufficient time or budget to complete the project activities. However, there was sufficient resources under the preceding ongoing Bank-financed Road project to complete the works of this project. Therefore, most of the road safety (component two) activities of this project was transferred to the ongoing project. With the fifth project restructuring on April 15, 2020, the project closing date was further extended by a year from June 30, 2020, to June 30, 2021, due to the delays caused by the restrictions in the wake of the COVID-19 pandemic. Split rating. The PDO remained unchanged throughout implementation. However, as the targets were revised rather substantially with the project restructurings, this assessment is based on a split rating of objectives when 41% (US$184.0 million) was disbursed before the first project restructuring in 2015, 39% (US$176.4 million) before the project restructuring in 2019, and the balance 10% (US$89 million) after the restructuring in 2019. 3. Relevance of Objectives Rationale Country context. Ukraine is a key transit country in view of its strategic geographic location between Russia and Europe. Although Ukraine traditionally produced and exported metals, heavy industry products and agricultural goods in the decade before appraisal, there was evidence of a gradual shift towards production and exports of high-value, time-sensitive, semi-finished goods. Given this shift, modernizing the road network was important for movement of high-value freight. Sector context. Roads and railways account for the major part of passenger and freight movement within and outside the country. Ukraine's road network included seven major transport corridors in East-West and in the North-South directions carrying transit freight to neighboring states and countries beyond (the EU and Central Asia). At appraisal, the EU and the EBRD were financing the rehabilitation of roads between Kyiv and Ukraine's western border. It was also important for the government to improve corridors east of Kyiv towards the industrial heartland of Ukraine and the border with Russia to get maximum returns on these road investments. The project area - the M 03 road - connected eastern Ukraine to Kyiv. The main challenges faced by the road sector were the poor ridership quality and road safety on a substantial portion of the network, due to the large backlog of required periodic maintenance and rehabilitation in the past. Government strategy. In the years before appraisal, the Government issued a transport policy articulating the need for a more balanced development of transport modes, with the rail mode retaining its role as the dominant mode for heavy bulk goods, while the road network was to increasingly facilitate the movement of high-value goods. The Cabinet of Ministers in 2010 adopted Government's Transport Strategy Until 2020. This strategy aimed to modernize the transport system, improve the quality and access to services, improve sector governance and increase road safety. Bank strategy. At appraisal, the second pillar of the Country Partnership Framework (CPF) of 2012 - 2016, articulated the need for more productive cooperation between the government and business Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) through focusing on growth, competitiveness, job creation and channeling public investment into critical public infrastructure. The PDO was well-aligned with the Country Partnership Framework (CPF) for 2017 - 2021. The CPF identified inadequate infrastructure as a bottleneck holding back private sector competitiveness and included a Focus Area "making markets work". This focus area specifically identified the need for improving transport and energy infrastructure. The PDO was also consistent with the first pillar of the Bank's Regional strategy for Europe and Central Asia updated in 2011: "deepening reforms for competitiveness", through transport and energy infrastructure investments and improving sector governance. The Bank's CPF for 2022 - 2026 is currently under preparation. Previous Bank experience. This project was a follow-on (repeater) project to an ongoing Bank-financed Road and Safety Improvement Project, with the same development objectives. The PDO formulation is output-oriented. However, the theory of change presents clear expected outcomes at higher levels.This project aimed to rehabilitate additional sections of the same M -03 road and improve road safety conditions on several road sections on the corridor. Given that the design was similar to the ongoing project and did not present significant new challenges, the relevance of the PDO is assessed as substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective An improved condition and quality of sections of the M - 03 road and increased road safety on roads. Rationale Theory of change. The outputs such as rehabilitation of the existing four lane sections, widening sections of the M - 03 road from two to four lanes and constructing bypasses, together with institutional strengthening activities (such as preparing the strategic program for road development and developing the capacities of Ukratodar and other institutions), were likely to improve the condition and quality of sections of the M 03 road. These outcomes were also likely to aid in the long-term outcome of improving domestic and regional land connectivity in Ukraine. The outputs of activities such as improved road safety features of 60 km of road and road survey assessment were likely to aid in reducing road fatalities and thereby in improving road safety. The causal links between the project activities, outputs and outcomes were logical and the outcomes could be monitored. The intended outcomes were predicated on the following assumptions: (i) the Government remains committed to road reforms, including improving road maintenance funding; (ii) the roads are developed to good technical specifications and standards: (iii) the Government remains committed to developing the capacities of the road agencies; and (iv) other regulatory, enforcement and technological factors contributing to reduction in fatalities are sustained at current levels, if not improved. Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) Outputs (ICR, pages 15 - 17 and pages 33 -36).  A total of 53.30 km of roads were rehabilitated/upgraded/constructed when the project closed as compared to the original target of 108km. This was significantly short of the original target of 108.3 km. This total included, rehabilitating 22.5 km roads, upgrading 10.8 km from two to four lanes and constructing 20.3 km of three bypasses (Pokrovskaya, Bogachka and Beltseryuka) that detoured traffic away from urban congestion. The roads were constructed and rehabilitated using asphalt concrete standards with a design speed of 120 km/hour.  53.50 km of roads were improved with road safety features. This was short of the target of 60 km.  The project financed the re-design of the village sections of the M - 03 and implemented measures to reduce traffic speed (such as traffic calming measures), pedestrian crossings, guardrails, road signs, pavement markings, lighting at critical intersections and bus stops.  The project developed a Strategic Program Sector Development until 2030, with recommendations and key pillars for developing the road sector over the medium and long terms. The Government adopted the Action Plan emanating from this strategy, and a number of actions of the plan were to be implemented over the year, including some under other Bank-funded projects.  The project provided capacity building activities for the institutional strengthening of Ukratodar (the agency in charge of managing the national road network), the Ukrainian State Institute for Design of Roads (Ukriprodor) and other Ukrainian institutions for designing and building safer roads.  A Technical Audit for Domestic funded Road Works was completed as targeted. Outcomes The outputs described above were intended to realize two outcomes: (i) improve road conditions and quality of sections of the M 03 road, measured through the International Roughness Index (IRI); this is a standardized method for measuring the riding quality for road users. A lower IRI number signifies smoother ride; and (ii) the outputs of activities described above were aimed to reduce the total number of fatal accidents along the road sections. .  The existing roads had an IRI value of five when works commenced. The rehabilitated roads achieved a target value of two when the project closed. Similarly all the newly constructed roads achieved an IRI target value of two.  The number of fatalities per 100 vehicle-km dropped from twenty four at the baseline in December 31, 2014 to six in November 2021. This exceeded the target of 14.  There was a 72% reduction in fatalities and serious injuries per 100 million vehicle-km. This was short of the target of 80%. However, it is difficult to attribute such outcomes to project activities, given that the outcome would also depend on factors such as public awareness, drivers' behavior, police enforcement and vehicle standards. Nevertheless, it is reasonable to conclude that improved road safety can contribute significantly to a reduction in fatalities and injuries. Given that the target with respect to the rehabilitation of roads was only partially realized, the efficacy of the PDO is assessed as modest. Rating Modest Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) OBJECTIVE 1 REVISION 1 Revised Objective An improved condition and quality of sections of the M - 03 road and increased road safety on roads. Revised Rationale The PDO did not change, but the targets were revised downwards with the project restructuring in 2015. Outputs The outputs described above were relevant to this objective, with one difference.  A total of 53.30 km of roads were rehabilitated/upgraded/constructed when the project closed as compared to the revised target of 86.4 km. Outcomes  The existing roads had an IRI value of five when works commenced. The rehabilitated roads achieved a target value of two when the project closed. Similarly all the newly constructed roads achieved a target value IRI of two.  The number of fatalities per 100 vehicle-km dropped from twenty four at the baseline in December 31, 2014 to six in November 2021. This exceeded the target of 14.  There was a 72% reduction in fatalities and serious injuries per 100 million vehicle-km. This was slightly short of the target of 80%. It is difficult to attribute the outcomes to project activities, given that the outcome would also depend on factors such as public awareness, drivers' behavior, police enforcement and vehicle standards. Nevertheless, it is reasonable to conclude that improved road safety can contribute significantly to reduction in fatalities. The efficacy is rated substantial, with moderate shortcomings. Revised Rating Substantial OBJECTIVE 1 REVISION 2 Revised Objective An improved condition and quality of sections of the M - 03 road and increased road safety on roads. Revised Rationale The PDO did not change, but the targets were revised downwards with the project restructuring in 2019. Outputs The outputs described above were relevant to this objective, with one difference. Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156)  A total of 53.30 km of roads were rehabilitated/upgraded/constructed when the project closed as compared to the revised target of 57.3 km. Outcomes  The existing roads had an IRI value of five when works commenced. The rehabilitated roads achieved a target value of two when the project closed. Similarly all the newly constructed roads achieved a target value IRI of two.  The number of fatalities per 100 vehicle-km dropped from twenty four at the baseline in December 31, 2014 to six in November 2021. This exceeded the target of 14.  There was a 72% reduction in fatalities and serious injuries per 100 million vehicle-km. This was slightly short of the target of 80%. It is difficult to attribute the outcomes to project activities, given that the outcome would also depend on factors such as public awareness, drivers' behavior, police enforcement and vehicle standards. Nevertheless, it is reasonable to conclude that improved road safety can contribute significantly to reduction in fatalities. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The efficacy rating as compared to the original project scope is modest. Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale The efficacy rating as compared to the revised project scope following the 2015 project restructuring is substantial, with moderate shortcomings. Overall Efficacy Revision 1 Rating Substantial OBJR2_TBL OVERALL EFFICACY REVISION 2 Overall Efficacy Revision 2 Rationale Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) The efficacy rating as compared to the revised scope following the 2019 restructuring is substantial. Overall Efficacy Revision 2 Rating Substantial 5. Efficiency Economic analysis. An economic analysis was conducted for the two project components (rehabilitation of roads and road safety measures on roads) using the recent version of the Highway Development Management (HDM) model (version 2.09). These components accounted for over 99% of the estimated cost at appraisal and actual cost. The potential economic benefits of the project were assumed come from: (i) savings in vehicle operating costs; (ii) travel time savings: and (iii) reduction in traffic fatalities and injuries. The overall ex-post Economic Internal Rate of Return (EIRR) was 14.0%, exceeding the ex-ante EIRR of 13.1%. The actual cost of rehabilitating the roads was US$213.4 million, as compared to the estimated cost at appraisal of US$199.0 million. However, although the actual cost was higher, this was offset by the higher-than-expected traffic flows in the project-affected areas. Administrative and operational issues. There were three major challenges contributing to the delays during implementation. The first was due to the procurement delays caused by the inadequate performance of the contractors, which led to the termination of their contracts in two cases. The other two reasons were due to external factors outside the project's control. The first was the tense political situation in the initial years of the project during 2013-2015 (conflict in the east of Ukraine). This contributed to the Government's inability to provide the planned counterpart funding for land acquisition activities. This contributed to delays in land acquisition. The second was the physical restrictions imposed in the wake of the COVID-19 pandemic in the final year of the project. These factors worsened the cost and time overruns, with the project closing four and half years behind schedule. Although there were operational issues during implementation, efficiency is assessed as substantial, given the solid economic justification for the project. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 99.00 Appraisal  13.10  Not Applicable Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) 99.00 ICR Estimate  14.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the original outcome targets. With substantial relevance of design to the Government and Bank strategies, substantial efficiency and modest efficacy, the overall rating is Moderately Unsatisfactory (3). Under the first outcome revised targets. With substantial relevance of design to the Government and Bank strategies, substantial efficacy with moderate shortcomings and substantial efficiency, the overall rating is Moderately satisfactory (4). Under the second outcome revised targets. With substantial relevance of design to the Government and Bank strategies, substantial efficacy and substantial efficiency, the overall rating is Satisfactory (5). A double split rating is applied based on the disbursement shares before and after the project restructurings in 2015 and 2019, when a disbursement share was at 41%, 39% and 20% (total financing was US$450.0 million). The overall outcome is Moderately Satisfactory, the weighted value is 4 (41*3+39*4+20*5=3.79). a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ongoing Russia-Ukraine war has put the development outcomes at high risk. 8. Assessment of Bank Performance a. Quality-at-Entry This project was a repeater project of the Road Safety Improvement Project. The design was straightforward and replicated the design of the ongoing project. The lessons incorporated at design included: (i) The original project had demonstrated the lack of interest by contractors in small road safety improvement contracts scattered over large geographical areas. This project incorporated road safety contracts into large contracts; and (ii) Bank's portfolio review for Ukraine during the past years showed the that sector policy difficulties are best addressed through policy lending operations and sector dialogue. Therefore, this project did not include a major policy reform component. The implementation arrangements were similar to that of the ongoing road project financed by the Bank. Ukratodar (the agency in charge of managing the national road network) was responsible Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) for overall implementation, while the Government-financed Project Implementation Unit - Ukrdorinvest- was in charge of day-to- day project management. The arrangements were appropriate as the performance of these agencies with the ongoing project was satisfactory. Several moderate risks due to the weak capacity of the implementing agency and social and environmental risk were identified at appraisal. Governance at the country level was also identified as a high risk. Mitigation measures incorporated at design included preparation of an updated governance and anti-corruption action plan. With mitigation measures, overall project risk was rated as moderate at appraisal. The arrangements made at appraisal for safeguards and fiduciary compliance were appropriate (discussed in section 10). There were moderate shortcomings at quality-at-entry. First, the communities were not adequately consulted during preparation (ICR, paragraph 69). This became evident when several complaints were voiced by persons affected by the project and necessitated fresh rounds of consultations with local-level stakeholders. This led to some design changes during project implementation (such as, modifications to interchanges, rehabilitation of access roads and construction of bus stops) to address the needs of the local communities. and two, there were in addition M&E design shortcomings (discussed in section 9). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank conducted twice-a-year supervision missions, with 18 Implementation Status Results (ISRs) filed over nine years. The ICR (paragraph 93) notes that there were five task team leaders (TTLs) during the project lifetime, with some based in the country office and some at headquarters. The ICR also notes that throughout implementation, there were country-based transport experts. The Bank team was proactive and flexible in introducing the "abnormally-low-bids" to the bidding process (discussed in section 10b). The team appropriately addressed the challenges that rose during implementation through the two major project restructurings in 2015 (in the wake of the government's inability to provide the planned counterpart funding for land acquisitions) and 2019 (following the physical restrictions on movement of people in the wake of the COVID pandemic). There were minor shortcomings. For example, the supervision team could have addressed the shortcomings in M&E design (discussed in section 9). Overall Bank performance is assessed as moderately satisfactory, given the moderate shortcomings at Quality-at-Entry. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E arrangements were similar to that of the ongoing project, with Ukrdorinvest in charge of monitoring performance (PAD, paragraph 37). The key outcome indicator of monitoring the quality of roads using the International Roughness Index (IRI) was appropriate. Some intermediate indicators did not sufficiently capture the key objectives. For instance, the intermediate indicator associated with the three bypasses did not fully capture their purpose. While the bypasses improve road safety, their primary benefit is travel time savings. The inclusion of such an indicator would have enabled better assessment of the impact of the bypasses, which accounted for about US$179.5 million of the project cost. Likewise, the widening of the road from two to four lanes in addition to improving riding quality, also increased road capacities. Using indicators, such as increased traffic flows or travel time reduction along the target road would have contributed to a more accurate assessment of the impact of these activities. Furthermore, while road safety features do contribute to reducing fatalities, there may be other factors such as public awareness, police enforcement, vehicle standards which could influence the outcome. Comparing the number of fatalities on the project intervened areas with the national trend in fatalities would have enabled better attribution to the project-financed road safety activities. b. M&E Implementation Ukratodar was responsible for collecting the data for monitoring project performance, The ICR (paragraph 81) notes that progress towards the achievement of the final targets was monitored through regular progress reports by Ukratodar and that most indicators were regularly updated without major issues. c. M&E Utilization The ICR (paragraph 82) notes that the M&E indicators were utilized by both Ukratodar and the Bank to gauge and assess performance and to prepare response measures to ensure that the project was on track towards achieving the development objectives. In sum, M&E is assessed as modest, given the shortcomings in M&E design. M&E Quality Rating Modest 10. Other Issues Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) a. Safeguards The project was classified as a Category A (full Assessment) project under the World Bank safeguard policies. Two safeguard policies were triggered at appraisal: Environmental Assessment (OP/BP 4.01); and Involuntary Resettlement (OP/BP 4.12). Environmental assessment. Except for the activity associated with construction of bypasses, the environmental impacts of the road rehabilitation activities was expected to be located within the existing right-of-way. These impacts included air pollution and noise from trucks, soil disturbance, and impacts on surface water networks (PAD, paragraph 64). Two Environmental Impact Assessments (EIAs) and three Environmental Management Plans (EMPs) were prepared and publicly-disclosed at appraisal to address the environmental issues (PAD, paragraph 66). The ICR (paragraph 84) noted that environmental performance was satisfactory throughout most of project life. Ukratodar and the Project Implementation Unit's in-house institutional capacity for environmental management was adequate with minor deficiencies. The ICR also notes that the contractor's and consultant's teams included safeguards experts. Involuntary resettlement. The PAD (paragraph 60) observed that agricultural land acquisitions were mainly required for constructing bypasses, interchanges and road widening. Since the existing-right- of-way of the road was wide enough to accommodate its widening from two to four lanes, land acquisition and resettlement were mainly required for constructing bypasses. Ukrdorinvest prepared and publicly-disclosed an updated Resettlement Policy Framework (RPF) and a Resettlement Action Plan (RAP) at appraisal for addressing resettlement issues (PAD, paragraph 61). The ICR (paragraph 86) related that social safeguard compliance was mainly rated as satisfactory. However, there were several health and safety-related events at some project sites, leading to some fatalities. There was inadequate reporting of these incidents during the earlier part of implementation. Data reporting on Health and Safety Improvements significantly improved under the Bank team's and the Project Implementation Unit's efforts following the November 2018 publication of the Bank's Environmental and Social Response Toolkit (ESRT). b. Fiduciary Compliance Financial management. The financial arrangements of this project were similar to those of the on-going Bank-financed project. The Bank conducted an assessment of the financial management assessments of Ukratodar and Ukrdorinvest at appraisal. They were deemed to be appropriate (PAD, paragraph 52).The ICR (paragraph 88) noted that the financial management performance was moderately satisfactory during the initial years, but was satisfactory from 2016 onwards. All project related interim financial statements and audits were submitted in a timely fashion. The audit reports were unqualified. Procurement. The PAD (paragraph 53) observed that procurement under the project was straightforward and included mostly large civil works contracts and consultancy contracts for technical supervision and capacity building. Ukrdorinvest had extensive experience in managing this type of procurement under the Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) ongoing Bank-financed project and other large projects funded by EBRD and EIB. The procurement arrangements were deemed to be adequate at appraisal. The ICR (paragraph 73) notes that a contractor failed to perform in one contract leading to the termination of the contractor. The contractor had presented a low bid price. The ICR (paragraph 74) notes that the Government voiced strong concerns about the low bid price but the Bank process at the time did not provide specific guidance about "abnormally low bid prices" (which was introduced in Bank operations at a later stage). When reviewing the bid evaluation results, the Bank team had followed the Bank's procedures and processes at the time. To avoid a repeat of the experience, the Bank team introduced the concept into the new bid process. The Bank also supported the Government in developing a Project Procurement Strategy for Development (PPSD). Based on the findings of the PPSD the procurement method was changed from International Competitive Bidding to Limited Competitive Bidding and this led to successful results. The ICR (paragraph 87) notes that there were no unresolved procurement issues when the project closed. c. Unintended impacts (Positive or Negative) The ICR does not report of any unintended impacts. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR draws the following main lessons from the experience of implementing this project, with some adaptation of language. 1. Flexibility in adjusting and improving design during implementation, while remaining focused on the overall objectives, can help realize the development results. This is particularly relevant in countries where certain risks are high. In this project, the Bank not only proactively Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) restructured the project in the face of implementation challenges but also through active implementation support and procurement approaches. 2. A Government-financed Project Implementation Unit (PIU) can be useful in sustaining implementation capacities and commitment at the national level. This approach presents a midway approach between a (typically) Bank-financed PIU designed to oversee a specific project and the use of the existing structure of national agencies/ministries. While still having technical independence the PIU in this project demonstrated eagerness to accelerate implementation, driven by the higher government objectives. 3. Weak indicators for monitoring road safety undermine attribution to development results . Better road safety outcomes are the result of various factors, with better road designs being one of them. For better attribution of results to the project activities, it would be useful if the Bank could more fully develop guidelines for measuring road safety improvements. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written and provides a candid description of the external events that impacted this project. The theory of change provided in the text elucidates the links between project activities, outputs and the intended outcomes. It carries out a split analysis as required in the guidelines. The ICR clearly identifies the shortcomings in the M&E framework and suggests possible ways as to how the framework could have been improved upon. The quality of evidence provided in the text is adequate. The photographs provided in page 47 helps the reader to visualize the changes made by the project activities. Overall, the ICR draws good lessons from the experience of implementing this project. A minor issue is the length of the ICR. The main text of the ICR is almost twice the recommended length of 15 pages. a. Quality of ICR Rating Substantial Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SECOND ROAD & SAFETY IMPROVEMENT PROJ. (P127156) Page 16 of 16