Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) Report Number: ICRR0022834 1. Project Data Project ID Project Name P118023 AZ HIGHWAY 3 Country Practice Area(Lead) Azerbaijan Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-78890,IBRD-86020,IDA-47230 31-Mar-2015 311,285,308.99 Bank Approval Date Closing Date (Actual) 25-May-2010 31-Mar-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 241,600,000.00 0.00 Revised Commitment 316,418,498.11 0.00 Actual 311,285,308.99 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Peter Nigel Freeman Victoria Alexeeva IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) as stated in the Financing Agreement dated August 11, 2010 (Schedule 1, page 6) and in the Project Appraisal Document (PAD, page 6): "To contribute to a more efficient and safer Baku-Shamakhi road and higher quality road services as part of the general network upgrading to motorway standard, and to improve the management of the nascent motorway network ". Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) The PDO was revised with the Additional Financing (AF) on April 7, 2016. The PDO as stated in the AF Agreement dated April 7, 2016 (Schedule 1, page 6): "To contribute to a more efficient and safer Baku-Shamakhi and Yenikend- Shorsulu road and higher quality services as part of the general network upgrading to motorway standard, and to improve the management of the nascent motorway network". The AF expanded the project scope. Therefore, this review is not based on a split rating of objectives. The revised PDOs is unpacked for this review as follows: (i) To contribute to a more efficient Baku-Shamakhi and Yenikend-Shorsulu road; (ii) To contribute to a safer Baku - Shamakhi and Yenikend - Shorsulu road; (iii) To contribute to higher quality services as part of the general network upgrading to motorway standard; and (iv) To improve management of the nascent motorway network. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 07-Apr-2016 c. Will a split evaluation be undertaken? No d. Components There were three components (PAD, page 7). 1. Motorway Improvement. The estimated cost at appraisal was US$296.0 million. The revised estimate with AF was US$419.4 million. The actual cost was US$374.3 million. This component planned to upgrade 100 kilometers (km) of the existing two-lane Baku - Shamakhi road to a four- lane motorway and related road safety activities. Activities in this component: (i) upgrading the Baku-Shamakhi road; and (ii) road safety works. The scope of this component was expanded with the project restructuring in 2013. 48 kilometers (km) of new motorway construction on the Yenikend - Shorsulu road was added. In parallel, the length of the Baku - Shamakhi road was reduced from 100 km to 77 km due the geotechnical issues at the western end of the road that was not foreseen at appraisal. The project scope was expanded with the AF, to upgrade 94 km of the Baku-Shamakhi road (from 77 Km). 2. Institutional development. The estimated cost at appraisal was US$5.0 million. The revised estimate after the AF was US$25.0 million.The actual cost was US$15.6 million. Activities in this component: (i) preparing a study to explore policy options on management and financing; (ii) institutional strengthening of the AzerRoadService (ARS - road administration unit in the Ministry of Transport (MoT), responsible for managing motorways; (iii) establishing guidelines for motorways; (ii) testing an experimental motorway Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) unit at ARS; and (iii) operations and maintenance pilot on sections of motorways; and (iv) financing traffic related equipment, training, and study tours on to the MoT, ARS, and officials from other government agencies. The scope of this component was expanded with the AF to provide additional support for establishing dedicated Regional Maintenance Units (RMUs) for motorway maintenance activities, a Motorway Management Unit, constructing RMU offices, road maintenance depots and road maintenance equipment. With RMUs, the activity of planned pilot on motorway operations and maintenance became irrelevant and was therefore cancelled. 3. Project Management. The estimated cost at appraisal was US$1.0 million. The revised estimate after the AF was US$4.0 million.The actual cost was US$3.2 million. This component planned to support the management capacity of the Project Implementation Unit (PIU). The scope of this component was expanded with the AF to increase institutional support to update the road administration's road database, and conduct a road user survey. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The estimated cost at appraisal was US$356.1 million (inclusive of contingencies). The revised estimate with the AF was US$477.8 million. The actual cost was US$394.0 million. Project financing. The original Bank financing for the project was US$241.6 million. AF of US$140.0 million (IBRD loan) was approved on April 7, 2016. With this, the total Bank financing for the project was US$381.6 million. The Bank disbursed US$311.2 million. US$64.0 savings realized due to the devaluation of the local currency between 2015 and 2017 were cancelled by the Bank at government request. There was parallel financing for complementary road rehabilitation activities on the corridors from the Asian Development Bank (ADB) and the European Bank for Reconstruction and Development (EBRD). Recipient contribution. The recipient contribution was planned at US$114.5 million at appraisal. Their contribution was more than planned at US$153.7 million during implementation. Dates. The Bank approved the project on 25 May, 2010, and the project became effective on 8 November 8, 2010, with an original closing date of 31 March, 31, 2015. AF was approved on 7 April, 2016. The project closed on 31 March, 2021. Other changes. Besides the AF, there were four Level 2 restructurings during the project lifetime. The changes made during the project are listed sequentially as follows. The changes made through the first restructuring on 4 December, 2013.  A new activity - upgrading 48 km of the corridor from Yenikend - Shorsulu - was added. The restructuring reduced the scope of the Baku-Shamakhi road from 100 km to 77 km, due to complex geotechnical issues that was not foreseen at appraisal. The savings realized from the reduced scope of the Baku-Shamakhi road were allocated to funding the Yenikend - Shorsulu road. The PDO was revised to reflect the new activity. Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023)  The government's co-financing share was increased from 20 percent to 25 percent, and the closing date was extended by 18 months from 31 March, 2015, to 30 September, 2016, for completing the new activity. The changes made through the AF approved for the project on 7 April, 2016.  The PDO was formally revised to reflect the increase in project scope.  As discussed in section 2d, the project scope of all components was increased (including the scope of the Baku-Shamakhi road from 77km to 94 km, and support for establishing the Regional Management Units).  The closing date for the original project was extended by eighteen months from 30 September, 2016, to 31 December, 2019, to coincide with the closing date for the AF project. The Bank supported the following changes through the second Level 2 restructuring on 31 December, 2019.  extending the closing date by six months from 31 December, 2019, to 30 June, 2020, for completing ongoing activities associated with the RMU. The Bank supported the following changes through the third Level 2 restructuring on 5 June, 2020.  US$64.0 savings realized due to the devaluation of the local currency between 2015 and 2017 were cancelled at government request.  The closing date was extended by nine months from 30 June, 2020, to 31 March, 2021, for completing the activities relating to establishing the RMUs that were delayed due to the restrictions following the COVID-19 pandemic, and for preparing a new project to further develop connectivity for about 250,000 people in the Salyan - Bilsuvar area in the southern part of Azerbaijan. 3. Relevance of Objectives Rationale Country and sector context. The PDOs were relevant to the development challenges facing Azerbaijan. Azerbaijan's economic growth before appraisal was mainly due to the oil and gas boom that was expected to last for about twenty years. Diversifying into the non-oil sector through developing the natural resources and the tourism potential of the country was important to the government strategy for transforming Azerbaijan to a sustainable market economy. The road mode dominated the transport sector, with roads accounting for 60 percent of freight, and almost all passenger transport. Although the length of the road network (25,000 kilometers (km) of the country was adequate, their quality was poor, with only 45 percent of the Major arterial (M) roads, 30 percent of the minor arterial roads (R) roads, and nine percent of tertiary (Y) unpaved roads in good condition. The network condition was fragile due to inadequate road maintenance, vehicle axle overloading, and Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) poor road safety. (Azerbaijan had a poor road safety record and the number of people hurt or killed in traffic accidents was expected to increase as car ownership and traffic increased). Government strategy. The PDOs were well aligned with the Government strategy, articulated in the " State Program for Rehabilitation and Development of Azerbaijan Republic's Road Network for 2006- 2015" at appraisal. This strategy aimed to develop the main arterial roads to international motorway standards for technical and geometrical specifications, traffic safety and signage, road-user amenities and a sustainable funding mechanism for road maintenance. The ICR does not provide details on whether this strategy was updated. Bank strategy. The PDO was well aligned with two pillars of the Bank's Country Partnership Strategy (CPS) for 2007 - 2010: (i) support sustainable and balanced growth of the non-oil economy; and (ii) increase social services and access. The CPF specifically highlighted the need for transport infrastructure investments to develop Azerbaijan's non-oil growth potential. The PDO was also well aligned with the current Country Partnership Framework (CPF) for 2016-2019. This CPF was current when the project ended in March 2021 (ICR, paragraph 34). The second focus area of the CPF articulated the need for improving economic competitiveness through improving connectivity along the main highways. The CPF recognized the importance of rehabilitating corridors in allowing producers to reach both local and international markets, and underscored the need for introducing modern operation and maintenance management practices in the road sector. Previous Bank experience. This project was the third phase of the Bank support to the road sector in Azerbaijan. The first Azerbaijan Highway project that closed on 2009, rehabilitated parts of the East-West Highway, and reorganized road maintenance activities. The second Highway project that closed on 2015, aimed to improve transit access and road safety within Azerbaijan's East-West and North South corridors through upgrading sections of the Alat -Hasalli road on the Baku-Iran Highway and sections of the Baku- Shamakhi road. The scope of this project was similar to the previous highway projects financed by the Bank. Accordingly, the relevance of the PDO is assessed as substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To contribute to more efficient Baku-Shamakhi and Yenikend-Shorsulu roads. Rationale Theory of change. The outputs such as upgrading sections of the two-lane Baku - Shamakhi and Yenikend - Shorsulu roads to four lanes, were expected to improve their quality, reduce road user costs, and aid in Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) the long-term development outcome of sustaining growth of the non-oil sector. The theory of change assumes that the government remains committed to motorway reform, including through improving road maintenance through setting a dedicated mechanism for road maintenance, and that the technical challenges could be economically solved in the mountainous region. The causal links between project activities, outputs and outcomes were logical. The outcomes were monitorable. Outputs (pages 9 - 11 and pages 32- 33).  94 km of the M4 highway from Baku to Shamakhi (linking Baku with central Azerbaijan) was widened from a single carriageway to a dual carriageway, as per the revised target, and slightly short of the original target of 100 km. The activities entailed stabilizing slopes in areas prone to landslides, and building footbridges and culverts and underpasses. Passenger car speeds along this section of the road increased by 40 percent, compared to the pre-project times.  48 km of the M3 highway from Yenikend - Shorsulu (linking Azerbaijan's northern central and central regions with the Iranian border in the south) were upgraded as targeted. The activities included a dual carriageway along a new alignment, grade-separated interchanges, and rest areas for road users.  144 km of roads were constructed, exceeding the target of 142 km.  The existing bridge over the River Kur was rehabilitated to maintain accessibility for local and non- motorway traffic.  About 25,000 person-months of employment were generated. Outcomes (ICR, paragraphs 39 and 41). The outputs were expected to reduce the average journey times and thereby reduce road user costs.  The average time for road users along the upgraded sections of the Baku - Shamakhi road reduced by 19 minutes, and the average road user cost reduced by 30%. This exceeded the target of 20%.  The average journey time for road users on the upgraded sections of the Yenikend - Shorsulu road reduced from 52 minutes to 32 minutes, and the average road user cost reduced by 23%. This exceeded the target of 20%.  A road user survey conducted at closure (discussed in detail below) indicated that 79% of the respondents were satisfied when the project closed, as compared to 19% at the baseline. The realized outcomes were attributable to project activities. Therefore, efficacy of this PDO is assessed as substantial. Rating Substantial OBJECTIVE 2 Objective To contribute to a safer Baku - Shamakhi and Yenikend - Shorsulu road. Rationale Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) Theory of change. The outputs of activities, such as improving road safety standards on the upgraded sections of the Baku - Shamakhi and Yenikend - Shorsulu highways, conducting road safety audits, were likely to reduce road accidents, and thereby contribute to the PDO of safer roads. The causal links between project activities, outputs and outcomes were logical. The intended outcomes were monitorable. Outputs (ICR, paragraphs 43 and 44).  Dual carriageway roads on the Baku - Shamakhi and Yenikend - Shorsulu highways were constructed, with traffic segregated by separating carriageways. Both metallic and concrete safety barriers were used in the median and at the roadside. Hazardous left turns were restricted, and replaced with lanes dedicated to turning traffic. In urban areas, footbridges were constructed, and sidewalks installed to separate pedestrians and traffic.  Road safety audits on design and safety audits on construction were completed as targeted. The ICR (paragraph 43) observes that such audits were not routinely used for state budget financed projects in Azerbaijan before the project. Outcomes (paragraph 44). The outputs of the activities described above, were expected to contribute to the outcome of safer roads. The ICR (page 19) observes that a World Bank study conducted in Azerbaijan in June 2021 (Socio-Economic Cost and Human Impacts of Road Accidents in Azerbaijan) estimated the socio-economic cost and human impacts of road accidents at 1.6 billion Azerbaijani Manat (AZM), equivalent to 2.0 percent of Gross Domestic Product (GDP). The study also notes that the number of road accidents in 2019 was 14 percent higher than in 2018.  The number of people killed or injured from road accidents (per 100 million vehicle- km) on the Baku - Shamakhi road decreased by 71%, far exceeding the benchmark of 30% when the project closed.  The number of people killed or injured from road accidents on the Yenikend - Shorsulu road decreased by 76%, far exceeding the benchmark of 30%. This outcome was mainly attributable to the project activities, as the reduction in road accidents on the project-intervened areas was substantially higher than the national trend, where fatalities due to road accidents decreased by only 25 percent over the past decade. The ICR (page 43) notes that nationwide road fatalities decreased by 21% from 2009 - 2019, and by 32% if measured from 2010 - 2020). The efficacy of this PDO is assessed as substantial. Rating Substantial OBJECTIVE 3 Objective To contribute to higher quality services as part of the general network upgrading to motorway standard. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) Rationale Theory of change. The outputs of activities aimed at institutional strengthening of the road agency for preparing policy options on management and financing, and developing guidelines for constructing motorways, their operations and maintenance, were likely to improve their capacity to provide higher road quality services. The theory of change explicitly assumes that government would remain committed to motorway reform including through for funding road maintenance. The causal links between the project activities, outputs and outcomes were logical. The intended outcomes were monitorable. Outputs (ICR, pages 34 - 35). In addition to the activities described above, the following outputs were relevant for this objective.  Motorway operational guidelines were developed as targeted.  Motorway management and maintenance strategy was developed as targeted.  The project aided in developing a road database and providing capacity building support for the road agency's in-house data collection unit. Outcomes. The activities described above were expected to contribute to higher road quality services. Roads in the project-intervened areas that were in good condition increased from 55% at the baseline to 90% when the project closed, exceeding the target of 70%. A Road User satisfaction survey was conducted on 800 residents and road users to provide feedback on the quality of road services, both before and after upgrading the roads (ICR, paragraph 51). Topics for the survey, included road and sidewalk condition, signage and markings, drainage condition, pedestrian safety features, and the availability of bus stops and pedestrian crossings. The main conclusions of the survey are as follows:  Overall 79% of the respondents expressed satisfaction when the project closed (as compared to a baseline of 19%).  89% percent of the respondents were satisfied with the upgraded Yenikend - Shorsulu road and 73% with the upgraded Baku - Shamakhi road.  73% and 72% of the female and male respondents were satisfied with the quality of roads.  77% of the residents and 80% of the travelers were satisfied with the quality of roads. The efficacy of this objective is assessed as substantial. Rating Substantial OBJECTIVE 4 Objective To improve management of the nascent motorway network. Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) Rationale Theory of change. The outputs of activities described above, together with the outputs such as establishing regional dedicated regional management units (RMUs) under a central motorway management unit for maintaining the magistral (motorway) roads, were expected to improve management of the nascent motorways (until then local units were responsible for maintaining the magistral roads). The causal links between the project activities, outputs and outcomes were clear, and the intended outcomes were monitorable. Outputs (ICR, paragraphs 47 - 49). In addition to the outputs described above, the following outputs were relevant for this objective.  The project provided technical assistance to support the planning and capacity building for establishing RMUs and for introducing performance-based service-level agreements (SLA) between the RMUs and the Motorway Management Unit of the road administration (MMU). According to the clarifications provided by the team, before the setting of RMUs, road maintenance was undertaken by more than sixty small locally based maintenance units responsible for the roads passing through the respective administrative areas. The reforms removed the specialized task of maintaining the motorways from the local units to the RMUs. Each RMU was responsible for the routine, emergency and winter maintenance along the corridor. The task team leader also clarified that the RMUs are managed and funded independently from the local maintenance units and the Road Agency approved designated budgets for each RMU based on the level derived by the project consultants.  Three priority RMUs were established and operational when the project closed, for maintenance planning and budgeting and maintaining Service Level Agreements (SLA) with the MMU for maintaining the Magistral roads. According to the clarifications provided by the team, the SLAs require the service provider (the RMUs) to undertake maintenance tasks and achieve a defined level of service (such as, replacing damaged sections of guard rail within 48 hours and filling potholes within seven days). The SLAs also contains penalty clauses that reduce monthly payment if service level targets are not met. The ICR (paragraph 48) notes that four more are operational and in the process of transitioning to SLAs. Outcomes. The outputs of the activities described above were expected to increase the km of roads under SLA with the MMU.  774 km of the highway roads of the RMUs were under SLA with the MMU, exceeding the target of 500 km. According to the clarifications provided by the team, the concept of delivering a defined level of service was new to the road sector in Azerbaijan and was in alignment with the way the Government intends to manage state-owned enterprises. The team also clarified that to the best of the team's knowledge, this way of delivering services was applied for the first time among the state-owned enterprises. Efficacy is assessed as substantial. Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The intended outcomes that were attributable to the project activities were realized. Therefore, overall efficacy is assessed as substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis. An economic analysis was conducted at completion separately for upgrading the Baku- Shamakhi and the Yenikend - Shorsulu segments of the highways, using the Highway Development and Management (HDM - 4) model. These components accounted for 88% of the appraisal estimate and 96% of the actual cost. The project benefits were assumed to come from vehicle operating cost savings, travel time savings for road users, and accident savings costs. Baku - Shamakhi segment. The ex-post Economic Internal Rate of Return (EIRR) was 13.2 percent, short of the estimated ex- ante EIRR of 14.5 percent. The Net Present Value (NPV) when the project closed was US$105.7 million. The NPV at closure is not comparable to the NPV at appraisal, as it was based on different assumptions about the cost of capital as the project progressed. Yenikend - Shorsulu segment. The ex-post EIRR was 13.2 percent, well short of the estimated ex-ante of 17.3 percent. The NPV when the project closed was US$13.4 million. The lower than expected returns on the Yenikend - Shorsulu road was due to factors such as, low traffic growth along the corridor as the economy contracted in 2015-2016 and its slow recovery, compounded by the restrictions on movements following the COVID - 19 pandemic. The ICR also notes that with the economy rebounding faster than expected (4.8% during January - September 2021), traffic on this road network is likely to increase faster than expected. The actual cost was well within the cost estimated at appraisal. The unit cost of upgrading the Baku - Shamakhi road was US$2.47 million per km, substantially less than the US$2.96 per km anticipated at entry, due to efficiencies that were achieved through engineering design that avoided complex and expensive technical solutions in the landslide areas and on sections with steep inclines (ICR, paragraph 57). 94 km of this highway were upgraded for a total cost of US$232.0 million, compared to the PAD estimate of US$296.0 (for 100 km) (ICR, paragraph 63). The unit cost of upgrading sections of the Yenikend - Shorsulu road was US$2.67 million per km, as compared to the estimated cost of US$3.12 million per km estimated in the October 2013 restructuring paper (ICR, paragraph 57). 48 km of this highway were upgraded for a total cost of US$129.0 million, US$24.0 million less than estimated in the restructuring paper. The yearly average supervision cost was US$121,422 as compared to the US$120,000 estimated at entry. The slightly higher supervision cost was due to Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) a combination of factors including upscaling of the project and the extended implementation period. Total savings of US$64.0 million were realized due to the devaluation of the local currency vis-à-vis the US$ from 2015 - 2017 and were cancelled at government request. Administrative and operational issues during implementation. The earlier stages of the project saw less than optimal implementation efficiency, as evidenced by slower than anticipated disbursement rate. The reasons for this were: (i) issues with technical design and procurement delays; (ii) the first works contract for the Baku- Shamakhi road was awarded six months after project effectiveness despite its readiness at appraisal and work on the subsequent sections (km 45 to 67 and from km 67 to 91) took a further two years to commence; (iii) there were serious problems due to the poor performance of a contractor, which the Bank was unable to resolve and the contractor had to be replaced; and (iv) issues relating to the financial capacity of the contractors due to the sharp devaluation of the domestic currency vis-à-vis the US$ (the devaluation led to steep increases in both domestic and foreign costs which was problematic because the contracts were denominated in only local currency and did not include price adjustment provisions. The ICR (page 22) notes that price adjustment provisions were not used as per the recommendations of the Ministry of Finance to minimize the price escalation under such contracts. This aggravated the cash flow problems being experienced by most contractors). This issue was eventually resolved by the Bank. The delays were exacerbated by adverse external factors over which the project had no control, such as a banking and economic crisis, currency volatility, country's involvement in a major military conflict, and physical restrictions following the COVID - 19 pandemic. However, these challenges were addressed and project activities completed were completed with extension of the closing date (the closing date for the original project was extended by 18 months and the closing date for the AF project was extended for a cumulative 15 months). Overall, efficiency is assessed as modest in view of the administrative shortcomings during implementation. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 88.00 Appraisal  14.50  Not Applicable 96.00 ICR Estimate  13.20  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) The overall outcome was moderately satisfactory, based on the substantial ratings for the relevance of the PDO, efficacy and modest efficiency. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Government commitment. This project supported activities for establishing Regional Management Units (RMUs) for managing motorways through performance-based road maintenance practices. There is however a modest risk to the development outcome, as the service level agreements still remain in their early stages in Azerbaijan, and so the long-term adoption of this approach will depend on the benefits realized by the road agency. Further, the development outcome is contingent upon support through annual budget allocations. The ICR notes that given the significant financing needs for post-conflict reconstruction, the state budget for road maintenance could be undermined due to shifting priorities towards rehabilitating infrastructure in the reacquired territories (the team clarified that the territories refer to those reacquired in Azerbaijan's during the country's 2020 conflict with Armenia). 8. Assessment of Bank Performance a. Quality-at-Entry The Bank prepared this project based on the experiences from the previous Bank-financed highway project and the ongoing Second Highway project. Lessons incorporated at design included; (i) ensuring readiness of works activities at the start of the project. Disbursements in the second Highway project had been slow initially, due to unforeseen technical and procurement delays. For the Baku- Shamakhi section in this project, a feasibility was undertaken in advance and design documents for the first 30 km of this section was ready at effectiveness (since no land acquisition was required for this section). Despite this, as explained in section 5, the first works contract for this section was delayed by six months due to the technical design and procurement delays; (ii) Limited technical assistance activities focused on managing motorways; and (iii) adequate implementation arrangements were in place (discussed below) (PAD, paragraphs 29 - 31). The Bank prepared this project in collaboration with ADB and EBRD, given the complementary road rehabilitation activities on other sections of the corridors financed by these donors.(PAD, paragraph 34). The implementation arrangement were appropriate. The AzerRoadService (ARS), under the supervision of the Ministry of Transport (MoT) was responsible for project implementation. ARS and the MoT had executed Bank-financed projects and were familiar with Bank policies. The Project Implementation Unit (PIU), which was executing the Second Highway project, was also retained for this project. Since the Second Highway project also entailed upgrading from two lanes to four lanes, the ARS and the PIU were familiar with the technical, geographical and safeguard issues (PAD, paragraph 34). Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) The preparation team identified several risks at appraisal, including substantial risks associated with the weak implementation capacity, fiduciary and safeguards risks. Mitigation measures at design, included hiring international consultants to support ARS and the PIU, and targeted technical assistance activities to strengthen motorway management. Regarding the mitigation measures, the overall project risk was rated as moderate at appraisal (PAD, paragraph 39). The arrangements made at appraisal for safeguards and fiduciary compliance were appropriate (discussed in Section 10). However there were minor shortcomings at Quality-at-Entry. One, the mountainous section of the M4 corridor had not been designed in detail at appraisal. This was probably due to the strategic and political pressure for the project to proceed before the detailed design had been completed. Nevertheless, a major geotechnical design had to be undertaken later; and two, there were some issues with the M&E framework (discussed in section 9). Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank supervision team conducted supervision missions twice a year, with twenty three missions during the project lifetime (ICR, paragraph 118). The continuity of leadership was maintained, with just two Task Team Leaders (TTLs) during the project lifetime. The TTL or the Co TTL and the procurement, financial, environmental and social specialists were located in the field. This helped the team to support the Government and the road administration to quickly react to changing circumstances, such as in dealing with the issue due to the devaluation of the local currency vis-à-vis the US$ between 2015 and 2017 (this issue was problematic, as the contracts were denominated in local currency with no provisions for price adjustments. The crisis also hit the banking sector, and aggravated the cash flow problems being experienced by most contractors). The Bank team in consultation with the implementing agencies, designed measures to support the heavily affected contractors. The team was proactive in making the appropriate decisions on project restructuring (such as, strengthening the institutional components of the project and adjusting the M&E design shortcomings (discussed in section 9). The support provided by the team aided in safeguards and fiduciary compliance (discussed in section 10). Overall Bank performance is assessed as satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) a. M&E Design The results framework was clear. Of the three key outcome indicators - reduction in road user costs - was appropriate for monitoring the PDO of more efficient roads. The second key indicator - the percentage reduction in accidents on the roads - was appropriate for monitoring the PDO of safer roads. However, this indicator needed modification, as the indicator did not differentiate between fatalities and injuries due to road accidents. The third key indicator measured the network condition and did not differentiate between improvements gained through capital investment (road construction) or improvements due to better managed maintenance. b. M&E Implementation The shortcomings in M&E design were rectified. The indicator pertaining to road accidents combined fatality and injury data and used multi-year average numbers (to disaggregate the benefits attributable to the project from broader national safety trends).A fourth indicator was introduced during the 2016 restructuring that was necessary to reflect the increased scope of this project. However, the indicator on the percentage of road safety audits was dropped due to weaknesses in the traffic police crash system data system (ICR, paragraph 104). The PIU was responsible for data collection. The ICR (paragraph 102) notes that as the PIU was initially overseeing two Bank projects, a full-time M&E specialist was hired after project restructuring. The strengthened PIU since then had the required knowledge for monitoring the project and this enabled accurate assessment of the implementation progress. The M&E arrangements were strengthened through undertaking a road user satisfaction survey, to assess the public's perception of road use and the new facilities. The Bank team also employed a tool to monitor the physical progress of civil works activities through the Innovative Project Assessment Tool (PAT). The ICR (paragraph 105) notes that this tool efficiently tracked progress. c. M&E Utilization The ICR (paragraph 106) notes that data on all indicators in the Results Framework were regularly updated and reported in the six-monthly Implementation and Status Results (ISR) reports. These data were used for tracking implementation progress and taking remedial actions, and the M&E system was sufficient to assess the achievement of the objectives. Overall, M&E is assessed as substantial. Although there were minor shortcomings in M&E design, they were rectified during implementation. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) The project was classified as a Category A (full assessment) project under the World Bank safeguard policies, triggering two safeguard policies at appraisal: Environmental Assessment (OP/BP 4.01); and Involuntary Resettlement (OP/BP 4.12). (PAD, page 17). Environmental Assessment. The PAD (paragraph 54) notes that an Environmental Management Framework (EMF), and a Regional Environmental Review (RER) prepared for the first Highway Project in 2005, were updated in 2009. The EMF outlined the procedures for environmental screening, management, consultation, and disclosure of sub-projects. The RER identified the potential environmental risks, and the issues to be addressed by the site-specific Environmental Impact Assessments (EIAs) and the Environmental Management Plans (EMPs). The EMF and RER were publicly-disclosed. As the detailed design had only been developed for 30 km of the road at appraisal, site-specific EIA and EMP were prepared for these roads and publicly-disclosed at appraisal. The ICR (paragraph 108) reports that site- specific EIAs and EMPs were prepared for all road sections and updated during implementation. The team clarified that the project complied with environmental safeguards and environmental compliance was rated as satisfactory when the project closed According to the ICR (paragraph 108), the site staff included a safeguards specialist to oversee environmental issues. Both contractors and consultants were contractually required to employ experienced staff to manage the environmental and social safeguards at the site level. Involuntary Resettlement. The PAD (paragraph 51) notes that for land acquisition was not required for most road sections, as the roads could be expanded within the existing right-of-way. Land acquisition was however foreseen for 9.75 km of road where the land was owned by private owners, and for constructing a new bypass around Shamakhi. Since the exact alignment was only known for the first 30 km road section which did not require land, a Resettlement Policy Framework (RPF) was prepared and publicly disclosed at appraisal. The ICR (paragraph 109) notes that the road administration prepared a Resettlement Action Plan (RAP) which was publicly-disclosed during implementation. The project complied with the social safeguards. The ICR (paragraph 110) reports that 441 people were affected (including fourteen people who were displaced). The affected people were compensated from state funds. The PIU ensured that owners and users of the impacted land were fully compensated before works commenced. However, the process was lengthy as some landowners could not be traced, and a few cases were resolved only after litigation. The project established a grievance redress mechanism (GRM), that was monitored by consultants under the oversight of the PIU. Resettlement and land acquisition-related grievances were also received directly by the land acquisition unit of the road administration. The ICR (paragraph 111) notes that most cases (such as accidental damages to irrigation or drainage pipes), were resolved on-site by the contractors, while the remainder were addressed by the PIU. b. Fiduciary Compliance Financial management. The AzerRoadService (ARS) was responsible for financial management. The Bank conducted an assessment of ARS's financial management arrangements. The arrangements were deemed to be satisfactory and the financial risk was rated as moderate at appraisal (PAD, paragraph 48). Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) The project complied with the financial management legal covenants (ICR, paragraph 112). Financial arrangements were strengthened during implementation through hiring an additional dedicated financial management specialist for the PIU. The financial audits were consistently submitted on time, and all payments under the project were completed with no debt remaining when the project closed. The team clarified that none of the audits were qualified. Procurement. The Bank conducted a procurement assessment of the Project Implementation Unit (PIU) at appraisal. The PIU was already executing the ongoing Bank-financed Second Highway project and had demonstrated capacity for managing procurement. The assessment concluded that the procurement risk was moderate (PAD, paragraph 47). The procurement activities was managed by experienced procurement specialists at the PIU (ICR, paragraph 114). Although some evaluation procedures took longer than planned, there were no serious issues and the ICR does not report of any misprocurement. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR draws the following main lessons from implementing this project, with some adaptation of language. 1. To ensure early disbursement, it is helpful to have more than one high value activity that can commence at effectiveness or shortly afterwards. This project became effective with the design and procurement ready completed for 30 km stretch of the Baku - Shamakhi road while other sections were still under preparation. 2. The introduction of modern management approaches for road maintenance is feasible for state-owned entities, but requires time. The development objectives of this project were initially Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) modest. However, the Bank provided options and engaged with the Government in a long-running dialogue on the maintenance of roads. The Government eventually adopted the recommendations on the modernization of motorway management and maintenance, and this led to the Service Level Agreements between the Regional Management Units and the Motorway Management Unit of the Road Administration. 3. Implementing agencies should strongly encourage all bidders for works contracts to reflect their foreign currency costs , regardless of historical exchange rate performance. The sudden devaluation of the local currency, during project implementation created cash flow problems for several contractors, as they had priced the work in local currency but had costs in foreign currencies. While the standard World Bank bidding documents used for the project allowed bidders to include foreign currencies, none did so as the local currency had previously been very stable. Consequently, contractors were over-exposed to the effects of currency fluctuations. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is clear and laid out with some useful photographs of the "before" and "after" situations. The theory of change provides a logical elucidation of the causal links between project activities, outputs and outcomes. The quality of evidence provided in the ICR (such as the data on road accidents) is derived from a credible source. Overall, the ICR is internally consistent and logically links the various parts of the report. It also candidly discusses the issues facing the contractors due to the unexpected devaluation of the currency and draws good lessons from the experiences in implementing the project. One minor issue is the length of the ICR. The main body of the text at 29 pages is almost twice the recommended length of 15 pages. The ICR would have benefitted from better editing. a. Quality of ICR Rating Substantial Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AZ HIGHWAY 3 (P118023) Page 18 of 18