Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) Report Number: ICRR0023491 1. Project Data Project ID Project Name P145545 Enhanced Road Access Project Country Practice Area(Lead) Samoa Transport L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H8840,TF-17563,TF-A6371 30-Apr-2019 27,602,364.24 Bank Approval Date Closing Date (Actual) 17-Oct-2013 31-Oct-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 20,000,000.00 12,225,008.00 Revised Commitment 29,629,479.18 9,629,479.18 Actual 27,671,096.16 9,629,479.18 Prepared by Reviewed by ICR Review Coordinator Group Gouthami Padam Vibecke Dixon Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The PDO, as stated in the PAD (Page 6) and Financing Agreement (Schedule 1, Page 5), was to “restore key road sector assets damaged by extreme weather events and enhance the climate resilience of critical roads and bridges in Samoa.” Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) For the purposes of this evaluation, the PDO will be assessed as is and will not be unpacked. This is because while building back better, restoring key road sector assets is an inseparable part of enhancing climate resilience. This is also noted in the ICR (Para 23). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components Road and Bridge Reconstruction: (estimated cost US$28.60 million; Actual Cost: 24.87 million) This component aimed to repair roads and bridges damaged by Tropical Cyclone Evan (TCE) and upgrade, rehabilitate, and/or reconstruct existing road sector assets to higher standards. The main activities included: (a) Widening of 2.3 km of Vaitele Street; (b) Repair to or replacement of Leone Bridge; (c) Reconstruction of other roads and bridges to be selected during implementation based on both economic and social factors; and (d) Detailed design and supervision of the civil works. Technical Assistance: (estimated cost US$2.00 million; Actual Cost: US$ 1.73 million). This component aimed to support institutional and regulatory reforms for road sector assets management and maintenance and to increase the sustainability of sector investments. This included: (a) Revision of standards for maintaining and constructing roads and bridges to be more climate resilient; (b) Update of legal, regulatory, and enforcement arrangements for enforcing axle load limits; (c) Strengthening the Land Transport Authority (LTA)’s technical capacity to effectively plan and manage the sector; and (d) Addressing emerging priority issues that could have an impact on the Government’s ability to create more climate resilient road sector assets. Project Management (estimated cost US$1.40 million; Actual Cost: US$ 1.07 million). This component aimed to support the Government in project management, including the services of a consulting firm to assist LTA in project implementation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The actual project cost is US$27.67 million, lower than the appraisal estimate of US$ 32 million. Project financing. The project was financed by IDA and trust funds. The total project cost at appraisal of US$ 32 million. The actual cost at closing was US$ 27.67 million. Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) Borrower contribution. There was no Borrower contribution for this project. Dates. The project was approved on October 17, 2013, became effective on February 20, 2014, and was scheduled to close on April 30, 2019. It was extended four times on October 25, 2018 (for 18 months), on April 22, 2020 (for 6 months), on April 20, 2021 (for 6 months) and on October 27, 2021 (for 12 months). The final closing date was October 31, 2022. Other changes. There were six level 2 restructurings during the project lifetime, including the restructuring and additional financing in May 2018. The main changes were as follows: First Restructuring: During the February 29, 2016 project restructuring, the scope of Component 1 was expanded to finance outstanding claims associated with the construction and supervision of relatively minor parts of the East Coast Inland Route, including pedestrian access routes. This restructuring also revised the end target of the second original PDO indicator on the number of ‘Bridges rehabilitated to improved standards’ from seven to six. Second Restructuring: During the second restructuring and additional financing approved on May 31, 2018, the following changes were made:  Provision of an additional financing in a net amount of US$3.36 million provided by GoA. This was needed to cover cost overruns for Leone Bridge and projected overruns for Mali’oli’o Bridge and support financing of the climate-resilient road network associated with the new access road for Mali’oli’o Bridge  All four original PDO indicators were dropped and two new PDO indicators were added: (i) improved connectivity through rehabilitation or construction of the road network to improved standards (which could include raising the road elevation, strengthening the pavement, and improving the drainage capacity) and (ii) improved connectivity through rehabilitation or replacement of water crossings to improved standards (through single span designs, raised crossings, and scour protection). The target value of the first revised PDO indicator was reduced to 3.9 km, down from 10 km for the original first PDO indicator for the kilometer of roads rehabilitated to improved standards. The target value of the second revised PDO indicator was 3, down from 7 for the original second PDO indicator for the number of bridges rehabilitated to improved standards. Third Restructuring: During the restructuring approved on October 25, 2018, the project’s closing date was extended by 18 months from April 30, 2019, to October 31, 2020. This was needed to allow sufficient time for completing primarily the civil works for Mali’oli’o Bridge, but also for the remaining activities, including the axle-load limit reforms and construction standards, and the design and construction of Moamoa Crossing, Aia Bridge, and Nu’usuatia Bridge. Fourth Restructuring: During the restructuring approved on April 22, 2020, the project’s closing date was extended by six months from October 31, 2020, to April 30, 2021. This was needed due to major implications of the COVID-19 pandemic which Samoan road projects were facing. It was needed to allow the civil works and other outstanding contracts to be completed. Fifth Restructuring: During the restructuring approved on April 20, 2021, the project’s closing date was extended by six months from April 30, 2021, to October 29, 2021. This was done to finance the rectification works of the most critical road safety hazards on the West Coast Road (WCR) financed under Enhancing Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) the Climate Resilience of the West Coast Road Project (CRWCR) and ERAP, which were identified through the post-construction road safety audit in March 2021. Sixth Restructuring: The restructuring approved on October 27, 2021 extended the project’s closing date by 12 months from October 29, 2021, to October 31, 2022. This was needed to rectify road safety hazards on the WCR and enhance the safety of Mali’oli’o Bridge, Moamoa Crossing, Aia Bridge, and Nu’usuatia Bridge, using additional project savings of around US$1.6 million identified by GoS in September 2021. 3. Relevance of Objectives Rationale Country and sector context. Samoa is a small and remote Pacific Island nation comprising two large islands, Upolu and Savai’i, and eight smaller islands. It is approximately 3,000 km from New Zealand and 4,000 km from Hawaii and Australia. The country faces significant vulnerability to climate change and natural disasters, experiencing an increase in extreme weather and climate events such as heavy rainfall, strong winds, storm surges, and rising sea levels. These events, along with earthquakes and tsunamis, have caused severe damage to infrastructure, economic assets, and livelihoods. Flooding, particularly in urban areas like the capital Apia, has been a recurring issue throughout Samoa's history. The Enhanced Road Access Project (ERAP) was prepared as an emergency response to Tropical Cyclone Evan (TCE), which hit Samoa on December 13 and 14, 2012. Over a two-day period, the cyclone resulted in five fatalities and approximately 4,250 people being temporarily displaced. The Government of Samoa (GoS) estimated the combined physical and economic losses to transport infrastructure, houses, and tourism facilities at US$204 million, equivalent to about 28 percent of the country's total value of goods and services produced in 2011. Several key road sector assets were damaged, including Central Cross Island Road (a key economic artery between North and South Upolu) and Leone Bridge (which could no longer be used by vehicles and was one of two bridges capable of carrying heavy vehicles between Apia Port and the principal industrial area at Vaitele). Alignment with Government strategy. The PDO of the ERAP project was relevant to Samoa's development priorities that were outlined in the country’s long term development strategy, “Samoa 2040”. It directly contributed to climate-resilient infrastructure investment, which was noted in Samoa 2040 as a key action to fully realize the transformative opportunities in tourism, agriculture and fishing, digital economy, and labor mobility. ERAP aligned with Key Priority Area 15 'Build Climate Resilience' and Key Priority Area 21 'Consolidated Infrastructure Management' in The Pathway for the Development of Samoa (FY2021/22– FY2025/26). The Transport and Infrastructure Sector Plan 2022/23–2027/28 emphasized improving connectivity, mobility, and accessibility through safe and climate-resilient road infrastructure, which ERAP supported. The LTA Corporate Plan 2023–2026 set strategic goals in line with ERAP's objective of enhancing and climate-proofing road networks. ERAP's focus on climate resilience and infrastructure development aligned with Samoa's broader vision for sustainable prosperity and a safe and efficient national land transport network. Alignment with Bank strategy: At project closing, the PDO of ERAP aligned with the World Bank Group's Regional Partnership Framework (RPF) for nine Pacific Island Countries, including Samoa, from FY2017 to FY2021. The RPF focused on maximizing economic opportunities, improving access to economic Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) opportunities, safeguarding incomes and livelihoods, and enhancing growth and opportunities through macroeconomic management, infrastructure, and knowledge development. ERAP was recognized as one of the projects contributing to achieving the RPF objectives, primarily supporting the fourth focus area of increasing access to basic services and improving connectivity infrastructure. It also aligned with the third focus area of strengthening preparedness and resilience to natural disasters and climate change. Previous World Bank Experience. At project appraisal, the World Bank had a successful and long-term involvement in Samoa's road sector through various past and ongoing projects, such as the Samoa Infrastructure Asset Management Program SIAM - Phase 1 (1999-2003) and Phase 2 (2004-2014), the Post-Tsunami Reconstruction Project (PRTP (2010-2016)). These projects supported reforms and technical assistance to help manage road maintenance and rehabilitation works, establishment of the LTA and other reforms that were in line with Samoa's strategic direction. The EARP also aimed to continue the Bank's engagement with Samoa in the road sector, specifically supporting post-TCE rehabilitation efforts. Level of the PDO. There is a clear alignment between the project’s development objectives and the country- and WB strategies. Even though the project was initially prepared as an emergency project, given Samoa’s high vulnerability to climate change and natural disasters, the objective of “restoring key road sector assets damaged by extreme weather events and enhancing the climate resilience of critical roads and bridges in Samoa” is highly relevant. While the PDO formulation of "restoring key roads" is pitched relatively at an output level and not at a higher development outcome level, this is justified given that the project was prepared as a response to an emergency. Hence, this review rates the relevance of the PDO as High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To restore key road sector assets damaged by extreme weather events and enhance the climate resilience of critical roads and bridges in Samoa Rationale Theory of change. ERAP aimed to restore key road sector assets damaged by extreme weather events and enhance the climate resilience of critical roads and bridges in Samoa by focussing on ‘hard’ infrastructure investments combined with ‘soft’ capacity development activities. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) The project was to finance activities including reconstruction and repair of roads and bridges in the areas affected by the cyclone; provision of technical assistance including the revision of national road and bridge standards, updating axle-load limit standards and enforcement; and other capacity development activities. The expected outputs from these activities included agreement on key roads for reconstruction and identification of specific bridges to be improved. These outputs were expected to lead to the outcomes of rehabilitated roads and water crossings to improved standards, reduction in disruptions in traffic flows at targeted locations, and project designs being informed by access audits. The anticipated long-term impact would be the provision of safer and more efficient transport services, improved reliability of road and bridge infrastructure in the event of a natural disaster and increased access to economic activities and social services. Overall, the theory of change (TOC) of the project was straightforward and the causal pathways from inputs to outcomes were valid and direct Outputs  Works for three roads out of the originally targeted five were finished (target not achieved). This indicator was dropped at the restructuring in May 2018.  Works for five water crossings of the originally targeted six were finished (target almost achieved). This indicator was dropped at the restructuring in May 2018. Outcomes:  5.72 km of the project roads were rehabilitated or constructed as opposed to the original target value of 10 km (target not achieved).  Five water crossings were rehabilitated to improved standards, compared to the original target of seven (target not achieved). This indicator was dropped, and a similar indicator measuring the number of water crossings replaced or rehabilitated was added to the intermediate level, both at the restructuring in May 2018.  Disruptions in traffic flows were reduced at seven locations (Vaitele Street, WCR Lot 2b, Mali’oli’o Bridge and access road, Leone Bridge, Moamoa Crossing, Aia Bridge, and Nu’usuatia Bridge), exceeding the original target of zero. This indicator was dropped, and a same indicator (with a baseline of 3 and target of 4) was added to the intermediate level, both at the restructuring in May 2018.  Eight project designs were informed by access audits, exceeding the original target of seven. This indicator was dropped, and a same indicator was added to the intermediate level at the restructuring in May 2018. The project managed to successfully reduce disruptions in traffic flows at the targeted locations and inform project designs by access audits, but fell short of achieving its targets for rehabilitation and construction of roads and bridges to improved standards. Therefore, efficacy before restructuring is rated as Modest. Rating Modest Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) OBJECTIVE 1 REVISION 1 Revised Objective To restore key road sector assets damaged by extreme weather events and enhance the climate resilience of critical roads and bridges in Samoa There was no change in the PDO formulation, but the PDO indicators and targets were revised during the restructuring in May 2018. It is to be noted that the initial purpose of the PDO was to respond to a particular disaster, however, as the project progressed, its focus somewhat shifted to enhancing the climate resilience of crucial roads and water crossings to improve connectivity. This shift was reflected in the updated PDO indicators in 2018. While the ERAP’s only emergency works for Leone Bridge were completed by then, the PDO remained unchanged due to the Samoa’s high and growing vulnerability to climate change and natural disasters. Revised Rationale Theory of change. While the project’s PDO remained the same through the restructuring and additional financing of 2018, the outcome indicators and intermediate indicators were revised. The key activities that the project was to finance remained the same and included reconstruction and repair of roads and bridges in the areas affected by the cyclone; provision of technical assistance including the revision of national road and bridge standards, updating axle-load limit standards and enforcement; and other capacity development activities. The expected outputs from these activities included key rehabilitated roads and water crossings to improved standards; reduction in disruptions in traffic flows at targeted locations, project designs being informed by access audits; gender responsive features incorporated into design; and grievances related to delivery of project benefits that are addressed. These outputs were expected to lead to the outcomes of improved connectivity through roads being constructed or rehabilitated to improved standards, improved connectivity through rehabilitation or replacement of water crossings to improved standards, improved road safety for roads and water crossings and strengthened enabling environment for planning and managing the road sector and other sector issues. The anticipated long-term impact would be the provision of safer and more efficient transport services, improved reliability of road and bridge infrastructure in the event of a natural disaster and increased access to economic activities and social services. Under the revised TOC, the causal pathways between activities, outputs and outcomes were clearer and more direct and there were no logical gaps. The revised output and outcome indicators were better aligned with the PDO, were more measurable and could be monitored. Outputs:  1.60 km of a road constructed, achieving the revised target of 1.60 km.  4.12 km of roads rehabilitated, exceeding the revised target of 2.3 km.  Five water crossings replaced or rehabilitated, exceeding the revised target of three.  Traffic flows at seven locations less disrupted, exceeding the revised target of four. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545)  Eight designs for Leone Bridge, Central Cross Island Road, Vaitele Street, Mali’oli’o Bridge and access road, WCR Lot 2b, Moamoa Crossing, Aia Bridge, and Nu’usuatia Bridge informed by access audits, exceeding the revised target of five.  100 percent of grievances registered related to delivery of project benefits were addressed (Revised target fully achieved)  Gender-responsive design features incorporated into seven designs for Central Cross Island Road, Vaitele Street, Leone Bridge, Mali’oli’o Bridge and access road, Moamoa Crossing, Aia Bridge, and Nu’usuatia Bridge, exceeding the revised target of 5. Outcomes:  5.72 km of roads were rehabilitated or reconstructed to improved standards, exceeding the revised target of 3.90 km. This included rehabilitation of 2.30 km of Vaitele Street and construction of 1.60 km of Mali’oli’o Bridge access road. In addition, 1.82 km of WCR Lot 2b, which was transferred from CRWCR to ERAP, was rehabilitated to improved standards. These investments in the project roads enhanced road network connectivity, as demonstrated in the fourth revised intermediate indicator ‘Targeted locations where traffic flows will have reduced disruptions’.  Five water crossings (at Leone, Mali’oli’o, Moamoa, Aia, and Nuusuatia) were replaced to improved standards, exceeding the revised target of three. These water crossings enhanced road network connectivity by providing year-round access to road users and is reflected in the fourth revised intermediate indicator ‘Targeted locations where traffic flows will have reduced disruptions’. In addition to the above outcomes listed in the Results Framework (RF), the ICR also reported on the following outcomes not covered in the Results Framework and hence without target values:  The ICR claims that the project enhanced road safety on roads and bridges by implementing engineering measures such as sealing road shoulders, developing bus stops, installing signages, and implementing safety features like thermoplastic line markings, rumble strips, guard rails, flexible bollards, and reflectors. A post-construction road safety assessment identified high-risk concerns related to drains and driveway culverts on WCR. To address this, the project repaired critical drains, culverts, and safety hazards on WCR, Mali’oli’o Bridge, access road, and three crossings. All of these measures adhered to global best practices that have been proven to reduce the number of serious road crashes. However, it must be noted that no specific indicators were identified to measure the outcomes and no data to support this claim was collected as part of the project. Gathering data on some indicators such as road crashes before and after implementation was challenging given that the project was also prepared as an emergency project. Nevertheless, it was communicated to the IEG review team [September 5, 2023] that the Bank is working on developing a crash database system through another ongoing project in Samoa which aims to map all road crashes in the country including historical data where available. This information should potentially allow for a future analysis on the impact of Bank investments on savings lives from road crashes in the country.  ERAP improved access to markets and social services, particularly in rural areas. This was mainly through improved connectivity by re-establishing direct routes such as the replacement of Leone Bridge, and by increasing climate resilience and helping to ensure access during extreme weather events such as at the Mali’oli’o river crossing and the three water crossings on Upolu. For instance, the Mali’oli’o river crossing used to be impassable for about ten days each year during the average wet season. This isolation cut off the local population from basic education, healthcare, cultural and religious services, job opportunities, and economic centers. A new bridge that was constructed at a Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) new location, which was more inland, has been able to provide safer and more reliable access to the local population, particularly during severe weather events. The project team has reported that this new bridge was able to hold up well and was fully passable during a recent severe storm and heavy rainfall and flooding over the past few months. In addition to reconstruction and rehabilitation of roads and bridges as targeted, the project was successful in reducing disruptions in traffic flows at the targeted locations, inform project designs by access audits, incorporate gender responsive features into design, and address grievances related to delivery of project benefits. The targets for the two main outcome indicators of improved connectivity through roads constructed or rehabilitated to improved standards and improved connectivity through rehabilitation or replacement of water crossings to improved standards were also achieved or exceeded. However, there were insufficient indicators for technical assistance activities; while some indicators covered access audits and gender- responsive design features, none of them addressed the revision of road and bridge design standards or vehicle loading laws and regulations. Based on this, efficacy of the revised objective is rated Substantial. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale As described above, the main outcomes were only partially achieved before the project was restructured. Hence, the overall efficacy is rated Modest Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale As described, the project either achieved or exceeded its PDO indicators and as well its intermediate results after the project was restructured. The overall efficacy is rated Substantial. Overall Efficacy Revision 1 Rating Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) Substantial 5. Efficiency Economic Efficiency. The efficiency analysis reported in the ICR was based on an ex-post economic analysis. The methodology used at project completion was the same as that used at appraisal and the restructuring and additional financing, and it focused on the economic appraisal of Component 1, which represented 90 percent of project costs. The economic model covered a 23-year period from 2013 to 2035, and used a discount rate of 5 percent. The quantifiable benefits included reduced vehicle operating costs, decreased passenger time costs, and fewer disruptions due to improved climate resilience. The results of the ex-post analysis showed that the ex-post economic internal rate of return (EIRR) was 18.5 percent, and the net present value (NPV) was US$29.7 million. This showed higher returns and increased net present value compared to earlier assessments both at project appraisal (EIRR of 16 percent and NPV of US$19.7 million) and at the time of restructuring and additional financing in May 2018 (EIRR of 15.6 percent and NPV of US$20.0 million) (ICR, para 35). Operational efficiency. Several aspects of the project’s design and implementation affected its efficiency. Despite some initial delays, which were mostly attributable to slow progress of design for Leone Bridge, performance challenges of the Project Management Company (PMC), and frequent changes in counterpart staff, the project managed to complete all activities before the revised closing date of October 31, 2022. The Government of Samoa (GoS) used additional time to address unexpected events like the measles outbreak, election challenges, and the COVID-19 pandemic. GoS was able to maximize the use of project savings through two restructurings, which proved to be more effective than canceling the savings and using them under Samoa Aviation and Roads Investment Project (SARIP), which started in October 2022, the same month ERAP closed. Component 2 and Component 3 of the project demonstrated efficiency, with actual costs lower than the initial estimates at appraisal, indicating successful cost management (ICR, para 36). Based on the above, efficiency is rated Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  16.00  Not Applicable 0 ICR Estimate  18.50  Not Applicable Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Assessment against the original outcome targets: Relevance of Objectives was rated High. Efficacy was rated Modest and Efficiency was rated Substantial. Based on this, the original Outcome is rated Moderately Unsatisfactory (3) Assessment against the revised outcome targets: Relevance of Objectives was rated High. Efficacy and Efficiency were both rated Substantial. Based on this, the revised Outcome is rated Satisfactory (5) A split rating is applied based on the disbursement shares before and after the project restructuring in 2018, when a disbursement share was at 63% and 37% respectively (amount disbursed was US$ 17.5 million before restructuring, and US$ 10.17 million after restructuring). The weighted value of the outcome rating under the original objectives is the outcome rating (3 for Moderately Unsatisfactory) times its weight (63%) = 1.89. The weighted value of the outcome rating under the formally revised objectives is the outcome rating (5 for Satisfactory) times its weight (37%) = 1.85. The weighted average score is the sum of the two: 1.89 + 1.85. = 3.74. Rounding up to the nearest whole number, it amounts to an overall outcome value of 4, i.e., Moderately Satisfactory on the 6-point scale. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Environmental Risk. At project completion, the risk to the development outcome was considered substantial due to the frequency of severe weather events in Samoa. ERAP significantly improved the climate resilience of the project's roads and bridges, strengthening the local capacity to develop a more resilient road network. However, natural hazards and climate risks continue to pose threats to the sustainability of the investments and the likelihood of future severe weather events in Samoa is very high, and their potential impact could be catastrophic. Even though the design of climate-resilient infrastructure investments under ERAP was guided by the Vulnerability Assessment and a Climate Resilient Road Strategy and the upgraded roads and bridges receive routine maintenance funded by the government, the increasing frequency and intensity of extreme weather events still pose a substantial risk to the project's infrastructure within its lifespan. This was already seen at Moamoa Crossing, where, as reported in the ICR, the newly completed crossing was overtopped during the severe floods in December 2020 and again after heavy rainfall in February 2022. Capacity and Institutional Risk. ERAP faced several challenges in the initial implementation stage as a result of poor capacity of the PMC and there was a need for extensive implementation support. Eventually the PMC’s responsibilities were cut back and it was replaced by a government agency, which also required intensive support. While the project was successful in strengthening the institutional capacity, there is a risk that changing circumstances and budget constraints may impact the availability of resources for local government institutions to fulfill their responsibilities in the future. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) 8. Assessment of Bank Performance a. Quality-at-Entry Despite being prepared under emergency procedures, the project was strategically relevant to the development priorities of the Government of the State (GoS) and aligned with the World Bank's guiding documents during preparation. The project design was straightforward and practical, and aimed to achieve the PDO by focusing mainly on infrastructure investments combined with technical assistance activities. During preparation, the task team took into account lessons learned from previous projects, including SIAM - Phase 2, PTRP, and CRWCR. These included designing structural investments to account for natural hazards and climate risks, using proven technologies and procedures for civil works, coordinating closely with other donors for efficient assistance delivery, and providing intensive implementation support for emergency operations. The institutional capacity risk was assessed as ‘Moderate’ at appraisal, however, the team was able to appropriately identify the risks associated with implementing a large transport infrastructure project and develop suitable mitigation measures. On the World Bank’s side, the project’s preparation was overseen by a TTL who had prior experience working on projects in similar contexts, including CRWCR. (ICR, para 72) The quality-at-entry was Satisfactory Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank team conducted 22 implementation support missions and for most of the implementation period these missions were conducted on a quarterly basis rather than semi-annually. This increased implementation support from the team was to provide support and address capacity gaps initially identified with the PMC. Eventually, the PMC's responsibilities were reduced, and a Government agency established through another WB project, took over, also requiring extensive implementation support. There were three changes in TTLs over the nine-year implementation period, which could have resulted in some loss of institutional knowledge. However, this was mitigated through strategies such as overlapping TTLs, maintaining continuity of TTLs and co-TTLs for subsequent projects, and assigning an incoming TTL who had previous experience as a TTL for a related project. The task team and the Country Management Unit actively collaborated with the GoS counterpart to address the changing needs of the country's transport sector throughout the implementation phase. The team monitored project progress, addressed issues as needed, identified opportunities for improvement and acted upon them, as reflected in the restructuring and additional financing in May 2018. The team facilitated six project restructurings, and demonstrated proactive engagement by leveraging synergies with other World Bank-funded projects and adapting the implementation modality. Despite some challenges encountered in implementation, safeguards, the need to modify the Results Framework, and multiple Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) project closing date extensions, the Bank team’s implementation support contributed to the project achieving objectives. The quality of supervision was Satisfactory Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The original M&E design was weak and inadequate for measuring project progress effectively. The initial PDO and output targets were not specific or easily monitorable. Consequently, all the original PDO and intermediate indicators were replaced with new ones (two PDO and seven intermediate indicators) during the restructuring and additional financing in 2018. These revised indicators were better aligned with the updated project scope under Component 1, and were measurable, time-bound, and did not entail additional cost of collection. However, the revised indicators had a few shortcomings:  There was an overlap of PDO and intermediate indicators, where the target for the first revised PDO indicator –“Improved connectivity through rehabilitation or construction of roads to improved standards” included the sum of two revised intermediate indicators, namely “Roads constructed” and “Roads rehabilitated”. Similarly, the second revised PDO indicator – “Improved connectivity through rehabilitation or construction of roads to improved standards” had the same target as the third revised intermediate indicator – “Water crossings replaced or rehabilitated”.  Insufficient indicators for technical assistance activities - while some indicators covered access audits and gender-responsive design features, none of them addressed the revision of road and bridge design standards or vehicle loading laws and regulations. In addition, there were no specific targets to measure the improvement in road safety.  Target values for some of the indicators could have been revised in the results framework – the length of roads or the number of water crossings financed under ERAP was longer or larger than what was planned during the restructuring in 2018 and this was not reflected in the target values in the RF. b. M&E Implementation The project’s M&E reporting was initially handled by the PMC until June 30, 2019, and later by the Ministry of Finance’s Aid Coordination and Debt Management Division (MOF ACDMD) and Transport and Infrastructure Sector Coordination Division (TISCD) through semiannual project reports. It was reported that these were often delayed, particularly due to the PMC's failure to provide the January 1, 2019, to June 30, 2019, semi-annual report, which hindered accurate project tracking. Although TISCD managed Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) to create the report using LTA's data in a new template developed for Samoa Climate Resilient Transport Project (SCRTP) and MOF ACDMD submitted it to the World Bank after review and verification, the lack of handover from the PMC to MOF ACDMD and TISCD resulted in gaps in project history, undermining M&E effectiveness (ICR, para 60). c. M&E Utilization The M&E findings of the project, which were regularly shared with the stakeholders through the World Bank's Aide Memoires and ISRs after each implementation support mission, had a positive impact on the project's implementation direction andled to the restructuring and additional financing in 2018. The M&E data provided evidence of achieving the project's goals of improving connectivity by upgrading roads and replacing water crossings to higher standards. In addition, the M&E arrangements under ERAP have been institutionalized beyond July 2019, with TISCD overseeing and supporting M&E activities (ICR, para 61). While there was sufficient evidence from the ICR for a Substantial rating of Efficacy post restructuring, the overall M&E quality is rated Modest due to shortcomings of original M&E design and reporting delays during implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was assigned Environmental Category "B" and three safeguards policies were triggered. These included - Environmental Assessment (OP/BP 4.01), Indigenous Peoples (OP/BP 4.10), and Involuntary Resettlement (OP/BP 4.12). However, it was later decided not to apply the Indigenous Peoples policy to generally homogeneous island nations like Samoa, and this was reflected in the restructuring and additional financing in 2018. The ICR stated that the project undertook various measures to address environment and social compliance. During preparation, the Environmental and Social Safeguards Assessment Framework and the Land Acquisition and Resettlement Framework were prepared and disclosed. During implementation, the Environmental and Social Management Framework was prepared and disclosed. In addition, Environmental Management Plans (EMPs) and Environmental and Social Management Plans (ESMPs) were prepared. The project was in compliance with the environmental safeguards. The social safeguards management system was also in compliance with WBG policies. A Social Impact Assessment and Abbreviated Resettlement Action Plans (ARAPs) were prepared. As specified in the ARAP, compensation for 13 affected lots due to the Mali’oli’o Bridge and access road projects was provided before construction began. Additionally, full compensation for land and non-land asset losses in WCR Lot 2b was also provided for. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) b. Fiduciary Compliance Procurement ERAP’s procurement complied with the World Bank's Procurement Guidelines, relevant provisions in the Financing Agreement, and the project's Procurement Plan. There were, however, a few procedural errors and delays in the initial project period. Measures were taken to address the capacity shortfalls and concurrent implementation progress. The project was also transitioned to using the Systematic Tracking of Exchanges in Procurement (STEP) tool. During implementation, one issue that came up was the excessive burn rate of the PMC contract when compared to the rate of project progress. Despite its time-based payment model, the PMC submitted monthly invoices without time-sheets to validate hours worked. This prompted an investigation by the Integrity Vice Presidency into consultant contract payments under ERAP and CRWCR. This investigation led to the identification of questionable spending, followed by an analysis of contract invoices, ultimately resulting in the World Bank identifying some expenses as ineligible. Following this, corrective actions were implemented under MOF supervision, resulting in the repayment of the ineligible expenditures. Financial Management There were a few issues with the project’s Financial Management, including report delays and ineligible costs, leading to a downgrade in the FM rating to Moderately Unsatisfactory in January 2017. However, the transfer of FM responsibilities to MOF ACDMD in June 2017 helped enhance the project’s financial management. All Interim Financial Reports (IFRs), were submitted and accepted, as were the required audited project financial statements up to June 30, 2022, which were conducted in compliance with International Standards on Auditing. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Modest Modest Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) Quality of ICR --- High 12. Lessons The following main lessons can be drawn from the experience of implementing this project.  In smaller and highly exposed economies, using a thematically structured yet flexible project design like ERAP's PDO could be more effective in accommodating potential changes. A relatively broad PDO can offer flexibility to address emerging needs that could come up during implementation. Having such flexibility in this project, allowed for the addition of various infrastructure components to ERAP, such as the Mali’oli’o Bridge and access road, three water crossings on Upolu, and WCR Lot 2b during its implementation. This was possible as the PDO for ERAP was not too restrictive by focusing on “critical roads and bridges in Samoa."  A design-phase road safety audit can help avoid extra cost and time for resolving road safety issues that may be identified after construction. A post-construction audit in March 2021 identified high-risk safety concerns with culvert driveways on WCR, leading to an extension of the project closing date to rectify and address this issue. While this contributed to improving road safety on WCR, it resulted in both rectification costs of US$0.49 million and an 18-month extension.  Intensive implementation support and sufficient budget allocation may be required for enhance the success of emergency operations. While the World Bank's prior engagement in Samoa's transport sector helped put in place the foundations for quick preparation of the emergency ERAP project, challenges including slow design progress, performance issues with the PMC, and staff changes led to a 45-month timeline from Board approval in October 2013 to the opening of Leone Bridge in July 2017. Large infrastructure projects, particularly emergency projects such as the ERAP, are often developed within tight timeframes. In such cases, certain project details may need to be refined during implementation, underscoring the need for robust implementation support and adequate funding. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is clear and provides a detailed overview of the project. The evidence and analysis presented are adequate for evaluating the performance of the project. It includes a clear discussion on the achievement of outcomes and appropriately carries out a split rating for objectives, given the original indicators and targets were changed during restructuring. The report is results-oriented, and the discussion is based on what the project was able to achieve on the ground. The ICR is candid about the M&E shortcomings and provides Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Enhanced Road Access Project (P145545) additional evidence to support the achievement of objectives. The ICR draws good lessons from the experience of implementing this project, and they were based on evidence and analysis. Overall, the quality of the ICR was High. a. Quality of ICR Rating High Page 17 of 17