Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) Report Number: ICRR0022495 1. Project Data Project ID Project Name P126346 HT Disaster Risk Mngmt & Reconstruction Country Practice Area(Lead) Haiti Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-D2090,IDA-H7460,TF-13014 31-Dec-2016 67,236,113.48 Bank Approval Date Closing Date (Actual) 01-Dec-2011 30-Jun-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 60,000,000.00 550,000.00 Revised Commitment 76,338,921.69 338,922.50 Actual 67,294,979.44 338,922.50 Prepared by Reviewed by ICR Review Coordinator Group Maria Shkaratan Fernando Manibog Victoria Alexeeva IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The original project development objective (PDO) was “to support the Recipient in improving disaster response capacity and enhancing the resiliency of critical transport infrastructure”. For the purposes of this ICR Review, the objective will be assessed as follows: PDO1: To support the Recipient in improving disaster response capacity. Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) PDO2: To support the Recipient in enhancing the resiliency of critical transport infrastructure. The PDO remained unchanged. The three PDO indicators were adjusted, which was necessary considering the difficult circumstances: between 2012 and June 2020, Haiti suffered from 16 floods, seven hurricanes (including a catastrophic Category 5 hurricane Matthew), two droughts, and an earthquake (ICR, page 19). Specifically, PDO indicator 3 was replaced to better reflect project outcomes and its target was later revised upwards, PDO indicator 2 was slightly reformulated; and the target for PDO indicator 1 was reduced. A split rating is not justified for this evaluation because the scope of the project was expanded while the commitments remained basically the same: the PDO was not revised; the PDO indicators were adjusted to better reflect project outcomes (but not to change project’s ambition); and the scope of the project was increased at additional financing to include new activities needed to respond to the destruction created by hurricane Matthew. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1. Natural Hazard Risk Assessment and Analysis (At approval: US$3.5 million/Actual: US$4.1 million). The component aimed at strengthening the institutional capacity of the Line Ministries to incorporate natural hazard risks into development planning and at collecting disaster risk data under a technical assistance program (TAP). It financed extensive technical assistance; collection and interpretation of seismic and hydrometeorological data; as well as procurement of servers, computers, software, technical monitoring, data collection equipment, and small works. Component 2. Support to Disaster Preparedness and Emergency Response (At approval: US$14.5 million/Actual: US$13.6 million). This component financed consultancies and services to improve institutional capacity of the Departmental Directorates of Public Works (DPC) and of the nation-wide network of Municipal Civil Protection Committees (CCPCs); the acquisition of goods for the national emergency communication network; and works to support the DPC training center and a pilot emergency shelter program. Component 3. Rehabilitation of Vulnerable and Damaged Critical Transport Infrastructure (At approval: US$37.0 million/Actual: US$49.6 million). The component financed (i) institutional capacity building of the Ministry of Public Works, Transport and Communications (MTPTC) and Road Maintenance Fund (Fonds d'Entretien Routier or FER); (ii) urgent construction works to rehabilitate damaged bridges, roads, and riverbanks to prevent their deterioration beyond repair; (iii) urban road repairs; and (iv) a pilot coastal road protection project. The focus was on three road corridors in the Grande Anse, South and South-East departments to maintain community access to the main Haitian road network and prevent isolation in the event of adverse natural events. In the Port-au-Prince area, the focus was on urban road repairs and pavestone works. The list of priority reconstruction activities was prepared in preliminary consultation with the MTPTC prior to project implementation. Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) Component 4. Emergency Response and Recovery (At approval: US$1 million/Actual US$1.5 million). Facilitating rapid response in case of emergency, including infrastructure reconstruction, rehabilitation, and associated studies. This component would allow the Government of Haiti (GoH) to request the Bank to reallocate financing from other project components to emergency response. Additional funds could also be made available through this window for the same purpose. If not disbursed 12 months before the closing date of the project, the component funds could be reallocated to other project components. Component 5. Project Management and Implementation Support (At approval US$4.0million/Actual US$4.2 million). Strengthening and developing the institutional capacity for project management, coordination, implementation, and monitoring and evaluation (M&E) of the two implementing units: the Central Implementing Unit at the DPC under the Ministry of Interior and Regional Authorities (MICT), responsible for components 1 and 2, and the Project Coordination Unit under the MTPTC, responsible for components 3 and 4. Project components remained the same throughout the project life. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The original project financing was US$60.0 million. Actual financing amounted to US$73.1 million (note: the actual cost figures reported here were provided by the Bank team to IEG on June 29, 2021). Project Financing. The only financing source was an International Development Association (IDA) grant. The original financing plan was revised three times:  Under Restructuring 2, additional financing (AF) was provided to add new IDA emergency funding of US$20 million to respond to the impact of Hurricane Matthew.  Under Restructuring 3, financing was reallocated among categories.  Under Restructuring 4, uncommitted funds (US$4 million) were reallocated to a World Bank’s COVID-19 emergency operation in Haiti. Recipient Contribution. The Recipient did not make any contribution. Project Dates. The project was approved on December 1, 2011 and became effective on April 3, 2012. The mid-term review was on November 24, 2014. The original closing date was December 31, 2016. After extensions totaling three and a half years, the actual closing date was June 30, 2020. The project went through four level two restructurings: Restructuring 1, July 2016, was needed to replace PDO indicator 3 (to better reflect project outcomes), adjust PDO indicator 2 (its target was later increased), and reduce the target of PDO indicator 1. The closing date was extended to June 30, 2017. Restructuring 2, May 2017, was needed to provide AF (a grant) to respond to the destruction created by the hurricane Matthew. Component 3 received 85 percent of the additional grant, while the parent project funds Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) were reallocated among disbursement categories. In addition, four intermediate results indicators (IRIs) were revised and eight were added. The closing date was further extended to June 30, 2020. Restructuring 3, June 2019. The PDO3 target (number of beneficiaries targeted by the road network improvement) was increased from 75,000 to 150,000. Seven IRIs were adjusted to reflect the changes in project scope. The scope of Component 3 was adjusted (two bridges were cancelled), and financing proceeds were reallocated among the various categories. Restructuring 4, June 2020 was triggered by the COVID-19 pandemic: some activities had been delayed, and uncommitted funds (US$4 million) were reallocated to a World Bank operation that supported Haiti’s emergency response to COVID-19. 3. Relevance of Objectives Rationale Haiti is highly exposed to multiple natural hazards: it has the top vulnerability to hurricanes among the region’s small island states and is located in a seismically active zone. High population density, many informal settlements, and weak infrastructure increase the risks to the population. Several major disasters before the project showed that the lack of institutional capacity has led the GoH to engage in short-term, reactive disaster related actions rather than developing long-term strategies and programs. The January 2010 earthquake severely diminished the already weak government capacity. At the time of approval, the PDO was highly relevant to the country’s conditions, development priorities, and aligned with the World Bank Group strategy in the country. The PDO aimed at strengthening GoH’s institutional capacity to respond to natural disasters. The PDO was also aligned with the GoH’s reconstruction strategy articulated in the Action Plan for National Recovery and Development of Haiti 2009-15 (Plan d’Actions pour la Réconstruction et le Développement d’Haïti - PARDH), specifically with pillar 3 Territorial reconstruction, focusing on investment in transformative infrastructure, and pillar 4 Institutional reconstruction. The PDO was consistent with the World Bank Group’s Interim Strategy FY12- F13 for the Republic of Haiti, December, 2011, where the top priorities included the strengthening of the country’s capacity to respond, manage and prevent disaster related crises; the restoration of the infrastructure damaged by the earthquake and hurricanes; and the strengthening of the infrastructure backbones. The PDO remained highly relevant at closing. The PDO was linked to the GoH’s National Disaster Risk Management Plan 2019–2030 and to the GoH’s DRM Legal and Institutional Framework (the National DRM System), both approved in June 2020. The latter was a direct and critical DRM achievement of the improved disaster response capacity element of the project. The PDO was also consistent with the Country Partnership Framework (CPF) FY2016–19, specifically with the resilience pillar, which focuses on strengthening natural disaster preparedness and prevention. The Performance and Learning Review (PLR) update of the CPF (FY2018) increased the focus on resilience by: (i) providing additional funding; (ii) targeting high-risk areas; (iii) focusing on local capacity to prepare for and respond to disasters; and (iv) mainstreaming DRM in other sectors. Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To support the Recipient in improving disaster response capacity. Rationale The theory of change (TOC) for this objective is as follows. Input 1 consisted of disaster risk data, risk assessment, and online data platform. These inputs, if successfully developed, were expected to causally lead to the strengthening of the institutional capacity of the line ministries to plan disaster response (an output). In turn, the strengthened institutional capacity would lead to the outcome of improved government disaster preparedness and response and disaster vulnerability reduction. Input 2 consisted of training in disaster risk management provided to communities and public officials, expansion of the network of the CCPC, improving the DRM communication network and decision-support system, and construction of shelters. These inputs were expected to lead to better capacity of the government and the communities to prepare and respond to disasters and, consequently, to reduced vulnerability of the population to disasters. This would support the achievement of Objective 1. The Emergency Project Paper did not include the TOC, it was created for the ICR. The TOC in the ICR did not indicate any critical assumptions for the TOC to be fully operable. However, the Bank team provided the following critical assumptions to IEG: “1. Performance improvements of the platform thanks to capacity building carried out before the MTR to better inform decision-making, policies, preparedness, and early warning capabilities. 2. Performance improvements of all levels of the MTPTC through capacity building before the MTR to increase the length of critical all-weather roads based on the Rural Index Methodology to connect the South to the rest of the country.” The results chain is logical and relevant to the country’s urgent needs. Disaster risk data and assessments are necessary for natural hazard decision making. Related training, the expansion of the CCPC network nation-wide, and improved DRM communication support the capacity to use the DRM information and make informed and strategic DRM decisions. Outputs: a. Outputs included at approval: IRI 1: Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) Number of technical assistance days completed, target 1,000. This indicator was to reflect the number of days of training delivered to line ministry officials. Consultants would be based in line-ministries to train and mentor selected staff. This indicator was removed from the results framework (RF) and replaced by IRI 6 at first restructuring, as suggested at MTR (see it below). IRI 2: Number of ministry officials who benefited from training in risk management, no target at approval. The project benefitted 127 ministry officials who received training in risk management, exceeding the original target of 75 (introduced at first restructuring) and the revised target of 70 (the target was revised during the third restructuring in June 2019, as suggested at MTR). This IRI appears to duplicate another one. IRI 3: Number of CCPC staff trained and certified, target 2,300. The project benefited 3,313 CCPC staff by training them and providing with level 2 CCPC certification, exceeding the original target of 2,300 staff and the revised target of 2,800 staff (revised during the third restructuring in June 2019). IRI 4: Number of operational Municipal Civil Protection Committees, target 144. The project increased the number of operational level 2-certified to 119, partially achieving (i.e., 83 percent) the original target of 144 or the revised one of 140 (the target was revised under the first restructuring in July 2016). The ICR explains that the COVID-19 pandemic disrupted the final phase of the training. It will continue under PGRAC. b. Outputs added at first restructuring: IRI 5: Number of female ministry officials who benefited from training in risk management, target 50 percent of IRI 2. The ICR reports: “The gender target was not reached as it was ambitious: women civil servants in Haiti do not exceed 30 percent of the public work force.” IRI 6 (replaced IRI1): Number of beneficiaries from DRM training, target 400. Approximately 1,520 line-ministry staff benefited from training in disaster risk management, exceeding the original target of 400 and the revised target of 799 (the target was revised during the third restructuring in June 2019). IRI 7 (gender specific part of IRI 6): Number of beneficiaries from DRM training, female, target 50 percent of IRI 6. This gender specific target was not achieved. The ICR (page 29) explains it as follows: “the gender target was not reached as it was ambitious: most women in targeted areas did not have free time for training as they are full-time housewives Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) handling domestic tasks (water and wood hauling, cooking, etc.) as well as sometimes other tasks such as tending and harvesting.” A confusion arises with this indicator, which was supposed to reflect training provided to line ministry employees, yet in the comments to the indicator, the ICR implies the targeting of women who are full-time homemakers. IRI 8: Percentage of the national territory covered by a high-resolution Digital Elevation Model and high resolution aero-photography, target 100 percent. The target was reached. IRI 9: A national radio communication system was initiated and at least four pilot departments were connected to the national level, reaching the target of four. IRI 10: A draft road map with a score card for a disaster risk management system was developed, as targeted. This IRI was strategically added during the first restructuring. The Project Coordination Unit and the MICT maintained the policy dialogue, developed and initiated a process review, and finalized the DRM institutional and legal framework that was enacted officially on June 15, 2020. c. Outputs added at second restructuring (related to hurricane Matthew): IRI 11: A first pilot detailed hydrological risk modelling was developed using the Digital Elevation Model, achieving the target. The ICR reports that the National Center for Geo-spatial Information (CNIGS) delivered georeferenced database, layers for modeling, open-data platform (www.haitidata.org), and national and regional workshops on the use of data. The task of rehabilitating the remaining five shelters was transferred to the World Bank’s Strengthening Disaster Risk Management and Climate Resilience Project (PGRAC), P165870, approved in May 2019. IRI 12: Seven pilot evacuation shelters were rehabilitated or reconstructed with multi-hazard resilient measures, short of reaching the original target of 25 or the revised target of 12 (the target was revised during the third restructuring in June 19). IRI 13: Pilot evacuation shelters’ tool to evaluate and prioritize shelters’ needs was developed and used in the area affected by the hurricane Matthew, reaching the target. d. Outputs added at third restructuring: Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) IRI 14: Number of risk maps produced, disseminated and available in an open-data online platform, target 35. No risk maps were produced before closure. The ICR reports that the “information to develop the risk maps was produced, but the risk maps were not finalized. Constraints…include payment delays related to the high turnover of ministers needing to be familiarized to the project, as well as to… the rainy seasons.” The finalization of the 35 maps was transferred to the World Bank’s PGRAC project, P165870. PDO-related Outcomes: PDO indicator 1 (as originally approved and not revised): Line ministries developing an action plan for vulnerability reduction. Two line ministries - Ministry of Public Health and Population (MSPP) and Ministry of National Education and Vocational Training (MENFP) - benefited from the formulation of the action plans, as well as from a tailor-made tool to provide safety to hospitals and schools and a related staff training. The original objective was five ministries. This target was reduced to two ministries at the first restructuring in July 2016 (following the mid-term review (MTR)) due to delays in finding local consultants to perform the training and produce the action plans (the allocated budget did not allow for hiring international consultants). Therefore, the original target was not achieved, while the revised target was achieved. This is the only PDO indicator with a reduced target; overall, the project’s scope increased. PDO indicator 2: Original PDO indicator 2 (included at approval): Share of population living in a municipality with a certified CCPC (Comité Communal de Protection Civile), target 40 percent. Revised PDO indicator 2 (at the first restructuring in July 2016): The share of the population living in a municipality with a CCPC of category 2 or higher was estimated at 77.3 percent at closure, reaching the target of 75 percent (the baseline was 45 percent). It appears that the revised indicator refers to non-certified CCPCs while the original one refers to certified CCPCs; it is unclear what the difference is. Data on the level of actual achievement of the original target is also unavailable. To summarize the conclusions regarding the achievement of objective 1: - The scope of the project under objective 1 was increased during restructurings, while the commitments did not materially change. Objective 1 remained the same while the PDO indicators were adjusted, which was necessary considering multiple disasters that hit Haiti during the project life and the need to expand the project to respond to the aftermath of the hurricane Matthew. Specifically, PDO indicator 2 was reformulated to better reflect project outcomes and the target for PDO indicator 1 was reduced. Overall, considering many additional IRIs, the scope of the project was increased. Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) - There were two original PDO indicator targets and two revised ones. Neither of the two original PDO indicator targets were achieved; however, both of the revised PDO indicator targets were achieved. - There were 14 original IRI indicator targets (introduced either at approval, or at the first or second restructuring) and seven revised ones. Eight out of 14 original IRI targets (57 percent of the total) were achieved and six were achieved only partially. Three out of five revised (reduced) targets (60 percent of the total) were achieved and two were achieved only partially (note that many IRIs were added and some revised at the restructurings; the original indicators/targets were reviewed together, whether they were included in the RF at approval or at the restructurings). Overall, project activities under objective 1 resulted in the following outcomes: (i) a better capacity to understand disaster risks and a higher preparedness at the central and municipal levels to respond to natural disasters; (ii) an improved capacity of the National Center for Geospatial Information to assess disaster risks by generating data and using modeling; (iii) an enacted National DRM system including a DRM Roadmap; (iv) a strengthened DRM capacity of the CCPCs; and (v) an expanded early warning system coverage. In particular, at the central level, the outputs include the following: two line ministries were equipped with action plans and a tool to provide safety to hospitals and schools, and received related training; a high-resolution digital elevation model was supplied and used to produce pilot hydrological risk modeling, related training was provided; a national radio communication system was developed and piloted. At the municipal level, the outputs include the following: the number of operational CCPC in municipalities rose from 73 to 119, increasing the share of the population living in a municipality with a CCPC from 45 percent to 77.3 percent. Rating Substantial OBJECTIVE 2 Objective To support the Recipient in enhancing the resiliency of critical transport infrastructure. Rationale The theory of change (TOC) for this objective is as follows. Input 1 consisted of rehabilitating and disaster-proofing of transport infrastructure (road sections and bridges), resulting in resilient critical transport infrastructure and therefore in accessibility of disaster-affected areas by transportation in the future. Input 2 consisted of capacity building at the MTPTC, resulting in its improved ability to build, rehabilitate, and maintain resilient transport infrastructure and therefore in the increased sustainability of the Outcomes under Objective 2. The results chain is logical and relevant to the country’s urgent needs. Rebuilding roads and bridges in a way that would make them resilient to future disasters, as well as increasing the MTPTC’s capacity to do so in the future were critical activities that helped the country’s recovery and led to the achievement of the project’s objective 2 of enhanced resiliency of critical transport infrastructure. Outputs: Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) a. Outputs included at approval: IRI 15: Roads km rehabilitated (rural and non-rural), target was to be determined. This indicator was removed from the RF (for unclear reasons) at the first restructuring in July 2016 and replaced by IRI 19 (see below). Therefore, achievement of the target is not reported in the ICR. IRI 16: Number of bridges built or rehabilitated with satisfactory technical standards, original target 5. The achievement was six bridges, exceeding the original target of 5 and reaching the revised (at second restructuring in June 2017) target of 6. IRI 17: Number of spot interventions to protect the local access executed with satisfactory technical standards, with no specific target. The project implemented 112 spot interventions, exceeding the original target (determined during first restructuring in July 2016) of 20 and the revised target (during second restructuring in June 2017) of 40. This achievement was linked to additional financing to support restoration after hurricane Matthew. b. Output added at first restructuring: IRI 18 (included at first restructuring): Number of departments with investment plan to improve all weather rural accessibility based on the Rural Index Access methodology, original target 3. The achievement was five departments, exceeding the original target of three and reaching the revised (during the second restructuring in May 2017) target of five. This was related to additional financing to support restoration after hurricane Matthew. b. Outputs added at second restructuring: IRI 19: Roads rehabilitated, non-rural, target 100 km. The project rehabilitated 120 roads (non-rural), exceeding the target. IRI 20: Number of bridges and road sections damaged by hurricane Matthew that were repaired or consolidated with satisfactory technical standards, target 15. The actual achievement is 27, exceeding the target. IRI 21: Number of bridges damaged by hurricane Matthew that have been rebuilt with satisfactory technical standards, original target three. The actual achievement is one bridge, short of the original target of three and achieving the revised target of 1. The ICR explains that due to delays in the bidding process and potential Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) security constraints on site, the target was reduced to 1 bridge under third restructuring. However, 18 emergency bridges were supplied with the spare parts under the project. IRI 22: Number of man/days of labor-intensive work (LIW) generated by project activities, original target 300,000. The project activities generated 370,000 man/days of LIW, exceeding the original target of 300,000 and the revised one of 350,000 (at third restructuring in June 2019). The project trained and empowered employees to perform LIW building, developing unique skills that will be needed for resilient local transport upkeep. IRI 23: Actions are taken in a timely manner in response to feedback received during consultation sessions with beneficiaries/project affected people (Feedback, Responses and Action are monitored and reported). The indicator was introduced following Hurricane Matthew to make sure recovery is proceeding in a timely manner in response to feedback received during consultation sessions by the beneficiaries. The target was achieved. PDO-related Outcomes: PDO indicator 3: Original PDO3 indicator (included at approval): Number of cumulative road closure days per year, target <25. Revised PDO indicator 3 (at the first restructuring in July 2016): Number of beneficiaries from an improved and resilient investments in the road network, target 70,000. The target was revised at the second restructuring to 150,000 beneficiaries (linked to additional financing to support restoration after Hurricane Matthew), and the gender sub-indicator was added (50 percent of the beneficiaries are women). At closure, the actual number was 150,000 beneficiaries. To summarize the conclusions regarding the achievement of objective 2: - The scope of the project under Objective 2 was significantly increased during the second restructuring (to include many additional IRIs related to the hurricane Matthew’s aftermath), while the commitment did not materially change. Objective 2 remained the same while PDO indicator 3 was adjusted, which was necessary to better reflect project outcomes. - There was one original PDO indicator and one revised one. It is unclear if the original indicator’s target was achieved. T, the revised indicator’s target was achieved. - There were nine original IRI indicator targets and six revised ones. Seven out of the original nine IRI targets (78 percent) were achieved. All six revised targets (100 percent) were achieved (note that many IRIs were added and some revised at the four restructurings; the original indicators/targets were reviewed together, whether they were included in the RF at approval or at any of the restructurings). Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) - The RF at closure was very complex. It included 23 IRIs (old and new together) and had very little in common with the RF in the Project Paper at approval. The ICR based its efficacy analysis on the RF after the third restructuring, without any comparison with the original RF. Overall, project activities under objective 2 resulted in restored connectivity and more resilient transport infrastructure including improved critical roads and emergency bridges. They also led to a better MTPTC’s capacity to build back better, as it relates to resilient transport infrastructure, and to maintain the resilient roads and bridges constructed by the project. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale For Objective 1, the efficacy is Substantial. The project developed a better capacity to understand disaster risks and prepare for future disasters; improved the country’s capacity to generate DRM data and produce DRM assessments; strengthened the DRM capacity of the CCPCs; and created a nation-wide early warning coverage. While some of the original IRI targets were not achieved or only partially achieved, and the original PDO indicators were replaced with other indicators, this should be evaluated in the context of the overall situation of multiple disasters that occurred in this low income fragile country during the project life, including the catastrophic Category 5 hurricane Matthew, a response which was included in the project at the second restructuring, thus increasing the project scope. For Objective 2, the efficacy is Substantial. The project added value to restored connectivity; improved resiliency of transport infrastructure; and improved the MTPTC’s capacity, thus supporting the sustainability of project outcomes. The PDO target of the number of beneficiaries was exceeded and all revised IRI targets were achieved. The project added significant value under both objectives, and, while many of the original intentions were not achieved, this was mainly because the project design had to be adapted considering difficult circumstances and the need to respond to a new disaster, hurricane Matthew. Since project implementation had shortcomings which led to the need to re-adjust the targets and to transfer the outstanding activities (unmet targets) to the successor project PGRAC (P165870), on balance, the overall efficacy is Substantial, with moderate shortcomings. Overall Efficacy Rating Substantial Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) 5. Efficiency Economic analysis. The project did not require an economic analysis at appraisal as it was an emergency operation appraised under OP/BP8.00. Economic analysis was carried out in the ICR for the project’s components 1 and 2, which are under Objective 1 To support the Recipient in improving disaster response capacity, and was not carried out for the component 3, which is under objective 2 To support the Recipient in enhancing the resiliency of critical transport infrastructure. The ICR reports that Component 3 ex-post economic analysis was not possible “due to the difficulty of collecting car flow statistics after a disaster while disaster occurrence and intensity are difficult to predict” (page 46). This is the only explanation, which is only partially convincing since data from previous disasters could have been utilized. In other words, the only activities covered in the ex-post economic analysis are the capacity building ones, constituting only 24 percent of the overall financing, while the tangible activities - road and bridge construction/rehabilitation, - constituting 68 percent of the total financing, are not covered. This constitutes an important gap in the economic analysis. The ICR indicates that the capacity building components are viable with a net present value (NPV) of US$6.9 million over 20 years at a discount rate of 6 percent, with an economic rate of return (ERR) of 22 percent, and a benefit cost ratio of 1.5. The valuation of benefits of the DRM components were based on (i) a per capita risk premium linked to the reduction of flood risks and (ii) estimations on the value of a statistical life in Haiti. It was assumed that the project would deliver the following direct benefits: (i) reduction in the average disruption days from fifteen days to eight days; (ii) reduction in average injuries by a third; and (iii) reduction in average deaths by a third (ICR, Annex 4, Efficiency Analysis). IEG also notes that, although not explicitly stated, the assumption is also being made that these benefits are causally linked and directly attributable to the project’s DRM institutional capacity-building activities (ICR, Annex 4, Efficiency Analysis). Operational/ Administrative efficiency. The project was prepared in a relatively short time, close to one year from identification to effectiveness, for a very poor and fragile country. The overall design supported implementation flexibility, critical in the emergency context. No additional financing (AF) was requested, and the only AF received was to support the reconstruction following Hurricane Matthew. At the same time, there were some inefficiencies in implementation. There were procurement delays, with poorly executed contracts, the project was restructured (at level 2) four times, and the closing date was delayed 3.5 years. Two out of four restructurings were caused by exceptional emergencies (Hurricane Matthew and COVID-19) and the other two involved adjustments to the RF. Overall, considering some inefficiencies in implementation and the fact that the economic analysis covered only the capacity building activities, constituting only 24 percent of the overall financing, in the absence of complete or at least adequate evidence to substantiate the whole project's efficiency, the project's efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) 0 Appraisal 0  Not Applicable 26.00 ICR Estimate  22.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Based on the high relevance of objectives, substantial efficacy with moderate shortcomings, and modest efficiency, the overall outcome rating is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The project risk status changed twice during project life: from “High Risk” at appraisal, to “Substantial Risk” at mid-term review, and to “Moderate Risk” at closure. The change was mostly due to the government’s (MTPTC’s and DPC’s) evident commitment on DRM. However, there are still several risks to project development outcomes, specifically: 1. Country’s high exposure risk to natural disasters and the increasing frequency and severity of weather events due to climate change. The project involved enhancing the resiliency of the critical transport infrastructure and increasing government capacity to respond to disasters, thus mitigating these risks. However, as the severity of extreme weather events increases globally, the country is expected to suffer worsening impacts, potentially requiring even more resilient roads, bridges and shelters. 2. Political and institutional capacity risks. Haiti is a poor, fragile, and politically unstable country, with low institutional capacity, which further deteriorated after the earthquake. Government commitment is even more critical for project sustainability than elsewhere. To ensure government ownership, the project involved consultations with the government at start and a close coordination during implementation. The project was based on a multi-sector approach (DRM and transport) and included resilience mainstreaming in transport. In addition, the project was encompassed a bottom-up participatory approach, resulting in a dynamic involvement of the communities, local organizations, and small enterprises in its activities. Capacity building activities were at the core of the project. 3. Implementation capacity risk, including procurement and financial management risk. Hiring competent personnel for the implementation unit was a challenge, especially after the earthquake. The project supported the preparation of manuals on FM and procurement at appraisal and the hiring of experienced consultants to continuously train the staff of the implementation unit. Also, there were 16 Bank supervision missions, which involved project adaptation to the difficult and changing circumstances. Expert guidance was provided on procurement, FM, safeguards, and monitoring and evaluation. Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) 4. Technical risk. The ICR reports (page 24) that the enacted National DRM System, the Earth observation (EO) knowledge base, communications and early warning systems, and transport sections developed under the project face moderate risk to sustainability. The continuity and further strengthening of these achievements are supported by four ongoing World Bank projects, designed to provide continued support to increase resilience and manage response to natural disasters. This includes the successor project PGRAC. 8. Assessment of Bank Performance a. Quality-at-Entry This was an emergency project prepared using a framework approach to ensure flexibility, an adequate design in an emergency. The preparation time was nine months—April to December 2011. Project preparation involved close consultations with the government of Haiti, specifically with the MITC and the MTPTC. The project was designed to provide extensive technical assistance on DRM and to finance increased resilience of the critical transport infrastructure. The project’s three-pronged approach - building DRM data and knowledge base, enhancing institutional capacity to respond to disasters, and investing in resilient roads and bridges – was sound and addressed the issue of high vulnerability of the population to disasters in a multi-faceted way, thus increasing the sustainability of outcomes. The project used the Bank’s global experience with DRM and transport projects as well as lessons from post- earthquake projects in Haiti. The approach was strategically relevant and technically sound; the design benefited from stakeholder consultations; the project included bottom-up participation and mitigation of various risks, and accounted for the capacity of the implementing agency. However, the project was too ambitious considering the conditions in the country. As such, it had to be adjusted at restructurings, extended, and some of the activities had to be transferred to the successor project. The adjustments included some changes in the indicators and targets. While the ambition of the project was not reduced, the RF was significantly modified. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project supervision should be evaluated in the context of multiple natural disasters that happened during the project life, including seven hurricanes (including the catastrophic hurricane Matthew), 16 floods, two droughts, and an earthquake. In addition, the project was affected by COVID-19. The project was significantly adjusted to include activities to mitigate the impact of the hurricane Matthew. Despite all these circumstances, the project achieved its objectives without requesting additional financing for the original activities (the only AF the project received was in relation to new activities aimed at reconstruction after hurricane Matthew). Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) Project supervision was adequate. There were 16 supervision mission, which included field visits, discussions with a wide range of stakeholders, and meetings with development partners. During supervision, close attention was paid to procurement and financial management, areas where the implementation unit had insufficient experience. The Bank provided continuous training to the implementation unit in those areas. The supervision missions involved decision making related to improving project management. One of the important decisions was to transfer the responsibility for the construction of the multipurpose center from the implementation unit to the United Nations Office for Project Services (UNOPS), and the setting up of a communication network to the World Food Program (WFP). However, there were shortcomings in implementation. In particular, there was a slow start of implementation due to delays in procurement, insufficient budgeting for resettlement compensation, and delays in fund release for the DRM contracts. The ICR explains (page 19) that this was mainly due to the postponed signing of the contracts at the MICT, which had a high turnover of its ministers. However, the Bank was also responsible for some delays: the MTR stated that the Bank should have been more pro- active in addressing issues with the implementation progress after the first and second restructurings (ICR, page 24). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E design was a weak part of the project. The original RF is barely discernible in the final RF, as there were multiple changes over the project life. Apart from adding the indicators to reflect new activities related to the additional financing for hurricane Matthew, the RFs were significantly re-designed over the restructurings 1 to 3, adjusting the indicators and their targets. While the original RF was logical and manageable (consisted of a reasonable number of indicators), many of the targets were not achievable, and either the indicators altogether (including two of the three PDO indicators) or the targets needed to be revised. Gender targets were set too high for the country and were artificial (e.g., 50 percent of the beneficiaries should be women). The number of IRIs in the final RF (after the third restructuring) amounted to 21 and was too high. b. M&E Implementation M&E implementation was challenging but improved over time, as the ICR states (page 21). Progress reports prior to the MTR did not provide updates on the indicators. This problem was resolved Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) after the MTR when the Bank provided M&E support to the implementation units. After the second restructuring (May 2017), M&E data were collected, measured, recorded, and analyzed, as appropriate. c. M&E Utilization M&E utilization was adequate. Reports submitted by the implementation units were used in the preparation of the MTR and ISRs. After adjusting the RF during the first and second restructurings, data were collected and analyzed on outcomes and outputs. This helped project rescoping during the third restructuring, as well as the systematic evaluation of the project. M&E Quality Rating Modest 10. Other Issues a. Safeguards At appraisal, the project was rated “Category B” and triggered the following Bank safeguard policies: Operations Policy (OP) 4.01 Environmental Assessment; OP 4.11 Physical and Cultural Resources; and OP 4.12 Involuntary Resettlement. The ICR reports (page 21) that all safeguards instruments were prepared as expected. There was one issue: a transport site showed environmental, health and safety (EHS) issues in terms of inadequate workers’ gear and occupational hazards; however, these were subsequently rectified. Social safeguard policies. The ICR reports (page 21-22) that there was no social safeguards issue until May 2018, when the project experienced extended delays in compensation for involuntary resettlement. The reasons included bureaucratic obstacles to legalize land titles, difficulties in identifying project affected people, and delays by the government agencies in paying compensation. These issues were resolved. The Grievance Redress Mechanism (GRM) was functional and received questions and grievances through focal points at the community level, as well as during bimonthly site visits and consultation meetings with beneficiaries. All grievances were addressed in a timely manner and to the satisfaction of the people involved. b. Fiduciary Compliance Financial management: FM was a project weakness throughout the project life, despite the continuing FM training provided by the Bank. Compliance with the project’s FM was moderately satisfactory until 2017, after which it was downgraded to “Moderately Unsatisfactory”. However, most audit reports and interim financial reports (IFRs) were provided on time and were accepted by the Bank. Difficulties in the FM, as the ICR reports (page 22) were: (i) delayed execution of the budget and low project execution; (ii) long delays in the submission of applications for advances to the designated accounts, and in obtaining refunds of unutilized Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) balances from UN agencies; (iii) discrepancy between the expenditures incurred in the project records and the expenditures recorded in the Bank system; (iv) use of different accounting systems that were not satisfactorily functional and updated; (v) delays in submitting IFRs as agreed in the financing agreements; and (vi) unclear staff-time allocation for the project. Procurement: Procurement was also a project weakness, although the Bank invested in continuous procurement training. There were delays with contracts, which were poorly implemented. The reasons cited in the ICR (page 22) are as follows: (i) an increase in costs, often due to the depreciation of the gourde versus the US dollar; (ii) financial proposals above the available budgets and unclear staff-time allocation by the contractors to the project; and (iii) delays in updating activities in Systematic Tracking of Exchanges in Procurement (STEP). c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory The original RF had to be significantly revised, as targets were set too high and some of the indicators were not Quality of M&E Substantial Modest applicable in the country context. The final RF was unmanageably complicated. Data were not collected until after the second restructuring. Quality of ICR --- Substantial 12. Lessons 1. Stakeholder consultations and continued collaboration (including bottom-up participation), as well as long-term engagement with the client are critical for achieving results, especially in the poor FCV countries. World Bank’s long-term engagement with the Haiti’s government and communities paid off and led to the formulation and adoption of the National DRM Strategy, and a Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) policy dialogue with the Client over the years led to the preparation and adoption of the DRM legal and institutional framework, and the enactment of the National DRM System after more than ten years in the making. This was a transformative outcome. 2. The multi-sectoral approach and DRM mainstreaming are critical to achieving sustainable DRM outcomes and promoting DRM in the core sectors of the economy, including transport. This approach was initiated in Haiti by the World Bank in 2005 and succeeded through several DRM projects. The transport component adapted to a complex and changing situation in Haiti through three successful approaches: (i) the earmarked funds in stand-by mode were efficiently disbursed after a catastrophic event (i.e., after Hurricane Matthew); (ii) innovative, reliable, less expensive engineering solutions, such as new gabion reinforcement techniques to increase the resilience of transport infrastructure in flood-prone areas; and (iii) trained and empowered men and women for Labor intensive work (LIW), building unique skills that would continuously be needed for resilient local transport upkeep. 3. DRM knowledge management is essential for informing decision making. The Bank helped build scientific capabilities and their use for the DRM decision making. When these systems are fully operational under the ongoing PGRAC, they will help formulate more effective strategies, design policies, support early warning systems (EWS), and prioritize programs and projects with real time monitoring capabilities that allow for proper evaluation and recalibration. 4. While there is limited time for the preparation of emergency projects, it is important to use accumulated experience in Haiti to pay more attention to the design of the results framework (RF). The experience of this project demonstrates that a poorly designed RF can cause a need for project restructuring and lead to complications with the evaluation of project achievements. 13. Assessment Recommended? No 14. Comments on Quality of ICR Overall, the ICR provides relevant and valid information. This includes the project context; assessment of Bank performance, risks, and compliance issues; key factors affecting implementation; and lessons learned. However, there are some shortcomings in the ICR’s analysis of the project’s efficacy and efficiency. The efficacy analysis did not take into account the original RF and the change in the project PDO indicators and the IRIs during the life of the project. At the same time, the issues with the RF are fully acknowledged in other sections of the ICR, which appears inconsistent. Moreover, the economic analysis only covers the capacity building components, which constitute 24 percent of the project cost and can only be evaluated on the basis of many assumptions. The analysis does not cover the only tangible component (road and bridge construction), which constitutes 68 percent of total project cost at closure. Given the extensive experience of the Bank and other donors in emergency operations in Haiti related to natural disasters, it should have been possible to collect some data related to transport infrastructure improvements. In addition, the ICR does not acknowledge that the economic analysis is provided for the capacity building activities and not provided for the construction Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HT Disaster Risk Mngmt & Reconstruction (P126346) component, these facts are not clear in the ICR. This unclarity exaggerates the impression that the economic analysis reflects more than it really does. The ICR Quality is rated substantial. a. Quality of ICR Rating Substantial Page 20 of 20