Approach Paper Reducing Disaster Risk from Natural Hazards: An Evaluation of World Bank Support, FY10–20 August 30, 2021 1. Background and Context 1.1 Disasters caused by natural hazards are a threat to development, and their costs are rising. Natural hazards include cyclones (also known as hurricanes and typhoons), earthquakes, tsunamis, floods, volcanic eruptions, landslides, and droughts. The impact and severity of disasters that follow a hazard event depend on the choices made over time by governments, the private sector, and others. The costs of disasters are staggering and continue to rise. In developing countries, disasters account for approximately a 9 percent loss to annual average gross domestic product (Tang et al. 2019). The annual average cost of disasters in developing countries has risen from US$23 billion to US$150 billion over the past 30 years, and the number of affected people has tripled to 2 billion (Hallegatte et al. 2017).1 Population growth, rapid and unplanned urbanization, poor-quality infrastructure, and ineffective disaster risk governance have contributed to these increased damages from natural hazards. 1.2 Climate change is exacerbating the costs of disasters and putting more people at risk from more powerful, more frequent, and more severe storms, floods, and droughts. Climate-related disasters, including extreme weather events, were twice as common during 2000–19 than during 1980–99 (United Nations Office for Disaster Risk Reduction [UNDRR] 2020a). Rising sea levels make coastal flood events more likely and more devastating, and they place the lives of 1.3 billion people and approximately US$158 trillion in assets at risk (World Bank 2018). Over the past two decades, droughts have contributed to the deaths of 11 million people and have negatively affected the livelihoods of some 2 billion people worldwide (Haile et al. 2019). In Sub-Saharan Africa, drought events have significantly increased over the past half-century and projections suggest that such severe drying will continue to increase due to climate change (World Bank 2013a). Reducing the risks of hydroclimatic disasters (especially cyclones, floods, and droughts) is one of the most important aspects of climate change adaptation and of building climate resilience (UNDRR 2020b). 1.3 People in developing countries, particularly the poorest and most vulnerable residents, are most at risk of losing their lives and livelihoods from disaster-related events. While low-income countries have experienced only 9 percent of hazard events, they have suffered 48 percent of all fatalities since 1980 (Global Facility for Disaster 1 Reduction and Recovery 2013). Within these countries, poor people are most at risk of experiencing disasters: they are twice as likely to live in poorly constructed housing and in highly exposed areas, including in riverbeds and on coastlines, and to work in disaster-prone sectors. It is estimated that disasters have pushed about 26 million people into poverty each year since 2017 (Hallegatte et al. 2017). Intersectionality also plays a role: Within these exposed areas, women and girls, older people, and people with disabilities face greater vulnerability and exposure to disasters (Holtsberg 2020). The impact of disasters can be particularly severe in contexts affected by fragility, conflict, and violence (FCV; Peters 2019). 1.4 Reducing disaster risk from natural hazards, the focus of this evaluation, can reduce the negative effects that disasters have on society and people’s lives. Disaster risk reduction (DRR) is defined as “the concept and practice of reducing disaster risks through systematic efforts to analyze and manage the causal factors of disasters, including through reduced exposure to hazards, lessened vulnerability of people and property, wise management of land and the environment, and improved preparedness for adverse events” (United Nations International Strategy for Disaster Risk Reduction 2009). The terms exposure and “vulnerability” are defined as follows: • Exposure: the situation of people, infrastructure, housing, production capacities, and other tangible and intangible (for example, cultural) assets located in hazard- prone areas (UNDRR 2016). Exposure may be dictated by mediating social structures and institutions (Sen 1983). • Vulnerability: the conditions determined by physical, social, economic, and environmental factors or processes that increase the susceptibility of an individual, a community, assets, or systems to the impacts of hazards (UNDRR 2016). 1.5 This evaluation interprets DRR broadly. It includes efforts to reduce exposure and vulnerability, as well as other ex ante risk-reduction interventions that can reduce the adverse impacts of disasters. Areas of DRR include (but are not limited to) identifying disaster risks; mitigating risks through protective works and resilient buildings and infrastructure (including resilient reconstruction); strengthening or integrating disaster risk considerations into policies and institutions; improving disaster risk preparedness (for example, early-warning systems); and financial protection through disaster risk finance. 1.6 DRR is operationalized throughout the different phases of the disaster risk management cycle. The four internationally recognized phases of the disaster management cycle are mitigation, preparedness, response, and recovery. Mitigation 2 takes place prior to a disaster to prevent or reduce the cause, impact, and consequences of disasters. Preparedness includes support for knowledge and capacity to anticipate, respond to, and recover from disaster events. The response phase is typified by emergency services and humanitarian responses designed to save lives and processes to understand, map, and calculate the costs of disaster-related losses, and to develop recovery plans, including through donor coordination. The recovery phase supports the restoration and improvement of facilities, livelihoods, and living conditions, including efforts to reduce disaster risks, and occurs concurrently with regular operations and activities. DRR occurs predominantly in the mitigation and preparedness phases but is also integrated in the recovery phase as part of efforts to build back better. The response phase is mainly focused on saving lives. 1.7 There is a strong economic and social rationale to invest in DRR. Investing in resilient infrastructure, for example, can provide a net benefit of US$4.2 trillion, with US$4 in benefits for each US$1 invested (Hallegatte, Rentschler, and Rozenberg 2019). When countries rebuild stronger and more inclusively after disasters, they can reduce the impact on livelihoods and well-being by as much as 31 percent (Hallegatte, Rentschler, and Walsh 2018). Investing in people by providing universal access to early- warning systems can reduce well-being losses from disasters by an estimated US$11 billion (Hallegatte et al. 2017). In addition, mortality from disasters has declined over time due in part to economic development and better disaster management, especially for disasters where early warning is possible (UNDRR 2019). 1.8 Despite these opportunities, underinvestment in DRR persists. There has been insufficient investment in DRR, especially disaster risk mitigation and preparedness (United Nations International Strategy for Disaster Risk Reduction 2015; World Bank 2013b). Between 2010 and 2019, only 6.5 percent of total official development assistance for disaster risk management was directed toward risk-reduction activities.2 The literature points to several reasons for this phenomenon: Countries lack resources to invest in DRR and may have a limited understanding of disaster risks and vulnerabilities, and their governments tend to favor politically visible postdisaster initiatives over predisaster risk reduction. Supply is also a problem: Much more international development assistance is available for disaster response and recovery, which has long been identified as a moral hazard in the sector (Keefer 2009; Tanner, Bahadur, and Moench 2017, Wilkinson 2012; World Bank 2013b). 2. The Role of the World Bank 2.1 DRR is at the core of the World Bank’s approach to support green, resilient, and inclusive development. Developing countries face the dual challenge and opportunity of repairing the damage to development gains brought about by the coronavirus (COVID- 3 19) pandemic while building back better, including by improving societal resilience and by using more inclusive approaches to better face future shocks posed by pandemics, climate change, disasters, and conflict. Building resilience requires DRR across the full set of potential disasters, including those caused by natural hazards and other environmental, technological, and biological hazards. However, this evaluation will cover only natural hazards; the Independent Evaluation Group (IEG) covers other resilience issues through multiple evaluations (see below). 2.2 The World Bank has placed DRR at the core of its commitments to address climate change. Climate change is a critical corporate priority for the World Bank, as laid out in the 2016–20 climate change action plan that identifies disaster risk management as a key aspect of climate change adaptation and makes specific commitments, for example on disaster risk financing (World Bank 2016b). The International Development Association (IDA) has identified climate change as one of its special themes since the 16th Replenishment of IDA (IDA16), with specific emphasis on disaster risk management since IDA17, and this emphasis has gained further prominence under IDA20 as a Cross-Cutting Issue with an emphasis on crisis preparedness and building back better. In 2016 the World Bank–International Monetary Fund Development Committee laid out the Forward Look: A Vision for the World Bank Group in 2030 and identified the need to strengthen resilience as one of the Bank Group’s three top priorities (World Bank and International Monetary Fund 2017). The 2018 capital package reinforced the Forward Look’s agenda, prioritizing the need to foster resilience to global shocks and threats and identifying climate change and related disasters as major threats that undermine development progress (World Bank 2018). 2.3 The World Bank’s support for DRR is aligned with several international agreements, including the Sendai Framework, the Paris Agreement, and the Sustainable Development Goals (SDGs). The World Bank is helping member countries to implement the Sendai Framework for Disaster Risk Reduction, an agreement adopted by 187 member countries in 2015 (United Nations 2015). The framework lays out four priorities for DRR: i. Understanding disaster risk; ii. Strengthening disaster risk governance to manage disaster risk; iii. Investing in disaster reduction for resilience; and iv. Enhancing disaster preparedness for effective response, and to “build back better” in recovery, rehabilitation, and reconstruction. 2.4 The Paris Agreement on climate change establishes a global goal on adaptation, with specific targets on enhancing adaptive capacity, strengthening resilience, and 4 reducing vulnerability. It also recognizes the importance of minimizing and addressing loss and damage from extreme weather events and slow-onset events, including through risk assessment and risk insurance facilities (United Nations Framework Convention on Climate Change 2015). DRR cuts across several aspects of the SDGs, with 25 DRR-related targets found in 10 of the 17 SDGs, particularly in building resilient infrastructure (SDG9) and making cities resilient (SDG11). The Sendai Framework and the SDGs are directly tied together and use the same indicators in several cases.3 2.5 Over the past decade, the World Bank has evolved its stated approach to disaster management by focusing more on ex ante risk reduction than on disaster response and by pursuing more integrated approaches. The World Bank has issued progress reports to its Board of Executive Directors on mainstreaming disaster risk management in Bank Group operations every two years since 2014. In all of these reports, the World Bank explains how it has shifted its approach to disasters to focus more on disaster risk mitigation and preparedness than on disaster response. It reports that it has mainstreamed disaster risk considerations into country programs across sectors and pursued more integrated approaches across instruments (combining analytical work, advisory services, partnerships, and lending) and key DRR areas (risk identification, resilient infrastructure, risk preparedness, financial protection, and resilient reconstruction). 3. Evaluation Purpose and Scope 3.1 The purpose of this evaluation is to learn how the World Bank has helped client countries undertake DRR from natural hazards and how and how well it has achieved DRR outcomes. Based on these lessons, the evaluation will identify how the World Bank can enhance its performance on supporting countries to reduce disaster risk from natural hazards. 3.2 The evaluation will focus on disaster risks caused by natural hazards rather than other types of hazards or chronic stresses. Natural hazards include rapid-onset hazards such as cyclones, earthquakes, floods, landslides, tsunamis, and volcanic eruptions, as well as slow-onset hazards that may take years to develop, such as droughts. Both types of hazards, if not mitigated, can cause disasters. The evaluation does not cover disasters caused by other hazards (for example, macroeconomic or financial shocks; biological shocks, such as pandemics or epidemics; industrial accidents; or conflicts). The evaluation also does not cover chronic stresses (for example, resource degradation and pollution). This scoping decision was made because natural hazards represent perhaps the largest and most sustained form of World Bank DRR, because the evaluation needs to have sufficient depth to generate useful findings, and because other forms of risk reduction are covered by other IEG evaluations.4 5 3.3 The evaluation covers activities in client countries that aim to reduce the risk from future disasters. The evaluation covers actions that contribute to reducing the frequency or impact of future disasters caused by natural hazards. Figure 3.1 illustrates the parts of the disaster risk management cycle that are included in the evaluation (see the dark blue quarters). The evaluation does not cover elements of disaster risk management (such as emergency disaster response) and parts of recovery (for example, immediate relief activities and restoration of basic services and assets) that do not support DRR. In deciding which activities are included, the guiding principle is to include activities undertaken before a disaster that will reduce the negative impact of future disasters—while noting that this work may also occur as part of disaster recovery. As noted above in paragraph 1.4, this broad definition includes but goes beyond exposure and vulnerability reduction. Examples of included and excluded activities are in appendix table B.2. The evaluation covers the World Bank’s global work on DRR only to the extent that this influences its country work or is an important factor in explaining the World Bank’s evolving approach. 3.4 The evaluation covers the DRR activities of the World Bank during fiscal years 2010–20. The 10-year period was chosen to capture evidence of effectiveness from the closed portfolio of lending projects with DRR activities. It was also chosen to assess the evolution of the World Bank’s approach in response to key commitments and milestones, namely the Sendai Framework for DRR 2015–30, which cemented client commitments related to pivoting toward more ex ante risk-reduction activities and institutions. 3.5 The evaluation will not focus on public health emergencies or pandemic risk reduction. While public health emergencies share some common elements with natural hazards, the activities taken to manage them often differ, and they involve different stakeholders in the World Bank and often in client countries. A preliminary portfolio review identified a relatively small number of projects addressing pandemic risk reduction approved during the period covered by this evaluation.5 IEG is addressing the COVID-19 pandemic through two other evaluations, one focused on the Bank Group’s health and social interventions and the other focused on the Bank Group’s economic and financial interventions to respond to the COVID-19 crisis. 3.6 This DRR evaluation focuses on the World Bank. It does not include the International Finance Corporation or the Multilateral Investment Guarantee Agency since disaster risk management is not a major corporate priority for them. This focus is partly because many DRR activities are classic public goods or are core functions of government. A preliminary portfolio review identified only a limited International Finance Corporation portfolio relevant to DRR, mainly related to advisory work on disaster and agricultural insurance (see appendix B). 6 Figure 3.1. Evaluation Scope across the Four Phases of Disaster Risk Management Source: Independent Evaluation Group, using globally accepted phases of disaster risk management (for example, UNDRR). Note:  Included in the evaluation scope.  Excluded from the evaluation scope. DRR = disaster risk reduction; NRM = natural resource management; WRM = water resource management. Listed activities are illustrative examples and not exhaustive. For example, community-based approaches can cut across all aspects of disaster risk management, not just community-based preparedness. 3.7 The evaluation builds on and contributes to IEG’s work stream on climate change and environmental sustainability. IEG’s 2006 evaluation on disaster risk management found that although the World Bank demonstrated flexibility and effective coordination in disaster response, its attitude to disasters was reactive and tactical rather than proactive and strategic (World Bank 2006). The 2013 evaluation on climate change adaptation found a clear shift toward risk reduction by 2008–10 (World Bank 2012). The 2016 cluster Country Program Evaluation on small states looked at some of the most disaster-vulnerable countries and found that support for resilient infrastructure had been helpful but limited (World Bank 2016c). The 2019 evaluation on urban resilience considered broader issues of resilience, including but not limited to those related to disasters (World Bank 2019). The evaluation found that despite increasing integration of resilience characteristics in assessed operations (robustness, inclusion, redundancy, coordination, and reflectiveness), this integration has been inconsistent across the portfolio. This evaluation will also draw on recent IEG field-based project evaluations of disaster risk management projects. 7 Evaluation Portfolio 3.8 The evaluation covers the full range of World Bank interventions that support DRR as defined above between fiscal years 2010 and 2020. Based on the previously outlined scope, a preliminary portfolio review using text analytics identified 556 lending projects (including 92 additional financing), with a total commitment of US$54.7 billion, of which 480 are investment project financing; 69 are development policy financing, including catastrophe deferred drawdown operations; and 5 are Program-for-Results projects. Lending has taken place across 107 countries and all Regions. The preliminary review also identified 769 nonlending activities, of which 251 have relevant and available documentation on the Bank Group’s public Documents and Reports site.6 While a more thorough search will be conducted in the operations portal as part of the portfolio review and analysis, it should be noted that many DRR nonlending activities in this space include trainings, workshops, and other advisory services that are not covered by evaluation systems and hence do not generate substantial documentation. 3.9 Almost half of World Bank lending and nonlending DRR activities are mapped to the Urban, Disaster Risk, Resilience, and Land Global Practice (figure 3.2). Also, a sizeable number of projects are mapped to the following Global Practices: Water, Transport, and Environment and Natural Resources. Table 3.1 describes commonly occurring activities across the Global Practices. Almost one-third of the portfolio (28 percent) is in the East Asia and Pacific Region, primarily due to the number of activities conducted in small island states. In many cases, particularly in small states, individual projects are part of larger regional programs with a common design. Figure 3.2. DRR Lending and Nonlending Evaluation Portfolio by Global Practice a. World Bank DRR lending (n = 556) b. World Bank DRR nonlending (n = 769) URL URL WAT WAT Global Practice or Theme Transport Climate Change Global Practice ENB FCI SPJ ENB AGR SPJ MTI AGR FCI Other 0 50 100 150 200 250 0 100 200 300 Projects (no.) Projects (no.) Source: Independent Evaluation Group. Note: Charts exclude Global Practices with very few projects. AGR = Agriculture; DRR = disaster risk reduction; ENB = Environment, Natural Resources and Blue Economy; FCI = Finance, Competitiveness, and Innovation; MTI = Macroeconomics, Trade, and Investment; SPJ = Social Protection and Jobs; URL = Urban, Disaster Risk, Resilience, and Land; WAT = Water. 8 Table 3.1. Typical Disaster Risk Reduction Activities within the Evaluation Portfolio by Global Practice Global Practice Illustrative Activities Urban, Disaster Risk, • Disaster-resilient infrastructure (for example, roads, housing, slum upgrading, schools, Resilience, and Land and medical facilities), including in postdisaster reconstruction • Protective works (for example, coastal protection, flood banks, and emergency shelters) • DRR capacity building (regional, national, local), including disaster preparedness • DRR policy reform (for example, mainstreaming disaster into planning and building codes) • Disaster risk identification (for example, vulnerability assessment and hazard mapping) • Emergency preparedness, including emergency management and planning • Early-warning systems (including data systems) Water • Flood management (for example, protective works, drainage, institutional strengthening) • Drought risk management (for example, drought response plans and drought-resistant technologies) • Water resource management with link to flood or drought risk reduction • Irrigation with explicit flood or drought risk reduction Transport • Disaster-resilient infrastructure (for example, roads, ports, and airports) Environment and • Landscapes approaches (for example, watershed management, coastal zone Natural Resources management, nature-based solutions) with risk-mitigation effects • Policy and institutional development for climate-resilient planning and development Social Protection and • Disaster-responsive social protection or safety nets Jobs Macroeconomics, • Pre- or postdisaster DPF (including CAT DDO) with DRR prior actions supporting Trade, and mitigation and preparedness Investment Agriculture • Agricultural disaster risk mitigation (for example, grain and seed storage, agro-climatic information systems, and agriculture sector risk assessment and management plans) • Climate-smart or -resilient agriculture with explicit DRR Finance, • Disaster risk finance (for example, insurance and catastrophe bonds) Competitiveness, and • Postdisaster support for economic recovery (for example, SME support) with DRR Innovation elements Source: Independent Evaluation Group, based on preliminary portfolio review. Note: Listed activities may also occur in other Global Practices. The examples listed are indicative and not exhaustive. CAT DDO = catastrophe deferred drawdown option; DPF = development policy financing; DRR = disaster risk reduction; SME = small and medium enterprises. 3.10 Only a small portion of the World Bank project portfolio has evaluations validated by IEG. Of the 556 lending projects, 253 have closed. Of these, 117 have completion reports, and 114 have been validated by IEG. Many projects (103 total, of which 80 are closed) are recipient-executed trust fund activities that do not always generate a completion report and are not always validated by IEG; these projects will be covered by the evaluation to the extent possible, but fewer data are available for small trust-funded activities under US$5 million.7 9 Theory of Change 3.11 This evaluation uses a theory of change to guide its understanding of the World Bank’s contribution to DRR in client countries (figure 3.3). It was developed based on an initial review of strategy and project documents and consultations with key stakeholders in the World Bank. The evaluation questions and methods have been designed to test many of the causal assumptions embedded in this theory of change. A set of relevance questions, supported by data and case analysis, has been designed to explore how the World Bank’s upstream engagement can lead to client uptake of DRR activities. A set of effectiveness questions has been developed to test assumptions about the way that various DRR activities lead to reduced exposure and vulnerability of people and assets. The evaluation will assess the completeness of the proposed theory and note where missing elements or false assumptions are made evident through the emerging evaluation evidence. A revised theory will be presented in the final report. 3.12 The World Bank engages upstream to identify disaster risks, raise client awareness, and enable clients to undertake DRR priorities. As shown in the blue boxes in figure 3.3, the World Bank uses its advisory services and analytics in its policy dialogue and convening to raise clients’ awareness of their vulnerability to disasters and of opportunities for DRR. This process is influenced by World Bank internal factors, such as corporate priorities for climate change adaptation and resilience; coordination and incentives for working across multiple relevant practices; and financing, including a reliance on trust fund resources (primarily through the Global Facility for Disaster Reduction and Recovery). Clients may lack awareness or knowledge of their disaster vulnerabilities, face fiscal constraints, act on DRR primarily when there are reform champions, and find that DRR is not a political priority except after a serious disaster event. When clients are aware of their disaster risks and have the capacity to act (for example, available fiscal space and institutional capacity), they undertake priority DRR actions and investments. Clients that undertake DRR may do so with World Bank support, with other partners, or on their own. 3.13 The World Bank supports its clients to reduce disaster risk through a wide range of activities. The yellow boxes in figure 3.3 show key DRR activities, including physical mitigation through protective works and resilient infrastructure; policy and institutional reform (using both investment project financing and development policy financing); disaster preparedness measures, including early-warning systems, emergency planning and management, and community-based approaches; and disaster risk finance, such as developing insurance mechanisms or markets that function at sovereign, firm, and household levels. DRR activities each have their own intervention logic and are expected to generate intermediate outcomes, as shown in the gray boxes in figure 3.3. Physical investments seek to reduce the area or share of people and assets affected by disasters. 10 Policy reforms aim to change the behavior of governments, firms, and other actors. Preparedness measures aim to improve the ability of actors to respond to a disaster. Financial disaster risk management helps actors cope with disasters by transferring or mitigating financial risk. 3.14 DRR activities contribute to reductions in disaster exposure and vulnerability and in turn to lower mortality, morbidity, and economic impacts from disasters. The orange and green boxes in figure 3.3 show these effects. DRR activities may reduce the share of the population, livelihoods, and assets that are exposed and vulnerable to disasters. Disaster risk finance seeks to reduce the financial stress from disasters. 3.15 Successful DRR activities may also have transformative effects. The most successful interventions can make significant contributions to country or sector outcomes. This evaluation defines transformative effects based on IEG's 2016 evaluation on transformational engagements (World Bank 2016a): An activity has transformative effects if it addresses a major developmental challenge (relevance), if it addresses root causes to support a change in trajectory (depth of change), or if it causes large-scale impacts at a national level (scale of change). These effects could occur through a range of mechanisms: A successful project model might be replicated or expanded; an important policy change that is fully operationalized might induce behavior change in households or firms nationally; and an innovative financial instrument could create a new market with self- sustaining expansion. 11 Figure 3.3. Evaluation Theory of Change Source: Independent Evaluation Group. Note: DRR = disaster risk reduction. 12 4. Evaluation Questions 4.1 This evaluation will answer the following two key evaluation questions, which are each facilitated by three subquestions: Question 1. Has the World Bank’s support for DRR been relevant, and what factors have facilitated or limited the relevance of this support? • 1a. To what extent has the World Bank supported DRR for hazards posing serious disaster risks in disaster-vulnerable countries? • 1b. What has worked in the World Bank’s efforts to influence clients to undertake DRR, including in partnership with other stakeholders? • 1c. To what extent has the World Bank evolved its approach to DRR in line with good practices? Question 2. How effectively has the World Bank supported DRR, and what factors explain this effectiveness? • 2a. How well does the World Bank articulate and capture DRR outcomes, including for whom they are intended, and how can this be improved? • 2b. For key DRR approaches and activities, how effective have they been? • 2c. What has worked to achieve transformative DRR effects in client countries in the most successful cases? 5. Evaluation Design, Methodology, and Limitations 5.1 The evaluation is designed to answer the main evaluation questions about the relevance and effectiveness of the World Bank’s support for DRR. As such, it includes three subquestions for each of the key relevance and effectiveness questions. The design uses a “building blocks” approach that features a round of data collection and analysis of portfolio trends followed by several deep dives to derive explanatory factors and generate enhanced learning. The design and the accompanying methods are included in figure 5.1, expanded on in the evaluation design matrix in appendix A, and explained in sequence in the sections below. 13 Figure 5.1. Evaluation Design Source: Independent Evaluation Group. Note: DRR = disaster risk reduction; EQ = evaluation question; M&E = monitoring and evaluation; PRA = portfolio review and analysis. Relevance 5.2 The evaluation seeks to assess three aspects of relevance regarding the World Bank’s support for DRR. 5.3 First, it asks whether the World Bank is engaging strategically in those places where different types of disasters pose, or will pose, serious threats. To answer this question, the evaluation will first undertake a global hazard and vulnerability analysis, disaggregated by disaster type, using both historical and predictive data from existing analyses. Using basic portfolio identification tools, the evaluation will then juxtapose these data against the World Bank’s lending and advisory portfolio. On that basis, the evaluation will analyze overlaps and gaps at the country level while seeking to mitigate data gaps and biases, such as for countries experiencing FCV. 5.4 Second, the evaluation will source lessons on what works to raise client awareness and support for DRR at the country level. The World Bank uses actions including analytical work, policy dialogue, different types of investments, and convening partners to help clients understand their disaster risk and act on priorities through investments and policy reforms. This question is related to relevance because it addresses how to build engagements to get to DRR action but not the effectiveness of those actions. To identify what works, the evaluation will use an explanatory case- analysis method (that is, a case method that aims to answer “how” and “why” 14 questions). The case method will be designed to derive explanatory factors about what it takes to influence clients to invest in DRR and to adopt DRR-sensitive policies, including how the World Bank works with partners. The unit of analysis will be the country. Two types of situations will be selected: those where there has been high uptake of World Bank advice and investment in DRR by clients, and those where the World Bank has sought to engage clients on DRR but has received little to no uptake. Roughly 12 cases, covering different disaster types and providing regional variability, are envisioned. The methodology entails a deeper review of high-uptake cases, since more information will be available, and a broader review of low-uptake cases. The evaluation will also consider how factors may vary across country contexts, particularly for small states and countries experiencing FCV. 5.5 Third, the evaluation seeks to assess the degree to which the World Bank has evolved its approach to DRR in line with good practices. The evaluation proposes to use a deductive approach. It will create criteria based on known good practices and use portfolio review and analysis to code the incidence and trends associated with these practices, noting when and where they have occurred over time, as well as gaps at the portfolio and country levels. Examples of good practices include a shift from disaster response to predisaster vulnerability reduction, pursuit of integrated approaches, mainstreaming of disaster considerations in sectors, and appropriate use of nature-based solutions. Explanatory factors about internal institutional barriers to achieving good practice solutions will be derived from follow-up interviews with World Bank staff to propose solutions to overcome these barriers in the future. Effectiveness 5.6 The evaluation seeks to assess three aspects of effectiveness regarding the World Bank’s support for DRR. The evaluation developed a cascading set of questions on effectiveness designed to first establish the type of information that is known to exist on DRR outputs and outcomes, and then to interrogate the cases where these data exist to derive more causal information. 5.7 First, the evaluation will identify how the World Bank articulates DRR outcomes in its project objectives and theories of change, and how it captures those outcomes with indicators. This aim will be achieved using portfolio review and analysis of the World Bank lending portfolio of projects with DRR activities, which will be disaggregated by disaster and project type. The evaluation will then highlight good practices to generate knowledge on what works to capture DRR-related outcomes, including for whom they are achieved. This step will include an assessment of the degree to which evidence exists on the distributional impact of DRR activities. 15 5.8 To complement the review of outcomes in the DRR portfolio, the evaluation will commission a selective review of indicators and measurement tools being used by “best- in-class” agencies globally. These agencies will be identified during the evaluation, based on recognition by technical experts and published literature. Recognizing that best-in-class agencies might be using tools that require high capacity, the evaluation will make this information available while placing it in the proper context. 5.9 Second, by capturing good practices (in evaluation question 2a), the evaluation will generate lessons on factors that support effectiveness for key activities in the DRR portfolio. The evaluation will also use explanatory case analysis to derive lessons about effectiveness for about six key DRR activity types to be determined. These key activities will be chosen based on (i) portfolio review, ensuring that they are representative of large bodies of work and relevant for the future pipeline, and (ii) consultations with World Bank staff and management (that is, where there are questions about effectiveness of certain approaches and where World Bank staff and management are seeking more evidence). Examples of key activities include the mainstreaming of DRR considerations into buildings and infrastructure; protective works, including nature- based solutions; insurance and disaster risk finance mechanisms; and early-warning systems and other community-based preparedness approaches. In assessing effectiveness, the evaluation will consider each activity in terms of its intended results based on the intervention logic and will report on distributional impacts when these data exist. 5.10 Third, the evaluation will identify and draw lessons from those instances when World Bank DRR activities had transformative effects. As defined above, engagements can have transformative effects based on their relevance, depth of change, and scale of change. The evaluation will undertake transformational case analysis to identify lessons for achieving these effects for DRR. Each case may span multiple World Bank interventions that contribute to a transformative effect in a particular country. Cases will be selected based on the presence of transformative effects. Scoping interviews will be used to generate candidate cases that may have transformative effects, which will then be screened for plausibility based on available evidence. Each case will then be assessed based on the evidence of its contributions to country or sector outcomes and success factors for generating transformational effects will be identified. Cases will draw on interviews and a review of relevant literature, including evaluations. Methodological Limitations 5.11 DRR outcomes are inherently difficult to measure because they are a reduction in the negative effects of a probabilistic future shock. Avoided losses cannot be directly measured. Reduced expected mortality and damage are a function of both the 16 probability distribution of natural hazards of varying intensities and the effectiveness of risk reduction activities. The effects of a disaster cannot be measured until an actual hazard strikes, and then measuring the effectiveness of DRR is dependent on a good counterfactual (Maxwell et al. 2009). Potential tools and methods for assessing DRR outcomes include information mapping, universal data sets to monitor trends, case study analysis (including both qualitative and quantitative data), literature review of effective DRR models, building on existing systems, evaluating local coping strategies, and assessing characteristics of a disaster-resilient community (Feinstein International Center 2011; Twigg 2009). Additional challenges include multiple scales of analysis leading to aggregation problems, the absence of objective benchmarks, and dynamic systems that involve different combinations of explanatory variables over time and place (Thomalla et al. 2006). 5.12 In contexts of FCV countries, data gaps can distort and thus understate disaster risk. An increasing number of countries are affected by recurring natural hazards and protracted crises associated with FCV. However, this association may not be overtly apparent from the data available on disaster risk in these countries. This data challenge, especially as it relates to data-poor countries—like those experiencing FCV—will be managed as part of the methodology that will be designed to answer question 1a. 5.13 In consideration of the constraints posed by the COVID-19 pandemic, the evaluation has been designed to be conducted “on desk.” The evaluation team will seek guidance from partner agencies on good emerging practices on conducting virtual missions. If the evaluation team chooses to partner with local agencies or individuals, necessary precautions will be taken to mitigate exposure risks. Care will also be taken to avoid putting unnecessary strain on already overstretched public systems (as part of planned consultations or interviews). 6. Quality Assurance Process 6.1 The Approach Paper and evaluation will undergo standard IEG quality assurance processes, including internal IEG and World Bank management review and external peer review. This evaluation will be peer-reviewed by the following experts on DRR: • Katie Peters, senior research fellow at the Overseas Development Institute since 2011. She leads the institute’s portfolio on the intersection of natural hazard– related disasters, climate change, and conflict, and her research focuses on DRR in fragile and conflict-affected states, the relationship between climate change and conflict, and the securitization of climate change. 17 • Mohamed Béavogui, an expert in agricultural finance, was elected as the first African general manager of the African Risk Capacity in January 2015. He served as director-general and United Nations assistant secretary general until 2020. Mr. Béavogui has over 25 years of international experience in development, and previous to his appointment with African Risk Capacity was the director of partnerships and resource mobilization and senior adviser to the president of the United Nations International Fund for Agricultural Development. Mr. Béavogui thus has extensive experience on international development, food security, and disaster risk management, including drought. • Paola Albrito, chief of the intergovernmental processes, Interagency Cooperation and Partnerships Branch, UNDRR. Formerly head of the UNDRR regional office for Europe, Ms. Albrito has over 15 years of experience at UNDRR. 7. Staffing and Resources 7.1 This evaluation will be task-managed by Lauren Kelly, lead evaluation officer, and Stephen Hutton, under the guidance of Marialisa Motta, manager of the Financial, Private Sector, Infrastructure, and Sustainable Development Unit, and José Carbajo Martinez, director of the Financial, Private Sector, and Sustainable Development Department. The team will include as core team members Joy Butscher, evaluation officer; Mees van der Werf, extended term consultant; and Romayne Pereira, program assistant. Estelle Raimondo, senior evaluation officer, and Harsh Anuj, data scientist, will also provide methodological and other inputs. 7.2 The evaluation report will be sent to Bank Group management for review and submitted to the Committee on Development Effectiveness in the fourth quarter of fiscal year 2022. 8. Expected Outputs, Outreach, and Tracking 8.1 Expected outputs. The main output will be a final evaluation report that will be delivered to the Board’s Committee on Development Effectiveness after integrating feedback from World Bank management. The evaluation will also produce intermittent outputs to communicate important findings and messages that can be used by key counterparts in a timely way (for example, brown-bag lunches, contributions to Learning Weeks, briefings, thematic papers, micro products, blogs, and so on). 8.2 Engagement. The evaluation will be conducted in close collaboration with internal stakeholders. Throughout the evaluation process, the team will engage with relevant technical counterparts across the World Bank (including Global Practices, Global Solution Groups, country teams, and so on) as identified through stakeholder 18 analysis. Regular consultations will be held at key stages of the evaluation to (i) seek feedback on preliminary findings; (ii) surface lessons that support operational learning; (iii) create ownership of the evaluation; and (iv) ensure the evaluation focus and findings are relevant and useful for the intended users. While developing the Approach Paper, the evaluation team consulted with 30 World Bank management and technical staff to inform the proposed scope and approach. 8.3 Audience. The primary audience of this evaluation is the Board and World Bank management and staff working on DRR. However, the evaluation findings will also be relevant to a broader audience, including disaster agencies, government officials, multilateral and bilateral agencies, donors, private sector actors, nongovernmental organizations, civil society, academia, and so forth. An external stakeholder mapping exercise identified approximately 30 relevant agencies and organizations whose mandates align with DRR. This mapping exercise will inform the external outreach strategy applied throughout the evaluation. 8.4 Outreach and tracking. A communications and influence strategy—including both internal and external forums—will be developed with the IEG’s Knowledge and Communications Department. This strategy will include launching and disseminating the evaluation once it is disclosed, as well as publicizing intermittent outputs such as brown-bag lunches, contributions to Learning Weeks, briefings, blogs, and so on. Formal venues will be sought to engage relevant actors to encourage uptake of evaluation products and findings. For example, key conferences and events that could be targeted for wider outreach include the European Forum for Disaster Risk Reduction and other UNDRR events.8 The evaluation peer reviewers will also help develop outreach suggestions as part of their wider networks. The communications and influence strategy will include detailed indicators to track the report’s influence. 1The statistic was calculated from data obtained from EM-DAT, the International Disaster Database, part of the Centre for Research on the Epidemiology of Disasters at Université catholique de Louvain (www.emdat.be). 2This statistic was calculated from data obtained from the Query Wizard for International Development Statistics, Organisation for Economic Co-operation and Development, Paris, http://stats.oecd.org/qwids/ (accessed 2021). 3For example, Sustainable Development Goal indicators 11.b.1 and 13.1.2 track the number of countries implementing national disaster risk-reduction strategies in line with the Sendai Framework. 4For example, pollution abatement is covered in the 2017 Independent Evaluation Group (IEG) evaluation Toward a Clean World for All: An Evaluation of the World Bank Group’s Support to Pollution 19 Management; work to address natural resource degradation is covered by IEG’s 2021 evaluation The Natural Resource Degradation and Vulnerability Nexus: An Evaluation of the World Bank’s Support for Sustainable and Inclusive Natural Resource Management; efforts to mainstream resilience are covered in IEG’s 2019 evaluation Building Urban Resilience: An Evaluation of the World Bank Group’s Evolving Experience (2007–2017); conflict is covered by an IEG work stream on World Bank engagement in countries experiencing fragility, conflict, and violence; pandemic response is covered by two forthcoming evaluations on support to protect human capital and support to address economic consequences; and financial resilience is covered by IEG’s forthcoming evaluation of efforts to address country-level fiscal and financial vulnerabilities. 5Previous IEG work assessed the World Bank’s pandemic preparedness efforts during 2006–13 and found that though the global program and 83 operations had many successes, support for zoonotic disease control and pandemic preparedness were not sustained (World Bank 2014). After 2013, relatively few projects supported pandemic risk reduction prior to the coronavirus (COVID-19) pandemic. 6 The Documents and Reports site is an official disclosure mechanism for the World Bank Group ’s final reports. The repository contains official documents and reports that are made available to the public in accordance with the World Bank’s access to information policy to better share the institution’s knowledge base. The Documents and Reports site contains final and official documents and reports from 1946 through the present, including Board documents (items concerning meetings of the Board of Executive Directors); country focuses (strategic priorities and directions for lending activities); economic and sector work (in-depth background studies); project documents (loan- and credit-related documents released to the public according to the project cycle, including legal agreements); and publications and research (formal publications, working papers, and informal series from departments around the Bank Group). 7IEG does not normally validate the self-evaluations for small trust-funded projects (below US$5 million). 8These might include regional ministerial disaster risk reduction conferences and disaster risk reduction platforms taking place throughout 2022; the Global Platform for Disaster Risk Reduction 2022 in Bali, Indonesia; the Understanding Risk Conference; and the Fragility Forum. 20 References Feinstein International Center. 2011. Examining Linkages between Disaster Risk Reduction and Livelihoods: Literature Review. Boston: Feinstein International Center, Tufts University. https://fic.tufts.edu/wp-content/uploads/DRR-lit-review.pdf. Global Facility for Disaster Reduction and Recovery. 2013. Annual Report 2013: Inroads to Resilience. Washington, DC: Global Facility for Disaster Reduction and Recovery. https://www.gfdrr.org/sites/default/files/publication/annual-report-2013.pdf. Haile, Gebremedhin Gebremeskel, Qiuhong Tang, Siao Sun, Zhongwei Huang, Xuejun Zhang, and Xingcai Liu. 2019. “Droughts in East Africa: Causes, Impacts and Resilience.” Earth- Science Reviews 193: 146–61. doi:10.1016/j.earscirev.2019.04.015. Hallegatte, Stéphane, Jun Rentschler, and Julie Rozenberg. 2019. Lifelines: The Resilient Infrastructure Opportunity. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/31805. Hallegatte, Stéphane, Jun Rentschler, and Brian Walsh. 2018. Building Back Better: Achieving Resilience through Stronger, Faster, and More Inclusive Post-Disaster Reconstruction. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/29867. Hallegatte, Stéphane, Adrien Vogt-Schilb, Mook Bangalore, and Julie Rozenberg. 2017. Unbreakable: Building the Resilience of the Poor in the Face of Natural Disasters. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/25335. Holtsberg, Maria. 2020. “Gender-Responsive and Disability-Inclusion in Disaster Risk Reduction in Asia Pacific.” PreventionWeb (blog), October 7, 2020. https://www.preventionweb.net/blog/gender-responsive-and-disability-inclusion- disaster-risk-reduction-asia-pacific (August 11, 2021). Keefer, Philip. 2009. “Disastrous Consequences: the Political Economy of Disaster Risk Reduction.” Working Paper. Washington, DC: World Bank. https://www.scribd.com/document/468141120/Keefer-Disastrous-Consequences. Maxwell, Daniel, Girum Tadesse, Mary Mukwavi, Shimelis Hailu, Wolde Gebreal Zewold, and Abraha Gebrekiros. 2009. Baseline Report: Africa Community Resilience Project. Tsaeda Amba Woreda, Eastern Tigray, Ethiopia. Boston: Feinstein International Center, Tufts University. https://fic.tufts.edu/publication-item/baseline-report-africa-community-resilience- project/. Peters, Katie. 2019. Disaster Risk Reduction in Conflict Contexts: An Agenda for Action. London: Overseas Development Institute. https://odi.org/en/publications/disaster-risk-reduction- in-conflict-contexts-an-agenda-for-action/. 21 Sen, Amartya. 1983. Poverty and Famines: An Essay on Entitlement and Deprivation. Oxford: Oxford University Press. Oxford Scholarship Online, 2003. doi: 10.1093/0198284632.001.0001. Tang, Rumei, Jidong Wu, Mengqi Ye, and Wenhui Liu. 2019. “Impact of Economic Development Levels and Disaster Types on the Short-Term Macroeconomic Consequences of Natural Hazard-Induced Disasters in China.” International Journal of Disaster Risk Science 10: 371– 85. Tanner, Thomas, Aditya Bahadur, and Marcus Moench. 2017. Challenges for Resilience Policy and Practice. London: Overseas Development Institute. https://cdn.odi.org/media/documents/11733.pdf. Thomalla, Frank, Tom Downing, Erika Spanger‐Siegfried, Guoyi Han, and Johan Rockström . 2006. “Reducing Hazard Vulnerability: Towards a Common Approach between Disaster Risk Reduction and Climate Adaptation.” Disasters 30 (1): 39–48. doi:10.1111/j.1467- 9523.2006.00305.x. Twigg, John. 2009. Characteristics of a Disaster-resilient Community: A Guidance Note, version 2. https://discovery.ucl.ac.uk/id/eprint/1346086/1/1346086.pdf. UNDRR (United Nations Office for Disaster Risk Reduction). 2016. Report of the Open-Ended Intergovernmental Expert Working Group on Indicators and Terminology Relating to Disaster Risk Reduction. Geneva: UNDRR. https://www.undrr.org/publication/report-open-ended- intergovernmental-expert-working-group-indicators-and-terminology-0. UNDRR. 2019. Global Assessment Report on Disaster Risk Reduction 2019. Geneva: UNDRR. https://gar.undrr.org/report-2019. UNDRR. 2020a. The Human Cost of Disasters: An Overview of the Last 20 Years, 2000–2019. Geneva: UNDRR. https://www.undrr.org/publication/human-cost-disasters-2000-2019. UNDRR. 2020b. Integrating Disaster Risk Reduction and Climate Change Adaptation in the UN Sustainable Development Cooperation Framework. Geneva: UNDRR. https://www.undrr.org/publication/integrating-disaster-risk-reduction-and-climate- change-adaptation-un-sustainable. UNFCCC (United Nations Framework Convention on Climate Change). 2015. Paris Agreement to the United Nations Framework Convention on Climate Change. T.I.A.S. No. 16-1104 (December 12, 2015). Bonn: UNFCCC. https://unfccc.int/sites/default/files/english_paris_agreement.pdf. UNISDR (United Nations International Strategy for Disaster Risk Reduction). 2009. 2009 UNISDR Terminology on Disaster Risk Reduction. Geneva: UNISDR. https://www.unisdr.org/files/7817_UNISDRTerminologyEnglish.pdf 22 UNISDR. 2015. Making Development Sustainable: The Future of Disaster Risk Management, Global Assessment Report on Disaster Risk Reduction 2015. Geneva: UNISDR. https://www.undrr.org/publication/global-assessment-report-disaster-risk-reduction- 2015. United Nations. 2015. Resolution 69/283, Sendai Framework for Disaster Risk Reduction 2015 –2030. A/RES/69/283 (June 3, 2015). New York: United Nations. https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030- 0. Wilkinson, Emily. 2012. Transforming Disaster Risk Management: A Political Economy Approach. London: Overseas Development Institute. https://cdn.odi.org/media/documents/7555.pdf. World Bank. 2006. Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/7001. World Bank. 2012. Adapting to Climate Change: Assessing World Bank Group Experience. Phase III of the World Bank Group and Climate Change. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/21106. World Bank. 2013a. Turn Down the Heat: Climate Extremes, Regional Impacts, and the Case for Resilience. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/975911468163736818/Turn-down-the-heat- climate-extremes-regional-impacts-and-the-case-for-resilience-full-report. World Bank. 2013b. World Development Report 2014: Risk and Opportunity —Managing Risk for Development. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/16092. World Bank. 2014. Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza 2006-13. Independent Evaluation Group. Washington, DC: World Bank. https://documents.worldbank.org/en/publication/documents- reports/documentdetail/688581468001166444/responding-to-global-public-bads-learning- from-evaluation-of-the-world-bank-experience-with-avian-influenza-2006-13. World Bank. 2016a. Supporting Transformational Change for Poverty Reduction and Shared Prosperity. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/supporting-transformational-change- poverty-reduction-and-shared-prosperity. World Bank. 2016b. World Bank Group Climate Change Action Plan 2016–2020. Washington, DC: World Bank Group. https://openknowledge.worldbank.org/handle/10986/24451. 23 World Bank. 2016c. World Bank Group Engagement in Small States: The Cases of the OECS, PICs, Cabo Verde, Djibouti, Mauritius, and the Seychelles. Independent Evaluation Group. Washington, DC: World Bank. https://documents.worldbank.org/en/publication/documents- reports/documentdetail/390181475038805949/world-bank-group-engagement-in-small- states-the-cases-of-the-oecs-pics-cabo-verde-djibouti-mauritius-and-the-seychelles- clustered-country-program-evaluation. World Bank. 2018. Sustainable Financing for Sustainable Development: World Bank Group Capital Package Proposal. Washington, DC: World Bank. https://www.ifc.org/wps/wcm/connect/280f4d9f-b889-4918-9eb2- bab6fc57c2f3/WBG+capital+package_post+DC+release.pdf?MOD=AJPERES&CVID=muW J8zA. World Bank. 2019. Building Urban Resilience: An Evaluation of the World Bank Group’s Evolving Experience (2007–2017). Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/evaluations/urban-resilience. World Bank. 2020. Additions to IDA Resources: Nineteenth Replenishment—Ten Years to 2030: Growth, People, Resilience. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/459531582153485508/Additions-to-IDA- Resources-Nineteenth-Replenishment-Ten-Years-to-2030-Growth-People-Resilience. World Bank and International Monetary Fund. 2017. Forward Look: A Vision for the World Bank Group in 2030—Progress and Challenges. Washington, DC: World Bank and International Monetary Fund. https://www.devcommittee.org/sites/dc/files/download/Documentation/DC2017- 0002.pdf. 24 Appendix A. Evaluation Design Matrix Table A.1. Evaluation Design Matrix Evaluation Question or Data and Measurement Subquestion Logic Methods Data Limitations and Mitigation Question 1. Has the World Bank’s support for DRR been relevant, and what factors have facilitated or limited the relevance of this support? 1a. To what extent has the World This question provides the basis • Global hazard and vulnerability Historical hazard data: • Biases in historical disaster Bank supported DRR for hazards for an assessment of whether the analysis (disaggregated by • Proportion of persons affected data and their quality may posing serious disaster risks in World Bank engages strategically disaster type) • Damage as a percentage of undercount low-income and disaster-vulnerable countries? in those places where disaster • Country portfolio review: ASA gross domestic product FCV countries and hazards that risk poses, or is likely to pose, and lending analysis of DRR are low frequency (for • Available from EM-DAT and serious threats. It includes an engagements example, earthquakes) or slow other sources (for example, analysis of whether the World • Gap analysis: Analysis of gap moving (for example, drought). databases for Sendai Bank helps clients reduce the risk between global data and • Predictive analysis will require Framework monitoring, the of high- and low-frequency country engagement. forward-looking estimates of DesInventar Consolidated hazards in high- and medium- worsening vulnerability based Losses Database, the UN vulnerability countries. on climate change and OCHA INFORM index, and so on) changes in the built environment. Predictive hazard data: • Existing models (climate change, urbanization, Maplecroft climate change vulnerability analysis, and so on) 1b. What has worked in the This question recognizes that Country case analyses: • World Bank portfolio data • Cannot measure or quantify World Bank’s efforts to influence there is underinvestment in DRR • Desk review of DRR (strategy, ASA, and lending) in total DRR investment clients to undertake DRR, and that policy frameworks do engagements selected countries • Data biases can be overcome including in partnership with not always facilitate risk • Interviews for case countries • Use of computer-assisted through triangulating sources other stakeholders? reduction. It aims to derive with key DRR and CMU staff, qualitative data analysis of interviews. explanatory factors about what it clients, key development software to organize and • The high-or-low method of takes to influence clients to partners, DRR experts, or DRR- analyze interview data case selection will “go deep” invest in DRR and adopt DRR- relevant nongovernmental on cases where there has been sensitive policies, with a focus on 25 Evaluation Question or Data and Measurement Subquestion Logic Methods Data Limitations and Mitigation specific World Bank organizations (to capture uptake of DRR, since there is contributions. The analysis aims influence of policy dialogue, likely more information; it will to understand enabling factors convening, and analytics) go “broad” and “light” on cases of influence and barriers in cases Case selection: where there has been modest where clients have or have not • Seek to identify cases of real or no uptake, since it is likely in invested. contributions, where World these cases there will be less Bank made a difference in information and therefore it is influencing prioritization. necessary to capture a larger number of cases. • Base potential case identification on (i) key World Bank staff expert interviews, (ii) presence of indicative portfolios, and (iii) use of high or low (prioritization or influence) selection criteria (that is, cases of successful and unsuccessful attempts to build investment) with enough breadth to derive explanatory factors germane to multiple cases. • Ensure coverage of multiple hazards and country types. • Include an FCV and small-state lens. 1c. To what extent has the World Counterparts report that the • PRA at two levels: (i) general • Portfolio-level data It is difficult to observe if some Bank evolved its approach to World Bank has sought to trend analysis at the portfolio • Project and key ASA design part of the integrated approach DRR in line with good practices? pursue integrated approaches to level using text analytics documents is not needed or is covered by DRR in client countries and has followed by manual screening, the client or other development shifted its approach to the DRR and (ii) country-level analysis partner, especially in high- portfolio by mainstreaming DRR for integrated approaches capacity International Bank for into sectors, shifting from using quantitative and Reconstruction and disaster response to predisaster qualitative methods Development countries. This vulnerability reduction, using (potentially with a sample) 26 Evaluation Question or Data and Measurement Subquestion Logic Methods Data Limitations and Mitigation nonstructural interventions, • Interviews with World Bank limitation will be mitigated adopting nature-based solutions, managers, leads, and through staff interviews. and so on. We seek to assess if coordinators for explanatory these changes have occurred factors and to learn about what internal • Review of relevant documents, factors have helped the World such as key analytical flagships, Bank move in this direction and disaster risk management what barriers remain. mainstreaming, and board updates Question 2. How effectively has the World Bank supported disaster risk reduction, and what factors explain this effectiveness? 2a. How well does the World Based on a preliminary PRA, • Indicator results analyses • Project documents and results Data types vary across hazards Bank articulate and capture DRR most interventions lack more within activity and hazard frameworks and activity types, which makes outcomes, including for whom than output data, and this types from PRAs (table of • Relevant external literature cross-comparisons difficult. they are intended, and how can deficiency varies across Global contents and results this be improved? Practices. frameworks) • Selective review and synthesis of external literature on how other “best-in-class” DRR organizations measure results, which could draw on work such as the Itad work on the BRACED program. 2b. For key DRR approaches and While there are evidence gaps Expanded PRA: • Portfolio data, including • Effectiveness analysis requires activities, how effective have on outcomes (see 2a), there is • Identify discrete activity types project documents for projects looking at results from projects they been? some evidence on effectiveness based on PRA component with each activity approved early in the across the broad range of analysis. • Existing evaluations (ICR, ICRR, evaluation period. different DRR activities. This • Select key activity types based PPAR) • The number of closed projects question will identify key on purposive selection criteria, • Other evaluations in each key activity type may activities that are useful for including presence in closed be limited. • Key informant interviews in counterparts and represent large project portfolio, innovative or • Limitations exist in the each activity type parts of the portfolio, and then it growing approaches, and effectiveness data and will assess the effectiveness of expressed stakeholder information on “why” in ICRs. those activities in terms of DRR demand. results and generate lessons on 27 Evaluation Question or Data and Measurement Subquestion Logic Methods Data Limitations and Mitigation factors that support • In each activity type, identify • PPARs cover projects that were effectiveness. For example, this and assess results and factors approved prior to the question might look into policy of effectiveness using a evaluation period, but they can lending instruments, insurance saturation method. Synthesize be used where relevant. and disaster risk finance, results. community preparedness, early- • Triangulate and corroborate warning systems, protective emerging findings with works, and resilient buildings technical experts. and infrastructure. 2c. What has worked to achieve Transformative interventions are • Transformational case analysis: Consultations for candidates transformative DRR effects in those that make substantial Using a structured template, may be biased; cases will be client countries in the most contributions to country or the analysis will assess validated. successful cases? sector outcomes through evidence of the World Bank’s positive spillover or indirect contribution to country or effects, such as demonstration sector outcomes by analyzing and scale-up or adoption, policy and identifying success factors or institutional changes, or for positive spillover or indirect market creation. This question effects, such as demonstration will identify lessons for achieving and scale-up or adoption, significant results that arise from policy or institutional changes, direct project effects. or market creation. • Sources of information: The analysis would draw on interviews with World Bank staff, clients, key development partners, DRR experts, or DRR- relevant nongovernmental organizations and a review of relevant literature, including relevant evaluations. • Case selection: Identify candidate cases with transformative effects from scoping interviews, screened 28 Evaluation Question or Data and Measurement Subquestion Logic Methods Data Limitations and Mitigation for plausibility based on available evidence. Source: Independent Evaluation Group Note: ASA = advisory services and analytics; BRACED = Building Resilience and Adaptation to Climate Extremes and Disasters; CMU = Country Management Unit; DRR = disaster risk reduction; FCV = fragility, conflict, and violence; ICR = Implementation Completion and Results Report; ICRR = Implementation Completion and Results Report Review; PPAR = Project Performance Assessment Report; PRA = portfolio review and analysis; UN OCHA = United Nations Office for the Coordination of Humanitarian Affairs. 29 Appendix B. Preliminary Portfolio Identification and Review A preliminary portfolio review and analysis was conducted to (i) identify the relevant portfolio based on the definition of disaster risk reduction (DRR) used by this evaluation; (ii) understand the range of DRR activities supported by the World Bank; (iii) assess their general theories and components; and (iv) take stock of DRR-related indicators and monitoring and evaluation frameworks (including to understand the level of outcome orientation in the portfolio). This preliminary review was used to determine the evaluation scope, develop the evaluation theory of change (see figure 3.3), and inform the evaluation questions and methodological design. Portfolio Identification World Bank To identify the relevant World Bank lending and nonlending portfolio, the evaluation used several methods and means of verification, including (i) project theme data, (ii) text analysis of operational data, (iii) manual inputs from technical consultations, and (iv) manual screening and verification. First portfolio identification method: Thematic coding. Relevant World Bank operational themes were identified (see table B.1) to generate an initial list of 743 lending and 715 nonlending projects. Table B.1. Theme Codes Relevant to the Evaluation Used for Portfolio Identification No. Theme Description 75 Disaster Risk Processes for designing, implementing, and evaluating strategies, policies, and measures to Management improve the understanding of disaster risk, foster risk reduction and transfer, and promote continuous improvement in disaster preparedness, response, and recovery practices, with the explicit purpose of increasing human security, well-being, quality of life, and sustainable development. 751 Disaster Activities supporting response, recovery, and reconstruction after a natural disaster in affected Response countries, equipping governments and disaster risk management practitioners with the and necessary skills and resources to conduct their own postdisaster assessments and resilient Recovery reconstruction planning, and supporting the implementation of large reconstruction programs. 752 Disaster Risk Technical advice, capacity building, and implementation assistance for governments, civil Reduction society, and the private sector to create and improve policies and legislation needed for better land-use planning and to drive investment aimed at reducing risk based on risk information. 753 Disaster Activities aiming to improve forecasting and early-warning systems, contingency and Preparedness emergency response plans, civil protection services, and protocols to help local communities anticipate, prepare for, and quickly respond to disasters. 754 Flood and Used to capture support for physical infrastructure investments, including both greenfield and Drought Risk rehabilitation projects, and institutional capacity-building support to strengthen flood and Management drought risk management. 30 No. Theme Description 331 Disaster Risk Agricultural Market Development: Development of micro- or meso-level insurance products Finance and markets in support of disaster risk financing for agriculture. Involves increasing the capacity to use domestic insurance markets to support financial protection of households and firms against disasters. Insurance-Based Solutions for Resilient Livelihoods: Application of insurance-based tools and approaches in disaster risk financing for resilient livelihoods. Involves applying actuarial skills and techniques to the design of shock-responsive safety net systems that provide financial protection to vulnerable households and communities. Responds to growing momentum to explore the use of cash transfers as a response mechanism to disasters and facilitating a greater role for national actors in humanitarian response. Sovereign Disaster Risk Financing: Increasing the capacity of sovereigns to better plan, prepare for, and manage the financial aspects of disaster-related risks. Links to work on public financial management, public debt management, macroeconomics and fiscal stability, and the structuring and execution of financial solutions. Subnational Disaster Risk Financing: Increasing the capacity of subnationals and state-owned enterprises to better plan, prepare for, and manage the financial aspects of disaster-related risks. Links to work on public financial management, public debt management, macroeconomics and fiscal stability, and the structuring and execution of financial solutions. Public Financial Management of Natural Disasters: Developing policy frameworks and implementation plans to support a more comprehensive approach to public financial management of natural disasters. Source: World Bank 2016 theme code definitions. Second portfolio identification method: Text analysis. To ensure comprehensiveness, the evaluation team used text analysis to supplement the theme code search. First, the team created a DRR text taxonomy: a list of keywords and phrases that frequently occur in the DRR space, such as the names of specific hazard types (disaster, flood, drought, hazard, catastrophe, earthquake, seismic, cyclone, hurricane, typhoon, landslide, mudslide, tsunami, and so on). The search was performed in key parts of project descriptions (for example, abstracts of project documents, project development objectives, project descriptions, activity summaries, component titles, indicator titles). Using text analysis, an additional 326 lending and 634 nonlending projects were identified. This increased the total number of projects for manual screening and verification to 1,069 lending and 1,349 nonlending projects. Third portfolio identification method: Manual inputs. Inputs from operations management and past evaluations were incorporated manually. For example, the nature-based solutions portfolio was imported manually if the projects were not already captured through themes and text analysis (see above). Fourth portfolio identification method: Manual verification. All lending and nonlending projects identified through the above searches (n = 2,418) were subsequently manually screened to verify their relevance to the evaluation scope (see inclusion and exclusion rules in table B.2.). Project development objectives, component titles, project abstracts, and key performance indicators were screened during this process. Projects outside the evaluation scope and false positives (for example, projects with phrases such 31 as “hazardous waste,” “flood the market,” and so on) were eliminated (n = 536 lending and 597 nonlending projects were removed). Table B.2. Portfolio Inclusion and Exclusion Rules Explained Included Content Excluded Content Mitigation Preparedness Recovery Response Othera Mainstreaming DRR Disaster risk and Resilient Postdisaster Projects with Contingency Emergency or DRM and climate emergency postdisaster needs Response Components but without DRR and disaster risk preparedness, reconstructionassessment General urban services (for example, water into strategy, policy, including with DRR (for example, supply and sanitation, water pollution, and planning emergency postdamage Postdisaster wastewater treatment, governance, (including land-use management and needs and recovery with municipal finance) planning) planning (for loss DRR example, assessment) Water resource management or natural Disaster-resilient communications, resource management without DRR infrastructure (for Disaster shelters, hospital example, roads, reconstruction Infrastructure without DRR preparedness, ports, airports, without DRR health shocks) Risk mitigation for non-disaster-related housing, slum or DRM shocks (for example, commodity, supply upgrading, schools, Strengthening Disaster chain) tourism, medical weather and response or facilities, protective climate Food and nutrition security (not disaster- recovery works) information related) without DRR systems, Identification of or DRM Social protection for conflict or other including disaster risk or nondisaster emergencies hydromet Locust and hazard (for example, other pest General CDD without DRR agricultural risk Early-warning control and assessment, systems Public health emergencies (for example, response vulnerability (including ICT or Ebola, COVID-19) assessment, hazard data systems, Disaster- Animal health and disease mapping) community- related food based early- and nutrition General energy security (not disaster- Knowledge and warning systems security related) learning (for example, disaster Financial disaster Spatial and land-use planning without DRR mitigation risk management General PFM without DRR evaluation, (for example, information contingency fund, Non–disaster-related emergency systems systems) disaster (for example, crime, medical, 911) insurance, Global DRR Dam safety catastrophe risk convening and insurance, Water scarcity and security without link to awareness raising sovereign, drought Water resource agricultural risk management, insurance) natural resource Capacity building management, for postdisaster nature-based needs assessment solutions, climate- and disaster relief smart and resilient (recovery phase) agriculture with disaster risk 32 mitigation (for Disaster example, drought, responsive social flood) protection and safety nets Source: Independent Evaluation Group Note: CDD = community-driven development; CERC = Contingency Emergency Response Component; COVID-19 = coronavirus pandemic; DRM = disaster risk management; DRR = disaster risk reduction; ICT = information and communication technology; PFM = public financial management. a. Many projects with DRM theme codes or that were identified by text analysis were found to be false positives. Portfolio Description The identified portfolio includes 556 lending projects (including 92 additional financing), with a total commitment of US$54.7 billion, and of which 480 are investment project financing, 69 are development policy financing, and 5 are Program-for-Results projects. The preliminary review also identified 769 nonlending activities, of which 251 have relevant and available documentation on the World Bank Group’s public documents and reports site.1 While a more thorough search will be conducted in the operations portal as part of the portfolio review and analysis, it should be noted that many DRR nonlending activities in this space include trainings, workshops, and other advisory services that do not have clear output documentation. Almost half of both lending and nonlending activities are mapped to the Urban, Resilience and Land Global Practice (see figures B.1 and B.2). Also, a sizeable number of projects are mapped to the following Global Practices: Water; Transport; and Environment, Natural Resources, and Blue Economy. 33 Figure B.1. Evaluation Lending Portfolio (n = 556) by Global Practice and Theme URL 46%, 255 WAT 14%, 79 Transport 10%, 56 ENB 9%, 52 SPJ 5%, 29 AGR 5%, 27 Global Practice MTI 5%, 27 FCI 1%, 8 EAE 1%, 7 HNP 1%, 5 SSI 1%, 3 GOV 1%, 3 IDD 0%, 2 POV 0%, 1 EDU 0%, 1 Climate Change 0%, 1 0 50 100 150 200 250 300 Projects (no.) Source: Independent Evaluation Group. Note: AGR = Agriculture; EAE = Energy and Extractives; EDU = Education; ENB = Environment, Natural Resources and Blue Economy; FCI = Finance, Competitiveness, and Innovation; GOV = Governance; HNP = Health, Nutrition, and Population; IDD = Digital Development; MTI = Macroeconomics, Trade, and Investment; POV = Poverty; SPJ = Social Protection and Jobs; SSI = Social Sustainability and Inclusion; URL = Urban, Disaster Risk, Resilience, and Land; WAT = Water. 34 Figure B.2. Evaluation Nonlending Portfolio (n = 769) by Global Practice and Theme URL 43%, 334 WAT 10%, 79 Climate Change 9%, 67 FCI 7%, 54 ENB 7%, 51 SPJ 6%, 47 AGR 5%, 38 Other 3%, 25 Global Practice Transport 3%, 21 (blank) 3%, 20 SSI 1%, 8 EAE 1%, 8 GOV 1%, 6 MTI 1%, 6 POV 0%, 3 HNP 0%, 2 0 50 100 150 200 250 300 350 400 Projects (no.) Source: Independent Evaluation Group. Note: AGR = Agriculture; EAE = Energy and Extractives; EDU = Education; ENB = Environment Natural Resources and Blue Economy; FCI = Finance, Competitiveness, and Innovation; GOV = Governance; HNP = Health, Nutrition, and Population; IDD = Digital Development; MTI = Macroeconomics, Trade, and Investment; POV = Poverty; SPJ = Social Protection and Jobs; SSI = Social Sustainability and Inclusion; URL = Urban, Disaster Risk, Resilience, and Land; WAT = Water. Lending has taken place in 107 countries and across all Regions. Almost a third of the lending portfolio (28 percent) is in the East Asia and Pacific Region (see figure B.3.), primarily due to the number of activities conducted in small island states. In many cases, particularly in small states, individual projects are part of larger regional programs with a common design. 35 Figure B.3. Evaluation Lending Portfolio (n = 556) by World Bank Region East Asia and Pacific 28%, 158 Latin America and Caribbean 20%, 112 World Bank Regions Eastern and Southern Africa 17%, 95 South Asia 15%, 84 Western and Central Africa 8%, 46 Europe and Central Asia 8%, 46 Middle East and North Africa 1%, 7 Regional/World 1%, 5 Africa 1%, 3 0 25 50 75 100 125 150 175 Projects (no.) Source: Independent Evaluation Group. Figure B.4. Evaluation Nonlending Portfolio (n = 769) by World Bank Region Regional/World 27%, 204 Latin America and Caribbean 20%, 150 World Bank Regions East Asia and Pacific 16%, 121 Eastern and Southern Africa 9%, 68 South Asia 9%, 66 Europe and Central Asia 8%, 65 Africa 5%, 37 Middle East and North Africa 4%, 31 Western and Central Africa 4%, 27 0 50 100 150 200 250 Projects (no.) Source: Independent Evaluation Group. The number of closed and validated portfolios is small. Of the 556 lending projects, 223 projects are closed, 114 of which have a completion report and 102 of which have been validated by the Independent Evaluation Group (IEG). A large number of projects in the portfolio (103, of which 72 are closed) are recipient-executed trust fund activities, which are not generally validated by IEG if under US$5 million.2 The World Bank uses its lending instruments to reduce disaster risk through a wide range of activities. Key DRR activities include protective works and resilient infrastructure; policy and institutional reform; early-warning systems; emergency planning and management; community-based approaches; and disaster risk finance, such as developing insurance mechanisms or markets that function at sovereign, firm, and household levels. Table 3.1 describes some common activities being implemented by different World Bank Global Practices. 36 International Finance Corporation An approach similar to the one described above was used to identify relevant International Finance Corporation (IFC) investment and advisory activities. A keyword taxonomy was used to screen the IFC advisory database for relevant keywords in project descriptive text using text analysis. As a result of technical consultations, the IFC climate adaptation co-benefits portfolio was also examined. Both the investment and advisory portfolios were manually screened in line with the project scope to separate relevant activities from false positives and miscoded projects. This approach led to the identification of 15 relevant investments and 77 relevant advisory projects, as seen in figure B.1. Partly because many DRR activities are classic public goods or core functions of government—and therefore outside IFC’s scope—the portfolio is modest in size and mainly related to advisory work on disaster and agricultural insurance. Because of the limited portfolio size, it was decided not to include IFC in the evaluation scope. Figure B.5. Evaluation Identification of International Finance Corporation Portfolio Source: Independent Evaluation Group. Note: DRR = disaster risk reduction; FY = fiscal year; IFC = International Finance Corporation. Preliminary Analysis of Results and Outcome Orientation of the Disaster Risk Reduction Portfolio The Approach Paper conducted a preliminary analysis to understand how the World Bank articulates DRR outcomes in its project objectives and theories of change, and how it captures those outcomes with indicators. It did this to scope its questions on effectiveness: What evidence exists and where are the gaps? What is feasible to expect from a desk review of project documents, and where does the evaluation have to look for other sources? 37 For all investment project financing projects in the evaluation portfolio (86 percent of total projects), the evaluation team conducted a preliminary review and analysis of the key performance indicators in the results frameworks to determine the type, frequency, and adequacy of different results being measured. The indicator database was downloaded from the Enterprise Data Catalog and was scanned for DRR-related indicators to better understand how DRR outcomes are measured. The emerging findings included the following: • DRR outcomes are difficult to measure: DRR activities seek to reduce the negative effects (relative to a without-project counterfactual) of a future event that has a probabilistic range of occurrence and severity. • DRR projects and their results frameworks provide little evidence on outcome and impacts. Most projects only include output indicators. Only approximately 20 percent of identified DRR projects included indicators that measured DRR outcomes; most indicators measure outputs. This lack of outcome evidence makes a direct effectiveness assessment difficult. • Water Global Practice projects addressing floods and water management have the highest share of outcome-oriented indicators. For example, reduced flood depth at the monitoring points compared with equivalent flood depth from before the project; reduction in average number of flooded days per flood event. • Projects largely lack disaggregated data on beneficiaries, which hinders questions about “who benefits.” • A range of quality of indicators exist across even the same intervention type, which suggests the potential for improvements in outcome measurements if projects with weaker indicators could learn from projects with stronger indicators. Methodological Limitations Measuring DRR Outcomes DRR outcomes are inherently difficult to measure because they are a reduction in the negative effects of a probabilistic future shock. Avoided losses cannot be directly measured. Reduced expected mortality and damage are a function of both the probability distribution of natural hazards of varying intensities and the effectiveness of risk-reduction activities. The effects of a disaster cannot be measured until an actual hazard strikes, and then measuring the effectiveness of DRR is dependent on a good counterfactual (Maxwell et al. 2009). Potential tools and methods for assessing DRR 38 outcomes include information mapping, universal data sets to monitor trends, case study analysis (including both qualitative and quantitative data), literature review of effective DRR models, building on existing systems, evaluating local coping strategies, and assessing characteristics of a disaster-resilient community (Feinstein International Center 2011; Twigg 2009). Additional challenges include multiple scales of analysis leading to aggregation problems, the absence of objective benchmarks, and dynamic systems that involve different combinations of explanatory variables over time and place (Thomalla et al. 2006). Disaster Data Biases Disaster data can be biased toward disasters that are bigger or that affect richer countries because they are more likely to be reported. Almost all published cross-country studies on disasters have used information about the economic or human damage of disasters from the EM-DAT International Disaster Database, provided by the Centre for Research on the Epidemiology of Disasters at Université catholique de Louvain. This evaluation will also use EM-DAT data to identify the historical frequency and intensity of disasters. EM-DAT documents disasters from 1900 to the present that conform to at least one of the following criteria: (i) 10 or more people dead; (ii) 100 or more people affected; (iii) the declaration of a state of emergency; or (iv) a call for international assistance. Disasters that do not meet these thresholds are not included. The inclusion of disasters is mostly based on insurance claims or news stories but not on primary geophysical or meteorological data, which means that disasters in countries with more developed insurance markets and better media coverage are more likely to be included and are likely to correlate with gross national product per capita. Disaster intensity measures from EM-DAT are also correlated with gross national product per capita, because the monetary damage of a given disaster is higher in a richer economy. These problems would lead to an upward bias in empirical estimates of disasters on growth or per capita income. As reporting has improved in many countries, more disasters have been documented, potentially leading to an overestimate of the degree to which disasters are becoming more frequent. Despite its limitations, the EM-DAT database remains the best available. While alternative high-quality data sets exist, they are either not public (Munich Re), do not cover the evaluation period (GeoMet), or cover fewer countries (DesInventar). Consequently, the EM-DAT database remains the highest-quality data set available. This evaluation will take the possible biases into account, especially when using EM-DAT data to aid case selection. 39 COVID-19 Risks and Travel Restrictions The evaluation design considered and adjusted to the ethical and methodological limitations of the ongoing coronavirus (COVID-19) pandemic. Specifically, the evaluation team used guidance from the IEG Methods Advisory team (Vaessen and Raimondo 2020) to explore challenges and mitigation strategies: Much of the proposed data gathering processes have been designed to be carried out at desk. Care will be taken to avoid putting unnecessary strain on already overstretched public systems (as part of planned consultations or interviews). From an ethical standpoint, the evaluation will carefully consider the risk-reward ratio of evaluation activities. Necessary precautions will be taken to protect staff and respondents in the event that local evaluation teams (for example, local consultants, client country counterparts, nongovernmental organizations, and so on) are engaged. 1 The Documents and Reports site is an official disclosure mechanism for the World Bank Group’s final reports. The repository contains official documents and reports that are made available to the public in accordance with the World Bank’s access to information policy to better share the institution’s knowledge base. The Documents and Reports site contains final and official documents and reports from 1946 through the present, including Board documents (items concerning meetings of the Executive Directors); country focuses (strategic priorities and directions for lending activities); economic and sector work (in-depth background studies); project documents (loan- and credit-related documents released to the public according to the project cycle, including legal agreements); publications and research (formal publications, working papers, and informal series from departments around the Bank Group). 2Where total contributions are greater than or equal to US$5 million for each programmatic trust fund and for each Global and Regional Partnership Program financed by trust fund(s), respectively, the task team leader arranges to have an independent evaluation carried out at least once every five years in accordance with the principles and standards laid out by the Independent Evaluation Group. The Independent Evaluation Group periodically reviews such evaluations and also reviews individual trust funds and related activities as part of its ongoing Implementation Completion and Results Report Reviews. 40 References Feinstein International Center. 2011. Examining Linkages between Disaster Risk Reduction and Livelihoods: Literature Review. Boston: Feinstein International Center, Tufts University. https://fic.tufts.edu/wp-content/uploads/DRR-lit-review.pdf . Maxwell, Daniel, Girum Tadesse, Mary Mukwavi, Shimelis Hailu, Wolde Gebreal Zewold, and Abraha Gebrekiros. 2009. Baseline Report: Africa Community Resilience Project. Tsaeda Amba Woreda, Eastern Tigray, Ethiopia. Boston: Feinstein International Center, Tufts University. https://fic.tufts.edu/publication-item/baseline-report-africa-community-resilience- project/. Thomalla, Frank, Tom Downing, Erika Spanger‐Siegfried, Guoyi Han, and Johan Rockström. 2006. “Reducing Hazard Vulnerability: Towards a Common Approach between Disaster Risk Reduction and Climate Adaptation.” Disasters 30 (1): 39–48. doi:10.1111/j.1467- 9523.2006.00305.x . Twigg, John. 2009. Characteristics of a Disaster-resilient Community: A Guidance Note, version 2. https://discovery.ucl.ac.uk/id/eprint/1346086/1/1346086.pdf . Vaessen, Jos, and Estelle Raimondo. 2020. “Conducting Evaluations in Times of COVID-19 (Coronavirus).” Independent Evaluation Group (blog), April 8, 2020. https://ieg.worldbankgroup.org/blog/conducting-evaluations-times-covid-19- coronavirus. World Bank. 2016. Theme Taxonomy and Definitions. Washington, DC: World Bank. https://thedocs.worldbank.org/en/doc/275841490966525495- 0290022017/original/NewThemeTaxonomyanddefinitionsrevisedJuly012016.pdf. 41