GLOBAL FACILITY FOR DISASTER REDUCTION AND RECOVERY: Synthesis Evaluation Report Final VERSION – September 2016 [B] ICF International GLOBAL FACILITY FOR DISASTER REDUCTION AND RECOVERY: Synthesis Evaluation Report FInal VERSION – September 2016 Prepared for Global Facility for Disaster Reduction and Recovery 1818 H Street NW Washington DC 20433 Prepared by ICF International 9300 Lee Highway Fairfax, VA 22031 ICF International [C] Contents 1. Introduction........................................................................................................ 1 1.1. Purpose and Scope............................................................................ 1 1.2. Approach............................................................................................. 1 1.3. Roadmap for the Report..................................................................... 2 2. Progress toward Results................................................................................... 3 2.1. Outputs................................................................................................ 3 2.2. Intermediate Outcomes...................................................................... 5 2.2.1. Knowledge Deepened....................................................................... 5 2.2.2. Client Capacity Increased.................................................................. 5 2.2.3. Innovative Approaches and Solutions Generated............................ 6 2.2.4. Policy/Strategy Informed..................................................................... 6 2.2.5. Development Financing Informed...................................................... 7 2.3. Outcomes and Impacts...................................................................... 9 2.4. Lessons Learned.............................................................................. 10 3. Monitoring and Evaluating Results................................................................. 13 3.1. Moving Toward a Revised Framework............................................. 13 3.2. Current State of Monitoring and Reporting...................................... 14 4. Recommendations.......................................................................................... 16 Appendix A. Terms of Reference............................................................................... 18 Appendix B. Supporting Analysis.............................................................................. 24 Appendix C. Preliminary Action Plan......................................................................... 28 ICF International [i] Acronyms and Abbreviations Acronym / Abbreviation AAA advisory and analytical assistance Indonesia Ministry of National Development Planning (Badan Perencanaan BAPPENAS Pembangunan Nasional) BIG Indonesia Geospatial Information Agency (Badan Informasi Geospasial) Indonesia National Disaster Management Agency (Badan Nasional Pen- BNPB anggulangan Bencana) CSO civil society organization DRFI disaster risk financing and insurance DRM disaster risk management DVRP Disaster Vulnerability Reduction Program GFDRR Global Facility for Disaster Reduction and Recovery InaSAFE Indonesian Scenario Assessment for Emergency IO intermediate outcome JICA Japan International Cooperation Agency M&E monitoring and evaluation PDNA post-disaster needs assessment PHRD Japan’s Policy and Human Resources Development Indonesia National Program for Community Empowerment (Program Nasi- PNPM onal Pemberdayaan Masyarakat) WINRIP Western Indonesia National Roads Improvement Project All monetary values are in U.S. dollars. [ ii ] ICF International Executive Summary At the request of the Global Facility for Disaster The consolidated evaluation findings and Reduction and Recovery’s (GFDRR) Consultative recommendations are presented below. Group, this report consolidates the findings and recommendations of two recent independent Findings evaluations of GFDRR, with a focus on identifying a potential path forward for improved results Progress toward Results measurement. The consolidation focused on Evidence across from the 2014 and 2015 evaluation synthesizing the common elements across both suggests that GFDRR has successfully delivered evaluations, namely GFDRR’s contribution to results outputs, and that those outputs were reasonable in achievement at the output, intermediate outcome, scope and scale given the size of the grants. Table and outcome and impact levels, including GFDRR’s ES-1 below illustrates outputs achieved in the case informing of larger investments by the World Bank study countries: Bangladesh, the Eastern Caribbean, and other partners, and observations on GFDRR’s Ethiopia, Guatemala, Indonesia, Malawi, Nepal, Sri monitoring and reporting systems. Lanka, and Vietnam. Table ES-1: Summary of Outputs in Case Study Countries Bangladesh Saint Lucia Dominica / Guatemala Indonesia Sri Lanka Ethiopia Vietnam Malawi Nepal Pathways Data sharing platform established       Model or tool developed or improved   Risk assessment conducted        Hazard mapping conducted         Policy products developed        Disaster risk reduction investment/financing studies developed       Building codes developed and/or implemented    Contingency planning or emergency preparedness services informed       Post disaster assessment conducted       Equipment procured and/or installed   The first, conducted by DARA in 2014, focused on Guatemala, Malawi, Nepal, Sri Lanka and Vietnam, while the second, conducted by ICF International in 2015, focused on Bangladesh, 1 Dominica, Ethiopia, Indonesia, and Saint Lucia. Both evaluations evaluated GFDRR activities between 2008 and 2014, and reported on results achieved in individual country case studies; examined the leveraging/influencing impact from GFDRR’s grants; and made recommendations related to improving GFDRR’s monitoring and evaluation (M&E) framework. ICF International [ iii ] Bangladesh Saint Lucia Dominica / Guatemala Indonesia Sri Lanka Ethiopia Vietnam Malawi Nepal Pathways Dialogue on climate change impacts and resilience facilitated   Developed and/or implemented pilot projects      Training sessions developed and/or administered         Study tours, conferences, forums, knowledge exchanges facilitated         Guidelines or training modules on land use planning developed   Monitoring and evaluation system for DRM developed  Public outreach materials developed and disseminated     Both evaluations found evidence that most of the strengthening policy dialogue and supporting policy observable results of GFDRR interventions are in development and implementation, including around the intermediate outcome step of the results chain. disaster risk financing and insurance; and influencing Most GFDRR activities in the ten countries visited are and leveraging significant resources for DRM. making valuable contributions to achieving process- oriented (i.e., intermediate) outcomes. Intermediate GFDRR has leveraged DRM resources primarily outcomes observed include: raising disaster risk through support for the preparation of post-disaster awareness at local and national levels and increasing needs assessments (PDNAs) and technical the availability of disaster risk information; building assistance that informed a World Bank lending capacity of national and local governments, as well as operations, as well as, to a lesser extent, by informing civil society, for disaster risk preparedness, reduction, recipient country government expenditures. Table and response; developing and demonstrating ES-2 summarizes these results for the 10 case study innovative tools and approaches for DRM; countries covered by the 2014 and 2015 evaluations. Table ES-2. Pathways for Informing Development Finance Bangladesh Saint Lucia Guatemala Indonesia Dominica Sri Lanka Ethiopia Vietnam Malawi Nepal Pathways Informing development finance through support for the preparation of post-disaster needs assessments       Informing World Bank operations (including financing from the World Bank and co-financers)           Informing recipient country government expenditures*    *The 2014 evaluation did not specifically identify instances of GFDRR activities informing or influencing recipient country government expenditures. [ iv ] ICF International Overall, GFDRR has been relatively successful in been important to enable high-level engagement and identifying entry points for small grant contributions provide opportunities for GFDRR’s relatively small to demonstrate or advance DRM activities that can grant activities to have a broader impact via World inform larger-scale investment operations. Combined, Bank operations. Other factors for success have been the 2014 and 2015 evaluations identified about $2 GFDRR’s strong partnerships, which have enhanced billion of project operations informed by GFDRR the scope of GFDRR’s potential results, as well as in the ten countries studied, including $1.7 billion strong choices for executing agencies, which has built of World Bank commitments with the remainder in capacity among local actors. In some countries, a co-financing from recipient governments, bilateral programmatic approach or cohesive strategy has also donors, and the Global Environment Facility, among supported results achievement. others. Approximately $800 million is associated with disaster risk and climate resilience projects, with the Challenges to success have included lack of remainder associated with mainstreaming disaster risk readiness or capacity to use some of the technologies considerations into infrastructure investments (e.g., piloted by GFDRR, long development periods for transport, water management, urban development) some technical assistance activities, and the use and poverty reduction programs. These values should of less-effective activities, such as one-time training be interpreted cautiously, however, because the scale events or conference attendance support. The or significance of GFDRR’s contribution to individual observation of these particular challenges suggests operations varies significantly. that a long-term approach is especially needed to solidify results for certain activity types, such as the Limited evidence was found of outcomes and impacts introduction of new technologies and support for achieved at-scale, although some activities show disaster risk financing and insurance. In addition, in strong potential. In all countries studied, the 2014 and Bangladesh, the evaluation observed that GFDRR 2015 evaluations found that sustained engagement is utilized a co-financing modality ineffectively, lacking needed to improve the likelihood that some activities’ strategic dialogue during the creation of that intermediate outcomes will proceed toward outcomes arrangement and engagement during implementation. and impacts. Monitoring and Evaluating Results Linking GFDRR small grants with larger World Bank investment operations or broader government GFDRR lacks a systematic process to monitor and initiatives may reinforce potential for downstream report results beyond the output level. Measuring results, but GFDRR’s contribution to those operations’ and evaluating the results of technical assistance outcomes and impacts is difficult to discern. Many programs—especially those focused on resilience—is World Bank investment operations for which GFDRR difficult. Moreover, monitoring and evaluation is in the has contributed to the incorporation or improvement early stages of implementation for climate and disaster of DRM components will achieve sizeable outcomes, resilience programs. Many global programs hosted if successfully implemented. However, these impacts by the World Bank also lack evidence of results cannot be directly attributed to GFDRR, and even beyond the output level. GFDRR’s challenge is further GFDRR’s relative contribution to these impacts is compounded by the broadening scope of its work difficult to establish. Current monitoring and reporting plan, with focus on a variety of issues from resilient systems are not designed to differentiate the impacts cities, to infrastructure, to gender, to climate change, of such contributions to larger investments—not only and the way that its technical assistance often informs for GFDRR, but for other comparator funds as well. broader investment operations. Both the 2014 and 2015 evaluations identified the A key shortcoming identified by the 2014 and 2015 important role of an in-country focal point as a driver evaluations is that there is too much “distance” of deeper engagement and conditions for results. between GFDRR’s output and outcome indicators, GFDRR’s partnership with the World Bank has also meaning that GFDRR’s results beyond the output level These amounts do not include the very large $2.6 billion Productive Safety New Program IV (PSNP IV) in Ethiopia, of which a relatively small proportion ($32) million is related to disaster 2 risk management. ICF International [v] are not being adequately captured. Intermediate (i.e., 2014 and 2015 evaluations found that DRM focal shorter-term) and longer-term outcomes have not yet points have helped ensure that activities maintain been conclusively defined for GFDRR, nor have they momentum and advance toward outcomes at- been clearly integrated into the program’s theory of scale. These staff have also been instrumental in change. A secondary issue is that some of the GFDRR mainstreaming DRM into World Bank operations, “output” indicators—as currently defined in Annex III particularly where such mainstreaming is a stated of the 2016-18 Business Plan—blur the lines between objective of the GFDRR program in that country. outputs and intermediate outcomes. For example, follow-up to ensure that communities of practice, technologies, and other GFDRR- A more robust assessment of GFDRR’s M&E system, supported activities continue to be implemented as well as the development and implementation after individual grants have closed may lead to of a refined M&E plan, could help better articulate better outcomes. GFDRR’s expected process of change, identify a set of suitable and relevant intermediate outcome ƒƒ Consider more focused or cohesive approaches indicators, as well as longer-term outcome indicators, within countries. Both evaluations found that and move the program toward a stronger results GFDRR is producing valuable results, but the size orientation. of GFDRR’s program is small compared to overall country needs. Within each country, focusing in on activities that provide added value and Recommendations build on GFDRR’s strengths could support better To improve future GFDRR results measurement and achievement of sustainable and higher-order achievement, the evaluations made the following outcomes. For example, limited evidence was recommendations. These recommendations are found by both evaluations of sustained results of based on evidence and findings from 2014 and one-time training events or conference attendance 2015; since then, GFDRR has taken decisions that not connected to other, ongoing GFDRR initiatives. affect its strategies and approaches, and that may In contrast, both the 2014 and 2015 evaluations or may not align with these recommendations. The noted that a cohesive strategy has supported recommendations below should be understood in this results achievement in countries where it has been historical context. used. At the country level, grants could be more purposefully designed to build on and reinforce Recommendation #1: Deepen and sustain each other; results are stronger in countries where engagement on the ground. Deeper and more there is a clearer linkage and trajectory among sustained engagement could improve potential for grants. Focusing efforts on building institutional achieving downstream results by addressing several capacity—rather than individual staff capacity— of the challenges identified by both the 2014 and 2015 may also be an effective strategy. evaluations, including limited readiness or capacity to use some of the technologies piloted by GFDRR, long Recommendation #2. Strengthen GFDRR monitoring and evaluation of results beyond development periods for certain types of interventions, high government turnover, and occasional lack of the output level. GFDRR needs a sound follow-up by GFDRR. Improved engagement could be methodology that clarifies the theory of change with a fostered as follows: straightforward results framework, identifies a limited number of meaningful and measurable indicators ƒƒ Prioritize interventions that link to broader at the outcome and impact level, and explains the initiatives and make use of GFDRR’s well- role of evaluation in answering questions that are of recognized technical expertise. Country studies interest to GFDRR stakeholders, and particularly the suggested that activities that are linked to World Consultative Group. Tying these elements together in a Bank, government, and other donor initiatives coherent M&E system would enable GFDRR to better and programs are more likely to have strong communicate how the program is delivering results stakeholder support, show better potential for through its engagement in countries. contributing to results at-scale, and achieve leverage or influence. Similarly, interventions that Some of these improvements could include: make use of GFDRR’s comparative advantages in ƒƒ Refine the existing program results framework the DRR community, including technical expertise to “close the gap” between current GFDRR and regional thematic initiatives, also show strong outputs and outcomes. Long time horizons promise for achieving results. for achieving disaster resilience outcomes ƒƒ Support and coordinate through DRM focal points. The and non-linearity makes monitoring long-term [ vi ] ICF International outcomes very challenging. Instead, monitoring at the intermediate outcome level (e.g., two-to- five years) is being increasingly recognized as a viable approach for resilience programs. The 2014 and 2015 evaluations also showed that this is the level of results where GFDRR’s contributions are most evident, given GFDRR’s valuable role as a facilitator/catalyzer of progress in DRM performance at the country level. Including and monitoring intermediate outcomes in the M&E framework would more accurately hold GFDRR accountable for its own performance. ƒƒ Make further refinements to the M&E framework to enable more robust M&E. These refinements could include defining a limited number of SMART (Specific, Measureable, Achievable, Relevant, and Time-bound) intermediate outcome indicators that are logically tied to the results framework; developing indicator definitions and measurement protocols, and identifying data sources to improve the reliability and validity of reporting; better aligning the M&E framework with the Sendai Framework indicators (under development) to improve relevance; developing approaches and tools to measure gender outcomes within GFDRR activities; and defining where and how evaluation can help address the questions and needs of GFDRR constituents. ƒƒ Address operational aspects of monitoring and reporting. Making improvements to GFDRR’s M&E system as described above may also require some operational changes to improve efficiency and ease of implementation. Systems or tools might be developed or improved, roles and responsibilities might be clarified, and different staff or resources might be required. In addition, clearly stating how monitoring information will be used is important— for example, how the information could be integrated into the Annual Report, and what other reports and learning products might be produced. ICF International [ vii ] [ viii ] ICF International 1. Introduction 1.1 Purpose and Scope Broadly speaking, the 2014 and 2015 evaluations followed similar methodologies: using primarily In 2014 and 2015, GFDRR underwent two qualitative approaches, drawing on evidence retrospective, independent, country evaluations. from desk review and key informant interviews, The first, conducted by DARA in 2014, focused on and using triangulation and other data analysis Guatemala, Malawi, Nepal, Sri Lanka and Vietnam, methods to identify evidence-based findings while the second, conducted by ICF International and recommendations. However, because they in 2015, focused on Bangladesh, Dominica, were conducted independently, each evaluation Ethiopia, Indonesia, and Saint Lucia. Both had differences in their approach and different evaluations evaluated GFDRR activities between Terms of Reference. As a result, to consolidate the 2008 and 2014, and reported on results achieved findings and recommendations, it was necessary in individual country case studies; examined to address some gaps and differences across the the leveraging/influencing impact from GFDRR’s two evaluations. grants; and made recommendations related to improving GFDRR’s monitoring and evaluation In particular, differences in how results were (M&E) framework. reported required some interpretive mapping. The 2015 ICF evaluation identified results at At its fall 2015 meeting, the GFDRR Consultative specific levels of the results chain (i.e., outputs, Group decided to have a report prepared that intermediate outcomes, outcomes, and impacts), consolidates the findings and recommendations while the 2014 DARA evaluation reported on of the previous two evaluations, leading “achievements” and “contributions to DRM toward a longer-term effort to strengthen the outcomes.” To assimilate these differences, ICF measurement of results. The Consultative Group reviewed the country-level findings from the 2014 also emphasized the importance of bridging the evaluation, mapped those findings along the gap between outputs and outcomes to better results chain (outputs, intermediate outcomes, report on how GFDRR’s activities contribute to the outcomes, and impacts), and consolidated them achievement of program results through in-country with the findings from ICF’s 2015 evaluation. The engagements. This report directly responds to the categories of outputs are those reported in recent Consultative Group’s request for a consolidated GFDRR annual reports, with some additional evaluation report and represents a first step at categories identified by the ICF team based on identifying the path forward to improved results the review of activities and GFDRR work products. measurement. The intermediate outcomes and outcomes are those included in the results framework in GFDRR’s 1.2 Approach M&E Framework Update, presented at the 15th This report consolidates the findings and Consultative Group Meeting. The interpretation of recommendations of the 2014 and 2015 country the indicators and the subsequent mapping is that evaluations. In doing so, the consolidation focuses of the ICF evaluation team. on synthesizing the common elements across both It is important to note that findings were evaluations, including: synthesized across the evaluations, but each ƒƒ G  FDRR’s contribution to results achievement at evaluation’s assessment of results achievement the output, intermediate outcome, outcome, and was not updated or validated. In other words, impact levels. this consolidation effort did not seek to determine whether outputs or intermediate outcomes  ontributing and detracting factors for ƒƒ C identified in the 2014 evaluation have since achieving success; progressed to the outcome or impact level. To  FDRR’s informing of larger investments by the ƒƒ G support the discussion around the intermediate World Bank and other partners; and outcome indicator “development financing ƒƒ  GFDRR’s monitoring and reporting systems. informed,” the ICF team did review supplemental materials (primarily World Bank project appraisal ICF International [1] documents, as well as interim deliverables from DARA, including a country report for Vietnam), to clarify the extent and nature of GFDRR’s contributions to World Bank operations. 1.3 Roadmap for the Evaluation The remainder of the evaluation report is divided into three main chapters: ƒƒ Progress toward results and lessons learned. ƒƒ An assessment of GFDRR’s monitoring and reporting systems. ƒƒ Recommendations to improve future GFDRR results achievement, measurement, and reporting. [2] ICF International 2. Progress toward Results In the GFDRR theory of change, results are conditions, those outputs contribute to outcomes, envisioned along a chain: GFDRR’s activities which further contribute to the achievement of (inputs) produce outputs; and under the right impacts. Figure 1 below illustrates this results chain. Figure 1: Overview of the GFDRR Results Chain Inputs/Activites Outputs Outcomes Goal/Impact Action by GFDRR Immediate result Mid-term result - Long term result (or grant recipient) of GFDRR action action taken by of collective action governments and Technical or DRM systems others Societal and advisory products capabilities economic change: and services: established: DRM systems lives and livelihoods performed by core new/improved capabilities in use: better protected, specialist teams or knowledge, behavior, systems, and losses ‘outsourced’ to capacity, or other or institutional mitigated grant recipients enabling factor change in received by country performance partners *Source: GFDRR. Implementation of GFDRR Monitoring and Evaluation Framework. To provide context for the analysis that follows, employed in the 2015 evaluation, using the World each level of the chain is briefly described, below. Bank’s intermediate outcomes for analytical and A further assessment of GFDRR’s monitoring and advisory assistance (AAA). The analysis in Section evaluation system is provided in Section 3. 2.2 consolidates the findings of the 2014 and 2015 evaluations using these AAA intermediate ƒƒ At the output level, GFDRR has developed a set outcome indicators. of common indicators that were employed for reporting results in the past two Annual Reports. ƒƒ Evaluation findings on outcomes and impact are These indicators inform the analysis of outputs provided in Section 2.3. in Section 2.1, although some adjustments have been made to account for other outputs observed 2.1 Outputs by the 2014 and 2015 evaluations. Evidence across from the 2014 and 2015 evaluation suggests that GFDRR has successfully delivered ƒƒ GFDRR’s current M&E framework (e.g., as outputs, and that those outputs were reasonable in presented in the GFDRR Work Plan 2015-17) scope and scale given the size of the grants. Limited provides outcome indicators (e.g., improved instances of non-completion of expected outputs identification and understanding of disaster risk, were observed in Bangladesh and Ethiopia, and avoided creation of new risk and reduced existing for a few grants in the Eastern Caribbean, evidence risks in society). The 2014 evaluation found that was not available to confirm delivery of some these outcomes did not adequately capture outputs. Table 1 below illustrates outputs achieved GFDRR’s facilitation role and suggested the use in Bangladesh, the Eastern Caribbean, Ethiopia, of shorter-term, intermediate outcome indicators, Guatemala, Indonesia, Malawi, Nepal, Sri Lanka, and which would be placed between outputs and Vietnam. outcomes in the results chain. This approach was ICF International [3] Both evaluations noted challenges in reviewing objectives and design during implementation; both GFDRR delivery against plan, however. In particular, of these documentation issues made it difficult for the many GFDRR grant proposals do not describe evaluators to identify and compare results to planned planned outputs, and in some cases, significant achievements. adjustments were made to the original program Table 1: Summary of Outputs in Case Study Countries Bangladesh Saint Lucia Dominica / Guatemala Indonesia Sri Lanka Ethiopia Vietnam Malawi Nepal Pathways Data sharing platform established       Model or tool developed or improved   Risk assessment conducted        Hazard mapping conducted         Policy products developed        Disaster risk reduction investment/financing studies developed       Building codes developed and/or implemented    Contingency planning or emergency preparedness services informed       Post disaster assessment conducted       Equipment procured and/or installed   Dialogue on climate change impacts and resilience facilitated   Developed and/or implemented pilot projects      Training sessions developed and/or administered         Study tours, conferences, forums, knowledge exchanges facilitated         Guidelines or training modules on land use planning developed   Monitoring and evaluation system for DRM developed  Public outreach materials developed and disseminated     [4] ICF International 2.2 Intermediate Outcomes quantified financial requirements for DRM. Some examples of improved availability of disaster risk Both evaluations found evidence that most of the information include: observable results of GFDRR interventions are in the intermediate outcome step of the results chain. The ƒƒ In Malawi, information and data sharing has been qualitative research conducted for the evaluations initiated in conjunction with the development of the (including interviews with project proponents and Malawi Spatial Data Portal (MASDAP); although beneficiaries) identified process-based results information sharing has been low so far, experts that could be mapped to the five intermediate from different sectors are actively participating in the outcome indicators: knowledge deepened; client upgrading process. capacity increased; innovative approaches and solutions generated; policy/strategy informed; and ƒƒ In the Eastern Caribbean, GFDRR has supported development financing informed. These results are the development of GeoNodes and socialized the discussed in more detail below. tool to garner national-level support for data sharing. ƒƒ In Sri Lanka, the GeoNodes are expected to 2.2.1 Knowledge Deepened facilitate sharing of disaster risk information across government departments, through a disaster data A commonly observed intermediate outcome of working group. Also in Sri Lanka, hydrological and GFDRR activities in the ten case study countries hydraulic modeling for Colombo provided technical was increased awareness of DRM at local and information that informed the prioritization of national levels. In Bangladesh, GFDRR contributed improvement works on the city’s canal system. to increased understanding and awareness of earthquake risk among key stakeholders in ƒƒ In Nepal, a national hazard risk assessment Dhaka. In Guatemala, increased understanding improved knowledge of the national risk profile and and acknowledgement of DRM was observed in a GeoNode platform was developed to improve the national government (including the National sharing of this risk information. GFDRR’s Open Coordinator for Disaster Reduction [CONRED], Data for Resilience Initiative mapped two wards Secretariat of Planning and Programming in Kathmandu, creating the basis for exposure [SEGEPLAN], Ministry of Education, and Ministry of mapping of schools and health facilities and Agriculture), as well as among local authorities in enabling informed decision-making in resource municipalities in which GFDRR financed mapping of allocation. hydrometereological hazards. In Ethiopia, awareness ƒƒ In Indonesia, disaster risk information has been was raised at the woreda (or district) level, through made increasingly available through participatory pilot programs. In Indonesia, awareness was raised mapping, InaSAFE, a national risk assessment study, in urban communities through facilitator training on and rapid diagnostics. DRR, and through safe school pilots. In Malawi, the 2014 evaluation observed increased understanding ƒƒ In Ethiopia, development of the woreda disaster and appreciation of DRM among policy officials and risk profiles made information available, and technical personnel in the disaster management Woreda-net, a digital interactive database of all agency, as well as in the Ministry of Housing and related information, improved information exchange Ministry of Agriculture. In Vietnam, DRM awareness between woreda-level government and the regional was raised in the Ministry of Transport, in the context and national levels. of highway construction. 2.2.2 Client Capacity Increased GFDRR has also contributed to increased availability GFDRR has contributed toward building capacity of disaster risk information, broader support for open of national and local governments, as well as civil data, and more informed decision-making (more society, for disaster risk preparedness, reduction, and efficient use of resources). Common GFDRR activities response. Compared to deepening knowledge, fewer leading to these intermediate outcomes included the instances of strengthened government or institutional development of open source disaster risk information capacity to manage disaster risk were observed by platforms (often powered by GeoNode), pilot the 2014 and 2015 evaluations. These include: mapping of urban and peri-urban neighborhoods, and conducting hazard, exposure, and risk ƒƒ In Bangladesh, GFDRR has begun to strengthen assessments. Post-disaster needs assessments the emergency preparedness and response (PDNAs) in Guatemala, Indonesia, Malawi, Saint capacity of Dhaka government authorities—an Lucia, Sri Lanka, and Bangladesh also contributed activity that will be further taken up by a World to greater availability of information about needs and Bank operation. ICF International [5] ƒƒ In Vietnam, a study and trainings on climate Bantul; according to interviews with the Ministry of proofing of rural roads increased the capacity Public Works, this was the first time in Indonesia of the Ministry of Transport, and guidelines that a community-based risk assessment had been were incorporated into the World Bank Third conducted and that people had been relocated Rural Transport project. GFDRR also helped the based on the mapping. In addition, the DRM national Department of Dyke Management and Flood knowledge hub that GFDRR is developing with the Control to develop emergency preparedness World Bank’s Leadership, Learning and Innovation plans for 14 central provinces, as well as the Group offers an innovative strategy for addressing development of a model provincial disaster Indonesia’s challenges in training all 340 of its local management center in Quang Tri province. disaster management agencies. ƒƒ In Indonesia, national capacity for independently Other intermediate outcomes observed related to conducting damage and loss assessments has innovative tools and approaches for DRM include increased through GFDRR support, and GFDRR GeoNodes in four of the ten countries, improvements also played a role in operationalizing the newly to the Livelihood Early Assessment and Protection formed national disaster management agency. In (LEAP) model in Ethiopia, and the creation of Indonesia, GFDRR has also increased the capacity the GEODASH platform with data for Dhaka, in of civil society to contribute to DRM through connection with the Bangladesh Urban Resilience training and pilot programs. Project. In all of the case study countries, GFDRR has ƒƒ In Nepal, a training on risk assessment pursued open source data or platforms to engage methodologies and PDNA led to strengthened civil society and government in issues relating to government capacities on these topics, although DRM; for example, the mapping and collecting of the evaluation noted that a lack of strategic seismic risk exposure data for public and private direction diluted the impact of this engagement. schools including public buildings in the Kathmandu ƒƒ In Ethiopia, GFDRR’s capacity building has Valley. focused at the local (woreda) level, for disaster risk identification, reduction, and preparedness. 2.2.4 Policy/Strategy Informed ƒƒ In the Eastern Caribbean, national government GFDRR has strengthened policy dialogue and capacities have been strengthened, but there supported policy development and implementation, is a risk of capacity loss unless follow-on including around disaster risk financing and insurance support is provided. In Dominica, the national (DRFI). Varying levels of support have been provided government showed some increased capacity to the ten case study countries through GFDRR’s around geospatial data and shelter vulnerability flagship DRFI program. In Guatemala, significant assessments. technical assistance led to increased awareness of the Ministry of Agriculture around risk transfer 2.2.3 Innovative Approaches and Solutions mechanisms for food security and sustainability of Generated livelihoods, and a larger engagement on the revision GFDRR has contributed to developing and of policy frameworks to support the development of demonstrating innovative tools and approaches an agriculture insurance market. In Nepal, GFDRR’s for DRM.3 Numerous examples were observed in agricultural insurance feasibility study was widely Indonesia. For example, the Indonesian Scenario discussed and helped the Government of Nepal Assessment for Emergency (InaSAFE) tool offers an recognize the institutional challenges that need to be opportunity to use the collected mapping data to addressed; and although the Government included support local-level contingency planning, which is a agricultural insurance as part of the 2009/2010 required activity for local disaster risk management budget policy framework, the 2014 evaluation found agencies under national regulation. InaSAFE was that a lack of GFDRR follow up funding meant that no awarded by Wired, an American magazine that expected outcomes were achieved. In Bangladesh, reports on emerging technologies, as one of the top Indonesia, and Vietnam, GFDRR has provided 10 “open-source rookies of the year” in 2013. There is analytical products and dialogued with ministries of interest also outside of Indonesia to adapt and use the finance on DRFI, to begin to socialize the concepts underlying software. Also in Indonesia, GFDRR piloted in what is widely recognized as a long development an approach for assessing and communicating process. landslide hazard risks in the peri-urban area of The 2014 DARA evaluation did not specifically identify GFDRR approaches or solutions as innovative. 3 [6] ICF International GFDRR has directly supported the development dialogue at the national level, GFDRR also contributed and implementation of DRM policy. In Ethiopia, the to integrating DRR into Indonesia’s National Medium- partnership between the World Bank and GFDRR Term Development Plan for 2010–2014. In Sri Lanka, has helped facilitate a transition in the policy dialogue GFDRR’s technical assessment on social protection and programmatic priorities toward risk reduction programs and disasters helped the Department of and preparedness. This is demonstrated by the shift National Planning draft a Social Protection Strategy in mandate of the Disaster Risk Management and and Natural Disaster Guidelines for Safety.4 In Food Security Sector, the National Policy and Strategy Dominica, GFDRR supported the development of on Disaster Risk Management, and DRM Strategic policy around information sharing. Programme and Investment Framework. GFDRR activities have supported this shift, including through 2.2.5 Development Financing Informed the provision of advisory services on the development GFDRR grants represent a very small portion of the of the national DRM policy. In Guatemala, GFDRR investments needed to reduce disaster risk in the ten supported developments in the DRM regulatory countries studied. Strategic application of GFDRR’s framework after the 2012 earthquake. grants, however, have potential to amplify results, GFDRR has also dialogued with line ministries either by directly leveraging larger investments by to support mainstreaming of DRM. In Indonesia, partners or by influencing how existing resources for GFDRR has frequent dialogue with the National resilience are spent. Table 2 shows the most common Disaster Management Agency (BNPB) and Ministry pathways through which GDFRR has been observed of National Development Planning (BAPPENAS), and to inform development finance, including World Bank also supported policy changes with the Geospatial operations, recipient country government budgets, Information Agency (BIG), the Ministry of Public and development partners’ resources. Appendix B Works, and the Ministry of Education and Culture. By presents more detail on the development finance complementing the preparation of the National Action informed through GFDRR activities. Plan for Disaster Risk Reduction with facilitation and Table 2. Pathways for Informing Development Finance Bangladesh Saint Lucia Guatemala Indonesia Dominica Sri Lanka Ethiopia Vietnam Malawi Pathways Nepal Informing development finance through support for the preparation of post-disaster needs assessments       Informing World Bank operations (including financing from the World Bank and co-financers)           Informing recipient country government expenditures*    *The 2014 evaluation did not specifically identify instances of GFDRR activities informing or influencing recipient country government expenditures. PDNAs. These assessments are intended to provide and DRM investments and improve the enabling a coordinated and credible basis for recovery and environment for DRM. Through grants for PDNAs in reconstruction planning, and for the international Bangladesh, Guatemala, Indonesia, Malawi, Saint community to assist the affected country in this Lucia, and Sri Lanka, GFDRR—in partnership with the process, including through providing funding. As World Bank, United Nations agencies, the European such, PDNAs often leverage emergency relief Union and other development partners—has helped At the time of the 2014 evaluation, however, further evidence was not available on the implementation of an action plan for the SPS. 4 ICF International [7] develop recommendations for key actions that are Third Rural Transport Project (the additional financing frequently funded by the World Bank and other component). In Malawi, GFDRR developed an donors. In Bangladesh, more than $1,600 million has Integrated Flood Risk Management Plan for the Shire been committed to World Bank projects based on Basin, which supported improved hydrologic and the PDNA that GFDRR supported after Cyclone Sidr. hydraulic modelling of the flood zones. In Saint Lucia, the Joint Rapid Damage and Needs Assessment, supported by GFDRR after the 2013 In Indonesia, Bangladesh, and Guatemala, GFDRR Christmas Rains, was used to leverage emergency staff also provided technical support on the inclusion response resources ($17 million) from the World of zero-dollar disaster contingency components in Bank’s Crisis Response Window and $10 million in World Bank projects. These components provide reconstruction support from the European Union, to an option for countries to rapidly access funding for support the Disaster Vulnerability Reduction Program emergency response in the event of a natural disaster. (DVRP). GFDRR’s technical expertise has lent Recipient country government expenditures. credibility to these assessments. GFDRR activities have influenced national and local World Bank operations. Overall, GFDRR has been government expenditures for DRM in Bangladesh, relatively successful in identifying entry points Ethiopia, and Indonesia.6 In Vietnam, GFDRR was for small grant contributions to demonstrate or less successful in influencing government spending advance DRM activities that can inform larger- because risk-proofing options identified by GFDRR scale investment operations. Combined, the 2014 studies were seen as too costly and challenging to and 2015 evaluations identified about $2 billion of implement. In Indonesia, certain activities currently project operations informed by GFDRR in the ten ongoing show potential for future influence of national countries studied, including $1.7 billion of World Bank government expenditures, including the DRM commitments with the remainder in co-financing knowledge management hub, which could influence from recipient governments, bilateral donors, and how BNPB allocates its budget to train disaster the Global Environment Facility, among others.5 management government staff around the country, Approximately $800 million is associated with disaster and the safe schools pilot program, which has risk and climate resilience projects, with the remainder potential to influence national education funding to associated with mainstreaming disaster risk improve structural and non-structural resilience. Also considerations into infrastructure investments (e.g., in Indonesia, GFDRR activities have also leveraged transport, water management, urban development) DRM funding on a smaller scale. For instance, the and poverty reduction programs. local government in Bantul spent its own resources to do structural mitigation works in a few villages as These values should be interpreted cautiously, a result of GFDRR’s landslide risk assessment, and however, because the scale or significance of some communities and businesses made in-kind GFDRR’s contribution to individual operations contributions to supplement GFDRR funding for varies significantly. At one end of the spectrum is safe schools and community disaster risk action the example of Bangladesh, where GFDRR actively plans under the National Program for Community leveraged investment through the Urban Resilience Empowerment (PNPM). Project (2015–20, $182 million). More than two years of sustained technical assistance under a $2.8 million Other channels of influence. GFDRR has also GFDRR grant led to the preparation and approval informed development finance through coordination of this large investment in early 2015 ($173 million with other development partners on DRM issues. For in World Bank loans and $9 million in co-financing example, in Bangladesh, GFDRR’s engagements from the Government of Bangladesh). In most facilitated close coordination and strategic cases, however, the scale of GFDRR’s contribution collaboration with the Japanese International has been more circumscribed. In Vietnam, Malawi, Cooperation Agency (JICA) on parallel investments Nepal, and Sri Lanka, GFDRR financed analytical in urban resilience (e.g., the World Bank will finance work that informed the project design process. For the procurement of search and rescue equipment example, in Vietnam, GFDRR funded a study on for Fire Service and Civil Defense, while JICA climate proofing of rural roads, which the Ministry of finances the earthquake retrofitting of fire stations). Transport will use as guidelines in the World Bank’s As another example, in Indonesia, GFDRR helped These amounts do not include the very large $2.6 billion Productive Safety New Program IV (PSNP IV) in Ethiopia, of which a relatively small proportion ($32) million is related to disaster 5 risk management. Not including co-financing for World Bank operations as described above. 6 [8] ICF International develop InaSAFE in close technical and financial intermediate outcomes will proceed toward outcomes cooperation with the Australia-Indonesia Facility for and impacts. In particular, long gestation periods Disaster Reduction. In Nepal, GFDRR conducted and continued GFDRR support will be needed to a risk assessment in two districts in Kathmandu realize outcomes for DFRI activities and technology- Valley, trained teachers and students in earthquake oriented solutions. These findings also suggest preparedness, and trained masons in seismic that the timing of the evaluations may be too early resistance construction techniques; an accompanying to see the expected outcomes. In some cases, grant from Japan’s Policy and Human Resources outcomes and impacts may not be achieved due Development (PHRD) fund provides for structural to significant obstacles encountered in the country, assessment, retrofitting of up to seven buildings, deficiencies in GFDRR’s support, or other factors. contingency planning, and awareness building. In Sri The 2014 evaluation identified a lack of follow-up Lanka, GFDRR support for developing a hydrological and a lack of enabling conditions to support the model for Metro Colombo also helped inform a GFDRR intervention as contributing factors to non- grant from PHRD to carry out a detailed flood risk achievement, as discussed further in Section 2.4 on assessment for the region. lessons learned, below. Broadly speaking, it is difficult to systematically 2.3 Outcomes and Impacts assess the extent to which the intermediate outcomes Limited evidence was found of outcomes and impacts observed will contribute to longer-term outcome achieved at-scale at the time of the 2014 and 2015 achievement, in part because the intermediate evaluations, although some activities show strong outcome indicators are not linked to GFDRR’s theory potential. Figure 1 describes some of the outcomes of change. For example, the theory of change does observed in the case study countries. not surmise about the extent or conditions under which deepening knowledge will actually lead to In all countries studied, the 2014 and 2015 outcomes. These issues are further taken up in evaluations found that sustained engagement is Section 3 on monitoring and evaluating results. needed to improve the likelihood that some activities’ Figure 2: Examples of Outcomes Observed ƒƒ In Vietnam, the establishment of a Provincial Disaster Management Centre in Quang Tri, for example, led to observable improvements in relation to a recent storm. ƒƒ In Indonesia, Bantul (Yogyakarta), where GFDRR funded an innovative community-based assessment for landslide risk, nearly 90 households have been relocated to safer ground, and the local government has also conducted structural mitigation works based on the assessment. In many of the 180 schools participating in GFDRR’s safe schools pilot, structural improvements have been financed through Indonesia’s education Special Allocation Fund (DAK), to better protect against earthquakes and other natural disasters. Linking GFDRR small grants with larger World Bank ƒƒ Saint Lucia and Dominica’s DVRPs—which GFDRR investment operations or broader government helped shape—are expected to benefit more than initiatives reinforces potential for downstream results 240,000 people combined. and sustainability. Many World Bank investment ƒƒ In Indonesia, the Western Indonesia National operations for which GFDRR has contributed to the Roads Improvement Project (WINRIP) will incorporation or improvement of DRM components improve road sections traversing 12 districts will achieve sizeable outcomes, if successfully with a total population of over 4 million, and implemented. For example: GFDRR’s assistance means the project should ƒƒ Building on GFDRR’s critical groundwork, the $182 now strengthen disaster risk mitigation in the road million Urban Resilience Project in Bangladesh has sector. potential to increase resilience to earthquakes for ƒƒ In Guatemala, the Disaster Risk Management the 15.5 million people living in Greater Dhaka and Development Policy Loan, building on Sylhet. GFDRR’s support at the municipal level, led to Guatemala City, Quetzaltenango, and Antigua ICF International [9] incorporating DRM in land use and territorial Table 3. Factors Supporting Success planning—representing 18% of the country’s urban population. In addition, the seismic safety Strengths and Factors for Success index that GFDRR produced with the Ministry of Education was applied in 43 schools and ƒƒ In-country presence of focal points. 4 hospitals. And, new seismic standards for designing and construction of public buildings ƒƒ Partnership with the World Bank. were adopted in two cities. ƒƒ Strong partnerships. ƒƒ In Ethiopia, expected benefits associated with reductions in drought and flood impacts and ƒƒ Technical expertise and regional thematic losses and long-term risk reduction efforts under programs. PSNP-IV are valued at roughly $300 million per ƒƒ Tailoring engagement strategies to country year. conditions. ƒƒ In Sri Lanka, the Metro Colombo Urban ƒƒ Programmatic approach / focused or cohesive Development Project is expected to directly strategy. mitigate the effects of floods on the lives of approximately 232,000 people in the Colombo ƒƒ Strong choices for executing agencies at the local Water Basin. level. ƒƒ In Malawi, the Shire River Basin Management Program is expected to result in 250,000 people with access to improve flood management by year Both the 2014 and 2015 evaluations identified the 15. important role of an in-country focal point as a driver of deeper engagement and conditions for results. It should be noted that these impacts cannot be GFDRR and DRM focal points have contributed to directly attributed to GFDRR, and even GFDRR’s building and maintaining good working relationships relative contribution to these impacts is difficult with key government partners, providing continuity to establish. As an example, GFDRR’s support and “keeping alive” policy discussions that have long in Indonesia for the WINRIP project has led to gestation periods (e.g., disaster risk financing and the incorporation of a component that provides insurance), supporting programmatic approaches technical assistance and capacity building support to GFDRR’s grant-making, identifying strategic entry to strengthen disaster risk mitigation in the roads points to inform development finance (most notably, sector, and the Ministry of Public Works has now World Bank operations). Supporting upstream policy funded a study with its own resources to do a changes and building capacity for disaster risk stocktaking of road segments prone to disaster, reduction and resilient recovery is often long-term based on maintenance records and hazard maps. work. Because much of this work requires ongoing Whether these components will lead to actions with interfacing with government, GFDRR and DRM implications for the affected population was yet to be focal points represent a critical element for progress seen at the time of the evaluation. Current monitoring toward impact. In Guatemala, Malawi, and Sri Lanka, and reporting systems do not have the capability to a GFDRR focal point was appointed during the 2014 differentiate the impacts of such contributions to larger evaluation’s time-scope, and significant improvements investments—not only for GFDRR, but for other trust in fostering DRM planning and action were observed funds as well, as discussed in Section 3 below. as a result of those appointments. 2.4 Lessons Learned GFDRR’s partnership with the World Bank has been important to enable high-level engagement and To better understand how and why GFDRR has, or provide opportunities for GFDRR’s relatively small has not, accomplished its goals, a cross-country grant activities to have a broader impact via World analysis was conducted of factors supporting and Bank operations. The World Bank’s access and detracting from success, as reported in the 2014 and convening power has helped GFDRR engage at 2015 evaluations. The factors described below are high levels of government, including ministries of those that were observed in multiple countries and in finance and planning as well as line ministries with both evaluations. responsibilities for DRM, which increases potential for achieving upstream results and results at-scale. Proximity of GFDRR to World Bank operations staff has also maximized the opportunity to influence [ 10 ] ICF International development resources (notably World Bank in responding to country needs and demands, projects). In Indonesia and Vietnam—where the World evidence from fieldwork also suggests that GFDRR Bank has a larger operational portfolio—the presence has worked most effectively when its support has of a GFDRR focal point facilitated connections with been part of a broader country program approach or World Bank operations staff, enabling GFDRR’s focused strategy. For example, in Indonesia, where influence of at least six projects during the evaluation programmatic grants had an express purpose of periods. In Bangladesh and Ethiopia, proximity is mainstreaming DRR into World Bank investments, taken one step further. The same World Bank staff the 2015 evaluation found more instances of that person serves as the task team leader for a GFDRR outcome being achieved (including across sectors). grant and the World Bank investment operation that Similar intentions and results were observed in the GFDRR grant informed. In Bangladesh, this Vietnam by the 2014 evaluation. Also, in Indonesia, tautology helped project leaders to think strategically where programmatic grants had a stated objective about how technical assistance could be linked to to mainstream DRR into development, evidence was investments. found of GFDRR contributions to this effect: at the national level through national development plans GFDRR’s strong partnerships have enhanced and government education budgets, and at the the scope of potential results to which GFDRR is local level through community-driven development contributing. From a theory of change perspective, planning. In contrast, in Nepal, the 2014 evaluation acting in coordination with other key development found that a lack of focus and strategic direction in the partners contributes to an improved enabling interventions promoted by GFDRR had hindered the environment and facilitates the effectiveness of program’s ability to effect national change. GFDRR’s DRM activities as well as the activities of other partners. In Sri Lanka, for example, GFDRR’s Finally, strong choices for executing agencies have relationship with UN agencies (such as UNDP and contributed to building the capacity of local actors UN-Habitat) has facilitated complementarities with and ultimately results achievement. For example, in DRM processes at the local government level, while Indonesia, civil society organizations (CSOs) and GFDRR focused on information technologies and local universities have been used to execute many strengthening DRM capacities at the national level. GFDRR grants because these organizations are able Other key partners have included the World Bank, to gain community trust and engagement, which have UNDP, and the European Commission in preparation been precursors for pilot-level success. Similarly, in of PDNAs, Australia-Indonesia Facility for Disaster Nepal, engagement with CSOs facilitated GFDRR’s Reduction in Indonesia, JICA in Bangladesh, and work; partnerships with civil society in Kathmandu, Japan PHRD in Nepal and Vietnam. in particular, were fundamental to the mapping of seismic risk in relation to public buildings. In Malawi, Another contributor to success has been GFDRR’s collaboration with local consultants and national use of engagement strategies that reflect individual universities (mostly for post-disaster assessments and country conditions. For example, GFDRR has taken a seismic risk identification) increased local capacities proof-of-concept and community-driven development and buy-in. approach in Indonesia, where DRM responsibilities and budgets are decentralized. GFDRR used Table 4. Weaknesses and Factors Detracting from participatory technical assistance in Dhaka Success (Bangladesh), where local government structures and dynamics are very complex and require long-term Weaknesses and Challenges to Success relationship building. In Ethiopia, GFDRR successfully used the evolving social protection agenda as an ƒƒ Lack of readiness or capacity to use technologies entry-point to advance the DRM agenda. Conversely, piloted. in Vietnam, interviews with government institutions suggested that risk-proofing options and technical ƒƒ Long development periods / lack of follow-up. proposals identified by GFDRR studies would have ƒƒ Staff turnover/rotations and competing demands for led to further improvements in risk reduction when staff time. better tailored to the country context, that is, when less costly and more attuned to political preferences. ƒƒ Use of less-effective activities like one-time training events or conference attendance support. A programmatic approach or cohesive strategy has also supported results achievement. While ƒƒ Ineffective use of funding modalities (co-financing, one of GFDRR’s strengths is its flexibility and agility procurement of equipment). ICF International [ 11 ] GFDRR often operates in a country context also limited the usefulness or impact of its analytical in which there are weak or insufficient legal or and capacity building work. For example, in Nepal, regulatory frameworks for DRM, lack of law or code GFDRR prepared a national hazard risk assessment enforcement, insufficient or unpredictable budgets with potential for influencing planning processes, but for DRM, and weak institutional capacity. Much of did not systematically follow up on its application. In GFDRR’s work aims at improving these enabling Sri Lanka, a lack of follow-up from GFDRR on further conditions, including through ministerial dialogue implementation of the PDNA methodology meant and analytical support for new policy development, that the methodology was an ad-hoc program for a capacity building for staff and institutions with specific event (2010 floods), and was not used in the DRM responsibilities, and mainstreaming DRM into 2011 floods. broader World Bank operations to reach scale. Thus, the evaluation focused on challenges to GFDRR’s Rotation of staff and competing demands for staff time success in translating its activities into longer-term have also been challenges to achieving sustainable results within these broader constraints. results through training, capacity building, and some technical assistance activities. This is especially Difficulties were observed in most countries true in the small island Eastern Caribbean context, associated with readiness or capacity to use some where ministries often operate with few staff. High of the technologies piloted by GFDRR. For example, staff turnover in Nepal also had repercussions for in Indonesia, local DRM agencies generally do not the continuity of dialogue with national ministries. In have staff with sufficient GIS programming skills to Malawi, only one person per ministry/department independently use InaSAFE. Geospatial platforms in participated in PDNA training, and representation in the Eastern Caribbean and the Woreda-net systems the national team has not be consistent to effectively in Ethiopia similarly suffer from software, hardware, undertake future PDNAs. and trained user challenges. In Malawi, poor internet connections and the lack of a law on data High government turnover and lack of follow-up are sharing have meant that data sharing was initially also contributing factors to the finding that some low. Governments in many of the countries visited GFDRR activities, such as one-time training events or showed interest in these technology-based tools and conference attendance support, appear less likely to in at least two countries (Indonesia and Ethiopia), achieve long-term results. In general, the 2014 and governments acknowledged the need to invest in 2015 evaluations struggled to find robust evidence human capacity and have started to hire staff with of enduring impacts of these types of activities. In necessary skills. contrast, for example, in Indonesia, GFDRR was able to demonstrate the effectiveness of damage and loss These types of obstacles to introducing new assessment trainings such that GFDRR’s training technologies are recognized in engagement module was eventually institutionalized in the national strategies—for example, the World Bank’s Strategic training center, ensuring its sustainability. These Engagement Framework for the Caribbean findings suggest that focusing efforts on building anticipates issues related to hardware, network, and institutional capacity—rather than individual staff software limitations, as well as information technology capacity—may be a more effective strategy. human support capacity.7 Still, for GFDRR, the observation of these challenges suggests that a long- Lastly, the use of co-financing or direct procurement term approach is needed to institutionalize the use of modalities does not take advantage of GFDRR’s these technologies. comparative advantages, including technical expertise and partnership with the World Bank. In Long development periods and the occasional lack Bangladesh, lack of strategic dialogue during the of follow-up from GFDRR have challenged success. creation of the co-financing arrangement, and a lack The development period for some upstream activities, of engagement with GFDRR during implementation, including on disaster risk financing and insurance, may have contributed to an ineffective use of co- is particularly long and requires ongoing GFDRR financing. In Nepal, GFDRR procured six motorized support. Ensuring strong government support boats for areas affected by Kosi river floods, can help maintain momentum for these longer but lacked in-house experience on this type of engagements; in one country (Bangladesh), an initial procurement and no reporting was required after lack of client demand for DFRI slowed progress. In equipment/service delivery. some countries, a lack of follow up from GFDRR has 7 World Bank. 2012. The Caribbean Region: Strategic Engagement Framework for Disaster Risk Management and Climate Resilience FY13-15. June 2012. [ 12 ] ICF International 3. Monitoring and Evaluating Results 3.1 Moving Toward a Revised Framework viable. DARA recommended that if GFDRR’s M&E framework incorporated “intermediate outcomes” Since 2013, GFDRR has been in the process of with “process-based indicators” to measure the implementing a revised monitoring and evaluation progress made in DRM, then specific contributions (M&E) framework. This new framework was from the different stakeholders could be better endorsed by the Consultative Group in 2013, and captured. defined the GFDRR results chain from outputs to outcomes to impacts. The first phase of this Building on this recommendation, the 2015 external work involved re-categorizing the GFDRR project evaluation was asked to “field-test” potential portfolio against activity types (capacity building, intermediate outcome (IOs) indicators.8 In response, analytical products, and technical assistance) and the ICF evaluation team gathered qualitative data developing activity and output indicators. In 2014, on IOs through desk review and interviews and an external evaluation by DARA was commissioned, mapped those data to the IO indicators, to provide one of the objectives of which was to draw out insights on the relevance and usefulness of the recommendations on the M&E framework and indicators. specifically whether (and how) to adjust indicators for improved program design and evaluation. ICF found that evidence of progress toward DRM results could be mapped against the IO indicators. DARA found that GFDRR succeeds in delivering Indeed, in the country case studies, most of the planned outputs and makes a valuable contribution observed results of GFDRR interventions were in the beyond the output level—but that the M&E IO step of the results chain. However, not all of the framework’s outcome indicators do not adequately IO indicators were directly relevant to the types of capture that contribution. DARA also noted that the process-based results being achieved by GFDRR— GFDRR theory of change was flawed in the sense i.e., the IO indicators could be more precisely that its “assumptions” were actually obstacles worded and tailored to GFDRR’s mission. For observed at the country-level, which GFDRR example, many GFDRR interventions raised disaster was actively trying to address to contribute to an risk awareness among stakeholders; the evaluation enabling environment for DRM improvements. In team mapped these intermediate outcomes to the this way, by assuming that a conducive setting indicator “best practices exchanged with clients,” exists at the country level, the theory of change although this is not a precise articulation of what bears the risk of making GFDRR’s contributions GFDRR actually achieved. less visible to its stakeholders and ultimately less Overview of Challenges for Monitoring and Evaluating Resilience Resilience monitoring and evaluation (M&E) is notoriously challenging due to a number of factors such as complexity, lack of predictability, and long time horizons. Widely recognized challenges for resilience M&E include the following: ƒƒ Given that disasters are unpredictable and their effects highly uncertain, traditional M&E systems that have predictable pre/post “testing points” to understand impact generally do not apply. ƒƒ Resilience itself is non-linear and rapidly changing, influenced by social, economic, climactic, and other factors. This further renders most linear dose-response types of M&E approaches as non-applicable. ƒƒ Resilience is particularly context specific, requiring specific M&E for each context. This in turn prevents higher-level M&E frameworks from being broadly applicable, posing additional challenges to consistency across portfolios and the ability to scale up context-specific indicators and results. These indicators were defined in Annex I to an update on the implementation of the M&E framework, as given at the 15th meeting of the Consultative Group, and are based on the World 8 Bank’s standard indicators for analytical and advisory assistance (AAA) work. ICF International [ 13 ] Overview of Challenges for Monitoring and Evaluating Resilience (continued) ƒƒ There are no widely agreed upon indicators to determine the effectiveness of resilience interventions. Even the most advanced indicators are being refined with field experience. ƒƒ Setting baselines and targets is often complicated due to the rapidly changing and complex nature of vulnerability and uncertainties about the future. ƒƒ Most resilience M&E currently focuses on immediate or near-term outputs, which are more readily identifiable, rather than outcomes and impacts, which can only be realized in the indeterminate future. Thus, for practical reasons, the ultimate impacts of resilience interventions are both unknown and unknowable. Proxies such as interim outcomes indicative of longer-term results may be the best measures available. ƒƒ For all the above reasons, with few exceptions, resilience M&E requires approaches that are typically more systems-oriented, qualitative, mixed method, and innovative. Findings from these most-suitable approaches tend to be more qualitative and nuanced, reflecting evidence of shifts in a system related to resilience rather than a “concrete” result. ƒƒ Related to this, it is often difficult or impossible to attribute a particular change or outcome to a specific intervention. Identifying contribution to a change/outcome is often more realistic, but is not considered satisfactory to some audiences, particularly when a quantitative results finding or “value for money” conclusion is sought. 3.2 Current State of Monitoring and frameworks that include different levels of outcomes Reporting (e.g., intermediate and longer-term outcomes) better illustrate the underlying paths by which the As of the writing of this synthesis report, GFDRR program intends to produce impact, are more lacks a systematic process to monitor and helpful for a managing-for-results approach, and report results beyond the output level. This is a can demonstrate step-wise progress toward longer- common challenge for resilience M&E field-wide, term results. as described in the text box above. In terms of reporting, the 2014 Annual Report provided Although IO indicators were defined in the Grant information on the number of countries with each Proposal and Progress Reporting Template type of output by pillar (e.g., in Risk Identification, included in the M&E update to the 15th Consultative the number of countries for which GFDRR delivered Group, as described above, the GFDRR Work hazard mapping or risk assessments or data Plan for 2016-18 includes in Annex II a legacy platforms). The 2015 Annual Report similarly tracks M&E framework that moves directly from outputs the number of outputs achieved by indicator and to outcomes to impacts, without distinguishing pillar, and also adds reporting on the “number of between intermediate or longer-term outcomes. large scale programs enabled by GFDRR activities At the same time, in Annex I, a program logic in FY15.” Both Annual Reports also include narrative is articulated at four levels, seeming to indicate descriptions of results achieved in some countries a distinction between intermediate and longer- in each pillar and thematic program. In both Annual term outcomes, although such a distinction is not Reports, some results are reported that go beyond specifically articulated in the text. the output level, but they are not necessarily identified as such, and they are not collected A secondary issue is that some of the GFDRR systematically across the portfolio. “output” indicators—as defined in Annex III of the 2016-18 Business Plan—blur the lines between As described above, a key shortcoming identified outputs and IO. In M&E best practice, outputs by the previous evaluations is that there is too much are generally understood as those contributions “distance” between GFDRR’s output and outcome that are entirely within the sphere of influence of indicators, meaning that GFDRR’s results beyond the program—for example, services delivered the output level are not being adequately captured. (e.g., producing a study) or number of people Intermediate (i.e., shorter-term) and longer- reached (e.g., through a training session). Some of term outcomes have not yet been conclusively GFDRR’s current “output” indicators move beyond defined for GFDRR, nor have they been clearly that sphere of influence to what other comparator integrated into the program’s theory of change. organizations would identify as IOs; for example, This is a significant gap. In program theory, results “policy and regulatory frameworks strengthened” is [ 14 ] ICF International defined as an output in the 2015 Annual Report, but into downstream outcomes, such as improved is also one of the key World Bank AAA IO indicators. identification and understanding of disaster risks. In this way, reporting on these AAA indicators provides One potential path forward is to adopt or adapt a reasonable view into the extent to which GFDRR some (or all) of the World Bank AAA indicators for is effectively serving as a facilitator and delivering reporting IO outcomes. An advantage of using the intermediate results, but offers less insight into World Bank AAA indicators is that they are pre- the likelihood that those results will proceed down existing indicators that have been adopted by other the chain and ultimately achieve outcomes and World Bank trust funds, including those in the Water impacts. and Energy and Extractives Global Practices. A more robust assessment of GFDRR’s M&E system, A disadvantage is that the indicators are not as well as the development and implementation integrated into GFDRR’s program theory. For of a refined M&E plan, could help better articulate example, if GFDRR deepens knowledge (one of GFDRR’s expected process of change, identify a the AAA indicators) in a given country, the program set of suitable and relevant IO indicators as well model does not provide a theory of change for as longer-term outcome indicators, and move the how or whether deepening knowledge translates program toward a stronger results orientation. What do comparator organizations do? A 2011 evaluation of the World Bank’s trust fund portfolio found a frequent lack of results frameworks with clear outcome objectives and indicators for monitoring progress. The majority of the 36 randomly selected trust fund programs lack outcome-level evidence; most had defined objectives in terms of inputs or outputs. A parallel evaluation of the World Bank’s global and regional partnership programs similarly found that few programs have generated systematic evidence about achievements at the outcome level, owing to generally poor monitoring and evaluation. Those that do monitor and report against indicators at the outcome level typically do so for intermediate, or lower-order, outcomes. For example, the Energy Sector Management Assistance Program (ESMAP) tracks and monitors outputs and intermediate outcomes, as defined by the AAA indicators. This approach was recently recognized by the World Bank’s Independent Evaluation Group (IEG) as being well-designed to track and report on the entire results chain of each activity. Sources: ƒƒ IEG. 2015. World Bank Group Support to Electricity Access, FY2000-2014: An Independent Evaluation. Volume II: Together for Energy: How Partnership Programs Support Energy Access. Washington, DC: World Bank. ƒƒ IEG. 2011. Trust Fund Support for Development: An Evaluation of the World Bank’s Trust Fund Portfolio. Washington, DC: Independent Evaluation Group, the World Bank Group. ƒƒ IEG. 2010. The World Bank’s Involvement in Global and Regional Partnership Programs: An Independent Assessment. Washington, DC: World Bank. ICF International [ 15 ] 4. Recommendations Each of the previous evaluations made a set of that DRM focal points have helped ensure that recommendations intended to improve future activities maintain momentum and advance GFDRR results measurement and achievement. toward outcomes at-scale. These staff have also These recommendations have been consolidated been instrumental in mainstreaming DRM into into two categories below; the first recommendation World Bank operations, particularly where such is directed at improving GFDRR’s contribution to mainstreaming is a stated objective of the GFDRR downstream results and the second is focused program in that country. For example, follow-up to on monitoring and evaluating those results. ensure that communities of practice, technologies, Appendix C presents a preliminary action plan that and other GFDRR-supported activities continue defines potential concrete steps to address these to be implemented after individual grants have recommendations and, particularly, to define a path closed may lead to better outcomes. forward towards strengthening GFDRR’s results ƒƒ Consider more focused or cohesive approaches measurement. within countries. Both evaluations found that These recommendations are based on evidence GFDRR is producing valuable results, but the size and findings from 2014 and 2015; since then, of GFDRR’s program is small compared to overall GFDRR has taken decisions that affect its strategies country needs. Within each country, focusing and approaches, and that may or may not align with in on activities that provide added value and these recommendations. The recommendations build on GFDRR’s strengths could support better below should be understood in this historical achievement of sustainable and higher-order context. outcomes. For example, limited evidence was found by both evaluations of sustained results of Recommendation #1: Deepen and sustain engagement one-time training events or conference attendance on the ground. Deeper and more sustained not connected to other, ongoing GFDRR initiatives. engagement could improve potential for achieving In contrast, both the 2014 and 2015 evaluations downstream results by addressing several of the noted that a cohesive strategy has supported challenges identified by both the 2014 and 2015 results achievement in countries where it has been evaluations, including limited readiness or capacity used. At the country level, grants could be more to use some of the technologies piloted by GFDRR, purposefully designed to build on and reinforce long development periods for certain types of each other; results are stronger in countries where interventions, high government turnover, and there is a clearer linkage and trajectory among occasional lack of follow-up by GFDRR. Improved grants. Focusing efforts on building institutional engagement could be fostered as follows: capacity—rather than individual staff capacity— may also be an effective strategy. ƒƒ Prioritize interventions that link to broader initiatives and make use of GFDRR’s well- Recommendation #2. Strengthen GFDRR monitoring recognized technical expertise. Country studies and evaluation of results beyond the output level. suggested that activities that are linked to World As discussed above, measuring and evaluating Bank, government, and other donor initiatives the results of technical assistance programs— and programs are more likely to have strong especially those focused on resilience—is difficult. stakeholder support, show better potential for M&E is in the early stages of implementation for contributing to results at-scale, and achieve climate and disaster resilience programs. Many leverage or influence. Similarly, interventions that global programs hosted by the World Bank also make use of GFDRR’s comparative advantages in lack evidence of results beyond the output level. the DRR community, including technical expertise GFDRR’s challenge is further compounded by and regional thematic initiatives, also show strong the broadening scope of its work plan, with focus promise for achieving results. on a variety of issues from resilient cities, to infrastructure, to gender, to climate change, and ƒƒ Support and coordinate through DRM focal the way that its technical assistance often informs points. The 2014 and 2015 evaluations found broader investment operations. [ 16 ] ICF International GFDRR needs a sound methodology that clarifies ƒƒ Make further refinements to the M&E framework the theory of change with a straightforward results to enable more robust M&E. These refinements framework, identifies a limited number of meaningful could include defining a limited number of SMART and measurable indicators at the outcome and (Specific, Measureable, Achievable, Relevant, impact level, and explains the role of evaluation in and Time-bound) intermediate outcome indicators answering questions that are of interest to GFDRR that are logically tied to the results framework; stakeholders, and particularly the CG. Tying these developing indicator definitions and measurement elements together in a coherent M&E system protocols, and identifying data sources to improve would enable GFDRR to better communicate the reliability and validity of reporting; better how the program is delivering results through its aligning the M&E framework with the Sendai engagement in countries. Framework indicators (under development) to improve relevance; developing approaches and Some of these improvements could include: tools to measure gender outcomes within GFDRR activities; and defining where and how evaluation ƒƒ Refine the existing program results framework can help address the questions and needs of to “close the gap” between current GFDRR GFDRR constituents. outputs and outcomes. Long time horizons for achieving disaster resilience outcomes ƒƒ Address operational aspects of monitoring and and non-linearity makes monitoring long-term reporting. Making improvements to GFDRR’s M&E outcomes very challenging. Instead, monitoring system as described above may also require some at the intermediate outcome level (e.g., two-to- operational changes to improve efficiency and five years) is being increasingly recognized as ease of implementation. Systems or tools might be a viable approach for resilience programs. The developed or improved, roles and responsibilities 2014 and 2015 evaluations also showed that this might be clarified, and different staff or resources is the level of results where GFDRR’s contributions might be required. In addition, clearly stating how are most evident, given GFDRR’s valuable role monitoring information will be used is important— as a facilitator/catalyzer of progress in DRM for example, how the information could be performance at the country level. Including and integrated into the Annual Report, and what other monitoring intermediate outcomes in the M&E reports and learning products might be produced. framework would more accurately hold GFDRR accountable for its own performance. ICF International [ 17 ] appendices Appendix A: Terms of Reference B. LESSONS LEARNED FROM THE 2014 AND 2015 EVALUATIONS Consolidated recommendations from the GFDRR country program evaluations (2014 and 2015) and 4. In 2014 and 2015, GFDRR underwent two action plan. retrospective independent country evaluations focused on a sample of five countries. The I. INTRODUCTION first, conducted by DARA in 2014, focused A. BACKGROUND on Guatemala, Malawi, Nepal, Sri Lanka and Vietnam, while the second, conducted by ICF 1. The Global Facility for Disaster Reduction and International in 2015, focused on Bangladesh, Recovery (GFDRR) is a global partnership Dominica, Ethiopia, Indonesia, and Saint Lucia. program administered by the World Bank Both evaluations focused on GFDRR activities Group. The mission of GFDRR, aligned with between 2008 and 2014, and reported on Sendai Framework for Disaster Risk Reduction results achieved in individual country case (SFDRR), is to help build resilient societies studies; examined the leveraging impact from that manage and adapt to emerging disaster GFDRR’s grants; and made recommendations and climate risks, and to contribute to the related to improving GFDRR’s monitoring and substantial reduction of disaster risk and evaluation (M&E) framework. losses in lives, livelihoods, and health, and in 5. The 2014 DARA evaluation provided a the economic, physical, social, cultural and long and detailed list of recommendations, environmental assets of persons, businesses, which were discussed with members of the communities, and countries. Consultative Group (CG) and GFDRR. The 2. GFDRR is a grant-making facility – not a main recommendations include the following: direct implementer – and as such works i. Measuring performance: The DARA primarily through the World Bank Group and evaluation found that GFDRR succeeds other partners to stimulate policy reform and in delivering planned outputs and makes implement public investment that can better a valuable contribution to the broader protect people from the natural hazard risks disaster risk management (DRM) they face. In line with geographic and thematic performance at the national level. GFDRR priorities set by its donors and partners, triggers policy processes, facilitates GFDRR has supported over 70 countries since some of the necessary conditions for 2006. Between 2007 and 2015, GFDRR’s risk reduction, promotes government annual grant making has grown from $6.4 readiness, and leverages support for million to over $70 million, respectively. DRM. Moving forward, the facilitation role 3. Over the past eight years, GFDRR has invested that GFDRR plays at country level should significant effort in defining and measuring be better captured in the M&E framework. results. The program has been the subject of a number of independent evaluations, and is ii. Leveraging strategy: The DARA well positioned to contribute to an evidence evaluation confirmed that the synergy base on effective management of risks, through between World Bank and GFDRR has better understanding the impact of its program. delivered results at scale, particularly in In particular, the dual focus of the program the areas of risk reduction and financial – on both stimulating institutional reform and protection. leveraging investment – provides an important 6. To further expand the understanding of opportunity to learn what works and accounts the way GFDRR is able to influence and for resources spent. leverage resources for resilience, the 2015 ICF evaluation produced several [ 18 ] ICF International recommendations regarding GFDRR’s i. External: the assignment will allow strengths and opportunities, including that: GFDRR to communicate externally with its Consultative Group, country partners, and i. GFDRR finds and pursues ways to the broader DRM community about the deepen and sustain engagement on lessons learned. the ground. Some options might include continued support for GFDRR focal ii. Internal: the deliverables will support points in-country, improved modalities the GFDRR Secretariat to incorporate for capacity building (e.g., on-the-job lessons learned into its internal decision- training), and designing grants to build on making processes, specifically related to and reinforce each other. (i) improving its reporting mechanisms; (ii) the design and implementation of future ii. GFDRR prioritizes interventions that link GFDRR grants; and (iii) improvements to broader initiatives and makes use of required to further maximize impacts. its well-recognized technical expertise. All ten country studies suggested that 9. This assignment will be supported by a interventions incorporating technical Technical Advisory Group to be established expertise and support are more likely to by the Secretariat. The advisory group will have strong stakeholder engagement, comprise of interested CG members and show better potential for contributing to donor partners, and can be complemented results at-scale, and achieve leverage or by M&E experts nominated by the CG. The influence. main purpose of the advisory group will be to guide the Secretariat in the review of the iii. GFDRR improves documentation of outputs from this assignment. activities and results to support further monitoring and evaluation. III. SCOPE OF SERVICES II. PURPOSE OF ASSIGNMENT 10. As requested by the Consultative Group, the Secretariat is commissioning the services of 7. At the Fall 2015 CG meeting, the Consultative an independent evaluation firm to carry out Group discussed the findings and the following tasks: recommendations of the 2015 ICF evaluation report. During the discussion, the CG i. Consolidate the findings and emphasized the importance of bridging the recommendations from the 2014 gap between outputs and outcomes to better and 2015 country evaluations. The report on how GFDRR‘s activities contribute consolidation will focus on synthesizing to achievement of program results through and prioritizing the common elements in-country engagements. The CG also noted across both evaluations, including: that instead of undertaking another country- level evaluation, which may articulate similar • GFDRR’s contribution to results findings to the previous two evaluations, achievement at the output, outcome, the Secretariat – with the support of an and impact levels. independent evaluation firm – should develop • Contributing and detracting factors for an action plan to systematically address the achieving success; findings and lessons learned from the past two evaluations. • GFDRR’s informing of larger investments by the World Bank and 8. This assignment will inform two key other partners (leverage/influence); and audiences: ICF International [ 19 ] appendices • GFDRR’s monitoring and reporting an M&E Plan, the necessary approaches and systems. tools to systematically assess intermediate outcomes across GFDRR’s portfolio. The ii. Develop a concrete and actionable second assignment will also focus on plan of how GFDRR can systematically how to strengthen GFDRR’s reporting address the findings and lessons learned mechanisms beyond measuring outputs, from the past two evaluations. The action and strengthening the understanding of the plan should, among other things, propose linkages between the GFDRR and World Bank actions to help bridge the gap between portfolios vis-à-vis leveraging of investments. outputs and intermediate outcomes using sound methodology. 11. The firm will need to address some gaps and IV. INFORMATION SOURCES inconsistencies across the two evaluations, while not updating the assessment of 14. In developing the synthesis report, the firm results achievement from either evaluation. will conduct a desk review of all relevant For example, compared to the 2015 ICF internal documents; including assessing evaluation, the 2014 DARA evaluation GFDRR activity documentation, World Bank included a limited analysis of how GFDRR’s operational project documents, and other activities are informing investment programs consultations with GFDRR and World Bank at the country and activity level. Similarly, staff. In additional, GFDRR will provide while the 2015 ICF evaluation identified all documentation related to World Bank results at specific levels of the results chain development policy lending and investment (i.e., outputs, intermediate outcomes, operations which can be directly and outcomes, and impacts), the 2014 DARA indirectly linked to GFDRR interventions (e.g., evaluation reported on “achievements” and GFDRR staff provided technical support to “contributions to DRM outcomes.” As a result, ensure risk was factored into the design of the some interpretive mapping will be required to operation or GFDRR financed analytical work assimilate these differences. which informed the design process). 12. The expected output from the two tasks is V. DELIVERABLES a draft synthesis report. The report will be 15. Draft Synthesis Report: The report will shared and presented at the GFDRR CG present the consolidated findings and meeting (April 25-28, 2016); focusing on the recommendations from the 2014 and priority actions to address the findings and 2015 evaluations, and an action plan recommendations from the 2014 and 2015 for addressing these consolidated evaluations. In developing the synthesis recommendations. The inception draft report report, the firm will conduct desk reviews to will be shared with GFDRR and the CG, assess GFDRR activity documentation, World including the Advisory Group, for review. Bank operational project documents, and other consultations with GFDRR and World 16. Workshop (Donor Advisory Group) and Bank. Presentation: The firm will present the draft synthesis report at the GFDRR Advisory 13. The final synthesis report, incorporating Group Meeting, tentatively scheduled to take the feedback from the CG members and place on April 25, 2016.The advisory group Advisory Group, will inform the TOR of a meeting, organized as a workshop, will be second assignment that will be commissioned an opportunity for the firm and GFDRR to after the Spring CG meetings. This second gather feedback on the synthesis report assignment will focus on inter alia developing from participating CG members. Following [ 20 ] ICF International the advisory group meeting, the firm will also Group and the GFDRR Secretariat, will be present the synthesis report to the broader published following the Spring CG meetings. donor members during the formal CG The report will include (but not limited to): a meetings (April 26-29, 2016). The feedback foreword, executive summary, synthesis of from the CG members will need to be the recommendations and findings from the reflected in the final synthesis report. 2014 and 2015 evaluations, an action plan for addressing the recommendations, and 17. Final Synthesis Report: The final report, relevant annexes. The total recommended incorporating feedback from the CG, Advisory length of the executive summary is 2-3 pages. VI. TIMELINES # Activity February March April May 1 First draft of Synthesis Report 2 Final Draft of Synthesis Report Workshop and Presentation at Spring 3 CG meeting 4 Final Synthesis Report VII. DEADLINES # DELIVERABLES DEADLINE 1 Initial draft of Synthesis Report March 21, 2016 Final Draft Synthesis Report (for circulation at the Spring CG, 2 April 1, 2016 2016) Advisory Group Meeting - April 25, 2016 (tentative) Workshop at Donor Advisory Group Meeting and Presentation at 3 Spring CG meeting (April 26-29, 2016) Date of presentation at Formal CG – TBC 4 Final Synthesis Report May 31, 2016 ICF International [ 21 ] appendices VIII. FIRM QUALIFICATIONS IX. PROJECT MANAGEMENT 18. The firm will be required to demonstrate: 20. The GFDRR team supporting the assignment will include the GFDRR Program Manager, ƒƒ Knowledge and experience with complex the Country Program and Operations Team quantitative and qualitative evaluations; Leader, and the GFDRR Task Manager. The ƒƒ Demonstrated experience with World Bank and GFDRR Task Manager will be the day-to- Trust Fund programs; day project manager to provide oversight on all aspects of the assignment. The firm ƒƒ In-depth knowledge of issues related to DRM shall report and communicate the status policies and operations; and products of the project to GFDRR’s Task ƒƒ Previous experience of theory-based Manager on a weekly basis after the project’s approaches to evaluation; initiation. In addition, there will be at least two ƒƒ Previous experience with the evaluation and/ project meetings via teleconference during or operation of multi-donor programs or global this assignment. The final deliverables will partnerships (preferred); and have to be endorsed by the Consultative Group (CG) and GFDRR Secretariat. ƒƒ Excellent written and verbal communication skills. X. RESOURCES TO BE PROVIDED BY GFDRR 21. GFDRR will provide the following support for 19. Firm qualifications for the assignment include: the purposes of this assignment: ƒƒ Minimum of 15 years of professional i. GFDRR Grant Proposals (including TORs) experience in evaluating multi-disciplinary ii. Outputs projects and programs; ƒƒ Experience with theory of change-based iii. Progress reports (RBMS reports, Aide- evaluations; and Memoirs, and BTORs) and Completion Reports ƒƒ A team comprised of the following specialists: –– A Resilience/Recovery Specialist with iv. Financial reports extensive experience in monitoring and evaluation, particularly in the fields of v. Access to key stakeholders international development, disaster risk vi. Inputs on the data interpretation and management, climate change adaptation, analysis policy influence, and organizational assessment; and vii. Access to any other available –– An Expert in monitoring, evaluation, and background information collected for both reporting who has a good understanding and the 2014 and 2015 evaluations. experience in implementing best practices for M&E; and –– A Communications Specialist with proven understanding of international development issues. [ 22 ] ICF International XI. PAYMENT SCHEDULE 22. The firm will be remunerated for the deliverables as follows: i. 10% upon contract signature ii. 20% upon delivery of first draft of Synthesis Report iii. 30% upon delivery of draft Synthesis report, circulated and presented at Spring CG, 2016. iv. 30% upon delivery of Final Synthesis Report v. 10% upon delivery of Edited Final Synthesis Report XII. OTHER 23. Selection procedure and form of contract: The firm will be selected following the World Bank’s Guidelines: Selection and Employment of Consultants by World Bank Borrowers (January 2011). 24. Duration of assignment: The duration of the contract will be for 3 months from mobilization. ICF International [ 23 ] appendices Appendix B: Supporting Analysis Table B-1. Evidence of Development Finance Informed (US$ Millions) Total Project Cost DRM Component World Bank Financing Country Resources or Activities How did GFDRR influence these resources? Influenced Coastal GFDRR contributed to the improvement of the ToR guiding Bangladesh Embankment long-term research and monitoring, which will be carried out Improvement 400 400 375 alongside the implementation of the CEIP-I, and will directly Project - Phase I inform the design of $300 million of investments under the (CEIP-I) project. Disaster Dominica Vulnerability GFDRR support for spatial data management and sharing Reduction 38 38 17 platform and a shelter vulnerability assessment helped to Program (DVRP) inform development of the DVRP. PSNP IV allocated a portion of their funds for DRM focused activities. GFDRR’s contribution is through supporting Productive Safety Ethiopia strategic initiatives that advance a specific activity or test Net Program IV 2,616 32 600 a concept that can help push the DRM policy dialogue (PSNP IV) forward. Used in this way, GFDRR grants have significantly informed the design of the PSNP and altered the World Bank’s relationship with the government. FDRR activities on technical and scientific information Guatemala Disaster Risk for municipal planning served as inputs for the DPL, and Management DPL 85 85 85 GFDRR funds supported the Government of Guatemala with with a CAT DDO capacity building to support results achievement under the DPL. Aceh-Nias A DRM strategy for the LEPD funded by GFDRR informed/ Livelihoods influenced the project design. During implementation, local and Economic and provincial government and beneficiaries received 8.2 - 8.2 Development training on how to integrate disaster resilience measures. Program (LEDP) These measures increased food security, mitigated against Indonesia future disasters, and increased resilience. The GFDRR focal point participated in project missions Community- and provided training to improve the DRR content of Based Settlement the community settlement plan (CSP) process. GFDRR Rehabilitation for 61 61 60 identified CSP good practices for DRR and provided special Yogyakarta assistance to learning villages as models that later informed community-based DRR investment under the PNPM. [ 24 ] ICF International Total Project Cost DRM Component World Bank Financing Country Resources or Activities How did GFDRR influence these resources? Influenced The GFDRR focal point provided expert consultation to the Western Indonesia World Bank project team and the Ministry of Public Works National Roads on the inclusion of a component that provides technical Improvement 350 1 250 assistance and capacity-building support to strengthen Project (WINRIP) disaster risk mitigation in the roads section. The project now also includes a component that serves as a contingency for DRR. Third National GFDRR provided co-financing for the project in the form of Program for grants to 16 pilot kelurahans in four cities to prepare and Community partially implement community disaster risk action plans. Empowerment 217 - 150 GFDRR also funded guidelines and training for PNPM in Urban Areas community facilitators on DRM. A provisional zero dollar Project (PNPM- component was added in coordination with the multi-donor Urban III) Callable Fund under GFDRR’s Track 3. Shire River Basic GFDRR developed an Integrated Flood Risk Management Malawi Management Plan for the Shire Basin—a tool that could be refined under 136.3 41.6 125 Program (Phase-I) the project to support improved hydrologic and hydraulic modeling of the flood zones. GFDRR funded the development of a pilot probabilistic risk assessment for floods and landslide hazards in the Kosi River Basin. According to the Project Appraisal Document, “[b]uilding on improved hydromet data, and combined Building Resilience with flood risk modeling, exposure and vulnerability to Climate-related mapping, [GFDRR’s] risk assessment should provide 31.3 31.3 -- Hazards (PPCR) additional information towards the development of a real- time flood early warning and decision support system for the Kosi which is also a goal of this project.” GFDRR also built capacity of the hydromet on the design of early Nepal warning systems, with these trainings intended to support PPCR project development. Pilot Program for GFDRR conducted a risk assessment in two districts Seismic School in Kathmandu Valley, trained teachers and students in Safety in the earthquake preparedness, and trained masons in seismic Kathmandu Valley 1.37 1.37 -- resistance construction techniques. The grant from Japan (Japan PHRD PHRD provides for structural assessment, retrofitting of up Fund) to seven buildings, contingency planning, and awareness building. ICF International [ 25 ] appendices Total Project Cost DRM Component World Bank Financing Country Resources or Activities How did GFDRR influence these resources? Influenced Metro Colombo GFDRR funded the update of the hydraulic model of the Urban Colombo Water Basin, to validate the viability and final Sri Lanka Development 321 147.55 213 design of proposed flood structures, and provide design Project engineering support during the final stage of the revision of structure design. Third Rural Transport Project GFDRR TA developed a study on climate proofing for rural (Additional 112.8 -- 97 roads, which MOT will approve for use on the additional Financing) financing (AF) work. GFDRR staff are part of the World Bank project team. GFDRR activities, such as software development for Vietnam Managing Natural mobile-based EWS will continue in this project, and parallel Hazards Project 167 167 150 TA from GFDRR is expected to strengthen the capacity of DRM staff in government line ministries and support national awareness raising efforts. GFDRR’s study on disaster resilience resulted in better Second Northern understanding of risk and greater attention to livelihood Mountains Poverty protection in the Northern Mountains Rural Development 165 -- 150 Reduction Project Project. A concrete example given was that the shelters for pig-rearing were built to a higher standard to withstand the rigors of winter. Note: Additional World Bank operations may have been informed by GFDRR in Vietnam, but they were not able to be confirmed for this synthesis report. In particular, the 2014 evaluation reported that GFDRR had “facilitated the clearance” of the Road Asset Management Project, Coastal Cities Environmental Sanitation project, Irrigated Agricultural Improvement Project, and Ho Chi Minh Environmental Sanitation project. However, because these projects were not mentioned in DARA’s back-to-office report for the Vietnam fieldwork and because GFDRR was not mentioned in World Bank’s project appraisal documents, these projects were not included in this table as being World Bank operations “informed.” [ 26 ] ICF International Sources: See ICF’s 2015 Technical Evaluation Report for sources for Nepal: World Bank. 2012. Nepal - Building Resilience to Bangladesh, Indonesia, Ethiopia, and Dominica. Sources Climate Related Hazards Project. Washington D.C. for Vietnam, Malawi, Guatemala, Nepal, and Sri Lanka : The World Bank. http://documents.worldbank. projects: org/curated/en/2012/12/17116662/nepal-building- resilience-climate-related-hazards-project World Bank. 2011. Vietnam - Additional Financing for the Third Rural Transport Project. Washington, DC: Nepal: Mei Wang. 2012. Official Documents- Agreement World Bank. http://documents.worldbank.org/ for PHRD Grant TF011452. Washington, DC: curated/en/2011/11/15499578/vietnam-additional- World Bank. http://documents.worldbank.org/ financing-third-rural-transport-project curated/en/2012/07/16571355/official-documents-- agreement-phrd-grant-tf011452 World Bank. 2012. Vietnam - Managing Natural Hazards Project. Washington, DC: World Bank. Forni, Marc S.. 2013. Nepal - Nepal: Pilot Project for http://documents.worldbank.org/curated/ Seismic School Safety in the kathmandu : en/2012/06/16484548/vietnam-managing-natural- P129177 - Implementation Status Results Report hazards-project : Sequence 01. Washington, D.C. : World Bank Group. http://documents.worldbank.org/curated/ World Bank. 2015. Vietnam - Second Northern Mountains en/2013/10/18438803/nepal-nepal-pilot-project- Poverty Reduction Project. Washington, DC : World seismic-school-safety-kathmandu-p129177- Bank Group. http://documents.worldbank.org/ implementation-status-results-report-sequence-01 curated/en/2015/02/23980566/vietnam-second- northern-mountains-poverty-reduction-project World Bank. 2012. Sri Lanka - Metro Colombo Urban Development Project. Washington, DC: World Malawi: World Bank. 2012. Malawi - Adaptable Program Bank. http://documents.worldbank.org/curated/ Loan for Shire River Basin Management en/2012/02/15873173/sri-lanka-metro-colombo- Program Project. Washington, DC: World Bank. urban-development-project http://documents.worldbank.org/curated/ en/2012/05/16462826/malawi-adaptable-program- loan-shire-river-basin-management-program-project Guatemala: World Bank. 2009. Guatemala - Catastrophe Development Policy Loan Deferred Draw Down Option Project. Washington, DC: World Bank. http://documents.worldbank.org/curated/ en/2009/03/10384066/guatemala-catastrophe- development-policy-loan-deferred-draw-down- option-project World Bank. 2013. Guatemala - Disaster Risk Management Development Loan with a Catastrophe Deferred Drawdown Option (CAT DDO) Project. Washington, DC: World Bank. http://documents.worldbank.org/ curated/en/2013/02/17425615/guatemala-disaster- risk-management-development-loan-catastrophe- deferred-drawdown-option-cat-ddo-project ICF International [ 27 ] appendix Appendix C: Preliminary Action Plan The preliminary action plan presented below provides potential steps to address the recommendations of the consolidated evaluations, as presented in Section 4 of this report, and to define a path forward towards strengthening GFDRR’s results measurement. Recommendation Action Suggested Steps and Sequencing ƒƒ Identify strategic ways to carry the DRM agenda forward on-the-ground and build stronger partnerships with World Bank country offices. This includes working through staff focused on DRM and staff in other World Bank Global Coordinate through Practices (e.g., energy, transport, water, etc.) that may staff focused on have relationships with DRM-related government agencies DRM and related and/or lending operations that could provide opportunities topics, located in for DRM mainstreaming. Find and pursue ways World Bank country offices ƒƒ Consider opportunities to provide continuity of staff support to deepen and sustain engagement on the for GFDRR activities. ground ƒƒ Seek opportunities for GFDRR technical expertise to inform World Bank, government, and other donor initiatives and programs, to support mainstreaming DRM and up-scaling. Consider ƒƒ Identify ways to prioritize and rationalize activities in more focused countries to identify a (possibly more limited) scope of approaches within activities with higher impact potential. countries, with an emphasis on longer- ƒƒ Avoid one-off events. Focus efforts on building institutional term engagement capacity, rather than individual staff capacity Step 1. Prepare short pre-assessments of the following to understand needs and approaches: Strengthen GFDRR ƒƒ Conduct a brief needs assessment to (a) understand tracking and reporting Develop and implement an who the target M&E users are (e.g., donors and GFDRR of results beyond the output level M&E plan management); (b) identify their priority questions and needs (e.g., information to meet internal reporting obligations, refine program strategy, and inform business planning); and (c) determine whether monitoring and/or evaluation can address those questions, and how. [ 28 ] ICF International Recommendation Action Suggested Steps and Sequencing ƒƒ Conduct a limited benchmarking assessment to understand the M&E approaches taken by other multi- donor global programs focused on technical assistance and trust-funded in the World Bank, as well as other resilience-focused programs. In particular, review approaches that comparator programs have taken to monitor or evaluate the effects of their technical assistance on the outcomes and impacts of larger lending operations. Step 2. Based on these assessments, develop and implement an M&E plan that will serve as a guide for GFDRR stakeholders, including program managers, donors, regional coordinators, and task team leaders. The M&E plan will address the following: ƒƒ How to revise GFDRR’s results framework to incorporate relevant and measurable intermediate outcomes to bridge the gap between outputs and outcomes; to capture the effects of GFDRR on larger lending operations; and to better align the M&E framework with the Sendai Framework indicators, which are still under development, focusing on a few core indicators. ƒƒ Development of indicators to be monitored that are logically tied to the results framework, and their definitions and data sources. ƒƒ Approaches and tools to measure the effect of GFDRR’s activities on gender. ƒƒ Specifications of how the monitoring information will be used (e.g., how the information could be integrated into the Annual Report, and what other reports and learning products might be produced). ƒƒ Operational aspects of monitoring and reporting (e.g., systems or tools that need to be developed or improved, roles and responsibilities, staff or resources potentially required). ƒƒ If, where, and how evaluation can help address the questions and needs identified in Step 1. ICF International [ 29 ]