celebrating 45 years of evaluation excellence growing our influence Growing Our Influence | Celebrating 45 Years of Evaluation Excellence c1 About IEG How IEG contributes to the work of the World Bank Group The Independent Evaluation Group (IEG) is an independent unit IEG seeks to accelerate World Bank Group development within the World Bank Group. It reports directly to the Boards effectiveness through independent evaluation. of Executive Directors, which oversee IEG’s work through the Committee on Development Effectiveness. IEG is charged with We aim to deliver: evaluating the activities of the World Bank (the International Bank • Independent, credible, and useful evaluations for Reconstruction and Development and the International Development Association), the International Finance Corporation, • Impartiality through the employment of rigorous methods that and the Multilateral Investment Guarantee Agency. promote triangulation of findings from a cross-section of sources • New insights through the choice of what we evaluate; how we The goal of IEG’s evaluation work is to influence the World Bank formulate evaluation questions; and which methodologies we Group’s ability to achieve development outcomes globally and use, including the use of the latest technology with its partner countries by providing impartial, evidence-based • Learning through outreach activities that include the Boards of assessments and lessons on drivers of success and failure. Executive Directors, World Bank Group management and staff, Sharing these insights through a variety of channels and engaging and the development and evaluation communities stakeholders in a discussion on follow-up actions ensure that lessons are learned and necessary improvements are made. • Evaluation capacity development for World Bank Group staff, clients, and partners. All evaluations discussed in this report are available on IEG’s website: http://ieg.worldbank.org c2 Independent Evaluation Group contents 2 Message from the Senior Vice President and Director-General, Evaluation 4 Growing Our Influence: 45 Years 6 Evidence to Inform Current Debates 16 Amplifying IEG’s Effectiveness 21 Deepening Influence and Expanding Outreach 25 Developing the Evaluation Practice  33 Future Directions 36 Appendixes message from the senior vice president and director-general, evaluation Forty-five years ago, evaluation became independent at the Over the course of this year, the International Development World Bank. Having been created a few years earlier, the evaluation Association (IDA) 18 replenishment, the largest in history, went into unit proved valuable to the World Bank’s then president—so much effect, and the Bank Group received the largest capital increase so that the Board of Executive Directors demanded that the function package in history. Shareholders entrusted the institutions of the report directly to the Board. Ever since, the Independent Evaluation Bank Group with their contributions while also expressing expecta- Group (IEG) has fulfilled its role to support the Board in holding the tions that it will improve its effectiveness and efficiency and deliver World Bank Group accountable and to help the Bank Group learn on an ambitious forward-looking agenda. from experience to enhance its development effectiveness. During my seven-year tenure as Director-General, Evaluation, Development evaluation has evolved since the 1970s. Starting as I have led IEG through a transformational process to enhance the a largely donor-driven effort to understand whether moneys being effectiveness of our work. We pursued a simple strategy of one spent were achieving expected results, it has now become widely clear IEG-wide goal supported by three strategic pillars and based accepted as a tool to support better governance. More and more on the foundation of effective resource management. countries are investing in national evaluation capacities. At the same time, evaluation faces increasing demands to sharpen relevance, Collectively, the measures of this strategy aligned IEG around the credibility, and influence. Data science, which promises fast turn- goal of influencing change to enhance the Bank Group’s development around insights, social media with its incredible reach and power to effectiveness. The strategic pillars—making strategic choices, influence significant population groups, and debates about whether enhancing quality and credibility, and improving outreach and facts matter, have created an environment in which evaluation needs follow-up—implemented together have resulted in the following: to step up to ensure its relevance in the future. • A work program that delivers evaluations well-timed to inform These heightened demands on evaluation apply to IEG, especially current debates. Many of our recent evaluations draw on now that the Bank Group has received an incredible financial boost. experience to help the Bank Group address immediate issues 2 Independent Evaluation Group to succeed in achieving expected results in the future. During this Our commitment to invest in people goes well beyond IEG. During fiscal year, we delivered evaluations that drew attention to pollution the year, we were excited to relaunch the International Program and climate issues and to stakeholder engagements. These and for Development Evaluation Training (IPDET) together with our new other evaluations complement those of previous years and, when partners, a consortium of the University of Bern, in Switzerland, added to future evaluations, will offer richer insights than any and the Center for Evaluation, in Germany. The two institutions offer single evaluation could. Europe’s longest-standing, demand-oriented master’s degree and continuing education programs in evaluation. IPDET 2.0 will deliver • Using mixed methods to strengthen the evidence base and its first course in July 2018. conduct evaluations in ways that facilitate learning through the evaluation process. This year we expanded our use of net-work IEG at 45 is now a leading analysis, pattern matching, diverse case-based studies, and other institution in development techniques to strengthen the evidence base of our evaluations. evaluation. I am grateful for the opportunity to serve with • A powerful outreach program that draws on strategic excellent colleagues and communications, social media, and a range of engagements. advance independent This fiscal year, the insights from several evaluations were shared evaluation at the World Bank during in-country events. Our flagship report, The Results and Group. I am confident that Performance of the World Bank Group 2017, broke online records with my successor at the for reach and downloads. helm, IEG will continue its work of enhancing Achieving these results required establishing a foundation built on development effectiveness effective resource management. This year, I am particularly proud through excellence in of the new IEG Academy, which provides a structured offering of independent evaluation. training programs to enhance the skills and knowledge of IEG staff. Investing in people to empower them, help them grow in their jobs, Caroline Heider, and advance their careers is important to me. Programs like the IEG Director-General, Evaluation Academy and our award-winning IEG Mentoring Program are critical to IEG’s continued effectiveness. Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 3 growing our influence: 45 years The World Bank Group began evaluating projects in 1970 when to evaluating development effectiveness. These approaches include President Robert McNamara created an Operations Evaluation Unit assessing outcomes against stated objectives, benchmarks, standards, in the World Bank’s Programming and Budgeting Department. In and expectations, or assessing what might have happened in the 1973, the unit became the Operations Evaluation Department, which absence of the project, program, or policy (counterfactual analysis). reported to the Board of Executive Directors and became the first Across projects, IEG looks at the patterns of what works under independent evaluation function in an international financial institution. what circumstances. After evaluation offices were established in the International Finance Corporation (IFC) in 1984 and the Multilateral Investment Guarantee IEG’s evaluation approach reflects and is harmonized with Agency (MIGA) in 2002, the three evaluation functions were merged internationally accepted evaluation norms and principles, such into the Independent Evaluation Group (IEG) in July 2006. as the quality standards for development evaluation of the OECD Development Assistance Committee, the good practice As the scope of World Bank Group operations and its portfolio of standards of the Evaluation Cooperation Group, and the norms products grows, IEG continues to develop and adapt its approaches and standards of the United Nations Evaluation Group. 1970 President Robert McNamara 1973 The unit became the Operations 1984 IFC established an evaluation office. created an Operations Evaluation Unit Evaluation Department, which reported in the World Bank’s Programming and to the Board of Executive Directors and Budgeting Department. became the first independent evaluation function in an international financial institution. 4 Independent Evaluation Group The New IEG Results Framework IEG adheres to a multilayered quality assurance model, which includes in-depth review of intermediate and final evaluation products TIER 1: Strategic Results by internal (IEG) and external peers. A Methods Advisory Function • Accountability was established in fiscal year (FY)16 to promote internal knowledge • Feedback loops, learning, and knowledge sharing on evaluation design issues and methodological innovation. • Evaluation capacity development This fiscal year, the Bank Group introduced a Bank Group–wide evaluation framework, which reiterated the independence of IEG TIER 2: IEG Outputs and made explicit our dual mandate of promoting accountability • Accountability and fostering learning. IEG’s new Results Framework aligns with • Feedback loops, learning, and knowledge the World Bank Group’s evaluation framework and the revised • Evaluation capacity development IEG mandate. TIER 3: IEG Performance • Evaluation planning, selection, and resources • Evaluation implementation • Evaluation reporting, dissemination, and follow-up • Contribution to corporate goals 2002 MIGA established an 2006 The World Bank, IFC, and evaluation office. MIGA evaluation offices were merged to form IEG. Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 5 evidence to inform current debates In FY18, IEG set out a strategy to align with the World Bank Group’s priorities as outlined in the “Forward Look” and the twin goals (eliminating extreme poverty and boosting shared prosperity). External factors also played a role in shaping IEG’s focus. In particular, helping the Bank Group and other stakeholders achieve the Sustainable Development Goals informs decisions on what to evaluate. Bearing these factors in mind, this year IEG selected evaluations in each of its three strategic engagement areas (fostering resilience to global shocks and threats, inclusive and sustainable economic growth, and investing in people) designed to enhance the World Bank Group’s development impact and address the most pressing development challenges facing its clients. In addition, IEG assessed the Bank Group’s engagement with citizens and IFC clients and evaluated two country programs and conducted three pilot meso evaluations. 6 Independent Evaluation Group STRATEGIC ENGAGEMENT AREA 1 pollution concerns in client countries. The evaluation revealed that while the Bank Group’s urban transport inter ventions to address air pollution Fostering Resilience to Global Shocks were relatively effective, its efforts in addressing pollution from solid waste and Threats and waste water treatment faced multiple challenges. The evaluation Growth, and the consequences and risks thereof (for example, also found that IFC’s Advisory Services to help client companies reduce climate change), is a leading concern among development agencies. waste, increase resource efficiency and introduce renewable energy were Two of IEG’s FY18 evaluations examined how effectively the Bank largely successful. To strengthen the Bank Group’s role in supporting Group supported the health of our planet. Toward a Clean World client countries to better address pollution, IEG’s report called for the for All reviewed the Bank Group’s projects centered on pollution Bank Group to better leverage country evaluation analyses and improve issues. Carbon Markets for Greenhouse Gas Emission Reduction national pollution monitoring systems. in a Warming World explored the evolution of the Bank Group’s role in carbon markets and how it can draw on this experience Carbon Finance for the future. The thematic chapter in this year’s Results and Climate change is one of the biggest threats facing the world today, and Performance of the World Bank Group report reviewed whether much of global warming is owing to the rise in greenhouse gas emissions. the World Bank Group portfolio has increased its focus on Carbon markets reduce greenhouse gas emissions through strategies environmental sustainability, brought together the findings of past such as credits, carbon offsets, or cap-and-trade schemes. The Bank evaluations on environmental sustainability, and assessed what Group has led in developing these strategies for the past 20 years. IEG’s has happened as a result of IEG’s recommendations. evaluation Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World found the Bank Group to be very effective in thought Pollution leadership and innovating in carbon finance and building capacity but Economic and population growth, increased urbanization and traffic, less successful in generating local development co-benefits and and a shift toward more industrialized economies has led to high documenting results. IEG recommended several paths to enhancing the levels of pollution in many developing countries. IEG’s evaluation Bank Group’s carbon finance support, including reinforcing coordination Toward a Clean World for All assessed the extent to which the World among projects, strengthening country focus, and continuing to pilot Bank Group has been relevant, effective, and efficient in addressing market-based and scalable approaches for reducing emissions. Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 7 STRATEGIC ENGAGEMENT AREA 2 Essential Health Care Services IEG’s evaluation World Bank Group Support to Health Services Inclusive and Sustainable Economic Growth indicated that Bank Group support is well aligned with the health IEG assesses the extent to which Bank Group projects generate needs and priorities of client countries and covers all key drivers growth that creates jobs and is inclusive, resilient, and sustainable. of universal health care—although improvement of access, quality, This year, IEG evaluated how well the Bank Group is helping the or health systems receives greater emphasis. Strong service delivery growth of international trade. mechanisms ensure equitable access to quality healthcare, but synergies between the World Bank and IFC remain weak for inte- Facilitating Trade grating public and private financing to support such mechanisms. In the evaluation World Bank Group Support to Facilitating Trade IEG recommended that this relationship be bolstered if the Bank 2006–17, IEG determined that the World Bank led in promoting trade Group is to fully contribute to the goal of universal health coverage. facilitation with a degree of success in lowering trade costs. The Bank Group helped countries strengthen trade facilitation, trade logistics, trade policy, and trade finance through its policy dialogue, Engaging Stakeholders investment and advisory work, and investment climate reform In FY18, IEG completed two evaluations focused on engaging with activities. However, reform efforts often failed, and simplifying reg- stakeholders: citizens in one and IFC strategic clients in the other. ulations to facilitate ease of doing business led, in some cases, to a reduction in protection for and monitoring of the social objectives of Engaging Citizens regulations. To address this, IEG recommended that the World Bank Engaging citizens in development operations can tangibly influence adopt a stronger programmatic approach and develop a framework the quality of services and development outcomes. When the for identifying and monitoring the effects of trade facilitation reforms. Bank Group introduced corporate indicators, mainstreaming citizen engagement became a priority in project design. The evaluation STRATEGIC ENGAGEMENT AREA 3 Engaging Citizens for Better Development Results found that most previous engagement had come as a by-product of safeguards Investing in People policy requirements, but the Bank Group has gradually embedded IEG has evaluated many dimensions of the Bank Group’s work noncompulsory beneficiary feedback mechanisms in many projects. in investing in people. In FY18, IEG turned its attention to health Citizens also participate more in the preparation of country strategies services. Universal health coverage can help to maintain healthy in the reformed country engagement model. However, design, populations, which in turn drive economic growth and development. implementation, and monitoring of citizen engagement is still lacking. 8 Independent Evaluation Group Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 9 10 Independent Evaluation Group Moreover, many of the Bank Group’s instruments, such as develop- Maximizing the Impact of Development Policy Financing ment policy financing consultation or convening power, remain in IDA Countries: A Stocktaking of Success Factors and Risks underused in promoting citizen engagement. IEG recommended that the World Bank encourage teams to establish regular and The evaluation sought to inform decisions on the use of develop- continuous engagement that uses multiple tools and is embedded in ment policy financing in International Development Association (IDA) country systems and to strengthen the monitoring of such activities. countries by providing evaluative insights into drivers of success and risks. In the context of the record replenishment for IDA18 and the IFC Client Engagement important role development policy financing plays in its implemen- IFC’s engagement with clients promotes economic development tation, the evaluation found that the relevance of design is a critical by supporting productive private enterprises in developing countries. factor influencing development outcomes of development policy Findings discussed in IEG’s evaluation IFC’s Approach to Engaging financing in IDA countries. Systematic analytical work, technical Clients for Increased Development Impact indicate that making client assistance, government ownership, and sound macroeconomic engagement a strategic priority beginning in the early 2000s led policy frameworks are also important. to improvements in IFC’s performance. Enhancing and applying IFC’s Experience with Inclusive Business: An Assessment strategic client engagement more methodically was shown to of IFC’s Role, Outcomes, and Potential Scenarios support development impact. IEG recommended that IFC identify In 2010, IFC recognized the inclusive business concept as important areas of integration with sector and country strategies and program- for reaching people at the base of the economic pyramid, but its matic approaches and establish clear accountabilities and incentives support has remained largely passive. IEG found that, on the basis for strategic client relationship management. of their development outcome ratings, IFC projects with inclusive business models perform no differently than the rest of IFC’s Meso Evaluations: A Pilot portfolio. IEG suggested IFC strengthen its analysis or knowledge of the income or welfare effects of its inclusive business projects In response to the strong demand for focused, just-in-time evalua- on people at the bottom of the pyramid. tions, IEG piloted the meso evaluation product line in FY18 with three reports. These evaluations aim to distill and share evaluative findings IFC’s Asset Management Company and lessons on well-defined topics from past IEG evaluations. And where needed, they include additional analytical work. IFC’s Asset Management Company (AMC) is one of the World Bank Group’s private capital mobilization platforms that facilitates private Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 11 equity investments for institutional investors to invest alongside engaged extensively with technical assistance engagement at the IFC in a wide range of sector, regional, and thematic funds. IEG’s subnational level. Although the World Bank has generally been at the pilot meso evaluation of AMC provided a deep-dive analysis of the forefront of the policy debate in key areas of engagement, contribut- effectiveness of the business model, AMC’s additionality to its clients ing with lending and knowledge services, its presence at the national and stakeholders (institutional investors, investees, and IFC), and policy debate level could have been greater. IEG recommended that outcome achievements from five of thirteen active funds managed the Bank Group ensure its presence in policy dialogue and continue by AMC since 2009. This evaluation contributed to real-time to engage broadly to support further policy and institutional reforms learning, using sophisticated methods such as benchmarking fund and capacity building, especially at the subnational levels, and help performance against public and private market equivalents and maximize effective private finance for development. capturing investors’ expectations and demands through IEG’S Global Institutional Investor Survey. Rwanda Country Program Evaluation Rwanda aspires to attain lower-middle-income status by 2035 by transforming the economy from one based predominantly on subsis- Country-Focused Evaluations tence agriculture to a knowledge-based, regional service hub. IEG’s In FY18, IEG reinstated country program evaluations as an important evaluation looked at the alignment of the Bank Group’s strategic accountability exercise. These evaluations focus on countries with objectives with and contributions to these efforts over FY09–17. The common factors that make lessons transferable to other communities. Bank Group offered sustained technical and financial support in the areas of social protection, agriculture, and energy, which are vital Mexico Country Program Evaluation to the structural transformation of the Rwandan economy and to Mexico, a middle-income country, experienced stagnating growth reducing social vulnerability. The Bank Group’s performance in over the past few years because of low competition, logistics and designing and implementing its program was strong. However, more infrastructure bottlenecks, and a financial sector with low pene- integrative analytical work could have made support for the 2035 tration. IEG evaluated the Bank Group’s support to the country goal more effective. IEG recommended that the Bank Group use between 2008–17, noting that engagements target several areas, analytical work strategically to draw timely attention to insufficiently including contributing to strengthening social safety nets and addressed binding constraints or policy errors that a client country education, providing timely budget support, bolstering infrastructure, must overcome to reach middle-income status. simplifying tariffs and customs, and supporting tax reform. It also 12 Independent Evaluation Group Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 13 Results and Performance of the World Bank achieved moderately satisfactory or better development outcome Group 2017 ratings. This represents a 3 percent improvement from the previous review period (FY11–13) but remains below the corporate scorecard IEG’s flagship report, the Results and Performance of the World target of 75 percent. Finally, IEG analyzed the Management Action Bank Group (RAP), is our annual review of the development effec- Record to determine how well recommendations are being trans- tiveness of the Bank Group. The 2017 report synthesizes existing formed into actions for change. IEG found that only 20 percent of evidence from IEG evaluations, validations, and other products, recommendations in their fourth year of implementation rated high complemented by relevant information from other sources. This or complete. To make more progress, the engagement process year’s report found that the Bank Group’s efforts to mainstream between the management of the Bank Group institutions and environmental IEG needs to be strengthened. sustainability in its country- and project-level work Validations and Project Reports show a 4 percent A significant part of IEG’s work involves validating the self-evaluations increase across required at the completion of all Bank Group projects. During FY18, its activities. IEG IEG validated 427 self-evaluations. In addition, IEG performed also looked at the in-depth assessments of a sample of 43 projects through Project overall development Results and Performance of the Performance Assessment Reports. IEG expects to maintain the outcomes of Bank World Bank Group 2017 coverage rate of validation products while introducing revisions to AN INDEPENDENT EVALUATION Group operations the Implementation Completion and Results Report Reviews (ICRRs) based on IEG’s in line with the World Bank’s institutional reforms in this area. ratings. Findings showed that 73 percent of World Bank projects completed between FY14 and FY16 14 Independent Evaluation Group MIGA 0% 10% 20% 30% 40% FY08–10 FY15–17 Share of Projects with Potential Environmental Benefits RAP 2017 Are World Bank Group World Bank 33% 37% Projects Getting Cleaner, IFC 31% 35% FY08–10 Greener, and More Resilient? 8% 36% FY15–17 MIGA 0% 10% 20% 30% 40% Results by Institution 100% 100% 100% 100% 100% 100% 100%100% World Bank 100% IFC 100% 100% MIGA 100% 80% 80% 80% 80% 80% 80%Investment 88% 88% 80% 80% 80% 86% 86% 86% Projects88% 73% Projects with development 80% outcome 86% 50% 80% with development 88% 80% 82% 82% 82% 62% Projects with development outcome rated 60% 60% MS+60%by number (FY14–16) 60% 77% 60%outcome rated 60% 77% MS+ 77% by number (CY14–16) 60% 60% 74% 74% 60% rated 74% S+ by number (FY11–16) 67% 67% 82% EFI EFIEFI 67%60% HD HDHD 60% 77% SD SDSD 60% 74% 67% 84% Projects 40% with EFI 40% 40% development outcome 49% Advisory 40% 40%HD projects 40% with development 40% SD 40% 40% 40% (FY14–16) rated MS+ by volume effectiveness40% rated MS+ by number (FY14–16) 40% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 0% 0% 0% 0% 0% 0% 0% 0% 0% Results by 0% Practice Group 0% 0% Rated Rated MS+ By MS+Rated By Number Number MS+ By Number Rated Rated MS+ MS+Rated By By Volume Volume MS+ By Volume Rated MS+ By Number Rated MS+ By Volume World Bank Projects Rated in FY14–16 100% 100%100% 100% 100%100% 100% 100%100% 100% 100% 100% 80% 80% 80% 80% 80% 80% 80% 80% 80% Rated Rated MS+ MS+ MS+Rated 86% 86% 86% 88% 88% 88% 80% 86% 80% 80% 82% 82% 82% By Number By Number By Number Rated MS+ 60% 60% 60% 60% 60% 60% 77% 77% 77% 88%60% 60% 60% 74% 74% 74% 82% EFI EFIEFI 67% 60% 67% 67% HD HDHD SD SD SD By Number 60% 77% 60% 74% 40% 40%EFI 40% 67% 40% 40%HD40% 40% 40%SD40% 40% 40% 40% Rated Rated MS+ MS+ MS+Rated 20% 20% 20% 20% 20% 20% 20% 20% 20% By Volume By Volume By Volume Rated MS+ 20% 20% 20% 0% 0% 0% 0% 0% 0% 0% 0% 0% By Volume 0% 0% 0% Note: MS+=Modestly Satisfactory (WB)/Mostly Successful (IFC), S+=Satisfactory, EFI=Equitable Growth, Finance, and Institutions, HD=Human Development, SD=Sustainable Development. Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 15 amplifying IEG’s effectiveness IEG seeks to employ the most rigorous and accepted methods in evaluation while pioneering approaches and learning from experience. The Bank Group and IEG have undergone a series of reforms to improve evaluations and their implementation. Enhancing the Enabling Environment A number of reforms are in progress to enhance the quality of IEG’s evaluations at every phase. In FY18, IEG worked to strengthen the enabling environment for evaluations in the Bank Group in general and for IEG in particular. A major milestone during the year was the drafting of a Bank Group–wide Evaluation Framework, which aims to ensure that the policy environment guiding the production and use of evaluations, including independent evaluations, in the Bank Group is clear, transparent, well disseminated, and adheres to best practices. IEG’s mandate was also updated, and the Management Action Record process continued to be adjusted and improved. Improving Quality IEG continues to seek improvement in the quality of its evaluations. In FY18, IEG reshaped its theory of change, bolstered quality assurance processes, and improved efficiency. 16 Independent Evaluation Group Theory of Change IEG has developed a theory of change to clarify how evaluation con- activities and corresponding key outputs to the potential direct tributes to the Bank Group’s development effectiveness by fostering outcomes (for example, outreach and behavioral influence among learning, accountability for results, and evidence-based decision key target audiences), indirect outcomes (for example, on the making. The theory of change has its basis in IEG’s mandate, and Bank Group’s learning and accountability processes) and, finally, maps the causal linkages between its major functions, its primary the causal linkages with the Bank Group’s two corporate goals. IEG IMPLEMENTS THE STAKEHOLDERS USE EVALUATIONS EVALUATION USE INFLUENCES CHANGE WORLD BANK GROUP EVALUATION PRINCIPLES (strategic selectivity, responsive STAKEHOLDERS USE EVALUATIONS • Improved internal and external accountability planning, adequate resources, TO INFORM for results collaborative approach, rigor, quality • Improved evidence-based strategic • Strategic decisions assurance, customized reporting, • Organizational decisions (e.g., staffing, decision-making broad dissemination, adequate follow-up) resources, processes) • Enhanced learning about what works, for • Operational decisions (e.g., design whom, under what circumstances regarding and implementation) policies, operations and processes • Decisions about ongoing and future • Strengthened capacities to design, conduct IEG LINES OF WORK engagement with the World Bank Group and use evaluations (inside and outside the Validations World Bank Group) Dissemination and Follow-Up Evaluations EVALUATIONS AND EVALUATION CAPACITY DEVELOPMENT INFORM • Enhanced relevance and development effective- Evaluation Capacity ness of the World Bank Group and its contribution Development • Good practices, standards and to achieving the twin goals of eliminating extreme approaches to evaluation inside poverty and boosting shared prosperity. and outside the World Bank Group Enhanced reputation of IEG as a competent and influential Enhanced reputation of World Bank Group as evaluation function a learning and accountable institution Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 17 Quality Assurance In FY18, the strengthened quality assurance process put in place in FY17 was further institutionalized and adapted to different types of evaluations. For major evaluations, existing quality standards for the internal IEG review process have been retained and comple- mented by the introduction of methodology workshops. Improving Efficiency This year, IEG significantly reduced the cost and time it takes to produce major evaluations, cutting the average time by five months when comparing FY18 with FY15. Significant efforts were implemented to bring the average cost of a major evaluation down, including narrowing the scope, limiting case studies, optimizing staff time, and enhancing the use of digital technologies. 18 Independent Evaluation Group Strengthening Methodology Expertise IEG identified five pillars to strengthening the methodological expertise within IEG: PILLAR 1 PILLAR 2 PILLAR 3 PILLAR 4 PILLAR 5 Including (specialized) Leveraging existing Identifying and bringing Building staff capacities: Building staff capacities: methodological skills methodological expertise external methodolog- learning fundamental learning (new) method- as one of the criteria within IEG (fostering ical experts into IEG and common principles ological skills through in IEG’s strategic knowledge-sharing (selectively) of methodological participation in (special- staffing plans among staff and improv- design, data collection, ized) training courses ing the strategic allocation and analysis in the provided by the World of IEG staff with specific framework of IEG Bank Group, universities, methodological expertise) evaluations through and specialized training participation in IEG institutions Academy learning events Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 19 20 Independent Evaluation Group deepening influence and expanding outreach Intensifying Influence requested that IEG and management pursue further reforms that will lead to better uptake and follow-through on evaluation In FY18, IEG pursued growing its influence in several arenas. The recommendations. first was to increase Bank Group stakeholder buy-in, the second to perform in-depth analysis of the uptake of IEG’s recommendations, Learning Engagements and the third to share learning through Learning Engagements. IEG continued piloting Learning Engagements in FY18 and approved 13 new proposals prepared and implemented in collaboration with Accountability: Buy-In and Follow-Up Actions a range of Bank Group operational and corporate teams. The During the fiscal year, IEG continued to pilot different engagement resources earmarked for this pilot product were fully allocated. mechanisms to increase ownership of evaluation findings and Topics included gender in evaluation, cosponsored by the Gender recommendations. cross-cutting solution area; strengthening country engagement practices, cosponsored by Operations Policy and Country Services; In addition, IEG piloted deep dives into the recommendations and the results framework and indicators for improved service of selected past evaluations. These analyses allowed a greater delivery in the water supply and sanitation sector, cosponsored engagement with Bank Group management to discuss systemic by the Water Global Practice. issues that explain the lack of appropriate follow-up actions. The 2017 RAP also analyzed the relationship between IEG’s recommen- Feedback shows high demand for the continuation of this product dations, management’s action plans, and actions taken. Findings line among Bank Group cosponsors and within IEG. On this basis, indicated that action plans do not always align with the intent of IEG suggested mainstreaming Learning Engagements. This has the recommendations, leaving issues unaddressed even when been seconded by the Board and World Bank Group management. management fulfills its action plan. The Board subsequently Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 21 Major Evaluations Go Abroad A workshop in Manila, the Philippines, titled Prospects, Opportunities, and Challenges IEG expanded international outreach by pre- for the Philippines—Insights from World Bank senting findings from several evaluations Group Evaluations, was jointly cohosted by the World Bank Group Engagement at international events. Locations in Situations of Fragility, Conflict, and Violence National Economic and Development Authority included China, Ethiopia, the of the Philippines, the World Bank Country Philippines, and Thailand. Office in Manila, and IEG, and was attended by individuals belonging to the planning, operations, and monitoring and evaluation departments of several government agencies and members of academia. The workshop provided an opportunity to share key insights from IEG’s evaluations. It covered three recent evaluations: World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from IEG Evaluations; Mobile Metropolises: Urban Transport Matters—An IEG Evaluation of the World Bank Group’s Support for Urban Transport; and World Bank Group Engagement in Situations of Fragility, Conflict, and Violence. A New Outreach Strategy To this end, IEG continued to invest in growing its online presence through its website and social media. In FY18, IEG’s website In FY18, IEG developed a new outreach strategy to expand reach, received a significantly higher number of users than it did in the visibility, and influence with key stakeholders. The strategy aims previous year (up 19.7 percent). IEG also produced several derivative to target the most relevant internal and external opportunities. products to make its findings more accessible and digestible for This more strategic and systematic approach to outreach enabled key stakeholders. IEG to better define its focus and proactively anticipate and capture the best opportunities to deliver these goals, as well as facilitate These efforts led to an increase in the number of IEG evaluation better decisions on where to allocate limited resources to get downloads. For FY18, the number of report downloads hit 65,299— the most value. 22 Independent Evaluation Group IEG presented its evaluation of World Bank In a joint event with the Asian Development Group Engagement in Upper-Middle- Bank in Manila, IEG presented the main Income Countries: Evidence from IEG findings of IEG’s evaluation Program for Evaluations to stakeholders in China, Results: An Early Stage Assessment of IEG SYNTHESIS REPORT Program-for-Results World Bank Group Engagement in Upper-Middle-Income Countries Evidence from IEG Evaluations including an event jointly organized with the An Early-Stage Assessment of the Process and Effects of a New Lending Instrument the Process and Effects of a New Lending Government of China and the Shanghai Instrument along with those from the Asian Institutes for International Studies with Development Bank’s report on results- the participation of members of the Bank based lending. Participants included Asian Group’s Executive Board, high-level Development Bank executive directors, Chinese officials, and international resource persons. The other advisers, operations directors, and staff. event was hosted by the World Bank’s Country Director in Beijing for government officials, development partners, and academics. a 10 percent increase from the previous year. IEG’s findings are Client Survey also increasingly being cited in major development reports and IEG’s annual client survey from FY18 confirmed that stakeholders academic research. value and use IEG’s work. Board members shared that they find IEG’s evaluations helpful in assessing the Bank Group’s develop- In addition to launching 11 major evaluations during FY18, IEG ment effectiveness. Half of World Bank staff surveyed reported that delivered or participated in dozens of learning and outreach events. they use IEG evaluations in their work. These included the presentation of two topics at the 2017 Annual Meetings: “Harnessing the Power of the Private Sector in Support of Sustainable Development” and “Civil Society and the World Bank Group’s Country Engagement Model.” Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 23 24 Independent Evaluation Group developing the evaluation practice IEG has undertaken several substantial initiatives to expand and strengthen the practice of evaluation globally. A key highlight for FY18 was an initiative joined by IEG to start a conversation on rethinking and possibly updating the evaluation criteria commonly known as the Development Assistance Committee Evaluation Criteria. The collaboration with the OECD Development Assistance Committee Evaluation Network has led to a global consultation process, collecting views from the evaluation community at large. These inputs will help shape the criteria in ways to ensure they will continue to be fit-for-purpose well into the future. IEG Academy FY18 also saw the launch of the IEG Academy, a comprehensive learning program especially designed and curated to support the 60 Number of IEG Academy events 615 Number of participants across all IEG Academy skills required for IEG staff. In recognition of the importance of just- in FY18 events in-time learning, the academy also provided IEG staff with tools, job aids, and coaching that allow them to develop and apply new skills on the job. Among these, the custom-designed budget manage- ment course has enabled IEG managers and task team leaders to better plan and monitor IEG’s resource allocations and expenses. 83 Number of IEG staff participating 6.3 Overall satisfaction, based on 33 event in at least one IEG surveys with the best Academy course possible score being 10 Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 25 IPDET RMES Together 2018 In FY18, IEG successfully supported the relaunch of the International This was the fourth year in which IEG cohosted the World Bank Program for Development Evaluation Training (IPDET 2.0) with a new Group–wide Results Measurement and Evidence Stream (RMES) delivery partner—a consortium of the Center for Evaluation (CEval), learning days. The initiative brings together talent, knowledge, and Germany, and the University of Bern, Switzerland. IPDET 2.0 will innovations to establish standards and provide operational solutions hold its first course in July 2018 in Bern, and brings a renewed focus on results measurement and evaluation. This year’s theme was on the global south, both in terms of trainees and through potential “Results at the Center: Rethinking How We Manage for Results.” partnerships with academic institutions in the developing world IEG staff presented in 14 technical sessions and 2 plenaries and for the delivery of future trainings. This year, 150 students from in the high-level panel discussion at the opening. IEG also hosted 67 countries will participate in the two-week session. two lunch and learn events during the meetings and held a booth at the Results Fair. CLEAR With the Centers for Learning in Evaluation and Results (CLEAR), an initiative supported by a multi-donor trust fund ending in 2021, IEG invested significant efforts to support the six CLEAR centers in advancing their activities. Together, centers delivered capacity building services to 23,954 individuals in 2017. Of the 185 activities delivered, close to 50 percent aimed to strengthen the evaluation capacity of government-sector clients. In FY19, together with the centers, IEG will help conceptualize the strategy beyond 2021, focusing on striking a balance between the delivery of public goods and promoting sustainability of the network. 26 Independent Evaluation Group Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 27 F Y18 highlights evaluations project-level assessments Project Performance 8 8 Major Evaluations Assessment Reports 33 0 Meso Evaluations 43 57 2 0 Country-Focused Evaluations (FY18 versus FY17) 0 (FY18 versus FY17) project validations media Number of product Number of 65,299 downloads Access to 25 Information 427 431 requests received and delivered Number (FY18 versus FY17) Number of 180,365 of unique 119 learning events website visitors 28 Independent Evaluation Group most downloaded reports IEG WO R K ING PA PE R 2016 / N O.3 Introducing a Framework for Evaluating Service Delivery in Sector Evaluations: Urban Transport, Data for Development Growing the Rural Nonfarm Economy Water and Sanitation, and Nutrition Results and Performance of the An Evaluation of World Bank Support for Data and Statistical Capacity to Alleviate Poverty An Evaluation of the Contribution of the World Bank Group World Bank Group 2017 A N I N D E PE N D E N T E VA LUAT I O N AN INDEPENDENT EVALUATION A N I N D E PE N D E N T E VA LUATI O N 1,963 1,684 1,242 1,238 Results and Performance of Data for Development Growing the Rural Service Delivery in Sector the World Bank Group 2017 Nonfarm Economy to Evaluations Working Paper Alleviate Poverty IEG SYNTHESIS REPORT Mobile Metropolises: Growth for the Bottom 40 Percent World Bank Group Engagement in Urban Transport Matters The World Bank Group’s Support for Shared Prosperity An IEG Evaluation of the World Bank Group’s Support for Upper-Middle-Income Countries Urban Transport A N I N D E PE N D E N T E VA LUAT I O N Evidence from IEG Evaluations 1,144 967 829 746 Growth for the World Bank Group Mobile Metropolises: Toward a Clean World for All Bottom 40 Percent Engagement in Upper- Urban Transport Matters Middle-Income Countries Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 29 IEG online dissemination & outreach — F Y18 highlights Social Media Stats Report downloads (up 10%) 65,299 14,756 followers 2,007 followers (up 7%) (up 28%) 85,531 followers 548 subscribers 260,206 website 180,365 website (up 6%) (up 42%) sessions (up 18%) users (up 20%) 65,253 blog views 18,976 sessions via (down 5%) email referral (up 47%) Most-Read Blogs The Power of Data Visualization — An Evaluator’s Experience 2,641 Rethinking Evaluation: Have We Had Enough... 2,522 Staying on Track with the Sustainable Development Goals — What Evaluation Can Teach Us 2,322 Rethinking Evaluation — Where to Next? 2,024 Evaluation for Policy Making: Are We There Yet? 1,848 30 Independent Evaluation Group 32 32 28 28 IEG Spending 24 24 20 20 Total Spending (FY14–18) Actual Actual Planned Planned Summary of IEG Expenses (FY18) 40 Administration Major Evaluations 36 32 27% 23% $, millions Country- Focused 28 4% Evaluations 5% 2% Meso 24 7% Evaluations Evaluation 16% 14% 20 Capacity 2% Project-Level FY14 FY15 FY16 FY17 FY18a Development Evaluations Actual Planned Knowledge and Validations a. IEG operated on a flat budget in FY18. However, the FY18 numbers appear Communications larger as a result of account changes in the application of institutional benefit Learning rates and price factors. Engagements Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 31 32 Independent Evaluation Group future directions IEG continues to lay strong foundations that strategically align its work with key priorities, and to focus on enhancing quality and credibility as key to increasing influence, while ensuring evaluation insights are accessible to stakeholders when needed to inform choices. It has created systems to invest in its human resources and manage its financial resources efficiently. The forward-looking work program contains many exciting and important evaluations that will contribute to current debates and help the World Bank Group make course corrections for improved results and development effectiveness. Further work will also be needed to increase the uptake of evaluation recommendations that have been lagging. IEG has also repositioned itself globally with its contributions to international debates on evaluation methods. It will need to continue its investments in applying new methods and eventually developing, testing, and rolling out new methods. Given the reach and resources of the World Bank Group, IEG is well placed to make significant contributions to the advancement of the evaluation practice well into the future. Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 33 FY19 Work Program Major Evaluations Other Evaluation Products • Conflict-Induced Displacement PPARs • Renewable Energy • World Bank • Public Finance for Development • IFC/MIGA • Decentralization and Effectiveness of Subnational Learning Engagements Government • Creating Markets Validation Products • Urban Resilience Country Strategy Completion Report • Fostering Regional Integration Validations • Knowledge Flow and Collaboration Project Completion Report Validations • World Bank Group Convening Power AAA/AS Completion Report Validations Country-Focused Evaluations • The Philippines • Albania Results and Performance Report • RAP 2018: Inclusive Growth Meso Evaluations • Drivers of Education Quality • Shaping Social Contracts • Sustainable Irrigation • MIGA Nonhonoring of Government Obligations Guarantee 34 Independent Evaluation Group FY20 Work Program FY21 Work Program Major Evaluations Only (Indicative) Major Evaluations Only (Indicative) • Managing Economic Transformations, I • Managing Economic Transformations, II • State-Owned Enterprise Reform • Water Resources Management • Cities, Growth, and Inclusion • Agricultural Transformation • Demographic and Population Issues • Corruption • Natural Resources Degradation and Vulnerability • Malnutrition and Stunting • Macrofinancial Stability and Crisis Preparedness • Capacity Strengthening in World Bank Group • Mobilization of Private Capital Operations • Development in the Digital Era • IFC in Low-Income IDA Countries and FCS • World Bank Group Agility, Adaptability, and Country-Focused Evaluations Responsiveness in a Changing World Results and Performance Report Country-Focused Evaluations • RAP 2019 Results and Performance Report Meso Evaluations • RAP 2020 Meso Evaluations Growing Our Influence | Celebrating 45 Years of Evaluation Excellence 35 appendixes IEG Deliverables (number, unless otherwise indicated) Deliverables FY14 FY15 FY16 FY17 FY18a Planned Actual Percent Total Major Evaluations 10 7 9 8 10 9 90 Corporate Evaluations 4 2 3 3 3 3 100 Sector and Thematic Evaluations 4 4 2 5 5 4 80 Country-Focused Program Evaluations 2 1 2 — 2 2 100 Impact Evaluations — — 2 — — — — Learning Products — — 6 — — — — Learning Engagements — — — 11 11 7 63 Project Evaluations and Global Program Reviews Project Performance Assessment Reports 47 36 51 50 52 43 83 Global Program Reviews 2 0 — — — — — Total Validations 573 571 442 431 461 427 93 Implementation Completion and Results Reports 361 392 442 285 265 273 103 Completion and Learning Report Review 22 13 267 18 20 11 55 Project Completion Reports 87 72 47 54 53 38 72 Expanded Project Supervision Reports 88 78 88 69 107 95 89 MIGA Validations 14 16 18 14 16 10 63 36 Independent Evaluation Group IEG Staffing (number) Staffing FY14 FY15 FY16 FY17 FY18 Total Staff 109 105 114 118 113 By Grade UC — 1 2 1 1 GA-GD 22 23 23 24 23 GE 6 7 8 11 6 GF 25 21 27 22 20 GG 30 30 30 36 38 GH 20 19 17 20 21 GI and above 6 3 4 4 4 Other (ETT, ETC, UC) 8 12 3 — — Note: — = not available a. For validation products, IEG makes a rough estimate of the planned numbers at the beginning of the year, but actual delivery is determined by how many reports the World Bank Group provides to validate. FY18 delivery of Project Performance Assessment Reports was below target values because of one-time effects from substantial turnover in staff and management. Internal coordinators for these reports have been put in place, and a robust pipeline has been developed for FY19 delivery. In the case of the Completion and Learning Report (CLR) Reviews, actual delivery to the Board is determined by the timing of the Country Partnership Framework board discussion. All CLRs received by IEG are reviewed and delivered to the Board on schedule in line with the Policy and Operations unit of the Secretariat lead time requirements. In other words, the percentage does not reflect slippages in CLR Review delivery but slippages or delays in Country Partnership Framework board discussions. 1818 H Street NW, Washington, DC 20433 http://ieg.worldbank.org ieg@worldbank.org