67657 APPROACH PAPER IEG ANNUAL REPORT 2012 RESULTS AND PERFORMANCE OF THE WORLD BANK GROUP Background 1. The Results and Performance of the World Bank Group (RAP) is the annual account of what recent IEG evaluations reveal about the effectiveness of the World Bank Group (IBRD/IDA, IFC, and MIGA) in addressing current and emerging development challenges. The report builds on and synthesizes evidence from recent IEG evaluations and is complemented by relevant information from other sources. RAP 2012 will be the third in a series that began in 2010 with the consolidation of annual reports prepared separately by IEG for the World Bank (IBRD/IDA), IFC, and MIGA. 2. RAP 2012 will describe the progress in and challenges facing WBG efforts to achieve development results and will point to the areas that require further enhancement as guided by evaluative evidence. The report will contribute to ongoing and future discussion within the WBG and in the international community about the directions the WBG can usefully take to fulfill its poverty reduction mission more effectively. Context 3. The report comes at a time when the WBG and its institutions face challenges and continued uncertainty in the global economy, which have far-reaching implications for how to best serve its clients. The WBG has increased the volume of operations over the past few years, especially IBRD operations in response to the crisis, but there are some indications of declining project performance for the World Bank. Internally, the WBG entities are implementing institutional reforms to address emerging challenges in development. The following paragraphs set out the general environment for RAP 2012. 4. State of Global Development. The achievement of development results remains a priority, as was underscored in the outcome document of the fourth High-Level Forum on Aid Effectiveness in Busan, South Korea, in November 2011. Considerable progress has been made in reducing poverty and achieving development goals globally. Preliminary World Bank estimates indicate that developing countries have achieved, ahead of time, the Millennium Development Goal (MDG) of cutting the 1990 extreme-poverty rate in half by 2015. Access to basic education has increased across countries: net primary enrollment in developing countries reached 89 percent in 2009, up from 82 percent in 1999. The World Bank for Results 2011 report highlights that 2009 average annual GDP per capita in developing countries reached $1,873 in constant 2000 dollars; gender parity in primary and secondary education reached 96 percent in 2008; and there was a turnaround in the trajectory of the AIDS epidemic, with fewer people newly infected and dying from AIDS-related illness. Yet significant challenges remain in broadening these gains and making sure they are sustained in a vulnerable economic environment. Though aggregate gains have been achieved, progress has not been equally realized by all developing countries, and income 1 inequalities within countries have risen. For example, nearly three-quarters of developing countries are off track for achieving the MDG for reducing under-five mortality. As a group, poor countries and regions lag on all the MDGs and are unlikely to reach a single target by 2015.1 In China and India, data indicate growing disparities in income among the population, which could pose risks to the long-term sustainability of economic growth. 5. Meanwhile, ongoing concerns over sovereign debt sustainability in developed countries and high rates of unemployment are increasing the volatility of the global economy. The risk of another global crisis or severe downturn is high. At the same time, a threat to long-term growth posed by climate change is real, but agreement on global solutions has been difficult. The ensuing uncertainties from these challenges will have significant impact on the course of development results, and thus on the direction of WBG operations. Moreover, it is important to recognize that these challenges are interlinked, and the measures to address them transcend traditional boundaries. For example, investments in infrastructure, a key measure for job creation and economic opportunities for the poor, could exacerbate climate change risk. The world could invest in developing sustainable energy sources, but the financial crisis has absorbed resources for such investments. 6. These challenges point to the highly complex nature of the environment in which the WBG operates. Complexity likely will continue to grow given the opportunities that may be presented by the surge in new forms of development support, such as public-private partnerships and South-South cooperation, and governmental and nongovernmental entities playing increasingly significant roles in achieving development results. In this environment, partnerships and collaboration with development partners and stakeholders are vital for the WBG to advance its mission. Foreign direct investment flows, though volatile, have increased significantly over the past decade, which points to the larger potential role for the private sector in development. The growing share of FDI originating from emerging markets and developing countries indicates the increasingly prominent role that the developing world plays in the world economy. However, the nature of these investments should be investigated to assess their sustainability (economic, social, and environmental) as well as sector concentrations. The evolving state of global development characterized by interlinkages of issues and multiplicity of players requires the WBG to work across sectors, countries, and institutional boundaries. 7. WBG Response and Performance. IBRD and IDA drastically increased their volume of assistance to developing countries in response to the 2008economic crisis. In FY09-11, IBRD and IDA commitments reached an annual average of $50 billion, compared to just $24 billion for FY06-08 (Figure 1). Gross disbursements followed a similar pattern, indicating rapid and timely provision of support during the global crisis, reaching a peak of $40.3 billion in FY10. The average annual amount over FY09-11 was $33.4 billion, approximately 67 percent higher than the average volume over FY06-08. 1 The World Bank (2011), Global Monitoring Report 2011: Improving the Odds of Achieving the MDGs. 2 8. Development policy operations Figure 1: IBRD and IDA Commitments (DPOs) contributed substantially to the surge (US$ million) in commitments and disbursements, 70,000 particularly for IBRD. The average 60,000 proportion of DPOs in IBRD commitments 50,000 during FY09-11 was 44 percent; a jump of 12 40,000 percentage points from the average in FY06- 30,000 08. DPOs accounted for 60 percent of the 20,000 total gross disbursements for IBRD in FY10, 10,000 0 the highest proportion in the past six years (Figure 2).2 IBRD IDA 9. The profile of activities supported by Source: World Bank data IFC shifted sharply after the economic crisis. Commitment volumes declined in FY09 but Figure 2: Proportion of DPLs in IBRD recovered to the pre-crisis level in FY10, Commitments and Disbursements 70 reaching a high of $11 billion (Figure 3). 60 Short-term trade finance products, established 50 in 2005, increased rapidly during the crisis 40 recovery phase, rising from 11 percent of total 30 net commitments in FY07 to 43 percent in 20 FY11. Trade finance is now by far IFC’s most 10 frequently used instrument. In addition, - IFC’s product mix has shifted from traditional project finance to corporate finance, and % DPL (Commitments) increased emphasis on equity investments. % DPL (Disbursements) IFC also increased capital mobilization from the private sector and other partners. The Source: World Bank data Asset Management Company (AMC), a Figure 3: IFC Commitments wholly owned IFC subsidiary, was (US$ million) 14,000 established in FY09 to act as a fund manager 12,000 for third-party capital to invest alongside IFC. 10,000 Advisory services, consolidated in four 8,000 business lines, continued to be an important 6,000 means of engagement for IFC, particularly in 4,000 the poorest countries and in those with more 2,000 difficult business environments. - IFC Net Commt Original Commitments Source: IFC and IEG 2 Development policy lending was introduced in 2005. Instruments under DPLs in Figure 2 prior to 2005 include Debt and Debt Service Reduction Loans, Poverty Reduction Support Credits, Programmatic Structural Adjustment Loans, Rehabilitation Loans, Sector Adjustment Loans, Structural Adjustment Loans, and Special Structural Adjustment Loans. 3 10. Similar to IFC, MIGA’s annual volume of guarantee issuance declined to Figure 4: MIGA Guarantee Issuance (US$ million) about $1.4 billion during the economic crisis, 2,500 reflecting a decrease in capital flows to developing countries. Guarantee issuance 2,000 was heavily concentrated in the financial 1,500 sector in the Europe and Central Asia region during FY09 and FY10. However, in FY11, 1,000 MIGA activities rebounded with guarantees 500 totaling $2.1 billion (Figure 4). MIGA has responded to changing demands in the - political risk market and needs of developing countries by modernizing its mandate and Source: IEG and MIGA databases product mix, which allowed it to expand the range of its guarantee products and the type of project it may cover.3 11. With respect to the performance of Figure 5: Development Outcome Ratings World Bank-financed projects, while the (% of MS+ by the number of projects) volume of operations entering the portfolio 85 has surged, the development outcome ratings 81 83 80 79 81 as validated in IEG’s reviews have been 77 78 trending downward in recent years. The 75 70 76 share of the number of World Bank 74 70 71 71 investment operations rated moderately 66 69 65 65 satisfactory or above (MS+) in IEG’s validation has declined significantly4 from 78 60 percent for FY06-FY08 exits to 71 percent for FY09-11 exits. There is a gap of 12 percent in the project performance ratings DPO Investment between DPOs and investment projects for Source: World Bank data FY09-11 exits (Figure 5). 12. The development outcome ratings when weighted by disbursements show that 84 percent of investments projects exited the portfolio in FY09-11 were rated MS+. This is a decline by one percentage point compared to the cohort of FY06-08 exits.5 This implies that the development outcome ratings for larger investment projects are generally better than those of smaller investment projects. The gap between the development outcome ratings of 3 Changes in MIGA’s Operational Regulations and Convention became effect ive in FY10 and FY11, respectively. 4 Significant at 95 percent confidence level; significance test based up on two-proportion z-test. 5 Significant at 95 percent confidence level. 4 investment and DPO operations weighted by Figure 6: Development Outcome Ratings disbursements is also significantly large (% of MS+ weighted by disbursements (Figure 6). 95 90 90 13. The development outcome ratings of a 85 83 85 sample of IFC’s operationally mature 80 80 83 84 73 79 76 investment operations improved during the 75 79 79 73 2008-10 period: 72 percent of projects 70 73 71 achieved satisfactory ratings for their 65 development outcomes compared to 63 60 percent in the preceding three-year period (2005-2007).6 Project development outcome ratings based on three-year moving averages DPO Investment peaked in 2009 and leveled off after that— Source: World Bank data possibly due to the global crisis and its effects on IFC operations, particularly in the financial sector. Previous IEG studies have observed that projects approved before the crisis tend to exhibit the effects of a crisis in terms of lower development outcome ratings one to two years after the crisis.7 A close examination of the 2011 cohort of evaluated projects, which will be available for RAP 2012, will determine whether a similar pattern can be detected for the 2008-2009 crisis. Despite the increasing volume and importance of trade finance for IFC’s operations in recent years, this product has yet to be evaluated by IEG.8 The ratings for evaluated MIGA projects showed similar results to those of IFC investments, achieving 70 percent satisfactory outcomes, based on a sample of projects evaluated during FY09-11 (Figure 7). 6 IEG Annual Report 2011: Results and Performance of the World Bank Group. 7 IEG Evaluation Brief: Lessons from Past Financial Crises. 2009. The development outcome ratings of IFC projects approved before previous crisis episodes were significantly below those for projects approved post- crisis. See also IEG’s Results and Performance Report 2010. 8 IEG and IFC management are currently developing an evaluative framework for the trade finance product. 5 Figure 7: IFC and MIGA Development Outcomes (Satisfactory Ratings) IFC Development Outcomes MIGA Development Outcomes ( Three-year Moving Average: % Mostly Successful or Higher) 100% 85% 80% 30% 75% 60% 65% 55% 40% 70% 45% 20% 35% 0% FY09-11 Satisfactory or better Less than satisfactory Source: IEG and IFC data. Source: IEG and MIGA data Note: The evaluation period in the charts is different for each institution because of the different approaches followed by each of them (also see footnote 16). Periods in Figure 7 refer to the time of project evaluation, rather than approval. Rationale for the RAP 2012 14. The external environment as described above indicates that the WBG is facing two challenges.  The WBG needs to build on the progress made in recent decades and help ensure that the gains are shared with a broader segment of the population in developing countries. Economic growth supported by rapid globalization and technological advances accelerated the gains in poverty reduction, but this progress has been uneven across regions and countries and among different groups of people. The 2008-09 economic crisis likely had an adverse effect on the rate of improvement in some of these indicators, particularly for vulnerable groups. Persisting gaps in progress on the MDGs in such areas as maternal health, child mortality, and gender equality are a few examples. Economic growth and policy matter greatly in achieving development results.9 The effectiveness of the WBG’s support for economic growth and policy environment thus becomes a crucial element in expanding the benefits to lagging countries and population groups.  The WBG’s mission of supporting its clients to achieve development results has become increasingly challenging in the face of growing uncertainty and volatility in the global economy. Apart from the complexity involved in providing advice on policy directions and investment priorities in a dynamic environment, unanticipated events that require a timely response (as experienced with the global financial crisis) have broad implications for WBG activities. Shifts in resources to address the unanticipated events create tension between competing demands. The 2008-09 economic crisis tested the WBG’s ability to adapt to changing needs. Since the last RAP, more evaluative information on operations during the crisis response period has 9 The World Bank (2011), Global Monitoring Report 2011: Improving the Odds of Achieving the MDGs. 6 become available. An examination of that data could lead to lessons for ongoing and future efforts to manage results under uncertainty. 15. Independent evaluations can provide insights into what has worked in a given area and how the existing approaches can be modified to meet new contingencies. These insights would be valuable for the WBG to address today’s challenges effectively and allocate its limited resources appropriately. RAP 2012 would thus aim to bring to bear available knowledge from recent evaluations to inform the WBG’s efforts to enhance its development results in an increasingly uncertain environment. It will build not only on the findings of IEG’s evaluations but also other relevant information such as evaluations and research conducted elsewhere in the WBG and beyond; project databases; and structured interviews and surveys as needed. Objectives 16. The purpose of the RAP series is to compile findings from recent evaluations to learn what evaluations reveal about WBG effectiveness in producing results and improving performance. RAP 2012 has two main objectives. First, it will examine the results and performance of the WBG’s activities as captured in recent independent evaluations. Second, the report will distill findings from evaluations to serve as inputs to the institutional efforts in the WBG to enhance its effectiveness. 17. To enhance focus on achieving development results, the WBG is undertaking efforts to improve institutional effectiveness supported by such initiatives as the Corporate Scorecard in the World Bank, the IFC Development Goals (IDGs), product innovations and organizational changes in IFC, and strengthening of the product mix and increased focus on measuring and monitoring results in MIGA. A strategic question for the WBG is how to maintain and enhance its relevance in the development architecture that is emerging in the post-crisis environment. RAP 2012 will review WBG activities, including key corporate initiatives, to assess how well the WBG is placed to deliver development results under continuing uncertainty. Scope 18. This year’s RAP will assess: WBG achievements and challenges in contributing to development results; factors influencing the effectiveness of the WBG in delivering results as well as the performance of WBG operations based on analyses of WBG’s record in supporting development; and progress in the WBG’s institutional initiatives to manage for results, including follow-up to IEG recommendations. 19. The WBG pursues development results at various levels. WBG-supported activities (projects, investments, and guarantees) are designed to achieve specific results. The WBG engages in knowledge work and partnerships with various actors to improve development results through contributing to in-country and global discussions on the development agenda. The results of country programs are being monitored and assessed based on their results frameworks. Sector and thematic results within countries are often pursued as part of the CAS framework, but some of the goals are aggregated at the global level, such as the MDGs. 7 There are also interfaces between results at different levels. Good project results would typically lead to good country program results, but the causality could flow in an opposite way. Satisfactory country program results could enhance the policy dialogue between the WBG and the recipient country, which could lead to better project outcomes. 20. The overall results and performance of the WBG are generated through complex processes of interactions between various levels of results and are influenced by external conditions. Institutional effectiveness would either foster or hamper the efforts to improve results at each level and to generate synergies between them (Figure 8). To draw lessons that are useful to enhance WBG results and performance, one needs to triangulate the findings at these respective levels. The assessment framework of RAP 2012 will encompass analyses at each of these levels of activity to identify the drivers of results. Figure 8: Building Blocks of WBG Results and Performance 21. Project performance ratings provide core inputs to IEG’s evaluations and to previous annual reports. RAP 2012 will continue to use the wealth of information accumulated in IEG over years to examine the portfolio performance by WBG institutions to identify institutional lessons and challenges. The declining trend in the World Bank’s portfolio performance, as indicated earlier, will be a major area of analysis in this exercise. For IFC’s operations, the implications of the shift in instruments as well as the outcome drivers of the results of investment and advisory services will be considered. For MIGA, the report will cover experience in implementing changes to its mandate and in strengthening business development. 22. A country program encompasses the diverse development objectives as determined based on country demand and through policy dialogue. The primacy of country programs as a means for the World Bank to influence development results has been pointed out repeatedly in past evaluations. IFC and, to a lesser degree, MIGA are beginning to establish some alignment, where appropriate, with the country programs through participation in the 8 Country Assistance Strategy (CAS) development process. Given its significance, the RAP will further examine IEG’s work on country programs to draw lessons to improve effectiveness in implementing country programs. The analysis will also review the contributions of knowledge products to country program results as identified in evaluations. It will also comment on IFC’s and MIGA’s unique capabilities and constraints in addressing the needs of the private sector via the CAS framework. 23. In examining the drivers for sectoral results, the report will organize relevant IEG findings and project evaluation ratings into four broad areas of WBG intervention: expanding economic opportunities and strengthening institutions; building infrastructure for growth; enhancing human development; and ensuring environmentally and socially sustainable development. Similar to RAP 2011, the report will cluster and discuss the findings related to all WBG institutions under each area of intervention. By doing so, the report expects to identify the differences and opportunities for collaboration in the operations of WBG institutions. 24. In addition, the report will undertake a similar exercise to identify lessons on cross- cutting themes that are not easily covered in the proposed organizing device. The report will explore selected themes that have been identified to be critical in development results in past evaluations and WBG corporate strategies such as the Post-Crisis Directions Strategy.10 These areas would likely include the WBG’s efforts in governance and anti-corruption as well as managing risks and enhancing resilience to unanticipated events (for example, climate change, economic crisis, and natural disasters). The sectoral/thematic organizing device proposed for RAP 2012 builds on the four development goals introduced in last year’s RAP. More specific discussion on this device and the differences from the approach in last year’s RAP is in Attachment I. 25. In analyzing WBG’s results and performance, IEG’s databases and ratings of the various aspects of operational results and performance at the project and country level will provide the analytical basis. The report will also use information from WBG’s databases and documents such as Implementation Status Results reports. The data will be analyzed from a variety of angles. Differences between countries with various income levels (middle income and low income), differing conditions (fragile and conflict-affected countries) and regions will be examined. The report will present findings for the WBG as a whole as well as for the individual institutions. 26. The report will also base its findings on recent sector and thematic evaluations, Country Assistance Strategy Completion Report Reviews (CASCR-Rs), Country Program Evaluations (CPEs), and project-level evaluations to synthesize and consolidate the findings that are relevant in today’s context. Other possible areas for assessment include the WBG’s role in global and regional efforts to embark on collective actions based on the information available in IEG’s review of Global and Regional Partnership Programs. The report will focus its attention on new IEG reports that were not covered in last year’s RAP and draw on 10 The Post-Crisis Directions Strategy included: targeting the poor and vulnerable; creating opportunities for growth; providing cooperative models; strengthening governance; and managing risks and preparing for crises. 9 the findings of other evaluations to examine whether similar findings have been consistently identified. More discussion on the sources of information for the report is available in the Methodology, Limitations, and Design Matrix section below. 27. As indicated in Figure 8, the WBG’s ability to achieve results and improve its performance depends much on its organizational effectiveness and incentive mechanisms. The report will also examine whether evaluations indicate any useful lessons for the ongoing effort to enhance institutional results and performance. Several institutional initiatives are being undertaken to strengthen the WBG’s capacity to deliver effective operations and services, improve client responsiveness, and establish strong knowledge sharing environment. The report will assess how some of the ongoing initiatives, such as the Corporate Scorecard, IFC’s initiatives after the economic crisis, and MIGA’s self-evaluation system and performance indicators help address the issues that the WBG faces in achieving development results in the current environment. The report will also describe the WBG’s use of monitoring and evaluation systems as a management tool. The experience in the reform of the Management Action Record being undertaken in collaboration between IEG and WBG management will be reviewed as well. 28. The tentative outline of the report is presented in Attachment II. Questions for RAP 2012 29. The key evaluation questions for RAP 2012 are consistent with those of the previous RAP and attempt to add additional dimension and insight. They are as follows:  How effective have the WBG’s contributions been in supporting progress on achieving development results? o Was the WBG’s assistance relevant to development priorities? o Has the WBG been realistic in defining its objectives, sequencing its activities, and tailoring its programs to the situation on the ground? o What do evaluations show as the key drivers of results and how effective has the WBG been in supporting these areas? o How WBG support has been complementary to that of other development partners?  How effective has the WBG been in improving the performance of its operations? o What are the emerging trends from the latest portfolio data and what are the reasons behind them? o Are the WBG instruments effective in addressing the needs on the ground given the variations across countries and environment? o What are the good practices and shortfalls emerging from independent evaluations, and which factors explain performance? 10  How well are the WBG institutions making adjustments in response to the evolving development challenges? o What does independent evaluation reveal about the quality of the monitoring and evaluation system in the WBG and how it can be improved? o Have the WBG’s responses to global challenges enhanced its preparedness to manage future crisis situations? o How well have WBG institutions capitalized on potential synergies from collaboration? Methodology, Limitations, and Design Matrix 30. RAP 2012 will synthesize the evaluative evidence produced by IEG at the project, country, sector, and corporate level. The analysis in this effort will be complemented by WBG and external data and evidence to inform IEG’s findings about the WBG’s effectiveness in delivering results and improving its performance. 31. The report will cover the activities of all WBG entities—the World Bank (IBRD and IDA), IFC, and MIGA—and will encompass a variety of interventions: (i) financial services, such as Bank development policy and investment lending; IFC investment services (loans, equity, and guarantees); and MIGA political risk insurance; (ii) nonlending services, including Bank analytical and advisory activities, IFC Advisory Services, and World Bank policy dialogue; and (iii) support to global and regional programs. The report will present the findings and lessons that are applicable to the three institutions of the WBG in an integrated manner, while recognizing the differing organizational and business models and the value of identifying the issues pertaining to the performance of individual institutions. 32. Development results encompass the output, outcome, or impact (intended or unintended, positive or negative) of a development intervention.11 Traditionally, monitoring and evaluation, including that practiced in the WBG, have focused on measuring outputs and meeting stated objectives. Accordingly, examining the performance of WBG activities, which can be measured by project outputs and the degree to which a given activity achieved the pre-defined objectives, will form an important part of RAP 2012. Clarifying how outputs and stated objectives translate into development outcomes and impacts would offer a crucial opportunity for learning from experience. However, it is often difficult to link the intervention and the observed outcomes given the extended causality chains of interlinked activities and effects. Recognizing this difficulty encountered in IEG evaluations to be used in this report, RAP 2012 will trace the causal chains as identified in evaluations and will seek to identify common factors that affect the outcomes and impacts. It will also aim to assess the nature of contributions that the WBG made to generate results. 33. The report will aim to derive findings using a mix of methods and approaches. It will aim to assess performance by going beyond ratings for individual operations and programs, aiming to identify generalizable findings and lessons learned that can inform the operations 11 Glossary of Key Terms in Evaluation and Results Based Management, May 2002, OECD. 11 and strategy of the WBG. It will build on recently completed IEG evaluations and validations of self-evaluations of WBG projects and activities. 34. At the project level, databases of the World Bank, IFC, and MIGA portfolios and project evaluations will be the basis of analysis, which will include a mix of statistical and qualitative analysis. The data analysis will guide more in-depth review of individual project approval, supervision, and evaluation documents based on the issues identified. Statistical analysis will be conducted to identify trends and determine confidence intervals of the data presented. As in previous annual reports, the explanatory power of these analyses will be bound by confidence levels and margins of error, depending on sample sizes. Qualitative analyses of relevant information collected from inside and outside of IEG will supplement these gaps.12 The document review will be facilitated by text analysis tools (such as atlas.ti). 35. Similarly, a database of CASCR-Rs and Country Program Evaluations will be used to identify country-level outcomes, trends and common themes. The team will also draw on recent (FY08-12) IEG sector, thematic, and corporate evaluation reports to identify patterns of performance and common issues and themes. As for FY2012 IEG deliverables, evaluations completed by March 31, 2012, are expected to be covered in this report. Attachment III provides a list of recent reports that this report will build on. 36. Analysis of initiatives to improve the effectiveness and the results focus of the WBG will be based primarily on a review of relevant documents and interviews with key staff. Additional information will be collected through the ongoing reform and update of the Management Action Record, including a survey of groups of actors with a stake in IEG’s work. Information and data generated directly from IEG evaluations or validation will be complemented by additional information collected from literature and databases, as well as background papers prepared on specific topics. The main sources of information will be:  Project evaluations such as Implementation Completion Report Reviews (ICR-Rs) and Project Performance Assessment Reports (PPARs) for World Bank-financed projects; Expanded Project Completion Reports (XPSRs) and IEG cluster notes for IFC investments; Project Completion Reports (PCRs) for IFC Advisory Services; and MIGA Project Evaluation Reports (PERs). Specifically, RAP 2012 will be based primarily on ICR-Rs for projects exiting the portfolio in FY09-11, PPARs prepared in 12 IEG validates 100 percent of the Implementation Completion Reports of closed projects in the World Bank’s lending portfolio. For IFC investments, IEG evaluates a random, representative sample of projects reaching early operating maturity, stratified by industry department. The sample is representative of the population in terms of various characteristics. The sampling rate is set at a level sufficient to make statistical inferences about success rates in the population at the 95 percent confidence interval with a sampling error of +/-5 percent or less based on a three-year rolling sample. On IFC Advisory Services, IEG validated 100 percent of Project Completion Reports during FY08-FY09. Beginning in FY10, IEG moved to a sampling approach, selecting a stratified (by business line) random sample from the population of PCRs (70 percent sampling rate). The sampling rate is set at a level sufficient to make inferences about success rates in the population at the 95 percent confidence interval with a sampling error of +/- 5 percent or less. With regard to MIGA’s operations, until FY11, IEG evaluated random samples from the population of MIGA guarantees issued 3 years before, covering about 50 percent of the eligible population. Starting FY12, the MIGA project evaluation program will cover 100 percent of all eligible mature projects. 12 FY09-11; 249 XPSRs for IFC investments reaching early operational maturity in CY2009-2011; 276 PCRs for IFC advisory services operations closed in FY09-10; and 21 MIGA PERs for guarantees reaching early maturity during FY08-11.  Country evaluations, including both CAS Completion Report Reviews (prepared for 63 countries during FY09-12, Q3)13 and more detailed Country Program Evaluations (10 prepared during FY08-11].  Sector and thematic studies produced by IEG since FY08, including some reports that are expected to be completed by March 31, 2012 (see Attachment III).  Management Action Records to capture the progress and gaps identified in degree of adoption of past IEG recommendations and to assess ongoing efforts.  Databases, including IEG’s ICR Review databases and the World Bank’s Business Warehouse.  Documents and records on relevant corporate strategies, initiatives, programs, and evaluations, including the World Bank’s Scorecard, Implementation Status Results reports of World Bank financed operations, IFC’s Development Goals and institutional reforms, and MIGA’s self-evaluation and monitoring initiatives.  Interviews of staff will be conducted as needed to collect the updated information on the progress and status of WBG activities. 37. For specific approaches pertaining to evaluation questions, please see the report’s design matrix in Attachment IV. Work Plan and Timeline 38. The report will be prepared by an IEG-wide team including Stephen Pirozzi (co-Task Team Leader; Senior Evaluation Officer, IEGCC), Jiro Tominaga (co-Task Team Leader; Senior Evaluation Officer, IEGCC), Stephan Wegner (Senior Evaluation Officer, IEGPE), and Melvin Vaz (Evaluation Officer, IEGPS). Vikki Taaka will provide administrative support. The team will be supported by short-term consultants and the IEG staff involved in the Management Action Record. The report will also draw on ongoing evaluative work undertaken in IEG, such as on trade finance. The report will be prepared under the direction and guidance of Ali Khadr (Senior Manager, IEGCC). 39. The peer reviewers for the report will be Amédée Prouvost (Senior Advisor to the WBG Chief Financial Officer), Shahrokh Fardoust (former Director for Strategy and Operations, DEC), Bert Hofman (Director and Chief Economist, EAP), and a reviewer with private sector background (TBC). The team also will explore the possibility of forming an external advisory panel to provide views on the draft report. Consultation with relevant Bank Group management staff will be conducted following the usual practice of IEG’s large-scale evaluations. 13 IFC and MIGA are included in CAS Completion Report Reviews only for countries where CASs are jointly developed and implemented by Bank, IFC, and MIGA. 13 40. The report will be prepared by late June 2012 as indicated in Table 1 below. Table 1: RAP 2012 Timetable TASK DATE Internal IEG One Stop Meeting—Approach Paper January 2012 Approach Paper to SEC for distribution to CODE March 2012 Internal (IEG) draft review May 2012 Management commenting period May-June 2012 Final report submission to CODE/Board June 2012 CODE/Board Meeting July 2012 Disclosure September 2012 41. The RAP previously has been discussed at a meeting of the Board of Directors, and it is assumed that this practice will continue for this year’s RAP. Following the Board discussion, the team will seek to undertake systematic communication of the report’s main findings to WBG staff and senior management. The team will also identify targeted venues to disseminate the report to external stakeholders selectively. 14 Attachment I: Classification of IEG Findings by Areas of WBG Operations In analyzing the evaluative information pertaining to sectoral and thematic results, RAP 2012 will organize IEG findings under several broad areas of WBG operations. The areas to be used in the report were developed building on the four core development goals introduced in the RAP 2011. The four core goals were derived through an effort to identify the underlying development goals and priorities that the WBG has sought to address through its strategies and operations. RAP 2011 used them as a framework to classify, review, and communicate the effectiveness of the operations of the WBG over time. This year’s RAP will retain the basic approach of undertaking analyses of the sectoral/thematic results around broadly defined areas to enable analysis of the operations of all the WBG institutions in a systematic manner. However, there will be modifications reflecting last year’s experience. In last year’s RAP, the framework was based on high-level goals pursued in WBG operations. For World Bank-supported projects, Sector Board assignments will be used this year to ensure better alignment with World Bank practices, mutual exclusivity, comprehensive coverage, and replicability in classifying the WBG activities into each area. More importantly, the use of broad areas of intervention in this year’s RAP has been undertaken primarily as one of the ways to organize the findings from diverse WBG operations systematically, while the last year’s RAP introduced them as an evaluative framework. RAP 2012 will use multiple levels of classification to undertake the analyses. The first set of organizing principles is based on Sector Board coding for the World Bank-financed activities to conduct comprehensive analyses of WBG operations based on traditional sectors. The second set of principles involves using the World Bank’s thematic codes to assess specific development themes which are often cross-sectoral. Efforts related to governance and anti- corruption as well as managing risks and enhancing resilience to unanticipated events are some of the examples. The comparison of goals in last year’s report and the organizing device areas to be used in RAP 2012 are indicated in the table below. Comparison of Organizing Device Used in RAP 2011 and Planned for RAP 2012 RAP 2011 FOUR DEVELOPMENT RAP 2012 GOALS AREAS OF WBG OPERATIONS CROSS-CUTTING THEMES  Expanding economic  Expanding economic opportunities and opportunities strengthening institutions Selected topics e.g.  Improving public sector  Building infrastructure for  Governance and Anti- effectiveness growth Corruption  Enhancing human development  Enhancing human  Managing risks and enhancing  Increasing resilience to development resilience to unanticipated socioeconomic and  Ensuring environmentally and events environmental risks socially sustainable development. The project codes to be included under each broad area of WBG operations are as shown below. 15 Classification of IEG Findings by Areas of WBG Operations AREAS OF WBG WORLD BANK CODING OF PROJECTS IFC SECTOR CODE/ADVISORY SERVICES OPERATIONS EVALUATED(*) BUSINESS LINE(*) Expanding  Economic Policy  Agriculture and Forestry economic  Poverty Reduction  Oil, Gas and Mining opportunities and  Finance and Private Sector  Construction and Real Estate strengthening Development  Food and Beverage institutions  Public Sector Governance  Chemicals  Legal Department (Law and Justice)  Nonmetallic Mineral Product Manufacturing  Agriculture and Rural Development  Primary Metals  Pulp & Paper  Textiles, Apparel & Leather  Plastics and Rubber  Industrial and Consumer Products  Finance & Insurance  Collective Investment Vehicles  Wholesale and Retail Trade  Professional, Scientific and Technical Services  Accommodation and Tourism Services  Other  Access to Finance AS Business Line  Sustainable Business Advisory AS Business Line (Corporate Governance; Linkages; SME Management Solutions)  Investment Climate AS Business line Building  Transport  Utilities infrastructure for  Global Information and  Transport & Warehousing growth Communication Technology  Information (Telecoms, Internet, Publication)  Energy and Mining  Public Private Partnerships AS Business Line  Water  Urban Development Enhancing human  Education  Healthcare development  Health, Nutrition, and Population  Education Services  Social Protection  Pharmaceuticals (**) Ensuring  Social Development Sustainable Business Advisory AS Business Line environmentally  Environment (Eco-standards & Sustainable Supply Chain; and and socially  Gender Development Sustainable Energy Market Development) sustainable development (*) 1.Sector board coding will be used for World Bank supported projects. IFC sector coding and Advisory Service business lines will be used for IFC supported activities. MIGA projects will be classified according to their respective institutional sector designations. 2. The organizing device presented in this table is introduced as a way to organize IEG findings into areas of WBG operations. The report recognizes that some of the cross-cutting themes that have emerged recently as key development challenges are not easily captured in this device. 3. The report will create additional clusters of IEG findings relevant to selected cross-cutting themes. Examples would include governance and anti-corruption as well as managing risks and enhancing resilience to unanticipated events. The latter category would include such topics as economic crises, social protection, climate change, and natural disasters. (**) Formally part of Chemicals 16 Attachment II: Tentative Outline of Evaluation Report I. Introduction and Context A. Recent Progress in Development Goals and Emerging Challenges 1. Overall Environment and Context for WBG Operations 2. Emerging Challenges for the WBG B. Objectives, Scope, and Approach of RAP 2012 II. Results and Performance of WBG’s Operations A. WBG’s Record in Supporting Development—what does evaluation show? 1. Overall performance of WBG institution a) Portfolio Trend and Drivers of Performance b) Shifting Instrument Mix 2. WBG’s Effectiveness by Areas of WBG Operations (update from RAP 2011) a) Expanding Opportunities for Growth b) Building Infrastructure for Growth c) Enhancing Human Development d) Ensuring Environmentally and Socially Sustainable Development e) Cross-Cutting Themes 3. Results in Countries: Achievement of high level development goals through country programs as assessed in CASCR-Rs, CPEs, and other evaluations to be complemented by non-IEG resources B. What Have We Learned? III. Toward Better Development Results A. Institutional Initiatives 1. WB Scorecard and IFC’s Development Goals 2. Other Initiatives B. Results Orientation and Independent Evaluations: M&E system and MAR Reform 1. M&E in Projects 2. MAR Lessons and Risk Assessment Framework IV. Conclusion 17 Attachment III: Main IEG Evaluations Recently Completed or Expected to be Completed in FY08-12 A. By Areas of WBG Operations Areas Sectoral/Thematic Evaluations (**) Expanding Economic Crisis Response II (FY12)* Opportunities and Strengthening Crisis Response I (FY10)* Institutions Earnings Growth and Employment Creation (FY09)* IFC Poverty Focus and Results (FY11) Poverty and Social Impact Analysis (FY08) (also in Social Development) Doing Business (FY08) WBG Guarantees (FY08) SME Finance (IFC) (FY08) GAC (FY12)* Public Sector Reform (FY08) Decentralization in client countries (FY08) Growth and Productivity in Agriculture and Agribusiness (FY10) Building infrastructure for growth ICT (FY11) China Utility-based Energy Efficiency Finance (CHUEE) Program (FY10)* Water (FY10) Municipal Development (FY09) Enhancing Human Development Education Portfolio Note (FY11) Nutrition Impact Evaluation Synthesis (FY10) Health, Nutrition and Population (FY09) Social Safety Nets (FY11)* Environmentally and Socially Poverty and Social Impact Analysis (FY08) (also in Expanding Economic Sustainable Development Opportunities) Environmental Sustainability (FY08)* Gender (FY10) (*) These are evaluations on cross-cutting themes that apply to multiple areas. (**) The report will draw on findings in other relevant evaluations and the MAR process as well as information from non-IEG sources as appropriate. B. By Cross-Cutting Themes Possible Themes Sectoral/Thematic Evaluations (*) Governance and anti-corruption GAC (FY12) Managing risks and enhancing Crisis Response II (FY12) resilience to unanticipated events Crisis Response I (FY10) Earnings Growth and Employment Creation (FY09) Climate Change I-II CHUEE (FY10) Social Safety Nets (FY11) Environmental Sustainability (FY08) (*) The report will draw on findings in other relevant evaluations and the MAR process as well as information from non-IEG sources as appropriate. 18 C. Country/Regional Evaluations Region Evaluations(*) Africa Africa Action Plan (FY11); Nigeria (CAE and CIR) (FY08), Uganda (FY09), Mozambique (FY10), East Asia and Pacific Timor Leste (FY11) Europe and Central Asia Georgia (FY09) Latin America and Caribbean Peru (FY10) Middle East and North Africa West Bank and Gaza (FY11); Egypt (FY08), South Asia Bangladesh (FY09), Nepal (FY10), (*) The report will draw on findings in other relevant evaluations and the MAR process as well as information from non-IEG sources as appropriate. D. Corporate Evaluations/Global and Regional Partnership Program Reviews(*) Year Evaluations FY08 Economic and Sector Work (ESW) and Technical Assistance FY09 Country Policy and Institutional Assessment (CPIA); Independent Evaluation of MIGA’s Development Effectiveness—2009 FY10 Poverty Reduction Support Credit (PRSC); Cost-Benefit analysis; MIGA Institutional Effectiveness; Safeguards and Sustainable Policies; IDA Controls FY11 World Bank Progress in Harmonization and Alignment in Low-Income Countries; Trust Fund Support for Development; (GRPPs) Marrakech Action Plan for Statistics, PARIS 21, Trust Fund for Statistical Capacity; The Mesoamerican Biological Corridor; Multi-Donor Trust Fund for the Extractive Industries Transparency Initiative FY12 The Matrix System at Work (to be completed); World Bank Group Impact Evaluations: Relevance and Effectiveness (to be completed) (GRPP) The Global Fund to Fight AIDS, Tuberculosis and Malaria; The World Bank’s Involvement in Global and Regional Partnership Programs (*) GRPP Reviews published/expected in 2011-2012 19 CONFIDENTIAL Attachment IV: Report Design Matrix Information Data Collection Data Analysis Key Question(s) Information Source(s) Limitations Conclusions Required Methods Methods How effective have the WBG’s contributions been in supporting progress on achieving development results? Was the WBG’s assistance  Evaluation  IEG evaluations  Desk reviews  Synthesis of  Data quality and  Examples of relevant to development findings and ratings (existing and ongoing)  Data evaluation reliability relevant and not priorities? (project/ sector/ and evaluators extraction from findings  Sample size very relevant thematic/ country/  WBG staff and IEG and other  Descriptive operations development goals) stakeholders relevant and inferential  Statements  WBG programs/  Project/ country databases statistics about the trends initiatives evaluation databases  Qualitative and the sample of  WBG documents analysis projects Has the WBG been  Evaluation  IEG evaluations  Desk reviews  Synthesis of  Assessment will be  Examples of realistic in defining its findings and ratings  IEG evaluators  Interviews findings constrained by the realistic and objectives, sequencing its (project/ sector/  WBG staff and  Qualitative depth of analyses in unrealistic (overly activities, and tailoring its thematic/ country/ stakeholders analysis evaluations and ambitious) programs to the situation development goals)  WBG documents documents on realism operations on the ground?  WBG programs/  Statements initiatives about the tends  Lessons learned What do evaluations show  Evaluation  Relevant IEG  Desk reviews  Synthesis of  May vary by  Statements as the key drivers of findings and ratings evaluations  Data findings program and sectors about the tends results and how effective (project/ sector/  IEG evaluators extraction from  Qualitative  Lessons learned has the WBG been in thematic/ country)  WBG staff and IEG and other analysis supporting these areas?  Information on stakeholders relevant partnerships/  WBG documents databases stakeholder engagement 20 CONFIDENTIAL Information Data Collection Data Analysis Key Question(s) Information Source(s) Limitations Conclusions Required Methods Methods How WBG support has  Evaluation  Relevant IEG  Desk reviews  Synthesis of  May vary by  Statements been complementary to findings and ratings evaluations (e.g.  Interviews findings program and sectors about the tends that of other development (project/ sector/ Harmonization, Trust  Qualitative  Lessons learned partners? thematic/ country) Fund, GRPP, IFC analysis  Information on Poverty) partnerships/  IEG evaluators stakeholder  WBG staff and engagement stakeholders  WBG documents How effective has the WBG been in improving the performance of its operations? What are the emerging  Evaluation  IEG evaluations  Desk reviews  Synthesis of  Quality of  WBG’s trends from the latest findings and ratings  IEG evaluators  Interviews findings documentation proactive actions portfolio data and what are (project/ sector/  WBG staff and  Qualitative and missed the reasons behind them? thematic/ country) stakeholders analysis opportunities  WBG programs  WBG documents and initiatives Are the WBG instruments  Evaluation  IEG evaluations  Desk reviews  Synthesis of  May vary by  Lessons to effective in addressing the findings (project/ (PPARs on results-  Interviews findings instruments inform the needs on the ground given sector/ thematic/ based lending,  Qualitative discussion on new the variations across country) harmonization, Crisis analysis instruments countries and  WBG programs II, PRSC, advisory environment? (e.g. P4R, IL reform, services) trade finance)  IEG evaluators  Information collected in MAR follow up process  WBG documents What are the good  Evaluation  IEG evaluations  Desk reviews  Synthesis of  Quality of lessons  Lessons that practices and shortfalls findings (project/  IEG evaluators  Interviews findings extracted through can be emerging from sector/ thematic/  Qualitative evaluations generalized to independent evaluations, country) analysis achieve better and which factors explain results performance? 21 CONFIDENTIAL Information Data Collection Data Analysis Key Question(s) Information Source(s) Limitations Conclusions Required Methods Methods How well are the WBG institutions making adjustments in response to the evolving development challenges? What does independent  Evaluation  Relevant IEG  Desk reviews  Synthesis of  Quality of  WBG’s evaluation tell us about the findings and ratings evaluations (e.g. Crisis  Structured findings documentation proactive actions quality of the M&E (project/ sector/ II, GAC, Climate interviews  Qualitative and missed system in the WBG and thematic/ country) Change III, Matrix, analysis opportunities how it can be improved?  WBG programs Private sector and initiatives evaluation) Have the WBG’s  IEG evaluators responses to global  WBG staff and challenges enhanced its stakeholders preparedness to manage  WBG documents future crisis situations? (Scorecard, IDG, MIGA self evaluations) How well have WBG institutions capitalized on potential synergies from collaboration? 22