Concept Note Results and Performance of the World Bank Group 2022 April 26, 2022 1. Background and Context 1.1 The Results and Performance of the World Bank Group (RAP) report presents an annual review of evidence from Independent Evaluation Group (IEG) evaluations and validation work on the development effectiveness of the World Bank Group.1 The 2022 RAP will be the 12th in the annual series. 2. Focus and Audience 2.1 This year’s RAP will focus on the country level. The Bank Group’s outcome orientation agenda emphasizes high-level outcomes, and by focusing on the country level, the 2022 RAP aligns with that agenda. This focus is also a response to the interest of members of the Bank Group’s Board of Executive Directors in reporting on country- level performance. In this context, the RAP team will conduct an in-depth analysis of country-level evidence contained in IEG’s Country Program Evaluations (CPEs) and Completion and Learning Report Reviews (CLRRs) through two types of analyses. First, overall country program performance will be assessed by tracking country program ratings over time. Second, the country program will be used as the entry point to examine the extent to which the Bank Group’s support (that is, project portfolio and advisory services and analytics) contributed to the achievement of the objectives of the Country Partnership Framework (CPF) and the intended development outcome. Furthermore, the extent to which there was a line of sight between the development outcome and high-level outcomes will also be examined. 2.2 The traditional update of project ratings trends will be undertaken. For the World Bank, the RAP team will also identify some drivers of performance by examining ratings at the two ends of the ratings spectrum (highly satisfactory or satisfactory and highly unsatisfactory or unsatisfactory) and ratings in the middle range. For International Finance Corporation (IFC) investments and advisory services and Multilateral Investment Guarantee Agency (MIGA) guarantees, the analysis of drivers of performance will examine ratings at the two extreme ends of the rating scale (for example, highly successful and highly unsuccessful for IFC investments and advisory services; and excellent and unsatisfactory for MIGA guarantee projects). 2.3 The main audience for the 2022 RAP will be the Bank Group’s Board of Executive Directors. Other important stakeholders include senior management; the Operations 1 Policy and Country Services Vice Presidency; management of the Global Practices (Global Industries for IFC) and Regions; directors and senior managers in IFC and MIGA; country directors and representatives; task teams of operational projects; and funders of trust funds. 3. Data Sources 3.1 The primary data sources for this year’s RAP will be IEG’s CPEs and CLRRs. Although all CLRRs covering the 10-year period from fiscal year (FY)13 to FY22 and comprising 152 CLRRs will be covered for the ratings trends analysis,2 a stratified random sample of these CLRRs will be examined to answer the evaluation questions pertaining to country-level performance. The nine completed CPEs will also be examined for the latter.3 For the World Bank, the analysis of findings from CPEs and CLRRs will be complemented by country-level insights from a purposive sample of sector and thematic evaluations (including corporate evaluations), Project Performance Assessment Reports (PPARs), and other relevant evaluative material. For IFC and MIGA, the analysis of findings from CPEs and CLRRs will be complemented by country-level insights from IFC Sector Highlights, Learning Engagements, Synthesis Notes. 3.2 At the project level, the project ratings trends will be updated from the previous RAP based on IEG’s project ratings database. 3.3 Tables 3.1–3.3 provide a count of available IEG evaluations by year. Table 3.1. World Bank Data Sources and Coverage Report Type FY13 FY14 FY15 FY16 FY17 FY18 FY19 FY20 FY21 FY22 Total CPEs 1 2 — 1 1 — 2 2 — — 9 CLRRs 19 23 13 22 23 16 15 11 8 2 152 Sector, thematic, and 5 6 10 11 11 10 9 6 10 7 85 corporate evaluations PPARs 33 47 36 43 35 61 41 20 28 15 359 ICRRs 322 374 413 283 331 306 237 276 221 147 2,910 Source: Independent Evaluation Group. Note: For CPEs, sector, thematic, and corporate evaluations, the fiscal year refers to the publication or document disclosure fiscal year. For CLRRs, PPARs and ICRRs, the fiscal year refers to the evaluation fiscal year. The final number of evaluations will be updated as of June 30, 2022. — = not available. CLRR = Completion and Learning Report Review; CPE = Country Program Evaluation; FY = fiscal year; ICRR = Implementation Completion and Results Report Review; PPAR = Project Performance Assessment Report. 2 Table 3.2. International Finance Corporation Data Sources and Coverage Report Type CY12 CY13 CY14 CY15 CY16 CY17 CY18 CY19 CY20 CY21 Total IFC XPSR EvNotes 88 65 80 90 79 92 99 81 77 9 760 and PES IFC PCRs EvNotes 66 70 54 61 39 45 85 54 64 4 542 Source: Independent Evaluation Group. Note: The table shows the number of completed evaluations (with IEG Evaluation Notes) as of January 31, 2022. Only the 10-year period spanning CY12–21 will be covered for IFC because no evaluations will be completed by the cut-off for this year’s Results and Performance of the World Bank Group, which is June 30, 2022. CY = calendar year; IFC = International Finance Corporation; PCR = Project Completion Report; PES = Project Evaluation Summary; XPSR = Expanded Project Supervision Report. Table 3.3. Multilateral Investment Guarantee Agency Data Sources and Coverage FY12 FY13 FY14 FY15 FY16 FY17 FY18 FY19 FY20 FY21 Total MIGA PER Validation 12 8 6 16 14 11 16 12 6 0 101 Notes and IEG PERs Source: Independent Evaluation Group. Note: FY = fiscal year; MIGA = Multilateral Investment Guarantee Agency; PER = Project Evaluation Report. 3.4 The 152 CLRRs cover 112 countries. The 112 countries are classified according to income level in figure 3.1. Figure 3.1. Countries with Completion and Learning Report Reviews Since Fiscal Year 2013 a. By income level b. By lending group c. By FCS status Low IDA Non-FCS Lower Middle Upper Middle IBRD FCS High 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 Percent Percent Percent Source: World Bank Enterprise Data Catalog; World Bank Classification of Fragile and Conflict-Affected Situations. Note: Country characteristics are determined based on the ending fiscal year of the latest Country Partnership Framework period reviewed by the Independent Evaluation Group. IDA – International Development Association; IBRD – International Bank for Reconstruction and Development; FCS – Fragile and Conflict-Affected Situations. 4. Evaluation Questions 4.1 The evaluation questions are presented in box 4.1. 3 Box 4.1. Evaluation Questions Country program performance (data sources): • For the World Bank, Country Program Evaluations (CPEs) and Completion and Learning Report Reviews (CLRRs) will be complemented by country-level insights from a purposive sample of sector and thematic evaluations (including corporate evaluations), Project Performance Assessment Reports (PPARs), and other relevant evaluative material. • For the International Finance Corporation (IFC) and Multilateral Investment Guarantee Agency (MIGA), CPEs and CLRRs will be complemented by country-level insights from sector highlights, learning engagements, and synthesis notes.. • How have Independent Evaluation Group (IEG) country program ratings changed over time and across types of countries (for example, classified by income level, lending group, or status of countries affected by fragility, conflict, and violence FCS status)?a • To what extent did the World Bank Group’s support (that is, project portfolio and advisory services and analytics) contribute to the achievement of the Country Partnership Framework’s objectives and intended development outcomes? To what extent was there a line of sight between the development outcome and high-level outcomes? • What role did World Bank advisory services and analytics play in informing the World Bank’s support and the policies and programs of governments? Project portfolio performance (data sources): • For the World Bank, PPARs and ICRRs. • For IFC and MIGA, IFC XPSR Evaluation Notes, IFC Project Evaluation Summaries, IFC PCR Evaluation Notes, MIGA PER Validation Notes, and IEG PERs. • For the Bank Group, how have IEG project ratings changed over time and across types of countries and operations?a For the World Bank, what explains variations in the ratings at the two ends of the ratings spectrum (that is, highly satisfactory or satisfactory versus highly unsatisfactory or unsatisfactory) and also, ratings in the middle range? For IFC and MIGA, what explains variations in the ratings at the two extreme ends of the rating spectrum (highly successful and highly unsuccessful for IFC; and excellent and unsatisfactory for MIGA). • For IFC investment and MIGA guarantee projects (in addition to the previous question), what was the relationship between development outcomes and work quality and between development outcomes and the role and contribution of IFC additionality or MIGA guarantees? For IFC investments, what was the relationship between development outcome and IFC’s investment outcome? What was the relationship between IFC advisory services and investments (in particular, those IFC investment projects that had prior or concurrent advisory services)? Source: Independent Evaluation Group. Note: a. Includes both outcome and Bank performance ratings. 4 5. Approach and Methodology 5.1 At the country level, in addition to examining country performance ratings trends emerging from all completed CLRRs, CLRRs for a stratified random sample of 50 countries (table 5.1) will be examined to answer the evaluation questions pertaining to country-level performance. For the Bank Group, country-level insights will also be gleaned from a purposive sample of sector and thematic evaluations (including corporate evaluations), PPARs, and other relevant evaluative material. For IFC and MIGA, the analysis of findings from CPEs and CLRRs will be complemented by country-level insights from sector highlights, learning engagements, and synthesis notes. Table 5.1. Fifty Randomly Sampled Countries Income Level Non-FCS FCS High- and upper-middle N = 17 N=2 income Argentina Kazakhstan Bosnia and Herzegovina Armenia Mauritius Marshall Islands, the Belarus Montenegro Brazil OECS Countries Bulgaria Panama China Poland Jamaica Romania Jordan Serbia South Africa Lower-middle income N = 16 N=4 Bhutan Nicaragua Comoros Bolivia Nigeria Kosovo Cameroon Pakistan Micronesia, Fed. Sts. Congo, Rep. Philippines Myanmar Côte d'Ivoire Samoa Egypt, Arab Rep. São Tomé and Príncipe Guyana Sri Lanka Lesotho Ukraine Low income N=7 N=4 Guinea Senegal Burundi Kenya Tajikistan Congo, Dem. Rep. Madagascar Uganda Gambia, The Rwanda Haiti Source: Independent Evaluation Group. Note: The sample size has 95 percent confidence level and 10 percent margin of error. FCS = fragile and conflict-affected situation 5.2 At the project level, the RAP will update the ratings trends. For the World Bank, the RAP will identify some drivers of performance by examining projects rated at the 5 two ends of the ratings spectrum (that is, highly satisfactory or satisfactory versus highly unsatisfactory or unsatisfactory) and also ratings in the middle range using machine learning (supplemented by NVivo as relevant). The machine learning algorithm will pull out from project documents what it identifies as common characteristics of projects in the satisfactory and unsatisfactory ranges. 5.3 For IFC investments and advisory and MIGA guarantee projects, the RAP will identify some drivers of performance at the two extreme ends of the rating spectrum (for example, highly successful and highly unsuccessful for IFC investments and advisory projects; and excellent and unsatisfactory for MIGA projects).4 Past RAP reports have conducted regression analysis of drivers of performance across the rating spectrum; the focus on the extreme ends aims to elicit new insights on what worked well and not so well. Owing to the small number of evaluated IFC and MIGA projects with development outcome ratings at the two extreme ends of the rating scale, the RAP will conduct manual document review of the projects. 5.4 The evaluation design matrix is presented in appendix A. 6. Engagement, Communication, and Dissemination 6.1 The 2022 RAP team will engage with Board members and advisers, members of World Bank, IFC, and MIGA management, and staff from the World Bank, IFC, and MIGA. The 2022 RAP will be e-submitted to the Bank Group’s Board of Executive Directors by August 31, 2022, for presentation to the Board in September, or on the first available Board date. 7. Team, Budget, and Timeline 7.1 The co–task team leaders for this RAP are Soniya Carvalho and Aurora Medina Siy. Other IEG staff and consultants will also contribute. 7.2 The report will be produced under the overall supervision of Alison Evans (Director-General, Evaluation) and the direct supervision of Oscar Calvo-Gonzalez (director, Human Development and Economic Management) and Galina Sotirova (manager, Corporate and Human Development). During the production of the report, the team will also consult with IEG staff, including ICRR coordinators, CLR coordinators, the Financial and Private Sector Micro Unit, and staff and consultants involved in the validation of self-evaluations. 7.3 The budget for the task is $695,000, including $40,000 for dissemination. Staff costs are estimated at 55 percent of the total task budget, and variable costs, mostly for consultants, at 45 percent. The timeline for the evaluation is included in table 7.1. 6 7.4 The report will also benefit from the advice of three external reviewers: Tamar Manuelyan Atinc (nonresident senior fellow at The Brookings Institution); Monika Huppi (principal adviser, Office of Evaluation and Oversight, Inter-American Development Bank); and Barbara Scott (deputy director general, External Cooperation Management and Project Development, Planning Institute of Jamaica). The reviewers will advise the team on methods and the interpretation of findings. This Concept Note and draft final report will be subject to internal IEG review and the standard process of Bank Group management comments. Table 7.1. Timeline Product Date Timeline Concept Note January 13, 2022 Cleared by director for One Stop January 24, 2022 One Stop review meeting for Concept Note February 25, 2022 Send to World Bank Group management for comments March 18, 2022 Bank Group management comments due April 15, 2022 Concept Note e-submission to Corporate Secretariat Report June 1, 2022 Cleared by director for One Stop June 9, 2022 One-Stop meeting July 11, 2022 Send to Bank Group management for comments August 9, 2022 Bank Group management comments due August 31, 2022 Report e-submission to Corporate Secretariat September 30, 2022, or first available date Board discussion 1The World Bank Group consists of the World Bank (the International Bank for Reconstruction and Development [IBRD] and the International Development Association [IDA]), the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) for the purposes of this evaluation. The evaluation does not cover the International Centre for Settlement of Investment Disputes, which is also part of the Bank Group. 2The Independent Evaluation Group (IEG) evaluated six Organization of Eastern Caribbean States (OECS) countries together in a single Completion and Learning Report Review (CLRR) and provided aggregated ratings. The final number of CPEs and CLRRs will be updated as of June 30, 2022. 3The nine completed Country Program Evaluations (CPEs) include two clustered CPEs: (i) World Bank Group Engagement in Resource-Rich Developing Countries, covering Bolivia, Kazakhstan, Mongolia, and Zambia, with separate full CPE for each country; and (ii) World Bank Group Engagement in Small States, addressing six OECS countries, nine Pacific Island countries, and four 7 African countries (Mauritius, the Seychelles, Cabo Verde, and Djibouti), with full CPEs for OECS and Pacific Island countries at regional level, and case studies for Mauritius and the Seychelles. 4 Development outcome of IFC investment projects and development effectiveness of IFC advisory projects are rated on a six-point scale: highly successful, successful, mostly successful, mostly unsuccessful, unsuccessful and highly unsuccessful. Up until fiscal year 2019, evaluated MIGA projects were rated excellent, satisfactory, partly unsatisfactory, and unsatisfactory. In fiscal year 2020, a six-point rating scale was adopted by MIGA and IEG for consistency with IFC and the World Bank. 8 Appendix A. Evaluation Design Matrix Table A.1.Evaluation Design Matrix Data Collection Strengths and No. Key Questions Information Required Information Sources Methods Data Analysis Methods Limitations Country program performance 1.1 How have IEG country program Country program data. CLRR ratings and CPEs. Compiling ratings data Quantitative analysis of Established methods ratings changed over time and from CLRRs and ratings trends and exist for analyzing across types of countries (for qualitative data from qualitative analysis of ratings trends. example, classified by income CPEs and CLRRs. CPE and CLRR findings. Explanatory power of level, lending group, FCS ratings trends is limited status)? What explains the without qualitative variations in the ratings? analysis. 1.2 To what extent did the Bank IEG assessments of the CPEs and CLRRs. For the Compiling findings on Qualitative analysis of IEG evaluations cover the Group’s support (that is, project Bank Group’s lending World Bank, sector and the Bank Group’s lending CPE and CLRR findings performance of the Bank portfolio and ASA) contribute and nonlending in CPEs thematic evaluations, and nonlending support on the Bank Group’s Group’s lending and to the achievement of the CPF’s and CLRRs. For the Bank PPARs, and other from CPEs and CLRRs. lending and nonlending nonlending support. strategic objectives and the Group, the analysis of relevant evaluative For the World Bank, support. For the World Coverage of the intended development findings from CPEs and material. For IFC and compiling country-level Bank, qualitative analysis performance of the Bank outcome? To what extent was CLRRs will be MIGA, CPEs and CLRRs insights from a purposive of findings from a Group’s nonlending there a line of sight between complemented by a will be complemented by sample of sector and purposive sample of support may be the development outcome and purposive sample of country-level insights thematic evaluations sector and thematic insufficient. high-level outcomes? sector and thematic from sector highlights, including corporate evaluations including Coverage of IFC and evaluations, including learning engagements, evaluations, PPARs, and corporate evaluations, MIGA in the CLRRs may corporate evaluations, and synthesis notes. other relevant evaluative PPARs, and other be inadequate. For IFC, PPARs, and other material. For IFC and relevant evaluative the number of evaluated relevant evaluative MIGA, compiling material. For IFC and investments and advisory material. For IFC and country-level insights MIGA, qualitative projects may be limited MIGA, the analysis of from sector highlights, analysis of findings from in a country as a result of findings from CPEs and learning engagements, sector highlights, the sampling CLRRs will be and synthesis notes. learning engagements, methodology used in complemented by and synthesis notes will selecting IFC projects for country-level insights complement the CPE and evaluation (stratified from sector highlights, CLRRs. sampling by industry learning engagements, group and region and and synthesis notes. not by country). 9 Data Collection Strengths and No. Key Questions Information Required Information Sources Methods Data Analysis Methods Limitations 1.3 What role did World Bank ASA IEG assessments of ASA CPEs and CLR Reviews. Compiling findings on Qualitative analysis of IEG evaluations cover play in informing the Bank in CPEs and CLRRs. IEG IEG assessments of ASA ASA from CPEs and CLR findings on ASA from World Bank ASA. Depth Group’s support and the assessments of ASA from from a purposive sample Reviews. Compiling CPEs and CLR Reviews. of coverage is variable. policies and programs of a purposive sample of of Sector and Thematic findings on ASA from a Qualitative analysis of governments? sector and thematic evaluations including purposive sample of findings on ASA from a evaluations, including Corporate evaluations, Sector and Thematic purposive sample of corporate evaluations, PPARs, and other evaluations including Sector and Thematic PPARs, and other relevant evaluative Corporate evaluations, evaluations including relevant evaluative material. PPARs, and other Corporate evaluations, material. relevant evaluative PPARs, and other material. relevant evaluative material. Project portfolio performance 2.1 For the Bank Group, how have World Bank project For the World Bank, Compiling project ratings For World Bank, Established methods IEG project ratings changed rating data from PPARs ratings from PPARs and data both for the World quantitative analysis of exist for analyzing over time and across types of and ICRRs. For IFC or ICRRs. For IFC or MIGA, Bank and IFC or MIGA. ratings trends using ratings trends. countries and operations? For MIGA, ratings from IFC ratings from IFC XPSR machine learning Explanatory power of the World Bank, what explains XPSR Evaluation Notes Evaluation Notes, IFC (supplemented by NVivo ratings trends is limited variations in the ratings at the ratings data, IFC Project Project Evaluation as relevant) for without qualitative two ends of the spectrum (that Evaluation Summaries, Summaries, IFC PCR identifying some drivers analysis. is, highly satisfactory or IFC PCR Evaluation Notes Evaluation Notes, MIGA of performance. For satisfactory versus highly ratings data, MIGA PER PER Validation Notes, IFC/MIGA, document unsatisfactory or unsatisfactory) Validation Notes, and and PERs. analysis will be used to and also in the middle range? PER ratings data. elicit new information on For IFC and MIGA, what the drivers of explains variations in the performance of projects ratings at the two extreme ends rated at the two extreme of the rating spectrum (highly ends of the rating successful and highly spectrum. unsuccessful for IFC; and Excellent and unsatisfactory for MIGA). 2.2 For IFC investment and MIGA IFC investment outcome, IFC XPSR and IFC PCR Compiling qualitative Quantitative and Although links or guarantee projects (in addition IFC work quality, IFC Evaluation Notes and IFC data. qualitative analysis of relationships among the to the above question), what investment outcome and Project Evaluation project ratings trends parameters can be was the relationship between IFC additionality; MIGA Summaries; MIGA PER from the respective discerned, it may not be development outcomes and Evaluation Notes; IFC ratings databases. 10 Data Collection Strengths and No. Key Questions Information Required Information Sources Methods Data Analysis Methods Limitations work quality, and between guarantees: outcome Sector Highlights and Document analysis of IFC possible to establish development outcomes and and MIGA effectiveness. learning engagements. Sector Highlights and causality. the role and contribution of IFC For IFC, project level learning engagements additionality or MIGA ratings of IFC and IEG sector/ thematic guarantees? For IFC investments, and evaluations and investments, what was the advisory services. Also, synthesis notes. relationship between IFC investment projects development outcome and with IFC advisory services IFC’s investment outcome? support to assess What was the relationship broader country-level between IFC advisory services effects. and investments (in particular, those IFC investment projects that had prior or concurrent advisory services)? Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; CLRR = Completion and Learning Report Review; CPE = Country Program Evaluation; CPF = Country Partnership Framework; FCV = fragility, conflict, and violence; IEG = Independent Evaluation Group; IDA = International Development Association; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; PCR = Project Completion Report; PER = Project Evaluation Report; PPAR = Project Performance Assessment Report; XPSR = Expanded Project Supervision Report. 11