97273 Approach Paper: Program for Results (PforR)—IEG Process Evaluation May 21, 2015 Table of Contents Background and Context 1 Purpose, Objectives, and Audience 7 Evaluation Questions and Coverage/Scope 8 Evaluation Design and Evaluability Assessment 11 Quality Assurance Process 13 Expected Outputs and Dissemination 14 Annex 1: References 15 Annex 2: Evaluation Design Matrix 16 Annex 3: Methodology Note for Review of the Bank’s PforR Portfolio 18 Annex 4: Outline of Evaluation Report 26 Attachment 1: Management Comments 27 Approach Paper Program for Results (PforR) – IEG Process Evaluation May 21, 2015 Background and Context 1. The PforR Instrument. The Bank introduced three years ago a new instrument – Program for Results (PforR) – as per the Board paper of December 29, 2011.1 As foreseen in the Board paper, World Bank Management has now provided a review of the first two years of operation (document dated March 17, 2015).2 This review assessed the early experience with the design and implementation of PforR operations and the challenges faced by borrowers, development partners, and Bank staff. Board members have requested an early IEG evaluation, and this was reiterated in the CODE discussion of the two-year review on October 24, 2014 and the Board discussion on April 9, 2015. This process evaluation will respond to that request. 2. The PforR instrument is intended to provide a new dimension to Bank operational support. With PforR , the Bank now has three complementary financing instruments: project support through investment project financing (IPF), policy and institutional support through development policy financing (DPF), and program support through PforR. PforRs will support the implementation of government programs, using the government’s own fiduciary, environmental and social safeguard systems. Bank management expects this new instrument to be central to delivering the Bank priorities around results, scaling up in poorer countries, and strengthening country systems. 3. Management’s Two-Year Review. The review addressed the initial experience of PforR operations from the management perspective. It summarized the Bank’s experience, drawing on a literature and desk review organized thematically and with participation of task teams and managers who had worked on the instrument, internal and external surveys, structured interviews with government officials and senior Bank managers, and consultations with key stakeholders. 1A New Instrument to Advance Development Effectiveness: Program-for-Results Financing. December 29, 2011, OPCS, Washington, DC: World Bank. 2 Program-for-Results: Two-Year Review, March 17, 2015, OPCS, Washington, DC: World Bank. 1  4. The two-year review concludes that the PforR instrument has been successfully rolled out across a broad range of countries and sectors, policy requirements have been met, with implementation broadly on track for all but one of the approved operations. On this basis the paper finds that no major changes in the design of the instrument appear to be necessary, but identifies some adjustments and areas for improvement. In this regard the report recommends that the five percent cap on PforR commitments (as a percentage of total Bank commitments) should be increased to 15 percent, although the volume of approved operations has not yet reached the five percent level, and that limited exceptions should be permitted to the procurement exclusions. The paper also suggests other editorial and technical changes to OP 9.00, the Anti-Corruption Guidelines and BP 9.00. 5. The planned IEG evaluation of the PforR instrument will provide an independent assessment of the early experience with the PforR instrument. It will make use of the two-year review, but go beyond it in several respects, most importantly through a careful desk review of all approved PforR projects supplemented by selected field reviews. 6. The evaluation will follow the OECD-DAC description of a program-based approach that has also been used in the Board paper: “…a way of engaging in development cooperation based on the principle of coordinated support for a locally owned program of development. It includes four elements: leadership by the host country organization; a single program and budget framework; donor coordination and harmonization of procedures; and efforts to increase the use of local procedures over time with regard to program design and implementation, financial management, and monitoring and evaluation.” 7. Prior to the introduction of the PforRs, the Bank was supporting government programs in a number of ways, primarily through programmatic-type investment lending operations such as sector-wide approaches, conditional cash transfers, output- based aid, results-based financing in health, and operations under the Education for All-Fast Track Initiative (now Global Partnership for Education). Annex A of the 2011 Board paper discussed the experiences of the Bank and other development partners with these other types of operations, finding that the numbers of program-based approaches had been increasing in response to a stronger focus on results and to demand from clients. While the Bank’s programmatic investment lending operations have not been coded separately, management samples indicated that such operations had performed as well as the overall lending portfolio. The Board paper also reported findings from development partners that performance-based and program-based financing can incentivize progress on results, but that flexibility in instrument design is important. Also, keeping project/program designs as simple as possible improves 2 operational focus and the ease of implementation. IEG has used this material, and in particular the issues identified by Bank management, as one input in designing the issues and questions for the PforR evaluation. 8. The PforR operations have the following key features:  They will finance and support borrowers’ programs, which can be ongoing or new, sectoral or sub-sectoral, national or subnational, as well as community development programs.  They will disburse upon achievement of program results, as determined by the achievement of monitorable and verifiable indicators rather than by inputs. Advances are allowed up to 25 percent of outstanding commitments. Together with funds from other sources, Bank disbursements will finance a borrower’s expenditure program rather than being linked to individual transactions.  The operations will thus provide support for the use of a government’s own systems to implement the program, including for financing plan, procurement, anti-corruption, and environmental and social standards.  They will provide assurance that Bank financing is used appropriately and that the environmental and social impacts of the programs are adequately addressed. The Bank will to this end assess a program’s fiduciary, environmental and social management systems, and agree as necessary with a borrower on any additional measures to provide assurance that potential impacts to the environment and affected people are adequately addressed.  The operations will focus on strengthening the institutional capacity needed for programs to achieve their desired results, thereby enhancing development impact and sustainability. The strengthening of capacity to implement a program will be a priority area for both preparation and implementation support.  In addition, the Board paper also sets out the expectation that PforRs would enhance the Bank’s ability to partner with other development organizations by pooling resources and focusing directly on capacity building and institutional strengthening.  Finally, the Board paper states that PforRs will support improvements in governance and transparency by making program information publicly available and monitoring the achievement of results, including through enhancing the role of beneficiaries and civil society organizations. 9. As of mid-May 2015, the Board had approved 25 PforR operations, providing $4.6 billion of Bank financing to support a total of $11.7 billion in government programs (Table 1 and 2). The total volume of approved operations at the end of the PforR’s initial two years after approval was $1.8 billion or 2.6 percent of total IBRD/IDA commitments over that period. Since inception, there has been a steady increase in the 3  share of PforR commitments from $0.4 billion in FY12 (1.1 percent) to $1.7 billion in FY14 (4.2 percent) 3. In addition, as of May 19, 2015, nineteen operations are at an early stage of preparation (i.e., have completed the concept stage), totaling $4.9 billion in Bank financing. Table 1. IBRD/IDA Lending, Total and for PforR Operations, FY12 to FY15 (US$ billion) Commitments FY12 FY13 FY14 FY15 estimates PforR - IBRD lending 0.3 0.1 0.5 0.90 PforR - IDA lending 0.1 0.7 1.2 1.27 Total PforR - IBRD/IDA lending 0.4 0.8 1.7 2.17* IBRD lending 20.6 15.2 18.6 20.4 IDA lending 15.1 16.3 22.2 16.6 Total IBRD/IDA lending 35.6 31.5 40.8 37.0 Source: Program-for-Results: Two-Year Review (March 18, 2015) and Business Intelligence (May 17, 2015). * As of May 15, 2015 a total of $1.7 billion PforR projects have been approved for FY15. Among them $900 million are for IBRD countries and $822 million for IDA countries. 10. All regions have at least two approved PforR operations; the Africa region leads the way, with ten operations (Figure 1). PforR operations cover most of the key sectors in which the Bank traditionally provides financing. In terms of global practices, health, nutrition and population; social, urban, and rural; and finance and markets are the leading practices (Figure 2). Figure 1. PforR Portfolio by Region (by amount as of May 15, 2015) 50% 44% 45% 40% 35% 30% 26% 25% 20% 15% 10% 9% 10% 7% 5% 3% 0% AFR EAP ECA LCR MNA SAR Source: Business Intelligence 3 Program-for-Results: Two-Year Review, March 17, 2015, OPCS, Washington, DC: World Bank. 4 11. Conceptual Aspects of the Evaluation. This will be a real time process evaluation of an instrument for which there are as yet no completed operations and thus no available IEG-validated data on final outcomes, and only partial data on disbursements, results and outputs. Through a review of the projects approved by end- FY15 the evaluation will assess project designs and – as possible – the early implementation experiences against the stated expectations (in the 2011 Board paper) and in particular the key issues highlighted in CODE and Board discussions: institutional capacity building, expected results, management of fiduciary risks, environmental and social aspects, and eventual evaluability.4 On this basis, the overall additionality so far of this instrument for the Bank and its borrowers will be compared to that of the other Bank lending instruments. Figure 2. PforR Portfolio by Global Practice (by amount as of May 15, 2015) 40% 37% 35% 30% 25% 18% 20% 15% 11% 10% 7% 8% 6% 5% 5% 2% 3% 3% 0% Source: Source: Business Intelligence 4As per the 2011 Board paper, all individual operations will in due course (separately from the present study) be evaluated upon completion. This self-evaluation by Bank teams (to be followed as usual by IEG validations and selective Project Performance Assessment Reports) will at completion stage analyze the results of each operation as well as efforts to achieve institutional capacity building (building on the various assessments and their outcomes). The self-evaluation will also look at the performance of the Bank and the borrower in carrying out their respective roles. 5  Table 2. Summary of Approved PforR Operations (as of May 15, 2015 and US$ million) Total Assoc Board IDA/ Bank Other Country Program name Global Practice Program gov. approval IBRD financing donors cost program Revenue Mobilization Program for Bangladesh 5/9/14 Governance IDA 60 0 73 73 Results: VAT Improvement Program Strengthening Service Delivery for Growth, Poverty Reduction and Trade & Brazil 11/21/13 IBRD 315 0 416 477 Environmental Sustainability in the State Competitiveness of Ceará Health System Quality and Efficiency Health, Nutrition & Croatia 5/8/14 IBRD 103 0 248 409 Improvement Program Population Egypt Inclusive Housing Finance Program 5/5/15 Finance & Markets IBRD 500 0 1,982 1,482 Health Millennium Development Goals Health, Nutrition & Ethiopia 2/28/13 IDA 100 556 676 4610 Program Population Local Government Development II Social, Urban, Ethiopia 5/2/14 IDA 380 0 557 557 Program Rural & Resilience Third Maharashtra Rural Water Supply India 3/12/14 Water IDA 165 0 235 2100 and Sanitation Program Social Protection & Kenya National Integrated Safety Net Program 7/23/13 IDA 250 261 952 952 Labor Oaxaca WSS Sector Modernization Social, Urban, Mexico 6/6/14 IBRD 45 0 93.5 800 Program Rural & Resilience Health, Nutrition & Moldova Health Transformation Program 5/22/14 IDA 28.7 0 114 1,964 Population National Initiative for Human Social, Urban, Morocco 6/28/12 IBRD 300 0 1115 2100 Development (INDH) Phase II Rural & Resilience Health, Nutrition & Morocco Improving Primary Health in Rural Areas 4/24/15 IBRD 100 111 226.2 15.2 Population Public Financial Management for Results Mozambique 6/24/14 Governance IDA 50 0 130.6 287.6 Program Results-Based Bridges Improvement Nepal 6/28/12 Transport & ICT IDA 60 0 148 1,250 Program Program to Support Saving One Million Health, Nutrition & Nigeria 4/23/15 IDA 500 0 1052 552 Lives Population Punjab Governance Reforms for Service Pakistan 11/14/13 Governance IDA 50 0 77 407 Delivery Transformation of Agriculture Sector Rwanda 10/31/14 Agriculture IDA 100 800 1200 1200 Program Rwanda Public Sector Governance Program 10/31/14 Governance IDA 100 30 172 250 Social, Urban, Tanzania Urban Local Government Strengthening 10/23/12 IDA 255 0 255 620 Rural & Resilience Tanzania Big Results in Education 7/10/14 Education IDA 122 130 416 416 Urban Development and Local Social, Urban, Tunisia 7/24/14 IBRD 300 0 363 751 Governance Program Rural & Resilience Support to Municipal Infrastructure Social, Urban, Uganda 3/28/13 IDA 150 0 160 341 Development Rural & Resilience Uruguay Road Rehabilitation and Maintenance 11/13/12 Transport & ICT IBRD 66 160.5 510 1127 Results-Based Rural Water Supply and Vietnam 11/1/12 Water IDA 200 0 260 1,313 Sanitation Social, Urban, Vietnam Northern Mountains Urban Program 6/5/14 IDA 250 0 300 300 Rural & Resilience Total 4,549.70 2,048.50 11,731.30 24,353.80 6 12. Links to Previous Evaluations and Existing Literature. The first PforR projects were approved in the second half of FY12, so for this evaluation there will be no completed projects, and therefore no IEG ICRRs or PPARs. Other evaluation links will be very modest – there have been no previous evaluations on this instrument and only modest IEG evaluative material is available pertaining to other programmatic instruments. However, as background the evaluation will draw on any identified project validations/evaluations for other programmatic investment projects to the extent that these might indicate similarities of some issues, including in the health sector.5 The only ongoing IEG evaluation that could be of some relevance for the PforR evaluation would be the Systematic Country Diagnostic and Country Partnership Framework Process Evaluation (SCD/CPF) (a concurrent process evaluation addressing the new World Bank Group approach to country engagement). 13. IEG will seek to make use of possible synergies with other ongoing evaluations. E.g., if there should be any field visits (meetings with governments or development partners) under the parallel SCD/CPF evaluation, we would arrange for appropriate questions to be asked also regarding PforR. Such synergies will also be sought for country visits under other IEG activities. Purpose, Objectives, and Audience 14. Purpose: This evaluation is being undertaken at the request of the Board, and will provide early feedback on how the PforR instrument is working and its adherence to the stated objectives and intentions in the Board paper. On this basis it will identify any progress achieved, opportunities and issues, and will present lessons and recommendations relevant for the use and possible improvements of this instrument. It will also help lay the foundation for subsequent evaluations of individual operations upon their completion. Since this is a new and untested instrument, the evaluation will also pay significant attention to possible risks – in particular the fiduciary, environmental and social risks that have been in the forefront during discussions so far. 15. Objectives: PforR is a new lending product. It is therefore expected that this evaluation will provide the Board, management, and staff with an assessment, findings and recommendations that will be useful to underpin any future decisions on the PforR 5 IEG (Independent Evaluation Group). 2009. Do Health Sector-Wide Approaches Achieve Results? Emerging Evidence and Lessons From Six Countries. IEG Working Paper 2009/4. Washington, DC: World Bank. IEG (Independent Evaluation Group. 2011. “Results-Based Health Programs in Argentina and Brazil: Project Performance Assessment Report.” Report No. 62571. Washington, DC: World Bank. IEG (Independent Evaluation Group). 2014. World Bank Group Support to Health Financing. Washington, DC: World Bank. 7  instrument including if relevant improving or developing the instrument. The IEG report will also address key questions previously raised in CODE and Board discussions. 16. Audience: Specific audiences are the Board, which is guiding the rollout of this new instrument, Bank senior management, which is implementing the instrument, and client country stakeholders, who would stand to benefit from the appropriate use of the instrument. The evaluation is also expected to be helpful to inform future Board decisions on the mainstreaming of this instrument. In addition, the evaluation should be useful for program task teams engaged in the identification, preparation, and supervision of PforR operations. Evaluation Questions and Coverage/Scope 17. The key evaluation questions are designed to explore the extent to which the features of this new instrument are delivering value to clients. These features include support for government programs, capacity building, results-based disbursements, employment of the country systems used for the program, and assurances for appropriate resource use, as detailed in para 8 above. The evaluation questions will necessarily cut across the areas of program emphasis. For example, the assessment of institutional capacity strengthening will be informed by that of the program action plans, results frameworks, monitoring and evaluation and disbursement linked indicators (DLIs) (all under Question 1), that of the quality of technical assessments (under Question 2), and that of risk assessments (under Question 3). 18. Key Evaluation Questions: Question 1: What has been the overall experience to date with the design, preparation and early implementation of PforR operations and the opportunities and challenges in that regard?  Relevance of PforR operations to Bank Group country programs.  Quality, strength, relevance and use of program action plans in relation to the underlying assessments.  Quality, relevance and use of DLIs as incentives for strengthening the results orientation of institutions and country programs. Question 2: What is the quality of the program assessments, including the technical, fiduciary systems, and environmental and social assessments?  Quality of fiduciary systems assessments including identification of key financial and procurement issues. 8  Quality of environmental and social assessments including identification of key issues.  Quality of technical assessments including their coverage of governance issues.  Quality and use of the assessment of institutional capacity.  Comparison on all of these aspects between PforRs and other recent projects in the same sector and country. Question 3: How effective has the PforR instrument (including policies, procedures, guidelines and their application) been in identifying, assessing and mitigating critical risks?  Identification and responsiveness to fiduciary risks.  Identification and responsiveness to social and environmental risks.  Identification and responsiveness to technical and other program risks including resource availability.  Identification and responsiveness to governance risks.  Identification and responsiveness to political and partnership risks.  Appropriateness and use of the integrated risk rating. Question 4: To what extent is the PforR instrument being used to strengthen national systems for financial management, procurement, environmental and social safeguards, and monitoring and evaluation?  How does the use of country systems in PforRs compare with other operations in each country?  To what extent are PforRs ringfenced, and to what extent do they use the same systems as the broader program being supported?  Has the Bank’s analytical work provided an adequate understanding of the extent to which the level of country system capacities could affect the programs’ implementation?  How adequate are the provisions for building the programs’ institutional capacity, particularly with respect to low income countries with weaker systems? 19. Based on the findings, and recognizing the limitations posed by the early stage of the PforR operations, the evaluation will seek as far as possible to draw some overall conclusions regarding:  How successfully is the instrument achieving so far the objectives and expectations in the 2011 Board document including focus on institutional 9  capacity building, progress towards results, managing fiduciary risks, incentives for learning and innovation?  What are the key parameters for appropriate, successful PforR operations?  What are the critical risks and how have these been managed so far?  How and why has the demand for PforRs developed in IBRD compared with IDA countries?  What are the issues for the evaluability of PforR operations (monitoring of results etc.)?  How do PforR operations compare with earlier projects in the same countries and sectors, including investment project financing (IPF) using DLIs?  How does the existence of the PforR instrument provide additionality to the Bank’s set of instruments for clients and for the Bank, compared to the other existing lending instruments? 20. Scope. The newness of the program and its short implementation experience set clear limitations for the evaluation. As of mid-May, 2015, twenty-five projects had been approved by the Board, and 20 of these operations are now effective. Few projects will have any significant results during the evaluation period, and none is likely to be closed. This evaluation will therefore not be able to discuss final results, but will describe and assess the objectives, structures, risks identified and addressed during preparation, and (where possible) early performance of the approved PforR projects – also paying attention to differences in the structure and operational modalities with respect to the Bank’s other instruments. 21. It is expected that the PforR evaluation report will include a substantial amount of descriptions of approaches, processes and practicalities, given the newness and unfamiliarity of the instrument. The report will also provide a systematic presentation of the key overlaps with and differences from other Bank instruments, for a better understanding of the possible additionality that this instrument may provide to Bank clients and to the Bank, and as an input for the preparation of further guidance material. 22. The evaluation will also devote special attention to the quality, relevance and use of the results framework and DLIs, including the verification protocols. It will assess the extent to which the programs’ monitoring indicators have been integrated into a logical results chain and appropriately designed and specified to incentivize the improvement of institutional performance and the achievement of the desired results. To the extent possible, given the limited implementation experience with PforRs, where there are predecessor operations (same country, sector and direction of objectives), the evaluation will also seek to identify strengths and risks of the PforR instrument and how it might complement other instruments and the extent to which the DLIs have been effective in 10 strengthening the programs’ results orientation, performance and outcomes. Finally, the evaluation will also review the extent to which PforRs are helping build a national system for the monitoring of results. 23. Evaluation Criteria: The evaluation will (as far as possible given the limitations) assess relevance, efficacy and effectiveness of this new instrument, and its contribution to a stronger Bank results focus, against the expectations and criteria of the 2011 Board document. 24. Assessing Performance: This is an evaluation of a specific instrument, based primarily on a real-time assessment of ongoing PforR operations. Accordingly, the evaluation will not rate Bank or borrower performance – neither at the aggregate nor at the level of the individual operations – although it will take into consideration the ratings in Implementation Status and Results Reports of ongoing PforR operations. Evaluation Design and Evaluability Assessment 25. Analytical Framework: The evaluation will consist of the following activities:  Systematic assessment of approved PforR projects. This is currently expected to cover all projects approved by end FY15 – such a 100 percent sample is possible and desirable because of the small number of projects. The assessment will be undertaken on a desk review basis by the evaluation team based on a standardized approach to focus clearly on the new aspects of PforR projects and their implications (see Attachment 3). The evaluation team will meet with key team members, and create a data base to facilitate analysis of DLIs and their classification (process, outputs, interim results and outcomes).  Field visits will be carried out, tentatively to six countries with PforR operations approved in FY2012-14. Their main purpose will be to seek the views of government counterparts and relevant other donors on the various aspects of their PforRs, what works well or not so well, and how they see advantages and disadvantages compared to other Bank lending instruments. The visits will also serve as a cross-check of the desk-based assessments. While additional field visits would have been useful, we believe that visits to six countries – one per region - will give an acceptable level of cross-checking of the findings from the desk reviews, as we plan to select the country visits systematically to cover a variety of regions, sectors, and types of program issues. To the extent possible, the visits will also seek findings from the early implementation experience.  Calculation of budgetary costs of PforR operations, based on available cost data. 11   Systematic consultations within the Bank, including with focus on experiences with PforR projects not yet approved or dropped.  Literature survey of relevant material concerning the results focus for development activities, the use of country systems, and political economy issues. This will include a review of literature from/about relevant program initiatives of other multilaterals (in particular IDB and ADB) and leading bilateral aid agencies (including DfID and the Millennium Challenge Corporation).  Brief Overview of other relevant Bank approaches to support programs, results-focus and use of country systems. This will draw primarily on any available IEG evaluative material and relevant Bank documents. 26. Answering the Questions: The bulk of the evaluation work will consist of the systematic assessment of approved projects together with the field visits – and the aggregation of the project-level findings for this process evaluation. This will provide the answers to the four main evaluation questions (para 18 above). It will also validate the findings of the two-year review. 27. Sampling Strategy: It is planned to review all of the operations approved by end FY15 – at most 30 of which 25 had been approved by mid-May 2015. This review of the universe of approved projects is possible because of the relatively small number of approved operations under the review period – but the projects approved in CY15 may be assessed using a simplified template (these projects would also not have an implementation history at the time of review). The countries to be visited will be selected to ensure a mix of countries (IBRD versus IDA) and regions (one visit per region), with preference for countries with PforRs that have been effective for some time, and for countries with more than one PforR. Brazil, Nepal, Morocco, Tanzania and Vietnam are among the countries being considered. Any country with only PforRs not yet effective would not be visited. 28. Data Requirements: There will be no special data requirements for this evaluation, since most of the inputs will come from the project reviews (primarily the PADs and the underlying assessments, ISRs and aide memoires) together with information collected in the field (views of governments, counterparts and development partners), and from cost and processing data available in the Bank’s systems. 29. Design Strength and Limitations: The evaluation will cover the universe of approved projects in sufficient detail and depth. But as noted above, the evaluation cannot assess final results since none will be available. The evaluation will be cautious in trying to project final results from early indications, but may be able to provide some indications from the assessment of project designs and early implementation. The 12 absence of final results will inevitably make difficult any discussion of project effectiveness, for which reason the focus of the evaluation will be on the design and early implementation of the PforR projects. Quality Assurance Process 30. The evaluation will be overseen by Nick York, Director, IEGCC and Geeta Batra, Manager, IEGCC. Peer reviewers are Alison Evans (Incoming Chief Commissioner of the Independent Commission for Aid Impact, UK), Anis Dani (former IEG Lead Evaluation Officer), Louise Fox (Woodrow Wilson International Center and Visiting Professor at University of California at Berkeley, former World Bank staff member), and Joanne Salop (former Vice President for OPCS). The critical skills required for this evaluation include a deep understanding of lending instruments including the PforRs, macro-economic and country knowledge, substantial prior operational and evaluation experience, and in-depth knowledge on environmental and social impacts, fiduciary, anti-corruption and governance policies. The highly experienced evaluation team possesses all these skills. 31. The team will be led by Ismail Arslan (extensive country, operational and evaluation experience including as task team leader for several substantial IEG evaluations). The core team includes Monika Huppi (focus on evaluation design; extensive operational and evaluation experience with a broad range of instruments, including first-hand knowledge of experience with instruments similar to PforRs in another multilateral institution); Nils Fostvedt (focus on evaluation design and preparation of the final report; extensive operational, country and evaluation experience); Xue Li (focus on economic and portfolio analysis; advanced skills in statistical methods and in data research and portfolio analysis); Albert Martinez (focus on program assessments; broad experience in assessments of quality at entry and of supervision, regional and sector portfolio reviews and economic and sector assessments); Andres Liebenthal (focus on environmental and social aspects; extensive operational and evaluation experience with environment, infrastructure, safeguards, natural resource policies and projects, and economic and sector work); and Clay Wescott (focus on fiduciary and anti-corruption aspects; wide-ranging experience in public sector management, public financial management, capacity development, governance and anti-corruption, procurement, monitoring and evaluation, and project management). The team would be supplemented by a procurement specialist on an as- needed basis. 13  Expected Outputs and Dissemination 32. The output will be in the form of a process evaluation report for CODE and the Board. There will be separate write-ups on the individual project assessments that will be shared directly with the relevant project teams for their information and comments. The final evaluation report will be disclosed in the normal manner. The demand for special internal or external dissemination will be assessed at a later stage. Resources 33. Timeline and budget. Work on project assessments has started based on an assessment template (see Attachment 3) that has been prepared and tested. The project assessments will be undertaken during the period up to October 2015 to have room for the assessment of any projects approved in the first half of 2015. Six field visits are planned between July and December 2015. The draft report will be shared with IEG management for a one stop review meeting in March 2016 and finalization during April-May 2016. The proposed budget is $750,000 (of which $ 257,000 in FY15) including dissemination. 14 Annex 1 References IEG (Independent Evaluation Group). 2009. Do Health Sector-Wide Approaches Achieve Results? Emerging Evidence and Lessons From Six Countries. IEG Working Paper 2009/4. Washington, DC: World Bank. ____. 2011. Results-Based Health Programs in Argentina and Brazil: Project Performance Assessment Report. Report No. 62571. Washington, DC: World Bank. ____. 2014. World Bank Group Support to Health Financing. Washington, DC: World Bank. World Bank. 2011. A New Instrument to Advance Development Effectiveness: Program-for- Results Financing. December 29, 2011, OPCS, Washington, DC: World Bank. ____. 2012. Program-for-Results Financing: Interim Guidance Notes to Staff on Assessment. June 18, 2012, OPCS, Washington D.C. World Bank ____. 2014. Program-for-Results: Two-Year Review, March 17, 2015, OPCS, Washington, DC: World Bank. 15  Annex 2 Evaluation Design Matrix 1. Purpose: This evaluation will provide early feedback on how the new PforR instrument is working and its adherence to the stated intentions in the 2011 Board paper. The report will address the key questions previously raised in CODE and Board discussions, such as regarding the fiduciary and environmental and social aspects. Question Data Sources Q1: What has overall been the experience to date with the design, preparation and early implementation of PforR operations and the opportunities and challenges in that regard? - Relevance of PforR operations to Bank PADs, country strategy documents Group country programs. - Quality, strength, relevance and use of Program action plans, PADs, technical, fiduciary program action plans in relation to the and environment/social assessments, ISRs and underlying assessments. aide memoires, field visits and Bank staff interviews - Quality, relevance and use of DLIs as Program action plans, PADs, ISRs and aide incentives for strengthening the results memoires, field visits and Bank staff interviews orientation of institutions and country programs. Q2: What is the quality of the program assessments, including the technical, fiduciary, and environmental and social assessments? - Quality of fiduciary assessments including Fiduciary assessments, ISRs and aide memoires, identification of key financial and procurement field visits and Bank staff interviews issues. - Quality of environmental and social Environmental and social assessments, ISRs and assessments including identification of key aide memoires, field visits and Bank staff issues interviews - Quality of technical assessments including Technical assessments, ISRs and aide memoires, their coverage of governance issues field visits and Bank staff interviews -Quality and use of the assessment of Technical assessments, program action plans, institutional capacity. PADs, ISRs and aide memoires, field visits and Bank staff interviews - Comparison on all of these aspects between PADs, ISRs and aide memoires, field visits and PforRs and other recent projects in the same country case studies. sector and country. Q3: How effective has the PforR instrument (including policies, procedures, guidelines and their application) been in identifying, assessing and mitigating critical risks? 16  - Identification and responsiveness to fiduciary PADs, ISRs and aide memoires, field visits and risks Bank staff interviews - Identification and responsiveness to social PADs, ISRs and aide memoires, field visits and and environmental risks Bank staff interviews - Identification and responsiveness to technical PADs, ISRs and aide memoires, field visits and and other program risks including resource Bank staff interviews availability - Identification and responsiveness to PADs, ISRs and aide memoires, field visits and governance risks Bank staff interviews -Identification and responsiveness to political PADs, ISRs and aide memoires, field visits and and partnership risks Bank staff interviews - Appropriateness and use of the integrated PADs, ISRs and aide memoires, field visits and risk rating Bank staff interviews Q4: To what extent is the PforR instrument being used to strengthen national systems for financial management, procurement, environmental and social safeguards, and monitoring and evaluation? -How does the use of country systems in PADs, ISRs and aide memoires, field visits and PforRs compare with other operations in each Bank staff interviews case study country? -To what extent are PforRs ringfenced, and to PADs, ISRs and aide memoires, field visits and what extent are they subject to mainstream Bank staff interviews national systems? -Has the Bank’s analytical work provided an PADs, ISRs and aide memoires, field visits and adequate understanding of the extent to which Bank staff interviews the level of country system capacities could affect the implementation of programs? - How adequate are the provisions for building PADs, desk reviews, field visits and interviews. the programs’ institutional capacity, particularly with respect to low income countries with weaker systems 17 Annex 3 Methodology Note for Review of the Bank’s PforR Portfolio Guidance Questionnaire 1. The design of the Guidance Questionnaire (template) is based on the 2011 Board PforR Board Paper,6 BP/OP 9.0 Program-for-Results Financing, and the guidance to staff.7 It also takes into account the findings and issues raised in the 2014 PforR two- year review. The Guidance Questionnaire will enable panelists to systematically come up with an overall assessment of the PforR programs with the following measures:  Likelihood of Achieving Development Objectives (Highly Likely, Likely, Moderately Likely, Moderately Unlikely, Unlikely, Highly Unlikely); and  Integrated Risk Rating (High, Substantial, Medium, Low). 2. In addition, the following key risk categories will be rated (Low, Medium, Substantial, High):  Operating environment risks  Country risk (the report will identify which were the main factors contributing to country risk, e.g. governance, macroeconomic, security etc.)  Stakeholder risk (e.g. vested interests) 3. Program risks  Technical risk (program design and governance, institutional capacity, sustainability, and M&E arrangements and capacity)  Fiduciary risk (program fiduciary system and performance monitoring, including audit arrangements and capacity)  Environmental and social risk (potential environmental and social impacts and ability of program systems to manage these)  Disbursement-linked Indicators risk (results framework, selection of DLIs, and verification protocols)  Other risks (to be identified, if any, by reviewers) 6 World Bank. 2011. A New Instrument to Advance Development Effectiveness: Program-for-Results Financing. December 29, 2011, OPCS, Washington D.C. World Bank 7 World Bank. 2012. Program-for-Results Financing: Interim Guidance Notes to Staff on Assessment. June 18, 2012, OPCS, Washington D.C. World Bank 18  4. The above ratings for Likelihood of Achieving Development Objectives and various risks will be based on information available up to the time of the review and will be informed by the a more detailed assessment of various program components across the following dimensions, with governance as a cross-cutting theme:  Strategic Relevance  Technical Aspects  Fiduciary and Fraud and Corruption Aspects  Environment and Social Aspects  Risk Assessment  Program Implementation (for programs that are in the implementation stage) 5. An important part of the program review is an assessment of the interplay between the program and its operating environment, e.g., the program’s environment and social systems vs. the national systems. The main dimensions or sections of the Guidance Questionnaire are discussed below. The Guidance Questionnaire is attached as Annex 1, and covers the following dimensions. 6. Strategic Relevance. This dimension assesses the strategic relevance of the program from different perspectives: government, Bank corporate and relevant GP priorities. The case for government action or intervention is also examined in this dimension. In addition, the program contribution to partnership coordination is also assessed. Borrower ownership is also reviewed; for programs under implementation, there is more evidence of the degree of Borrower ownership and commitment (see also Box 1.3 of the PforR Guidance Notes for illustrative indicators for assessing Borrower commitment) based on actual implementation experience. Finally, Bank value added will be reviewed. 7. Technical Aspects. This dimension assesses the technical soundness of the program – its structure, implementation arrangements, results framework, monitoring and evaluation framework, and capacity to implement and monitor the program. This dimension also includes an assessment of: the economic evaluation of the program; the quality of the DLIs and the associated verification protocols; the quality of the Program Action Plan; and the appropriateness of the program conditionalities and the technical risk rating. The main sources of information will be the Technical Assessment as well as the actual implementation experience and ISRs for programs under implementation. 8. Fiduciary and Fraud and Corruption Aspects. This dimension assesses the fiduciary system including procurement, financial management, compliance with the Bank’s Anti-Corruption Guidelines (ACG), and capacity for implementation and 19 monitoring. With respect the ACG, reviewers will use Bank Guidelines on Preventing and Combatting Fraud and Corruption in Program-for-Results Financing in February 2012, as well as Attachment 1 of the PforR Interim Guidance Notes. The reviewers will also use actual implementation experience, where applicable, in assessing this dimension. 9. Environment and Social Aspects. This dimension assesses the program arrangements for managing environment and social effects consistent with OP/BP 9.0. Unlike in an ILF operation where the client bears full responsibility for managing all environment and social consultations with stakeholders, for PforR, the Bank is responsible for managing the consultation process for the program-specific environment and social systems (see Attachment 4.4 of the PforR Guidance Notes). The reviewers will also use actual implementation experience, where applicable, in assessing this dimension. 10. Risk Assessment. This dimension assesses the appropriateness of the integrated risk rating and the ratings for various operating environment and program risks. In addition, the impact on risk to DO of various fiduciary, environment and social exclusions are assessed. 11. Program Implementation. This dimension assesses the quality of program implementation for PforR operations that are in the implementation stage. Reviewers will use supervision documents, including ISRs, as main sources of information. In assessing program implementation, the reviewers will identify the various factors contributing to implementation issues, both program-related and those outside the scope of the program. 12. Ratings. The reviewers will use a six point rating scale (Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory) for the above dimensions. Risk ratings will follow the IEG scale (Low, Medium, Substantial, High). Panel of Reviewers 13. Each PforR operation will be reviewed by a panel consisting of the following areas of expertise: (i) the lead reviewer with expertise in the relevant GP area; and (ii) one specialist to cover fiduciary aspects, including anti-corruption; (iii) one specialist to cover environment and social aspects. Governance will be a cross-cutting area that will have to be assessed by the reviewers. 20  14. The lead reviewer will be responsible for the assessment of the operation with the specialists providing inputs in their specific areas. The choice of the lead reviewer for each operation will be dependent on the content of the operation. However, the specialist reviewers will cover several operations to ensure consistency of the assessments. 15. An IEG staff/consultant will oversee the portfolio review, including quality control of the individual assessments and consistency of basis for ratings. The IEG staff/consultant will consolidate the individual PforR program assessments into a report of findings that would include lessons learned and recommendations for improving various aspects of the PforR instrument. 16. The reviewers will have to be familiar with the PforR instrument, guidelines, and issues, in addition to having extensive operational experience. Reviewers will be provided with all relevant materials on the PforR instrument, including the PforR Board reports, BP/OP 9.0 Program-for-Results Financing, Interim Guidance Notes to Staff on Assessments, and Bank Guidelines On Preventing and Combatting Fraud and Corruption in Projects. Documentation 17. An IEG staff assistant will provide the relevant documents for the portfolio review. While most of these are in the Bank system, there may be need for getting in touch with the task team for some documents, especially correspondence that may not have been included in the document system. The main documents are the following:  Background Documents o Country Assistance Strategy (or similar instrument) o Sector Strategy (or similar GP document if available) o Poverty Reduction Strategy Paper (for IDA countries) o Relevant ESW (the staff assistant will generate the list of ESW during the past five years and the reviewers will determine which outputs are relevant, e.g., Public Expenditure Reviews) o Operational documents for operations with links to the PforR operations (the staff assistant will generate the list of Bank lending operations during the past five years and the reviewers will determine which operational documents are relevant) o Other documents (e.g. evaluations) that the reviewers may find relevant  PforR Operation Document 21 o Program Concept Note and supporting documents o Project Information Document o Management Decision Meeting minutes o Program Appraisal Document o Legal Agreements o Negotiation Minutes o Technical Assessment o Fiduciary Systems Assessment o Environmental and Social Systems Assessment o Program Action Plan o Implementation Support Plan o ISRs o Audit Reports o Relevant Communications within Bank and between Bank and Borrower Interviews with Management, Staff, Borrower, Stakeholders, and Partners 18. After a desk review and an initial assessment of the PforR program, the reviewers will identify the information gaps and determine whether interviewing specific persons is necessary, specifying the proposed agenda and questions. The interview can take the form of emails, phone calls, face-to-face meetings, or short questionnaires. A limited number of operational reviews will benefit from a country visit by a PforR team. The task manager of the PforR evaluation will approve all interviews. Quality Assurance of PforR Program Assessments 19. The IEG staff/consultant overseeing the portfolio review will be responsible for the quality of the assessments. There will be a peer review of the individual assessments; peer reviewers will be from the set of panelists used for the portfolio review. 22  Project Name and Number Lead Reviewer Fiduciary Reviewer E&S Reviewer Table 1. PforR Review Guidance Questionnaire Summary Assessment Comments Rating Likelihood of Achieving DO Overall Program Risk Assessment Dimensions Comments Rating 1. Strategic Relevance 2. Technical Aspects 3. Fiduciary and Fraud and Corruption Aspects 4. Environment and Social Aspects 5. Risk Assessment 6. Program Implementation 7. Bank Performance at Entry 8. Bank Performance during Supervision Comments Rating 1. Strategic Relevance 1.1 Program rationale 1.2 Clarity, realism, and scope of program objectives 1.3 Appropriateness of partnership arrangements with donors 1.4 Borrower ownership Comments Rating 2. Technical Aspects 2.1 Appropriateness of program structure and implementation arrangements, including Borrower capacity 2.2 Appropriateness of the program expenditure framework, including financial sustainability and funding predictability 2.3 Quality of the economic evaluation of the program 23 2.4. Quality of the results framework and M&E arrangements, including capacity 2.5 Quality of the disbursement linked indicators and verification protocols 2.6 Adequacy of Program Action Plan and conditionalities Comments Rating 3. Fiduciary and Fraud and Corruption Aspects 3.1 Adequacy of program procurement system 3.2 Adequacy of program financial management system 3.3 Adequacy of program-specific system to handle issues relating to fraud and corruption based on the Bank's Anti- Corruption Guidelines 3.4 Adequacy of stakeholder involvement in program implementation and oversight, including verification of disbursement linked indicators and establishment of a complaint and grievance mechanism 3.5 Level of transparency in program decision making processes and performance reporting 3.6 Appropriateness of measures for strengthening program's fiduciary system and governance, including in the area of anti-corruption 3.7 Appropriateness of exclusions Comments Rating 4. Environment and Social Aspects 4.1 Adequacy of program systems for managing environmental and social effects 4.2 Appropriateness of measures for improving program systems for managing environmental and social effects 4.3 Appropriateness of exclusions 4.4 Adequacy of national systems for managing and environmental and social effects of government programs 24  Comments Rating 5. Risk Assessment 5.1 Appropriateness of the program integrated risk rating 5.2 Appropriateness of technical risk rating 5.3 Appropriateness of fiduciary risk rating 5.4 Appropriateness of environment and social risk rating 5.5 Appropriateness of DLI risk 5.6 Appropriateness of other program risk(s) if any 5.7 Appropriateness of country risk 5.8 Appropriateness of stakeholder risk 5.9 Appropriateness of risk mitigation measures 5.10 Adequacy of assessment of impact on risk to DO of fiduciary, environment, and social exclusions, if applicable Comments Rating 6. Program Implementation 6.1 Quality of overall program implementation 6.2 Quality of implementation of fiduciary systems 6.3 Quality of implementation of environment and social systems 6.4 Quality of implementation of Program Action Plans 6.5 Quality of partnership engagement during program implementation 6.6 Appropriateness of stakeholder engagement and program transparency during implementation Notes: 25 Annex 4 Outline of Evaluation Report Outline of Evaluation Report Executive Summary Background and Evaluation Approach Results-based lending before the PforRs – inside and outside the World Bank Overview of Progress  The PforR Portfolio – Status and Trends  World Bank Engagements (relations to other lending instruments and country programs, relations to previous and parallel operations, packages of preparation and implementation support, resource requirements) Experience with Key Features of PforR Operations  Technical assessments  Fiduciary assessments including procurement  Environmental and social assessments  Capacity building  Results frameworks and DLIs  Program Action Plans  Partnerships  World Bank additionality Key Issues, Conclusions and Findings Recommendations Annexes to include:  PforR portfolio  Program analysis  Summaries from country visits and partner consultations. 26  Attachment 1 Management Comments on IEG Approach Paper: Program for Results (PforR) – IEG Process Evaluation Management thanks the members of the Committee on Development Effectiveness (CODE) and the Independent Evaluation Group (IEG) for the opportunity to contribute to CODE’s discussion of the IEG Approach Paper Program for Results (PforR). In addition to direct interactions with IEG, management would like to share the following points with the members of the Committee on Development Effectiveness. First, IEG’s process evaluation will add to our collective knowledge about how the PforR instrument is working in practice, including the design and implementation of PforR operations in a broad range of countries and sectors. It is almost a year since much of the analysis for the PforR Two-Year Review was completed, which included desk reviews of each of the dimensions of PforR, internal and external surveys, and structured interviews with government officials and senior Bank managers. In this context and in the spirit of the adaptive learning approach to PforR endorsed by CODE, management believes that further valuable lessons can be provided through IEG’s process evaluation. Second, management believes that it is important to be realistic with respect to the objectives and expectations for the evaluation and to ensure that any objectives can be adequately supported by evidence. As the approach paper notes, “This will be a real-time process evaluation of an instrument for which there are, as yet, no completed operations and thus no available IEG-validated data on final outcomes, and only partial data on disbursements, results and outputs.” Indeed, there has been a steady increase in the number of PforR operations since the inception of the instrument; all regions have now approved at least two operations, and PforR operations cover most of the key sectors in which the Bank traditionally provides financing. Nonetheless, as of early June 2015 there were still only 27 approved PforR operations, and no PforR operation is expected to be completed before June 2016. In this context, management believes that it is very important that the process evaluation is careful to avoid coming to premature conclusions that cannot be supported by adequate evidence at this stage, for example, with respect to the “key parameters for appropriate, successful PforR operations” and PforR’s “additionality ….compared to other existing lending instruments” (para 19 of the approach paper). Finally, management notes the importance of grounding the evaluation in a thorough understanding of the nature and dimensions of the PforR instrument. This is not easy, especially given the newness of the instrument and the fact that relatively few people have direct experience with it. During the PforR Two-Year Review, management received clear feedback from a broad range of stakeholders, including in client countries, as well as from Bank managers and staff, of the desire to learn from teams with direct PforR experience in order to better understand the instrument. In these circumstances it is important that the IEG core team has the opportunity for a full exchange of knowledge and perspectives with Bank staff and counterparts in country who have worked directly on the instrument, and management stands ready to work with IEG to facilitate such interactions. 27