72606 IEG Working Paper 2012/1 Comparison of the Monitoring and Evaluation Systems of the World Bank and the Global Fund Cheryl Cashin © 2012 Independent Evaluation Group, The World Bank Group 1818 H St., NW Washington, DC 20433 1 2 3 4 5 14 13 12 IEG: Improving World Bank Group Development Results Through Excellence in Evaluation The Independent Evaluation Group is an independent unit within the World Bank Group; it reports directly to the Bank’s Board of Executive Directors. IEG assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country’s overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. IEG Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation . The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent, or IEG management. IEG cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. ISBN 13: 978-1-60244-216-0 ISBN 10: 1-60244-216-9 Contact: IEG Communication, Learning and Strategies E-mail: ieg@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http://ieg.worldbankgroup.org i Contents Preface...................................................................................................................................... iv Executive Summary .................................................................................................................. v 1. Introduction and Objectives .................................................................................................. 1 2. World Bank Approach to M&E ............................................................................................ 3 Objectives of M&E ............................................................................................................... 4 Framework and Indicators for Project M&E ........................................................................ 4 M&E Implementation and Utilization .................................................................................. 6 Evaluation ............................................................................................................................. 9 Link Between Project M&E and National Systems ............................................................ 12 3. Global Fund Approach to M&E ......................................................................................... 14 Objectives of M&E ............................................................................................................. 14 Framework and Indicators for Global Fund M&E ............................................................. 15 M&E Framework and Indicators at the Global Fund Level ........................................... 15 M&E Framework and Indicators at the Country Level .................................................. 17 M&E Framework and Indicators at the Project/Grant Level .......................................... 19 M&E Implementation and Utilization ................................................................................ 23 Evaluation ........................................................................................................................... 26 Link Between Project M&E and National Systems ............................................................ 28 4. Country-Level Comparison of the M&E Design, Implementation, and Utilization .......... 29 Burkina Faso ....................................................................................................................... 29 World Bank: HIV/AIDS Disaster Response Project ...................................................... 29 Global Fund: Project for Enhancement of HIV/AIDS Control ...................................... 30 M&E Comparison ........................................................................................................... 30 Lesotho ................................................................................................................................ 32 World Bank: HIV and AIDS Capacity Building and Technical Assistance Project (HCTA) ........................................................................................................................... 32 Global Fund: Strengthening Prevention and Control of HIV/AIDS in Lesotho ............. 32 M&E Comparison ........................................................................................................... 32 Russian Federation .............................................................................................................. 33 World Bank: TB/AIDS Control Project .......................................................................... 33 Global Fund: Stimulating an Effective National Response to HIV/AIDS in the Russian Federation ....................................................................................................................... 34 M&E Comparison ........................................................................................................... 34 ii 5. Comparison of the World Bank and Global Fund M&E Approaches ................................ 35 Objectives and Approach .................................................................................................... 35 Design, Implementation, and Utilization ............................................................................ 38 Evaluation ........................................................................................................................... 38 Link to National M&E Systems .......................................................................................... 40 6. Conclusions and Lessons Learned ...................................................................................... 41 References ............................................................................................................................... 44 Figures Figure 1. World Bank M&E Traceable Pathway from Objectives to Conclusions about Effectiveness ................................................................................................................. 4 Figure 2. Global Fund Four-Tiered Corporate Performance Management Framework ......... 16 Figure 3. Conceptual Framework for Global Fund Country Impact Evaluations................... 19 Figure 4. Global Fund Project/Grant M&E Requirements ..................................................... 20 Figure 5. Selected Ratings of World Bank-Financed HNP Projects, in Comparison with Bank-wide Averages ................................................................................................... 39 Tables Table 1. Sample World Bank Project Results Framework: Burkina Faso HIV/AIDS Disaster Response Project ............................................................................................. 5 Table 2. Performance Indicators in the Lesotho 2004 Global Fund Grant Agreement .......... 21 Table 3. Global Fund Recommended ―Top Ten‖ Grant Performance Indicators................... 22 Table 4. Summary of World Bank and Global Fund M&E Comparison ............................... 37 Appendixes Appendix A. Stakeholder Interviews: Interview Guide .......................................................... 51 Appendix B. World Bank Stated Policy on Monitoring and Evaluation ................................ 52 Appendix C. Global Fund 2004 Monitoring and Evaluation Operational Plan ...................... 54 Appendix D. Burkina Faso M&E Comparison ....................................................................... 62 Appendix E. Lesotho M&E Comparison ................................................................................ 68 Appendix F. Russia M&E Comparison .................................................................................. 72 Appendix G. Selected Ratings of World Bank-Financed HNP Projects, in Comparison with Bank-wide Averages ........................................................................................... 76 iii Abbreviations AIDS Acquired immunodeficiency syndrome ARV Antiretroviral drug CBO Community-based organization CCM Country Coordination Mechanism (Global Fund) DOTS Directly Observed Treatment Short-Course (for tuberculosis) FYE Five-Year Evaluation of the Global Fund GAMET Global HIV/AIDS Monitoring and Evaluation Support Team GSC Grant Scorecard (Global Fund) GPR Grant Performance Report (Global Fund) HIV Human immunodeficiency virus HNP Health, nutrition, and population ICR Implementation Completion and Results Report (World Bank) IEG Independent Evaluation Group, formerly OED (World Bank) ISR Implementation Status and Results Report (World Bank) KPI Key performance indicators (Global Fund) LFA Local Fund Agent (Global Fund) M&E Monitoring and evaluation MDGs Millennium Development Goals MDR-TB Multidrug-resistant Tuberculosis MOH Ministry of Health in client countries of the Global Fund MTR Mid-Term Review (World Bank) NGO Nongovernmental organization PBF Performance-based funding (Global Fund)) PAD Project Appraisal Document (World Bank) PIU Project Implementation Unit PPAR Project Performance Assessment Report (IEG) PSC Policy and Strategy Committee (Global Fund) PSR Project Status Report (World Bank) PUDR Progress Update and Disbursement Request (Global Fund) QER Quality Enhancement Reviews (World Bank) SIM Strategic Information and Measurement (SIM) Team (Global Fund) TB Tuberculosis TERG Technical Evaluation Reference Group (the Global Fund) UNAIDS Joint United Nations Program on HIV/AIDS UNDP United Nations Development Programme WHO World Health Organization Fiscal Years Global Fund: January 1 – December 31 World Bank: July 1 – June 30 Director-General, Evaluation Ms. Caroline Heider Director, Public Sector Evaluations Mr. Emmanuel Y. Jimenez Manager, Public Sector Evaluations Mr. Mark Sundberg Global Programs Coordinator and Task Team Leader Mr. Chris Gerrard Consultant Ms. Cheryl Cashin iv Preface The Independent Evaluation Group (IEG) has recently completed a review, The Global Fund to Fight AIDS, Tuberculosis and Malaria and the World Bank’s Engagement with the Global Fund, which focuses on the engagement between the World Bank and Global Fund- supported activities at the country level. This comparison between the monitoring and evaluation systems of the two organizations represented an input into that larger review. The purpose of this study is to document the approaches of the two organizations to monitoring and evaluation (M&E), to systematically and objectively compare the principles and objectives of their M&E systems, and how these systems are implemented and used in practice at the country level. The report also discusses the relationship of the M&E systems to the two different business models of the World Bank and the Global Fund. The study methods involved an intensive desk review of World Bank and Global Fund policy documents related to monitoring and evaluation, and analysis of actual M&E products of both agencies. The study also includes an in-depth comparison of M&E implementation by the two agencies in three countries: Burkina Faso, Lesotho, and Russia. Similar World Bank health projects and Global Fund grants were implemented concurrently in these countries and have reached completion. The study was prepared by Cheryl Cashin under the supervision of Chris Gerrard. The draft was peer-reviewed by Judy L. Twigg, Associate Professor at the Wilder School of Government and Public Affairs, Virginia Commonwealth University, and by Martha Ainsworth, Adviser, IEG Public Sector Evaluation. v Executive Summary 1. IEG has undertaken a detailed comparison of the project-level monitoring and evaluation (M&E) systems of the Global Fund to Fight AIDS, Malaria and Tuberculosis (Global Fund) and the World Bank (a) to identify whether and how the findings and conclusions that emerge from the two organizations‘ M&E systems can be compared, and (b) to contribute to the ongoing process of identifying good practices for project-level M&E. The comparison is based on actual experience in three countries: Burkina Faso, Lesotho, and the Russian Federation. Burkina Faso and the Russian Federation were chosen from the six countries visited for IEG‘s overall Review of the Global Fund because of the existence of World Bank-supported projects and Global Fund grants with similar objectives being implemented during roughly the same time period in these countries, thereby enabling a comparison of project-level M&E in the two organizations. Lesotho was chosen for the same reason and because IEG has recently completed a Project Performance Assessment Report of the World Bank project that was specifically intended to increase the capacity of Lesotho ―to use effectively the resources provided through the Global Fund grant to support the implementation of HIV and AIDS programs‖ in Lesotho. 2. Grant-level M&E in the Global Fund is specifically tied to its Performance-Based Funding (PBF) system. Grants are initially approved for two years (Phase 1) and renewed for up to three additional years (Phase 2), based on the performance of the grant-funded activities. Tied to PBF are detailed and documented requirements and outputs for grant-level monitoring. Each grant agreement contains a disease-specific performance framework outlining the performance expected over the lifetime of the grant and containing key indicators and targets that are used to measure outputs and coverage on a routine basis. Funding is disbursed incrementally every three to six months throughout the life of the grant. The Principal Recipient prepares Progress Update and Disbursement Requests (PUDRs), which link the historical and expected program performance with the level of financing to be provided to the Principal Recipient. The Local Fund Agent (LFA) reviews these periodic requests for funding, undertakes site visits to verify results, reviews the Principal Recipient‘s audit reports, and then makes a confidential recommendation to the Global Fund Secretariat to disburse (or not to disburse) the funds. When the initial two-year grant commitment period is completed, the Country Coordinating Mechanism (CCM) requests further funding for the remaining three years of the approved grant. The LFA again reviews these requests before the Global Fund Secretariat instructs the trustee to release additional funds. 3. Project-level M&E in the World Bank aims to create a traceable pathway from a project‘s intent and objectives to inputs and activities, to performance against indicators, and ultimately to conclusions about effectiveness—both by the project team and by independent evaluators. This includes an assessment of the Bank‘s own performance and that of the borrower, in addition to the outcome of the project as a whole. A results framework, which describes the pathway from project activities to intermediate outcomes and ultimately to the project development objective, is a required annex in the Bank‘s project appraisal documents. 4. IEG found that both World Bank projects and Global Fund grants in the three countries suffered from weak M&E design at the beginning of the projects/grants. There was vi a particular problem regarding performance indicators. Typically there were too many indicators, they lacked validity, and they often did not fit into a logical framework of inputs, outputs, outcomes, and impacts. The Global Fund has attempted to address the inadequacy of performance indicators by developing a set of ―Top Ten‖ indicators that it recommends to its grantees, but these indicators often were not routinely available in the countries. 5. Neither the World Bank nor the Global Fund was successful at identifying data sources up front. The indicators relating to outcomes and impacts were difficult to report due to inadequate data sources in the countries. In general, the performance indicators provided little added value for assessing project/grant performance, for contributing to periodic summative evaluations, or for enhancing policy dialogue. Good monitoring systems do all three—assess progress in implementing activities, facilitate a cumulative assessment of project performance, and identify issues that require policy responses and other solutions beyond the scope of the projects. 6. Both the World Bank and Global Fund M&E products were more useful when they were supplemented with other analysis and when results were synthesized and interpreted more broadly. In the projects and grants reviewed, this was done more frequently in World Bank projects. There were also examples of more analytical M&E in Global Fund grants (for example, the Russian Central Public Health Research Institute database used for M&E of the HIV/AIDS grant). 7. The World Bank aims to overcome some of these deficiencies in project monitoring with a standardized evaluation process that combines internal self-evaluation and independent review of individual projects. Each project team undertakes a self-evaluation at the completion of every project using a standardized Implementation Completion and Results Report (ICR) submitted to the Bank‘s Board within six months of the project closing date. Project M&E data, performance-related reports, and other relevant operations documentation provide input into the ICR. The performance of the project is assessed against standard criteria. Then IEG undertakes an independent review of all completed projects and their ICRs using a standardized desk review that assesses both the project experience, based on information in the ICR, and the quality of the self-evaluation. 8. An emphasis on learning from implementation has led to a World Bank culture of acceptance of critical evaluations. The overall outcomes of 38 percent of Bank-financed health, nutrition, and population (HNP) projects approved since 1997 have been rated moderately unsatisfactory or worse. The traceable pathway in the World Bank‘s M&E system from project inputs/activities to outcomes made it possible for IEG to complete its 2009 evaluation of the World Bank Group‘s support to HNP based on cumulative self- evaluations and independent reviews of individual project outcomes. The conclusions of the evaluation reflected the aggregate performance of projects, which did not lend itself to reinterpretation and subjective conclusions. Given the real challenges that have been faced by the complex nature of World Bank HNP projects in challenging environments, and the willingness to rate projects as unsatisfactory, the evaluation, based on cumulative project performance, was unable to paint an overly positive picture. vii 9. By contrast, evaluation at the Global Fund has a conspicuous gap—the lack of an evaluation at the completion of individual grants. There has been no policy or process until recently within the Global Fund M&E system to determine the overall effectiveness of individual grants, or to generate lessons for future Global Fund activities in the country or in other programs. There was also no contribution of the grant-level M&E of Global Fund grants to the summative assessment in Study Area 3 of the Five-Year Evaluation. The FYE was an independent and quality evaluation, but it was constrained by the absence of assessments of the outcomes of individual grants, both because there was no framework in place to do so and because few grants had been completely implemented at the time of the FYE. Therefore, the FYE was based on other information, studies, and analysis, including the 16 country studies for Study Area 2 and the 18 country studies for Study Area 3. The lack of a framework and cumulative assessment of grant performance made it possible to draw conclusions—both positive and negative—about the overall efficacy of Global Fund grants that were not necessarily supported by objective criteria. 10. The World Bank, the Global Fund, and other multilateral organizations have expressed good intentions to coordinate and streamline M&E processes at the country level. They have endorsed the Three Ones principles and they have jointly prepared an M&E Toolkit in 2004 (revised in 2006, 2009, and 2011) to establish norms and identify indicators to be used by all the agencies. In terms of developing frameworks and identifying indicators, there has been some progress. The approach and the indicators in the M&E Toolkit make a lot of sense, but these have been difficult to achieve in practice because each agency has its own project-level M&E requirements, which often provide very little value for program assessment, program management, or policy dialogue. Achieving the third ―One‖—one county-level M&E system in each disease area—is also dependent on achieving the first two ―Ones‖—a common action framework with a single coordinating authority. 11. Both the Global Fund and the World Bank could contribute to improved M&E at the project and country levels by making a stronger commitment to the Three Ones principles. Then project-level M&E—including the Global Fund‘s PBF approach to disbursements— could focus on accountability for achieving the specific outputs of each project, and country- level M&E could focus on tracking the higher-level outcomes and impacts collectively. To build the knowledge base about which approaches most successfully contribute to achieving collective outcomes, the Global Fund could also consider undertaking evaluations of a random sample of the single streams of funding now taking place under its ―new grant architecture‖ and institutionalizing regular country-level evaluations, both of which could feed into subsequent evaluations of the overall program. The World Bank should continue to provide technical assistance to strengthen national M&E capacity through components of health projects and through the Global HIV/AIDS Monitoring and Evaluation Support Team, as it has done in the past. 1 1. Introduction and Objectives 1.1 The massive scale-up in global health investments over the last decade has been accompanied by a call for more serious attention to monitoring and evaluation of the effectiveness of these investments. The use of results-based financing by major global donors has further increased the demand for more systematic use of data and monitoring tools to strengthen program management and improve accountability (Chan and others 2010; The Lancet 2010). In spite of the consensus around the urgent need for valid, reliable assessments of the effectiveness of health inputs and programs, current approaches to health sector monitoring and evaluation (M&E) remain inadequate. Impact evaluations using rigorous methods are rare, and monitoring systems often have failed to provide timely information in a useful format with transparent processes for feeding results into project design, management and scale-up (Center for Global Development 2006). 1.2 The World Bank and the Global Fund to Fight AIDS, Tuberculosis and Malaria (Global Fund) have been at the forefront of incorporating systematic monitoring and evaluation into their global health activities. Both agencies have established M&E requirements for their projects and grants, and prepared detailed guidelines for each major component of their M&E systems. There are differences, however, in the way M&E systems are implemented across the two agencies, and how M&E results are produced and used. These differences reflect both divergence across the two agencies in their objectives for M&E and their contrasting underlying business models. 1.3 The different approaches to M&E by the World Bank and Global Fund have had consequences on the ground in countries that receive support from both agencies. The policies of both agencies explicitly call for coordinating and streamlining M&E activities across international agencies and programs and with national efforts, which is reiterated in project and grant preparation documents. Nonetheless, duplication and added reporting burdens have been difficult to avoid in practice (Biesma and others 2009; Atun and others 2010). 1.4 The different approaches to M&E by the World Bank and Global Fund also have led to some conflicting conclusions about the overall effectiveness of the global health investments of these agencies. For example, IEG‘s 2009 evaluation of the World Bank Group‘s support to Health, Nutrition, and Population (HNP; IEG 2009a) and the Five-Year Evaluation (FYE) of the Global Fund were published at about the same time in 2009. Although both evaluations pointed out successes and failures, the interpretations of the results of the two evaluations in the media were dramatically different. The media generally interpreted IEG‘s evaluation as critical of the World Bank‘s performance—based in part on flat trends in IEG outcome ratings for HNP projects compared with projects in other sectors, but particularly based on the poor performance of HIV/AIDS projects (Faiola 2009). On the other hand, the FYE was interpreted as complimentary of the Global Fund‘s performance. But the different approaches of the two agencies to M&E make such comparisons of their conclusions in relation to overall effectiveness potentially misleading. 1.5 The purpose of this study is to document the approaches of the World Bank and the Global Fund to M&E, and to systematically and objectively compare the principles and 2 objectives of the M&E systems and how these systems are implemented and used in practice at the country level. The report also discusses the relationship of the M&E systems to the two different business models of the World Bank and the Global Fund. The first goal is to identify whether and how the conclusions that emerge from their M&E systems on the effectiveness of their respective global health activities can be compared. The second goal is to contribute to the ongoing process of identifying good practices for developing M&E policies for global health programs, setting up M&E frameworks, planning and programming evaluations, and using M&E results more effectively to manage programs and strengthen the health policy process in partner countries. 1.6 The study methods involved an intensive desk review of World Bank and Global Fund policy documents related to monitoring and evaluation, analysis of actual M&E products of both agencies, and an in-depth comparison of M&E implementation in three countries (Burkina Faso, Lesotho, and Russia) where a similar World Bank health project and Global Fund grant were implemented concurrently and have reached completion. The in- depth comparison was based on a desk review, which was intended to be supplemented by stakeholder interviews (Appendix A). The stakeholder interviews were limited, however, by the difficulty of locating and engaging key individuals long after projects and grants had closed.1 1.7 The report is organized as follows. Section 2 summarizes the World Bank‘s stated policies and approach to monitoring and evaluation. Each element of standard M&E systems—framework; system of indicators and performance measurement; data collection and analysis; feedback and use of monitoring findings; and evaluation—is described for the World Bank‘s approach in this section. Section 3 summarizes the Global Fund‘s stated M&E approach and policies according to the same structure. Section 4 compares the application of the approach to M&E of the two agencies in Burkina Faso, Lesotho and Russia. Section 5 provides a summary of the comparison between the two approaches to M&E. Section 6 identifies conclusions and lessons learned. 1. Although ten individuals were contacted in the three countries for this study, only one stakeholder interview was completed. 3 2. World Bank Approach to M&E 2.1 Throughout its history, the World Bank has made monitoring and evaluation an integral part of its governance and operations. As an example, the Bank‘s Independent Evaluation Group celebrated its 30th anniversary in 2003. In the past decade the focus on results-oriented assistance has figured even more prominently, and has driven changes in the business model of the Bank from a project orientation to the country program approach (World Bank 2004). In 2002 the Bank issued a conceptual framework for managing for results, and the Results Secretariat was established within the Operations Policy and Country Services (World Bank 2002). Results-based M&E was identified as a key aspect of good governance both for planning and managing for results and for improving government accountability and transparency. This sharper focus on results has raised the profile of M&E throughout the Bank and has brought to light the real challenges of achieving effective M&E implementation in capacity-constrained countries. 2.2 The World Bank has an objectives-based approach to M&E, which relies on a combination of monitoring, self-evaluation, and independent evaluation. Project-level M&E provides the foundation for higher-level aggregate evaluations. The objective is to create a traceable pathway from a project‘s intent and objectives to inputs and activities, to performance against objectives using indicators and all available evidence, and ultimately to conclusions about the project‘s overall outcome—both internal by the project and external by independent evaluators, and both at the project level and more broadly at the country or sector level and beyond. 2.3 The Bank‘s M&E approach includes an assessment of the Bank‘s own performance and that of the borrower in addition to the outcome of the project as a whole. This approach to M&E stems from the World Bank‘s business model of serving not only as a lending institution, but also as a technical assistance institution that supports its borrowers in the implementation and supervision of its projects. In such a role, the World Bank not only needs to know if its investments are effective, but it also needs to know why and how. 2.4 To support the traceable pathway from inputs to impacts (Figure 1), the main building blocks of World Bank project M&E approach are: ï‚· Clearly articulated statement of objectives, reflected in the design documents and lending agreements ï‚· Results framework with output and outcome indicators capable of measuring the results chain leading to achievement of the objectives, specified during project design ï‚· Regular supervision and supervision reports ï‚· Self-evaluation by the managing units: Implementation Completion and Results Reports (ICR) completed within six months of project closing ï‚· Independent validation of the ICRs by the Independent Evaluation Group (ICR Reviews) and independent field evaluations of about one in five projects: Project Performance Assessment Reports (PPARs) ï‚· Project evaluations also feed into higher level evaluations, including country-level and sector-level evaluations, as well as meta-synthesis evaluations. 4 Figure 1. World Bank M&E Traceable Pathway from Objectives to Conclusions about Effectiveness Intent and Project Inputs Performance Against Conclusions about Objectives and Activities the Project Performance (based on Development relevance, efficacy, ï‚· Statement of ï‚· Implementation Objectives efficiency) objectives and of project results components ï‚· Regular supervision ï‚· ICR (self-evaluation) framework and activities reports (same as ï‚· ICR Reviews during project ISRs) (independent validation) design with ï‚· Implementation ï‚· PPARs independent evaluation output and status reports ï‚· Country-level and outcome ï‚· Mid-term review sector-level evaluation indicators ï‚· Meta synthesis evaluations Source: IEG. Objectives of M&E 2.5 According to the World Bank‘s stated M&E policy (Appendix B), the Bank monitors and evaluates its operational activities with the objective of ―assessing the extent to which the Bank‘s inputs together with the borrower‘s efforts assist its borrowing member countries, individually and collectively, to reduce poverty and achieve sustainable growth‖ (World Bank 2007a). The World Bank policy also states that M&E provides information to verify progress toward and achievement of results, supports learning from experience, and promotes accountability for results. Framework and Indicators for Project M&E 2.6 The framework guiding project monitoring and evaluation is developed during the project design phase. At the time of project preparation, the Team Leaders of new World Bank projects are required to draft a Project Appraisal Document (PAD) outlining the strategic context and rationale of the project, project description, implementation arrangements, and appraisal summary. According to the guidelines, the PAD should specify the principal outcomes for the primary target group as well as project-related outcomes for each project component, and how progress toward achieving the project outcomes will be measured (World Bank, undated-c). 2.7 A results framework, which describes the pathway from project activities to intermediate outcomes and ultimately to the project development objective, is a required annex of the PAD. An example of a World Bank project results framework is provided in Table 1. 2.8 According to the PAD guidelines, the results framework should address the following: ï‚· Where will the data for the project‘s outcome and results indicators come from? ï‚· Where will the capacity and responsibility for collection of indicator data and analysis of results be located? Do capacities have to be strengthened? If so, how? ï‚· What additional costs are required, if any? 5 Table 1. Sample World Bank Project Results Framework: Burkina Faso HIV/AIDS Disaster Response Project Project Development Objective Project Outcome Indicators/Targets To assist the government in implementing the 2001–05 1: Decrease the % of non-married individuals who report HIV/AIDS strategic plan, in order to slow the spread and having 2 or more sexual partners [females; males] mitigate the impact of the HIV/AIDS epidemic. Specific objectives are to: 2: Increase % of men who report having used condom in (a) Scale up, expand and improve preventive activities in the past 12 months an effort to lower the risks of transmission; 3: Increase by 20% condom utilization rates at previous (b) Strengthen capacity to provide care, treatment and high-risk sexual contact [females (15–45); males (15–45); support to those infected or affected by the epidemic; and females (15–24); males (15–24); female sex workers; (c) Mitigate the socio-economic impact on affected truckers; miners] households and communities 4: Decrease by 20% the # of persons living with HIV/AIDS and have had declared cases of discrimination and stigmatization. Intermediate Outcomes Intermediate Outcome Indicators 1: Increase the percentage of sexually active individuals 1: Increase the percentage of sexually active individuals who know they can avoid contracting HIV by using a who know that they can avoid contracting HIV by using a condom. condom [females; males] 2: Increase the number of pregnant, HIV positive women 2: # of pregnant women sero-positive who receive who receive treatment to prevent mother-child transmission treatment to prevent mother-child transmission 3: Increase the number of HIV infected people receiving 3: Number of HIV infected people receiving ARV treatment. ARV treatment. Outputs Output Indicators 1: Increase percentage of sexually active individuals who 1: Increase percentage of sexually active individuals who know that a person who looks well can be infected by HIV know that a person who looks well can be infected by HIV [females (rural/urban); males (rural/urban)] [females (rural/urban); males (rural/urban)] 2: Increase by 20% the number of sexually active 2: # of sexually active population receiving voluntary testing population receiving voluntary testing and counseling. and counseling. 3: Increase by 20% the number of infected people treated 3: # of infected people treated for opportunistic infection in for opportunistic infection in participating provinces participating provinces 4: Increase by 20% the number of people living with 4: # of people living with HIV/AIDS who are receiving home- HIV/AIDS who are receiving home-based care in based care in participating provinces participating provinces 5: Increase by 25% the number of communities providing 5: # of communities providing orphan care and support in orphan care and support in participating communities participating communities 6: Increase by 25% the number of orphans who receive 6: # of orphans who receive care and support (ministries; in care and support (ministries; in 13 provinces; national) 13 provinces; national) 7: Increase the number of villages and Community-based 7: # of villages and Community-based Organization (CBO) Organization (CBO) carrying out sub-projects carrying out sub-projects 8: Increase the number of HIV/AIDS awareness activities 8: # of HIV/AIDS awareness activities carried out by carried out by Ministries. Ministries. Source: World Bank (2007b). Note: This framework was substantially modified between the original Results Framework in the PAD, which included 34 indicators, and the ICR. 6 ï‚· What mechanisms will allow the indicators to be used by managers and policy- makers to assess the project‘s effectiveness during implementation and after completion 2.9 Results frameworks included in PADs should be assessed through Quality Enhancement Reviews (QERs) prior to project appraisal (World Bank, undated-c). The QER is intended to provide quality assurance on the technical aspects of project design, and specifically on how Bank-financed inputs can be expected to translate into results and outcomes.2 2.10 In practice, some difficulties with M&E systems in World Bank projects can already begin to emerge at this stage. Project M&E systems are evaluated in the ICR at the completion of the project and are often found lacking already in the design phase. The process for developing the results framework is not documented, and although the borrower is involved it is not clear to what extent the full range of country implementers is involved in selecting indicators and take ownership of the framework. Performance indicators are not always well defined, and data sources are rarely identified in the Results Framework. 2.11 Looking back from several ICRs to the original PAD also shows that indicators can be dropped or changed multiple times throughout the life of the project, and the reason is not always documented. Indicators may change if the objectives of the project change or if the key outcome targets change (for example, in the Russia TB/AIDS Control Project). In both cases the project has to be restructured with Board approval. In other cases, however, indicators may be changed without major changes in the structure of the project. In the Burkina Faso HIV/AIDS Disaster Response Project, for example, the initial PAD included 34 indicators. Two of the four outcome indicators included in the original credit agreement were eliminated from the amended credit agreement, and only partial data were available for the remaining two. The ICR found that indicators were selected and eliminated ―haphazardly‖ throughout the project period (World Bank 2007b). M&E Implementation and Utilization 2.12 M&E implementation involves data collection and management, and M&E utilization involves the feedback and use of monitoring findings. There are no established policies or guidance to improve these critical aspects of M&E systems in World Bank projects. The ICR guidelines specify that M&E design, implementation and utilization should be assessed and ratings applied, but there are no clear criteria for this assessment and it appears to be dealt with in an ad hoc way in the ICRs. 2.13 From the review of M&E documents from the three projects examined for this study, one of the main problems with M&E implementation and utilization stems from poor data availability. The Results Framework in the PAD does not consistently identify known data sources for performance indicators. Throughout implementation data sources and flows are weak and often non-existent for a significant share of the indicators. The 2009 IEG evaluation of World Bank support for health, nutrition, and population (HNP) found that 2. However, the QERs are not part of a standard World Bank project cycle and are not applied universally. 7 fewer than half of approved projects had baseline values for any of the outcome indicators in 2007 (IEG 2009a). This is a general problem of inadequate health information systems in the borrowing countries, which may need to be addressed directly through investment and technical support. 2.14 When the indicators are reported, data sources are typically not provided, so it is difficult to assess the validity of the information. In the Russia TB/AIDS Control Project, for example, the data sources for key indicators such as the incidence of new TB cases and treatment success rates are not given, and although the ICR states that the M&E system for the project was guided by the World Health Organization (WHO), the values are different from those reported by the WHO for the years the project was active (World Health Organization 2009). Using the WHO figures would give a different interpretation of the achievements of the project, with new case notification and treatment success rates actually declining. 2.15 Another potential weakness in the M&E implementation and utilization is the type of monitoring products produced by the Bank. The main monitoring processes and outputs are the Implementation Status and Results Report (ISR) and the Mid-Term Review (MTR), which are completed by Task Team Leaders for World Bank projects during supervision missions in conjunction with the local project implementing unit and other technical team members as needed. 2.16 The ISR is a standard report completed every six months, with ratings given to the different components of the project, as well as an overall rating of progress toward the development objective. The ISR replaced the original monitoring instrument, the Project Status Report (PSR), because the PSR was found to be ―not fully effective in reporting on implementation performance during [project] supervision‖ and ―few line managers used the PSR as a management tool because it did not report adequately on project outcomes‖ (IEG 2006). 2.17 The ISR is a more streamlined document than the PSR, but it can also be criticized as being more of a World Bank bureaucratic step than a useful tool to the Bank or the borrower for day-to-day project management and policy dialogue. The text of the report only addresses ―critical issues and actions to be brought to the attention of [Bank] management‖ (World Bank, undated-a). Based on the ISRs from the three projects reviewed for this study, project ratings in the ISR appear to be more driven by disbursement progress than achievement of intermediate outcomes, and the M&E indicators are not consistently used in assigning project ratings. The ISRs for the Lesotho HIV and AIDS Capacity Building and Technical Assistance Project, for example, focused almost exclusively on disbursement progress in upgrading or downgrading the project ratings given in the ISRs. The ISRs for the Burkina Faso HIV/AIDS Disaster Response Project assessed progress and provided ratings based on broader outcomes (for example, epidemic stabilization and access to treatment), although the M&E indicators identified in the PAD were unavailable for most of the life of the project and did not contribute to monitoring assessments. 2.18 The ISRs also appear to vary in the objectivity of the self-ratings. The ISRs (and previously PSRs) for the Russia TB/AIDS Control Project consistently awarded the M&E 8 component of the project a satisfactory rating in several early PSRs, although no data for the performance indicators were available until after two years of implementation. In the Lesotho HIV and AIDS Capacity and Technical Assistance Project, on the other hand, delays in procurement during a six-month period resulted in immediate downgrading the project rating on several occasions. The M&E component, however, was consistently awarded a moderately satisfactory rating throughout the life of the project without justification in any of the ISRs. 2.19 An MTR is completed for each project jointly by the World Bank and the borrower. There are no guidelines for conducting a MTR, and the process for carrying out the review appears to be unstructured. The MTR is process-oriented and typically involves stakeholder meetings, field visits, and/or public presentation and discussion of the project status and results (World Bank 2006b, 2006c). 2.20 The systematic use of monitoring results in World Bank projects is difficult to assess. Although the PAD Results Framework includes a statement about how the results of each monitoring indicator will be used, this is not documented in practice in the course of implementation. The ISRs seem to be most successful at identifying and documenting implementation and disbursement bottlenecks. Use of the ISRs for broader policy and program purposes is less apparent. For example, there was no discussion in the Russia TB/AIDS Control Project ISRs about possible reasons for the increase in the case fatality rate for new TB cases during the life of the project. 2.21 The MTR is more process-oriented and participatory, offering the opportunity to analyze project progress more deeply, using multiple sources of data and evidence from implementation to date, and to engage a variety of stakeholders in policy dialogue. It seems that together, the standardized ISR and unstructured MTR could provide a systematic and thorough picture of project status, but how effectively they are used varies. 2.22 One issue with the use of monitoring results is the transparency and availability of the monitoring products. The ISRs were not made public until recent changes in the World Bank‘s information disclosure policy (World Bank 2010). It remains unclear, however, how widely the ISRs are distributed and discussed among stakeholders. The output of MTRs is an Aide Memoire3 submitted to the government client, which is made public only if there is agreement between the client and the Bank (World Bank 2010). 2.23 The evaluation of the implementation and utilization of M&E systems in the ICRs identifies and documents these issues, but thus far effective solutions remain a significant M&E challenge. The ICR for the Ukraine Tuberculosis and HIV/AIDS Control Project, which was given an award by IEG for its candor, found that ―the M&E for the project was persistently one of the weakest areas and no complete set of indicators was ever produced with updated information‖ (World Bank 2009b). The ICR for the Burkina Faso HIV/AIDS Disaster Response Project found that the M&E system was ―generally ignored as a 3. Aides Memoire are post-mission summaries of findings, joint agreements with the government, and recommendations submitted by the World Bank to the government agency that is the loan/grant recipient. 9 management tool throughout much of the project period (at least by the project)‖ (World Bank 2007b). 2.24 These problems in M&E implementation and utilization have been overcome in some projects with a narrow, targeted scope and set of indicators (for example, the Lesotho HIV and AIDS Capacity and Technical Assistance Project; World Bank 2009a) or, more effectively, where project M&E is carefully integrated with support to strengthen national systems, and is carried out in partnership with other global partners (see below, ―Link Between Project M&E and National Systems‖). Evaluation 2.25 End-of-project evaluation is the strongest and most rigorous component of the World Bank institutional M&E approach, although within projects there are few incentives to use evaluation as a management tool for improving implementation and impact. The systematic guidelines and criteria for evaluating and rating projects upon completion often lead to candid assessments. It is not uncommon for projects to receive ―moderately unsatisfactory‖ or ―unsatisfactory‖ ratings. This can be interpreted as a high prevalence of poor performance among the Bank‘s global health projects, but also as acknowledging the reality of the difficulties in achieving results in countries receiving Bank assistance. The frequent low rating scores also can be interpreted as demonstrating a willingness to be self-critical in order to learn lessons for future global assistance investments and activities. 2.26 Evaluation of World Bank projects includes both internal project self-evaluation by the project team and independent review by the Independent Evaluation Group (IEG). Self-evaluation is undertaken at the completion of every project through a standardized ICR. The ICR is drafted according to ICR guidelines and submitted to the Board no later than six months after the closing date of the project (World Bank 2006a). Project monitoring and evaluation data, performance-related reports, and other relevant operations documentation provide input into the ICR. The performance of the project is assessed against standard criteria, and a separate assessment of the project M&E system (design, implementation and utilization) is included. Ratings are assigned to each criterion, typically six possible ratings from highly unsatisfactory to highly satisfactory (three possible positive ratings and three possible negative).4 The harmonized evaluation criteria for closed projects, used both by management and IEG, include: (a) Outcome: The extent to which the operation‘s major relevant objectives were achieved, or are expected to be achieved, efficiently. This is a rating of the project‘s overall performance, based on the relevance of the project‘s objectives and design, the extent to which it achieved its objectives, and the efficient use of resources in pursuing those objectives. Efficiency is assessed by whether the costs involved in achieving project objectives were reasonable in comparison with both the benefits and with recognized norms (―value for money‖). The project outcomes are rated in 4. The ICR discusses M&E but does not rate it. Since FY07, IEG‘s ICR Reviews have rated M&E in an effort to raise incentives for implementing M&E. 10 the ICR as highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, or highly unsatisfactory. A rating of highly satisfactory is given if there were no shortcomings in the achievement of project objectives, efficiency, or relevance. A rating of satisfactory is given if there were minor shortcomings, moderately satisfactory if there were moderate shortcomings, moderately unsatisfactory if there were significant shortcomings, unsatisfactory if there were major shortcomings, and highly unsatisfactory if there were severe shortcomings in the project‘s achievement of its objectives, efficiency or relevance (World Bank 2006a). (b) Risk to development outcome: The risk, at time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). The rating scale is negligible to low risk, moderate, significant, or high. This rating also can be interpreted as assessing potential for sustainability. (c) Bank performance: The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan/credit closing), toward the achievement of development outcomes. The rating scale is highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, or highly unsatisfactory. (d) Borrower performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating scale is highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, or highly unsatisfactory. (e) M&E quality: The quality of M&E design implementation and utilization are each assessed. M&E design is assessed by the extent to which adequate indicators were identified to monitor progress toward the project‘s development objective using effective collection methods. M&E implementation is assessed by the extent to which appropriate data were actually collected using appropriate collection methods to ensure data quality, and M&E utilization is assessed by the extent to which appropriate data were evaluated and used to inform decision-making and resource allocation (World Bank 2006a). 2.27 Independent review of all projects is undertaken by the IEG through a standardized desk review of all ICRs to independently assess the project experience based on the information in the ICR. The ICR Review is a validation exercise that validates the project ratings in the ICR using assessment criteria and guidelines that are harmonized with those of the self-evaluation completed in the ICR. IEG‘s project ratings may differ from those of the self-evaluation and those assigned by IEG become the final project ratings in the Bank‘s information system. The ICR Review also assesses the quality of the self-evaluation 11 according to the following criteria: quality of evidence, quality of analysis, extent to which lessons are based on evidence and analysis, the results orientation of the ICR, internal consistency, consistency with ICR guidelines, and conciseness (World Bank 2006a). 2.28 This self-evaluation and independent review process has been able to compensate for some of the deficiencies implementing monitoring systems in World Bank projects. For example, when the ICR team for the Ukraine Tuberculosis and HIV/AIDS Control Project discovered that the monitoring indicators had never been collected during the course of the project, the team collected extensive data on baselines, targets and actual achievements, working with government counterparts as well as with international agencies to retroactively build up the monitoring system (World Bank 2009c). 2.29 For 20–25 percent of closed projects each year, IEG conducts an independent Project Performance Assessment Report (PPAR), which involves a review of all project documents, field visits to project sites, data collection, and interviews of multiple stakeholders, including the government, World Bank staff, other donors, and civil society. Projects are purposively selected for a PPAR based on one or more the following criteria: (a) as a contribution to a cluster/country sector review; (b) as an input into an IEG sector/thematic study; (c) as an input into a Country Assistance Evaluation; (d) as a potential impact evaluation; (e) as an innovative project/new instrument; (f) due to disagreement on the project performance rating between IEG and management and/or poor ICR quality; (g) as an under-evaluated country/sector/theme; (h) a major safeguard compliance issue; and/or (i) requests by Executive Directors, cofinanciers, or regions. 2.30 The PPAR can give a more in-depth picture of what happened during implementation and why certain outcomes were observed. Sometimes the ―reality check‖ of field visits and stakeholder interviews can give quite a different picture from that presented in the ICR. In the ICR for the Lesotho HIV and AIDS Capacity Building and Technical Assistance Project, for example, the assessment of the M&E system was positive, pointing to the appropriateness of a limited set of indicators related to the project‘s objective of supporting the implementation of the Global Fund grant. The ICR also highlighted the success of the project supporting the development of a National HIV/AIDS M&E plan and strengthening national capacity in M&E (IEG 2010c). The PPAR, however, pointed out that in fact the M&E system in the health sector in Lesotho remains fragmented, with data and information systems for HIV/AIDS, TB, financial management, and human resources management still being run as vertical systems. The PPAR also concluded that the sophistication of the M&E planning and design in the project did not match the capacity of staff and systems implementing them, and that redundant systems at the district level remained highly inefficient (IEG 2010c). The overall ratings given by the PPAR were slightly lower than those for the self-evaluation (ICR) of the project. 2.31 A major weakness in the evaluation of World Bank health projects identified in the 2009 IEG evaluation of HNP support is the failure to carry out evaluations of pilot activities. The study found that 65 projects (about 30 percent of the total HNP portfolio approved from 1997 to 2006) were labeled as pilot projects, had pilot interventions or pilot regions, had an objective to test an approach or intervention, declared the intent to evaluate the impact of an intervention or program, or had the intent to conduct an impact evaluation. Of those projects, 12 32 had closed and only four actually conducted the planned pilot or impact evaluation (IEG 2009a, p. 25). The IEG evaluation cites lack of incentives for M&E in World Bank operations as the main reason for weak monitoring in many projects and failure to complete internal evaluations of pilot activities. The World Bank management response to IEG‘s evaluation also noted that countries are typically reluctant to borrow for M&E in general, and for expensive large-scale evaluations in particular (IEG 2009a). 2.32 The World Bank and IEG also carry out some impact evaluations using rigorous methods. These are less frequent due to their high cost and the difficulty of meeting the methodological demands, including control groups, adequate baseline, and ensuring the external validity of the findings, although there are new initiatives to expand the number of evaluations (World Bank, undated-b). 2.33 The main constraint facing IEG on impact evaluations besides resources is that IEG does ex post impact evaluation; it is not involved in the design of Bank projects. Therefore, it is necessary to be opportunistic about identifying promising datasets or field work that is often already underway. This also means that IEG primarily uses quasi-experimental methods (like propensity score matching). Nonetheless, impact evaluations are important for contributing to the pool of evidence about effective approaches for global health partners and how best to target investments. 2.34 There are a number of other initiatives that support impact evaluations in the World Bank, including the Development Impact Evaluation Initiative (DIME) from the Bank‘s Development Economics Department, the Spanish Trust Fund for Impact Evaluation, which is dedicated to the evaluation of innovative programs to improve human development, and the Africa Impact Evaluation Initiative. The 2009 IEG evaluation of HNP support identified 101 ongoing or complete HNP impact evaluations, most of which are based on an experimental design (IEG 2009b, p. 24). Link Between Project M&E and National Systems 2.35 The World Bank stated M&E policy requires that the project M&E systems ―[rely] on the borrower‘s M&E systems to the extent possible and, if these systems are not strong, assisting the borrower‘s efforts to strengthen them‖ (World Bank 2007a). World Bank Health projects often provide technical assistance to develop national M&E plans and build capacity in national M&E structures, which are then used as the structures for project reporting and M&E. In the Lesotho HIV and AIDS Capacity Building and Technical Assistance Project, for example, technical support was provided to the Ministry of Health and Social Welfare to strengthen its Monitoring and Evaluation Unit, which also had responsibility for project performance monitoring. The project supported the development of a National HIV/AIDS Monitoring and Evaluation Plan in 2004 and the later National HIV/AIDS Monitoring and Evaluation Plan 2006–2011, and provided significant support to build capacity to implement the plans (World Bank 2009a). 2.36 In fact, World Bank health projects often dedicate entire components or sub- components to strengthening national M&E systems, and project M&E is designed to be integrated into those activities. In the Russia TB/AIDS Control Project, strengthening 13 surveillance and monitoring was a sub-component of both the TB and AIDS components of the project, and a third component was dedicated to M&E of the project (World Bank 2003). The M&E activities were well integrated and carried out in partnership with other global health agencies such as WHO and UNAIDS. Through this approach, the project was effective in strengthening the overall systems in the country. The Russia TB/AIDS Control Project ICR found that ―data collection and reporting were conducted as part of the regular surveillance system that was strengthened with Project support, using data and information and assessments made at the regional and federal levels. The Project strengthened existing systems rather than attempting to establish a project-specific system‖ (World Bank 2009b). 2.37 The award-winning Kyrgyz Republic Health Sector Reform Projects ( HSRP I and II) (IEG 2009b) 5 are particularly good examples of how effective coordination and integration across global health partners and country policy-makers can fully exploit and leverage the role of M&E in the management of the project and health sector as a whole. The PPAR noted that ―One effective mechanism for countering political resistance to reform is strong and consistent M&E, where analytic results can be generated and disseminated rapidly and effectively, building support for project activities in a politically contentious environment‖ (World Bank 2008). 2.38 The ICR of the second project found that ―instead of simply developing a project monitoring framework, HSRP II sought to put in place and support a sector performance monitoring framework that could be used to track progress under the government‘s health reform program. The goal was to use a sub-set of indicators developed under this framework to track progress against the project‘s development objectives‖ and ―The design of the project also aimed to build capacity of MOH staff in M&E and integrating this function permanently into the MOH...At the end of the project, the monitoring function was institutionalized under the MOH as was planned while the evaluation function was institutionalized within the Center for Health Systems Development‖ (World Bank 2007c). 5. The projects were given an award by IEG to acknowledge ―exemplary design and implementation in World Bank projects.‖ The award was based on the results of the PPAR of the projects. 14 3. Global Fund Approach to M&E 3.1 Like the World Bank, the Global Fund has made monitoring and evaluation an integral part of its governance and operations. The 2004 Global Fund M&E Operations Plan (Appendix C) calls for an approach that integrates monitoring and evaluation at the grant- funded program, disease and Fund level (Global Fund 2004). In practice, however, the Global Fund began by largely separating the monitoring and evaluation of the performance of its grant recipients from the monitoring and evaluation of its own performance as a global health agency. Performance measures for grants could not be easily aggregated into measures of performance at the country or agency level. Whereas the M&E approach and requirements for grantees were elaborated in great detail and tied to the disbursement of funding, ―the broader institutional responsibilities for ensuring the Fund achieves its purpose are less well understood‖ (Global Fund 2004). 3.2 This situation is evolving, however, particularly with the new Global Fund strategy of 2012–2016 (Global Fund 2011c), which aims to improve alignment with national strategies and systems, and the new single-stream grant architecture (one funding agreement per Principle Recipient per disease), which includes Periodic Reviews of country-level program outcomes and impacts. The new Global Fund Strategy is accompanied by a new Evaluation Strategy (Global Fund 2011b), which aims to address a number of shortcomings in M&E processes identified in the Five-Year Evaluation. The new Evaluation Strategy includes: (a) Developing a systematic approach to the monitoring and evaluation of each of the three diseases with strengthened country-level information systems as the foundation (b) Addressing data quality at the Secretariat to improve grant management (c) Implementing continuous and more frequent evaluations that are integrated with country systems and rely more closely on partners (d) Building M&E capacity at the country level (e) Undertaking each year five to six country-level studies of Global Fund-financed programs (f) Filling information gaps related to data quality, equity, gender, marginalized groups and financing. Objectives of M&E 3.3 The Global Fund has different objectives for monitoring and evaluation at the ―corporate‖ level, the country level and the project/grant level: ï‚· Corporate (Global Fund) level: To ensure that raising, investing and proving the contribution of funds are closely related; to guarantee that money is well spent 15 relative to program goals and contributes to impact on the three diseases; and to develop an evidence-based platform to advocate for sustained funding6 ï‚· Country level: To link performance with disease impact, including the principle of additionality and the performance of the portfolio architecture of the Fund (Global Fund 2004) ï‚· Project/grant level: To support performance-based funding (Global Fund 2004). 3.4 The new Global Fund Evaluation Strategy articulates more specific objectives (Global Fund 2011b): (a) To assess the effectiveness, efficiency, equity and impact of the Global Fund‘s strategic initiatives and the programs it funds, to regularly inform grant management, investments, and resource mobilization efforts (b) To assess the implementation progress and success of the Global Fund Strategy 2012–16 and the recommendations of the High Level Independent Review Panel (c) To inform the 12-year evaluation of the Global Fund (d) To estimate the contribution of Global Fund investments to the Millennium Development Goals (MDGs) 4, 5, and 6. 3.5 To strengthen the independence of the Fund‘s M&E work and the soundness of the approaches, the Board established an independent evaluation advisory group—the Technical Evaluation Reference Group (TERG). The TERG is accountable to the Board ―for ensuring independent evaluation of the Global Fund business model, investments and impact‖ (Global Fund 2010e). The TERG meets at least twice a year and regularly reports to the Board through the Policy and Strategy Committee. The Board ―may request the TERG to consider commissioning or overseeing independent evaluations in areas identified. The TERG shall report the findings on such evaluations to the relevant Board committee. . . All recommendations of the TERG are advisory and are not binding on the Board or any of its committees. The PSC, other Board committees and the Secretariat shall not revise TERG recommendations or prevent those recommendations from reaching the full Board‖ (Global Fund 2010e). Framework and Indicators for Global Fund M&E M&E FRAMEWORK AND INDICATORS AT THE GLOBAL FUND LEVEL 3.6 At the corporate level, there is a framework and indicators for a global M&E system, which is loosely based on the aggregation of the performance of individual projects/grants in achieving their service, output and outcome targets. The Global Fund has developed a four- tiered performance management framework to establish the conceptual link between operational performance and the overall impact of the agency (Figure 2). The framework 6. http://www.theglobalfund.org/en/performance/kpi/?lang=en. 16 Figure 2. Global Fund Four-Tiered Corporate Performance Management Framework KPI: ï‚· MDG 4: Reduce by 2/3 under-5 child mortality by Ultimate measures of the success in 2015; MDG 6: Combat HIV/AIDS, malaria and fighting the three diseases and other diseases and halve the prevalence by 2015 reaching the MDGs ï‚· % of GF countries reporting positive trends Impact towards the MDGs 4 and 6 ï‚· % of disease-specific country programs with adequate systems to measure impact ï‚· ï‚· % of:disease-specific country programs with KPI Global Fund aid effectiveness in adequate systems ï‚· % of funds to civil impact to measure allocated society organization achieving development results, ï‚· Amount of approved funding for health system strengthening health and Effectiveness strengthening and cross-cutting interventions community systems, promoting ï‚· % of GF funding to listed in the national or health gender equity and getting value sector budget and/or annexes for money ï‚· % gap in achieving Paris Declaration targets KPI: Programmatic ï‚· % of grants with improved performance achievements and ï‚· % of well performing grants performance of the Grant Performance ï‚· % of Top 10 output targets achieved in GF Global Fund’s supported programs portfolio of grants ï‚· % of grants reporting complete information through Price Quality Reporting KPI: Performance related to resource ï‚· 15 KPIs; for example: mobilization; administrative Operational Performance ï‚· % of annual pledges donated effectiveness; portfolio management ï‚· % of staff rated below expectations and special initiatives. ï‚· Average time for disbursement of grant funds Global Fund Source: http://www.theglobalfund.org/en/performance/kpi/. Contribution Note: This framework is currently being updated by the Global Fund with new KPI to be consistent with the new strategy for 2012–2016. reflects the business model of the Global Fund as a funding agency rather than implementer, as it ties performance to the results of collective action rather than the sole impact of the Global Fund (Nahlen and Low-Beer 2007). 3.7 The framework starts from operational performance, to grant performance, which leads to effectiveness, and ultimately impact on the burden of the three diseases. The Global Fund accepts full responsibility for its operational performance, but results at higher levels of the framework are increasingly due to the collective effort of a number of partners in addition to the Global Fund.7 7. http://www.theglobalfund.org/en/performance/kpi/lang=en. 17 3.8 The Global Fund identifies key performance indicators (KPIs) to monitor progress along the four dimensions of the performance management framework.8 The Policy and Strategy Committee of the Board of Directors develops and approves the KPI with baselines and targets, which are periodically updated.9 There are currently 26 KPIs, 15 of which measure operational performance, and a large number of sub-indicators, most of which are also output indicators. 3.9 Although the pathway from operational performance to grant performance to effectiveness to impact makes intuitive sense, the performance indicators do not seem to capture these steps adequately. The link between effectiveness and impact as represented by the KPI in particular is poorly developed. Furthermore, the indicators focus on inputs, the definitions are vague in many cases, and the data sources are not obvious. Three of the four KPIs for effectiveness, for example, represent inputs rather than effects. There is no discussion of how many of the targets were set, whether they are valid, and how meeting the targets of each of the indicators contributes to overall impact. 3.10 To assess aggregate grant performance, the Global Fund focuses particularly on three ―people reached‖ indicators (funded by Global fund grants): (a) the number of HIV-positive people receiving anti-retroviral drugs; (b) the number of new confirmed tuberculosis cases treated through evidence-based DOTS therapy; and (c) the number of insecticide-treated bed nets distributed for protection against malaria. Targets for grants across these indicators are consolidated to provide annual targets for the total Global Fund grant portfolio (Global Fund 2005), so the validity of the global targets depends on the validity of the project/grant targets. The FYE of the Global Fund found that standardized information across the grant portfolio was not available for any other indicators of service delivery (Macro International 2009a). Furthermore, these indicators do not reflect outcomes, although there has been an attempt by the Global Fund in its latest progress report to quantify the contribution of Global Fund grants to reaching MDGs and attribute declines in mortality related to the three diseases at least partially to Global Fund activities (Global Fund 2010a). M&E FRAMEWORK AND INDICATORS AT THE COUNTRY LEVEL 3.11 The Global Fund requires national M&E plans to be submitted as part of the grant application process, although there was no specific country level M&E activities undertaken as part of Global Fund activities prior to recent changes in Global Fund grant architecture. The Global Fund considered it the responsibility of the countries to carry out M&E for the three disease areas in the context of national M&E systems. The Global Fund requires that grant applicants submit an M&E plan at the time of grant signing. Typically a Principal Recipient is only required to submit the national M&E plan (specific to a disease or for a combination of the three diseases, depending on the country context) for monitoring the national strategy expected to be supported by the Global Fund program(Global Fund 2010d). 3.12 Although the national M&E plan is not used to directly monitor and evaluate grant performance, it does provide background information for the specific performance 8. http://www.theglobalfund.org/en/performance/kpi/lang=en. 9. http://www.theglobalfund.org/en/performance/kpi/lang=en. 18 framework of the grant (see below). The M&E plan should be based on a self-assessment and accompanied by an annual costed workplan describing the planned M&E implementation and strengthening activities. If a national M&E plan is not available or appropriate, a grant- specific M&E plan can be submitted in its place. The applicants may also request funds to update or develop a national M&E plan (Global Fund 2011d). The Global Fund has provided a number of guidelines and tools to assist countries to assess, design and implement national M&E systems (Global Fund 2011d, 2009a, 2009b, 2008a). 3.13 In 2009 the Global Fund Secretariat introduced the M&E country profile tool to systematically track progress on M&E system performance at the country level, which will be rolled out globally in 2012 (Global Fund 2011d). The M&E country profiles provide detailed information on various aspects of the national M&E system at the health sector, disease program, and Principal Recipient levels. The tool also aims to track progress on M&E investments and the implementation of key M&E systems-strengthening interventions. The M&E country profiles will be completed by Local Fund Agents during the negotiation for new grants and at the Phase 2 periodic review. 3.14 Although many countries have produced elaborate M&E plans for the three diseases, it is difficult to find evidence about the extent to which these plans are implemented and used for decision making. There is also a concern that the focus on disease-specific M&E undermines broader efforts to strengthen health systems. The FYE identified lack of support for integrated country level M&E systems as an area of improvement for the Fund, recommending ―the Global Fund and its partners should reorient investments from disease-specific M&E toward strengthening the country health information systems required to maximize data quality and use for decision-making‖ (Macro International 2009b). 3.15 Prior to the recent changes in Global Fund grant architecture, country level M&E specific to the Global Fund had been limited to the 18 country impact evaluations conducted as part of the Five-Year Evaluation. These country-level evaluations followed a similar conceptual framework, which did not mirror the corporate performance framework and did not attempt to attribute results to the performance of individual projects/grants in the countries. The conceptual framework underlying the country evaluations related the additional resources contributed by the Global Fund to address the three diseases to increases in coverage of core interventions, which was assumed to lead to impacts on disease incidence, prevalence, and morbidity and mortality outcomes (Figure 3). The framework did not attempt to distill the contribution of the Global Fund, but rather assesses the impact of the collective donor efforts to address the three diseases. 3.16 To address the previous lack of country-level M&E activities in the Global Fund M&E approach, several country-level M&E activities have been added in the new Global Fund Evaluation Strategy. For example, the new strategy calls for national program reviews, in which the Global Fund will support and actively participate in multi-partner national program reviews conducted by national disease programs (Global Fund 2011b). The new evaluation strategy also includes 10–12 program evaluations annually, which will incorporate more robust methodologies than the program reviews. 19 Figure 3. Conceptual Framework for Global Fund Country Impact Evaluations Contextual Factors Other international resources Provision and Disease incidence utilization of core and prevalence interventions against AIDS, TB and Disease Global Fund malaria consequences: resources mortality; ill health Domestic resources Resource tracking and Service delivery and coverage Surveys and surveillance of National Health Accounts assessment and facility data and disease and mortality by disease surveys Source: Macro International (2009c). M&E FRAMEWORK AND INDICATORS AT THE PROJECT/GRANT LEVEL 3.17 In contrast to the Fund level and country level M&E systems, the Global Fund has very detailed and well documented requirements and outputs for project/grant M&E, which is tied to performance-based funding (Figure 4). Unlike the mostly ad hoc implementation of corporate and country-level M&E, the link between project/grant level M&E and disbursement of funds has ensured the full implementation of this part of the Global Fund‘s overall M&E strategy. Performance-based funding is also meant to serve as a project management tool, creating an opportunity for self-assessment and implementation decisions by the recipients to improve the effectiveness of the project and success of scale-up. The extent to which this happens in practice is not well documented. 3.18 As shown in Figure 4, the grant-level M&E framework is based on the national M&E self-assessment process and action plan (discussed above). A grant-level performance framework is the main monitoring tool throughout the life of the grant. The Global Fund provides numerous guidelines and toolkits to assist principal recipients in the preparation of the performance framework. The Global Fund also offers a planning process called Monitoring and Evaluation Systems Strengthening Tool (MESS) workshops, and technical assistance is available through the World Bank, the USAID Grant Management Solutions project, and other sources. 20 Figure 4. Global Fund Project/Grant M&E Requirements DQA - Data quality audit LFA - Local fund agent OSDV - On-site data verification PF - Performance framework PR - Principal recipient Source: http://www.theglobalfund.org/en/me/?lang=en. 3.19 The performance framework for each project forms part of the grant agreement and is the legal statement of the performance expected over the lifetime of the grant. It contains a summary of key indicators and targets, which are used to measure output and coverage on a routine basis. The performance framework also includes indicators to measure outcomes and impacts, which are used in funding renewal decisions for phase 2, while output and coverage indicators are used for routine disbursement of funds. All performance frameworks are disease-specific.10 The Global Fund tracks performance against the targets from the original proposal and uses them for performance-based funding decisions (Global Fund 2009a). 3.20 Selecting appropriate and meaningful performance indicators generally has been a challenge in Global Fund projects/grants. Indicators selected in the earlier grant rounds commonly suffered from a number of deficiencies: there were too many indicators, which often were not well defined, did not come from routine information sources and so required special data collection, and focused mainly on outputs rather than outcomes. In the Lesotho 2004 Global Fund grant agreement (HIV/AIDS only), for example, 17 performance indicators were specified, many of which were ―count‖ indicators reflecting outputs (Table 2). Furthermore, indicators such as ―HIV/AIDS curriculum in all schools institutionalized‖ and ―number of registered orphans and other vulnerable children receiving basic package of care and support‖ were not clearly defined, and measurement was even more problematic. 3.21 Even more problematic, however, is the setting of targets for grant performance assessment. Although the Global Fund provides guidance on setting targets in grant performance frameworks, it is not transparent in any of the grant documentation how targets 10. http://www.theglobalfund.org/en/me/guidelines_tools/?lang=en#pf. 21 Table 2. Performance Indicators in the Lesotho 2004 Global Fund Grant Agreement Indicator From Current Top Ten List? HIV/AIDS curriculum in all school institutionalized No # of young adults (15–29) reached by peer education No # of youth resource centers established regionally No # of Adolescent Reproductive Health corners increased No # of condoms distributed to youth corners No # of condom vending machines procured and installed in youth centers and adolescent No corners # of babies testing negative on PMTCT treatment No # of women counseled/trained on PMTCT No # of HSAs with PMTCT program No # of all HSAs staff trained on PMTCT Yes (―routine reporting‖) # of HSAs with continuum of care services No # of PLWHA with access to community home-based care support Yes (―routine reporting‖) # of health workers trained on diagnosing and providing care and support to PLWHA Yes (―routine reporting‖) Establishment of ARV program and number of PLHWA on ARV treatment Yes (―routine reporting‖) # of sites with functional Voluntary Counseling and Testing centers No # of registered OVC receiving basic package of care and support Yes (―routine reporting‖) # of children enrolled in the school program No Source: Program Grant Agreement between the Global Fund to Fight AIDS, Tuberculosis and Malaria and the Ministry of Finance and Development Planning of the Government of the Kingdom of Lesotho (2004). Note: PMTCT = prevention of mother-to-child transmission. are set. Some grant monitoring reports indicate a suspicion that the targets were set too low (for example, the Grant Scorecard for the project Stimulating an Effective National Response to HIV/AIDS in the Russian Federation). A transparent and critical assessment of this part of Global Fund grant assessment, however, seems to be missing, particularly in light of the consistent achievement and common over-achievement of targets across the entire grant portfolio. 3.22 To streamline the indicators and ensure their relevance and validity, the Global Fund has developed a set of ―Top Ten Indicators‖ that it recommends to its grantees (Table 3). The Global Fund recommends a different set of indicators for routine Global Fund reporting than for assessing medium-term outcomes and impacts. Examining the M&E frameworks for grants in Burkina Faso, Lesotho and the Russian Federation, it appears that the indicators for routine reporting and performance-based funding continue to focus on outputs. This standardization of at least a portion of the indicators tracked by individual projects/grants, however, has made it possible to begin to aggregate overall performance across Global Fund projects through the KPI. But the pathway remains tenuous from project/grant inputs, to service coverage (the main indicators tracked at the grant and global level), to global outcomes (country outcomes are poorly tracked). 3.23 The heavily planned and structured approach to developing M&E frameworks and indicators for Global Fund grants can appear burdensome, but it has led in some cases to indicator sets that are more closely aligned to grant objectives and that reflect the full range of outputs, outcomes, and impacts. The Moldova M&E plan for the TB component of its Global Fund grant, for example, includes 36 carefully chosen indicators related to each 22 Table 3. Global Fund Recommended ―Top Ten‖ Grant Performance Indicators Disease Indicators Indicators for Routine Global Fund Reporting 1 HIV Number of adults and children with advanced HIV infection currently receiving antiretroviral therapy 2 TB Number of (a) new smear-positive TB patients detected, (b) new smear-positive TB patients who were successfully treated and (c) laboratory-confirmed MDR-TB patients enrolled in second-line anti-TB treatment 3 Malaria Number of (a) insecticide-treated nets or re-treatment kits distributed to people and (b) households (or structures and walls) in designated target areas sprayed by indoor residual spraying in the past 12 months 4 Malaria Number of people with fever receiving antimalarial treatment according to national policy (specify artemisinin-based combination therapy versus other therapy) 5 HIV Number of women and men aged 15–49 years who received an HIV test in the last 12 months and who know their results 6 HIV Number of HIV-positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child transmission 7 HIV Number of condoms distributed 8 HIV, TB, Malaria Number of people benefiting from community-based programs: specify (a) care and support including orphan support, home-based management of malaria and directly observed therapy (DOT); (b) behavior change communication outreach activities including specific target groups; and (c) disease prevention for people most at risk (except behavior change communication) 9 HIV, TB Number of TB patients who had an HIV test result recorded in the TB register 10 Health systems Number of people trained strengthening for HIV, TB, Malaria Indicators to Assess Medium-Term Outcomes and Impacts Indicators recommended for generalized Indicators recommended for concentrated epidemics and high-endemicity areas epidemics and low-endemicity areas 1 HIV Percentage of women and men aged 15–24 Percentage of populations most at risk who years who are infected with HIV are infected with HIV 2 HIV Percentage of adults and children with HIV known to be receiving treatment 12 months after initiation of antiretroviral therapy (extend to two, three and 5 years as the program matures) 3 HIV Reduced mother-to-child transmission of HIV: percentage of infants born to mothers who are HIV infected 4 HIV Percentage of people aged 15–49 years with Percentage of populations most at risk with more than one sexual partner in the past 12 more than one sexual partner in the past 12 months reporting the use of condoms during months reporting the use of condoms during their last sexual intercourse last sexual intercourse 5 TB TB case detection rate and treatment success rate 6 TB TB prevalence rate: estimated number of TB cases (all forms) per 100,000 population 7 Malaria All-cause mortality rate among children Malaria-specific mortality: proportion of deaths younger than five years of age attributed to malaria among children younger than five years of age (other than target groups) 8 Malaria Number of (confirmed) malaria cases seen by a. Annual parasite index health workers (in facilities and/or outreach) b. Slide-positive or rapid diagnostic test- positive rate 9 Malaria People sleeping under an insecticide-treated net the previous night (specify the target population: all household residents, children younger than five years of age, pregnant women) 10 Health systems All-cause mortality rate among children younger than five years of age. strengthening for HIV, TB, Malaria Source: http://www.theglobalfund.org/documents/me/TopTenIndicatorsCard_en.pdf. 23 service area in the grant, with clear indicator definitions and data sources and flows (Ministry of Health of the Republic of Moldova 2007). It also appears that the indicators in a number of earlier grants were revised based on the Top Ten list. In Burkina Faso and Lesotho, for example, the indicators in the original grant agreements amounted to long lists of output indicators with little relationship to the performance framework.11,12 The revised list of indicators in the most recent Grant Performance Reports for both grants are organized along ―impact,‖ ―outcome,‖ and ―output‖ indicators (Global Fund 2009c, 2011a). 3.24 Nonetheless, the report of the Global Fund‘s Office of the Inspector General found that the M&E frameworks for grants in a large share of the seven audited countries were not designed effectively with, for example, approved M&E plans showing when, how and by whom monitoring should be undertaken, and with targets that were unattainable (Office of the Inspector General 2009). M&E Implementation and Utilization 3.25 At the corporate level, M&E appears to be mostly utilized to highlight the success of the Global Fund, although there has been a recent attempt to be more public about identifying shortcomings and revising policies and procedures in areas where poor performance is indicated (Global Fund 2009d, 2010a). Previously, M&E utilization was limited to the Policy and Strategy Committee (PSC) of the Global Fund‘s Board of Directors reporting annually on KPIs and highlighting success in the public annual progress report. Although areas of weakness are more openly discussed in recent progress reports, the conclusions remain highly positive, and targets are typically exceeded (Global Fund 2008b, 2009d, 2010a). According to the most recent KPI report, 84 percent of KPIs are meeting their targets or performing adequately (Global Fund 2010c). The report claims that targets were met by 105 percent for output and service indicators. 3.26 Areas of under-performance identified by the KPIs focus on grant-signing and disbursement processes. The Global Fund did act based on weaknesses identified by the KPI in these areas, conducting an assessment to identify bottlenecks in grant-signing and disbursement based on the indicator results, making changes to its grant negotiation process, and increasing the automation of disbursement monitoring (Global Fund 2010c). The FYE also found that the attention of Board was heavily weighted toward monitoring internal matters and grant management (Macro International 2009a). 3.27 At the country level, there is no evidence to assess the extent to which national M&E systems supported by the Global Fund are used to inform project planning and implementation. This is part of a general problem development partners have faced trying to assess the value of investments in M&E systems. A recent study examining the effectiveness of investments of global health partners in M&E for HIV/AIDS was unable to assess whether data were being used to inform decision making. The study did find that fewer than half of the countries reporting had self-assessed their data use as above average. 11. Program Grant Agreement between The Global Fund to Fight AIDS, Tuberculosis and Malaria and the United Nations Development Programme, Burkina Faso (2003). 12. Program Grant Agreement between The Global Fund to Fight AIDS, Tuberculosis and Malaria and the Ministry of Finance and Development Planning of the Government of the Kingdom of Lesotho (2004). 24 3.28 There also is little evidence that the extensive country impact evaluations conducted for the FYE are being used for policy and project purposes in the countries studied. The World Bank Burkina Faso country report for IEG‘s Global Program Review of the Global Fund found that many stakeholders interviewed in the country were aware of the FYE, but few had seen the report for Burkina Faso or the global synthesis report, neither of which was translated into French. The process of preparing the country report was seen as a one-time exercise conducted primarily by outside researchers, and the data generated from this exercise was not used for programming. In Tanzania most respondents were unaware of the FYE. 3.29 The implementation and utilization of the Global Fund M&E system at the project/grant level is focused on supporting performance-based funding (PBF). During the lifetime of the grant, the Global Fund periodically disburses funds to the Principal Recipient based on demonstrated program performance and financial need (Global Fund 2011a). Information collected is used at two main stages of performance evaluation: (a) Regular disbursements (every six months is the default). Agreement on a few indicators of progress is used for regular financial release every three to six months. Funds are released based on disbursement requests accompanied by progress updates of the results against targets. (b) Periodic reviews (after up to three years of implementation). Periodic reviews are used for additional financial commitment decisions under the new single-stream grant architecture. All Principal Recipients in same disease/HSS program are reviewed at the same time, aligned with country cycles. Performance assessment includes analysis of program impacts/outcomes.13 3.30 For each reporting period, the Principal Recipient is required to provide the Global Fund with a Progress Update and Disbursement Request (PUDR). This document consists of a progress report on the implementation of a grant, and a request for funds for the next reporting period. The progress report includes information on the program being implemented (including results against targets) and information on expenditures. The PUDR is reviewed by the Local Fund Agent and submitted to the Global Fund Secretariat. The Secretariat reviews the PUDR and assesses: ï‚· Programmatic achievements: Have programmatic targets been reached? ï‚· Financial performance: Are expenditures in line with budgets? ï‚· Grant management: Are there issues related to monitoring and evaluation, procurement and/or financial management? 3.31 Based on the assessment of the PUDR, the Secretariat assigns a performance rating to the grant from the following scale (three out of five ratings are positive):14 - A1: Exceeded expectations - A2: Met expectations 13. http://www.theglobalfund.org/en/grantlifecycle/3/?lang=en. 14. http://www.theglobalfund.org/en/performance/grantportfolio/performance/?lang=en. 25 - B1: Adequate - B2: Inadequate but potential demonstrated - C: Unacceptable. 3.32 The Secretariat then decides whether to allow the requested disbursement of funds, to allow partial disbursement of funds, or to deny the disbursement request. An outright denial of the request is rare and would only happen if a grant was in serious trouble. 3.33 The Grant Performance Report (GPR) is prepared by the Global Fund Secretariat when the Grant Agreement is signed, and it is updated with every PUDR received throughout the life of the grant. Before the end of Phase 1 of the grant, the Global Fund decides whether to continue funding for Phase 2. A Grant Scorecard (GSC) is prepared with a structured assessment of the grant performance, the decision about whether to continue funding Phase 2 of the grant, and justification for the decision. 3.34 The review prior to entering Phase 2 (now called the Periodic Review) is an integral part of the Global Fund‘s system of performance-based funding. The Global Fund will only approve Phase 2 funding if a project is performing adequately. In practice, it is rare to award a grant a ―no go‖ and completely discontinue the grant in Phase 2. Over the period 2005– 2009, only 1.9 percent of grants were discontinued after Phase 1 (Global Fund 2010a). It is more common for a portion of the funding to be reallocated to better performing grants. Over the same period, 13.7 percent of total funding was reallocated from poorer performing grants (including ―no-go‖) to better performing grants (Global Fund 2010a). 3.35 The PUDRs, GPRs and GSCs are completed consistently and made public. In 2009, 85 percent of grants performed well according to the KPIs and were rated as A2 (met or exceeded expectations) in 2009.15 The Global Fund FYE found some inconsistencies, however, in how grant ratings were awarded (Macro International 2009a). An independent study reviewed the performance information in 59 Global Fund grant scorecards in 37 sub-Saharan African countries. The results showed that M&E concerns were explicitly listed for remedial action in more than half of the grants. Systematic weaknesses in M&E systems (such as insufficient human capacity, lack of planning, ill-defined performance frameworks and indicators, weak health information systems and lack of survey data) were indicated in 51 percent of the grants, and specific data quality concerns were raised for 34 percent of the grants (Peersman and others 2009). 3.36 The Global Fund provides tools and support to continually identify weaknesses in M&E systems and support actions to strengthen them. For example, the Global Fund is currently implementing a corporate data quality strategy aimed at strengthening data quality at all stages of the grant life-cycle (Global Fund 2010b). As part of this strategy the Global Fund has introduced a new data quality assessment tool used by Local Fund Agents (LFAs) to conduct periodic on-site data verification (OSDV) at service delivery points for key service delivery indicators. The proportion of eligible grants verified through OSDV increased from 27 percent in 2008 to 84 percent in 2009 (Global Fund 2010b). The Global Fund also implements independent data quality audits for three to five grants per quarter. 15. http://www.theglobalfund.org/en/performance/kpi/2009/#gp. 26 3.37 Linking performance to funding puts additional pressure on M&E systems and demand for data quality. The FYE found that the Global Fund has emphasized both country ownership and performance-based financing, and the scale at which the Global Fund has attempted to implement PBF is unprecedented in the international health arena. However, the external evaluation found that this ―focus on results‖ remains a work in progress and has evolved into a complex and burdensome system that has so far focused more on inputs and outputs than on development outcomes (IEG 2010a). 3.38 Nevertheless, there are some examples of the reporting requirements of the performance-based financing being used for wider project management purposes on the ground. In Burkina Faso, for example, the process was initially considered burdensome, but over time the perceptions among grant participants changed dramatically. IEG‘s country report for the Global Program Review found that ―All Global Fund Recipients consulted were in agreement with both the principles and practices of PBF.‖ And several grant recipients told of how they have integrated the GF performance based indicators into their own planning processes and now rely on them for their own decision-making and planning (IEG 2010a). Evaluation 3.39 The role of evaluation in the Global Fund is still evolving, with the latest iteration articulated in the new Global Fund Evaluation Strategy 2012–2012. The Global Fund M&E plan did not establish a specific approach to evaluation of the Global Fund activities at the Fund level, and the need for evaluation was left to the TERG to determine on an ad hoc basis. To date the TERG has overseen five assessments in addition to the Five Year Evaluation, including an assessment of the CCM approach in 2005 (Technical Evaluation Reference Group 2005), assessment of the proposal development and review process in 2006 (Wilkinson and others 2006), an assessment of stakeholder perceptions of the Global Fund (Technical Evaluation Reference Group 2006a), a portfolio review (Technical Evaluation Reference Group 2006b), and an evaluation of the Local Fund Agent system in 2007 (Euro Health Group 2007). The Global Fund has not regularly conducted evaluations at the country level to determine the impact of its collective efforts on the three diseases in the countries in which it operates, nor does it conduct evaluations at the conclusion of individual grants. 3.40 Thus there has been no evaluation ―system‖ of the Global Fund. The Global Fund FYE published in 2009 was the first real attempt to look critically at the performance of the Global Fund in three ―study areas‖: (a) Study Area 1: Organizational efficiency and effectiveness of the Global Fund (Macro International 2009a): ï‚· Does the Global Fund, through both its policies and its operations, reflect its critical core principles, including acting as a financial instrument (rather than as an implementation agency) and furthering country ownership? ï‚· In fulfilling these principles, does it perform in an efficient and effective manner? (b) Study Area 2: Effectiveness of the Global Fund partner environment (Macro International 2008a): 27 ï‚· How effective and efficient is the Global Fund‘s partnership system in supporting HIV, tuberculosis and malaria programs at the country and global level? ï‚· What are the wider effects of the Global Fund partnership on country systems? (c) Study Area 3: Impact on the three diseases (Macro International 2009c) ï‚· What is the overall reduction of the burden of AIDS, tuberculosis and malaria? ï‚· What is the Global Fund‘s contribution to that reduction? 3.41 The overall evaluation framework linking the three study areas was based on the assumption that effective and efficient Global Fund operations, together with effective partner relationships, would lead to increased provision and utilization of core interventions against HIV/AIDS, TB and Malaria, which in turn would reduce disease incidence, prevalence and mortality. What is missing from the framework is a link between project/grantee performance and the overall impact of the Fund as an agency. The FYE was also not based on cumulative internal/external evaluations. 3.42 In examining the impact of the Global Fund on service coverage and disease impact, the evaluation focused on results of collective action rather than attempting to attribute results directly to Global Fund performance (Macro International 2009b). Therefore, although Study Area 3 assessed impact indicators, it cannot be considered a valid impact evaluation since there is no causal link established between activities and impacts (White 2009). 3.43 The Global Fund FYE design, including lack of attribution and lack of a framework or cumulative assessments linking grant performance to impacts on the three diseases, made it unclear what criteria the FYE used to draw conclusions. The study exposed many shortcomings of the operations, performance and outcomes of the Global Fund activities, but the overall conclusion was positive. After more than 50 pages summarizing achievements that were more than balanced by significant shortcomings and risks, the final paragraph of the syntheses of the evaluation concluded: Nevertheless, the overall efforts of the first five years of the Global Fund can only be termed as extraordinary. It has demonstrated tremendous flexibility in adjusting grant disbursement strategies and operational policies, while urgently addressing the global funding gap for HIV/AIDS, TB, and malaria. The Five- Year Evaluation found that at a global level, collective efforts have resulted in increases in service availability, better coverage and reduction in disease burden. With these great gains come the challenge of maintaining momentum while correcting a range of major inadequacies and addressing new challenges that are emerging in terms of health system capacity and sustainability. The impressive capacities of the Global Fund leadership and staff inspire confidence that they will embrace and excel in meeting these new challenges in the months and years to come (Macro International 2009b). 3.44 The new Global Fund Evaluation Strategy articulates more specific objectives and aims to create a more systematic approach to evaluation of both the Global Fund‘s strategic initiatives and the programs it funds. This new strategy includes more rigorous assessment of grant performance through Periodic Reviews and assessment at the country disease program level. 28 Link Between Project M&E and National Systems 3.45 The Global Fund‘s M&E strategy states that its M&E approach ―builds on existing programme, country and global level capacity, and will wherever possible utilise available tools, systems and expertise at country level and among global partners‖ (Global Fund 2004). Furthermore, ―the Global Fund needs to access and systematize such data and information for its own internal use and will only strengthen data collection systems if required. In that sense, the Global Fund is primarily a user of monitoring information‖ (Global Fund 2004). 3.46 Case studies supported by the Global Fund in five countries as a follow-up to the FYE, however, found that in these countries the Global Fund often has contributed to parallel M&E structures, and it has reinforced verticalization of data systems, monitoring and evaluation. Problems were identified in three of the countries studied (Indonesia, Lao PDR, Papua New Guinea) (Desai and others 2010; Mounier-Jack and others 2010; Rudge and others 2010)16 The exceptions were Nepal, where moderate levels of integration had been achieved for Global Fund M&E and national M&E systems for the three disease areas (Tragard and Shrestha 2010) and Thailand, where Global Fund M&E was at least partially integrated into national systems (Hanvoravongchai, Warakamin, and Coker 2010). An earlier study by the Gates Foundation also highlighted that the Global Fund M&E systems at the project/grant level contributed to the fragmentation of country M&E systems: ―Officials often collect surveillance metrics for global health partnership (GHP)-funded programs in a fragmented manner. These metrics are not consistently integrated into national systems, and consequently GHPs may be duplicating efforts to collect metrics (for example, through NGOs)‖ (McKinsey and Co. 2005). 3.47 The fragmentation in M&E is exacerbated by the many grant recipients—principal recipients and sub-recipients—many of which are NGOs, which do not integrate their M&E with national systems. For example, the Gates Foundation study found that ―In Zambia, two of the four principal recipients of Global Fund resources are NGOs and do not currently share the metrics they collect for Global Fund programs with the National Statistics Program. Officials in the Office of Statistics report, ‗collecting data outside of the national systems undermines our planning efforts‘ ‖ (McKinsey and Co. 2005). 3.48 The new single-stream grant architecture, consolidated periodic reviews, and the new Global Fund Evaluation Strategy are all focused on better streamlining many aspects of grant implementation and management and better aligning with country systems, including monitoring and evaluation systems. 16. The Global Fund collaborated with partner institutions to conduct the cases studies to provide an analysis of the interaction between the programs it finances and key health system functions as part of larger global efforts to better understand the effects of disease-specific funding on health systems and to follow up on recommendations in the FYE. 29 4. Country-Level Comparison of the M&E Design, Implementation, and Utilization 4.1 In this section, the M&E systems of World Bank and Global Fund projects are compared in three countries that had similar activities across the two agencies that have been completed. The M&E systems are systematically compared across four dimensions: (a) M&E design: project/grant goals and objectives and performance indicators (b) M&E implementation: implementation process, data sources and collection, data quality, and monitoring products (c) M&E utilization: project/grant management and policy input; performance assessment (d) Overall performance of the M&E system: link with the national M&E system; assessment of the project/grant M&E system. 4.2 The comparison is based on an in-depth review of M&E products in each of the projects/grants. Burkina Faso WORLD BANK: HIV/AIDS DISASTER RESPONSE PROJECT 4.3 The HIV/AIDS Disaster Response Project, an integral part of the Multi-Country AIDS Program for Africa, aimed to ―assist the Government of Burkina Faso to implement the 2001–05 HIV/AIDS strategic plan, in order to slow the spread and mitigate the impacts of the epidemic‖ (World Bank 2001). The specific objectives were to: ï‚· Scale up, expand and improve preventive activities in an effort to lower the risk of transmission ï‚· Strengthen capacity to provide care, treatment and support to those infected or affected by the epidemic ï‚· Mitigate the socioeconomic impact on affected households and communities 4.4 The main components of the project included: (a) Line Ministry Work Programs ($10.6 million): Preparation, and implementation of annual HIV/AIDS work programs of government line ministries to fund preventive interventions, policy development, psycho-social counseling activities, and capacity building; (b) Provincial activities and community subprojects ($5.10 million): to assist in the planning, and coordination of HIV/AIDS activities, screen, and monitor community subprojects, and, build capacities of communities, and associations for the 30 implementation of subprojects. Community subprojects were to include awareness, and education promotion on preventive practices; home-based care for those affected by the disease; care of orphans, and, income-generating activities to assist, and empower affected families economically. Technical support, and training to build capacity was provided; (c) Targeted interventions ($4.2 million): to curb the epidemic in vulnerable and high- risk groups. Activities included preventive interventions, treatment, and care of sexually transmitted diseases, and HIV testing, and counseling; and (d) Coordination, monitoring and evaluation ($3.6 million) (World Bank 2001). 4.5 The project was approved in July 2001, became effective in March 2002, and closed in June 2007. The project was implemented by the National AIDS Council, the national HIV/AIDS/STI coordinating body. The total project cost was $28.3 million funded through an IDA credit, and the lending instrument was an Adaptable Program Loan (World Bank 2007b). GLOBAL FUND: PROJECT FOR ENHANCEMENT OF HIV/AIDS CONTROL 4.6 The Project for Enhancement of HIV/AIDS Control was a grant under the Global Fund Round 2 HIV/AIDS grants. The objectives of the project supported by this grant were to ―reduce illness and death associated with HIV and AIDS by accelerating access to antiretroviral drugs through subsidies for antiretroviral treatment; strengthening the capacities of structures and the expertise of personnel involved in medical and psycho-social treatment of patients living with HIV and AIDS; and expanding the national program for prevention of mother-to-child transmission of HIV. The project targets people living with HIV and AIDS, pregnant women and HIV-positive children.‖17 4.7 The principal recipient18 for the grant was The United Nations Development Programme (UNDP). UNDP managed the project, which was implemented by a number of sub-recipients. The project was effective from December 2003 to December 2005. Phase 2 ended in 2007. The total grant amount was US$10.6 million, 83 percent of which was disbursed. M&E COMPARISON 4.8 A detailed comparison of the World Bank project and Global Fund program M&E is presented in Appendix D. Both the World Bank and Global Fund projects suffered from poor design of the M&E system. The performance indicators were output indicators that have not been validated (shown to measure what they intend to measure), that were not clearly related to the objectives, and that relied heavily on count indicators. The Global Fund indicators, 17. Program Grant Agreement between the Global Fund to Fight AIDS, Tuberculosis and Malaria and the United Nations Development Programme, Burkina Faso (2003). 18. The Global Fund signs a legal grant agreement with a principal recipient, which is designated by the CCM. The principal recipient receives the financing directly or passes it on to other organizations (subrecipients). 31 however, were at least partially harmonized with the National M&E Plan for HIV/AIDS Control. 4.9 In terms of M&E implementation, the end-of-project self-evaluation (ICR) of the World Bank project noted that the M&E component was never fully implemented. There was no such end-of-grant evaluation to determine how well the Global Fund project M&E system was implemented. Although the World Bank and Global Fund projects both produced regular standardized monitoring products, only the Global Fund monitoring products appear to have been used beyond implementation supervision for policy/program purposes. It is not clear to what extent this was useful, however, given the lack of validated indicators, which measured only outputs, and low targets. 4.10 Neither of the ratings assigned to the World Bank or Global Fund projects appear to be completely objective or clearly justified. For example, in the World Bank project, the first 10 monitoring reports (ISRs) covering the first two years of the project consistently awarded the project a satisfactory rating in the absence of any data on monitoring indicators other than two output indicators. Furthermore, the monitoring reports found that 80 percent of the funds of the largest component (Line Ministries Work Programs) were allocated to travel and per diems for ―awareness-raising‖ activities among the staff of the ministries. Subsequent ISRs downgraded to moderately satisfactory. The final rating in the ICR, however, was moderately unsatisfactory. 4.11 Upon completion of the project, the World Bank produced a critical evaluation (ICR) that exposed shortcomings in the design and implementation of the project but also highlighted its positive contributions. Although the monitoring system (ISRs) consistently rated the project ―satisfactory‖ or ―moderately satisfactory,‖ the end-of-project self- evaluation (ICR) rated the project as ―moderately unsatisfactory‖ overall. No comparable end-of-project evaluation was conducted for the Global Fund grant. A country ―impact evaluation‖ was completed as part of the Global Fund FYE. This study did not attempt to attribute impact specifically to the Global Fund, however. Furthermore, few stakeholders interviewed in Burkina Faso had seen the report for Burkina Faso or the FYE Synthesis Report. 4.12 Although both the World Bank and Global Fund projects had weak M&E systems in the Burkina Faso case, the linear and cumulative M&E process in the World Bank project resulted in an evaluation that drew overall conclusions about the effectiveness of the project, exposed both achievements and shortcomings, and generated lessons for future projects. The only documentation available to assess the effectiveness of the Global Fund project is the performance of the grant against targets reported in the GPR. It is not possible to assess the adequacy of the targets and whether achieving them is in fact a significant contribution to meeting the overall objectives of the grant. How and why results are achieved and obstacles encountered are not discussed, so the overall effectiveness of the grant is difficult to assess. 32 Lesotho WORLD BANK: HIV AND AIDS CAPACITY BUILDING AND TECHNICAL ASSISTANCE PROJECT (HCTA) 4.13 HCTA was a first-of-its kind project explicitly designed to enhance country capacity to absorb a significantly larger amount of resources offered by the Global Fund. The project development objective was to ―Increase the recipient‘s capacity to use effectively the resources provided through the Global Fund grant to support the implementation of HIV and AIDS programs within the recipient‘s territory‖ (World Bank 2009d). 4.14 The World Bank project provided support to build capacity in the Ministry of Health and Social Work, the Ministry of Finance and Development Planning, the National AIDS Commission, and civil society organizations to absorb the Global Fund grant of $29 million that aimed to support the implementation of HIV and AIDS programs. The $5 million project was approved on July 6, 2004 and became effective on January 31, 2005. The project closed on December 31, 2008. The grant was 98 percent disbursed (World Bank 2009a). GLOBAL FUND: STRENGTHENING PREVENTION AND CONTROL OF HIV/AIDS IN LESOTHO 4.15 The project supported by this grant aimed to contribute to the reduction of HIV incidence, particularly among youth, and improve the care and support of people living with HIV and AIDS and those affected by the epidemic. The project sought to improve life skills and peer education training and HIV and AIDS prevention services to adolescent and pre- adolescent young people, with a specific emphasis on girls; promote access to condoms for sexually active youth by installing condom vending machines in youth friendly corners; reduce the number of infants infected by establishing a prevention of mother-to-child transmission program in the 18 health service areas; and provide antiretroviral treatment, care and support for people living with HIV and AIDS.19 4.16 The Principal Recipient was the Ministry of Finance and Development Planning. The total grant amount was US$29.3 million. Phase 1 of the project operated from January 2004 to December 2009, and Phase 2 ended in June 2009. The grant was 99 percent disbursed. M&E COMPARISON 4.17 A detailed comparison of the World Bank and Global Fund project M&E is presented in Appendix E. Since the World Bank project was designed to support the implementation of the Global Fund grant, the M&E frameworks of the two activities were complementary. The frameworks and indicators of both the World Bank and Global Fund projects can be considered inadequate. The Global Fund performance indicators were largely weighted toward outputs, and of the three impact indicators, only one could truly be considered an impact (percent of infants born to HIV-infected mothers who are infected). The World Bank project performance indicators measured disbursement activity of the Global Fund grant but 19. http://portfolio.theglobalfund.org/Grant/Index/LSO-202-G01-H-00?lang=en. 33 did not include any indicators to measure the extent to which the GF grant was used effectively. 4.18 The implementation of the M&E systems has been considered a failure in both activities, as reported by both the Global Fund GPR and the IEG PPAR. Although it was the mandate of both projects to coordinate data collection activities, integrate with the national M&E system and strengthen it, this was never achieved. In addition, serious concerns about data quality were raised in the Global Fund GSC. 4.19 Overall the M&E system in Lesotho remains fragmented. M&E planning and design took place at a sophisticated level, but those plans were grossly mismatched with the capacity of the staff and systems implementing them. HIV/AIDS data, financial management data, TB data and human resources data all still run as vertical systems (Macro International 2008b). Redundant systems at the district level place huge burdens on limited staff. As the country‘s M&E capacity was strengthened, several parallel systems were created without proper plans for coordination or integration (IEG 2010c). 4.20 As in the case of Burkina Faso, both the World Bank and Global Fund projects had ineffective M&E systems, but only the World Bank conducted end-of-project evaluations (both the self-evaluation in the ICR and the independent evaluation in the PPAR), which documented and exposed these weaknesses and drew conclusions about the overall performance of the project with lessons for future activities. The overall effectiveness of the Global Fund project is difficult to assess with available M&E documentation. Russian Federation WORLD BANK: TB/AIDS CONTROL PROJECT 4.14 The TB/AIDS Control Project was the fourth World Bank-financed health sector project in the Russian Federation. The development objectives of the project were to: (1) contain the growth of the epidemics of TB and HIV/AIDS in the short term; and (2) halt and reverse the course of these epidemics in the medium term (World Bank 2003). The project had three components: (a) Component 1—Control of Tuberculosis: (76% of project funds) to (i) improve policies, strategies and protocols; (ii) strengthen surveillance, monitoring, quality control and quality assurance; (iii) improve the detection of TB cases, and (iv) improve TB treatment. (b) Component 2—Control of HIV/AIDS (23% of project funds) to (i) improve policies, strategies and public information for HIV/AIDS control; (ii) strengthen surveillance and monitoring (iii) improve laboratory services and blood safety, (iv) prevent and control STIs; (v) deliver preventive services against HIV/AIDS, with emphasis on high-risk groups, and (vi) prevent the transmission of HV from mothers to children. 34 (c) Component 3—Project Management, Monitoring and Evaluation (1% of project funds) to (i) support the operation of the Project Implementation Unit (PIU) in the Russian Health Care Foundation (RHCF); (ii) provide training and study tours for RHCF; (iii) finance project audits, and (iv) ensure that monitoring and evaluation was carried out as appropriate. 4.21 The project had a total budget of $150 million, which was revised to the final disbursed amount of $110.3 million. The project, which was implemented by the Ministry of Health and Social Development, became effective in December 2003 and closed in April 2009. GLOBAL FUND: STIMULATING AN EFFECTIVE NATIONAL RESPONSE TO HIV/AIDS IN THE RUSSIAN FEDERATION 4.22 The project supported by this grant aimed to improve the knowledge of youth and the general population about HIV/AIDS and promote prevention among specific vulnerable groups including injecting drug users, sex workers, street children, men who have sex with men, and prisoners. In addition, a treatment, care, health promotion and social support module aimed to address the needs of people living with HIV/AIDS. The project also included an advocacy module to improve the participation and effectiveness of government counterparts in HIV/AIDS policy.20 4.23 The Principal Recipient of the grant was the NGO Open Health Institute, and the activities were implemented by a wider NGO consortium. Phase 1 of the project started in August 2004 and ended in August 2006. Phase 2 is scheduled to end in December 2011. The total grant amount was $109.2 million. 4.24 At the same time there was also a tuberculosis project financed through the Global Fund—Tomsk Oblast Comprehensive Strategy to Contain and Control Tuberculosis. This project was much smaller at only $11 million, but the performance framework included more than 50 indicators. The M&E system for the Global Fund TB project is not reviewed for this study because of its small geographic coverage. M&E COMPARISON 4.25 A detailed comparison of the World Bank and Global Fund project M&E is presented in Appendix F. The objectives of both the World Bank and Global Fund projects were clearly oriented toward having an impact on the dynamics and effects of the TB and HIV/AIDS epidemics in Russia. As such, the M&E frameworks and indicators included intermediate outcome and impact indicators. The Global Fund M&E framework, however, also included a detailed set of output measures reflecting the specific activities funded by the grant. (for example, reaching people with treatment and improving treatment structures). 20. Program Grant Agreement between the Global Fund to Fight AIDS, Tuberculosis and Malaria and Partners in Health, the Russian Federation (2004). 35 4.26 Both the World Bank and Global Fund project M&E systems generated routine monitoring products to meet the bureaucratic requirements of the respective institutions. They both also generated supplemental reports, however, that provided additional information or synthesized results. These supplemental products appear to have been more useful for project management and policy purposes than the routine products. In both cases, the performance indicators themselves seemed to provide little value for either project management or policy purposes, but the more analytical reports and syntheses proved to be valuable (World Bank 2009b; IEG 2010d). 4.27 At least for the TB component, the World Bank project M&E system was fully integrated with the national M&E system. M&E was used under the project not only as a management tool for implementation activities but also to inform policy, particularly for assessing the evolution of the epidemic (World Bank 2009d). For the Global Fund project, the M&E system appears to have been established in parallel to generate indicators for the Global Fund performance-based funding requirements. 4.28 Both the World Bank and Global Fund projects appear to have had well functioning M&E systems that were useful beyond project/grant implementation management. As in the previous two cases, however, only the World Bank conducted an end-of-project evaluation that documented the implementation experience, conducted an objective and systematic assessment of overall performance of the project, and generated lessons for future activities. The overall effectiveness of the Global Fund project is difficult to assess with available M&E documentation. 5. Comparison of the World Bank and Global Fund M&E Approaches Objectives and Approach 5.1 Differences in the objectives of M&E between the World Bank and the Global Fund lead to differences in their approaches and how conclusions are generated and interpreted. These differences are summarized in Table 4, which is based on the conclusions from the three country case studies included in this review. 5.2 The World Bank approach to M&E is designed to generate a traceable pathway from objectives all the way to impacts — project M&E forms the foundation and monitoring inputs are aggregated to produce evaluation conclusions. This M&E approach leaves a ―paper trail‖ that leads to more objective conclusions about the effectiveness of project inputs and activities. In practice, however, the paper trail often breaks down during project implementation. ICRs in a number of projects state that they were not able to trace indicator performance. But this trail is picked up again with the ICR, and particularly in IEG‘s ICR Review and in its independent evaluation (PPAR), when the latter is conducted. 5.3 The World Bank approach is grounded in the Bank‘s business model and its role as both a lending institution and also as a technical assistance institution that supports its borrowers in the implementation and supervision of its projects. It is critical for M&E to both 36 remove bottlenecks for disbursement and generate lessons for implementation. From the World Bank M&E documents, it is clear that there is value placed on learning from unsuccessful efforts. 5.4 The Global Fund‘s approach to M&E divides along two very different objectives: (1) to inform funding decisions for individual projects/grants; and (2) to demonstrate the effectiveness and impact of the agency to generate continued high levels of funding. The Global Fund approach is grounded in its business model and its role as only a financing agency. It is critical for M&E to remove bottlenecks for disbursement and demonstrate impact to continue to generate large volumes of funding. Both objectives create an incentive to highlight success rather than expose failure. 5.5 Grant-level M&E for performance-based funding is highly systematized and functions well for its purpose. The monitoring products are produced regularly and considered to be of adequate quality. However, the value of Global Fund monitoring products as a management and policy dialogue tool, or even for the assessment of overall grant effectiveness, has been questionable. The Global Fund ratings used for disbursements are based mainly on outputs, and the incentive is to set targets low (for example, 6,400 people treated with antiretroviral (ARV) drugs by the end of Year 4 in Burkina Faso‘s $10 million grant) and ensure achievement rates are high (often over 100% on average). Assessment of outcomes and impacts is done as part of the funding renewal process in Phase 2 when data are available. 5.6 The second objective of M&E to demonstrate the impact of the agency to generate funding creates the incentive to overstate achievements. The Global Fund appears to be acknowledging this and is making attempts to examine its activities more critically. The approach to M&E at the Global Fund level is evolving and changing frequently. Nonetheless, the approach still lacks a conceptual foundation, so there is no systematic pathway for drawing conclusions about the effectiveness of Global Fund activities. 37 Table 4. Summary of World Bank and Global Fund M&E Comparison M&E Characteristic World Bank Global Fund Underlying objective ï‚· Remove bottlenecks to disbursement ï‚· Remove bottlenecks to disbursement stemming from business ï‚· Learn lessons for future implementation ï‚· Demonstrate impact to secure future model funding Integrates M&E of project performance with (Largely) separates M&E of project/grant overall institutional performance performance from M&E of overall institutional performance Link between M&E and Does not link M&E results to funding Links funding decisions to M&E results financing decisions (except in rare cases when projects are canceled) M&E design M&E design process includes getting M&E design process includes getting stakeholder buy-in? stakeholder buy-in? Indicators: too many, lack validity, and do not Indicators: too many, lack validity, and do always fit into a logical framework not always fit into a logical framework But this may be changing with better standardization of valid indicators (“Top Tenâ€?) Data sources often not identified in advance Data sources often not identified in advance M&E implementation and Regular monitoring products are more a Regular monitoring products are more a utilization bureaucratic tool than a programmatic tool bureaucratic tool than a programmatic tool But this is may be changing as grantees get more experience and report using PBF data for project purposes Project monitoring systems are often seen as Monitoring progress is of dubious value for useless for program management program purposes because of the incentive to set low targets and report high achievement rates Use of monitoring tools to assign Use of monitoring tools to assign performance rating during implementation is performance rating during implementation not transparent. is moderately transparent. Evaluation Internal, independent evaluations, like the External, independent evaluations, like the 2009 HNP evaluation, are conducted FYE, are conducted Standardized evaluation process No standardized evaluation process Independent assessments of project Project/grant performance is not foundation performance are the foundation of evaluation of evaluation Critical evaluations are frequent at the Critical evaluations are infrequent at the project level. grant level. Link to National M&E Stated policy is to use and/or strengthen Stated policy is to use national data and systems national data and M&E systems M&E systems Often provide resources and technical Rarely provide resources and technical assistance to strengthen national M&E assistance to strengthen national M&E systems systems Do not typically establish parallel, disease- Typically do establish parallel, disease- specific information and M&E systems specific information and M&E systems Source: IEG. 38 Design, Implementation, and Utilization 5.7 Both World Bank projects and Global Fund grants suffer from weak M&E design at the beginning of projects/grants. There is a particular problem regarding performance indicators. Typically there are too many indicators, they lack validity, and they often do not fit into a logical framework of inputs, outputs, outcomes, and impacts. The Global Fund has attempted to address the inadequacy of performance indicators with better standardization using valid indicators from the ―Top Ten‖ list, but these indicators are often not routinely available in the countries. 5.8 Neither the World Bank nor the Global Fund has been successful at identifying data sources up front. The World Bank and Global Fund project M&E products reviewed for this study show that indicators related to outcomes and impacts continue to be difficult to report due to inadequate data sources in many of the countries. In general, performance indicators have been of only modest value for assessing project/grant performance or for policy dialogue purposes. IEG has rated project-level M&E as modest or negligible (as opposed to substantial or high) in 76 percent of HNP projects closing since 2006, consistent with the findings of the three projects examined in depth in this study in Burkina Faso, Lesotho, and Russia. This compares with the Bank-wide average of 67 percent modest or negligible for all Bank-financed projects closing since 2006 (Figure 5).21 5.9 Data quality has been an issue, inhibiting the value of M&E products and processes for both agencies. This is being addressed by the World Bank and Global Fund both by tools and investments to strengthen national M&E structures and systems overall, and by the Global Fund with a specific data quality strategy that involves more rigorous verification of data used for performance-based funding. 5.10 The World Bank and Global Fund both generate M&E products that often serve agency-specific objectives rather than feeding into a broader process of system strengthening and policy dialogue. Both the World Bank and Global Fund M&E products are more useful for wider purposes when they are supplemented with other analysis and results are synthesized and interpreted more broadly. From the projects and grants reviewed, this is done more frequently in World Bank projects. There are also examples of more analytical M&E in Global Fund projects (for example, the Russia Central Public Health Research Institute database used for M&E of the HIV/AIDS grant). Evaluation 5.11 The World Bank has a highly refined and standardized end-of-project evaluation process combining internal self-evaluation with independent review. The clear guidelines and criteria for assigning ratings make the process objective. The emphasis on learning from implementation has created a culture of acceptance of critical evaluations. The overall outcomes of 38 percent of Bank-financed HNP projects approved since 1997 have been rated 21. IEG started rating the quality of project M&E in July 2007 for projects closing in fiscal year 2006, in an effort to raise incentives for implementing M&E. The self-evaluation (ICR) prepared by the project team discusses M&E but does not rate it. 39 moderately satisfactory or worse (Figure 5). This compares to 21 percent of all World Bank- financed projects during the same time period.22 Figure 5. Selected Ratings of World Bank-Financed HNP Projects, in Comparison with Bank-wide Averages HNP Projects All World Bank Projects 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Outcome ICR Quality M&E Quality Source: World Bank databases. See Appendix G. Note: For project outcomes, percent of projects rated ―moderately satisfactory‖ or better. For ICR quality, percent of Implementation Completion Reports rated satisfactory or exemplary. For M&E quality, percent of project M&E systems rated substantial or high, as opposed to modest or negligible. 5.12 The traceable pathway from project inputs/activities to impacts made it possible for IEG to complete the 2009 evaluation of the World Bank Group‘s support to Health, Nutrition, and Population based on cumulative self-evaluations and independent reviews of individual project outcomes. The conclusions of the evaluation reflect the aggregate performance of projects, which does not lend itself to re-interpretation and subjective conclusions. Given the real challenges that have been faced by the complex nature of World Bank HNP projects in challenging environments, and the willingness to rate projects as unsatisfactory, it was unavoidable that the evaluation based on cumulative project performance did not paint a very positive picture. 5.13 The Global Fund has lacked a systematic approach to evaluation that is grounded in a conceptual framework with an identified pathway from objectives/inputs to impacts. For 22. IEG has rated 92% of ICRs in the HNP sector as satisfactory or better, compared with 90% for all Bank- financed projects, notwithstanding the deficiencies in project M&E during the implementation of the projects. 40 example, the Global Fund FYE was not based on the four-tiered corporate performance framework, and there was no systematic approach to linking the project/grantee performance and the overall impact of the Fund as an agency. Furthermore, the evaluation was not based on cumulative internal/external evaluations. The lack of a framework and cumulative assessment of grant performance made it possible to draw conclusions that were not necessarily supported by objective criteria. The study exposes many shortcomings of the operations, performance and outcomes of the Global Fund activities, but the overall conclusion was positive. The new Global Fund Evaluation Strategy is a step in the direction in strengthening the conceptual foundations of evaluation at all levels of Global Fund activities and in better aligning them with country systems. 5.14 Evaluation activities in the Global Fund have previously had a conspicuous gap with the lack of evaluation at the completion of individual grants. There was no policy or process within Global Fund M&E to determine the overall effectiveness of grants or to generate lessons for future Global Fund activities in the country or in other programs. It is expected that this gap will be filled with the use of periodic reviews under the single streams of funding, as well as the national program reviews and program evaluations planned under the new evaluation strategy. Link to National M&E Systems 5.15 This review has shown that project/grant M&E is most useful when it is completely and successfully integrated with national M&E systems and coordinated with other international health partners. Although both the World Bank and Global Fund have stated policies to use existing national data sources and M&E systems to the extent possible, this review of projects of both agencies in three countries found that the World Bank has been more effective in achieving such integration of project M&E with national systems. 5.16 World Bank health projects often dedicate entire components or sub-components to strengthening national M&E systems, with significant resources and technical assistance, and project M&E is designed to be integrated into those activities. Although integration with national systems is not always achieved, the World Bank rarely has been criticized for establishing parallel and disease-specific information and M&E systems. 5.17 The way Global Fund grants are structured (vertically along specific diseases), the objectives of M&E at the grant level (to support performance-based funding), and the wide array of grant recipients and sub-recipients, often from the non-governmental sector, has made it difficult to achieve integration of grant M&E with national systems. The Global Fund has been criticized for establishing parallel and disease-specific information and M&E systems. 5.18 The most recent Global Fund progress report shows that only 4 percent of grant expenditures were allocated to M&E up to 2008, although the Global Fund recommends that 5–10 percent of proposal funding be allocated to M&E (Global Fund 2010e). The share of this funding that has been allocated to strengthening national systems cannot be estimated, but it is likely to be small given the demands of grant-specific performance-based funding. Technical assistance has been funded, however, in some cases (for example, M&E advisor to 41 the MOHSW in Lesotho), and investments in M&E are also made through the Health Systems Strengthening grant window, so the contribution of the Global Fund to strengthening national M&E systems may be under-estimated. 5.19 The new single-stream grant architecture, consolidated periodic reviews, and the new Global Fund Evaluation Strategy are all focused on better streamlining many aspects of grant implementation and management and better aligning with country systems, including monitoring and evaluation. The new national program reviews and program evaluations will provide an opportunity or the Global Fund to support and actively participate in multi-partner national program reviews conducted by national disease programs, which may also serve to strengthen national M&E systems and drive better integration of Global Fund M&E with national systems. 6. Conclusions and Lessons Learned 6.1 This review has demonstrated that it is possible, and necessary, for the M&E activities of the World Bank and the Global Fund to contribute to strengthening national systems and generating lessons for improved project/grant management and more effective health sector activities overall. Some lessons have emerged that may contribute to the ongoing process of identifying good practices for developing M&E policies for global health projects, setting up M&E frameworks, planning and programming evaluations, and using M&E results more effectively to manage programs and strengthen the health policy process in partner countries. 6.2 Both the World Bank and the Global Fund have made notable progress toward more systematic M&E in their global health activities. The investment in M&E, however, has tended to be heavy on up-front design, such as developing M&E frameworks and plans and selecting indicators, with insufficient attention to strengthening underlying data systems and analytical processes. In general, the performance indicators used to monitor projects/grants by both agencies have provided little added value for assessing project/grant performance, for contributing to periodic summative evaluations, or for enhancing policy dialogue. Good monitoring systems do all three—assess progress in implementing activities, facilitate a cumulative assessment of project performance, and identify issues that require policy responses and other solutions beyond the scope of the projects. As a result, assessments of the effectiveness of inputs and activities supported by these agencies can be in danger of being subjective and politicized. In the worst-case situations, the M&E systems have placed an added burden on fragile health sector management structures and information systems and have detracted efforts away from program implementation and service delivery (Biesma and others 2009). 6.3 The M&E systems of both agencies reviewed for this study, both the successful and less successful examples, point to several key lessons. Lesson 1: Avoid complicated, burdensome M&E systems. 6.4 Both the World Bank and Global Fund have elaborate M&E guidelines and approaches, but what is feasible on the ground is quite simple and limited. Both agencies 42 could help drive a movement to streamline, focus and concentrate M&E systems and, for example, to use more limited data and indicators as a focal point to strengthen data quality, HMIS and M&E. Lesson 2: Emphasize analytical processes rather than indicators and targets. 6.5 There has been an over-emphasis in both agencies on indicators and targets rather than on processes for using M&E results. For example, the Global Fund FYE has a large number of recommendations focused on strengthening health information and streamlining monitoring systems, but no recommendations to make better use of M&E (Macro International 2009b). Both the World Bank and Global Fund M&E systems have been more useful when they are process-oriented and supplemented with other analyses, and when results are synthesized and interpreted more broadly. Lesson 3: Work toward integrating and strengthening national M&E structures. 6.6 This review has again demonstrated that project/grant M&E is only useful beyond narrow project/grant supervision when it is integrated with national M&E systems and coordinated with other international health partners and supported by technical assistance. Most countries are reluctant to borrow or use grant funds to specifically support M&E systems or conduct costly evaluations. The global health community could consider funding these efforts in creative ways to ensure that both the costs and benefits are shared more widely. Monitoring systems that provide timely information in a useful format and impact evaluations using rigorous methods are public goods with benefits that extend well beyond initiative-specific assessment. Lesson 4: Find a balance between demonstrating agency-specific effectiveness and combined impact. 6.7 There is an inherent tension between the need for global health agencies to demonstrate their own impact and value for money, and the acknowledgment that any progress toward broad-based objectives such as poverty reduction or the MDGs will reflect concerted action of many national and international actors. Disproportionate concern with demonstrating initiative-specific successes can distort M&E approaches and generate misleading conclusions that provide little value for better understanding which initiatives and strategies are most effective. More consistent approaches are needed to identify the complementary contributions of different agencies and initiatives toward achieving broad- based goals and to measure their individual and combined impact. Integrated sector-wide M&E frameworks are needed that can serve the multiple purposes of assessing the performance of specific activities, strengthening national data and reporting systems, and monitoring the impact of the combined effort on broader national objectives. 6.8 The Global Fund, the World Bank, and other agencies have endorsed the ―Three Ones‖ principles of a common action framework with a single coordinating authority and one monitoring and evaluation framework to monitor collective efforts in each disease area. They have jointly prepared an M&E Toolkit in 2004 (revised in 2006, 2009, and 2011) to establish norms and identify indicators to be used by all the agencies, but these have been difficult to 43 achieve in practice because each agency has its own project-level M&E requirements. Both the Global Fund and the World Bank could contribute to improved M&E at the project and country levels by making a stronger commitment to the ―Three Ones‖ principles. Project- level M&E could focus on accountability for achieving the specific outputs of each project and country-level M&E on tracking the higher level outcomes and impacts collectively. 44 References Atun R, Lazarus J, Van Damme W, and Coker R (2010). Interactions between critical health system functions and HIV/AIDS, tuberculosis and malaria programmes. Health Policy and Planning 25(suppl 1): 11–13. Biesma R, Brugha R, Harmer A, Walsh A, Spicer N, and Walt G (2009). The effects of global health initiatives on country health systems: A review of the evidence from HIV/AIDS control. Health Policy and Planning 24(4): 239–252. Center for Global Development (2006). When will we ever learn? Improving lives through impact evaluation. Report of the Evaluation Gap Working Group. Washington, D.C.: May 2006. Chan M, Kazatchkine M, Lob-Levyt J, Obaid T, Schweizer J, Sidibe M, Veneman A, and Yamada T (2010). Meeting the demand for results and accountability: a call for action on health data from eight global health agencies. PLOS Medicine 7(1): 1–4. Desai M, Rudge J, Adisasmito W, Mounier-Jack S and Coker R (2010). Critical interactions between Global Fund-supported programmes and health systems: a case study in Indonesia. Health Policy and Planning 25 (supplement): i43–i37. Euro Health Group (2007). Evaluation of the local fund agent system of the Global Fund to Fight AIDS, Tuberculosis, and Malaria. Euro Health Group: Soborg, Denmark. Faiola A. (2009). Some World Bank health programs ineffective, report says. Washington Post. May 1, 2009. Global Fund (2011a). Grant Performance Report . Lesotho LSO-202-G01-H-00 http://www.theglobalfund.org/grantdocuments/2LSOH_246_197_gpr.pdf. Global Fund (2011b). Global Fund evaluation strategy 2012–2016. Sixteenth Policy and Strategy Committee Meeting. Geneva 24–26 October 2011. GF/PSC16/05. Global Fund (2011c). Global Fund strategy 2012–2016. Twenty-fifth Board Meeting. Accra, Ghana 21–22 November 2011. GF/B25/7. Global Fund (2011d). Monitoring and evaluation toolkit. HIV, tuberculosis, malaria and community systems strengthening. The Global Fund M&E requirements. Geneva: The Global Fund. Global Fund (2010a). The Global Fund 2010 innovation and impact. Geneva: The Global Fund. Global Fund (2010b). The Global Fund data quality strategy 2010–2013. Geneva: The Global Fund. Global Fund (2010c). Key performance indicators: Year-end report on results for 2009 and proposed modifications for 2011. Global Fund (2010d). M&E plan guidelines. Geneva: The Global Fund. Global Fund (2010e). Technical Evaluation Reference Group for the Global Fund to fight AIDS, Tuberculosis and Malaria: Terms of reference, membership and procedures. December 2010. 45 Global Fund (2009a). Monitoring and Evaluation Toolkit. Part 1: The M&E system and Global Fund requirements. Third edition. Geneva: The Global Fund. Global Fund (2009b). Monitoring and Evaluation Toolkit. Part 2: Tools for monitoring programs for HIV, tuberculosis, malaria and health systems strengthening. Third edition. Geneva: The Global Fund. Global Fund (2009c). Grant Performance Report . Burkina Faso BUR-202-G01-H-00 http://www.theglobalfund.org/grantdocuments/2BURH_203_205_gpr.pdf. Global Fund (2009d). Scaling up for impact results report. Geneva: The Global Fund. Global Fund (2008a). Monitoring and Evaluation Toolkit. Data quality audit tool: guidelines for implementation. Third edition. Geneva: The Global Fund. Global Fund (2008b). Report of the Policy and Strategy Committee. Global Fund 17th Board Meeting. April 28–29, 2007. Geneva. Global Fund (2005). Sustaining performance, scaling up results: third progress report 2005. Geneva: The Global Fund. Global Fund (2004). Monitoring and evaluation operations plan for the Global Fund 2004. Geneva, February 2004. Hanvoravongchai P, Warakamin B, and Coker R (2010). Critical interactions between Global Fund-supported programmes and health systems: a case study in Thailand. Health Policy and Planning 25 (supplement): i53–i57. IEG (Independent Evaluation Group). (2010a). Approach paper: Global Program Review of the Global Fund to Fight AIDS, Tuberculosis and Malaria. Independent Evaluation Group (IEG). Washington, DC. IEG. (2010b). Burkina Faso Country Report for the Global Program Review: The Global Fund to Fight AIDS, Tuberculosis and Malaria. Independent Evaluation Group (IEG). Washington, DC. IEG (2010c). Project Performance Assessment Report: Lesotho Health Sector Reform Project and HIV and AIDS Capacity Building and Technical Assistance Project. Sector Evaluations (IEGSE). Independent Evaluation Group (IEG). Washington, DC. IEG (2010d). Russian Federation country report for the Global Program Review: The Global Fund to Fight AIDS, Tuberculosis and malaria. Independent Evaluation Group (IEG). Washington, DC. IEG. (2010e). Tanzania Country Report for the Global Program Review: The Global Fund to Fight AIDS, Tuberculosis and Malaria. Independent Evaluation Group (IEG). Washington, DC. IEG. (2009a). Improving effectiveness and outcomes for the poor in health, nutrition, and population: an evaluation of World Bank Group support since 1997. Independent Evaluation Group (IEG). Washington, DC. IEG (2009b). 2009 Award Winners: Kyrgyz Republic Health Sector Reform Projects I and II. 46 http://web.worldbank.org/external/default/main?noSURL=Y&theSitePK=1324361&pagePK =64253958&contentMDK=22327748&piPK=64252979. IEG (2006). 2005 Annual Report on Operations Evaluation. Washington, DC: World Bank. The Lancet (2010). Editorial—Evaluation: the top priority for global health. The Lancet 375: 526. Macro International Inc. (2009a). Evaluation of the organizational effectiveness and efficiency of the Global Fund to Fight AIDS, Tuberculosis, and Malaria. Results from Study Area 1 of the Five-Year Evaluation. Submitted to the Global Fund to Fight AIDS, Tuberculosis, and Malaria Geneva. Macro International Inc. (2009b). The Five-Year Evaluation of the Global Fund to Fight AIDS, Tuberculosis, and Malaria: Synthesis of Study Areas 1, 2, and 3. Submitted to the Global Fund Geneva. Macro International Inc. (2009c). The Global Fund Five-Year Evaluation: Study Area 3—the impact of collective efforts on the reduction of the disease burden of AIDS, tuberculosis, and malaria. Submitted to the Global Fund to Fight AIDS, Tuberculosis, and Malaria. Geneva. Macro International Inc. (2008a). Evaluation of the Global Fund partner environment, at global and country levels, in relation to grant performance and health systems effects, including 16 country studies. Final report ubmitted to the Global Fund to Fight AIDS, Tuberculosis, and Malaria Geneva. Macro International Inc. (2008b). Multi-country evaluation study: health impact of the scale- up to fight AIDS, Tuberculosis, and Malaria with special reference to the Global Fund. Country Impact Evaluation Report: Lesotho. Submitted to the Global Fund to Fight AIDS, Tuberculosis, and Malaria Geneva. McKinsey & Company (2005). Global health partnerships: assessing country consequences. Bill and Melinda Gates Foundation. Ministry of Health of the Republic of Moldova (2007). Monitoring and evaluation plan for TB component to Round 6 GFATM grant: strengthening tuberculosis control in the Republic of Moldova. Mounier-Jack S, Rudge J, Phetsouvanh R, Chanthapdith C, and Coker R (2010). Critical interactions between Global Fund-supported programmes and health systems: a case study in Lao People‘s Democrtic Republic. Health Policy and Planning 25 (supplement): i37–i42. Nahlen B, and Low-Beer D (2007). Building to collective impact: the Global Fund support for measuring reduction in the burden of malaria. American Journal of Tropical Medicine and Hygiene 77(Suppl 6): 321–327. Office of the Inspector General (2009). Report on lessons learnt from the country audits and reviews undertaken. Geneva: The Global Fund to Fight AIDS, Tuberculosis and Malaria. Peersman G, Rugg D, Erkkola T, Kiwango E, Yang J (2009). Are the investments in national HIV monitoring and evaluation systems paying off? Journal of Acquired Immune Deficiency Syndrome 52(suppl 2): S87–s96. 47 Program Grant Agreement between The Global Fund to Fight AIDS, Tuberculosis and Malaria and the Ministry of Finance and Development Planning of the Government of the Kingdom of Lesotho (2004). Program Grant Agreement between The Global Fund to Fight AIDS, Tuberculosis and Malaria and Partners in Health. The Russian Federation (2004). Program Grant Agreement between The Global Fund to Fight AIDS, Tuberculosis and Malaria and the United Nations Development Programme. Burkina Faso (2003). Rudge J, Phuanakoonon S, Nema K, Mounier-Jack S, and Coker R (2010). Critical interactions between Global Fund-supported programmes and health systems: a case study in Papua New Guinea. Health Policy and Planning 25 (supplement): i48–i52. Technical Evaluation Reference Group (2006a). 360o stakeholder assessment: perceptions and opinions of stakeholders of the Global Fund. The Global Fund to Fight AIDS, Tuberculosis, and Malaria: Geneva, Switzerland. Technical Evaluation Reference Group (2006b). Review of the Global Fund grant portfolio – funding the right things? The Global Fund to Fight AIDS, Tuberculosis, and Malaria: Geneva, Switzerland. Technical Evaluation Reference Group (2005). Assessment of Country Coordinating Mechanisms performance baseline. The Global Fund to Fight AIDS, Tuberculosis, and Malaria: Geneva, Switzerland. Tragard A and Shrestha I (2010). System-wide effects of Global Fund investments in Nepal. Health Policy and Planning 25 (supplement): i58–i62. White H (2009). Some reflections on current debates in impact evaluation. 3ie Working Paper No. 1 New Delhi: International Initiative for Impact Evaluation. Wilkinson D, Brugha R, Hewitt S, Trap B, Erikson J, Nielsen L, and Weber W (2006). Assessment of the proposal development and review process of the Global Fund to Fight AIDS, Tuberculosis and Malaria: Assessment Report. Submitted by Euro Health Group to the Global Fund. Geneva. World Bank (2010). Access to information: staff handbook. The World Bank: Washington, DC. World Bank (2009a). Implementation completion and results report: Lesotho HIV and AIDS Capacity Building and Technical Assistance Project. Human Development I; Southern Africa Country Department 1; Africa Region. Washington, DC. World Bank (2009b). Implementation completion and results report: Russia TB/AIDS Control Project. Human Development Sector Unit. Europe and Central Asia Region. Washington, DC. World Bank (2009c). Implementation Completion and Results Report: Ukraine TB and HIV/AIDS Control Project. Human Development Sector Unit; Ukraine, Belarus and Moldova Country Unit; Europe and Central Asia Region, Washington, DC. 48 World Bank (2009d). Project Appraisal Document for Lesotho HIV and AIDS Technical Assistance Project. Human Development Sector Unit. South Africa Country Cluster 1, Africa Region. Washington, DC. World Bank (2008). Project Performance Assessment Report: Kyrgyz Republic Health Sector Reform Project and Second Health Sector Reform Project, Washington, DC. World Bank (2007a). OPv 13.60 – Monitoring and Evaluation. World Bank Operational Manual http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/EXTPOLICIES/EXTOPMAN UAL/0,,contentMDK:21345677~menuPK:64701637~pagePK:64709096~piPK:64709108~th eSitePK:502184,00.html. World Bank (2007b). Implementation completion and results report: Burkina Faso HIV/AIDS Disaster Response Project. Human Development I; Country Department AFCW1; Africa Region. Washington, DC. World Bank (2007c). Implementation completion and results report: The Kyrgyz Republic Second Health Sector Reform Project. Human Development Sector Unit. Europe and Central Asia Region. Central Asia Country Unit. Washington, DC. World Bank (2006a). Implementation Completion and Results report: guidelines. Washington, DC: The World Bank Operations Policy and Country Services. Updated February 9, 2007. World Bank (2006b). Lesotho Mission Aide Memoire. Health Sector Reform Program Phase II (Cr. 4118-LSO) and HIV/AIDS Capacity Building and Technical Assistance (Gr. H113- LSO) Project. September 25 – October 6, 2006. World Bank (2006c). Russian Federation Mid-term Evaluation Mission for TB/AIDS Control Project October 1–14, 2006 Mission Report. World Bank (2004). Managing for development results in the World Bank. Progress Report. Operations Policy and Country Services. Washington, D.C. World Bank (2003). Project Appraisal Document for the Russia TB/AIDS Control Project. Human Development Sector Unit. Europe and Central Asia Region. Washington, DC. World Bank (2002). Better Measuring, Monitoring, and Managing for Development Results. Washington, D.C. World Bank (2001). Project Appraisal Document for the Burkina Faso HIV/AIDS Disaster Response Project. Human Development Sector III. Country Department 15. Africa Regional Office. Washington, DC. World Bank (undated-a). Implementation Status and Results Report for Investment Projects. (template). World Bank (undated-b). OED and impact evaluation: a discussion note. Operations Evaluation Department. http://www.worldbank.org/oed/docs/world_bank_oed_impact_evaluations.pdf World Bank (undated-c). Template and Guidelines for the Project Appraisal Document (PAD), Operations Policy and Country Services, Washington, DC. 49 World Health Organization (2009). Russian Federation country profile. Global Tuberculosis Control WHO Report. World Health Organization: Geneva. 51 Appendix A Appendix A. Stakeholder Interviews: Interview Guide Questions for the Field about Grant/Project Monitoring and Evaluation 1. How was the initial M&E framework for the grant/project developed? ï‚· Which stakeholders were involved? ï‚· Was there a ―logframe‖ or explicit pathway from project inputs/activities to results that underlies M&E activities? 2. What documentation is available on grant/project M&E? ï‚· Is there a formal M&E plan? ï‚· If yes, who drafted the plan? ï‚· Who was responsible for implementing the M&E plan — collecting and analyzing indicators, drafting reports? ï‚· To what extent was the plan followed? 3. Where did the data for the monitoring indicators come from? ï‚· To what extent was the grant/project reporting system integrated with existing reporting/information systems? ï‚· Did data come from existing reporting/information systems, or were new reporting systems created? ï‚· What was the cost of establishing/strengthening data and reporting systems for the grant/project? ï‚· Were there problems with timeliness or completeness of reporting? ï‚· Were there improvements in data quality over the period of the grant/project? If yes, to what extent were these improvements been driven by the grant/project? 4. How often were monitoring indicators analyzed/reported and in what form? ï‚· Who was involved in assessing the progress? ï‚· How were results disseminated? 5. How were the monitoring indicators used by managers and policy-makers? ï‚· Was there a systematic process that links results to action (other than performance- based funding in the case of the GF grants)? ï‚· Were results used by others in the health sector beyond grant/project managers? 6. To what extent did the monitoring system and indicators form the basis for evaluation of the grant/project? 7. Did the M&E/data/reporting systems continue to be used beyond the life of the grant/project? Appendix B 52 Appendix B. World Bank Stated Policy on Monitoring and Evaluation http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/EXTPOLICIES/EXTOPMANUAL/0,,contentMDK:2 1345677~menuPK:64701637~pagePK:64709096~piPK:64709108~theSitePK:502184,00.html OP 13.60 – Monitoring and Evaluation These policies were prepared for use by World Bank staff and are not necessarily a complete treatment of the subject. OP 13.60 June, 2007 Note: This OP 13.60 replaces OD 13.60, Dissemination and Utilization of Operations Evaluation Department (OED) Findings, OD 10.70, Project Monitoring and Evaluation, and the following Operational Memoranda: Management Responses to OED Reports, February 2, 1994, and Project Implementation Plan and Performance Monitoring Indicators, January 23, 1996. It draws from Mandate of the Director-General, Evaluation and from Independent Evaluation Group — Terms of Reference (DGO Mandate and Terms of Reference for OED, OEG and OEU Proposed Name Changes ) (CODE2005-0088), September 27, 2005. Questions about OP 13.60 may be addressed to opmanual@worldbank.org 1 1. The Bank’s objective is to assist its borrowing member countries, individually and collectively, to 2 reduce poverty and achieve sustainable growth. To assess the extent to which its efforts and those of borrowers are making progress toward that objective, the Bank monitors and evaluates its operational 3 activities. 2. Monitoring and evaluation provides information to verify progress toward and achievement of 4 results, supports learning from experience, and promotes accountability for results. The Bank relies on a combination of monitoring and self-evaluation and independent evaluation. Staff take into account the findings of relevant monitoring and evaluation reports in designing the Bank’s operational activities. Monitoring and Self-Evaluation 3. Monitoring and evaluation (M&E) requires formulating the expected results of Bank support; selecting indicators of outputs and outcomes; gathering baseline data on outputs and outcomes; setting milestones and a timeline for progress; establishing a system for collecting, analyzing, and reporting data; monitoring progress; evaluating the activity to determine its relevance, efficacy, and efficiency; and establishing a framework for using M&E findings. These elements are tailored to the scale and scope of the operational activity. 5 4. The designs of Bank operational activities incorporate a framework for M&E. The Bank monitors and evaluates its own contribution to results using this framework, relying on the borrower’s M& E systems to the extent possible and, if these systems are not strong, assisting the borrower’s efforts to strengthen 6 them. For Country Assistance Strategies and sector/thematic strategies, the Bank monitors and evaluates progress toward achieving the results identified in the strategy. For lending operations, the borrower monitors progress towards results during implementation and evaluates the achievement of results upon completion; the Bank reviews the borrower’s M&E reporting. For analytic and advisory services, the Bank monitors and evaluates results on completion. 5. In addition to working with borrowers, the Bank works with other development partners to agree on the results expected from development activities and to harmonize monitoring, reporting, and evaluation requirements. 53 Appendix B Independent Evaluation 6. Independent evaluation validates self-evaluation activities, verifies their results, and/or undertakes separate assessments of the relevance, efficacy, and efficiency of Bank operational activities and processes. Independent evaluation is carried out by the Independent Evaluation Group (IEG) under the oversight of the Director-General, Evaluation (DGE), who reports directly to the Board, which approves the DGE’s mandate and IEG’s terms of reference. IEG’s work program is endorsed annually by the Board, following consultations with management. In this context, the DGE is directly responsible for (a) assessing whether the Bank’s programs and activities are producing the expected results; (b) incorporating evaluation assessments and findings into recommendations designed to help improve the development effectiveness of the Bank’s programs and activities, and their responsiveness to countries’ needs and concerns; (c) appraising the Bank’s operations self-evaluation and development risk management system; (d) reporting periodically to the Executive Directors on actions taken by the Bank in response to 7 evaluation findings, and on the measures being taken to improve the overall operations evaluation system including dissemination and outreach activities; and (e) encouraging and assisting developing member countries to build effective monitoring and evaluation associations, capacities and systems. 7. To discharge these functions, IEG has unrestricted access to the staff and records of the Bank. In carrying out its activities, IEG consults with borrower governments, beneficiaries, cofinanciers, and other stakeholders, as well as with operational managers, peer reviewers, and, as appropriate, advisory committees of specialists. IEG endeavors to maintain close contact with Bank staff so that their views are adequately considered in the preparation of IEG reports, and the analyses and findings of these reports are understood. However, IEG reports and findings are not subject to management approval. As an integral ingredient of its independence, IEG discloses its reports in accordance with IEG’s disclosure 8 policy statement approved by the Board. It disseminates evaluation findings within the Bank and the wider development community. 8. In addition, the Independent Evaluation Group (IEG) provides advice and support to operational units engaged in evaluation capacity development, cooperates with other international financial institutions and development assistance agencies to promote evaluation and harmonize evaluation standards, and assists member countries and development partners to develop effective operations monitoring and evaluation capacities and systems. 1. ―Bank‖ includes IBRD and IDA; ―loans‖ includes IDA credits and grants; and ―borrower‖ includes IDA grant recipient. 2. OP 1.00, Poverty Reduction. 3. For the purposes of this OP, the Bank’s operational activities include country strategies, sector and thematic strategies, lending operations, and analytic and advisory services. 4. Results are the outputs and outcomes of operational activities; outputs are the products, goods, services, or actions associated with an operational activity; and an outcome is a direct change in condition or behavior of a target group or institution that results from outputs. 5. See BP 2.11, Country Assistance Strategies; OP/BP 8.60, Development Policy Lending; BP 10.00, Investment Lending: Identification to Board Presentation; BP 13.05 Project Supervision; and BP 13.55, Implementation Completion Reporting. For IDA-financed operations, see Additions to IDA Resources: Fourteenth Replenishment (IDAR2005-0029), March 2005, Section II D, and any adjustments to the results measurement system agreed in future IDA replenishments. For analytic and advisory activities, see guidelines available to staff on the Bank’s intranet. 6. Such support includes, for example, support for strengthening statistical capacity. 7. Bank management comments on draft IEG reports and is responsible for preparing and implementing management responses to major IEG recommendations — those that are tracked in the aggregate annual Management Action Record (MAR), and those marked as ―MAR‖ in Country Assistance Evaluations. 8. IEG’s disclosure policy statement is approved directly by the Board, and then incorporated into the Bank’s overall disclosure policy. Appendix C 54 Appendix C. Global Fund 2004 Monitoring and Evaluation Operational Plan http://www.theglobalfund.org/documents/board/07/gfb78_annex5.pdf The Global Fund to Fight AIDS, Tuberculosis and Malaria Seventh Board Meeting Geneva, 18–19 March 2004 Seventh Board Meeting GF/B7/8 Annex 5 MONITORING AND EVALUATION OPERATIONS PLAN FOR THE GLOBAL FUND 2004 February 2004 A. Introduction The Monitoring and Evaluation Strategy presents an overall framework for the Fund’s M&E work and calls for an operational plan that integrates monitoring and evaluation at program, disease and Fund level. It attempts to build in Fund principles, for example additionality, in monitoring of activities and in how M&E itself works with other partners, and includes: a) Prioritization and implementation of evaluations of key performance areas for the Fund — including scope and timing of specific evaluation studies to be conducted by external evaluators and in collaboration with key partners. b) Operational details for M&E at the program level — including a description of the Secretariat’s review process for periodic Disbursement Reports and Progress Updates, Fiscal Years Progress reports and Requests for Continued Funding. This will focus on the methods envisioned and criteria for determining assessment of data quality, provision of feedback to grantees and ongoing funding of grants. c) Guidance on the selection of indicators for grantees, with particular attention to harmonizing with ongoing global initiatives and building on available tools. d) Indicators for performance monitoring for the Fund as an organisation. Indicators, timelines and targets will be specified for the Fund’s overall performance against its go als and principles as well as for the responsibilities of its different executing parties, including the Fund’s Secretariat. e) Incorporating fund principles, for example additionality, into the monitoring and evaluation of its activities, its impact, and relations with partners A tabular version of the operational M&E plan with detailed objectives, deliverables, time frames and budgets has been developed and is attached. To operationalize the Global Fund’s M&E strategy, the M&E plan follows a functional appro ach, as many of the specific functions and products cut across the different levels described in the strategy. The key objectives and sub objectives of the plan are: 1. Build M&E Capacity and Competence in the Secretariat and Global Fund System: 55 Appendix C - Build capacity and competence of the Strategic Information and Measurement (SIM) Team - Support Portfolio Teams in Implementing the Performance Based Funding System - Support Programme Level M&E – Link Performance with Disease Impact 2. Ensure Independence and Soundness of Approaches – Technical Evaluation Reference Group (TERG) 3. Establish the Information Platform of the Global Fund 4. Address Key Evaluation Questions: - Assess the Organizational Architecture of the Global Fund - Assess Strategies, Approaches and Results - Assess broader evaluation questions including the Global Fund’s potential effects in disease control and mitigation activities in promoting development 5. Monitor Global Fund Performance as an Organisation Guiding Principles for the Development of the M&E Plan • The operational plan serves as the guiding document and work plan for the Strategic Information and Measurement Unit (SIM). It describes how the Global Fund intends to operationalize its M&E strategy across its key functional areas. Furthermore, it seeks to define clear, realistic priorities and activities to be carried out with available human and financial resources. It is organized around functional monitoring and evaluation tasks, making a distinction between programme, country and global level M&E. • The Global Fund’s M&E strategy builds on existing programme, country and global level capacity, and will wherever possible utilise available tools, systems and expertise at country level and among global partners. Activities in the work plan will also be carried out in close collaboration and consultation with relevant stakeholders. • The M&E strategy makes a distinction between monitoring and evaluation. The definition of indicators, baselines and targets, and the measurement of activities and immediate results (monitoring) will build on and utilize existing systems established for monitoring of HIV/AIDS, malaria and tuberculosis. The Global Fund needs to access and systematize such data and information for its own internal use and will only strengthen data collection systems if required. In that sense, the Global Fund is primarily a user of monitoring information. In addition to data collection systems, the Fund will strongly support countries and partners to ensure that information collected is analyzed and used routinely for programme adjustments and planning, and is of adequate quality for these purposes. • In the area of evaluation, the M&E strategy specifies the requirements for programme and country level evaluations, which should be addressed by country partners. However, critical information needs, which are not currently being addressed, will be supported by the Fund, as feasible per resource levels, in coordination with other partners. There are, however, a number of evaluation questions which are crucial and unique in the assessment of Global Fund policies, tools and approaches — questions about their Appendix C 56 relevance, effectiveness and sustainability at global and country level (the organisational architecture, principles of performance based funding, additionality, national ownership, system wide effects, etc.). The MEFA Committee is charged with identifying these critical evaluation needs, supporting at least one external evaluation per year. The Strategic Information and Measurement Unit has a special responsibility for preparing, initiating and coordinating studies and evaluations of strategic questions for the Fund. The implementation of evaluation studies generally will be through external partners and/or in collaboration with relevant stakeholders. Resources of the Global Fund will be catalytic in nature and co- funding will be sought from other sources in many cases. • The current plan covers a two year period with more specific activities for the first year and indicative activities for the second year. A precondition for successful and implementation of the plan (and effective measurement processes) is to establish an effective Unit — recruit and introduce a sufficient number of qualified staff and consultants to organize and carry out the work. It also requires the establishment and strengthening of working relations with other evaluation and monitoring organisations. • Top priorities in 2004 will include: - To review and consolidate the internal monitoring systems — the flow of data and information between the various functional entities of the Fund (including principal recipients. LFAs, CCMs and the Fund Secretariat) and develop or adapt transparent systems for effective management and utilization of information (MIS). - To establish a systematic approach with well-defined criteria, roles and responsibilities of pertinent actors for the review of grant performance through periodic Disbursement Reports and Progress Updates, prior to the end of the first two-year funding period. - To define the disbursement criteria for the Requests for Continued Funding, which will soon begin to be processed, and any other elements of the system through which these requests will be processed. - Building on existing efforts and together with relevant partners, to finalize an M&E toolkit including reference to key indicators, data quality assurance and measurement tools at process, coverage and impact levels. - To finalize TOR for the Technical Evaluation Reference Group (TERG), select members and have the TERG fully functional. ï‚· A number of important evaluation studies will also be carried out during the year and others identified for later implementation. Since some of the activities in the plan last for more than one year and others are ongoing, the M&E strategy and work plan requires a multi-year approach. ï‚· There are an increasing number of studies initiated by Global Fund stakeholders (donors, NGOs, etc.) which provide useful information and complement the M&E work initiated and carried out by the Global Fund. The Global Fund has not always been able to include its own concerns and questions in such studies. MEFA has been tasked to define and coordinate mechanisms through which topics for new studies are reviewed and collaborative consultations take place to define the contents of these. 57 Appendix C SIM will support MEFA in this function and will work closely with and seek necessary and appropriate support from stakeholders when planning evaluations of relevance to the Global Fund. The following section presents and explains briefly the main components of the plan. B. Main Components of the M&E Operational Plan 1. Building Capacity and Competence in M&E within the Secretariat and the Global Fund System An immediate priority is to build the capacity and competence of the SIM team as the core M&E competence of the Fund. Terms of reference covering an appropriate mix of skills have been developed, with a total of three fixed term staff members in addition to the Director. The positions include an M&E manager (recruited), an M&E specialist, and a Knowledge Management Specialist. Short-term staff will be hired as and when certain tasks require additional capacity and resources, including for data entry, management and analysis. For the Global Fund to fully develop and support the various M&E functions and products, it is also planned to develop a small network of experienced consultants with appropriate experience and language skills. They will supplement in a flexible way the capacity of the Unit. It will also be critical to support M&E skills and capacity of other units of the Fund to fully capacitate the Fund Secretariat to implement the M&E strategy at all stages of the portfolio process. The SIM unit will work closely with and will advise other units of the Global Fund, in particular the portfolio managers, on M&E issues related to performance based funding decisions, including the development of appropriate tools and criteria. The M&E unit will provide resources for such activities throughout the Fund units. In collaboration with main international and national partners, the Global Fund will also support programme level M&E efforts through catalysing and coordinating technical support through a partner network (including WHO, UNAIDS, Bilaterals, Health Metrics Network, GAMET, and others), as well as the documentation and dissemination of Global Fund M&E best practices. SIM will also work closely with its partners to ensure access to existing country level information and databases. The development of an M&E toolkit addressing monitoring and evaluation frameworks and consolidated lists of key indicators, as well as tools for the collection of these indicators, is already well underway in partnership with WHO, UNAIDS, The World Bank and bilateral partners. This toolkit will be an essential component for harmonized support to monitoring progress at programme and country level. It will, to the degree possible, build on existing tools, while the effort is also identifying any gaps, to be filled during the next year. The toolkit will be an essential component of the guidelines for Round 4 proposals. 2. Strengthen the Independence and Soundness of Approaches To strengthen the independence of the Fund’s M&E work and the soundness of the approaches, the Board has decided to establish a Technical Evaluation Reference Group (TERG). Draft Terms of Reference (TOR) for the TERG are attached. The TERG shall provide independent assessment and advice to the Board of The Global Fund through its MEFA Committee, and to the Global Fund Secretariat, on technical and managerial aspects of monitoring and evaluation. Specifically, the TERG shall provide input and conduct regular reviews of progress towards the implementation and refinement of the M&E Strategy of the Global Fund. It will also provide a link to broader international discussion of monitoring and evaluation of HIV/AIDS, TB and Malaria. This includes reviewing monitoring and evaluation activities in relation to: - Programme level: performance based funding. - Country level: linking performance with disease impact, including the principle of additionality and the performance of the portfolio architecture of the Fund. Appendix C 58 - Global level: added value of the Global Fund, including monitoring of key performance indicators for the grant portfolio and the organization’s administrative performance, harmonisation with existing M&E systems and positive or negative health-system wide effects, if any are identified. SIM will serve as the secretariat to the TERG, in particular with regards to the arrangements for TERG meetings, sending of invitations and providing logistical support. The Global Fund Secretariat will be assisted by an external institution to support specific functions of the TERG, such as the creation and servicing of electronic discussion groups and the preparation of analytical reviews of existing studies and work to facilitate TERG discussions. The supporting institution will be selected through an independent search and assessment based on expertise, logistical and cost considerations 3. Establish the Information Platform of the Fund There is a need to develop an information system for the Global Fund to transparently support proposal and grant related processes, as well as to provide a platform to compile, analyse and disseminate relevant information, monitoring progress at all levels using key performance indicators. The basic monitoring framework has been defined, and the work towards an integrated information system will build on and drive the further development of key performance indicators and related tools (which are described in a different section of this plan). This work will include the following activities: a) Review and consolidate the process, coverage and impact indicators suggested for the three diseases together with, and building on, work done by development partners (e.g., WHO, UNAIDS, international initiatives such as Roll Back Malaria and StopTB, bilaterals). b) Review and revise country programme guidelines for monitoring processes (M&E toolkit). c) Develop indicators to measure the performance of the Fund at Global level, including portfolio and organizational performance. d) Develop indicators to measure processes and performance areas of special interest to the Fund, such additionality and sustainability. The development of the information system is well under way. This system will, in its first phase, support the proposal and grant agreement processes including key objectives, baselines, targets and indicators to monitor progress (target: phase one to be launched together with Round 4, January 2004). Further work will be undertaken by SIM in collaboration with other units in the Secretariat to build additional modules of the system to ensure the phased inclusion of the different aspects of monitoring at programme, country and global levels, and to develop procedures for validation and quality assurance of data and information. The Global Fund will also develop mechanisms and provide resources, through grant proposals and coordination with host countries’ and other donors’ efforts, to ensure utilization of inf ormation collected and disseminated through the information system. Such mechanisms may include meetings at all levels to analyze information and make needed programme adjustments, and to follow up systematically on earlier recommendations, using participative and dynamic methodologies for information analysis and action planning. 4. Address Key Evaluation Questions The M&E strategy lists a number of evaluation questions and thematic areas which should be assessed. The following presents evaluation priorities for 2004 and some already identified priorities 59 Appendix C for 2005 and 2006. It also describes briefly processes for preparing and implementing such evaluations. (a) Assessing the Organisational Architecture of the Global Fund The Fund was established with a new and innovative organizational architecture — with CCMs, LFAs and principal recipients at country level. There is a need at this early stage to assess the relevance, efficiency and effectiveness of the new structures in order to suggest and introduce changes and improvements, and to strengthen these country processes. ï‚· Studies of CCMs have already been completed and others are ongoing with support from external partners (primarily with a focus on inclusion and participation of stakeholders, like civil society, private sector, people living with AIDS, etc.). The M&E unit will undertake a review of all such studies in order to identify areas of further work for a comprehensive assessment of CCMs. Additional work will strengthen and utilise ongoing work, as and when appropriate. ï‚· A review of the effectiveness of the arrangement with Local Fund Agents (LFA) will be prepared and carried out in 2004. The LFA component is among the most innovative features of the organizational architecture in the Fund and needs special attention. LFAs represent and carry out defined tasks on behalf of the Global Fund at country level and were selected through an international tender. The LFA system is supported by the operational budget of the Global Fund and currently requires roughly half of the budget. There is a need to assess how well the LFA system is functioning and whether it addresses the multiple needs of the Fund in an effective manner, including cost effectiveness. There is also a need to assess to what extent the roles and functions of LFAs are clearly defined and supported within the broader Fund organization, in particular through guidance from the Fund Secretariat. Terms of reference for such an evaluation will be prepared in early 2004, and the evaluation will be carried out through a desk review, a user survey (among Portfolio Managers, CCMs, etc.) and case studies in selected countries. ï‚· An evaluation of the Fund’s governance structure will be prepared in 2005. (b) Assessing Strategies, Approaches and Results ï‚· Assessing the Performance Based Funding System Performance based funding is vital for the Global Fund and requires effective systems and procedures for measuring performance. An assessment should analyse and discuss to what extent the system is clearly defined, operationalized, and used according to intentions. What are country experiences? Is there capacity to collect, analyse and use quality information in a timely fashion? Can data be used to assess outcomes and measure performance of Global Fund interventions (issues of attribution)? Terms of reference for such an evaluation will be prepared in 2004. ï‚· Ex Post Reviews of Country Performance prior to the end of the first two-year funding period should be carried out. There are so far no clear criteria or agreements on systems for carrying out these reviews. Relevant countries would need to be informed soon about the criteria on which their performance will be assessed and future funding decided. These criteria should also be included in the Round Four documentation. The Strategic Information and Measurement Unit will assist in the preparation of systems and procedures for the review process, which may also include responsibilities to carry out or coordinate a Appendix C 60 systematic and independent assessment of the group of grants for which performance and decisions about future funding are uncertain — and would need to be informed and assisted by an independent 23 review. (c) Assessing Broader Evaluation Questions Addressing the Global Fund as a Resource for Development ï‚· Assessing Additionality According to its policy, the Global Fund will only finance programmes when it is assured that its assistance does not replace or reduce other sources of funding, i.e. the Fund will only provide additional resources. The fungibility of foreign aid has been a source of controversy for a long time — in particular for donors seeking to support specific development goals and make sure that their funds do not displace existing expenditure. There are also conceptual and practical problems in defining and measuring programmatic and financial additionality. These are new challenges which the Global Fund needs to address, which may contribute more widely to monitoring and evaluation. An evaluation should analyse and discuss how the principle of additionality is defined, measured, and accomplished by the Fund in countries. Issues may include (a) the measurement of resource flows to health from national budgets (e.g., through national health accounts), directly from global partners, and/or a combination of these, and (b) to what extent funding has been in line with the principle of increasing overall investments in health. Before an evaluation is carried out, a framework and criteria for analysing and measuring additionality will be prepared and discussed, followed by the development of terms of reference for the study. It is proposed to establish a task group under the oversight of the TERG, including additional experts from countries, multi- and bilateral organizations, and research institutions, to take this work forward in 2004. ï‚· Assessing Health System-wide Effects of the Global Fund A Research Protocol has been developed by the Research Network on System Wide Effects of the Global Fund (SWEF, November 2003) to document and assess the health systems effects of Global Fund Grants in seven countries with the possibility of three additional ones. SIM will review the proposal from a Global Fund perspective to make sure that the study benefits countries and the Secretariat. SIM will work closely with the study group to ensure that the objectives and the products of the study will fit synergistically with activities proposed in the Fund’s M&E plan, including relevant work of partners such as WHO. SWEF also seeks to establish an international advisory board. SIM will coordinate with MEFA and SWEF to what extent the TERG, or members of the TERG, could serve on such a body. Additional evaluation questions may come up and will be considered, such Global Fund versus SWAP or basket funding, poverty reduction strategies, harmonization processes at country level within health sector reform processes or sector wide coordination through national AIDS councils. ï‚· Assessing Sustainability It is the policy of the Fund to promote the sustainability of its investments. While it is premature to assess results, in terms of achieved institutional and financial sustainability, it is timely to monitor factors and processes contributing to sustainability and begin analyzing the likelihood for sustaining Global Fund benefits beyond the funding period. 23. The assumption is that the review will be based primarily on existing sources of information which might be sufficient to assess most countries (the clear yes and no countries), while a special assessment is required for the group in the middle. 61 Appendix C The first step of the assessment will be to prepare a framework and criteria for analyzing sustainability, building on a review of existing sustainability analysis models. This will be discussed with relevant stakeholders before terms of reference for an evaluation are prepared. It is proposed to establish a task group under the oversight of the TERG, including additional experts from countries, multi- and bilateral organizations and research institutions, to take this work forward in 2005. The evaluation is scheduled for 2006. ï‚· Studying Effects of Services on Poor and Vulnerable Groups, including People Living with the Diseases Global Fund investments are expected to reach poor and vulnerable groups and the actual distribution of benefits from grants needs to be assessed. Demonstrating that grants reach poor and vulnerable groups, including funding for scaling up capacity and strengthening systems, which support these populations in the periphery, requires longitudinal data according to socio-economic status. Such studies are long-term and should be properly prepared. SIM will in 2004 develop a framework and plan for how to carry out such benefit incidence studies, after first analyzing existing models for such research. It is proposed to establish a task group under the oversight of the TERG, including additional experts from countries, multi- and bilateral organizations and research institutions, to take this work forward in 2005 and 2006. 5. Assessing Performance of the Secretariat The Strategic Information and Measurement Unit will, in 2004, prepare the tools and systems for assessing performance of the Secretariat, in collaboration with the key functional leadership of the Secretariat. Performance indicators will be developed based on key functions and processes fulfilled by the Secretariat and data supportive of this process will be incorporated into a Secretariat management information system (part of information platform, described above). The following illustrate indicator categories for different key functions of the Secretariat and serve as a basis for finalizing work in this area: (a) Resource Mobilization • Annual funding targets versus pledges versus contributions • Timeliness of pledge conversions to contributions Appendix D 62 Appendix D. Burkina Faso M&E Comparison M&E Element World Bank (2002–2007) Global Fund (2003–2007) Comparison M&E Design Results Project/program The Deveopment Objective was to ―assist the 1. To accelerate access to HIV/AIDS treatment Both sets of goals/objectives are Framework Goals and Government of Burkina Faso to implement the with particular reference to ARVs vague and poorly defined, Objectives 2001–05 HIV/AIDS strategic plan, in order to slow 2. Strengthen the capacity of structures and the particularly what is meant by the spread and mitigate the impacts of the expertise of personnel involved in medical and ―capacity,‖ and are focused mainly epidemic.‖ psychological treatment of PLWHA on access to preventive activities The specific objectives in the PAD were: 3. Expansion of the national program for the and treatment. 1. Scale up, expand and improve preventive prevention of mother-to-child transmission activities in an effort to lower the risk of (PMTCT) transmission 2. Strengthen capacity to provide care, treatment and support to those infected or affected by the epidemic 3. Mitigate the socioeconomic impact on affected households and communities Performance [from PAD] Goal Indicators: The WB indicators appear to be a Indicators Outcome/Impact Indicators: 1. # of patients benefiting from antiretroviral ―laundry list‖ with unclear 1. % of non-married individuals who report having treatment relationship to each other and the 2 or more partners 2. # of PLWHA receiving at least 2 occasions development objectives. There are 2. % of men who report having used a condom community-based treatment support too many indicators, the list is during the past 12 months 3. % of sero-positive women treated with dominated by output/count 3. Condom utilization rates among priority groups Nevirapine at PMTCT facilities indicators, and many of them are 4. % of PLWHA reporting discrimination and Output Indicators: poorly defined. These deficiencies stigmatization 4. # of health districts with at least 1 approved were noted in the QER, but the 5. Differentials in school drop-out rates between treatment structure project went forward anyway. orphans and non-orphans 5. # of treatment structures approved, renovated The GF list of indicators found in Output Indicators: and equipped for the dispensing of ARVs the 2005 Grant Scorecard was 6. % of line ministries that have prepared work 6. # of health workers at approved sites trained substantially changed from the programs according to current standards and protocols of original grant agreement. The final 7. % of line ministries that have carried out work ARV treatment set is more streamlined, but also programs 7. # and % of patients still receiving treatment 6 exclusively output/count indicators. 8. % of line ministries that offer AIDS education in months after initiating ARV therapy According to the Grant the workplace 8. # of health districts with at least 1 CBO for Performance Report, at least some 9. # of communities that have prepared sub- medical and psychosocial treatment support of of the indicators were harmonized projects in participating provinces PLWHA with the National M&E Plan for 10. # of associations and PLWHA that are 9. # of CBO equipped and strengthened HIV/AIDS Control. 63 Appendix D M&E Element World Bank (2002–2007) Global Fund (2003–2007) Comparison participating in the project 10. # of CBO personnel trained and retrained (in Although there are too many 11. # of communities that have mapped high providing psychosocial support, in palliative output indicators, at least the WB transmission areas in participating provinces care, etc.) project has indicators for the full 12. % of sexually active individuals who know they 11. # of PLWHA receiving on at least two results chain (both outputs and can avoid contracting HIV by using a condom occasions community-based treatment outcomes), while the GF list has 13. % of rural females who report not knowing any support only outputs means of contracting HIV 12. #r of health facilities for PMTCT equipped and Neither set of indicators was 14. % sexually active individuals who know that a operational validated. person who looks well can be infected by HIV 13. #of community health workers involved in 15. # sexually active individuals who receive PMTCT trained at selected sites voluntary testing and counseling 14. # and % of seropositive women treated with 16. # of health workers trained in counseling Nevirapine at PMTCT facilities 17. # of infected people treated for opportunistic 15. % of seropositive children born to HIV positive infections in participating provinces mothers 18. # (% ?)of STI cases treated with appropriate drugs 19. # of training sessions attended 20. # of food packages distributed 21. # of drugs and basic supplies distributed 22. # of house visits made 23. # of orphans who are being looked after 24. # of counseling sessions organized 25. # of referrals made 26. # of income-generating activities organized 27. # of PLWHA who have food and shelter in participating provinces 28. # of PLWHA who receive home-based care in participating provinces 29. # of solidarity schemes established involving widows and poor women in participating provinces 30. # of communities providing orphan care and support in participating provinces 31. Correct knowledge of the means of HIV transmission among the priority groups targeted 32. Priority groups targeted who know that condoms provide protection against AIDS 33. % of population group targeted being treated Appendix D 64 M&E Element World Bank (2002–2007) Global Fund (2003–2007) Comparison for STIs and opportunistic infections 34. AIDS Program Effort Index [from ICR] With respect to the development of means to measure achievement, despite considerable additional resources mobilized by the Task Team (from GAMET12 and other Sources) to assist the PMU and the SP-11. The QAG assessment concluded that the Task Team should, at the Mid-term review, agree with the government on a smaller and simpler set of indicators for measuring project progress, but there is no indication that this was done. M&E Implementation M&E Implementation Process Although an entire component of the project was The Principal Recipient drafted a M&E plan and The end-of-project evaluation dedicated to M&E, according to the ICR, it was hired a full-time M&E expert. identified that the M&E component never implemented successfully. of the World Bank project was never fully implemented. There was no such end-of-grant evaluation to determine how well the Global Fund program M&E system was implemented. GF devoted human resources to implement M&E. The World Bank project benefited from 5 missions from the Global Monitoring and Evaluation Team (GAMET), but the M&E activities produced ―no discernible results.‖ Data Sources/ [from PAD] No data source listed for most No information given. Data sources not defined for either Collection indicators WB or GF indicators [from ICR] the management information system for Data were collected and reported for output the national strategic framework, which indicators only—no data reported for impact incorporates the project indicators, was never fully indicators as part of Global Fund KPI exercise. operational and only a few of the proposed surveys and studies were actually implemented Data Quality No data sources were provided in the ISRs, so [from Grant Performance Report]There is no major Although GF indicators were difficult to judge data quality or validity. issue with the quality or validity of data. As per the reported regularly throughout the 65 Appendix D M&E Element World Bank (2002–2007) Global Fund (2003–2007) Comparison LFA, the quality of reported results has gradually life of the program, no information improved during phase I. Progress updates are is given on the data sources, informative. Data quality is good. calculations or limitations. It is assumed that the LFA examined this, but it is unknown how rigorously or critically Monitoring Products 15 ISRs produced and a Mid-Term Review Regular progress updates (PUDRs); Grant WB produced regular monitoring Performance Report (regularly updated); Grant products (ISRs), which were made Scorecard available internally. The Quality Assurance Group review noted that: (i) the ISRs could be shorter and more strategic; and (ii) "ratings occasionally appear to have been disconnected from actual project progress or not clearly enough explained" especially for the M&E and FM ratings, which were "sometimes overestimated"). GF produced regular monitoring products, which are posted on the GF web site. M&E Utilization Program Management and M&E products used to further project Monitoring results used for funding disbursement. GF monitoring system was used Policy Input implementation, but not clear how effectively used Several grant recipients report that they have for both funding decisions and for policy input. integrated the GF performance based indicators program management. It is unclear into their own planning processes and now rely on how useful the results were in them for their own decision-making and planning. practice, however, since the targets were consistently achieved or exceeded. Project/program Performance In terms of monitoring assessments, the ICR Grant Rated B1 (adequate). It is not clear how this WB has standard criteria for rating Assessment indicated that performance assessments in the rating was determined, as performance against performance and written ISRs may not have been accurate. The ICR noted targets was consistently high and often over 100% justifications are given. The validity that ―a more accurate rating probably should have of project ratings in the ISRs were downgraded the PDO more substantially from still questioned, however, by the 2004 on and maintained the [Implementation ICR. GF rating is less transparent. Progress] at [moderately satisfactory] (which is where it finished). Appendix D 66 M&E Element World Bank (2002–2007) Global Fund (2003–2007) Comparison [ICR] In terms of evaluation assessments, the ICR found that basis of 14 potentially quantifiable outcome/intermediate outcome indicators, 8 were achieved, 3 were not, and 3 had no data. The ICR concluded that ―Whether using strict outcome results or the combined outcome/intermediate outcome results, the [achievement of Project Development Objectives] must be rated as moderately unsatisfactory. The project was also rated unsatisfactory on efficiency. Evaluation An ICR was completed according to standard Country impact evaluation was done. WB produced a critical evaluation World Bank guidelines. (ICR) that exposed shortcomings [from ICR] With strong support from the Task in the design and implementation Team, DECRG carried out a remarkable number of of the project but also highlighted operational research studies, including: (i) a survey positive contributions. of health facilities delivering ARV treatment (2006); A country ―impact evaluation‖ was (ii) the addition of questions related to treatment to done as part of the Global Fund the 2007 CWIS; (iii) a survey of HIV/AIDS patients FYE. This study did not attempt to and their households; (iv) an analysis of the attribute impact specifically to the HIV/AIDS data in the Burkina Faso DHS 2003; and Global Fund, however. (v) an impact evaluation on cash transfers to Furthermore, few stakeholders orphans. interviewed in Burkina Faso had However, although the ISRs indicated an intention seen the report for Burkina Faso or to conduct evaluations of the various pilot activities the global synthesis report. in the project, it does not appear that any of these were carried out. Overall Performance of M&E System Link with National M&E System [from ICR] (US$ 3.60 million equivalent) was [from Grant Scorecard] The indicators in the The WB intended to integrate intended to finance: principal recipient M&E plan are part of key project and country M&E and í¯€a comprehensive monitoring and evaluation indicators of National M&E Plan for HIV/AIDS strengthen the overall system. This system for the national program; and control. component did not achieve its í¯€a Project Management Unit to work closely with The principal recipient intended to contract whole overall objective, but the National the SP-CNLS on technical issues and to be activity of M&E with the Permanent Secretariat of AIDS Council did produce a M&E responsible for the administrative and financial the National Council for AIDS (SP/CNLS). Since framework and some analyses. management aspects of the project. the SP/CNLS is expected to take over the The GF was working with the same 67 Appendix D M&E Element World Bank (2002–2007) Global Fund (2003–2007) Comparison Although the M&E sub-component never management of the project by 2006, both the institution. completely achieved its objectives at the outcome, principal recipient and SP/CNLS agree that output, or process levels; at the same time, SP/CNLS will take major responsibility in M&E however, SP-CNLS (National AIDS Council) was: [note: not clear if this was done] í¯€developing a comprehensive M&E framework with 18 impact indicators, 25 results indicators, and more than 100 process and input indicators; í¯€producing (at least for 2004 and 2005) the UNGASS country report on the HIV/AIDS situation in Burkina Faso; and í¯€consolidating large quantities of data from disparate sources for presentation (2002 through 2007) at the annual meeting of contributors and participants in the fight against HIV/AIDS. Assessment of M&E System [from ICR] In practice, there were problems with The M&E plan was assessed by the LFA in March WB rates M&E in the ISRs and the both the selection of indicators and the means used 2004 (by Chemonics, Previous LFA) and ICR but assessments in the ISRs to measure their achievement (ICR). Indicators subsequently approved by the Global Fund. In appeared subjective in this case were selected and eliminated, measured (or not) addition, the M&E system was reviewed by the with no justification for positive haphazardly, and generally ignored as a LFA and considered as satisfactory for program ratings when the M&E system was management tool throughout much of the project implementation. found seriously lacking in the ICR. period (at least by the project) GF gave no criteria for assessing [from Grant Scorecard] The quality of reported GAMET's support for the development of M&E results has gradually improved during the M&E plan or system. capabilities deserves more study to determine why implementation of phase 1. Progress updates the component was not satisfactorily implemented; are correctly written and provide good insight into the ICR Team cannot say whether the proposals activities and problems encountered. Overall, data were too complex, lacked adequate incentives, fell quality is good. However, there is still a need to victim to evolving institutional arrangements, etc. improve the quality of indicators selected and their validation Appendix E 68 Appendix E. Lesotho M&E Comparison M&E Element World Bank (2005–2008) Global Fund (2004–2009) Comparison M&E Design Results Project/program ï‚· Objective (Development Grant Agreement): Contribute to the reduction of HIV incidence, The objectives were Framework Goals and Increase the recipient’s capacity to use particularly among youth, and improve the care complementary. Objectives effectively the resources provided through the and support of PLWHA and those affected by the Global Fund grant to support the implementation epidemic of HIV and AIDS programs within the recipient’s territory. ï‚· Objective (PAD): To increase the capacity of government and non-governmental institutions responsible for the national response to HIV and AIDS to utilize and assess the impact of funds received for HIV/AIDS. Performance 1. 80% of the Global Fund grant disbursed by the Impact WB list is targeted and streamlined Indicators end of the project 1. % of infants born to HIV-infected mothers who but did not include any indicators 2. 30% of the Global Fund grant disbursed are infected to measure the extent to which the through civil society organizations 2. % of people expressing accepting attitudes GF grant was used effectively, nor 3. 50% of HIV/AIDS sub-projects financed toward PLWHA, of all people surveyed aged did the GF grant include such through the Global Fund grant implemented in 15–49 measures. a satisfactory manner 3. % of people aged 15–24 reporting the use of The GF list of indicators found in 4. Annual reports on all key HIV/AIDS program condom the last time they had sex with a non- the Grant Performance Report was indicators produced and disseminated. regular sexual partner substantially changed from the Output original grant agreement. The final 4. Total # of young people in school taught life set is more streamlined, but still skills education heavily weighted toward output 5. Total # of young people out of school taught indicators. The output-oriented life skills education indicators did not capture the 6. # of condoms distributed to youth corners and quality, effectiveness or impact of the general population interventions 7. % of infants born to HIV-infected mothers who receive ARV prophylaxis as per WHO protocol 8. HIV-infected pregnant women receiving a complete course of antiretroviral prophylaxis to reduce the risk of MTCT 9. # of pregnant women visiting an ante-natal PMTCT site counseled/tested for HIV 10. # of health care service providers trained on 69 Appendix E M&E Element World Bank (2005–2008) Global Fund (2004–2009) Comparison diagnosing and management of HIV and AIDS 11. # of people receiving ARV treatment 12. # of sites with functional VCT centers 13. # of people counseled and tested for HIV 14. # of registered OVC receiving basic package of care and support 15. # of PLWHA receiving advocacy skills training on their human rights M&E Implementation M&E Implementation Process [PPAR] Due to lack of manpower, the M&E [from Grant Scorecard] M&E plan was not yet M&E implementation was a failure system was not fully implemented. implemented 2 years into the program due to lack in both activities. of capacity at all levels. Finally done in 2007. No Data Manager or M&E specialist was in place at that time (prior to the WB project) Data Sources/ Management information system (MIS) was never [from Grant Scorecard]There was no data manager [From stakeholder interview]: Collection developed). or M&E technical support person in Phase I to ï‚· Parallel reporting system — manage data collection and collation, data quality separate on for HIV driven by assessment and analysis of data. GF and separate for MOHSW; By Phase II the National database was installed for competition for data and human HTC, antiretroviral therapy and PMTCT in order to resources. develop the management information system (MIS [From stakeholder interview]: was never developed). ï‚· UN has worked with MOHSW on forms for data collection; developed and distributed but not used Data Quality [from Grant Scorecard] Results are not properly [From stakeholder interview]: verified, and are therefore unreliable. ï‚· High level of suspicion about 1) Secondary extrapolation of data from the health quality of data; MOHSW wants facilities is problematic; to be the sole custodian of the 2) The quality of the data cannot be established or data verified as a data quality assessment and reliability report has not been generated (extrapolated and analyzed manually); 3) Data quality varies and is often incomplete; 4) Capacity is minimal at the HSAs and the central level to ensure that data generated is accurate and reliable; Appendix E 70 M&E Element World Bank (2005–2008) Global Fund (2004–2009) Comparison Monitoring Products The project supported the MOHSW to produce GF recipient collected data on key indicators and GF produced regular monitoring quarterly activity monitoring reports and annual use those data to prepare quarterly reports and products, which are posted on the joint sector reviews annual reports, including Joint Annual Review GF web site. (from stakeholder interview) Joint reviews good for reports. Improved capacity was also used to bringing partners together but no consistency in conduct research studies and surveys. the data that is presented from year to year Regular progress updates (PUDRs); Grant Performance Report (regularly updated); Grant Scorecard M&E Utilization Program Management and Not clear. These M&E reports were used to formulate key Not clear. Policy Input policy and strategy documents, such as the National HIV/AIDS Policy, etc. However, data use remains limited due to system fragmentation, the planning cycle not being synchronized with the reporting cycle, and limited advocacy using strategic information. Performance Assessment The project outcome was rated ―satisfactory‖ by By December 2008 the grant was rated A2 (―met WB has standard criteria for rating the ICR, ―moderately satisfactory‖ by the ICR expectations‖) based on exceeding expectations performance and written Review, and ―moderately satisfactory‖ by the for performance indicators and adequate justifications are given. GF rating is PPAR, based on high relevance of the objectives, performance on management issues less transparent but possibly modest relevance of design, substantial related to low disbursement rates. achievement of raising the capacity of the borrower to use Global Fund money but modest achievement of the objective to use the funds effectively, and on substantial efficiency. Evaluation An ICR (self-evalaution) and PPAR (independent Impact evaluation was done as part of the Global The WB and IEG produced critical evaluation by IEG) were completed. Fund FYE, but that was not part of program M&E. evaluations that exposed shortcomings in the design and implementation of the project but also highlighted positive contributions. The GF country impact evaluation was done as part of the Global Fund FYE. It is not clear whether/how this evaluation was used by the country 71 Appendix E M&E Element World Bank (2005–2008) Global Fund (2004–2009) Comparison Overall Performance of M&E System Link with National M&E System An M&E Unit was established in the NAC with 10 [from Grant Scorecard] the HIV/AIDS National [From stakeholder interview]: data officers (one from each district). Two strategic plan does not have an M&E section Although mandate of both international advisors assisted the M&E Unit. An programs was to coordinate, electronic database system was developed to integrate with national M&E system improve overall data management. The M&E team and strengthen it, it seems that this under the MOHSW HIV/AIDS directorate took failed and added burden created parallel capacity-building steps, now having Development partners and several units with staff. Division of labor and data government have not sat down and sharing remains problematic between NAC an agreed on data needs — different MOHSW. programs/projects collect different data; M&E Unit is not showing leadership Assessment of M&E System IEG’s PPAR rated the M&E system as modest Rated ―C‖ at the end of Phase 1 (―unacceptable‖). [From PPAR]: Upgraded to B1 by end of Phase 2 (―adequate‖). Overall the M&E system remains fragmented. M&E planning and design have taken place at a sophisticated level, but those plans are grossly mismatched with the capacity of the staff and systems implementing them. HIV/AIDS data, financial management data, TB data and human resources data all still run as vertical systems (Macro International 2008). Redundant systems at the district level place huge burdens on limited staff. As the country’s M&E capacity was strengthened, several parallel systems were created without proper plans for coordination or integration (multiple data officers); insufficient human resources capacity to manage a burdensome M&E system. [PPAR] Appendix F 72 Appendix F. Russia M&E Comparison M&E Element World Bank (2003–2009) Global Fund (2004–2011) Comparison M&E Design Results Project/program [PAD] (1) contain the growth of the epidemics of (i) improve the knowledge of youth and the general The World Bank project objectives Framework Goals and TB and HIV/AIDS in the short term; and (2) halt population about HIV/AIDS; (ii) promote prevention are impact-oriented, whereas the Objectives and reverse the course of these epidemics in the among specific vulnerable groups including objectives of the Global Fund medium term injecting drug users, sex workers, street children, program are related more to men who have sex with men, and prisoners; (iii) outputs and what could be address the needs of people living with HIV/AIDS considered to be intermediate through treatment, care, health promotion and results. social support; (iv) improve the participation and effectiveness of government counterparts in HIV/AIDS policy Performance Development Objective Indicators Impact WB list is targeted and streamlined Indicators 1. Approval by MOH of protocols for TB diagnosis, 1. % of injecting drug users who are HIV-infected but definitions are not clear in all treatment and surveillance consistent with in target regions cases. The M&E system for both evidence-based approaches agreed with the 2. # of new HIV cases registered in targeted the TB and HIV components were World Health Organization regions based on WHO and UNAIDS Improvements in effectiveness of TB diagnosis and 3. % of infants born to HIV infected mothers who guidelines. treatment: are infected 2. case notification rate 4. % of adults with HIV still alive 12 months after The GF list of indicators found in 3. smear positive case notification rate; initiation of ARV treatment the Grant Performance Report was 4. TB mortality rate Outcome substantially changed from the 5. % of HIV-positive infants born to HIV-infected 5. % of youth always using condoms with non- original grant agreement. Impact women regular partners and Outcome indicators were 6. Incidence of other specified STIs among the 6. % of general population who report receiving added in Phase 2, but were not general population HIV post-test counseling n the last 12 months part of performance-based funding 7. Rate of increase of HIV prevalence 7. % of commercial sex workers in targeted decisions in Phase 1. The final set Intermediate Outcome Indicators regions who report using a condom during last of indicators is more streamlined, 8. % of HIV-positive pregnant women who receive commercial sex but still heavily weighted toward anti-retroviral prophylaxis to reduce mother-to- 8. % of men having sex with men in targeted output indicators. There are many child transmission in accordance with nationally regions who report condom use last anal sex count indicators and no indicators approved treatment protocol with a male partner of service quality. 9. % exposed neonates who received prophylactic 9. % of injecting drug users (clients of harm anti-retroviral drugs reduction programs) in targeted regions avoided 10. Treatment success rate among new TB cases sharing non-sterile syringes 10. % of people that support the idea that HIV positive people should be separated from the 73 Appendix F M&E Element World Bank (2003–2009) Global Fund (2004–2011) Comparison total population. Output 11. # of condoms distributed among targeted groups (youth, men who have sex with men, injecting drug users, CSWs, migrant workers, prisoners, and the general PLWH) 12. # and % of CSW reached by HIV/AIDS prevention outreach services in targeted regions 13. # and % of street children reached with HIV prevention services in selected cities 14. # and % of men who have sex with men in targeted regions reached by outreach programs 15. # of prison personnel in targeted regions trained in HIV prevention methods 16. # and % of injecting drug users reached by HIV/AIDS prevention services, including needle exchange, in targeted region 17. # of prisoners reached by HIV prevention services 18. # and % of migrant workers reached by HIV prevention services in targeted regions 19. # of prisoners reached by transitional HIV prevention case management services 20. # of youth prisons implementing the ―Life Skills‖ HIV prevention program 21. # of people living with HIV reached through information centers 22. # of people living with HIV who receive legal services 23. # of activists trained to recognize and document human rights violations M&E Implementation M&E Implementation Process Implementation of M&E of the TB component of A web-based M&E tool was used that was Both the World Bank project and the project was carried out in close collaboration considered ―exemplary‖ for the entire Eastern Global Fund program appear to with WHO [ICR]. The WHO TB Control Program European region [GSC] have had well functioning M&E assisted by Russian experts provided support for systems that were useful beyond Appendix F 74 M&E Element World Bank (2003–2009) Global Fund (2004–2011) Comparison independent monitoring visits to participating project/program implementation regions to supplement health sector statistics and management. surveillance data. There is no discussion in the ICR of the implementation of M&E for the HIV/AIDS component of the project. Data Sources/ Data collection and reporting were conducted as Data were not reported regularly for impact and Data for the World Bank project Collection part of the routine surveillance system that was outcome indicators. Only output indicators were M&E were generated by routine strengthened with project support. consistently reported in the Grant Performance surveillance systems, so the Report. validity and reliability of the data were more widely accepted than [from Grant Scorecard] Sub-recipients self- the self-reported programmatic reported programmatic results into web-based tool data in the Global Fund program. operated by the Central Public Health Research Institute (under the MOH), to which the M&E function was outsourced. The self-reporting raised concerns about data reliability. Data Quality Not possible to assess. [from Grant Scorecard]. Local Funding Agent Not possible to assess objectively. performed on-site data verification, and the Global Fund Secretariat believes there were no systematic weaknesses in M&E. Monitoring Products In addition to ISRs, the PIU produced annual Regular progress updates (PUDRs); Grant Both the World Bank project and reports integrating the results of M&E activities Performance Report (regularly updated); Grant Global Fund program not only including (i) progress of each component and sub- Scorecard generated routine monitoring component; (ii) broader impact of the project; and products, but also supplemental [from Grant Scorecard] In addition to the (iiii) effect on the overall control of TB and synthesis products or other reports monitoring products required by the Global Fund, HIV/AIDS epidemics in the Russian Federation. that appear to have been more the Central Public Health Research Institute useful. database provided a wealth of performance data and additional reports from all regions. M&E Utilization Program Management and [ICR] The M&E process (particularly the mid-term Not clear. The World Bank project clearly Policy Input review) and products were used effectively to fine- made use of M&E products both tune project performance indicators/targets and for project management and policy approaches in spite of obstacles early in the dialogue. It is not clear to what project related to government administrative extent this was the case in the changes and resistance to approaches in both Global Fund program. disease areas. 75 Appendix F M&E Element World Bank (2003–2009) Global Fund (2004–2011) Comparison [ICR] M&E was used under the project not only as a management tool for implementation activities but also to inform policy, particularly for assessing the evolution of the epidemic (this appears to be true for the TB component, but less so for the HIV/AIDS component) Performance Assessment The performance assessments in the ISRs did not Consistent over-achievement of indicators The use of monitoring indicators always appear to be linked to indicator/target corresponds to consistent high rating of the and products for performance achievement and seemed subjective in some program. assessment in both the World cases. Bank project and Global Fund program is not clearly consistent Evaluation An ICR and ICR review were completed. No evaluation completed. The World Bank project has an end-of project evaluation. There is no such evaluation for the Global Fund program Overall Performance of M&E System Link with National M&E System The project strengthened existing systems rather Not clear. The World Bank project integrated than attempting to establish a project-specific with and strengthened the national M&E system. Assistance was provided by WHO M&E system, whereas it appears and UNAIDS to strengthen national M&E efforts. that a parallel M&E system was M&E has become an important tool providing the created by the Global Fund authorities at the federal and regional levels with program for its own purposes. timely and relevant data. [ICR] Assessment of M&E System The ICR assessed the M&E system as B1 (adequate) satisfactory. Appendix G 76 Appendix G. Selected Ratings of World Bank-Financed HNP Projects, in Comparison with Bank-wide Averages Table G-1. Outcome Ratings for HNP Projects, Approved from Fiscal Year 1997 by Year of Exit Share of rated Rating 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Total projects Highly Satisfactory 1 1 1 3 2% Satisfactory 1 4 9 7 6 6 4 2 2 41 28% Moderately Satisfactory 2 1 3 4 8 3 10 9 7 47 32% Moderately Unsatisfactory 1 5 3 10 4 6 29 20% Unsatisfactory 1 5 5 1 5 3 3 1 24 17% Highly Unsatisfactory 1 1 1% Not Rated 2 1 1 4 Grand Total 2 5 5 19 21 19 26 22 20 10 149 Source: World Bank database. Table G-2. Outcome Ratings for All World Bank Projects, Approved from Fiscal Year 1997 by Year of Exit Share of rated Rating 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Total Projects Highly Satisfactory 9 4 5 9 15 9 6 8 6 3 74 4% Satisfactory 32 46 82 112 133 114 101 77 56 26 779 43% Moderately Satisfactory 15 28 47 57 67 89 71 79 85 43 581 32% Moderately Unsatisfactory 8 3 12 13 22 21 38 26 36 14 193 11% Unsatisfactory 6 16 22 27 25 19 15 20 15 15 180 10% Highly Unsatisfactory 1 2 2 4 5 1 2 1 18 1% Not Rated 7 6 4 4 6 5 8 1 4 3 48 Grand Total 78 105 174 226 273 258 241 211 203 104 1873 Source: World Bank database. 77 Appendix G Table G-3. ICR Quality Ratings for HNP Projects, Approved from Fiscal Year 1997, by Year of Exit Share of rated Rating 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Total projects Exemplary 1 1 1 3 1 1 3 11 7% Satisfactory 1 3 4 18 20 15 20 19 18 7 125 84% Unsatisfactory 1 1 1 3 3 2 1 12 8% Not Rated 1 1 Grand Total 2 5 5 19 21 19 26 22 20 10 149 Source: World Bank database. Table G-4. ICR Quality Ratings for All World Bank Projects, Approved from Fiscal Year 1997, by Year of Exit Share of rated Rating 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Total projects Exemplary 8 7 7 9 14 16 13 13 6 5 98 5% Satisfactory 63 81 147 186 236 210 198 177 185 92 1575 85% Unsatisfactory 5 17 20 31 23 30 27 19 10 6 188 10% Not Rated 2 2 3 2 2 1 12 Grand Total 78 105 174 226 273 258 241 211 203 104 1873 Source: World Bank database. Appendix G 78 Table G-5. M&E Quality Ratings for HNP Projects, Approved from Fiscal Year 1997 by Year of Exit Share of rated Rating 2006 2007 2008 2009 2010 Total projects High 1 1 2 2% Substantial 3 6 4 4 2 19 22% Modest 4 13 14 10 7 48 56% Negligible 1 6 3 6 1 17 20% Non-evaluable 1 1 Grand Total 9 26 22 20 10 87 Source: World Bank database. Table G-6. M&E Quality Ratings for All World Bank Projects, Approved from Fiscal Year 1997 by Year of Exit Share of rated Rating 2006 2007 2008 2009 2010 Total projects High 4 10 3 2 3 22 3% Substantial 39 76 69 58 19 261 30% Modest 71 108 95 107 67 448 51% Negligible 22 36 39 31 12 140 16% Non-evaluable 3 6 4 4 2 19 Grand Total 139 236 210 202 103 890 Source: World Bank database.