74056 MAY 2013 ABOUT THE AUTHORS JUMANA QAMRUDDIN Using Evidence to Scale Up Innovation: works in the Africa Region Health Unit (AFTHE) of the World Bank. Formally trained as a Insights from a Results-Based Financing Public Health Specialist, Jumana is currently the Task Team leader (RBF) Project for Health in Zambia of the RBF Grant in Zambia and the World’s Bank’s health and nutrition portfolio in Several developing countries face the challenge of attaining sufficient population- Madagascar. level impact to meet health-related Millennium Development Goals (MDGs) 1c, 4, COLLINS CHANSA 5, and 6. 1 This situation is partly attributable to constraints in their health systems, is a Consultant for AFTHE. Collins is a Health Economist with including: severe shortages in human resources for health; inequalities in service considerable experience working at managerial level at the provision and utilization; limited financial resources; and inefficiencies in resource Zambia Ministry of Health, and at technical level at the World allocation and use. Even in countries with adequate financial resources, health Bank Zambia office. indicators are sometimes poor due to inefficiencies and a lack of performance- ASHIS DAS related initiatives. is a Consultant for AFTHE. Ashis has been trained in medicine and public health. In an attempt to strengthen health systems and improve health-service delivery, The team is indebted to Harrison Mkandawire, Caesar Cheelo, Jos several countries in Africa are increasingly using results-based financing (RBF) Dusseljee, and Dr. Caroline Phiri for earlier data on the external approaches in their health programs. The premise is that linking financing verifications and the technical review of the Katete pre-pilot. to results will lead to improvements in health systems and health-outcome Mambwe Kabaso, Brivine indicators. This SmartLesson captures emerging lessons from Zambia’s drive to Sikapande, Chishimba Nakamba Mulambia, Teniford Shawa, improve service delivery in the public-health sector with support from the World and Dr. Pricilla M. Zulu are also commended for facilitating the Bank’s RBF initiative. data collection process in Katete District. APPROVING MANAGER Background The Zambian RBF model was designed Olusoji Adeyi, Sector Manager, to address some of the health-system AFTHE. In April 2008, Zambia was awarded a US$17 weaknesses identified as preventing better million grant from the World Bank’s Health health outcomes, including: an insufficient Results Innovation Trust Fund (HRITF). The and poorly motivated human-resource base; grant was used to launch a pilot results- an erratic supply of essential medicines based financing (RBF) project, which was and medical supplies; limited autonomy in expected to catalyze efforts to reduce decision-making at decentralized levels of maternal and child mortality in 10 districts in the health system; weak monitoring and the country. evaluation systems; and poor quality of service delivery. The project commenced with a 24-month pre-pilot implemented in Zambia’s Katete A fee-for-service model linked performance District between 2009 and 2011. The pre- payments to the delivery of nine facility- pilot provided an opportunity to test and based Maternal and Child Health (MCH) and refine the RBF model in a real-world setting, HIV/AIDS indicators. Performance payments while at the same time helping to strengthen were determined by multiplying the volume the health system, and improve the quality of of services delivered by a specified fee health care delivery and health outputs. The for each indicator, and by a quality score. project’s implementing team was also able In addition, the District Medical Office to contextualize RBF in Zambia and to draw (DMO) was rewarded if it fulfilled a set of lessons from the pre-pilot, which were then supervision and management functions. The used to stimulate policy dialogue between 1 MDG 1c: Halve, between 1990 and 2015, the proportion of people the Zambian government and its partners. who suffer from hunger; MDG 4: Reduce by two thirds, between These lessons also guided the revision of the 1990 and 2015, the under-five mortality rate; MDG 5: Improve maternal health; MDG 6: Combat HIV/AIDS, malaria, and other model’s design before roll-out of the pilot. diseases (Source: The United Nations). SMARTLESSONS — MAY 2013 1 Figure 1: The Katete District RBF Pre-Pilot Project Model incentives were aimed at strengthening the DMO’s role in went through a series of adaptations and modifications supporting health facilities’ efforts to increase the delivery during this phase. Throughout, joint technical reviews were of high-quality services. conducted by the Ministry of Health (MOH), cooperating partners, external consultants, and the World Bank. Based Evaluations2 of the pre-pilot showed a positive impact on on recommendations from these reviews, improvements service uptake in both incentivized and non-incentivized were made to the implementation model for the pilot as indicators between 2009 and 2010. The increase in the well as to the design of an impact evaluation covering 30 incentivized indicators ranged from 7 to 54 percent, while districts. Specifically, the pre-pilot helped to: the increase in the non-incentivized indicators ranged from 6 to 53 percent (See Tables 1 and 2). Additionally, 1. establish eligibility criteria for the inclusion of health there were improvements in the accuracy of routinely facilities; reported Health Management Information System data. 2. streamline the quality tool, a checklist used to assess External verification exercises undertaken one year apart strutural and process aspects of quality of care at showed a significant decline, to almost negligible levels, in health facilities; misreported data. 3. refine the process of setting fees per indicator; 4. kick-start the process of developing a hospital-level Lesson 1: Pre-piloting is a useful approach to incentive package; embedding evidence into policy and planning. 5. incorporate a performance target for the distribution of clinical health workers into the incentive package for the District Medical Offices; The pre-pilot was a key element in introducing a systematic 6. develop and test contracts and contracting mechanisms; learning-by-doing reform, and it provided implementers and with an opportunity to test and refine the model at relatively 7. improve research tools for the impact evaluation study. low cost before scale-up. The design of the RBF project 2 The evaluations were descriptive studies that all used case-study designs. The surveys The Zambian government has demonstrated ownership relied on a combination of qualitative (semi-structured interviews and group discussions) and leadership of the RBF initiative at the policy- and quantitative (document reviews, financial, and Health Management Information System data) data-collection methods. development level. Following the success of the pre-pilot, 2 SMARTLESSONS — MAY 2013 the central-level fund–holder, while Provincial RBF Steering Table 1: Impact on RBF Indicators: Katete Pre-pilot Committees approve and purchase services. District-level RBF Steering Committees, which draw their membership from the community, government, donors and civil-society Indicator 2009 2010 % change organizations, act as external regulators. Their main Outpatient functions are to verify that services are provided, monitor 522,000 647,000 +24% Consultations the quality of service provided, and ensure compliance with Antenatal Care standards and the proper functioning of the system. District Follow-up Visit 17,600 21,700 +23% Medical Offices and second-level hospitals act as internal regulators and conduct quantitative and qualitative Postnatal Care < assessments, and health centers provide health services. 6,900 7,700 +12% 6 days An externally recruited firm independently verifies the Intermittent accuracy of reported data and confirms through patient Preventive 4,100 6,300 +54% tracing that health services have been provided. For the Treatment (IPT) Katete pre-pilot, the University of Zambia was contracted 3rd dose as the external verifier. Institutional Deliveries by 6,000 6,400 +7% Though novel, there have been challenges in implementing Skilled Personnel a model entirely through the public-health sector. For ANC Tested after instance, the design phase lasted much longer than it 1st Visit 11,000 12,100 +10% does in countries where the contractors are international NGOs. That said, there are also clear benefits with the public-sector model in Zambia. Among these are national ownership of the project at all stakeholder levels and integration of the RBF model and concepts into the health the MOH adopted RBF as a strategy in its fifth National system from the initial stages of development. These are Health Strategic Plan, covering the period 2011-2015. The critical foundations for building longer-term institutional pre-pilot also helped to initiate broader health-sector and financial sustainability for this approach in the country. discussions about RBF among key stakeholders in Zambia at a very early stage. For example, the World Bank and the MOH have been holding discussions with other partners Lesson 3: Investment in capacity and regular in the health sector, including the UK’s Department for dissemination of results are useful in enhancing International Development, the European Union, the ownership and promoting sustainability. Japan International Cooperation Agency, the Swedish International Development Cooperation Agency, and the Prior to tapping the HRITF, Zambia had limited experience Churches Health Association of Zambia on how to scale up with RBF, in part due to low implementing capacity. RBF initiatives in the country. Building capacity from the start of the pre-pilot’s design phase was a critical element in enhancing ownership and Some partners have expressed interest in potentially co- creating an environment that would promote longer-term financing the program, and discussions over the modalities sustainability of the model. The implementers of the Katete are underway. Given the success of the Katete pre-pilot, pre-pilot and other MOH personnel were guided by a series the HRITF grant is now being regarded as a potential entry of learning workshops and evaluations on RBF held at the point to achieve broader health-sector reforms. national, regional, and international levels. These were aimed at developing an understanding of RBF and its role Lesson 2: In designing an RBF approach primarily in strengthening health systems. Furthermore, external RBF executed by the public sector, key elements such as the experts provided technical assistance to the MOH during the design and implementation phase in order to build separation of different functions needs to be thought the necessary capacity. This included a participatory RBF through carefully, which may result in the model Training of Trainers (ToT) course in which the first national- taking longer to develop and implement. level RBF training materials were developed by government and NGO representatives. The RBF model used in Zambia (and its precursor in Katete) is a fee-for-service, performance-based financing model The ToT produced the first cadre of National RBF Trainers implemented in the public-health sector. The public- in the country, and was followed by a countrywide roll- health sector’s role as the main contractor is unusual in out of training in health centers, districts, hospitals and Africa, where the private sector and NGOs are typically provincial medical offices. Community representatives were contracted to implement RBF initiatives in public-health also trained so that they could understand and actively facilities. The Katete pre-pilot and the current pilot show participate in the project. In addition, the formation of that the public-health sector can be used as the primary RBF Steering Committees at all levels (district, provincial, contractor as long as there is a clear separation of functions and national) created an opportunity to share knowledge to ensure the integrity of the system. This is extremely and ideas among stakeholders and communities on all RBF important, because conflicts of interest may arise over the initiatives in the country. Eight RBF intervention specialists management of resources in instances where the division were also recruited locally (one per RBF district) to provide of roles and responsibilities is not clearly defined. support to districts implementing the RBF pilot. From very early in the design phase, the Zambian The MOH also disseminated the initial results from the government was eager to implement RBF primarily through Katete pre-pilot within Zambia and internationally, the public-health system as a means to accelerating progress garnering more inputs to influence the design and on key health outcomes through adressing key constraints implementation. The MOH and the World Bank also intend in the health system. In the Zambian model, the MOH is to publish the results of the pre-pilot in a scientific journal to SMARTLESSONS — MAY 2013 3 Table 2: Positive Impact on Non-RBF Indicators Indicator 2009 2010 % change Children < 5 Years Weighed 170,000 215,400 +27% Vitamin A Supplementation (12-59 months) 50,300 53,100 +6% HIV Test (excl. ANC) 19,600 29,900 +53% Inpatient Discharge 24,600 28,000 +14% further share experiences on best practices The results of the Katete pre-pilot also for financing and strengthening health influenced the decision that process systems in developing countries. As a result, evaluations would be included during considerable national implementation implementation as a means to monitor capacity has been built in the country, which and document changes that occur during will be critical to the broader sustainability implementation to inform decision- of RBF-initiated reforms beyond the pilot. making. These process evaluations will also complement the impact evaluation, helping Lesson 4: RBF can increase flexibility in to contextualize the results of the RBF in the health facility management, encourage country. innovation and creativity, and improve Conclusion accountability in service delivery. Pre-piloting in Zambia has been valuable During the pre-pilot, Zambian health in identifying the key requirements for facilities received enhanced autonomy over implementing RBF, as well as for testing the finances, planning, and management and resolving implementation bottlenecks of their centers. Separate bank accounts before finalizing the design of the pilot. were opened under the RBF program with a Pre-piloting has also made it possible to joint signatory from the community, which adapt RBF to the Zambian context, to enhanced community participation and build consensus on its implementation, increased accountability and transparency. and to embed evidence into policy and A qualitative review identified a wide range planning. The project is already providing of innovations, such as: hiring of local staff an opportunity for partnerships, knowledge to fill human-resource gaps; procurement of sharing and learning, and discussions minor equipment and commodities; and use of broader health reforms in Zambia. of demand-side incentives to boost service Nonetheless, continuous identification utilization. In line with the overall project and timely resolution of implementation objectives, the pre-pilot also contributed to bottlenecks remain critical to the success of improved staff motivation and productivity the RBF program in Zambia going forward. in Katete District. The pilot phase, which commenced in Lesson 5: Impact evaluations should be April 2012, is operational in 11 rural designed early and contextualized to districts (including Katete) representing the country setting. eight provinces out of a total of nine, 204 health facilities, and a total catchment DISCLAIMER SmartLessons is an awards In order to adequately evaluate the impact population of 1,691,240. The intended program to share lessons learned of RBF, there is a need for more rigorous direct beneficiaries are 67,650 children in development-oriented case studies detailing field experiences, aged 0-11 months, 338,248 children aged advisory services and investment success factors, and potential advantages between 0-59 months, and 372,073 women operations. The findings, and disadvantages. The Zambia RBF pilot of child-bearing age. interpretations, and conclusions project combines both an implementation expressed in this paper are those and impact evaluation component, the of the author(s) and do not design of which greatly benefited from necessarily reflect the views of the Katete pre-pilot. The current impact IFC or its partner organizations, the Executive Directors of The evaluation design includes quantitative and World Bank or the governments qualitative segments aimed at answering they represent. IFC does not key questions about RBF’s impact on: i) assume any responsibility for the health outcomes, ii) health systems, and iii) completeness or accuracy of the health workers’ attitudes and behavior. The information contained in this conceptualization, setting of the questions document. Please see the terms for the study, and development of the survey and conditions at www.ifc.org/ instruments were extensively informed by smartlessons or contact the the Katete pre-pilot. program at smartlessons@ifc.org. 4 SMARTLESSONS — MAY 2013