Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005130 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA 50940 and TF013432; AF: IDA 58510 and TF0A2591 ON A CREDIT IN THE AMOUNT OF SDR 184.6 MILLION (US$275 MILLION EQUIVALENT) AND A GRANT FROM THE MULTI-DONOR HEALTH RESULTS INNOVATION TRUST FUND IN THE AMOUNT OF US$21.7 MILLION AND A GRANT FROM THE MULTI-DONOR GLOBAL FINANCING FACILITY IN THE AMOUNT OF US$ 20 MILLION TO THE FEDERAL REPUBLIC OF NIGERIA FOR THE NIGERIA STATES HEALTH INVESTMENT PROJECT June 24, 2021 Health, Nutrition & Population Global Practice Africa West Region CURRENCY EQUIVALENTS (Exchange Rate Effective {Jun 23, 2021}) Currency Unit = Nigeria Naira (NGN) NGN411.5 = US$1 US$1.428554 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Ousmane Diagana Country Director: Shubham Chaudhuri Regional Director: Dena Ringold Practice Manager: Magnus Lindelow Task Team Leader(s): Gyorgy Bela Fritsche, Fatimah Abubakar Mustapha ICR Main Contributor: Onoriode Ezire ABBREVIATIONS AND ACRONYMS AAA Analytical Advisory Activity AF Additional Financing BHCPF Basic health care provision fund CCT Conditional Cash Transfer CHEW Community Extension Workers CMVA Contract Management and Verification Agency CPA Complementary Package of Activities CPS Country Partnership Strategy DFF Decentralized Facility Financing NDHS National Demographic and Health Survey DLI Disbursement Linked Indicators DSF Demand Side Financing EEP Eligible Expenditure Program EU European Union FGN Federal Government of Nigeria FMF Federal Ministry of Finance FMoH Federal Ministry of Health FP Family Planning GDP Gross Domestic Product GoN Government of Nigeria HCW Heath Care Workers HD Human Development HIV Human Immunodeficiency Virus HMB Hospitals Management Board HMIS Health Management information system HRITF Health Results Innovation Trust Fund HSDP Health system development project IBRD International Bank for Reconstruction and Development ICER Incremental Cost Effectiveness Ratio ICR Implementation Completion and Result Report ICT Information, Communication and Technology IDA International Development Association IHP International Health Partnership IPC Infection Prevention and Control ISR Implementation Son Review Report IVAs Independent Verification Agents LGA Local Government Area LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MCH Maternal and Child Health MDA Ministry Department and Agencies MDG Millennium Development Goals MPA Minimum Package of Activities MTR Mid-Term Review NBS National Bureau of Statistics NE North East NETSP NE Emergency Transition and Stabilization Program NHIS National Health Insurance Scheme NPHCDA National Primary Health Care Development Agency NPopC National Population Commission NSHDP National Strategic Health Development Plan NSHIP Nigeria States Health Investment Project PAD Project Appraisal Document PBF Performance Based Financing PDO Project Development Objective PER Public Expenditure Review PHC Primary Health Care PHCUOR Primary Health Care Under One Roof PIU Project Implementation Unit QALY Quality Adjusted Life Year RBF Results Based Financing RPBA Recovery and Peace building assessment SCD Systematic Country Diagnosis SLA Subsidiary Loan Agreement SMoH State Ministry of Health SPHCDA/SPHCB State Primary Health Care Development Agency / State Primary Health Care Board TA Technical Assistant TF Trust Fund ToR Terms of Reference TTL Task Team Leader UHC Universal Health Coverage WBG World Bank Group WDC Ward Development Committee TABLE OF CONTENTS DATA SHEET .................................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ................................................................6 A. Context at Appraisal ...............................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION .............................................................. 18 II. OUTCOME .................................................................................................................................. 26 A. RELEVANCE OF PDOs ............................................................................................................ 26 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 27 C. EFFICIENCY ........................................................................................................................... 34 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 36 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 37 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ...................................... 40 A. KEY FACTORS DURING PREPARATION ...................................................................................40 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 41 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ..... 45 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 45 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 47 C. BANK PERFORMANCE ........................................................................................................... 48 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 49 V. LESSONS AND RECOMMENDATIONS ....................................................................................... 50 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ................................................................ 53 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ............................ 68 ANNEX 3. PROJECT COST BY COMPONENT .................................................................................. 70 ANNEX 4. EFFICIENCY ANALYSIS ................................................................................................... 71 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS .... 74 ANNEX 6. SUPPORTING DOCUMENTS (CLIENTS ICR REPORT) ..................................................... 75 The World Bank Nigeria States Health Investment Project (P120798) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P120798 Nigeria States Health Investment Project Country Financing Instrument Nigeria Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Related Projects Relationship Project Approval Product Line Supplement P149936-Nigeria State 20-Mar-2014 Recipient Executed Activities Health Investment Project (NSHIP) Additional Financing Additional Financing P157977-Additional 07-Jun-2016 IBRD/IDA Financing Nigeria State Health Investment Project Organizations Borrower Implementing Agency National Primary Health Care Development Agency Federal Ministry of Finance (NPHCDA) Project Development Objective (PDO) Original PDO To increase the delivery and use of high impact maternal and child health interventions and improve quality of care at selectedhealth facilities in the participating states. Page 1 of 84 The World Bank Nigeria States Health Investment Project (P120798) Revised PDO To increase the delivery and use of high impact maternal and child health interventions and improve quality of care available to the people in Nasarawa and Ondo and all the States in the NE. FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing P120798 IDA-50940 150,000,000 147,676,343 135,934,558 P120798 TF-13432 21,700,000 19,833,514 19,833,514 P120798 TF-A2591 20,000,000 11,330,397 11,330,397 P120798 IDA-58510 125,000,000 87,211,222 85,067,403 Total 316,700,000 266,051,476 252,165,872 Non-World Bank Financing 0 0 0 Total 0 0 0 Total Project Cost 316,700,000 266,051,476 252,165,872 KEY DATES Project Approval Effectiveness MTR Review Original Closing Actual Closing P120798 12-Apr-2012 20-Aug-2013 27-Nov-2017 30-Jun-2018 31-Oct-2020 Page 2 of 84 The World Bank Nigeria States Health Investment Project (P120798) RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 30-Jun-2018 182.94 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Institutional Arrangements Change in Procurement Change in Implementation Schedule 08-Aug-2019 204.46 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories 12-Apr-2020 234.58 Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories 22-Jun-2020 245.46 Change in Results Framework Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 02-Jul-2012 Satisfactory Moderately Satisfactory 0 02 25-Dec-2012 Moderately Satisfactory Moderately Satisfactory .10 03 18-Jun-2013 Moderately Satisfactory Moderately Satisfactory 1.70 04 30-Dec-2013 Satisfactory Satisfactory 35.73 05 25-May-2014 Satisfactory Satisfactory 44.91 06 18-Nov-2014 Satisfactory Satisfactory 47.96 07 28-May-2015 Satisfactory Satisfactory 58.00 08 01-Dec-2015 Satisfactory Satisfactory 74.01 Page 3 of 84 The World Bank Nigeria States Health Investment Project (P120798) 09 08-Jun-2016 Satisfactory Satisfactory 88.21 10 13-Dec-2016 Satisfactory Moderately Satisfactory 107.44 11 28-Jun-2017 Satisfactory Moderately Satisfactory 133.12 12 22-Dec-2017 Satisfactory Satisfactory 169.64 13 21-Jun-2018 Satisfactory Satisfactory 183.19 14 19-Dec-2018 Satisfactory Satisfactory 190.09 15 17-Apr-2019 Satisfactory Satisfactory 197.29 16 17-Jun-2019 Satisfactory Satisfactory 201.81 17 27-Dec-2019 Satisfactory Moderately Satisfactory 219.34 18 29-Jul-2020 Satisfactory Moderately Satisfactory 246.43 19 18-Nov-2020 Moderately Satisfactory Satisfactory 255.15 SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 24 Sub-National Government 24 Health 76 Public Administration - Health 7 Health 69 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Public Sector Management 10 Public Administration 10 Transparency, Accountability and Good 10 Governance Page 4 of 84 The World Bank Nigeria States Health Investment Project (P120798) Human Development and Gender 90 Health Systems and Policies 90 Health System Strengthening 30 Reproductive and Maternal Health 30 Child Health 30 ADM STAFF Role At Approval At ICR Regional Vice President: Obiageli Katryn Ezekwesili Ousmane Diagana Country Director: Marie Francoise Marie-Nelly Shubham Chaudhuri Director: Ritva S. Reinikka Dena Ringold Practice Manager: Jean J. De St Antoine Magnus Lindelow Dinesh M. Nair, Ayodeji Oluwole Gyorgy Bela Fritsche, Fatimah Task Team Leader(s): Odutolu Abubakar Mustapha ICR Contributing Author: Onoriode Ezire Page 5 of 84 The World Bank Nigeria States Health Investment Project (P120798) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. Context at Appraisal Country Context 1. Nigeria accounts for about half of West Africa’s population with about 202 million people and one of the largest youth populations in the world. Multi-ethnic and culturally diverse, with more than 400 ethnolinguistic groups, Nigeria consists of 36 autonomous states and the Federal Capital Territory (FCT). The country has experienced robust economic growth and yet modest poverty reduction. Dynamic urban growth centers and isolated rural areas and widening social and income disparities in the context of abundant natural and human resources are documented . 2. Oil has been the dominant source of government revenue since the 1970s and has been at the heart of the political debate as well as a major driver of the creation of new states in the Federation. Nigeria shares governance challenges with other developing countries with a large amount of natural resource revenue. 3. At project appraisal, the overall macroeconomic outlook for Nigeria appeared strong, assuming that oil output stabilizes, and oil prices remain strong. Under these conditions, the foreign inflows that generated the balance of payments surplus and reserve accumulation were expected to continue and help stimulate domestic demand that had been driving growth. Stronger oil production alleviated the then tight financial constraints on Government programs. There was strong potential for accelerated growth and job creation if key infrastructure and institutional constraints were to be successfully addressed, along with a clarification of regulatory conditions for oil and gas. 4. States in Nigeria operate with a high degree of autonomy. States and local governments in Nigeria control more than half of the nation’s revenue and are responsible for the provision of services that directly influence growth and poverty, such as primary and secondary education, health care, water and sanitation, rural infrastructure, and community services. The high degree of autonomy provides states, particularly those with dynamic and progressive leadership, an opportunity to move ahead on their own. This autonomy, however, also poses a challenge to building national consensus across different levels of government, including in the areas of macroeconomic stability, prioritizing public resource allocation, and meeting minimal national standards in service delivery. 5. Limited inclusiveness implies wide divergences in service performance between geopolitical regions and states. Three trends emerge concerning the geographic distribution of the poor. First, in the North East and North West, poverty rates remain very high: on average above 50 percent and in some cases above 70 percent with a tendency towards stagnation. Second, the North Central is doing well in particular around Abuja. Finally, the picture in the South is more heterogeneous. On average, this area is performing better, poverty rates are well below those in the North and practically all standard poverty correlates indicate the generally higher level of well-being. Furthermore, most of the important economic activities are in the South and the level of urbanization is higher. However, mainly the South Western part, where Lagos is located, enjoys relatively inclusive growth which is credited with giving that region the fastest poverty reduction rate in the country. Page 6 of 84 The World Bank Nigeria States Health Investment Project (P120798) Sector and Institutional Context 6. Nigeria fared poorly on key health indicators. At the time the Nigeria States Health Investment Project (NSHIP) was designed, Nigeria’s progress on key health indicators was deemed critical to the continent. Health outcomes in Nigeria were poor. With 545 maternal deaths for 100,000 live births in 2008, 10 percent of global maternal deaths occurred in Nigeria. The under-five child mortality rate, at 157 per 1,000 live births (2008), was also declining too slowly to achieve the MDG4 target of less than 67 per 1,000 live births by 2015. This was despite relatively high levels of health spending, compared to other parts of Africa, both in absolute terms and as a proportion of gross domestic product (GDP): total health spending in Nigeria was estimated at around US$30 per capita, amounting to about 6.5 to 7.4 percent of GDP. Comparing the data from 2003 and 2008 Demographic and Health Surveys (DHSs), it was evident that Nigeria had made limited progress in delivering critical health services. 7. Inter-regional and inter-state disparities in health outcomes were also stark. In Adamawa, Nasarawa and Ondo states, for example, there was a wide variation in key maternal health indicators which is typical of the country as a whole. Table 1: Selected health indicators in project states (source: 2008 NDHS report) States Received antenatal Received protection Delivered by a Delivered in a care from a health against tetanus during health health facility professional last pregnancy professional Adamawa 61.2 42.2 14.6 10.7 Nasarawa 72.6 35.3 33.8 32.9 Ondo 70.1 63.4 50.5 46.9 8. There were large income related inequalities in health outcomes: There were severe income related inequalities in health outcomes and in health care utilization: the poor had far worse health outcomes than the wealthy in Nigeria. These inequalities were worse than what was encountered in the rest of West and Central Africa. Table 2: Income Inequality in health outcomes and health coverage Nigeria West & Central Africa Indicators Poorest 20% Richest 20% Rich/Poor ratio Rich / Poor ratio Under 5 mortality (1,000) 219 87 2.5 1.7 Full immunization coverage (%) 4.8 52.7 11.0 2.5 Antenatal coverage (%) 23.5 93.8 4.0 1.8 Skilled birth attendance (%) 8.3 85.7 10.3 3.8 Source: Gwatkin 2011 based on DHS 2008. (lifted from the NSHIP original PAD). 9. Poor quality of care: Most Primary Health Care (PHC) facilities were overstaffed but often lacked skilled practitioners and a large percentage of facilities did not have basic pharmaceuticals in stock. Not surprisingly, more than 50 percent of the households were dissatisfied with the services in public facilities and used them infrequently. In addition, there was a shortage of supervisory staff, supervision budgets were insufficient, and responsibilities had not been clearly laid out. As a result, there was little monitoring of service delivery at health facilities. Page 7 of 84 The World Bank Nigeria States Health Investment Project (P120798) 10. Low coverage of effective, simple, and low-cost interventions: There was low coverage of effective interventions that can prevent many maternal and child deaths. Although the PHC system aims to provide basic PHC interventions, utilization of these services was low due to problems in service delivery at health facilities. The government health expenditures focused more on building of hospitals, payment of salaries and provision of specialist services rather than more on cost-effective public health activities or the financing of an essential health service package delivered close to the community (NSHIP PAD 2012). Population groups who need these services to reduce the burden of disease, and particularly women and children were too often neglected. 11. Lack of Accountability and Transparency in the Allocation and Management of Resources: States, LGAs and facilities allocated inadequate resources towards the provision of basic health care. Both state and LGA governments receive funds via sub-financing over which they have considerable autonomy. However, there was little accountability for how these funds are allocated across sectors and what the investment of these funds has achieved. There was limited federal oversight of state budgets and minimal state level review of LGA budgets. Furthermore, performance monitoring of service providers was weak as are structures to ensure quality and accountability in service delivery. 12. Lack of incentives for achieving results: There were several constraints to productivity within the PHC delivery system, including: • The allocation of too high a proportion of sector spending to unskilled compared to skilled staff (and this was largely due to discretion of hiring unskilled staff by the LGAs); • The failure to allocate staff resources between facilities based on their productivity or need; • The allocation of a major portion of total spending to staff costs leaving facilities with a shortage of non- staff resources; and • Lack of motivation among health workers and government officials and little consideration for results. 13. Medical and Health Workers' Union. Nigeria has a highly organized health workers’ union that aims to protect the economic interests of its members along with promoting high standards of workmanship and professional practice. While the existence of such union has many benefits for healthcare works and is meant to enhance quality of health services, it has been evident that staff remuneration takes top priority in Government spending even when the system is delivering a very low level of service to its clients. 14. Despite several decades of financial and human capital investments, coverage of key interventions was low even by Sub-Saharan African standards, quality of care was inadequate, and services do not reach the poor. Funding for health was largely input-based. Continuing input-based financing was considered as not a viable option. Bold innovations and reforms were necessary to shift the focus to strengthening service delivery and improve health outcomes. There was thus an urgent need to shift the focus to results-oriented service delivery, geared towards enhancing both quantity and quality of services. Equally, there was a need to decentralize funding to health facilities while at the same time enhancing their decision rights on this money and holding health facilities accountable for results. Furthermore, there was a need to hold the sub- national health administrative structures accountable for their supportive functions while at the same time enhancing their funding to strengthen their core functions related to supervision and coaching. Government Response 15. The Federal Government of Nigeria (FGN) endorsed the National Strategic Health Development Plan 2010-2015 (NSHDP) which marked the first clear shift by the government towards focusing resources and Page 8 of 84 The World Bank Nigeria States Health Investment Project (P120798) measuring results. State plans were developed along the lines of the NSHDP. As a signatory to the International Health Partnership (IHP+) Compact (2008), FGN committed to: • Actively support a national sector planning process; • Harmonizing government and donor funding to ensure long-term predictable financing for the health sector; • Adopting a uniform result monitoring framework, with a strong focus on achieving outcomes; and • Using the Compact (a signed expression of FGN commitment) that harmonizes donor funding commitments with Government inputs as the platform for measuring results against agreed outcomes. The Government used the NSHDP as a foundation for the Compact, thereby reinforcing national priorities. 16. In the last decade preceding the commencement of NSHIP, there had been a growing focus on enhancing key health outcomes and achieving the defunct MDG targets 4 (reducing child mortality) and 5 (enhancing maternal health). Having established the National Health Insurance Scheme (NHIS) in 1999, the FGN set aside 5 percent of total debt relief funds received to launch the NHIS-MDG initiative in 2008 to accelerate the achievement of the defunct MDGs 4 and 5. The initiative tested the viability of the NHIS mechanism to significantly reduce maternal and U5 child mortality through a combination of demand side financing, public-private partnerships, and a strong referral system. The NHIS/MDG scheme, which was piloted in six states prior to being scaled up country-wide, addressed several of the same institutional requirements as a Performance-based Financing (PBF) program such as separation of functions, increased autonomy at the health facility level, effective monitoring and paying for performance. An added sign of the Government’s continued commitment to the health sector was the health bill which created the Basic Health Care Provision Fund (BHCPF) to be financed by 2 percent of the Federal Consolidated Fund. The Fund would be used to provide a basic minimum package of health services including essential drugs, facility maintenance, equipment, transport and staff capacity building. Relevance to country priorities and relationship to Country Partnership Strategy 17. High Level of Ownership: The FGN was committed to expanding use of Result-based Financing (RBF) approach and participated in extensive discussions with the World Bank prior to and during preparation of the project. Before project preparation, the Governors of the participating states (Adamawa, Nasarawa and Ondo) signed a policy note endorsing the project for their individual states. The three Governors also initiated reforms that provided health facility autonomy and promote community participation in management of health facilities. Notably two of the states had established with legislative backing their State Primary Health Care Development Agencies (SPHCDA) and the third state was at final stages of obtaining legislative and executive assent. The State governments had also set up a RBF Technical Support Unit in each state. The units were staffed by government officials that had been trained in the implementation of PBF. The PBF pre-pilot interventions commenced in three LGAs, one per state, in December 2011. The state governments provided the equivalent of US$100,000 each to support implementation of the pre-pilot which lasted for about two years. 18. The NSHIP was aligned with the Africa Strategy (2011) and Country Partnership Strategy 2010-2013 (CPS). The Africa Strategy – Africa’s Future and the World Bank’s Support to it – is founded on strengthening governance and public sector capacity, including through enhancement of incentives within the civil service. In particular, the Africa Strategy recognizes that critical services are too often either not delivered or delivered poorly due to weak management of public funds. Notably, the Strategy supports initiatives to empower citizens to get information on entitlements, as well as voice grievances when services are not Page 9 of 84 The World Bank Nigeria States Health Investment Project (P120798) properly delivered. The 2010-2013 CPS for Nigeria focused on improving governance and promoting human development by creating improved access to quality health services, increase vaccination coverage and better maternal care services. 19. The NSHIP therefore came at an opportune time to demonstrate that RBF approaches can accelerate progress on health outcomes in Nigeria. The Project: • Tested the effectiveness of results-based management tools through programmatic financing Box 1: Brief description of Result Based linked to two different modalities of decentralized Financing Used under the NSHIP financing for health facilities and a set of Project disbursement linked indicators (DLIs) at state and local government levels; Results-Based Financing: RBF, is any • Built social accountability through community program that rewards the delivery of management of health facilities and performance one or more outputs or outcomes by measurement of health outputs; and one or more incentives, financial or • Invested in strengthening monitoring and evaluation otherwise, upon verification that the (M&E) mechanisms. agreed-upon result has actually been delivered. 20. NSHIP built on lessons from the Health Systems Development Projects 2 (HSDPs) P070290 and principles of Performance-Based Financing: PBF, is a fiscal decentralization to support targeted health systems form of RBF distinguished by three reforms in three states. Specifically, it built on the lesson on conditions. Incentives are directed only the need for a results-focused approach in both strategic to providers, not beneficiaries; awards planning and project implementation (HSDP 2 ICR report). are purely financial--payment is a fee The HSDP 2 was a $153.22m project that was implemented for service; and payment depends in Nigeria between 2002 and 2012. The overriding goal of the explicitly on the degree to which project was to assist the Nigerian Health authorities in their services are of approved quality. efforts to redress the serious deterioration in the delivery of basic health care services following decades of neglect and Disbursement Linked Indicator (DLI) build institutional capacities, paving the way for a more approach provides funds to sustained development of the Nigerian health care system. government based upon achieving specific and measurable goals or Theory of Change carrying out specific objective actions. 21. At both project appraisal and restructurings, the project did not specifically outline its theory of change as it was not required then. Hence, the ICR team had to extrapolate the theory of change at appraisal from the project description as illustrated below (figure 1). 22. The NSHIP was designed primarily to address the challenges mentioned above (paragraphs 6-14). The project aimed at increasing the delivery and use of high impact maternal and child health interventions and improving the quality of care at selected health facilities in the participating states. To achieve these outcomes, key sets of inputs were identified with sets of activities. As depicted in figure 1 below, the key inputs are result-based financing and technical assistance to different levels of governments to improve PHC supervision and transparent and accountability of use of health resources. Page 10 of 84 The World Bank Nigeria States Health Investment Project (P120798) Figure 1: NSHIP Project Theory of Change Project Development Objectives (PDOs) and Beneficiaries 23. NSHIP’s original objective was to increase the delivery and use of high impact maternal and child health interventions and to improve the quality of care at selected health facilities in the participating states. The project was designed to benefit the entire population of the three states: Adamawa, Nasarawa, and Ondo States (about 9.4 million). Specifically, the Project aimed to annually benefit about 400,000 pregnant women in three States; about 2 million women between 15 and 49 years (795,000 in Adamawa, 400,000 in Nasarawa and 850,000 in Ondo); and about 1.8 million children under five years of age (636,000 in Adamawa, 375,000 in Nasarawa and 775,000 in Ondo). Key Expected Outcomes and Outcome Indicators 24. The three expected outcomes and outcome indicators are indicated in Table 3 below. Page 11 of 84 The World Bank Nigeria States Health Investment Project (P120798) Table 3: NSHIP Outcomes and indicators (PDO and intermediate indicators) Outcomes Outcome Indicators (PDO level and Intermediate Result/IRI) Increase in the delivery and use of • Proportion and number of births attended by skilled health providers high impact maternal health (PDO) interventions • Direct Project Beneficiaries (number), of which female (%) (PDO) • TB Case Detection Rate (IRI) Increase in the delivery and use of • Proportion and number of 12-23 months old children fully immunized high impact child health (PDO) interventions • Number of outpatient visits by children under five (PDO) Improvement in the quality of care • Average Health Facility Quality of Care Score (PDO) at selected health facilities in the • Proportion of on duty technical staff present at health facility on the participating states day of the survey (IRI) • Proportion of primary health facilities having essential medicines and commodities in stock (IRI) Table 4: Original PDO baseline and targets Original Indicator Unit Original Baseline Original Target Data Source PDO level indicators Nasarawa: 86,237 Nasarawa: 120,731 No. Adamawa: 78,180 Adamawa: 109,452 HMIS Proportion and Number of 12-23- Ondo: 106,227 Ondo: 148,717 month-old children fully immunized Nasarawa: 16.1% Nasarawa: 45% Household % Adamawa: 19.1% Adamawa: 49% survey Ondo: 22.6% Ondo: 53% Nasarawa: 23,193 Nasarawa: 26,671 No. Adamawa: 24,520 Adamawa: 28,198 HMIS Proportion and Number of births Ondo: 29,247 Ondo: 33,634 attended by skilled personnel Nasarawa: 33.8% Nasarawa: 45% Household % Adamawa: 14.6% Adamawa: 25% survey Ondo: 50.5% Ondo: 60% Average health facility quality of Baseline + 20 % Baseline not available Facility Survey care score percentage points Nasarawa: 36,591 Nasarawa: 40,982 Number of out-patient visits by No. Adamawa: -- Adamawa: -- HMIS children under five Ondo: 301,372 Ondo: 347,616 Direct project beneficiaries (no) No. 0 Baseline +10% points HMIS female beneficiaries (%) % 0% Baseline +10% points HMIS Intermediate Result Indicators (IRIs) Reports of TB Case Detection Rate % 30.50% 43% NTBLCP Proportion of on-duty technical Baseline + 25 staff present at health facility on % 0% Facility Survey percentage points the day of the survey Proportion of primary health Baseline + 15 Quality facilities having essential medicines % Baseline not available percentage points Checklists and commodities in stock Number of LGAs with ongoing PBF PBF not scaled-up: 27 State report to No. 3 pilot PBF scaled-up: 53 WB Number of LGAs receiving grants State report to No. 0 30 linked to performance WB Page 12 of 84 The World Bank Nigeria States Health Investment Project (P120798) Original Indicator Unit Original Baseline Original Target Data Source Number of states disclosing report cards detailing LGA level State report to No. 0 3 performance made available in WB public domain Components 25. The original project was financed by an IDA Credit (US$150 million) and HRITF Grant (US$20 million) for a total of US$170 million (actual cost: IDA-US$135.93 and HRITF-19.83 for a total: US$155.76 million). The original NSHIP project had two components: Result Based Financing and Technical Support. These are summarized in table 5 below. Table 5: Original NSHIP project components and the budget allocated to them. Components Description US$ m US$ m US$ m 1 Results Based Financing 122.4 1A Strengthening Service Delivery 93.5 1.A.1 Performance Based Financing (PBF) 57.3 1.A.2 Decentralized Facility Financing (DFF) 31.5 1.A.3 PHC Departments 4.7 1.B Strengthening Institutional Performance 28.9 1.B.1 DLI LGAs Support 14.6 1.B.2 DLI State Support 14.3 2 Technical Support 34.0 2.A Technical Assistance to States 9.0 2.B Technical Assistance for Results Based Financing 15.0 2.C Monitoring and Evaluation 10.0 Total 156.4 Project preparation 3.0 Contingencies 10.6 Total project cost 170.0 • Component 1: Results Based Financing (original estimated cost: US$122.4 million). 1A: Strengthening Service Delivery: This sub-component was designed to support the implementation of performance-based funding for front line health workers in selected health facilities and agencies. This sub-component was to be delivered through: 1.A.1 Performance Based Financing (PBF): The PBF facility activity paid for quantity and quality of service delivered. At health facility level, the project tested and implemented PBF in which targeted health facilities were rewarded with cash for the quantity and quality of services they provide (e.g. the number of outpatient visits and births attended). Performance (service delivery and quality of services delivered) were verified using robust and independent methods. The facilities were given considerable autonomy in how they use the funds they earn, including for (i) health facility operational costs (about 50 percent), maintenance and repair, drugs and consumables, outreach and other quality-enhancement measures; and (ii) performance bonus for health workers (up to 50 percent). Page 13 of 84 The World Bank Nigeria States Health Investment Project (P120798) In order to incentivize improvements in quality of care at the secondary level, including referral from PHC facilities, the project tested a similar PBF approach in general hospitals and in a specialized secondary hospital for maternal and child health. This started with one general hospital in each LGA and the Mother and Child Hospital in Ondo. Based on the lessons learned from the Mother and Child Hospital in Ondo, this approach was rolled out to one MCH specialized secondary hospital each in Adamawa and Nasarawa. The quantity of services delivered was verified prior to making payments. Each PBF facility reported quarterly on delivery of agreed outputs through a standard invoice. The quantities reported were systematically verified by SPHCDA. The SPHCDA contracted the PHC departments of the LGAs to verify quality of service delivery. A peer review mechanism for carrying out quality verification at the hospital level was established. Based on performance data (both quantity and quality), SPHCDA approves the quarterly amounts to be paid to each PBF health facility. ICT solutions include online entry of information and cloud computing to improve transparency, allow faster processing, and facilitate continuous monitoring. Community based ex-post verification was carried out in two ways. First, grass-root organizations visited homes of randomly chosen clients (selected from the registers in facilities) to determine whether they existed, whether they received the services that have been paid for, and what their opinion is on these services. The SPHCDA had program officers dedicated to this task. Each quarter, a random sample of health facilities and services was assessed. Each defined service was recorded with a client address and where possible a mobile phone number through which a client can be reached. 1.A.2 Decentralized Facility Financing (DFF): The design of PBF and DFF were almost the same but with some variations. In selected LGAs, the Project supported one health center in each ward and one general hospital in each LGA with DFF for minor facility improvements and / or operational costs. By design, DFF facilities received 50 percent of the funding that PBF facilities received, as PBF facilities used half of their income for health facility incentive payments, while DFF facilities were not supposed to use income for health worker incentive payments. 1.A.3 PHC Departments: The PHC Departments in the participating LGAs received performance bonuses and operational support based on internal contractual arrangements and a performance matrix, which importantly measured prompt and correctly applying the quarterly quantitative quality checklists to PBF and DFF facilities. These checklists were submitted on a quarterly basis as evidence of performance to and were vetted in turn by SPHCDAs. Before submission, these checklists were discussed and approved at an LGA level steering committee meeting. LGA-PHC departments were under two types of incentive schemes: the internal performance contracts through the PBF and DFF systems, and the DLI approach (see below). 1.B Strengthening Institutional Performance: This sub-component disbursed against State/LGA eligible expenditure program expenditures conditioned on compliance with annual DLIs. This subcomponent supported LGAs and the state. B.1 DLI LGAs Support: The LGAs entered into a performance contract with the state that provided the LGAs with an annual sub-financing of up to US$50,000 based on their performance against DLIs in the LGA Performance Scorecards. The funding reimbursed the LGAs for specified eligible expenditures. Table 6 below shows the LGA performance score cards. Page 14 of 84 The World Bank Nigeria States Health Investment Project (P120798) Table 6: LGA Performance Scorecards S/N Disbursement Linked Indicators Weight A Health Budget Approved as per Chart of Accounts and published 20% B LGA facilities receive quarterly supervision as verified by quarterly 20% supervision checklists C LGA prepares quarterly HMIS reports on 4 key indicators and describes % of 20% facilities reporting D Maintain staffing norms 20% E Quality assured drugs (i) NAFDAC registered and (ii) PCN registered 20% 1.B.2 DLI State Support. States entered into an agreement with the Federal Government that provided the States with an annual sub- financing of up to US$500,000 per state, based on the performance against the state level DLI matrix (see Table 6 below). It was agreed that additional sub- financing of up to US$1,000,000 per State will be made in years 2 and 4 for DLIs including skilled birth attendance, immunization coverage, and health facility quality score as verified by independent household and facility surveys. Table 7: State Performance Scorecards S/N Disbursement Linked Indicators Weight A % of PBF Health facilities receiving payments in a timely manner 20% B Outpatient visits per capita 15% C Fully immunized children 15% D Deliveries at PBF health facilities 15% E Health Budget Approved as per Chart of Accounts and published 15% F Publishing of annual State DLI Report and LGA DLI Scorecard 10% G Release of LGA DLI grants in a timely manner 10% At the end of each year, each State was expected to prepare a report justifying the corresponding value of each DLI that the State and LGA have complied with. It was planned that the NPHCDA and the Bank will carry out annual review that assesses the state report, carry out an independent assessment and make proposals for disbursements. In order to incentivize States and LGAs to achieve unmet DLI results, a protocol was agreed to allow payments, at the Bank's discretion, for verified partially or late achieved DLI results. The Bank, in such cases, would determine the amount and timing of payments based on evidence provided. • Component 2: Technical Support (original estimated cost: US$34 million): The project also provided technical assistance (TA) to build the capacity of institutions at the federal, state, local government, and facility levels to implement both PBF and the DLI approaches. A robust M&E system was to be developed based on household and health facility surveys carried out by local institutions. The surveys were seen to be particularly important to track progress and provide evidence on whether the RBF approaches should be scaled up. This component has three subcomponents: 2.A Technical Assistance to States: Technical support for RBF implementation in the states funded hiring of consultants to strengthen monitoring and capacity building activities as well as for covering operational costs for day-to-day project management. This was managed by the SPHCDAs. 2.B Technical Assistance for Results Based Financing: This supported intensive technical support to Page 15 of 84 The World Bank Nigeria States Health Investment Project (P120798) build capacity for managing the RBF program, specifically its PBF component and its long-term sustainability through rigorous learning, institutional development, community engagement, and dissemination of results. The TA was also used to engage grassroots organizations to conduct community surveys and receive support for operating costs and other expenditures for day-to-day management of its activities under the project. 2.C Monitoring and Evaluation: This TA sub-component, managed by the FMOH, had two objectives: (i) collect data to measure progress on the Results Based Financing component and the project results framework and carry out the impact evaluation of NSHIP; and (ii) manage and disseminate lessons from NSHIP to benefit wider Nigerian health policy and practice. This sub-component financed: the carrying out of health facility and household surveys; operating costs for strengthening oversight and management of the project; and knowledge dissemination and studies. Implementation and Institutional Arrangements 26. The NSHIP Project was implemented largely at the state, LGA and health facility levels with a smaller but critical role at the federal level. The Project leveraged and strengthened existing institutions to enable them to take on additional responsibilities under the project. The ultimate sectoral authority for the project was with the Federal Minister for Health. The two lead agencies that implemented the project at the federal level were: (i) the Federal Ministry of Health (FMOH); and (ii) the National Primary Health Care Development Agency (NPHCDA). The ultimate authority for the project at the State level was with the Commissioner of Health. Institutions involved in the implementation of the project at the state level included: (a) State Ministry of Health; (b) State Primary Health Care Development Agency (SPHCDA); and (c) LGA primarily through its PHC Department. 27. Implementation and institutional arrangements at Federal level (i) The Federal Ministry of Finance is the primary interlocutor for the Bank and performed the following functions: (a) provided guidance and stewardship to the project on all financial matters; (b) participated in the annual and other review missions; (c) was a member of the project steering committee; and (d) facilitated inter-ministerial and federal-state coordination. (ii) The Federal Ministry of Health (FMOH) is the lead administrative authority which: (a) commissioned NBS and NPoPC to undertake the periodic household and facilities surveys; (b) worked with the NPHCDA on the Health management Information System (HMIS) and operations research; (c) managed the project impact evaluation; and (d) chaired and provided the secretariat for the Project Steering Committee (PSC). (iii) The NPHCDA is the lead technical body which: (a) oversaw RBF TA through the hiring of a consulting firm; (b) verified the DLI results achieved by the State and LGAs; and (c) was responsible for the technical and financial audits. Project implementation at this level was also be supported by the PSC with representation from federal, state, and local government institutions and members of civil society. The PSC was responsible for overall monitoring of project implementation by the project teams and various contracting agents. A Technical Working Group (TWG), comprising senior technical persons in NPHCDA and FMOH assisted with monitoring project implementation, refining project design, troubleshooting implementation issues, overseeing RBF Technical Assistance, and providing oversight for the evaluations. 28. Implementation and institutional arrangements at State level: At the State level the State Ministry of Health (SMOH), provided overall stewardship to the Project. SPHCDA was the agency charged with the implementation of the Project, including planning, management, and monitoring of project activities. A State Page 16 of 84 The World Bank Nigeria States Health Investment Project (P120798) RBF Technical Support Unit, headed by a Coordinator seconded from the SMOH, was located within the SPHCDA. The RBF unit liaised with all the various implementing agencies, continuously tracked and reported on progress on PB/DFF facilities, DLIs and EEP, verified that all fiduciary requirements (financial management (FM) and procurement) and safeguards were being observed, acted as the focal point for all communications/reporting to the Bank, coordinated a dissemination campaign, troubleshoot and mobilized technical assistance as necessary. The State Project Financial Management Units (SPFMUs) were the principal fund holder agencies for the PBF, DFF and LGA PHC Departments. SPFMUs were also provided required information on the EEP expenditures for the fiscal year to enable the Bank to make DLI payments. The State Ministries of Finance and Local Governments and Chieftaincy Affairs, the Local Government Service Commission, and the Governor‘s office were expected to be involved in Project management at the state level, through the State RBF Steering Committee. State level PSC and TWG were to play a role similar to those established at the federal level. 29. Implementation and institutional arrangements at LGA level: At the LGA the LGA through the LGA Chairman and Supervisory Councilor for health were responsible for overseeing the activities to ensure smooth implementation of the project, providing the annual DLI report, and liaising with SPHCDA. The LGA PHC Department as the lead institution for PHC in the LGA was responsible for overseeing implementation of the project and provided quality supervision to selected PBF/DFF facilities. An RBF steering committee was set up at the LGA to review and validate the measures obtained by the SPHCDA (quantity) and the LGA PHC Department (quality). Specifically, the committee was setup to: • Present and discuss the data and information related to the PBF/DFF including activity level, quality of care level and other relevant information and review strategies in place for enhancing results and follow-up on previous decisions of the committee. • Give an opportunity to committee members and representatives of the health facilities to express any challenges in implementing the program and address stakeholders‘ concerns. • Review, discuss and eventually approve the final consolidated quarterly invoices of PBF/DFF prior to transmission to the SPHCDA. This entails validation of every single original monthly PBF/DFF invoice, and all the quality scores, with the consolidated quarterly LGA PBF invoice. • Review and discuss the performance of the LGA PHC department which is under a performance contract to carry out supervision. 30. This LGA RBF Steering Committee provided the nexus for community level accountability and governance as the LGA, SMOH and SPHCDA were represented, alongside civil society organizations. Objectively verifiable performance data were presented. This was particularly critical because the information was presented at a level where important resource allocation decisions were expected to be made, where the knowledge on local circumstances is best, and where the accountability for the LGA health sector performance lies. 31. Implementation and institutional arrangements for NSHIP at the facility level: PHC Facility, RBF Committees and Hospital RBF Committees are to monitor service quality and delivery at health facilities and ensure that communities were well served by their health care providers. The PHC Committees have a strong linkage with the Ward Development Committee (WDC). The Committees were expected to oversee the quantity and quality of performance of the facility, approve and monitor the implementation of the facility level business plan, approve utilization of funds received under PBF and DFF with the Chair of the Committee designated as the co-signatory for the facility bank account and review and approve the performance appraisal of health workers. 32. Other coordination modalities: The project design required: a coordination mechanism to be established Page 17 of 84 The World Bank Nigeria States Health Investment Project (P120798) in each of the three Project states to support implementation. An extended team was meeting monthly on a predetermined date and time, and consisted of SPHCDA verifiers, LGA PHC coordinators, SMOH and selected development partners with field presence at the LGA levels. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 33. Changes during implementation of the project: PDO change; provision of additional HRITF, GFF and IDA funds to scale-up the project; partial cancelation of the project budget; addition of new components and dropping of some sub-components; extension of the closing date; and revision of the results framework. The changes were in response to the changing realities on the ground and were requested by the Government of Nigeria. 1st Restructuring and AF (March 2014): Additional HRITF Grant ($1.7m) 34. A qualitative analysis of major barriers to accessing health care found that availability and cost of transport, and cost of services as well as cultural barriers were the main barriers to increasing facility utilization of healthcare services in the study area1. These findings were found to be relevant in the context of NSHIP and to address this need early on in project implementation, an additional financing (AF) from the HRITF was needed which resulted in the first restructuring of the project. The AF led to the inclusion of sub- component 1C. “Strengthening Community Demand” for services. The interventions under this sub- component targeted poor households and poor communities in the participating LGAs, and were also based on results-based financing approaches, in line with the original results framework of the NSHIP. This first restructuring was a level two restructuring, which started in July 22, 2013, and was approved by the RVP on March 20, 2014. This first restructuring did not lead to revision of the PDO or PDO level indicators. Both the HRITF GA and IDA FA were amended to reflect these changes. 2nd Restructuring and AF (June 2016)2: Additional IDA Credit $125m and GFF Grant ($20m) 35. In 2015, in response to the Boko Haram insurgency that affected the NE part of Nigeria, the Nigeria Government in collaboration with the EU, WBG and UN undertook the Recovery and Peace Building Assessment (RPBA). The RPBA informed the design of the North East Emergency Transition and Stabilization Program (NETSP) which initially targeted the short and medium-term recovery, stabilization and peace building needs. Five HD operations and one Agriculture project, National FADAMA Development Project (FADAMA) III AF, were included in the NETSP and structured as a coordinated umbrella program that included interventions /approaches that can rapidly be scaled-up or re-engineered to benefit the vulnerable populations in the North East (NE). The coordinated approach was aimed at ensuring the greatest synergy and avoidance of duplication. The RPBA found that about 20 percent of health facilities were damaged or destroyed in the six NE states at a replacement cost of about US$150 million. The RPBA observed that health facilities were deliberately targeted by the insurgents and besides damage to the infrastructure and equipment, drugs were stolen, and health workers threatened. The RPBA identified infrastructure and social services (including health and nutrition) as some of key priority needs of the NE. Results from the implementation of NSHIP in the 3 pilot states as well as its implementation in Adamawa State during the insurgency led to it being considered as an appropriate strategy to respond to the situation in the NE. Using AF instead of initiating a new project reduced the project preparation time and allowed the Bank to quickly respond to the request of the Government. The AF implemented results-based approaches in-line with the 1 Mabuchi, S., et al. (2018). "Pathways to high and low performance: factors differentiating primary care facilities under performance-based financing in Nigeria." Health Policy and Planning 33: 41-58. 2 The date in the system should read June 2016 and not June 2018. Page 18 of 84 The World Bank Nigeria States Health Investment Project (P120798) original theory of change of the project. 36. This first level restructuring led to the revision of the PDO to include all the NE States. The revised PDO was : “to increase the delivery and use of high impact maternal and child health interventions and improve quality of care available to the people in Nasarawa and Ondo and all the States in the North East”. All the PDO indicators were revised splitting them into Ondo and Nasarawa and North East states and new ones were added to capture the NE States together. The results framework was revised and baseline values for both revised and new indicators were updated to reflect the actual period the project became effective. Other main changes included revision of components one and two and the addition of a new third component (Partnerships to Strengthen Service Delivery), to adapt to the specific needs of the NE region, and the extension of the original closing date from June 30, 2018 to June 30, 2020. The AF was approved by the Board on June 30, 2016 and the amended FA became effective on February 27, 2017 after delays in the signing of subsidiary loan agreements by the states. Box 2: Using PBF in Conflict-Affected Areas: Lessons from Adamawa Boko Haram invaded and captured seven of the 21 LGAs in Adamawa State in October 2014. The insurgents were in complete control of these areas until early February 2015 when five of the occupied LGAs were secured by the Nigerian military. Basic health services were completely shut down with many health facilities damaged with equipment and drugs stolen. In February 2015, the leadership of Adamawa State Primary Health Care Development Agency (SPHCDA) was faced with the problem of providing health care to the civilians returning to these LGAs as the military was gaining the upper hand in the five affected LGAs. Fortunately, staff in one of the affected LGAs, Mubi South, had been trained in performance-based financing in September 2014 and they were in the process of developing their business plans. To address the situation in Mubi South: (i) initial investment was doubled for all affected health centers (from US$5,000 to US$10,000), in the district hospital (from US$10,000 to US$20,000); and (ii) health workers were supported to return to their duty posts. As in other PBF LGAs, decision making was decentralized, there was a high degree of community involvement, and the performance bonuses were used to improve outreach as well as infrastructure. Since there were secure LGAs which began implementation of PBF at about the same time (Yola South and Song), it’s possible to judge how well PBF performed in a conflict affected LGA, like Mubi South. Mubi South performed at least as Restructuring 3rd well (August as the other 2019) LGAs and made rapid progress in attracting patients. The findings were similar for skilled birth attendance, immunization, and family planning, indicating that PBF can work well The 37. in Project an LGA whichwaswasrestructured in August 2019 following the mid-term review of the Project just recently secured. implementation in November/December 2017 as follows: (i) the baselines and targets in the results framework were revised to match the start of project implementation which began in 2014; (ii) changes were introduced to the verification methodology in Ondo State as agreed between the Government and the Bank as a mitigating measure to prevent over-reporting of outputs and to ensure that (a) the services provided by the PBF facilities in Ondo state were actually reaching their catchment population; (b) the services were 100 percent verified across all the PBF LGAs; (c) there was complete separation of function between independent verification agency (IVA) and the firm providing technical assistance; and (d) the Ondo PBF facilities continued to be incentivized based on their performance; (iii) change project components and component costs (stop payments against LGA and State DLIs) and disbursement estimates; (iv) reallocate funds between disbursement categories under Credit No. 50940 and TF013432; create new disbursement sub-categories and reallocate funds within IDA 58510 to optimize service delivery by States that have been performing well in improving PDO indicators, mainly Adamawa and Nasarawa; and (v) convert the method of payment to PBF facilities in Ondo state based on 100 percent verification through household-based Lot Quality Assurance Sampling (LQAS) surveys (changed from facility-focused to population-focused counter- Page 19 of 84 The World Bank Nigeria States Health Investment Project (P120798) verification procedures). Payments against all LGA and State Disbursement-Linked Indicators (DLIs) were stopped as the indicators have been achieved prior to 2019. Additionally, a World Bank financed P4R project Saving One Million Lives (SOML) was also monitoring the same indicators under NSHIP; thus, post the MTR, the Government and the World Bank agreed for the NSHIP DLIs to be monitored under the SOML project. 4th Restructuring (April 2020) 38. Following a request by the Government on March 12, 2020, the project was further restructured on April 12, 2020 through a level II restructuring to implement a partial cancellation of IDA 58510 in the amount of US$25 million equivalent. The cancellation was due to the team’s assessment and in discussion with the Government that the project would not to be able to implement the remaining project activities before its closing date of June 30, 2020. This was mainly due to the delay in implementation of certain project activities, a consequence of initial delays in project effectiveness as well as limited ability to implement in Borno and Yobe states due to insecurity. 39. This restructuring also entailed changes to components costs as well as reallocation between disbursement categories as summarized below: Change in amount allocated to Category schedule: i. Category 1A: Goods, works and services required for Part 1.A.1 of the Project and to be financed out of PBF Grants and LGA PHC Department Grants on an output basis at the respective Unit Price for Borno state from 20,600,000.00 to 8,156,357.08 expressed in SDR; ii. Category 1E: Goods, works and services required for Part 1.A.1 of the Project and to be financed out of PBF Grants and LGA PHC Department Grants on an output basis at the respective Unit Price for Yobe state from 11,500,000.00 to 8,253,832.28 expressed in SDR; iii. Category 6A: Goods, works and services required for Part 1.A.1 of the Project and to be financed out of PBF Grants and LGA PHC Department Grants on an output basis at the respective Unit Price for Adamawa State from 11,000,000.00 to 8,655,545.54 expressed in SDR 5th Restructuring June 2020 40. The project was further restructured in June, 2020 to enable the client to better deliver the project services amid the COVID-19 epidemic in the six additional project states (NE States) to: (i) compensate for the delay in implementation of project activities for the second quarter of 2020 due to stringent lock-downs in project states and FCT following the Covid-19 pandemic, effectively bringing about a halt to many planned project activities; and (ii) allow for the continuation of the delivery of RMNCH services within a Covid-19 pandemic by procuring necessary protective personal equipment, medicines and other medical equipment and commodities that would contribute to the achievement of the project development objective. The operations in two parent states Ondo and Nasarawa were closed on 31 May, 2020. The restructuring included: (a) extending the NSHIP AF (IDA Credit 58510 and GFF TF0A2591) closing date by four months to October 31, 2020; (b) reprogramming selected planned activities; and (c) changing the Performance-based financing (PBF) to Decentralized Facility Financing (DFF) (the latter intervention was also applied to the health facility financing in Ondo and Nasarawa states; all PBF was converted to DFF). The restructuring also included adjustment of the financing percentage, as determined in the annual workplans, based on the remaining balance of IDA and GFF TF financing available. This was driven by the understanding that re-purposing some funds was to enable health facilities to continue to deliver core health and nutrition services better within the COVID-19 pandemic. Finally, financing of health facilities was changed from PBF to DFF for the period January 1, 2020 – October 31, 2020. Allocations for health facilities were now based on four criteria: catchment population (30 percent); average quantity earnings over 2019 (20 percent); average quality score Page 20 of 84 The World Bank Nigeria States Health Investment Project (P120798) (35%); and average reporting rate over Q1-20 (15 percent). Guidelines on what to spend these decentralized funding on (related to recurrent health facility expenses; interventions for community activities; personal protective equipment; general infection prevention and control measures, measures to install triage) were created. In addition, the NPHCDA was tasked to assist States to document health facility historical monthly expenditures to enable follow up on the DFF. Cancelation of funds at project closure 41. At the closure of the project, the remaining funds were canceled and allocated to support the Multi- Sectoral Crisis Recovery Project for North Eastern Nigeria (MCRP). The canceled funds were used to further develop MCRP’s multi-sectoral approach in the BAY states (Borno, Adamawa and Yobe), to allow it to absorb complementary capacity and activities from the five HD operations, which cover health and education service delivery, agriculture and food security, social protection and community development — NSHIP, P120798, State Education Program Investment Project (SEPIP, P122124), Community and Social Development Project (CSDP, P157898), FADAMA (P131075) and Youth Employment and Social Support Operation (YESSO, P157899). Revised Components 42. The components were revised three times during project implementation. (a) 1st Restructuring and AF (March 2014) - Additional HRITF Grant ($1.7m): introduced interventions to address demand-side barriers to accessing health care through a new sub-component (Strengthening Community Demand), targeting poor communities and households. Specific interventions included conditional cash transfers, transport vouchers and performance-based incentives to traditional birth attendants and mission home birth attendants for referral services (Community PBF). (b) 2nd Restructuring and AF (June 2016) - Additional IDA Credit $125m and GFF Grant ($20m): revised components one and two and added a third new component as follows: Component 1 (Strengthening Service Delivery). Total estimated budget US$209.1 Original budget estimate: US$122.4m (IDA US$106.8m and HRITF US$15.6m), 1st AF (US$1.7m (HRITF) and 2nd AF US$85m (IDA: US$71 and GFF: US$14m). Actual expenditure: NGN43.86 billion3. Component 1 was revised to “Strengthening Service Delivery” and had additional interventions targeting the challenges of the NE such as increasing focus on nutrition services, introduction of psychosocial and mental health interventions, free pediatric and obstetric care, strengthened community outreach and provision of larger initial investments to eligible health facilities. These were aimed at reinforcing services under Performance-Based Financing (PBF) component 1.A.1 of the original project. Within this period, the (SOML) project was introduced. The SOML project was a P4R project which included similar DLIs to NSHIP. Because of this reason and the fact that all the disbursement linked indicators (DLIs) for the reporting period had been achieved, the eligible expenditure program (EEP) approach under component 1.B.1 of the original project was accounted for resulting in no further disbursement for DLI results under component 1.B.A following this 2nd restructuring . Component 2 (Technical Assistance) Total estimated budget US$57m. Original budget estimate: US$34m (IDA US$29.6m and HRITF US$4.4m); 2nd AF US$23m (IDA: US$19.5 and GFF: US$2.5m). Actual expenditure: NGN23.12 billion4: The approach to Technical Assistance (TA) in component 2 of the original project was modified to support contracting with non-state actors to provide TA and verification services. This was aimed 3 The project reported actual spent in the system by categories and not by components. During the ICR, we reached out to the PIUs who sent actual spent by components but in NGN and not in US$. We could not directly convert the total amount by a given exchange rate as the total was over a period of time with different exchange rates. 4 See footnote 2 for explanation Page 21 of 84 The World Bank Nigeria States Health Investment Project (P120798) at strengthening governance and accountability by separating the functions of TA, verification and counter- verification which was identified as a weakness of the original project, as well as to develop local capacity for PBF implementation. Component 3 (Partnerships to Strengthen Service Delivery). Total estimated budget US$18 (IDA: US$15.5m and GFF: US$2.5m). Actual spent was NGN176.55 million5: This was a new component that was added in June 2016 as part of the AF and it supported partnerships with non-state actors and ensures flexibility to respond to changing circumstances in the NE. Interventions under this component include partnerships to provide mobile health services in underserved communities where these services are limited or non-existent; re-establishing health services (at least one PHC per ward) through non-state actors (CMVAs) where health facilities are non-functional and when security is deemed acceptable; and strengthening LGA management through trainings, mentoring, as well as by strengthening accountability by implementing performance management systems. (c) 3rd restructuring and AF (August 2019). The third restructuring dropped two subcomponents and introduced another. Sub-component 1.B1 and 1.B2 (DLIs to LGAs and States). This 3rd restructuring also allowed to stop payments against all DLIs results as elaborated under the restructuring section above. Sub-component 2.B2. Implementation of result based approaches: A new sub-component (2.B.2) was introduced and read as follows: in the implementation of performance based financing (PBF) in Ondo State, NPHCDA through a service provider, whose qualification, experience and terms and conditions of employment shall be satisfactory to the Association shall carry out lots quality sampling (LQAS) for assessment or performance of LGAs. (d) The 4th and 5th restructurings did not include any revisions to the components. 5 See footnote 2 for explanation Page 22 of 84 The World Bank Nigeria States Health Investment Project (P120798) Figure 2: NSHIP Theory of Change after restructuring Implication of changes on the Original ToC 43. The TOC derived during the preparation of this ICR (based on the description of the project during appraisal) focused on three components of the project. The changes during the restructurings (all five) broadened the TOC to include demand generation to address barriers to use of health care services. 44. The original and revised PDO indicators with baseline and targets are presented in tables 4 above and 8 below. Targets did not change for most of the indicators as the indicators were either revised or new ones were added. The baseline for most changed reflecting the baseline value of most of the indicators as at the time when the project became effective. Page 23 of 84 The World Bank Nigeria States Health Investment Project (P120798) Table 8: New and revised PDO indicators, baseline and targets6 Indicators When Baseline Target PDO Indicators 1. Proportion of children sick in the last month who New 2nd used a government hospital or clinic (average of 61.70% 72% restructuring Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) 2. Proportion of children sick in the last month who New 2nd used a government hospital or clinic (average of 55.70% 85% restructuring Nasarawa and Ondo) 3. Average Health Facility Score - structural Quality of New 2nd Care(average of Adamawa, Bauchi, Borno, Gombe, 24% 72% restructuring Taraba and Yobe) 4. Average Health Facility Score - Structural Quality of New 2nd 45% 72% Care (average Nasarawa and Ondo) restructuring 5. Proportion of births attended by skilled health Revised 3rd 62.50% 75% personnel (Nasarawa) restructuring 6. Proportion of births attended by skilled health Revised 3rd 55.50% 75% personnel (Ondo state) restructuring 7. Proportion of births attended by skilled personnel - Revised 3rd NE (average of Adamawa, Bauchi, Borno, Gombe, 55.50% 75% restructuring Taraba and Yobe) 8. Percentage of 12-23 months old children vaccinated Revised 3rd 27.40% 49% with Penta 3 (NE) (Percentages) restructuring 9. Percentage of 12-23 months old children vaccinated Revised 3rd 39.40% 65% with Penta 3 in Nasarawa state) restructuring 10. Percentage of 12-23 months old children Revised 3rd 67.30% 75% vaccinated with Penta 3 in Ondo state restructuring 11. Number of 12 – 23 months old children vaccinated Revised 3rd 86,237 450,000 with Pentavalent (Nasarawa state ) restructuring 12. Number of 12 – 23 months old children vaccinated 3rd 106,227 700,000 with Pentavalent (Ondo state ) restructuring 13. Number of 12 – 23 months old children vaccinated Revised 3rd 78,180 1,350,000 with Pentavalent (NE states ) restructuring 14. Number of 0-12 months old children vaccinated New 2nd 206,090 270,000 with Pentavalent (Nasarawa and Ondo states ) restructuring 15. Number of 0- 12 months old children vaccinated New 2nd 327,278 500,000 with Pentavalent 3 in NE states restructuring 16. Number of births attended by skilled health Revised 3rd personnel (NE) (Sum of Adamawa, Bauchi, Borno, 24,520 400,000 restructuring Gombe, Taraba and Yobe) 17. Number of births attended by skilled health Revised 3rd 29,247 150,000 personnel (Ondo state) restructuring 18. Number of births attended by skilled health Revised 3rd 23,193 150,000 personnel (Nasarawa state) restructuring Revised 3rd 19. Average Health Facility Quality of Care Score 41.90% 61% restructuring 6The 3rd restructuring revised the baselines and targets in the results framework to match the start of project implementation which began in 2014 using SMART data for the proportion related indicators. To ensure consistency, the data from SMART (baseline and actual) is used in this report. Four PDO indicators target values (indicators 11, 12, 13 and 22) were revised. For others, the revisions were related to aligning baseline values to the current value when the project fully commenced intervention (2014) and grouping some indicators into Nasarawa and Ondo and North East states. Page 24 of 84 The World Bank Nigeria States Health Investment Project (P120798) Indicators When Baseline Target 20. Number of outpatient visits per year, children and New 2nd adults (sum for Adamawa, Bauchi, Borno, Gombe, 409,786 500,000 restructuring Taraba and Yobe) Revised 3rd 21. Number of outpatient visits by children under five 346,990 13,500,000 restructuring Revised 3rd 22. Direct Project Beneficiaries (number) 0 12,000,000 restructuring Revised 2nd 23. …......of which female (%) 0 60% restructuring Intermediate Indicators Revised 3rd 1. Number of LGA with ongoing PBF (NE) 1 64 restructuring Revised 3rd 2. Number of LGAs with ongoing PBF (Nasarawa state) 1 7 restructuring Revised 3rd 3. Number of LGAs with ongoing PBF (Ondo state) 1 9 restructuring 4. Proportion of health facilities in the project area New 2nd with functioning management committees having 0% 30% restructuring community representation 5. Number of mobile clinics conducted per year in New 2nd 0% 75.00 project area restructuring 6. Number of women receiving counselling on gender- New 2nd based violence or mental health services from a 45 20,000 restructuring trained provider 7. Proportion of pregnant women tested for HIV and New 2nd 39.50% 50% who received test result restructuring 8. Proportions of mothers aged 15 to 19 years of age New 2nd who deliver in the last two years who receive skill birth 20% 35% restructuring attendance. 9. Number of children treated for severe acute or New 2nd 0 25,000 chronic malnutrition per year in project area restructuring 10. People who have received essential health, New nutrition, and population (HNP) services (CRI, Number) 0 0 6,900,000 11. People who have received essential health, New 3rd nutrition and population services (numbers) females 0 3,800,000 restructuring (RMS) Revised 3rd 12. Number of immunized children 0 2,500,000 restructuring 13. Number of women and children who received New 3rd 0 3,700,000 basic nutritional services (Number) restructuring 14. Number of deliveries attended by skilled birth New 3rd 0 700,000 personnel (Number) restructuring 15. Proportion of health facilities having essential New 3rd 0 35% medicines and commodities in stock (percentages) restructuring 16. Proportion of on duty technical staff present at Revised 3rd 0 84% facility on the day of the survey (percentages) restructuring Page 25 of 84 The World Bank Nigeria States Health Investment Project (P120798) II. OUTCOME 45. The changes made to the project increased its scope and hence no split rating evaluation methodology is applied to the assessment of the PDO achievement. A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 46. The relevance of the PDO is rated High. Both the original and revised PDO remained highly relevant during the project and remain relevant today. 47. The original and revised PDOs aligned with the Country Partnership Strategy (CPS) 2014-20177 and the Country Partnership Strategy that was approved in late 2020. The 2014-2017 CPS for Nigeria which was extended to 2019 focused on improving governance and promoting human development. One of the CPS’s goals was to increase access to and utilization of quality health services. The health-related CPS indicators included: a) The share of child deliveries that are assisted by trained health personnel in 3 states (Adamawa, Nasarawa and Ondo) increases to 43 percent by 2017 b) The share of children 12-23 months old who are fully immunized in 3 states (Adamawa, Nasarawa and Ondo) increases to 45.4 percent by 2017 c) The percentage of under-5 children sleeping under insecticide treated net the night preceding the survey increased from 44.6 percent in 2010 to 60 percent. d) At least 80 percent coverage with oral polio vaccine sustained in every state. e) The percentage of pregnant women living with HIV who receive a complete course of antiretroviral prophylaxis to reduce the risk of mother to child transmission increased from 18 percent to 40 percent. The NSHIP’s original and revised PDO indicators contributed to the achievements of two out of the five health related indicators (indicators 1 and 2 above) of the 2014-2017 CPS extended to 2019, an evidence of the relevance of the project to the CPS. 48. The original and revised PDOs also aligned with Government’s NE engagement Strategy and other World Bank initiatives to address the developmental issues of the NE region. As mentioned in the significant changes section above, in 2015, the Nigeria Government in collaboration with the EU, WBG and UN undertook the Recovery and Peace Building Assessment (RPBA). The RPBA informed the design of the North East Emergency Transition and Stabilization Program (NETSP) which initially targeted the short and medium-term recovery, stabilization, and peace building needs. Five HD operations and the Agriculture project, FADAMA III AF, were included in the NETSP and structured as a coordinated umbrella program that included interventions/approaches that can rapidly be scaled-up or re-engineered to benefit the vulnerable populations in the NE. The coordinated approach was aimed at ensuring the greatest synergy and avoidance of duplication. 49. The PDO also aligns with the 2019 Nigeria Systematic Country Diagnostic (SCD) which identifies improving access to quality healthcare services delivery, addressing issues of financing, accountability and the architecture of development assistance for health as options to address the challenges facing the Nigeria 7 Nigeria Country Partnership Strategy 2014-2017. Page 26 of 84 The World Bank Nigeria States Health Investment Project (P120798) health care sector8. At every point along the way from preparation till close of the project, deliberate efforts were made to ensure the project responds to the needs of the Country and of Nigeria WB strategic direction. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 50. The revised PDO is “to increase the delivery and use of high impact maternal and child health interventions and improve quality of care available to the people in Nasarawa and Ondo and all the States in the North East”. For clarity, within the context of the NSHIP, “delivery and use of” high impact maternal and child health interventions has the same intended meaning as “improving utilization of” maternal and child health interventions, and this is consistent with the project’s key associated outcome targets and component descriptions in the legal agreements and the PAD. Therefore, the PDO is assessed against three key expected outcomes: • To increase delivery and use of high impact maternal interventions; • To increase delivery and use of high impact child health interventions; and • To improve quality of care available to people in the Nasarawa and Ondo and all the States in the North East. Rating: Substantial 51. In 2015, a mid-term impact evaluation of the project was conducted using a quasi-experimental design study (Gertler, P., et al. (2016). Impact Evaluation in Practice, 2nd. Washington DC, World Bank). The result shows significant improvement in selected indicators in facilities and communities where the project was implemented compared with facilities and communities where the project was not implemented. Through a rigorous evaluation processes, the evaluation was able to attribute the observed changes to the impact of the project. More specifically, based on the results of the impact evaluation, it is concluded that changes in the project indicators in selected states can be attributed to the project intervention activities. 52. The project has 23 PDO and 16 intermediate indicators to measure the performance of the expected outputs. Of these 39 indicators, seven are related to improving quality of care while the other thirty-two are related to increase in delivery and use of high impact maternal and child health interventions. The main source of data for the reporting the performance of the PDO indicators was from the 2020 National Health facility Survey, the SMART survey, national HMIS and the project PBF portal data. Outcome 1: To increase delivery and use of high impact maternal health interventions - Rated Substantial 53. Thirteen indicators (8 PDO and 5 IRI) were used to report on the performance of this outcome. As can be seen in table 9 below, eleven indicators’ target were surpassed (more than 100 percent achieved), one was partially achieved while one was not achieved. Ondo state surpassed the target for the PDO indicator proportion of births attended by skilled health personnel by almost 10 percentage points while Nasarawa and the NE states did not meet the target. Number of births attended by skilled personnel for the eight states, increased to over 2 million from a baseline of 76,790. At the end of the project, the project HMIS data shows that the project reached 22,390,040 people, or 67 percent of the total population of the targeted states (NPopC 2020 population estimate). The number surpassed the target set for the project. Of the number reached, 63 percent were females (surpassing the target of 60 percent female coverage). HIV testing and receiving results during antenatal care can contribute to keeping mothers healthy as early detection of the 8 Nigeria 2019 Systematic Country Diagnostic draft report. Page 27 of 84 The World Bank Nigeria States Health Investment Project (P120798) virus can lead to early commencement of treatment. The intermediate indicator target for proportion of pregnant women who got tested for HIV and received their results increased from about 40 percent to over 58 percent for the eight states, surpassing the target by about 15 percentage points. The end-line value of this indicator for the NSHIP states is more than the national average of 54.7 percent. The target for the IO level indicator number of births (deliveries) occurring in a health facility in all States was surpassed. The target for the PDO indicator number of direct project beneficiaries and the target for the proportion of female beneficiaries were surpassed. It is important to note that “by total number of direct beneficiaries”, the project describes the number of outpatient visits by project beneficiaries at public primary health facilities9 and not necessarily the number of individuals who benefitted from the project overall. Due to a dearth of data, the achievement of the target for the intermediate indicator (number of women receiving counselling on gender-based violence or mental health services from a trained provider), could not be verified for two reasons: the implementation of psychosocial support was delayed and when it was implemented, the national HMIS system which the project depended on did not capture it. Based on the assessment of the results above and as summarized in Table 9 below, Outcome 1: to increase delivery and use of high impact maternal health interventions is rated Substantial. Table 9: NSHIP outcome 1 performance Per. Data Indicators Baseline Target Achieved achieved Source Comments PDO Indicators Proportion of births attended by SMART Not 62.5 75.0 64.4 15.2% skilled health personnel (Nasarawa) Survey achieved Proportion of births attended by SMART 55.5 75.0 82.7 139.5% skilled health personnel (Ondo state) Survey Surpassed Proportion of births attended by skilled personnel -NE (average of 55.5 75.0 68.6 67.2% Adamawa, Bauchi, Borno, Gombe, SMART Partially Taraba and Yobe) Survey achieved Number of births attended by skilled health personnel (NE) (Sum of 24,520 400,000 1,206,195 314.7% Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) HMIS Surpassed Number of births attended by skilled 29,247 150,000 543,136 425.6% health personnel (Ondo state) HMIS Surpassed Number of births attended by skilled 23,193 150,000 334,171 245.2% health personnel (Nasarawa state) HMIS Surpassed Direct Project Beneficiaries (number) - 12,000,000 22,390,040 186.6% HMIS Surpassed …......of which female (%) - 60.0 63.0 105.0% HMIS Surpassed Intermediate Indicators Number of women receiving 45 20,000 HMIS No data counselling on gender-based violence and so not or mental health services from a used in the trained provider analysis Proportion of pregnant women tested for HIV and who received test result 39.0 50.0 58.7 179.1% SMART (NE) Survey Surpassed 9 NSHIP 2012 PAD Page 28 of 84 The World Bank Nigeria States Health Investment Project (P120798) Per. Data Indicators Baseline Target Achieved achieved Source Comments Proportion of pregnant women tested for HIV and who received test result 37.0 50.0 63.7 205.4% SMART (Ondo and Nasarawa) Survey Surpassed Proportions of mothers aged 15 to 19 years of age who deliver in the last 20.0 35.0 47.2 181.3% two years who receive skill birth SMART attendance. Survey Surpassed People who have received essential health, nutrition and population - 396.4% 3,800,000 15,063,069 services (numbers) females (RMS) HMIS Surpassed Number of women and children who received basic nutritional services - 3,700,000 19,620,631 530.3% (Number) HMIS Surpassed Box 3: Impact of the project on Skill birth attendants Proportion of births attended by a skilled provider was 28.7% for the NE states while the national average was 47.3%, a difference of about 18 percentage points. By 2020, this difference in percentage points have been reduced to just about 9. In other regions and states where the project was not implemented, some states and regions actually recorded a drop from the 2015 baseline data. For example, SE region dropped from 91.4% to 87.5%. While this is not conclusive, it is indicative of the contribution of this project to increasing skill birth attendants in the supported states. Sharing her experiences on how the project contributed in increasing skilled birth attendants, one member of the PIU shared that, providers went out of their way to mobilize pregnant women to deliver in their facilities. They incentivize pregnant women by giving them free delivery kits when they come to the health center to deliver. The expectation of a delivery kit and in some cases, cash to celebrate successful delivery in the center, encouraged pregnant women to come to the facilities. The healthcare providers also targeted traditional birth attendants and a token higher than what they (TBA) usually get if they deliver a pregnant woman, was given to them (TBAs) for every pregnant women referred to the primary healthcare facility for delivery. The healthcare providers used part of their earnings from quality and performance base financing to do this. Outcome 2: To increase delivery and use of high impact child health interventions – Rated Substantial 54. Fourteen indicators (12 PDO and two intermediate result indicators) were used to assess this outcome. Nine exceeded their targets, two were achieved, two were partially achieved while one was not achieved. Of the fourteen indicators, nine were measured using numbers while the other five were measured using percentages as described in Table 10 below. 55. According to data from HMIS the project contributed in massively scaling up access to immunization services for children under five and increase in number of outpatient visits per year. For example, Page 29 of 84 The World Bank Nigeria States Health Investment Project (P120798) outpatient visits were as high as over 30 million over the lifetime of the project. This achievement was about 16.5 million over the target. Although performance of all the number related indicators surpassed their targets, the targets appear a bit unambitious. For example, the target for the PDO indicator number of outpatient visits per year, children and adults (sum of NE states) was just 500,000 from a base of 409,786. The achievement for this indicator was over 20 million. For the IRI indicator “Number of women and children who received basic nutritional services”, the baseline was zero, the target, 3,700,000 and achievement was over 19 million. Of the fourteen indicators, only one PDO indicator was not achieved i.e., proportion of children sick in the last month who used a government hospital or clinic (average of Nasarawa and Ondo). 56. The increase in vaccination of children in the supported States can be linked with the strategic efforts of the project. Motivated in part by incentives, providers worked hard to mobilize community members to access vaccination services. The demand generation efforts, especially the conditional cash transfer, motivated community members to take their children to be vaccinated. Table 10: Outcome 2 performance Percentage Indicators Baseline Target Achieved achieved Comments Data source PDO Indicators Proportion of children sick in the last month who used a government SMART hospital or clinic (average of 61.7 72.0 69.1 71.8% Survey Adamawa, Bauchi, Borno, Gombe, Partially Taraba and Yobe) achieved Proportion of children sick in the last month who used a government SMART 55.7 85.0 55.4 -1.0% hospital or clinic (average of Not Survey Nasarawa and Ondo) achieved Percentage of 12-23 months old SMART children vaccinated with Penta 3 27.4 49.0 68.6 190.7% Survey (NE) )Percentages) Surpassed Percentage of 12-23 months old SMART children vaccinated with Penta 3 in 39.4 65.0 82.9 169.9% Survey Nasarawa state) Surpassed Percentage of 12-23 months old SMART children vaccinated with Penta 3 in 67.3 75.0 86.6 250.6% Survey Ondo state Surpassed Number of 12 – 23 months old children vaccinated with Pentavalent 86,237 428,729 94.2% 450,000 in Nasarawa state ) Achieved HMIS Number of 12 – 23 months old children vaccinated with Pentavalent 106,227 944,999 141.3% 700,000 3 in Ondo state Surpassed HMIS Number of 12 – 23 months old children vaccinated with Pentavalent 78,180 1,350,000 1,339,222 99.2% in NE states (sum of NE states) Achieved HMIS Number of 0-12 months old children vaccinated with Pentavalent 206,090 270,000 1,337,779 1770.8% (Nasarawa and Ondo states ) Surpassed HMIS Number of 0- 12 months old children vaccinated with Pentavalent 3 in NE 327,278 500,000 1,087,090 439.9% states Surpassed HMIS Page 30 of 84 The World Bank Nigeria States Health Investment Project (P120798) Percentage Indicators Baseline Target Achieved achieved Comments Data source Number of outpatient visits per year, children and adults (sum for NE 409,786 500,000 20,936,636 22753.5% States) Surpassed HMIS Number of outpatient visits by Partially 346,990 13,500,000 0,586,618 77.9% children under five achieved HMIS Intermediate Indicators Number of children treated for severe acute or chronic malnutrition 0 304,343 1217.4% 25,000 per year in project area Surpassed HMIS Number of women and children who received basic nutritional services 0 19,620,631 530.3% 3,700,000 (Number) Surpassed HMIS Box 4: Effect of NSHIP project in overcoming phobia associated with vaccine administration In 1996, a pharmaceutical company conducted a clinical trial of its new antibiotic. One hundred children received the new antibiotic while another 100 received the gold standard anti meningitis treatment. Five children given the new antibiotic died. It was later found that the approval for the human trial of the drug in Nigeria was falsified. The Government of Nigeria (GoN) called the trial “an illegal trial of unregistered drug”. There were allegations that participants and their families were not told that they were part of a trial. These led to litigations and eventually, an out of court settlement. This happened almost the same time the NSHIP project was being designed. This and similar cases created a general concern of the safety of vaccines and ensuring vaccine hesitancy. Individuals, communities and even some religious leaders condemn vaccination of children. The RBF approach of the project motivated health care workers to be advocates of vaccination in their communities. Sharing her experiences, one health care worker in Adamawa state said: “I brought women in my community together and explain the benefits of vaccinating. I vaccinated my own children under the age of five to give them confidence that vaccines are safe. Today most children in my family are vaccinated and they call me small Doctor”. The experience of this healthcare worker was one of so many who committed to motivate families to get their children vaccinated. The conditional cash transfers also incentivized families to get their children and wards vaccinated. Outcome 3: To improve quality of care available to people in the selected states – Rated Modest 57. Seven indicators were used to measure the performance of this outcome (three PDO and four IO indicators). As can be seen in table 11 below, one of the indicators (average of health facility quality of care score) exceeded its target; another (average of health facility score – structural quality of care – North East states- Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) was partially achieved, while Average Health Facility Score - Structural Quality of Care (average Nasarawa and Ondo) was not achieved. Three of the IRI indicators (number of LGAs with ongoing PBF, proportion of facilities with functioning management committees and Page 31 of 84 The World Bank Nigeria States Health Investment Project (P120798) Proportion of health facilities having essential medicines and commodities in stock), surpassed their targets. The indicator “Proportion of on duty technical staff present at facility on the day of the survey” was not achieved (partially achieved). 58. Two quality of care metrics in the results framework (Average Health Facility Score - structural Quality of Care (average of Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) and average Health Facility Score - structural Quality of Care (average of Nasarawa and Ondo) are drawn from independent surveys, using idiosyncratic survey instruments. These two indicators were added during the AF, which was part of the 2nd Restructuring. These two PDO indicators for quality are drawn from the annual health facility Quality of Care (QoC) surveys carried out under the SOML. The survey results showed an increase from a baseline of 24 percent in 2016 to 61.6 percent by 30 June 2020 (NE States) an increase of 37.6.4 percentage points (pp) and from 45 percent in 2016 to 60.2 percent by 30 May 2020 in Ondo and Nasarawa States, an increase of 15.2 percentage point. The national average for the QoC survey was 51.7 percent, which was bypassed by both the NE states and Adamawa and Ondo States. The survey metrics were composed of five components which were weighted differently; (i) Content of Care Quality was weighted 30 percent; (ii) Drug availability 20 percent, (iii) Readiness to Deliver Services 15 percent; (iv) Quality of Supervision 20 percent and (v) Financial Management 15%. The Structural Quality of Care were the elements (ii) to (v), and these were unweighted and averaged. These quality metrics are different from the (more comprehensive) quality indices which are used in the strategic purchasing for the essential and basic health care packages and which are independently counter-verified. Data for end 2019, drawn from the data base (publicly accessible at https://front.rbf- ng.org/data#section-info), show across the 8 states an average CPA quality of 68.36 percent and an average MPA quality of 70.26 percent. The CPA quality data vary between 83.23 percent (Gombe) and 48.13 percent (Ondo), and the MPA data vary between 84.08 percent (Taraba) and 62.78 percent (Adamawa). Adamawa state health facilities suffered from long payment delays (24 months from March 2018 to April 2020) due to Bank-internal procedural issues related to the restructuring process and this impacted on the quality of health services in Adamawa state. In Ondo State, the PBF approach was discontinued in 2018 and switched to a payment based on percentage point coverage achievement as measured through Lot Quality Assurance Sampling in the community. Table 11: Outcome 3 performance Percentage Indicators Baseline Target Achieved achieved Comments Data Source PDO Indicators Average Health Facility Score - Annual health structural Quality of Care (average facility survey, 24.0 72.0 61.6 78.3% of Adamawa, Bauchi, Borno, Partially unweighted Gombe, Taraba and Yobe) achieved average of Average Health Facility Score - Structural Structural Quality of Care (average 45.0 72.0 60.2 56.3% Not Quality Nasarawa and Ondo) Achieved elements Average Health Facility Quality of 41.9 61.0 67 131.4% PBF database Care Score Surpassed Intermediate Indicators Number of LGA with ongoing PBF 3 80 113 142.9% Surpassed Project HMIS Proportion of health facilities in the project area with functioning 0 30.0 100 333.3% Project Report management committees having community representation Surpassed Page 32 of 84 The World Bank Nigeria States Health Investment Project (P120798) Proportion of health facilities having essential medicines and 0 35.0 53 151.4% commodities in stock (percentages) Surpassed Proportion of on duty technical staff present at facility on the day 0 84.0 63.6 75.7% Partially of the survey (percentages) achieved PBF, DFF and Business as usual (for clarify business as usual in this context means other health financing mechanisms the Government has been and is using in Nigeria outside PBF and DFF): Results from the Impact Evaluation of the NSHIP project. 59. In 2017, a mid-term evaluation was conducted testing the effectiveness of the NSHIP intervention10. It was a quasi- experimental design that compared PBF and DFF with a control group (business as usual) and an experimental design that compared PBF with DFF. The Impact evaluation randomly allocated 52 LGAs in the experimental states of Adamawa, Nasarawa and Ondo states to either the PBF or DFF arms thus constituting a randomized trial comparing two approaches. A control group was established by selecting states similar to the experimental states along thirteen observable demographic characteristics that are associated with maternal and neonatal health. Before and after household surveys and health facility surveys were undertaken in all the LGA of the experimental and the control states. Baseline data were collected between February and April 2014 while end line data were collected between August and October 2017. The evaluation answered the following questions: a) Whether NSHIP’s interventions improved the availability, utilization, and coverage of maternal, child and reproductive health services particularly among the poor. b) Whether they improved the quality of care. c) The cost effectiveness of the three arms. d) Whether NSHIP financial resources were used appropriately. 60. The results of the impact evaluation demonstrated a remarkable increase in uptake of services in NSHIP supported facilities and the difference was statistically significant when compared to control states . PBF facilities did better in institutional deliveries (increased coverage by 9.1 percent (p<0.05)) , modern family planning (increased coverage by 5.7 percent (p,0.05)) and curative consultations for children (not statistically significant) while DFF facilities did better in childhood vaccinations (increased coverage for DPT by 13.4 percent (p<0.001) and BCG by 9.7 percent (p<0.05)). In terms of equity, it was observed that the largest improvements seen under NSHIP in skilled birth attendance and fully immunized children were observed in the 3rd, 4th and 5th income quintiles in the study sample which was more rural and poorer than the national average. Overall, the quality of care indicators increased significantly more in the NSHIP arms than in the control arm. The increase in quality of care was statistically significant for 20 out of the 26 quality of care indicators. 61. There was an increase in contraceptive prevalence rate in NSHIP supported states especially in the PBF arm) compared with the control sites. The difference was statistically significant at 10 percent level of significance for NSHIP states and at 5 percent level for PBF supported facilities. Family planning intervention has been shown as a remarkably high impact intervention in reducing maternal and under five mortality. Also, institutional deliveries were statistically significantly increased by the NSHIP as compared to control, and more so for PBF than for DFF. 10 Federal Ministry of Health, Impact evaluation of Nigeria State Health Investment Project. Page 33 of 84 The World Bank Nigeria States Health Investment Project (P120798) Justification of Overall Efficacy Rating In arriving at the overall efficacy rating of substantial, the indicators that surpassed their targets were weighted (1.25%). Each outcome and the overall efficacy rating are presented in table 12 below. Table 12: Overall efficacy rating Outcomes Outcome 1 Indicators Outcome 2 Indicators Outcome 3 Indicators Level of Indicators PDO IO PDO IO PDO IO Surpassed (>100%) 6 5 7 2 1 3 Weighted surpassed (*1.25) 7.5 6.25 8.75 2.5 1.25 3.75 Achieved (80 - 100%) 0 0 2 0 0 0 Partially achieved (65 - 79%) 1 0 2 0 1 1 Not achieved (<65%) 1 0 1 0 1 0 Total (unweighted) 8 5 12 2 3 4 Weighted Total 9.5 6.25 13.75 2.5 3.25 4.75 % surpassed or achieved (Weighted) 78.9% 100.0% 78.2% 100.0% 38.5% 78.9% Ratings Substantial Substantial Modest Overall Efficacy rating Substantial 62. Based on the evidence for efficacy presented above, the first outcome rating is substantial, the second is substantial while the third outcome rating in modest. Therefore, on balance, the overall rating for Efficacy is Substantial. C. EFFICIENCY Assessment of Efficiency and Rating Rating: Modest. Economic analysis: 63. The project supported health service packages consisting of very cost-effective health interventions. The cost-effectiveness analysis based on the impact evaluation data showed that the quality-adjusted incremental cost effectiveness ratio (quality-adjusted ICER) of PBF compared to control is $224 per quality adjusted QALY gained, and of the overall NSHIP project versus control is $271 spent per quality adjusted QALY gained. Whereas PBF is nearly twice as expensive as DFF ($3 per capita per year versus $1.70 per capita per year), it saves many more years of life as compared to DFF during the same period, about twice as much. This phenomenon is due to substantially more deliveries and family planning under the PBF arm as compared with DFF. Both PBF and DFF’s cost-effectiveness are well within Nigeria GDP per year, estimated at $2450 nominal USD for 2019 and therefore a good buy for policy makers. Adjusting for quality (using a Delphi methodology) improves the ICERs for each scenario significantly; with the quality-adjusted ICER of PBF 12.2 percent of Nigeria’s per capita GDP and the quality-adjusted ICER of DFF 9.3 percent of Nigeria’s per capita GDP. With all ICER values way below Nigeria’s GDP per capita, PBF and DFF, and NSHIP overall, are highly Page 34 of 84 The World Bank Nigeria States Health Investment Project (P120798) cost-effective as compared to control11. These results hold both with and without incorporating quality improvements and suggest that DFF and PBF are among the cost-effective interventions for strengthening MCH services in Nigeria. 64. The purpose of the following analysis is to assess whether the dollar benefit of the RBF project implemented in Nigeria between 2012 and 2020 outweigh its dollar costs. To do so, this analysis monetizes the major benefits and costs associated with the project, and reports on three measures: the benefit to cost ratio, the net present value and the internal rate of return. 65. The Benefit-Cost Ratio (BCR) is 34.92 for the verified data. This suggests that every dollar invested in NSHIP in Nigeria yields an economic return of 34.92 dollars. The investment in NSHIP of US$253.4 million generated economic benefits with a net present value of US$7.75 billion. The internal rate of return was 18 percent. If the result is adjusted for improvement in quality, the BCR, NPV and IRR would have been higher. Overall, the project is economically beneficial. On the basis of the above, economic efficiency rating of the project is rated substantial. Implementation Efficiency 66. Implementation of the project experienced some setbacks, one of which was delays in implementation of key activities: Cumulatively, the project lost over 30 months as a result of delay in implementation. The first of these delays was in 2012 after the project was approved by the Board. It took about a year before the project became effective. The delay in effectiveness was because the credit was not in the Federal government borrowing plan which must be approved by the National Assembly. The second delay was when the AF was approved. It took almost 18 months for some of the states to sign the subsidiary loan agreement. Ondo state lost about a year when funding to the state was temporarily put on hold because of improper use of the project funds. Activities only resumed after the state had refunded the misused funds. Adamawa state health facilities suffered from long payment delays (24 months from March 2018 to April 2020) due to Bank-internal procedural issues related to the restructuring process and this impacted on the quality of health services in Adamawa state. The implementation delays both after approval of the original project and after AF are a Nigeria wide issue. Effectiveness delays are a persistent challenge. Yet despite these effectiveness delays , the project was able to fully implement the project. 67. When the project was waiting for the National Assembly to approve the borrowing plan, the project started the pilot of the NSHIP RBF approaches in selected LGAs in the original NSHIP states. Funds for the pilot were provided by the states. By the time the borrowing plan was approved, and the project declared effective, lessons were learnt in the pilot LGAs which were used to improve the implementation of the project. 68. Effective institutional arrangements: The project’s institutional arrangements ensured that the bureaucracy of government did not slow down the project approval processes at the state and national levels. Once the PIUs received a No Objection from the Bank, the NSHIP PIU project Manager could approve as they were signatories to the Designated Account. This also ensured that the project funds were not used for other activities of the state. 69. Partial Project Budget Cancelation: During the implementation, the project partially cancelled funding (US$25m of the project funds was canceled in 2020). The cancellation was due to the team’s assessment that the project would not to be able to implement the remaining project activities before the original closing date of June 30, 2020. This was due to the delay in implementation of certain project activities for the following reasons: (i) a one-year delay in effectiveness of the Additional Financing; (ii) limited ability to 11 Mid-term evaluation report. Page 35 of 84 The World Bank Nigeria States Health Investment Project (P120798) implement at scale in Borno and Yobe states due to insecurity; and (iii) slowing down of project activities due to the impact of COVID-19 pandemic including lockdowns, restrictions on movement, etc. 70. Part of the cancelled funds was used for the financing of the MCRP AF, a program designed to respond to the crisis in the NE States of Borno, Adamawa and Yobe (BAY) as well as to consolidate the World Bank- financed operations in the NE (closing IDA North-East AF operations in the BAY states), by strategically absorbing objectives, funding, and capacities that are well aligned with MCRP’s core objectives, the wider recovery goals for the NE and regional stabilization goals for the Lake Chad region. Overall, the participation of Nigeria, the fourth country in the region, is crucial for the Lake Chad Region Recovery and Development Project (PROLAC) program to have complete regional coverage. Additionally, Nigeria is by far the largest country of the four and the epicenter of the Boko Haram conflict, the impacts and drivers of which PROLAC seeks to address. The cancelled IDA from the NSHIP project, provided an opportunity to include activities to strengthen service provision for health and education, absorbing activities underway through the Nigeria States Health Investment Project (NSHIP -P120798) and State Education Program Investment Project (SEPIP – P122124), while taking stock of the projects’ strengths as well as the challenges experienced during implementation and identifying areas for improvement / synergy with ongoing MCRP activities. Expansion of health-related service delivery activities (absorption of some NSHIP activities) include: • Decentralized facility financing of health centers and district hospitals, focused on quality of health services; • Internal performance contracting of LGA-PHC departments for key essential public health functions; • Engaging in internal contracting of State Primary Health Care Development agencies and Hospital Management Boards, to strengthen state level public health governance; • Institutionalizing follow up of internal LGA contracts, investment units (state primary health care development agencies, SPHCDA) and quality of hospital services (Hospital Management Board, HMB); and • Decentralized facility financing using investment grants through business planning processes for rapid structural improvements 71. Cost overruns. While the overall project did not experience cost overrun, some States did experience cost overrun. The project had to reallocate funds from States that were spending less to States that were overspending. The five restructurings were used as opportunities to reposition the project to address the country’s emerging needs. Considering the delays experienced before the project was effective (both original and additional financing), cancellation of the project budget and cost overrun experienced by some states, the implementation efficiency of the project is rated as Modest. 72. With economic efficiency rating of Substantial and implementation efficiency rating of Modest, the project Efficiency is rated as Modest. D. JUSTIFICATION OF OVERALL OUTCOME RATING 73. As per Bank Guidance on ICR12 (Appendix H), the overall outcome rating is Moderately Satisfactory, as one objective in the assessment of Efficacy is rated as modest. (Table 13). Table 13: Overall project rating (relevance, efficacy and efficiency) Sub-Ratings Overall Project Rating Relevance Efficacy Efficiency High Substantial Modest Moderately Satisfactory 12 Bank Guideline March 2020. Page 36 of 84 The World Bank Nigeria States Health Investment Project (P120798) E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 74. The project deliberately targeted improving health outcomes among women and children. It also had an indicator reporting the proportion of project direct beneficiaries who are females. The target for this indicator was 60 percent and was surpassed (63 percent). At the institutional level, the project made attempts to encourage females taking leadership positions. Of the two PIUs at the national level, one was at one time headed by a female while at the state level Borno and Ondo PIUs were at one time headed by a female. In all the states, females were key members of the state PIU. On the average, about 40 percent of PIUs membership across the seven states were females. This is very commendable viewed against the background that six out of the eight states are in the North East, where the top leadership of the public service is male dominated. More female PIU members in leadership roles gives them the opportunity to attend key project decision making meetings. At facility level, over 80 percent of HCW training beneficiaries were females. This is expected as most health care providers at PHC (Nurses, CHEWs) are females. Most of these facilities are also headed by females. 75. The project supported the constitution of facility management committees with at least three memberships drawn from members of the Ward Development Committee (WDC) formed from community(ies) - PHC serves. The formation of the WDC was well detailed in the Participatory, Learning and Action (PLA) manual of the National Primary Health Care Development Agency (NPHCDA). The project recruited community-based organizations (CBOs)/civil society organizations (CSOs) to carryout community client satisfaction surveys quarterly. Women representation was one of the conditions for a CBO to be selected for this task. The project ensured that a woman leader in the community is included as a member of the facility management committee. These women would have benefitted from some form of trainings that would have equipped them to make their voices better heard. A review of the training materials at both state and facility level shows that a module on gender was not included. This is a missed opportunity as that would have equipped those at the state level with skills to design and formulate gender responsive health related policies and at the facility level, this would have equipped the provider with knowledge and skills to provide a more gender sensitive services. 76. Interviews with providers and PIUs show that most deliberate efforts were made to target women than men, and activities were thus largely focused on women. It is important to note that the 2018 NDHS reported that women had to request permission from their husband before accessing healthcare services. This is still one of the major barriers for women to access health care services. Institutional Strengthening 77. Institutional strengthening is a key component of the project. Sub-component 1.B focused on strengthening the institutional capacity of state and local governments. Sub-components 1.A and 1.B strengthened the capacity of: PHCs to effectively deliver services through direct fund transfers to facilities; health care providers; and facility management committees and ward development committees. The project contributed significantly to improving Nigeria’s healthcare delivery capacity on various levels. 78. National. The DFF approach piloted and implemented under the NSHIP project has been adopted by the Government as a national strategy for the provision of essential health care service at the PHC level. The Government is doing this through the Basic Health Care Provision Fund. This is revolutionary for the Nigerian Health System as one of the main challenges PHC has been facing is limited operational funds, which in turn weakens its ability to provide quality health care services. As demonstrated under the NSHIP project, Page 37 of 84 The World Bank Nigeria States Health Investment Project (P120798) Box 5: Transformational effect of the NSHIP at the National Level and overall Nigeria Health care System. ● National Health Act 2014: The Basic Health Care Provision Fund (BHCPF) component of the National Health Act adopted the Decentralized Facility Financing (DFF) component from NSHIP (NHA, 2014). ● Oversight/Supervisory Tools: Similarly, the tremendous improvement in the quality of care recorded at PHCs through the use of the NSHIP Quality Supervisory Checklist (QSC), led to the adoption of the document during the revision of the Integrated Supportive Supervision (ISS) checklist in Nigeria. Presently, the country’s ISS checklist is heavily influenced by the content of NSHIP QSQC in the country’s effort to improve the quality of care across all PHCs in the country. ● Influence on National Documents: ● Essential Medicine Manual: In the same light, the success of NSHIP as regards to the management of essential medicines, led to the development of the Essential Medicine Manual, which has been adopted across all PHCs in Nigeria. ● Financial Management Manual: The financial management at the health facilities is a key success factor for NSHIP, with key community involvement in the form of Ward Development Committees (WDCs). Consequently, a similar approach to financial management was adopted for the BHCPF implementation. ● Health Care Waste Management (HCWM) Manual: The effectiveness of health care waste management within NSHIP - which is in line with global waste practice - also made the adoption of the HCWM manual by all PHCs across the country possible. direct fund transfer to facilities with the autonomy to use the resources to meet operational needs of the facility can significantly improve provision of quality health care services. The National PHC Supervisory Document (ISS) adopted NSHIP quality checklist. The National ISS document was developed for the secondary and tertiary health facilities following the NSHIP quality checklist. Development of business plan has been adopted for health facilities (HFs) operational activities. 79. Development of National Health Strategic Development Plan. The project supported and drove the development of the first national strategic health development plan. Today the national strategic health development plan has been institutionalized, providing strategic direction for the Nigeria healthcare sector. 80. State and Local Government levels. In 2011, the Primary Health Care Under One Roof (PHCUOR) policy of the Government was introduced. The Primary Health Care Under One Roof (PHCUOR) is a policy to reduce fragmentation in the delivery of Primary Health Care (PHC) services. It involves the integration of all PHC services under one authority (State Primary Health Care Board at state level). Fragmentation was identified as the most significant problem facing PHC services, and significantly affects utilization rates and health indices. While the policy was approved in 2011, it was very challenging to implement. When NSHIP was approved, the PIU at state level was domiciled in the state primary health care board. This propelled the states to ensure that their state primary health boards were in place. The NSHIP project committed resources to strengthen these agencies as described in sub-component 1.B. When the project started, only one state had a primary health care board. At the end of the project, all the states had primary health care boards. The result of the National PHCUOR scorecard four (a matrix used to measure the performance of states with respect to meeting PHCUOR standards) shows that seven out of the eight NSHIP states were among the 18 states that have a score of 55 percent and above with Gombe one of the NSHIP state having the highest score of 76%. Borno state is the only NSHIP state with a score less than 55 percent and this is largely due to the impact of insecurity in the state. The PHCUOR score card assesses nine indicators: governance and ownership, legislation, minimum package of services, repositioning, system development, human resources, funding sources and structure, operational guidelines and office setup. Page 38 of 84 The World Bank Nigeria States Health Investment Project (P120798) 81. Facility Level (Autonomy of PHCs). The greatest evidence of the impact of NSHIP institutional strengthening was at the facility level. The direct fund transfer to facilities either through the PBF or through the DFF approaches were used to renovate the facilities, hired more human resources for health at that level, procure health commodities and consumables, powered the facilities and meet other operational needs of the facilities. Healthcare workers also benefitted from a series of capacity building efforts and performance- based incentives. All these resulted in increased motivation for the staff, availability of health commodities and consumables at the facilities and better working environment. These led to increased productivity evidenced by increased number of clients the facilities were serving and improvement in quality of services being provided. In addition, the managerial capability of the health facility Officer-In-Charges (OICs) have been improved significantly as quite a number of health facilities engaged in income generating activities that will sustain the facility operational activities in the post NSHIP implementation. 82. Another indication of how the project strengthened PHCs is in the area of increasing the number of PHCs with essential medicines and commodities. The target was 35 percent from a baseline of 0 percent. The target had been exceeded (53 percent) by the end of the project. Similar improvement was recorded in the area of technical staff present at the health facility. The baseline value was 0 percent with a target of 84 percent. Though the target was not achieved largely because of the insurgency in the North East, it grew to 72 percent at the end of the project. Mobilizing Private Sector Financing 83. While the NSHIP project was not set out primarily to mobilize private sector finances, it did contribute to improving service provision capacity of the private sector, thus increasing profitability. The project did this in two ways: 84. Decentralization of pharmaceutical supply to PHCs. PHCs source for pharmaceutical supply from both select private and public sources that were pre-qualified as a result of the autonomy granted to them through DFF. To qualify to supply these commodities, minimum standards, including quality of the products but also pricing, were set that the suppliers were required to meet. There was a guaranteed supply once those criteria are met. This made private supplies to investment in their business to meet the set criteria and created a healthy competition between suppliers (government and public) which resulted not just in improved supply of pharmaceuticals down to the last mile but also of standard medical supplies. This broke down the monopoly of government owned enterprises with its associated inefficiencies and brought more revenue to the private facilities that were selected to supply health commodities and consumables to these PHCs. The project assisted with crafting guidelines for States related to private pharmaceutical outlets. At the onset, each state selected 2-3 certified private distributors (known for their reliable quality of products). Contracted health facilities were obliged to source their products from these certified distributors. In some states, most extensively in Ondo State, handheld spectrophotometers (TruScan)13 were procured to assist with regulating the quality not only at these certified distributors, but also at the contracted health facilities. Within the project implementation period, commodities worth over $2m were sourced from private suppliers. 85. In some cases, private health facilities were selected as PBF/DFF facilities in the absence of government owned facilities that meet minimum criteria for inclusion. In urban settings where the population density was high, private health facilities were also contracted to complement the efforts of public health providers through the urban PBF approach. These private facilities benefitted from the series of capacity building efforts of the project that targeted health care providers. They were also recipients of direct funding from the project to improve service delivery. At the end of the project, 77 private health facilities and 3 faith- 13 https://www.thermofisher.com/order/catalog/product/TRUSCANRM#/TRUSCANRM Page 39 of 84 The World Bank Nigeria States Health Investment Project (P120798) based health providers were engaged. While the financial records of private facilities were not reviewed to see if this resulted to increased earnings for the facilities, it is expected that the increased service delivery contributed to increased earnings in these facilities. Poverty Reduction and Shared Prosperity 86. Nigeria’s out of pocket expenditure for health of over 70 percent14 is one of the highest in the world. When poor households pay more for health, it makes them poorer and increases inequality between them and the rich. The design of the project ensured that the burden of payment for healthcare services was largely borne by the public purse, thus removing the financial barrier most poor households do face. By design, the project offered fee exemptions for the poorest 10 percent of the population for minimum package of activities (MPA) services – OPD consultation for the indigent and continuum of care. In addition, the Northeast States subsidy fee was set in such a way that MPA services were provided free of charge, which is also in line with the Government policy. More so, it also targeted the poorest of the poor for the demand- side financing interventions (CCT and transport voucher programs) where these were implemented. It is however challenging to tell if proper segmentation was done to ensure that these targeted efforts reached only those they were meant to serve. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 87. Realistic objectives. The NSHIP project development objectives were clear, realistic and had the right level of ambition. The objectives were also formulated based on a substantial amount of analytical work, which clearly identified the bottlenecks in the health systems and the measure needed to affect the necessary changes. Prior to the NSHIP, the Bank supported a health system strengthening project - HSDP, which focused primarily on supporting health systems nationwide to position the Government to provide the needed leadership for improved healthcare service delivery. The focus of the NSHIP was thus on actual delivery of health services at the PHC level with a component focusing on health system strengthening. The project introduced a novel approach to service delivery which pays for services delivered versus against inputs which hitherto was the practice. The introduction of incentives tied to performances was aimed at motivating health care providers to ensure that quality services are provided. The Federal and state governments also appeared ready and committed to the achievement of the project objectives. 88. During the design of the project, there were some implicit assumptions: (i) that states had some systems strong enough to implement a performance based intervention; (ii) that there are strong fiduciary processes in place to check misuses; and (iii) that government verbal and written commitments will automatically translate to real commitments when implementation commences. This turned out not to be the case. For example, the original project lost one year before it became effective due to regulations requiring the National Assembly to approve borrowing plans prior to effectiveness of the NSHIP. The NE receiving AF also lost close to 18 months in 2017 due to delays in the signing of some state subsidiary loan agreements (SLAs) with the FMF. Unlike the AF, the signing of the SLA in the original states was done in a better coordinated manner as the legislators and the governors were properly engaged while waiting for the National Assembly to approve the borrowing plan. Additionally, due to lack of states’ fiduciary capacity, considerable efforts were committed to strengthening fiduciary processes across the states. Health is on concurrent list as such decisions at national level do not automatically translate to acceptance at sub national level. 14 FMoH: National Health Account 2017. Page 40 of 84 The World Bank Nigeria States Health Investment Project (P120798) 89. Complex design. Although the project logic appears quite clear, the design was complex, and the approaches were very new to the Nigeria context. The focus was on PHC which was the first of its kind in the country. An evaluation was built into the design comparing PBF and DFF and both with a business as usual scenario. The main objective of the evaluation was to identify if both approaches were effective in achieving the project objectives (measured using some selected indicators) compared with a business as usual scenario and if one was more effective than the other. The lessons learned from the M&E checks and balances, informed the adjusted M&E separation of functions of the parent states (the introduction of the CMVAs and the IVAs). 90. Appropriate selection of stakeholders to engage or beneficiary groups to target. The preparation of the project was informed by a very extensive consultation with relevant stakeholders. These included the federal ministry of health and its agencies such as the National Primary Health Care Development Agency; State Ministries of Health, the private sector, Executive and legislative arms of government at the state level, Civil Society Organizations and Implementing Partners in Nigeria. These series of engagements of key stakeholders were important for the delivery of the project when it eventually became effective. A good example is the passage of relevant laws by state legislative arm to set up state primary health care boards as required for the implementation of the project. 91. Institutional Arrangements. The NSHIP project set up a very strong and efficient institutional arrangement from the national to the facility levels. The governance and coordination mechanisms were embedded within the Ministries of health and the primary health care agencies at federal and state level, as well as at the Local government health authority at the local government level. These institutions were responsible for setting standards and ensuring that minimum standards are adhered to by health providers. Project finances and payment functions was managed by the project finance management units to assure separation of function and promote good governance. For accountability and to minimize project financial risks, verification, counter verification, audits and community satisfaction surveys were assigned to independent agencies, recruited from the private sector. B. KEY FACTORS DURING IMPLEMENTATION (a) Factors subject to the control of government and/or implementing entities 92. Coordination and engagement: The implementation arrangement of the project ensured that the project implementation unit both at the state and national levels have the authority to effectively dispatch their duties. The PIU leadership did not need extra approvals from their supervisors within government to implement the project approved workplan and budget, thus freed them from the interferences of the implementing Agency and other government officers which could have slowed down processes or possibly lead to inappropriate use of project funds. Also, under the parent project, a contracted firm, Oxford Policy Management (OPM), was providing the TA to national and states. In the AF, this was replaced by local TA, that had been trained during the project. Over a ten-year period, 257 Nigerians, both technicians and politicians were trained in high quality two-week certificate courses 2010 to2019 in Enugu, Yola, Nairobi and Mombasa for instance. 93. There were periodic meetings between states and the National PIUs. The meetings were used to build capacity, ensure alignment with overall project goal and objectives, evaluate project performance and propose course corrective strategies to implementation. At the beginning of the year, the states and national PIUs come together to develop their annual workplan. These joint planning sessions ensured that the quality of the plans developed were of a good standard, realistic and align with the project objectives and resources Page 41 of 84 The World Bank Nigeria States Health Investment Project (P120798) available. The periodic meetings also provided opportunity for cross learning among states especially for the additional states who joined the project in 2017/2018 when the AF became effective. 94. The project rewarded different layers of health system based on their performances. Leaving the evaluation of these performances in the hands of the implementers in some cases resulted to manipulation of the data to favor those who will benefit from the rewards associated with the performances. To mitigate this, the project used independent verifying agents (third party agents) to verify data reported by the facilities and the PIUs. The ToRs of the CMVAs and IVAs were well spelt out and there were minimal interferences. Despite this arrangement, suspected gaming was reported, and it was addressed by the project as a result of the checks and balances in the project which ensured that gaming was intercepted by national (NPHCDA) and the Bank team. Penalties were applied as appropriate and as described in the project implementation manuals. 95. While stakeholders were actively engaged during the preparation of the project, this did not continue during the implementation. Key government officers outside the project knew little about the project approach. The results and impact of the project were largely not known to critical stakeholders and key decision makers especially at the state level. This affected the project’s effort to sustain the scaleup of the project design using state funds. Almost at the end of the project, some attempts were made notably in Gombe state to get the state government to appreciate result-based financing for health. This resulted in the state government committing funds to scale up the intervention to additional LGAs. Gombe state became the first State where the Governor committed funding to extend PBF from half the state to statewide. 96. The project adopted the hub and spoke arrangement with primary health care providing primary healthcare services and secondary providers as referral centers. The PHCs are managed by the SBHCB while secondary facilities are managed by the state Hospital Management Boards (HMB). At the state level, the project focused on SPHCDAs as implementing agencies, being a project focused in PHC. This led to Institutional conflicts with the MoH and HMBs who were not directly engaged at the outset. Hospitals did not have autonomy over the resources, medicines, and staff, coupled with non-engagement of the HMBs led to many problems. This affected referral services as without a mechanism to engage the HMBs, project implementation in the secondary health facilities became a challenge. This challenge was overcome by engaging the HMBs through a performance contract that finances supervision activities and quality reviews of hospitals, leading to further problems of hierarchies between the two institutions. 97. Commitment and leadership: At the implementing agency’s level, there was commitment of the agency’s leadership for effective implementation of the project. This was demonstrated by ensuring that the right personnel were recruited and or seconded as PIU members. The state government of the original three states of the project, committed state funds for the pre-pilot of the NSHIP design before the actual scaleup when the project became effective. The pre-pilot was used as a proof of concept for the approach before actual effectiveness and ensured that adequate capacity was built to support the scale-up. This was also an opportunity for learning, for instance training needs for the local health administrators were identified as these were originally focused on the health care workers. The gaps in engagement with the hospitals, i.e. issues with the unclear roles of the HMBs and State Ministries of Health were also identified during this period. Funding for the pilot was from the states demonstrating their commitment to the project. 98. Human resources and organizational capacity: Performance Based Financing concept and practice was relatively new in the Nigerian Health sector when the project was designed. TAs from Rwanda were recruited to jump start the project and build in-country capacity through sharing of their experiences and on the job training. The delayed recruitment of the RBF TA agency (OPM) responsible for carrying out independent verification of performance as well as building organizational capacity for RBF meant the States had to take Page 42 of 84 The World Bank Nigeria States Health Investment Project (P120798) on these roles with support from the Rwandan consultants. This caused significant delays in the timeliness and quality of the role out in the 3 original states. Similar delays were encountered with the recruitment of CMVAs and IVAs, responsible for putting in place independent verification systems during the additional financing, leading to further delays in the roll out of the AF. 99. Capacity building of health care workers was sustained throughout the implementation of the project. This contributed in increasing delivery of quality services. These capacity building efforts largely focused on public health care providers. Some attempts were made to include the private sector as much as possible but there was pushback internally (within the Bank) and externally (the client). Considering that private sector provides more than 50 percent health coverage, more engagement of the private sector would have been a boost to improving quality of health care services in Nigeria. 100. Fiduciary: The project had in place adequate procurement, financing, budgeting, and financial management mechanisms. This ensured that project funds were efficiently utilized. 101. Environmental and social: As a Category B project, the environmental impacts associated with the project were minor, site specific, non-cumulative and relatively easy to mitigate. These included health care wastes (including sharps) and the risks associated with handling them. The main factors that affected implementation (from the perspective of environmental safeguards) was poor system of and inadequate capacity on Health Care Waste Management (HCWM). However, these were largely addressed as part of project implementation. (b) Factors subject to World Bank control 102. Adequacy of supervision: The Bank provided the government with close technical support over the entire implementation period. The Bank team provided hands on implementation and technical support to the PIUs. Formal implementation support visits involved experts from each sector, fiduciary team members and environmental safeguards specialists. All implementation support visits systematically included a thorough review of the implementation status, including joint sessions to update the results framework. Finally, technical assistants were provided to states and at the national level during the entire implementation period, ensuring just-in-time dialogue and regular follow-up with the Government, as well as monthly field visits. 103. The quality of reporting, which was based on the Government’s progress reports and intermediate financial reports, was good. The aide-memoires (AMs) systematically captured implementation progress and provided action plans to address issues related to components, results framework, fiduciary and environment and social arrangements. The AMs also included updates on the issues discussed in the previous aide-memoire. The ISR reports candidly summarized the findings in the AMs, which were mostly submitted in a timely. 104. The relationship between the Bank and the states and NPHCDA was very cordial. Each party could reach the other with ease. This was particularly helpful as it facilitated open conversation and improve response time to request. 105. There was continuity in the composition of the project team members. Three TTLs / co TTL led the team throughout the course of the project. The first was from the preparation of the project till 2014. Between 2014 and 2019, a second TTL led the project and oversaw the expansion of the coverage of the project. Between 2019 and 2020 when the project closed, a third TTL / Co-TTL task team lead the project. The changes in the project leadership had minimal impact on the project as the new TTLs that took over the Page 43 of 84 The World Bank Nigeria States Health Investment Project (P120798) leadership of the project were members of the project task team from the earliest conception. The safeguards and fiduciary teams experienced minimal changes in the team composition. (c) Factors outside the control of the Government and/or implementing entities 106. Macroeconomic environment: During the life of this project, Nigeria experienced two economic recessions: one in 2015 and the other in 2020; the latter due to COVID 19 which affected economic activities globally. The impact of COVID-19 was and continue to be significant in Nigeria as the price of crude oil, the major source of revenue for the federal Government dropped significantly. This affected the ability of state government to commit their own government raised revenues to scaling up the NSHIP model. 107. Conflict and instability: Insurgency, communal clashes armed banditry, etc. significantly affected the project delivery especially in the NE region. As a result of insurgency, several primary health facilities were destroyed by the insurgents and those that were not destroyed had to stop provision of services because of insurgency and insecurity. Supervision of providers also became a huge challenge. Despite these challenges, the NSHIP team at the state level continued to provide services as much as possible and remained resilient. 108. COVID-19: The pandemic posed challenges to primary health systems and essential service utilization in Nigeria. Social distancing measures, fear and distrust in the system, and reduced household income due to the pandemic may have contributed in limiting utilization of basic health services towards the tail end of the project. On the supply side, disruptions in supply chain, mobility restrictions and lack of infection prevention and control (IPC) training and commodities may have impacted health worker health and motivation, and provision of these essential services. As seen in the table 14 below, between March and July 2020, there was significant drop in uptake of essential services in Nigeria. While this drop may not have direct impacts on the overall result of the project, it might have instilled in decision makers the importance of a strong PHC foundation and a good pandemic response system to be more resilient to similar shocks. 109. However, the health facilities under the project were the first public health facilities to respond to COVID-19 control and prevention. As at when efforts were being made to control community transmission, project facilities reviewed their business plan in which personal protective equipment (PPE), temperature scanner, triage principles were proposed for procurement and obtained approval. To remove the bureaucracy, the National team wrote to the Bank for approval of business plan review to carter for COVID- 19 activities. Following the guidance from the Bank, business plans were reviewed and approved using emergency mode. In addition, hand hygiene was highly promoted through which non-contact pedal handwashing machines were given to all the health facilities under the project. Table 14: Estimated impact of COVID-19 on essential services in Nigeria15 Number of FP clients First Total Third dose of BCG Consultations counselled Antenatal deliveries Pentavalent vaccination (Outpatient Visit (#) vaccine (#) #) attendance) March 2020 -3.4 -1.2 -3.1 -0.6 -3.7*** 2.4** 15 Source: Analysis from GFF and DEC in collaboration with FMoH and Nigeria HNP team, using HMIS data. This analysis is a part of the multicountry-support to GFF countries. Notes: Analysis incorporates facility type or size, difference by first subnational unit and seasonality —output from a regression analysis that adjusts for the factors above. Negative values indicate an estimated increase in service delivery levels compared to the expected level. * p-value<0.100, ** p-value< 0.050; ***p-value<0.001 1Drops in July may be affected by under reporting Page 44 of 84 The World Bank Nigeria States Health Investment Project (P120798) April 2020 -17.5*** -10.7*** -15.5*** -0.9 -12.3*** -5.7*** May 2020 -19.8*** -14.8*** -14.6*** -6.2*** -11.7*** -6.0*** June 2020 -14.7*** 0.2 15.7*** -5.8** -1.4 4.7*** July 20201 -21.3*** -9.6*** -13.9*** -6.5** -7.4*** -5.3*** IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design: Rating: Modest 110. When the project was prepared in 2011, there was no theory of change as ToC was not a requirement then. But the logic of the project was presented in the result framework. Also, the design and implementation, including theory of change were exposed in the PBF toolkit, which took the NSHIP design as one of its worked examples throughout the toolkit16. The project result framework was clear and sound. The objectives were clearly specified, and all expected outcomes were reflected in results framework indicators, which also adequately captured the operation’s activities and outputs contributing to the achievement of outcomes. The project (both original and the additional financing) has a detailed M&E plan that clearly defines each indicator, sources of data, frequency of reporting, method of data collection and person responsible. At the close of the project, there were 21 project development objectives indicators and 16 intermediate result indicators. The project developed a project specific HMIS (PBF portal) to track and report routine data. This was later harmonized with the national DHIS system. The project included a mid-term evaluation with the objective of testing the effectiveness and efficiency of the project design. However, some of the project AF indicators were not aligned with the project HMIS early enough. Though this was aligned later, it led to having different indicators across the years. It took the project approximately two years to collect the relevant data to complete the baseline section of the result framework. One of the indicators has a target lower than the baseline and the routine service-related indicators were under targeted. The project results framework was not set up in the portal early enough (over two years after the AF became effective) and even when eventually set up, it was not correctly and completely filled. M&E reporting was thus largely paper based. Finally, considering the PDO was to scale up the provision and use of high impact maternal and under five interventions, the non-inclusion of family planning indicators technically means family planning was not seen as a high impact intervention. Based on the above, the M&E design of the project is rated Modest. M&E Implementation. Rating: Modest 111. At the facility level, project routine data was collected periodically which in turn informed the payment of subsidies to the health facilities (to cover the cost of health facility operations as well as payment of performance bonuses to the staff). CMVAs provided the first level of verification of health facility claims, and Independent verification agents were engaged to provide second level of review of these data before they were validated. Each PIU had a dedicated M&E officer for the project throughout the life of the project. At LGA level, the HMIS officers were responsible for collecting routine health facility data and entering same into the DHIS2 portal. Because the project indicators were different from the routine HMIS in the initial phases of the project, these had to be collected separately and uploaded unto a dedicated RBF portal for NSHIP. 16Fritsche, G., et al. (2014). Performance-Based Financing Toolkit. Washington DC, © World Bank. https://openknowledge.worldbank.org/handle/10986/17194 License: CC BY 3.0 IGO Page 45 of 84 The World Bank Nigeria States Health Investment Project (P120798) 112. The result framework for the project was routinely updated, however the additional financing led to the revision of some indicators in the results framework and inclusion of new ones which were not updated on the WB operations portal early, making tracking of progress difficult. This was remedied through an ISR in December 2019 . The indicator number of 12-23 months old children vaccinated with Penta 3 was not included in RF in the portal at the time of writing the ICR report. 113. Following the AF and the restructuring in 2018, the project harmonized project indicators with those of HMIS and link the NSHIP database with DHIS2 to allow for inter-operability of the two platforms. The completion of the project HMIS / DHIS2 platform was however not completed until the last year of the project which significantly affected its usefulness and sustainability of the intervention. This new database and dashboard came out of a perceived need to have a link to the national HMIS system, which was also DHIS2 based. A fully functional online database with a public dashboard had been functional since the early pre-pilot in December 2011, up to 2017 when its use was abandoned pending the introduction of a new database and online public dashboard system. The latter took a long time to complete, and in the meantime the states reported their results in a customized MS Excel worksheet, a situation which was far from ideal. Surveys which were to provide data for reporting the progress of selected indicators were not conducted early enough. the SMART survey was supposed to be conducted annually. The 2019 SMART survey was conducted in late 2019 and early 2020. Data was not available to measure the performance of one of the intermediate result indicators. All these affected adequate reporting of the project. The conduct of M&E of the project was seconded to FMoH’s department of health planning, research and statistics. This however did not improve the delivery of the M&E component of the intervention neither was skill transferred to the point that the government can do these without the support of a TA. Based on the above, the M&E Implementation is rated Modest. M&E Utilization. Rating: Substantial 114. Use of project data: There are several data sets utilized in the project which include Quantity data, Quality data, Community Client Satisfaction Survey (CCSS), quality counter verification for Health facilities and, performance framework assessment for CMVAs, IVAs, HMBs and LGAs. On the portal, data sets were designed based on the type of service offered by the facility {(minimum package of activities (MPA) and complimentary package of activities (CPA)}. These data sets were further categorized based on the periods (including changes in the data collection tools - quantity and quality) over the course of the project. Quantity Quality Quantity MPA/CPA for NSHIP (Until June 2019) Quality MPA/CPA for NSHIP (Until June 2019) Quantity MPA/CPA for NSHIP (From July 2019) Quality counter verification MPA/ CPA for NSHIP Quantity MPA/CPA for AF-NSHIP (Until March 2018) Quality MPA/CPA for NSHIP/ AF-NSHIP Quantity MPA/CPA for AF-NSHIP (From April 2018) Quality counter verification MPA/CPA for NSHIP/ AF-NSHIP 115. Bluesquare connected the PBF DHIS2 to the federal HMIS using D2D tool. The D2D was used to get claimed quantity of service values and population data from the HMIS into the PBF DHIS2. Effectively, data from quarter 1, 2019 until quarter 1, 2020 have been entered while historical data from quarter 4, 2018 (NSHIP) and quarter 2, 2017 (AF) have been uploaded. In addition, Bluesquare imported NSHIP data from inception from open RBF into the project dhis2 portal. Although all backlog data have been uploaded there is no specific component that tests data completeness unless through a pivot table that will require further analysis. To ensure data completeness, data harmonization exercise was conducted which has improved the completeness of the data on the platform. Page 46 of 84 The World Bank Nigeria States Health Investment Project (P120798) 116. Invoicing: Invoicing was done using Hesabu as a calculating engine that runs on the DHIS2 portal . The portal can be used to generate invoices for AF-NSHIP and NSHIP facilities by setting up the PBF parameters for each facility. Parameters such as Facilities category (1, 2, 3, 4 and 5), Quantity Unit fee and Quality Available score. The project DHIS2 can generate invoices from quarter 4, 2018 (NSHIP) and quarter 1, 2019 (AF) while other periods can be generated from the Open RBF portal. From the data harmonization exercise conducted, the teams identified need some adjustment in sanction application. 117. Training and Capacity building. Several trainings have been conducted both physically and virtually to build capacity on DHIS2, mobile tool collection, invoicing, and data visualization. Activities involving DHIS2 customization were conducted in 2019 to customize the Portal with the project objectives. During these activities, several data elements / project indicators were reviewed to match the National indicators, aggregation and annualize were carried out. Also, capacity building on DHIS2, mobile data collection and Invoicing for National PIU team, states teams and LGA teams, contracted firms (CMVAs, IVAs) and end users have been conducted. The National team is conversant with fee setting as well as quality (max) score settings. 118. Data visualization: The Bluesquare team completed the dashboards for the project based on the outcome of the capacity building during FY2020 Detailed Implementation Plan – DIP (National & States). Also, Bluesquare team updated Dataviz to show new indicators (public portal). 119. The NSHIP Impact Evaluation. An impact evaluation was embedded in the parent project of the NSHIP and reflected results in the three NSHIP Parent States of Ondo, Nasarawa and Adamawa. The (strongly) positive results on quantity and quality of maternal and child health services were presented during the mid- term review in November 2017. The positive results informed the design of the Basic Health Care Provision Fund with its three gateways, and also the Board approved HNP Nigeria 2020 MPA project design, while operational lessons from the parent project led to changes to the institutional design of the NSHIP additional financing intervention in the five states of Borno, Yobe, Gombe, Bauchi and Taraba. The project would have benefited from an end-line evaluation. Based on the above, the M&E Utilization is rated Substantial. Justification of Overall Rating of Quality of M&E 120. With a Modest rating in M&E design, Modest rating for M&E implementation and Substantial for utilization, the project overall M&E rating is Modest. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 121. The project strengthened the Health Care Waste Management (HCWM) system in the participating states. It also ensured the application of necessary safeguard requirements and practice at primary health care facility level, built capacity on HCWM at state, local government and facility levels. A HCWM manual based on the National HCWM plan was developed and distributed to all stakeholders. Train the trainer workshop on HCWM was held at the National level and the training was stepped down to state levels. The project ensured conformance with the HCWM plan at the facility level. Incinerators, colour-coded bins, sharp boxes and PPEs were supplied to all facilities throughout the period of project implementation. Environmental health officers were engaged, trained, and equipped to identify and resolve HCW issues. In some states, private health care wastes collectors were accredited to collect, treat and dispose of HCW according to the World Bank and National Standards. 122. Financial management supports were provided to the PIUs both at the state and National levels. During missions, review of PIUs financial documentations were done. In most cases, issues Identified included unretired advance, inadequate documentations, and potential ineligible expenditures. The agreed Page 47 of 84 The World Bank Nigeria States Health Investment Project (P120798) actions on these issues were usually communicated to the relevant PIUs and were resolved. Overall, the internal controls were considered adequate, as authorization and approvals reviewed were satisfactory, and internal audit functions were in place at the FMOH, NPHCDA and the state PIUs. The PIUs were trained and re-trained on financial reporting and audit was conducted periodically consistent with the World Bank requirements. In one of the states, a case of fraud was detected, and the state was made to pay back the money and thereafter, stronger measures were put in place to improve financial processes. The Bank provided hands-on procurement support to the NSHIP PIUs to ensure compliance to the Procurement Regulations and Financial Handbook. Overall, the procurement under the NSHIP with respect to internal controls, record keeping, use of STEP etc. was substantially carried out in accordance with the Procurement Regulations, the PAD and financing agreements. C. BANK PERFORMANCE Quality at Entry : The quality at entry is rated Satisfactory 123. The preparation of the project was very consultative and developed based on a set of project preparation studies which included the Political Economy and Institutional Review studies, Human Resource studies, and Public Expenditure Reviews (PER) in Ondo and Nasarawa States. The dissemination of the studies engendered systematic consultations with key stakeholders and beneficiaries and brought international perspectives. Similarly, the various preparation missions, workshops and training exposed the clients to the basis and principles of RBF. Apart from the wide participation of stakeholders in these activities, the engagement of three Rwandan consultants stationed in the states during the preparatory activities for the pre-pilot gave the clients on the job training on PBF and enhanced the design of the project. This translated to learning by doing and local application of PBF. Learning from the impact evaluation that was conducted as part of the project MTR were incorporated into the design of the project. The lessons learned were incorporated into the final design of the project. 124. The development objective of the project had the right level of ambitiousness, particularly in the context of a long-lasting macroeconomic challenges of the country. In the design, opportunities were created for refinement as it was planned that during implementation that the project will be reviewed carefully twice each year by joint government/Bank review missions using the agreed upon M&E framework. 125. The project introduced a new concept to service provision in Nigeria taking resources away from payment of inputs to payments for results. This was innovative and revolutionary in Nigeria. There was strong ownership of the operation largely due to the stakeholders’ engagement that was well done during the preparation of the project. The institutional arrangement was pragmatic, ensuring government ownership and cross-sector coordination. In addition, plans for appropriate capacity strengthening of the PIUs were included and the quality of the monitoring and evaluation system was good. 126. Appropriate measures to mitigate operational risks were identified which included performance- based payment for implementing agency risk (original NSHIP) and capacity building of the implementing agency. Quality of Supervision. The quality of supervision is rated Moderately Satisfactory. 127. The Bank provided the Government with close support over the entire implementation period. Formal implementation support visits included experts from each sector - fiduciary team members, environmental safeguards specialists and government counterpart. Overall, these visits were held every six months and created opportunities for both the Bank and the Government to address emerging issues right there on the field. Implementation challenges were identified and included in the periodic ISRS. Despite Page 48 of 84 The World Bank Nigeria States Health Investment Project (P120798) security challenges, the COVID 19 pandemic and changes in the roles of the team members, continuity in the Bank team composition was ensured and support to the PIUs was provided throughout the implementation period. 128. The project did not merge the result framework of the original project with the AF result framework for over two years in the portal. This created a gap in reporting. The indicators number of 12-23 months old children vaccinated with Penta 3 in Nasarawa and Ondo and NE states were not included in the RF as at the time of writing the ICR report. 129. There were two levels of annual project workplan approval before actual implementation. The first was for the entire annual workplan while the other was when each activity was being implemented. The reason for this double approval was to ensure that approved annual plans are responsive to possible changes in the context in course of the year. Most state PIUs felt that the response time for the approval of the activities was delayed and that affected delivery of some approved annual plans. The wait was however necessary as the WB task team needed to double check the plan to ensure that what was approved in the annual plan was what was actually being implemented. 130. Supervision of safeguard or environmental and social aspects: Environmental issues were identified and resolved through supervision and monitoring of the project implementation at PIU and facility levels. In general, compliance with the Bank’s safeguards requirements was adequate during the course of project implementation. The Project did not trigger the Safeguard Policy on Involuntary Resettlement since all sub- project activities were limited to existing health facilities. The Social Safeguard requirements under the Project focused extensively on the implementation of a grievance redress mechanism (GRM). A GRM focal person who maintained a GRM logbook coordinated the collection of grievances and facilitated grievance resolution or escalation to the appropriate channel for resolution. Quarterly GRM reports were generated by the LGAs and disseminated to the state, while consolidated state reports were transmitted to the Federal level and shared with the Bank. No significant issue was reported throughout the life of the project. 131. The quality of reporting, based on the Government’s progress and intermediate financial reports, was high. The aide-memoires adequately reflected issues and action plans. The ISRs were informed by discussions with the CMU and HNP management and accurately highlighted the issues and action plans for Management attention. Management comments confirmed the key issues that the team highlighted in the ISR and summarized guidance that had been discussed with the team as part of an active dialogue on the operation. The only limitation is the difference in the values in the results Framework included in the ISRs and the body, especially in the last ISR. 132. During the MTR, a lot of hand holding was done by the Bank for those collecting the impact evaluation data as the technical agency contracted was not as effective as expected. The Bank technical team spent quality time cleaning that data as the baseline data had some quality issues which were addressed. Justification of Overall Rating of Bank Performance 133. Based on the above, the rating of Bank performance is Moderately Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 134. The Government of Nigeria appears committed to sustaining the gains of the NSHIP project. There are however some current realities that pose challenges to this intent as described below. Page 49 of 84 The World Bank Nigeria States Health Investment Project (P120798) 135. COVID-19 impact on essential services: As noted earlier (109 above), the COVID-19 pandemic affected uptake of essential services and further weaken the already weak healthcare system. If this impact continuous, it can erode some of the gains of the project. 136. Security challenges: Insurgency continue to remain a huge challenge Nigeria is facing. Due to security, some facilities in Borno, Adamawa and Yobe states in particular were either destroyed or abandoned. Functional PHCs in a safe environment are needed if quality health care services are to be provided to millions of Nigeria. Until this is guaranteed, the impact of the NSHIP project stand risk of losing some of its impact. 137. Macroeconomic challenges limiting the fiscal space especially at the state level. Nigeria is currently experiencing economic challenges. This has made it very challenging for Government to meet its financial obligation. As of December 2020, over 400 MDAs owe staff salaries for over two months. The federal Government is thus struggling to support states. With the dwindling resources at the disposal of the national and state governments, it is becoming challenging for the governments to commit the needed funds to sustain the impact of the project. While the Government has released funds for the implementation of the BHCPF, there are a lot of uncertainties with respect to its implementation as the fiduciary processes of the fund are still weak. Continuous and adequate financing of the heath care sector is critical to maintain the gains of NSHIP project. 138. Inability of the Government to sustain payment of incentives because of the structure of payment of civil servants’ salaries and remunerations. Health workers do not work in isolation of the other employees of government. While there are different salary scales for different categories of workers, the payment principles are largely the same and the determination of the income, wages and salaries of all healthcare workers is outside the Ministry of Health. It is practically impossible to place healthcare workers alone on performance-based payments as the health care workers union and management agencies will vocally oppose this. General civil service rules may have to change to accommodate this model. This is outside the control of the health sector. The strength of the NSHIP model is the payment of incentives for workers. It is uncertain whether workers will be motivated to improve productivity without an incentive system in place. This poses risk to the sustainability of the project impact. 139. Integrity of the system not yet at a level that sustains a performance-based programming. Performance based programming relied heavily on the accuracy of data. When the quality of data that informs payment is weak, the process can be compromised. To some extent, this was experienced in the life of the project. The 2020 SMART survey was characterized by poor quality of data. While attempts were made to address these largely at the instance of the Bank follow-on TA, policy dialogue, and the Government’s commitment to sustain the PBF/outcome of the project through BHCPF/other strategies/donor engagements will be required. With the project ending, the Bank will not be in a position to ensure these happen. The integrity of the system (data and financial management) is crucial to maintain the gains of the project. V. LESSONS AND RECOMMENDATIONS 140. Keeping project designs straightforward and learning from implementation experience: The NSHIP design was complex due to the mix of incentive approaches (State level and district level DLIs; two types of decentralized health facility financing approaches; PBF and DFF). While the logic appears to be simple (paying for outputs and not inputs), the concept and process were not only new, but required a 360-degree change from what the Nigerian Health Care system was used to. Because of the newness of the concept of RBF, international RBF experts were invited to build in country capacity to be able to implement the project. PBF Page 50 of 84 The World Bank Nigeria States Health Investment Project (P120798) and also DFF approaches are complex systemic interventions and both approaches need significant handholding and learning from implementation. 141. Lessons on good governance for decentralized health facility financing in the Nigerian context. The foremost lesson from 10 years of NSHIP implementation is the need to work on a ‘ground zero’; i.e. the political economy which determines how much funding is allocated for health in Nigeria. Without enhanced domestic funding pre-conditions met, no health project funded by loans or grants will sustainably impact Nigerian health indicators. Sustainability plan should be clearly stated in such a way that transition is clear from inception. This will ease the sustainability challenges projects may face despite their significant impacts. 142. The current NSHIP way of verifying quality (through LGAs), and counter-verifying through IVAs is vulnerable to fraud. The ‘carrot and stick’ method of paying providers puts too much incentive on the quality measure and can lead to fraud.” Different methods to verify will need to be tried. 143. Approaches to scaling access to quality health care in fragile and conflict areas: By tailoring the NSHIP project and services offered towards the context of the insurgency in the North East, this has allowed for more effective provision of services. The NSHIP project introduced very innovative approaches to scaling up provision of health care services in a fragile and or conflict areas. Some of these approaches include mobile clinics, direct facility financing which guaranteed regular funding to the facility and introduction of psychosocial support in routine health care delivery. While the impacts of each of these approaches was not evaluated, the number of persons reached in the fragile and insecure states of Adamawa, Yobe and Borno continue to increase even with the security challenges of the areas. Results in the NSHIP were achieved by decentralizing funding to health facilities, enhancing autonomy for financial management including for drug purchases, and rigorous monitoring and coaching using structured and quantified quality checklists. The NSHIP example can be tweaked to scale up services in these areas. 144. Incentives of improve healthcare workers motivation and engagement, however payment of separate incentives to healthcare workers is not sustainable in Nigeria unless there is a change in how civil servants are rewarded in Nigeria. NSHIP bonus payments improved the financial situation among health workers both that of contract and also the LGA staff deployed to the NSHIP facilities. NSHIP also focused on the working environment, improving infrastructure and general working conditions at the facility level. While PBF paid incentives directly to health care workers, DFF created an enabling environment for health care workers. Both resulted in increased service delivery. In a resource constraint environment with rigid service structures which may make it challenging to pay performance incentives to health care workers, DFF can be a cost-effective way of scaling access to quality health care. 145. Where does power lie? In the placement of a project PIU, it is very important that a political economy analysis of relevant actors is conducted to inform where the project PIU should sit. The NSHIP PIU at the state level was seated in the State Primary Health Care Board. While SPHCBs oversee PHCs, secondary facilities are under the state Hospital management Board and the Commissioners of Health are the ones that gives account of health in the state. In states where there was understanding between the state’s Ministry of Health and the SPHCD, key stakeholders in the state were properly briefed of the project. The reverse was the case in states where this understanding needed improvement. A good understanding of the political economy of the state do improve project performance. 146. Enhanced separation of functions (institutional arrangement) and rigorous counter-verification mechanism improves quantity and quality of health services. The NSHIP has experimented with various institutional arrangements for strategic purchasing of quality basic health services at scale. The first iteration of institutional arrangements in the parent project suffered from an inadequate separation of functions and weak counter-verification mechanism, while still showing noteworthy impact as documented by the impact Page 51 of 84 The World Bank Nigeria States Health Investment Project (P120798) evaluation. In the additional financing states, a new and much enhanced separation of functions and more rigorous verification and counter-verification mechanisms were introduced, and this second institutional set- up showed strong increases in quantity and quality of health services as independently verified by two separate non-governmental agencies. Box 6: Rigorous data validation, a case in Bauchi and Yobe. A rigorous data-revalidation exercise introduced in Bauchi and Yobe states, to re-validate and certify the large performance increases which had been signed off by the non-governmental agencies has shown excellent concordance with reported and validated data. This data-revalidation exercise which focused on two key services Institutional Deliveries and Implants and Quality of services reported has documented that the large volume increases were correctly reported, while there were only discrepancies on the Quality metrics (for instance instead of 98.4% to 99.7% average in Gamawa, Bauchi and Katagum LGAs as validated by the IVAs, about 69% to 86% average Quality scores as re-validated). . Page 52 of 84 The World Bank Nigeria States Health Investment Project (P120798) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS17 A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To increase the delivery and use of high impact MCH interventions and improve quality in 8 states Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 1. Proportion of children sick in Percentage 61.70 72.00 69.10 the last month who used a government hospital or clinic 01-Jan-2016 31-Jul-2020 31-Jul-2020 (average of Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 2. Number of outpatient visits Number 169,666.00 500000.00 20,936,636.00 per year, children and adults (sum of Adamawa, Bauchi, 31-Dec-2014 31-Jul-2020 30-Sep-2020 17The indicators number of 12-23 months old children vaccinated with Penta 3 is not included in the RF as the RF was not updated to include this indicator before the time of writing the ICR report. Page 53 of 84 The World Bank Nigeria States Health Investment Project (P120798) Borno, Gombe, Taraba and Yobe) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 3. Proportion of children sick in Percentage 55.70 85.00 55.40 the last month who used a government hospital or clinic 01-Jan-2014 31-Jul-2020 31-Jul-2020 (average of Nasarawa and Ondo) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 4. Average Health Facility Score Percentage 41.00 72.00 61.60 - Structural Quality of Care (average of Adamawa, Bauchi, 30-Dec-2016 31-Jul-2020 30-Sep-2020 Borno, Gombe, Taraba and Yobe) Comments (achievements against targets): Page 54 of 84 The World Bank Nigeria States Health Investment Project (P120798) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 5. Average Health Facility Score Percentage 45.00 72.00 60.20 - Structural Quality of Care (average of Nasarawa and 30-Dec-2016 31-Jul-2020 30-Sep-2020 Ondo) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 6. Proportion of skilled births Percentage 56.65 75.00 73.55 attended by skilled health personnel (average of 31-Dec-2014 31-Jul-2020 30-Sep-2020 Nasarawa and Ondo) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 7. Proportion of skilled births Percentage 22.20 35.00 75.00 68.60 attended by skilled health personnel (average of 31-Dec-2014 31-Jul-2020 31-Jul-2020 30-Sep-2020 Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) Page 55 of 84 The World Bank Nigeria States Health Investment Project (P120798) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 8. Percentage of 12-23 month Percentage 27.40 35.00 49.00 68.60 old children vaccinated with Penta3 (NE) 31-Dec-2010 31-Jul-2020 31-Jul-2020 30-Jun-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 9. Number of 0-12 month old Number 533,368.00 770000.00 2,424,869.00 children vaccinated with Penta3 31-Dec-2014 31-Jul-2020 31-Jul-2020 9A. Number of 0-12 month Number 96,000.00 110000.00 468,224.00 old children vaccinated with Penta3 (Nasarawa) 31-Dec-2014 31-Jul-2020 30-Jun-2020 9B. Number of 0-12 month Number 110,090.00 160000.00 869,556.00 old children vaccinated with Penta3 (Ondo) 31-Dec-2014 31-Jul-2020 31-Jul-2020 Comments (achievements against targets): Page 56 of 84 The World Bank Nigeria States Health Investment Project (P120798) The baseline, target and actual achieved values for this indicator are the sum of Nasarawa, Ondo and NE states Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 10. Number of 0-12 month old Number 327,278.00 500000.00 1,087,090.00 children vaccinated with Penta3 (NE) 31-Dec-2014 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 11A. Percentage of 12-23 Percentage 39.40 65.00 82.90 month old children vaccinated with Penta3 (Nasarawa) 31-Dec-2010 31-Jul-2020 31-Jul-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 11B. Percentage of 12-23 Percentage 67.30 75.00 86.60 month old children vaccinated 31-Dec-2010 31-Jul-2020 31-Jul-2020 Page 57 of 84 The World Bank Nigeria States Health Investment Project (P120798) with Penta3 (Ondo) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 12. Births (deliveries) attended Number 26,960.00 88503.00 700,000.00 2,083,502.00 by skilled health personnel (number) 31-Dec-2014 31-Jul-2020 31-Jul-2020 31-Aug-2020 Proportion of births attended Percentage 22.20 35.00 75.00 68.60 by skilled health personnel (NE) Proportion of births attended Percentage 62.50 75.00 64.40 by skilled health personnel (Nasarawa) Proportion of births attended Percentage 55.50 75.00 82.70 by skilled health personnel (Ondo) 13A. Number of births Number 23,193.00 32000.00 150,000.00 334,171.00 attended by skilled health personnel (Nasarawa) 31-Dec-2014 31-Jul-2020 31-Jul-2020 30-Jun-2020 Page 58 of 84 The World Bank Nigeria States Health Investment Project (P120798) 13B. Number of births Number 29,247.00 32000.00 150,000.00 543,136.00 attended by skilled health personnel (Ondo) 31-Dec-2014 31-Jul-2020 31-Jul-2020 31-Jul-2020 14. Number of births Number 24,520.00 150000.00 400,000.00 1,206,195.00 attended by skilled health personnel (NE) 31-Dec-2014 31-Jul-2020 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 15. Average Health Facility Percentage 41.90 61.00 67.00 Quality of Care Score 31-Mar-2014 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 16. Number of outpatient visits Number 346,990.00 13500000.00 10,586,618.13 by children and adults (Nasarawa and Ondo states) 31-Dec-2014 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Page 59 of 84 The World Bank Nigeria States Health Investment Project (P120798) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 17. Direct project beneficiaries Number 0.00 1400000.00 12,000,000.00 22,390,040.00 31-Dec-2010 31-Jul-2020 31-Jul-2020 30-Sep-2020 Female beneficiaries Percentage 0.00 60.00 63.00 Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: Strengthening Service Delivery and Institutional Peformance Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion People who have received Number 0.00 6900000.00 23,909,634.00 essential health, nutrition, and population (HNP) services 02-Oct-2013 31-Jul-2020 30-Sep-2020 People who have received Number 0.00 3800000.00 15,063,069.00 essential health, nutrition, and population (HNP) services - Female (RMS requirement) Page 60 of 84 The World Bank Nigeria States Health Investment Project (P120798) Number of children Number 0.00 2500000.00 2,712,950.00 immunized 02-Oct-2013 31-Jul-2020 30-Sep-2020 Number of women and Number 0.00 3700000.00 19,620,631.00 children who have received basic nutrition services 02-Oct-2013 31-Jul-2020 30-Sep-2020 Number of deliveries Number 0.00 700000.00 2,083,502.00 attended by skilled health personnel 02-Oct-2013 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 18. Proportion of health Percentage 0.00 30.00 100.00 facilities with functioning management committees 30-Dec-2016 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Page 61 of 84 The World Bank Nigeria States Health Investment Project (P120798) 19. Number of mobile clinics in Number 0.00 75.00 44.00 project areas 30-Dec-2016 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 20. Number of women Number 45.00 20000.00 0.00 receiving counseling on gender based violence or mental 30-Dec-2016 31-Jul-2020 30-Sep-2020 health services from a trained provider Comments (achievements against targets): This indicator was not reported by the national HMIS and so no date to capture its performance at the end of the project Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 21. Proportion of pregnant Percentage 39.00 50.00 58.70 women tested for HIV and who received test results (NE) 30-Dec-2016 31-Jul-2020 30-Sep-2020 Comments (achievements against targets): Page 62 of 84 The World Bank Nigeria States Health Investment Project (P120798) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 22. Proportion of pregnant Percentage 37.00 50.00 63.65 women tested for HIV and who received test results (Ondo and 30-Dec-2016 31-Jul-2020 30-Sep-2020 Nasarawa) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 23. Proportion of mothers aged Percentage 20.00 35.00 47.20 15 to 19 years of age who delivered in the last two years 31-Dec-2014 31-Jul-2020 30-Sep-2020 and who received birth attendance Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 24. Number of children treated Number 0.00 25000.00 304,343.00 Page 63 of 84 The World Bank Nigeria States Health Investment Project (P120798) for severe acute or chronic 30-Dec-2016 31-Jul-2020 30-Sep-2020 malnutrition per year in the project area Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 25. Number of LGA with Number 3.00 53.00 80.00 113.00 ongoing PBF 31-Dec-2011 31-Jul-2020 31-Jul-2020 30-Sep-2020 Number of LGA with ongoing Number 1.00 64.00 82.00 PBF (NE) 02-Oct-2013 31-Jul-2020 30-Sep-2020 Number of LGA with ongoing Number 1.00 7.00 13.00 PBF (Nasarawa) 02-Oct-2013 31-Jul-2020 30-Jun-2020 Number of LGA with ongoing Number 1.00 9.00 18.00 PBF (Ondo) 02-Oct-2013 31-Jul-2020 31-Jul-2020 Comments (achievements against targets): Page 64 of 84 The World Bank Nigeria States Health Investment Project (P120798) Component: Component 2: Technical Support Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Proportion of primary health Percentage 0.00 35.00 53.00 facilities having essential medicines and commodities in 02-Oct-2013 31-Jul-2020 30-Sep-2020 stock Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Proportion of on-duty technical Percentage 0.00 84.00 63.60 staff present at health facility on the day of the survey 02-Oct-2013 31-Jul-2020 31-Jul-2020 Comments (achievements against targets): Page 65 of 84 The World Bank Nigeria States Health Investment Project (P120798) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: 1: Increase in the delivery and use of high impact maternal health interventions 1. Proportion of births attended by skilled health personnel (Nasarawa) 2. Proportion of births attended by skilled health personnel (Ondo state) 3. Proportion of births attended by skilled personnel -NE (average of Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) 4. Number of births attended by skilled health personnel (NE) (Sum of Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) Outcome Indicators 5. Number of births attended by skilled health personnel (Ondo state) 6. Number of births attended by skilled health personnel (Nasarawa state) 7. Direct Project Beneficiaries (number) 8. …......of which female (%) 1. Proportion of pregnant women tested for HIV and who received test result (NE) 2. proportion of pregnant women tested for HIV and who received test result (Ondo and Nasarawa) Intermediate Results 3. Proportions of mothers aged 15 to 19 years of age who deliver in the last two years who receive skill birth attendance. Indicators 4. People who have received essential health, nutrition and population services (numbers) females (RMS) 5. Number of women and children who received basic nutritional services (Number) Component 1: 1. Increase in number of women who access PHC services Key Outputs by Component 2. Increase in number of women who received CCT (linked to the achievement of Component 2: the Objective/Outcome 1) 3. HCW trained on MH 4. Increase in number of people served through mobile clinic Objective/Outcome 2: Increase in the delivery and use of high impact child health interventions 1. Proportion of children sick in the last month who used a government hospital or clinic (average of Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) 2. Proportion of children sick in the last month who used a government hospital or clinic (average of Nasarawa and Ondo) 3. Percentage of 12-23 months old children vaccinated with Penta 3 (NE) )Percentages) Outcome Indicators 4. Percentage of 12-23 months old children vaccinated with Penta 3 in Nasarawa state) 5. Percentage of 12-23 months old children vaccinated with Penta 3 in Ondo state 6. Number of 12 – 23 months old children vaccinated with Pentavalent in Nasarawa state ) 7. Number of 12 – 23 months old children vaccinated with Pentavalent 3 in Ondo state Page 66 of 84 The World Bank Nigeria States Health Investment Project (P120798) 8. Number of 12 – 23 months old children vaccinated with Pentavalent in NE states (sum of Adamawa, Bauchi, Borno, Gombe, Taraba, and Yobe) 9. Number of outpatient visits per year, children and adults (sum for Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) 10. Number of outpatient visits by children under five Intermediate Results 1. Number of children treated for severe acute or chronic malnutrition per year in project area Indicators 2. Number of women and children who received basic nutritional services (Number) Component 1: 1. Increase in number of children under five who access PHC services Key Outputs by Component 2. Increase in number of children vaccinated (linked to the achievement of Component 2: the Objective/Outcome 2) 3. HCW trained on CH 4. Increase in number of people served through mobile clinic Objective/Outcome 3: Improvement in the quality of care at selected health facilities in the participating states. 1. Average Health Facility Score - structural Quality of Care(average of Adamawa, Bauchi, Borno, Gombe, Taraba and Yobe) Outcome Indicators 2. Average Health Facility Score - Structural Quality of Care (average Nasarawa and Ondo) 3. Average Health Facility Quality of Care Score 1. Number of LGA with ongoing PBF Intermediate Results 2. Proportion of HFs in the project area with functioning management committees having community representation Indicators 3. Proportion of health facilities having essential medicines and commodities in stock (percentages) 4. Proportion of on duty technical staff present at facility on the day of the survey (percentages) Key Outputs by Component 1. Improved service delivery (linked to the achievement of the Objective/Outcome 2) Page 67 of 84 The World Bank Nigeria States Health Investment Project (P120798) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Ayodeji Oluwole Odutolu Task Team Leader(s) Daniel Rikichi Kajang Procurement Specialist(s) Adewunmi Cosmas Adekoya Financial Management Specialist Alexandra C. Bezeredi Social Specialist Ayodeji Oluwole Odutolu Team Member Caroline Mary Sage Team Member Caroline Mary Sage Social Specialist Joseph Ese Akpokodje Team Member Joseph Ese Akpokodje Social Specialist Ogo-Oluwa Oluwatoyin Jagha Team Member Amos Abu Social Specialist Ugonne Margaret Wunyi Team Member Supervision/ICR Gyorgy Bela Fritsche, Fatimah Abubakar Mustapha Task Team Leader(s) Daniel Rikichi Kajang, Bayo Awosemusi, Mary Anika Procurement Specialist(s) Asanato-Adiwu Uchechi Chizomam Oloba Financial Management Specialist Arigu Yusufu Kudu Financial Management Specialist Onoriode Ezire Team Member Cindy Ijeoma Ikeaka Social Specialist Elijah Abiodun Siakpere Social Specialist Vivian Obianujunwa Mbusu Team Member Page 68 of 84 The World Bank Nigeria States Health Investment Project (P120798) Mayowa Oluwatosin Alade Team Member Fanen Ioryisa Verinumbe Team Member Elina Pradhan Team Member Ogochukwu Joy Medani Team Member Daniela Hoshino Team Member Amos Abu Environmental Specialist Joseph Ese Akpokodje Social Specialist Joyce Chukwuma-Nwachukwu Procurement Team B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY10 3.347 30,855.02 FY11 51.130 410,694.08 FY12 90.982 659,100.63 Total 145.46 1,100,649.73 Supervision/ICR FY12 0 306.47 FY13 78.572 368,120.22 FY14 52.986 334,378.23 FY15 57.741 367,350.11 FY16 76.990 635,675.62 FY17 84.029 374,209.14 FY18 86.513 444,358.20 FY19 76.030 427,266.96 FY20 76.519 407,493.02 Total 589.38 3,359,157.97 Page 69 of 84 The World Bank Nigeria States Health Investment Project (P120798) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval (US$M) Actual at Percentage (US$M) Project Closing of Approval Original NSHIP 1st AF 2nd AF (US$M) (US$M) IDA HRITF Total HRITF IDA GFF Total Total Component 1: Result Based Financing 106.8 15.6 122.4 1.7 71 14 85 209.1 225.6 66.0% Component 2: Technical Assistance 29.6 4.4 34 19.5 3.5 23 57 69.6 18.0% Component 3: Partnership for Service Delivery 0 0 0 0 15.5 2.5 18 18 18 5.7% Pre-Financing 3 3 0 3 0.9% Contingencies 10.6 10.6 19 19 29.6 9.3% Total 150 20 170 1.7 125 20 145 316.7 313.2 100% Page 70 of 84 The World Bank Nigeria States Health Investment Project (P120798) ANNEX 4. EFFICIENCY ANALYSIS The Purpose of the Cost Benefit Analysis. The project development outcome of the NSHIP is to increase the delivery and use of high impact maternal and child health interventions and improve quality of care available to the people in Nasarawa and Ondo and all the States in the North East. The purpose of this analysis is to assess whether the dollar benefit of NSHIP implemented in Nigeria outweighs its dollar costs. To do so, this analysis monetizes the major benefits and costs associated with the project, and reports on three measures: 1. Benefit to cost ratio (BCR): the ratio between the benefits and costs of performance-based financing, expressed in monetary units at discounted present values. A ratio greater than one indicates that project benefits outweigh its costs. 2. Net present value (NPV): the sum of the present values of a cash flow stream. An NPV above zero indicates that PBF was profitable. 3. Internal rate of return (IRR): the discount rate that equates the present value of the project’s cash inflow to the present value of its outflow. While the NPV measures the project’s dollar profitability, the IRR measures its percentage profitability. Beneficiaries When the project was designed in 2011, it was designed to benefit the entire populations of the original three NSHIP states with an estimated population of 9.4 million. Specifically, the Project aimed to annually benefit about 400,000 pregnant women in 3 States; about 2 million women between 15 and 49 years (795,000 in Adamawa, 400,000 in Nasarawa and 850,000 in Ondo); and about 1.8 million children under five years of age (636,000 in Adamawa, 375,000 in Nasarawa and 775,000 in Ondo). In 2018, the project was scaled up to additional four states in North East Nigeria. The combined population of the seven states as at 2020 was 33,272,578 (NPopC 2020). The project implemented RBF across the seven states primarily in PHCs with secondary facilities as referral centers. At the end of the project, the project was estimated to have reached a target population of 22,390,040, people, or 67% of the total population of the seven states (NPopC 2020 population estimate). The basic package of health services at health center and community levels covered 20 services, which aimed to primarily reduce maternal and child mortality. Project Benefits To assess the impact of the project in the population served by the contracted health facilities, changes in health service utilization, as reported from surveys conducted and HMIS date for the period 2015 to 2020 were used to estimate the number of additional child and maternal lives saved. The methodology used by the Lives Saved Tool to attribute additional lives saved to changes in service utilization is described elsewhere.18 We propose that January 2015 be considered as the baseline quarter, and then assume that service increases in the project areas were due to the project intervention (as demonstrated by the mid- term review conducted in 2017). A robust data reporting and verification system was established with project support. Health facilities submitted monthly invoices as to the number of services delivered during the reporting period. Independently contracted agencies visited the health facilities monthly to verify that the claimed amount was accurate. In Nigeria, as well as other countries where PBF has been implemented, the difference between the claimed and verified amount decreases as the system develops over time. To generate an estimate of the additional lives saved, the verified data are modeled. Coverage estimates were made using 18Winfrey, W., McKinnon, R., Stover, J. (2011). Methods used in the Lives Saved Tool (LiST). BMC Public Health, 11(Suppl 3): S32. Page 71 of 84 The World Bank Nigeria States Health Investment Project (P120798) the verified data in the numerator, and the estimated population size within the states targeted by the project in the denominator. We used health outcome data that was pre-populated for Nigeria in the LiST tool (using national estimates from the Demographic Health Survey and Multiple Indicator Cluster Survey), as well as effectiveness assumptions established within the LiST platform and applied in multiple countries. We did not model all services delivered through RBF, but only those that fit within the LiST. Table 16 presents the estimated additional lives saved due to greater health service coverage within the catchment area of health centers and hospitals with RBF contracts. A total of 3,478 maternal lives were saved while 38,413 children’s lives were saved. Table 15: Estimates of additional lives saved due to increased service coverage in the seven NSHIP states Years Maternal Deaths Child Death Total 2015 184 1,285 1,469 2016 370 2,975 3,345 2017 480 3,810 4,290 2018 560 5,469 6,029 2019 633 9,852 10,485 2020 1,251 15,022 16,273 Total 3,478 38,413 41,891 To assign a monetary value to the additional lives saved, we multiplied the number of productive years saved by Gross Domestic Product (GDP) per capita. We assumed that GDP would grow at a rate of 3.6 percent, which is the estimated economic growth rate in Nigeria between 2010 and 2020. We applied a discount rate of 3 percent per year. We assumed that individuals would contribute to the economy from the time they were 18 years old until their time of death, which we took as the life expectancy at birth in Nigeria (54.3 years for all, and 55.2 years for mothers). We took the median age between 12 and 59 months to calculate the average age of child death. We added half the total fertility rate (of 5.39 births per women) to the average age at first birth in Nigeria (21 years) to calculate the average age of maternal death. Table 2 summarizes the benefits, expressed in monetary terms, for the additional lives saved through RBF. Table 2: Monetary value of additional lives saved, modeled using verified and quality adjusted coverage Verified Benefit of PBF program (USD 2020) $579,234,672.61 Project Costs Table 17 summarizes the project costs from project inception to close of the project 2021. The covered the entire cost of the program as reported as funds disbursed from inception to the end of the project. Table 17: NSHIP Project costs 2015 2,016 2017 2,018 2019 2,020 2021 USD (Cumulative) 64,423,159 92,678,282 132,866,687 182,946,477 203,919,323 241,887,575 252,165,908 USD (Annual) 64,423,159 28,255,123 40,188,405 50,079,789 20,972,847 37,968,251 10,278,333 Results Table 18 presents the results of the analysis. the Benefit-Cost Ratio (BCR) is 34.92 for the verified data. This suggests that every dollar invested in the NSHIP project in Nigeria yields an economic return ranging from 34.92 dollars. The investment of US$252.165 million generated economic benefits with a net present value Page 72 of 84 The World Bank Nigeria States Health Investment Project (P120798) of ranging from US$ 7.74 billion. The internal rate of return is 18 percent. Table 18: Project results, considering cost of PBF Internal Rate of Return Benefit Cost Ratio (BCR) Net Present Value ($b)- (NPV) (IRR) Baseline 34.9154573 7.736 18% Sensitivity Analysis This analysis is based on a series of assumptions. A sensitivity analysis was therefore conducted to alter key assumptions to assess the impact on results. The baseline case presented uses the cost of the project only and assumes a 3 percent discount rate and economic growth of 3.6 percent per year. Only the verified results are considered. When the discount rate increases from three to five percent, the project is still economically viable. If the assumption of a lower long-term economic growth rate is used (of 1.19 percent, average GDP growth rate 2015 to 2019), the project is also economically viable. Adjusting for quality was not included in the analysis. Including quality would have made the project more economically viable. Table 19: Sensitivity analysis Benefit Cost Ratio Net Present Value ($b) Internal Rate of Return Baseline 34.92 7.736 18% Discount factor: 5% 17.98 3.75 18% GDP growth rate: 1.2% 16.05 3.474 15% Limitations This analysis has limitations. First, the population that received the intervention was not compared with a control population. Therefore, this analysis assumes that any increase in coverage is attributed to the intervention, but this may not be the case. Second, coverage was calculated using the number of services delivered each quarter as reported in the PBF database as the numerator, and an estimate of the target population served in the denominator. This approach to estimate service coverage provides a rough estimate but may not be accurate. Thirdly, the LiST does not model all services provided through the PBF package and it only measures lives saved for mothers and children under 5. The interventions provided through PBF do target women and children, but also have had benefits for men and people of all ages. The number of lives saved generated from the model is likely underestimated. Furthermore, the model only considers the financial benefits of saving lives, and not of other program benefits like decreasing the length of illness or improving wellbeing. Finally, estimates for lives saved as a result of improvement in quality were not included in the analysis. Page 73 of 84 The World Bank Nigeria States Health Investment Project (P120798) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Page 74 of 84 The World Bank Nigeria States Health Investment Project (P120798) ANNEX 6. SUPPORTING DOCUMENTS (GOVERNMENT ICR REPORT) GOVERNMENT IMPLEMENTATION COMPLETION AND RESULT REPORT (G-ICR)1920 Section One: Description of Project Background 1.1 Project Background 1.1.1 Project Context The Federal Government of Nigeria (FGN), with credit from the World Bank, introduced a Results-Based Financing (RBF) approach through the Nigeria State Health Investment Project (NSHIP) as a way of strengthening the health system. This commenced in December 2011 as a pre-pilot in one Local Government Area (LGA) from each of the three participating States of Adamawa, Nasarawa and Ondo. The project aimed to increase the delivery and use of high impact maternal and child health interventions and improve the quality of care at selected health facilities. This was in response to poor health indices and inability of the country to meet the Maternal and Child Health (MCH) component of the Millennium Development Goals (MDGs). The project also aimed to strengthen institutional performance at the Federal, State and LGA levels by decentralizing health facility financing to address structural issues and motivate health worker performance. NSHIP built on the principles of fiscal decentralization to support health systems reform and improve service delivery at primary and secondary levels through a paradigm shift from input to output financing. Following the success of the pre-pilot, the project was then scaled-up to all 52 LGAs in the three project States by 2014. Subsequently, in 2017, the FGN secured Additional Financing (AF) from the World Bank to expand the NSHIP implementation to the remaining five North East (NE) States of Bauchi, Borno, Gombe, Taraba and Yobe. This marked an important milestone for the project by bringing the total States implementing PBF in Nigeria to eight. The rationale for the AF was to reestablish the service provision network that was initially destroyed by the growing insurgency in the North East. The total financial commitment from the World Bank to NSHIP and AF sums up to $316.7 million ($171.7 million and $145 million for NSHIP and AF respectively) out of which is a grant of $21.7 million for NSHIP and $20 million for AF. 1.1.2 Rationale Prior to project commencement, Nigeria had made limited progress in delivering key health interventions particularly in MCH. Nigeria health indices remained among the worst even within Sub-Saharan Africa. Progress towards meeting MDGs such as infant and under-5 mortality rates, maternal mortality rates and proportion of births attended by skilled health personnel were also falling behind international standards. NSHIP was therefore introduced to shift the focus to strengthening service delivery and improving health outcomes through innovative approaches to health financing. The introduction of AF was to scale-up the success of PBF and to restructure the original operation to better adapt to the NE context. This was particularly in response to the insurgency in the NE, at a time of serious deterioration in the fiscal space in Nigeria. The ICR allows for self-evaluation following the completion of a project for the benefit of both internal and external stakeholders. This enables participants to be held accountable for the project performance and 19 The Government ICR was completed in March 2020 about seven months before the project closed in October 2020. As at the time Government wrote its ICR, 2020 SMART survey data was not yet ready as such data for some indicators were not available. We copied and pasted exactly what was written by government. 20 The reported number of 0-12 months old children vaccinated with Penta3 is higher than what is reported in the ICR report. The ICR report used data obtained from national HMIS. Page 75 of 84 The World Bank Nigeria States Health Investment Project (P120798) results, as well as provide systematic reflection of lessons learnt, and facilitating the sharing of knowledge for future projects. 1.1.3 Project Objectives Project Development Objective (PDO): The original PDO was: “to increase the delivery and use of high impact maternal and child health interventions and to improve the quality of care available to people in the participating States.” This was updated to all States in the North East and Nasarawa and Ondo with the introduction of Additional Financing. PDO Indicators: At the inception of the project, the following indicators were outlined to evaluate the impact of the project within NSHIP states: 1. Proportion and number of 12-23 months-old children immunized with pentavalent 3rd dose. 2. Proportion and number of pregnant women attended to by skilled birth attendance at delivery. 3. Structural quality of care measurements, such as availability of drugs, skilled personnel, basic equipment, and proper waste management. 4. Number of outpatients’ visits, and proportion of under-five children accessing health services; and 5. An HNP core indicator about the number of people with access to essential health services. Following the introduction of AF, NSHIP indicators were reviewed, and additional PDO indicators were added to reach a total of 26 PDO indicators21. Intermediate Indicators A number of changes were made in the intermediate indicators to reflect the broader systems-strengthening approach engendered in the AF and to address citizen engagement. These changes included the addition of new indicators: ● The proportion of health facilities in the project area which have functioning management committees with community representation on them: ● Number of women receiving counselling on gender-based violence or mental health services from a trained provider. ● Number of mobile clinics conducted per year in project area. ● Number of pregnant women tested for HIV during antenatal care; and ● Number of children treated for severe acute or chronic malnutrition per year in the project area. 1.1.4 Relevance of objectives during preparation and at completion The PDOs remained relevant both at preparation and completion to the project and country context. Due to the high rates of child and maternal deaths in Nigeria, project objectives were aimed at targeting the improvement of child and maternal health. For the AF, additional indicators such as psychosocial support, mobile clinics and health facilities with functional committees were added as a response to health needs in the context of the insurgency in the North East. Following the scale-up in 2014, baseline surveys were also updated from 2008/10 National Survey data used in the initial NSHIP project to more recent National Survey data from 2013/14. Although indicators were modified and updated to fit the NE context and to reflect changes in the Baseline surveys, the PDO remained the same at project completion. Section Two: Assessment of Outcome of Project Implementation 2.1 Assessment of the Achievement of Project Development Objectives (PDOs) This section outlines an assessment of the outcomes of the operation against the agreed PDOs, with a focus Page 76 of 84 The World Bank Nigeria States Health Investment Project (P120798) on providing evidence of the achievements of the operation’s objectives along with the contribution of the supported activities and outputs to the PDOs. 2.1.1 PDO Indicators Within the project, there are 5 broad PDO indicators, which are further broken down into 26 more detailed PDO sub-indicators. Out of these, NSHIP has successfully met targets for 18 sub-indicators and this represents 69%. It is important to note a number of limitations within the data. In order to assess a number of the PDOs, National Survey data was used. To ensure the accountability of this data, surveys using the SMART framework were used. However, the data sources used for validating some of the project performance indicators are reliant on surveys which date back to 2018. This includes the Nigerian Demographic Health Survey (NDHS), for which the most recent data was collected in 2018 published in 2019. Similarly, the most recent data from the National Nutrition and Health Survey (NNHS) was collected in 2017 and published in 2018. Furthermore, surveys such as Annual Health Facilities, SDI and MICS were available only for 2016. The use of national survey data for measuring a number of the PDOs also presents the issue of causality. The survey data captures health indicators for the States as a whole, of which we cannot attribute the whole effect of any increases in indicators solely to NSHIP/AF health facilities. For the 8 PDO sub-indicators target not met, it is estimated that with more up to date national survey data, 6 of them will successfully meet the target. From preliminary information from the AF States, the remaining 2 targets not met are as a result of the activities not being activated. It is therefore recommended that an end of the project survey be conducted to give more focused information about the PDO indicators and reflect the July 2020 targets. This will give a more accurate picture of the impact of the project performance. In addition, Borno, Taraba and Yobe should also commence mobile clinic service provision as planned. A. Proportion and number of 12-23 months-old children immunized with pentavalent 3rd dose This indicator was successfully met. The PDO target for the proportion of children from 12-23 months that were immunized with their 3rd dose of the pentavalent vaccine in the North Eastern States was 35%. By mid-2018 the proportion of immunized children in the North East was 49.05%, therefore successfully reaching the target (NNHS, 2018). Similarly, in Ondo and Nasarawa States, the target was 65% for 2020, by mid-2018, an average of 65.5% of children were immunized with their 3rd dose of the pentavalent vaccine across the 2 States (NNHS, 2018). This target was therefore successfully reached in all NSHIP-AF States even prior to the endline. The PDO target for the number of children 0-12 months immunized with pentavalent 3 vaccine per year in the 2 original states of Nasarawa and Ondo was 270,000 children. Data from the HMIS found that a total of 2,810,541 children were vaccinated in 2019, successfully meeting this target (HMIS, 2020). In the North East, this target rose to 500,000 infants vaccinated. This target was successfully met, with a total of 12,584,844 children receiving the 3 doses of Penta vaccine (HMIS, 2020). B. Proportion and number of pregnant women attended to by skilled birth attendance at delivery This indicator was not met. Page 77 of 84 The World Bank Nigeria States Health Investment Project (P120798) The target for births attended by skilled birth personnel in the North East was 35% for 2020. In 2018, this figure was 25.8% therefore not meeting the target, however this is due to lack of up to date data and limitations in the use of National Survey Data (NDHS, 2018). A higher target of 75% of births attended by skilled personnel was in place for Nasarawa and Ondo States for July 2020. This figure was at 71.7% in June 2018, not meeting the target (NDHS, 2018). This is due to lack of more recent data from the National Surveys. The number of births occurring in a health facility across all NSHIP-AF States reached 1,886,618 at the end of 2019 (DHIS, 2020). This is well above the baseline target set at 88,503 by mid-2020. For Nasarawa and Ondo, 481,976 deliveries occurred in health facilities, and 1,404,642 in the North Eastern States (DHIS, 2020). This exceeds the targets of 62,000 and 150,000 respectively. This target was met as we are using up to date national DHIS data. C. Structural quality of care measurements, such as availability of drugs, skilled personnel, basic equipment, and proper waste management This indicator was not met. For quality of care, the overall target was set at 61.0% for July 2020, however, available data from 2016 health facility survey only reached 41.66%, therefore not meeting the target. The average health facility quality of care was set at 72.0% for July 2020 for both Nasarawa and Ondo; and the North East States. While the baseline was 45% (National Health Survey, 2016) for Nasarawa and Ondo; it was 24.0% (National Health Survey, 2016) for the North East States. From the available recent survey (Health Facility Survey, 2016), the figure was 42.3% for Nasarawa and 41.2% which indicated that both sub-indicators were not met. The baseline scores were 24% in 2015 and rose to 42% in 2016 (Annual Health Facility Survey, 2016) for the North East States while it was 45% for Nasarawa and Ondo in 2014 (IE Report). Hence, more up to date data is needed to properly evaluate whether these targets have been met or not. D. Number of outpatients’ visits, and proportion of under-five children accessing health services This indicator was not met. To measure the proportion of children who were sick in the last month who used a government hospital or clinic, a proxy indicator of care seeking for children aged 0-59 months with fever was used (NDHS, 2018). The latest available data from 2018 found that the average in the North East was 72.6%, meeting the target for 2020 of 72%, whilst in Nasarawa and Ondo the average was 54.7%, below the target of 85% (NDHS, 2018). The number of outpatients per year treated in project facilities within the North East, including both children and adults, came to a total of 29,304,417 across the period, and well exceeded the target of 500,000 outpatients to be treated in the project States (DHIS, 2020). In Nasarawa and Ondo, the total was 8,162,025, again exceeding the target of 300,000 outpatients to be treated. The cumulative total of outpatients treated across all NSHIP-AF States over the period of 2016-2019 was therefore 37,466,442 (DHIS, 2020). E. An HNP core indicator about the number of people with access to essential health services This indicator was successfully met. Using facility attendance as a proxy for direct project beneficiaries, over the course of the project implementation from 2015-2019, there were 77,202,485 people benefitting from NSHIP/AF services which Page 78 of 84 The World Bank Nigeria States Health Investment Project (P120798) has already met the 2020 target of 1,400,000 beneficiaries. This figure includes both male and female attendees of all ages and of all services. The proportion of those who were female came to 62.7%% compared to the target of 60% for 2020 (estimated using male & female direct beneficiaries). The figure excludes August 2016 data from a facility (Rimin Zayam) from Bauchi with computation error which will be included once the error is corrected. Intermediate Indicators include 8 SUB PDOs: There are 8 indicators under the intermediate results under the project and 6 of them were met. The remaining 2 (mobile clinic and mental health counselling to pregnant women) that were not met were as a result of non-availability of data. ● The number of LGAs that implemented PBF by project closure is currently 88, well exceeding the target of 60 (HMIS, 2020). ● Furthermore, available data (project data) indicates that 100% of the health facilities implementing the project have functioning management committees with community representation, well above the target of 30%. ● The target for the proportion of pregnant women tested for HIV and who received test results was 50% for Nasarawa and Ondo States, this proportion reached 67.7% in December 2019 meaning the target was met (DHIS2, 2020). ● The target for the proportion of pregnant women tested for HIV and who received test results was 50% for the North East States the proportion reached 55.8% as at December 2019 meaning the target was met (DHIS2, 2020). ● An additional intermediate indicator was introduced to measure the proportion of mothers aged 15 to 19 years of age who delivered in the last two years and received skilled birth attendance. The target was to reach 35% by June 2020 of young mothers delivering with a skilled health personnel. By June 2018 this number was 38.31% - meaning the target was met (NNHS, 2019). ● Similarly, the number of children treated for severe acute or chronic malnutrition per year in the project area successfully met its target of 25,000 treated quarterly, reaching 596,732 children (DHIS, 2019). The targets that were not met due to non-availability of data: ● An intermediate indicator was introduced to measure the number of mobile clinics conducted per year in the project area specifically for the States of Borno, Taraba and Yobe. However, data around mobile clinics was limited due to delays in approval and implementation. Initial findings in Taraba found that the estimated number of mobile clinics conducted is twice per week. If this projection continues in Borno and Yobe as well, the average number of mobile clinics conducted in the project area across 3 months would be 208. ● Currently data for the number of women receiving counselling on gender-based violence or mental health services from a trained provider is not yet available, due to delays in implementation of psycho-social support. Page 79 of 84 The World Bank Nigeria States Health Investment Project (P120798) 2.2 Impact of Project Implementation on Policies and Interventions The implementation of output-based financing with NSHIP has had remarkable and far-reaching positive effects in the Nigerian health system, particularly the PHC system. Consequently, over the years, NSHIP has become a reference point for the implementation of many other projects and policies at national and subnational levels. ● NSHIP Approach Adoption by States: ○ The impact of NSHIP resonates sub nationally in terms of PHC funding. The recent commitment of 605 million naira by Gombe state demonstrated the confidence of the state government in the reform brought about by NSHIP implementation. The Fund will be used to scale up PBF in 5 LGAs within Gombe state. ○ The Government of Nasarawa State also piloted DFF components from the NSHIP in those health facilities that were not included in the NSHIP implementation. Following this, an institution was set up to oversee fund usage in this regard. ● SOML Project: SOML is the first health programme in Nigeria that adopted one of the crucial elements of NSHIP/AF, focusing on institutional strengthening. As used in NSHIP, SOML adopted Disbursement Linked Indicators (DLIs) to incentivize States to meet set targets. In order to avoid duplication of incentives, the NSHIP DLI component was discontinued, to be merged with the country wide SOML DLIs (P4R-SOML PAD, 2015). ● National Health Act 2014: The Basic Health Care Provision Fund (BHCPF) component of the National Health Act adopted the Decentralized Facility Financing (DFF) component from NSHIP (NHA, 2014). The BHCPF is also a nationwide programme intervention in the health sector. ● Oversight/Supervisory Tools: Similarly, the tremendous improvement in the quality of care recorded at PHCs through the use of the NSHIP Quality Supervisory Checklist (QSC), led to the adoption of the document during the revision of the Integrated Supportive Supervision (ISS) checklist in Nigeria. Presently, the country’s ISS checklist is heavily influenced by the content of NSHIP QSQC in the country’s effort to improve the quality of care across all PHCs in the country. ● Influence on National Documents: ○ Essential Medicine Manual: In the same light, the success of NSHIP as regards to the management of essential medicines, led to the development of the Essential Medicine Manual, which has been adopted across all PHCs in Nigeria. ○ Financial Management Manual: The financial management at the health facilities is a key success factor for NSHIP, with key community involvement in the form of Ward Development Committees (WDCs). Consequently, a similar approach to financial management was adopted for the BHCPF implementation. ○ Health Care Waste Management (HCWM) Manual: The effectiveness of health care waste management within NSHIP - which is in line with global waste practice - also made the adoption of the HCWM manual by all PHCs across the country possible. Section Three: Assessment of factors and events that affected Project Performance and Outcome This section outlines the key factors and events pertaining to the Bank, borrower, other partners, and the external environment during preparation and implementation, that affected performance and outcomes. Page 80 of 84 The World Bank Nigeria States Health Investment Project (P120798) 3.1 Factors that Positively Impact on the Performance and Outcome of the Project 3.1.2 Positive Impacts: health systems strengthening ● Strengthening of Management Systems; improvements in management processes, including financial transactions and higher transparency, as well as use of data and greater autonomy in decision making for example hiring staff, procurement of drugs and equipment. It included a better understanding of the context and sense of ownership of the facilities. ● Reinforced Supportive Supervision; the project recorded significantly improved integrated supportive supervision both in frequency as well as in its nature in the activities of health facilities. ● Improvements in Staff Knowledge, Motivation and Engagement; NSHIP bonus payments improved the financial situation among health workers both that of contract and also the LGA staff deployed to the NSHIP facilities. More so, improvements in the knowledge and service delivery skills of staff. ● Improvement in Motivation due to Non-Financial Incentives; NSHIP focused on the working environment, improved infrastructure, and enhanced motivation premised on skilled acquisition. ● Increase in Punctuality, Discipline and Commitment; the use of individual evaluation frameworks, which took into account health worker punctuality and discipline in order to calculate their bonus payments at the end of each quarter (although this does not directly speak of DFF facilities. This led to improved commitment of health workers and strengthened team engagement. ● Better Use of Data for Decision-making High Degree of Fidelity of NSHIP Implementation in Study Sites; the contractual procedures mandated by NSHIP’s project design, staff receiving trainings for learning about PBF and DFF principles, and creating new institutional arrangements, in the form of management committees, as well as reinvigorating existing institutional structures and verification process. 3.2 Negative Impacts: Challenges and Gaps that Affected the Outcome of Project Whilst NSHIP has brought about many positive changes at the facility and community levels since its inception, its implementation has also exposed some gaps in the implementation process, as well as recurrent challenges deterring it from achieving its full potential. This includes problems faced by communities in accessing health services, both in terms of geographic location and financial bearings, perceptions of poor attitudes of health providers and socio-cultural norms preventing the uptake of family planning services. Health workers also expressed concerns about excessive workload especially in the presence of systemic shortages in manpower. 3.2.1 Implementation Gaps: Structures, Manpower and Processes ● Communal Clashes and Security Challenges: This is one of the greatest challenges of the project implementation. Security issues in Borno, Yobe, Adamawa state with respect to Boko Haram as well as kidnapping, banditry and communal clashes in other States created additional hurdles in effective implementation of the project. In some cases, the project could not be scaled to these areas while in other cases, focal points were relocated to another facility. ● Poor and Inadequate Structures and Manpower: The implementation approach of NSHIP required adequate structure and sufficient manpower as health facilities are expected to produce output that Page 81 of 84 The World Bank Nigeria States Health Investment Project (P120798) will be incentivised. However, it was discovered that many health facilities do not even have adequate rooms as well as dearth of frontline health workers. These inhibited LGAs taking full advantage of the intervention. In addition, many health facilities had to resort to hiring additional personnel (technical & non-technical) thereby negatively impacting the project implementation. ● Poorly Functional Indigent Committees: The fact that there is no existing database of indigent people in Nigeria made the effectiveness of this component very difficult. In most cases, traditional and cultural norms also prevented people from voluntarily indicating that they are indigent and also due to fear of stigmatisation. ● Poor Programme Knowledge among Junior Cadres of Health Providers: In general, the lower cadres of health providers appeared to be less aware of various procedures and mechanisms of NSHIP. This is partly due to information not being effectively passed down or shared. ● Perception of Excessive Workload in Absence of Adequate Staff: Some staff felt that project implementation severely increased their workload as result of the increased utilization of health facilities, as well as a marked increase in documentation requirements brought about by the project. In addition, since the project also did not recognise Junior Cadres below Community Health Extension Workers (CHEW) as health care providers; schedules that were usually handled by these categories of personnel were transferred to Senior Cadre. ● Geographical Access to Health Facilities: Access to health care and facilities remains a critical challenge in some of the project areas. This is especially so in rural areas as a result of bad roads and transportation challenges, a situation which is further intensified during the rainy season. The project is not permitted to construct physical structures and hence, where facilities do not exist, project intervention in such areas becomes unattainable. ● Financial Access: Despite significant lowering of costs of services, some communities reported that the cost of services at NSHIP facilities remained expensive for them thereby deterring their use. ● Social and cultural norms with regards to family planning: across States, in particular Adamawa and Nasarawa, there remained persistent preference for traditional methods of family planning due to cost, desire to have more children, as well as social and cultural beliefs. In particular, those who were of the opinion that family planning practices were attempts at hindering “God’s work”, among others. Section Four: Borrower Performance and Risk to Development Outcomes This section outlines the country’s preparedness before, during and after the project. It provides details on the evaluation of the borrower’s performance and implementation of the project with emphasis on what worked well, and lessons learned for future programming, as well as a description of proposed arrangements for future operations of the project. To evaluate the country’s preparedness prior to the project, consultations between the FGN and World Bank were held to determine if the country-setting was suitable for project implementation, and whether the principles of RBF aligned with national priorities. Poor health indicators as well as systemic bottlenecks within the Nigerian health system at the time made PBF an appropriate model for implementation. Participating States were chosen on the basis that they had a strong governance capacity and expressed commitment and willingness to use RBF approaches. Selection was also made on the basis of health needs, as well as ensuring geo-political representation and filling gaps in donor support. As the States represented Page 82 of 84 The World Bank Nigeria States Health Investment Project (P120798) a cross-section of Nigerian states, this ensured that the experience under the project would be able to inform planning and future investments by the Government. The FGN and World Bank decided to set up the Project Implementation Units (PIUs) both at State and National government agencies. These agencies include the Federal Ministry of Health (FMoH), National Primary Health Care Development Agency (NPHCDA) and State Primary Health Care Development Agencies/Boards (SPHCDAs/Bs. By involving both State and National government agencies in PBF implementation, this helped to end fragmentation in the health sector and strengthen the government role in the participating States. The implementation of PBF in Nigerian therefore demonstrated the capacity of National and State actors in adopting innovative health systems financing models and taking on new roles in PBF design and implementation. National PBF training was conducted in Nigeria to prepare the national team technically for the preparation of the project. Three independent consultants were brought in from Rwanda to aid with the commencement of the verification process. The Nigerian team were also actively engaged in the creation of important documents such as the PIM and User Manual. At the State level, key actors were assigned including the Project Coordinator, the Project Accountant, and the Procurement Officer. Oxford Policy Management (OPM) was contracted to provide technical support to the country during NSHIP. With the introduction of AF, in-country technical staff with support from the World Bank modified the OPM role into indigenous CMVA and IVAs, and this technical support was then fully provided by the country. Nigeria also initiated an internship programme which conducted multiple training of verifiers, as well as initiating its own TA fellowship, from which 13 in-country TAs are now certified. From all of these documented cumulative experiences, the government ICR is also being drafted by in-country personnel. Section Five: Recommendations Over the course of project implementation, NSHIP has recorded a number of successes with PBF in Nigeria, notably structural changes, a more motivated health workforce, and improved patient flow. It has demonstrated how PBF can serve as an innovative strategy to increase the impact of investments in health and improve efficiency. As well as the improvements to both quality and quantity of care offered, the application of NSHIP has offered valuable lessons throughout the course of project implementation, used both internally to strengthen the design and application of NSHIP and influence national policy. A number of recommendations from the NSHIP project that can be taken forward to future projects are outlined in more detail below: 1. Sustainability: Sustainability plan should be clearly stated in such a way that transition is clear from inception. This will ease the sustainability challenges NSHIP faced despite its significant impacts at the implementing States. 2. Alignment of PDO Indicators with Scope of Implementation: The PDO indicators should be systematically designed to reflect the scope of project implementation. For instance, there are RBF and DFF in Adamawa, Nasarawa and Ondo States with different design and implementation approaches. PDOs should reflect each implementation approach. 3. Project Evaluations: Great emphasis should be placed on the evaluation designs; baseline, midline and endline, as they are crucial in project performance evaluation. In addition, irrespective of intervention approach, baseline should always be conducted in order to ascertain the likely impact attributable to the project (AF scenario). Following project closure and data availability challenges, an endline survey should be conducted to evaluate the full impact of the project in Nigeria. Page 83 of 84 The World Bank Nigeria States Health Investment Project (P120798) 4. Use of National Surveys and Project Closure Timing: Understanding of the nature of national surveys in implementing countries is essential if they will be used to evaluate project performance at closure. In addition, the institutions conducting national surveys could be requested to disaggregate data/analysis that could be used to evaluate project performance. More than 50% of the AF States are currently implementing the project in 50% of the LGAs while survey data covers the entire State. 5. Separation of functions: The project should always ensure full compliance of separation of functions from inception in order to ensure smooth implementation. For instance, some secondary health facilities are managed by other bodies like Hospital Management Boards (HMBs), who manage pharmacies, laboratories and records. This produced great hindrances as all sources of revenue are managed outside the secondary health facilities. 6. Maintenance of Supervisory and managerial roles: The managerial autonomy created by NSHIP has contributed to strengthening the overall health system, in particular PHC. The inbuilt coaching and mentoring at every level of project implementation, as well as internal training and guidance within States has led to improved capacity at every level. 7. Maintenance of Strong community participation: by involving the community at every stage of project implementation, this has allowed for more insightful improvements within the project, as well as building a strong community voice. The introduction of programmes such as the Grievance Redress Mechanism has created channels for patients and NSHIP staff workers to file any complaints in a systematic way. 8. Specific response to crises: by tailoring the project and services offered towards the context of the insurgency in the North East, this has allowed for more effective provision of services. This includes the introduction of mobile clinics and psychosocial support services. Team Technical/Managerial Oversight Haj Binta Ismail (NPC, NPHCDA); Mr. Bolaji Oladejo (PI, NPHCDA) and Pharm. Mohammed A. Mohammed (DNPC, NPHCDA) FMoH Mr. Osakwe (Chairman), Mr. Ahmed Ibrahim, Nwokedi N. Nwodo, Dr. Ayoola Olusola, Mr. Akin Fayinminu and Nkiruka Onwuchekwa NPHCDA Muhammad Mashin (Technical Lead), Dr. Ismail N. Salihu, Mercy Lewis, Hyeladzira Garnvwa-Pam, Dr. Michael Ajuluchuku, Adeleye Adekusibe, Sirajo Ibrahim and Sarah Martin. Page 84 of 84