Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) Report Number: ICRR0022675 1. Project Data Project ID Project Name P120798 NG-States Health Prog. Invest. (FY12) Country Practice Area(Lead) Nigeria Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-50940,IDA-58510,TF-13432,TF- 30-Jun-2018 252,165,871.82 A2591 Bank Approval Date Closing Date (Actual) 12-Apr-2012 31-Oct-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 150,000,000.00 41,700,000.00 Revised Commitment 266,051,476.24 31,163,911.27 Actual 252,165,871.82 31,163,911.27 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado P149936_TBL Project ID Project Name P149936 NSHIP Additional Financing ( P149936 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0 Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) Bank Approval Date Closing Date (Actual) 20-Mar-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The objectives of the project were to increase the delivery and use of high impact maternal and child health interventions and improve quality of care at selected health facilities in the Participating States (Financing Agreement, April 16, 2013, p. 5). The statements of objectives in the PAD and ICR were identical. The three states that were included in the original design were Adamawa (North-Eastern Region), Nasarawa (North-Central Region), and Ondo (South-Western Region). A Level-1 project restructuring with additional financing on June 30, 2016 revised the objectives to include all six North Eastern (NE) states: Bauchi, Borno, Gombe, Taraba, and Yobe, in addition to the originally-included NE state of Adamawa. Revised objectives were stated as follows: to increase the delivery and use of high impact maternal and child health interventions and improve quality of care available to the people in Nasarawa and Ondo and all the states in the NE. The PDO term of "selected facilities" was related to involvement of facilities in the project’s performance- based financing. The majority of primary care facilities and all general hospitals (first referral) were included (see Section 4 for the ratios used for contracting rural and urban facilities by a given catchment population number). With project expansion in 2016, the revised PDO statement modified the objectives from "increasing delivery of interventions and improving the quality of care at selected health facilities" to "increasing delivery of interventions and improving the quality of care available to the people" in the participating states. The revised PDO statement highlighted the expansion of the project in its geographic scope and did not entail a material change in terms of substance or outcome assessment. Also, outcome targets were revised upward, and both original and revised targets were largely met and/or exceeded. Therefore, this ICR Review did not apply a split evaluation methodology, as it was considered moot in this case, with no value added. Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 30-Jun-2016 c. Will a split evaluation be undertaken? No d. Components 1. Results Based Financing (Appraisal: US$122.4 million; AF2014: US$1.7 million; AF 2016: US$85 million; Total: US$209 million; Actual in local currency: NGN43.86 billion [see note below]). a. Strengthening Service Delivery i. Performance-Based Health Facility (PBF) Financing. Carrying out of a program of specific activities to deliver Packages of Health Services by PBF Health Facilities, with a focus on maternal and child health, through the provision of quarterly quantity- and quality-adjusted output-based grants to said facilities. ii. Decentralized Health Facility Financing (DFF). Carrying out of a program of specific activities to deliver Packages of Health Services by DFF Health Facilities, with a focus on maternal and child health, through the provision of quarterly output-based grants to said facilities. (The ICR, p. 14, explained that the design of PBF and DFF were almost the same, with some variations. In selected Local Government Areas (LGAs), the project supported one health center in each ward, and one general hospital in each LGA with DFF for minor facility improvements and/or operational costs. By design, DFF facilities received 50 percent of the funding that PBF facilities received, as PBF facilities used half of their income for incentive payments, while DFF facilities were not supposed to use income for health worker incentive payments.) iii. Local Government Area Primary Health Care (PHC) Department Performance-Based Financing. Carrying out of a program of specific activities by LGA PHC Departments to support, supervise, and verify the delivery of Packages of Health Services by Participating Health Facilities through the provision of quarterly quantity- and quality-adjusted output-based grants to said LGA PHC Departments. b. Strengthening Institutional Performance Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) i. Disbursement-Linked Indicators (DLIs)-Based LGA Financing. Carrying out of a program of specific health administration activities by the Participating LGAs including, inter alia, preparation, execution, and publication of transparent LGA health budgets in accordance with the updated charts of accounts for Participating LGAs, carrying out of enhanced systematic supervision of services delivered by health facilities, preparation and analysis of improved quarterly health management information system reports on key health indicators, development and implementation of appropriate health facility staffing norms and human resource policies, and implementation of improved procurement arrangements for acquiring quality assured drugs; all through the provision of annual output-based grants to Participating LGAs. ii. DLI-Based State Financing. Carrying out of a program of specific health administration activities by the Participating States, including preparation, execution, and publication of transparent state health budgets in accordance with updated charts of accounts for the Participating States, systematic verification of performance by the LGAs and health facilities and administration of related result-based coordination and administration mechanisms, and other essential institutional strengthening measures supporting the implementation and sustainability of results-based financing mechanisms; all through the provision of: (a) annual output-based sub-financings; and (b) bi-annual output-based sub-financings to the Participating States. 2. Technical Assistance (Appraisal: US$34 million; AF 2016: US$23 million; Total: US$ 57 million; Actual in local currency: NGN23.12 billion [see note below]). i. Project Implementation at the State Level. Carrying out of specific activities to build the capacity at the level of the State Ministry of Health, State Primary Health Care Development Agency, and other state agencies involved in the implementation of the project to carry out the day-to-day coordination, management, audit, monitoring, and evaluation of project activities within the Participating States, all through the provision of technical advisory services, training and workshops, goods, small works, and operating costs to the Participating States. ii. Implementation of Results-Based Approaches. Carrying out of specific activities to: (i) build the capacity of federal and state agencies, LGAs, facilities, and civil society stakeholders to implement and sustain results-based financing mechanisms supported under the project through development and implementation of result-based financing processes, protocols, and systems, and learning, knowledge sharing, communication, and community engagement activities; and (ii) assist the National Primary Health Care Development Agency to carry out its day-to-day functions under the project; all through the provision of technical advisory services, training and workshops, goods, small works, and operating costs. Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) iii. Overall Project Management, Monitoring and Evaluation. Carrying out of specific activities to: (i) establish and implement a monitoring and evaluation system to improve routine data collection and utilization, ensure accurate reporting, and evaluate the impact of project interventions in improving health service delivery; (ii) conduct household and health facility surveys for the purposes of the project; and (iii) assist the Federal Ministry of Health to carry out the overall coordination, management, and audit of the project at the federal level; all through the provision of technical advisory services, training and workshops, goods, small works, and operating costs to the Federal Ministry of Health, National Bureau of Statistics, and the National Population Commission for undertaking periodic household and facility surveys. Note on actual component costs: The ICR (p. 21) reported difficulties in obtaining actual component costs in USD, as expenditures were recorded in NGN and by category. The ICR stated that the mission could not convert the total amount by a given exchange rate as the cumulative total was spread over a period of time with different exchange rates. Component Revisions: The 2014 restructuring added interventions addressing demand-side barriers as detailed below (Section 2e, restructurings). The 2016 restructuring introduced the following changes:  added a third component on Partnerships to Strengthen Service Delivery at an estimated cost of US$18 million, with an actual cost of NGN176.55 million (see above note on actual costs by component). This third component aimed at supporting partnerships with non-state actors to ensure flexibility to respond to changing circumstances in the NE. Interventions included partnerships to provide mobile health services in underserved communities; re-establishing health services through non-state actors where health facilities were non-functional and when security was deemed acceptable; and strengthening LGA management;  reinforced services under PBF by including psycho-social support and mental health, nutrition; and extensive community outreach;  used independent Contract Management and Verification Agencies in support of mobile clinics, strengthening LGA management, and re-establishing health services in LGAs where services had been destroyed; and  alleviated demand-side constraints through selective implementation of free care for children under 5 and pregnant women. Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) The third restructuring in 2019 dropped the two sub-components related to LGA DLIs and State DLIs. Payments against all LGA and State DLIs were stopped, as the indicators had been achieved prior to 2019 (ICR, p. 20). Selection of participating states As stated in Section 2a, the three states of Nasarawa, Ondo, and Adamawa were selected after several rounds of consultations with partners and in accordance with the broad principles of the Country Partnership Strategy (CPS) and the following criteria: (i) strong commitment; (ii) prioritization of states with greater health needs; and (iii) working with states expressing a willingness to use RBF approaches (PAD, p. 5). Selected states represented a cross section of Nigerian states, and the experience gained under the project was expected to be helpful in planning future similar investments. Before project preparation, the governors of the three states signed a policy note endorsing the project for their individual states and initiated reforms that would provide health facility autonomy and promote community participation. State governments also set up RBF Technical Support Units that were staffed by government officials who were trained in PBF implementation. PBF pre-pilot interventions commenced in three LGAs, one per state, in December 2011, and were financially supported by state governments. In 2016, and as noted in Section 2a, five NE states with vulnerable populations were also included in the project: Bauchi, Borno, Gombe, Taraba, and Yobe. Therefore, in addition to Adamawa that was already included at the outset, all six NE states were included in project operations. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost and financing. At appraisal, project costs were estimated at US$170 million (consisting of an IDA Credit of US$150 million and a grant of US$20 million from the Health Results Innovations Trust Fund or HRITF). An additional HRITF Grant of $1.7 million was provided in March 2014 to support community demand. In June 2016, additional financing (AF) was provided to support project expansion in the NE, consisting of an additional IDA Credit of US$125 million and a Global Financing Facility (GFF) Grant of US$20 million. The total actual cost recorded in the portal was US$283.3 million. On March 12, 2020, the project had a partial cancellation of US$25 million, explained by difficult implementation in insecure areas (see below under restructurings), and the remaining funds at project closing were allocated to support the Multi Sectoral Crisis Recovery Project for North Eastern Nigeria. Dates. Appraisal was finalized on March 19, 2012 and the project was approved on April 12, 2012. It became effective on August 20, 2013. A Mid-Term Review was carried out on November 27, 2017. The project closed on October 31, 2020, two years and four months beyond its original closing date of June 30, 2018. Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) The project underwent five restructurings: 1. March 15, 2014: Additional HRITF grant of US$1.7 million was provided to support community demand. It introduced interventions to address demand-side barriers to accessing health care in targeted poor communities and households. Interventions included conditional cash transfers, transport vouchers, and performance-based incentives to traditional birth attendants for referral services. 2. June 30, 2016: Additional IDA Credit of US$125 and GFF Grant of US$20 million were provided to include all the NE states under a PDO revision, pursuant to the Recovery and Peace Building Assessment that was undertaken by the government in collaboration with the World Bank Group, European Union, and United Nations in response to the Boko Haram insurgency that affected NE states (ICR, p. 18). The closing date was extended by two years. 3. August 8, 2019: The third restructuring introduced changes to the verification methodology and revised the results framework. 4. April 12, 2020: The fourth restructuring reallocated funds between disbursement categories and cancelled US$25 million equivalent, as some activities could not be completed in view of limited implementation ability due to insecurity in several geographic areas (ICR, p. 20). 5. June 22, 2020: The fifth restructuring extended the closing date by four additional months to facilitate service delivery that was affected by the COVID-19 pandemic. 3. Relevance of Objectives Rationale The project was responsive to the needs of the country as it aimed at enhancing the effective use of public resources to deliver essential health services to underprivileged communities. Primary health care facilities often lacked skilled practitioners, even when they were overstaffed with other personnel categories, and facilities did not have basic pharmaceuticals in stock. There was a shortage of supervisory staff and supervision budgets, with no clearly defined responsibilities, resulting in little monitoring of service delivery. These factors contributed to poor quality of care (ICR, p. 7) that was also linked to a larger sectoral issue whereby overall health expenditures focused on tertiary care and specialized services (ICR, p. 8). More than 50 percent of households were dissatisfied with the services in primary public facilities and used them infrequently. Hence, there was low coverage of effective interventions that can prevent many maternal and child deaths (ICR, p. 8). It is also understood that there were inter-regional and inter-state disparities in health outcomes, and the project focused on disadvantaged states. The objectives were and remained aligned with CPS 2014-2017 that was extended until the advent of the Country Partnership Framework for the Period FY21-FY25, approved on November 16, 2020, after project Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) closing. The CPS focused on improving governance and promoting human development. One of its aims was to increase access to and utilization of quality health services. The project contributed to the progress made in health-related CPS indicators in the three states, such as attended deliveries and share of immunized children. Per Project Paper of May 24, 2016 (Report No: PAD1741, p. 4), the AF was aligned with the second CPS cluster aiming to improve the “effectiveness and efficiency of social service delivery at State level for greater social inclusion.” By addressing the challenges of fragility, conflict, and violence, the AF contributed to Nigeria’s efforts in dealing with the shocks associated with the insurgency in the NE. Both the original and revised objectives were aligned with the Government’s NE engagement Strategy and other World Bank initiatives to address developmental issues of the NE. As noted in Section 2, the Federal Government of Nigeria, in collaboration with the EU, WBG, and UN, carried out the Recovery and Peace Building Assessment that informed the design of the North East Emergency Transition and Stabilization Program targeting short- and medium-term recovery, stabilization, and peace building needs. Five human development operations and an agriculture project were included in the stabilization program and were structured under a coordinated umbrella that included interventions and approaches that could rapidly be scaled-up or re-engineered to benefit vulnerable NE populations. The objectives were also aligned with the 2019 Nigeria Systematic Country Diagnostic that identified improving access to quality health services delivery, and addressing issues of financing, accountability, and architecture of development assistance, as options to address the challenges facing Nigeria's health care sector. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase the delivery and use of high impact maternal and child health interventions in Nasarawa and Ondo and all the States in the North East. Rationale Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) Theory of change The theory of change was based on the premise that low coverage of effective and low-cost health interventions, lack of incentives for achieving results, and low accountability continue to deny millions of Nigerians access to safe and quality health care services (ICR, p. 23). It was reasonably expected that the following mutually reinforcing activities would bring out the following corresponding outputs:  results-based financing with periodic verification of results and targeted technical assistance would bring out an increase in the number of facilities with operational funds, minimal stockout of essential services, and increased worker motivation;  reducing demand-side barriers through targeted mass media communications, transport vouchers, and performance-based incentives for attended and referred deliveries would bring out an increase in community knowledge on where to access services, and in referred deliveries (these activities were added in the first restructuring of March 2014, as illustrated in Section 2e);  technical assistance to improve supportive supervision; and  technical assistance to improve accountability with disclosed information, financial reports, and utilization of charts of accounts. The above activities and outputs would collectively contribute to intermediate results that were expected to show increases in people receiving essential health services. In turn, these results would plausibly contribute to increased delivery of high-impact maternal and child health interventions in participating states. Outputs and Intermediate results  In terms of coverage and facility selection, public health centers were included in the PBF approach by using a ratio of one contracted health center per 7,000 population in rural areas, and one contracted health center per 12,000 population in urban areas, while all general hospitals were contracted (Task Team clarification, August 19, 2021). Practically, this meant that the majority of primary care facilities and all first referral level hospitals were included in project areas. In some cases, private health facilities were also selected as PBF/DFF facilities in the absence of government-owned facilities that met minimum criteria for inclusion. In urban settings where population density was high, private health facilities were contracted to complement the efforts of public sector providers. These private facilities benefitted from capacity building efforts targeting health care providers. At the end of the project, and in addition to public facilities, 77 private health facilities and 3 faith-based health providers were also engaged by the project (ICR, p. 39).  In the NE, the proportion of pregnant women tested for HIV and who received test results reached 58.7%, exceeding the target of 50%.  In Ondo and Nasarawa, the proportion of pregnant women tested for HIV and who received test results reached 63.65%, exceeding 50%. Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798)  The proportion of mothers aged 15 to 19 years of age who delivered in the last two years and who received skilled birth attendance reached 47.2%, exceeding the target of 35%.  The number of women and children who received basic nutritional services reached 19.6 million, exceeding the target of 3.7 million.  The number of immunized children reached 2.7 million, exceeding the target of 2.5 million.  The number of deliveries attended by skilled health personnel reached 2.1 million, exceeding the target of 0.7 million.  The number of children treated for severe acute or chronic malnutrition per year in the project area was 304,343 children, exceeding the target of 25,000 children.  The number of LGA with PBF reached 113, exceeding the original target of 53 and the revised target of 80 LGAs. Outcomes Outcomes for the delivery and use of maternal and child health interventions were largely reflected by increased proportions of children's vaccinations and attended deliveries, supported by the above intermediate results along the results chain pathway. The ICR reported that an impact evaluation was undertaken in 2017 at mid-term and compared the effectiveness of the project interventions with control groups in non-project states. The ICR (p.33) reported, with supporting data, that the results of the impact evaluation demonstrated a remarkable increase in the uptake of services, and that the difference was statistically significant when compared to control states. Children's immunization outcomes consisted of the following:  In the NE (Adamawa, Bauchi, Borno, Gombe, Taraba, and Yobe), the percentage of 12-23 months old children vaccinated with Penta3 increased from a baseline of 27.4% in 2010 to 68.6% in 2020, exceeding both the original target of 35% and the revised target of 49%.  In Nasarawa, the percentage of 12-23 months old children vaccinated with Penta3 increased from a baseline of 39.4 in 2010 to 82.9% in 2020, exceeding the target of 65%.  In Ondo, the percentage of 12-23 months old children vaccinated with Penta3 increased from a baseline of 67.3% in 2010 to 86.6% in 2020, exceeding the target of 75%. Outcomes for attended deliveries/births consisted of the following: Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798)  In the NE (Adamawa, Bauchi, Borno, Gombe, Taraba, and Yobe), the proportion of births attended by skilled health personnel increased from a baseline of 22.2% in 2014 to 69% in 2020, exceeding the original target of 35%, but short of the revised target of 75%.  In Nasarawa and Ondo, the proportion of births attended by skilled health personnel increased from a baseline of 56.65% in 2014 to 73.6% in 2020, moderately short of the target of 75%. The proportion of children sick in the last month who used a government hospital or clinic in the NE increased from a baseline of 61.7% in 2016 to 69.1% in 2020, short of the target of 72%. The number of outpatient visits per year (children and adults) in the NE increased from a baseline of 170,000 visits in 2014 to 20.9 million in 2020, exceeding the target of 0.5 million. The number of direct project beneficiaries was 22.4 million in 2020, exceeding the original target of 1.4 million and the revised target of 12 million beneficiaries, out of whom 63% were female beneficiaries, exceeding the target of 60% for females. (The Task Team clarified on August 19, 2021, that this comprehensive indicator may have included some double counting, i.e., counting the number of encounters for the same beneficiary in some instances). Rating Substantial OBJECTIVE 2 Objective Improve the quality of care available to the people in Nasarawa and Ondo and all the States in the North East. Rationale Theory of Change The theory of change built on that of Objective 1 by integrating quality measures to determine PBF payments to facilities. Linking PBF payments directly to quantitative performance and qualitative performance based on well-defined and institutionalized checklists would plausibly improve the quality of care available to the people in participating states. Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) Outputs and intermediate results Two quality checklists were developed and applied by the project in its operations: one at the health center level with the basic health services package, and the other for the first-level referral hospitals with the complementary health services package. These quality checklists were more elaborate and more granular than the ones used in the complementary annual health facility surveys that were limited to five components [(i) Content of Care Quality weighed at 30 percent; (ii) Drug Availability weighed at 20 percent, (iii) Readiness to Deliver Services weighed at 15 percent; (iv) Quality of Supervision weighed at 20 percent; and (v) Financial Management weighed at 15%]. The comprehensive quality checklists under PBF had lower targets than the annual surveys, and included the following: i. Quality Checklists for Health Centre Services with numerous indicators related to each of the following areas and their corresponding weights: General Management 24 points; Business Plan 18 points; Finance 23 points; Hygiene 57 points; Out-Patient Department 100 points; Family Planning 17 points; Laboratory 16 points; Inpatient Wards 7.5 points; Essential Drug Management 20 points; Tracer Drugs 22.5 points; Maternity 25 points; Immunizations 19 points; Ante-Natal Care 11 points; and HIV/TB 8 points. ii. Quality Checklists for General Hospital Services with numerous indicators related to each of the following areas and their corresponding weights: General Management 36 points; Business Plan 18 points; Finance 46 points; Hygiene and Medical Waste Disposal 57 points; Out-Patient Department 102 points; Family Planning 28 points; Laboratory 14.5 points; Inpatient Wards 140 points; Essential Drug Management 20 points; Tracer Drugs 32.5 points; Maternity 25 points; Ante-Natal Care 6.5 points; HIV/TB 8 points; and Surgery 52 points. The proportion of health facilities in the project area with functioning management committees and having community representation reached 100%, exceeding the target of 30%. The proportion of on-duty technical staff who were present at the health facility on the day of the survey was 63.6%, short of the target of 84%. The proportion of primary health facilities having essential medicines and commodities in stock reached 53%, exceeding the target of 35%. Outcomes Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) The PBF database showed that the Average Health Facility Quality of Care Score increased from a baseline of 41.9% in 2014 to 67% in 2020, exceeding the target of 61%. The annual health facility surveys that were limited to only five components (see above under outputs) showed the following: the Average Health Facility Score - Structural Quality of Care (average of Adamawa, Bauchi, Borno, Gombe, Taraba, and Yobe) increased from a baseline of 41% to 61.6%, short of the target of 72%, and for Nasarawa and Ondo, it showed that the score increased from a baseline of 45% to 60.2%, short of the target of 72%. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project almost fully achieved its two objectives to increase the delivery and use of high impact maternal and child health interventions, and to improve the quality of care available to the people in participating states. The impact evaluation showed that comparisons with non-PBF control groups in other states had statistically significant differences, reflecting positively on attribution aspects. The aggregated level of achievement is consistent with a substantial efficacy rating. Overall Efficacy Rating Substantial 5. Efficiency The PAD’s economic and financial analysis focused on the economic rationale for interventions using results- based financing. Based on global experience, it argued that the service packages of high impact maternal and child health interventions provided by the project were known to be cost-effective and accessible to the community. The ICR’s economic analysis monetized major benefits and costs associated with the project and reported on three measures: benefit to cost ratio, net present value, and internal rate of return. The analysis estimated a benefit-cost ratio at 34.9. It estimated that the investment generated economic benefits with a net present value Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) of US$7.75 billion, using a discount rate of 3 percent. The internal rate of return was estimated at 18 percent. A sensitivity analysis was undertaken using a discount rate of 5 percent, indicating that the project remained economically viable. The ICR’s analysis also argued that, if the result was adjusted for improvement in quality, the benefit to cost ratio, net present value, and internal rate of return would have been higher. It concluded that the project was economically beneficial. On the other hand, there were significant shortcomings in the efficiency of implementation, with extended delays for key activities and cost overruns. Effectiveness was delayed by one year pending Nigeria’s Assembly approval, and for the AF, it took 18 months for some states to sign the subsidiary loan agreement. Funding for one state was put on hold for one year because of improper use of project funds. The project underwent cancellation of funds due to the cumulative effects of implementation delays and limited implementation ability in some states resulting from the insecurity environment and the understandable slowing down of activities caused by the COVID-19 pandemic, including lockdowns and mobility restrictions. Some states experienced cost overruns, and the project had to reallocate funds from low-spending states to other states (ICR, p. 36). The ICR (p. 42) noted that adequate stakeholder engagement was limited to the preparation stage only. It noted that, during implementation, key government officers, stakeholders, and decision makers at the state level knew little about the project approach, and that this limited stakeholder engagement during the larger part of the implementation phase, thus affecting efforts in promoting state funding. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 70.00 ICR Estimate  18.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated high, as the objectives remained fully aligned at project closing with Bank and country strategies. Efficacy is rated substantial, as the objectives were almost fully achieved. Efficiency is rated modest because of significant shortcomings in the efficiency of implementation. Therefore, the overall outcome Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) is rated moderately satisfactory, indicative of moderate shortcomings in the project’s overall preparation, implementation, and achievement. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Main areas of risks to development outcomes are related to macro-economic challenges, the COVID-19 pandemic, insecurity, and sustenance of performance-based programming. The COVID-19 pandemic affected the uptake of essential services and further weakened an already fragile healthcare system. If the pandemic impact continues, it can erode gains made by the project. Several geographic areas suffer from insecurity, notably in Adamawa, Yobe, and Borno, where a number of facilities were destroyed or abandoned. Nigeria is currently experiencing significant economic challenges. As of December 2020, over 400 ministerial departments and agencies owed salaries to staff. The ICR noted that the federal government is struggling to support the states (ICR, p. 50). With limited resources at the disposal of national and state governments, it may be challenging for the states to commit sufficient funds to sustain project achievements. The ability to sustain the payment of incentives is uncertain because of the structure of salaries and remunerations for civil servants. Nevertheless, according to the ICR (pp. 37-39), the project resulted in substantial institutional strengthening and transformational effects. The DFF approach (without incentive payments) was adopted as a national strategy for the provision of essential health care services, supported by the Basic Health Care Provision Fund that was created by the government. Oversight and supervisory tools are being increasingly used in the country. The Essential Medicine Manual and Health Care Waste Management manual were adopted nationally. The approach to financial management used by the Financial Management Manual, with key community involvement in the form of Ward Development Committees, was adopted by the Basic Health Care Provision Fund. The project supported and drove the development of the first national strategic health development plan, providing strategic direction for the country's healthcare sector (ICR, p. 38). The project also mobilized the private sector in pharmaceutical supply and health services delivery (see Section 4, Objective 1, Outputs). 8. Assessment of Bank Performance a. Quality-at-Entry Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) The project design was based on a growing global understanding that transforming input-based health systems to result-based health systems can change the persistent under-performance of health services in many countries. In Nigeria, poor health outcomes have persisted despite substantial investments over several decades (PAD, p. 25). The introduction of PBF was a new concept to service provision in Nigeria, but the ICR (p. 48) noted that there was strong ownership of the operation (see Selection of Participating States in Section 2). The preparation process was consultative and participatory. It developed a set of project preparation studies that included aspects of the political economy, institutional reviews, human resource studies, and Public Expenditure Reviews in the states of Ondo and Nasarawa. The dissemination of the studies engendered systematic consultations with key stakeholders and beneficiaries and brought international perspectives. Preparation missions, workshops, and training exposed the Client to the principles of result- based orientation. The preparation process also engaged three Rwandan consultants who were stationed in the states. Institutional arrangements focused on the state level, LGAs, and health facilities (PAD, p. 16). The ultimate authority for the project at the state level was the Commissioner for Health, and the main institutions involved in project implementation were at the level of the State Ministry of Health, the State Primary Health Care Development Agency, and the LGA primarily through PHC Departments. There were PHC Facility RBF Committees and Hospital RBF Committees to monitor service quality and delivery, performance, facility level business planning, and approval of fund utilization (PAD, p. 19). At the federal level, the two lead agencies responsible for the project were the Federal Ministry of Health and the National Primary Health Care Development Agency. Implementation arrangements fully clarified the roles and responsibilities of various actors. Financial management and fund flow arrangements were well prepared (PAD, pp. 26-30). Project financial management units were established in the three states in the Offices of the State Accountants- General (PAD, p. 26). The provision of technical assistance was planned at various levels to strengthen capacity for managing financial aspects of the project (PAD, p. 23). The ICR (p. 43) noted that the project had in place adequate procurement, financing, budgeting, and financial management mechanisms. Risks were adequately identified, and mitigation measures were planned, including performance-based incentives and capacity building. Environmental safeguards addressing healthcare waste management were adequately prepared. However, there were moderate shortcomings, such as in design complexity (ICR, p. 41) and in M&E design (ICR, p. 45), including the lack of family planning indicators and the lack of early alignment of additional financing indicators with the project information system. Quality-at-Entry Rating Moderately Satisfactory Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) b. Quality of supervision The Task Team was reportedly proactive and provided close and regular implementation support over the entire implementation period (ICR, p. 43). The Task Team provided hands-on implementation support to the Project Implementation Units (PIUs), and technical assistance was provided to states and at the national level during the entire implementation period. In addition to monthly field visits (ICR, p. 43), formal supervision missions were undertaken every six months and included expertise from fiduciary team members, environmental safeguards specialists, and government counterparts (ICR, p. 48). The Task Team held joint sessions to update the results framework and promoted dialogue and follow-up with the government. The ICR reported that the relationship between the Task Team, the states, and the National Primary Health Care Development Agency was cordial. The project did not merge the results framework of the original project with the AF results framework for over two years in the portal. This created a gap in reporting (ICR, p. 49). The indicator for the number of 12-23 months old children vaccinated with Penta 3 in Nasarawa and Ondo and NE states was not included in the results framework at the time of writing the ICR. Nevertheless, the ISRs accurately highlighted emerging issues and action plans for Management’s attention and guidance. The aide-memoires captured implementation progress and provided action plans, with updates on issues discussed in previous aide- memoires (ICR, p. 43). Fiduciary supervision and monitoring of safeguards were adequate. Safeguard issues were identified and resolved at the PIU and facility levels. There were two levels of workplan approvals. The first was for the entire annual workplan, while the other was when each activity was to be implemented. The reason for this double approval was to ensure that approved annual plans were responsive to possible arising changes, and for checking concordance of activities with annual plans. But most of the State PIUs felt that the response time for approving of activities was delayed, thus affecting the delivery of some approved annual plans (ICR, p. 49). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The objectives were clearly specified and were reflected by the indicators. The results framework was clear. Both the original project and AF designed elaborate M&E plans and arrangements. However, there Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) were multiple shortcomings that extended to M&E readiness of the AF during project implementation. There was a long delay in establishing baseline data and resulting mis-setting of targets, and inaccuracies in the early setting of the results framework in the PBF portal. In the context of maternal and child health interventions, the design lacked family planning indicators. Sources of data consisted of (a) routine Health Management Information System data; (b) administrative reports by LGAs to the ministries of health at the state and federal levels; (c) quarterly quality checks; (d) household surveys at baseline, at mid-term, and at end-line; and (e) health facility surveys to be carried out by the National Population Commission (PAD, pp. 20-21). The project also included a mid-term evaluation to test the effectiveness of the project at mid-course (ICR, p. 33). b. M&E Implementation At the facility level, project data were regularly collected and informed the payment of subsidies to the health facilities covering operations and staff performance bonuses. Contract Management and Verification Agencies provided the first level of verification of health facility claims, and Independent Verification Agents provided second level reviews before data validation. Each PIU had a dedicated M&E officer for the project throughout the life of the project. At LGA level, information system officers collected health facility data and entered them into the portal. But there were delays in key surveys and misalignment of indicators across systems during most of the project’s lifetime. Pursuant to these delays that affected timely reporting (ICR, p. 46), the project harmonized project indicators with those of the health management information system and linked the project database with the open-source web-based District Health Information Software 2 to allow inter- operability of the two platforms. Pending resolution of information system delays, the states reported their results in a customized MS Excel worksheet. The PBF District Health Information Software 2 was also connected to the federal health management information system. An impact evaluation was carried out in 2017 prior to the Mid-Term Review of the parent project in the three states of Ondo, Nasarawa, and Adamawa, and, according to the ICR (p. 47), it showed strongly positive results on quantitative and qualitative aspects of maternal & child health services. c. M&E Utilization The project benefited from several data sets that included quantitative and qualitative data, community client satisfaction, quality counter-verification, and performance assessment for Contract Management and Verification Agencies, Independent Verification Agencies, Hospital Management Boards, LGAs, and from the portal data that offered information based on the type of service offered, including the minimum package of activities and the complementary package of activities. The project also benefited from M&E for invoicing, training and capacity building, and impact evaluation. Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) The experience of the parent project contributed to the design of the AF that expanded the project to five additional NE states. The ICR (p. 47) reported that the favorable results of the mid-term impact evaluation informed the design of the Basic Health Care Provision Fund, that is part of the National Health Act, and the design of the first phase (P167156, Credit of US$650 million equivalent) of the US$1.5 billion Multiphase Programmatic Approach of the Nigeria Improved Child Survival Program for Human Capital. In addition, the project supported the development of the first national strategic health development plan that provides strategic direction for Nigeria's health sector (ICR, p. 38). M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Category B for Safeguard Policy OP 4.01 on Environmental Assessment because of potential risks generated by healthcare waste. A Health Care Waste Management (HCWM) Plan, based on the national plan, was prepared and disclosed (PAD, p. 36). A manual was distributed to stakeholders. Related training, equipment and supplies were provided. The ICR noted that the project strengthened HCWM in participating states and ensured the application of safeguard requirements and practices. The ICR (p. 47) reported that, in some states, private health care waste collectors were accredited to collect, treat, and dispose of health care waste according to World Bank and National Standards. Compliance with the Bank’s safeguards requirements was adequate during the course of the project (ICR, p. 49). The overall safeguards rating was recorded as satisfactory. The project had a grievance redress mechanism (GRM). A GRM focal person maintained a GRM logbook, coordinated the collection of grievances, and facilitated resolution or referral to the appropriate channel for resolution. Quarterly GRM reports were generated by LGAs and disseminated to the states, while consolidated state reports were transmitted to the Federal level and shared with the Bank (ICR, p. 49). No significant issues were reported throughout the life of the project. b. Fiduciary Compliance Overall, internal controls were adequate, and the reviewed authorizations and approvals were satisfactory. Internal audit functions were in place at the federal and state levels. When issues were identified, they largely included unretired advances, inadequate documentation, and potential ineligible expenditures. Such issues were resolved by agreed actions with relevant PIUs. The PIUs were trained and re-trained on financial reporting. External audits were undertaken as agreed, and, in one of the states, a case of fraud was detected and resolved, as the state paid back related funds (ICR, p. 48). Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) The Bank Task Team provided hands-on procurement support to the PIUs to ensure compliance with the financial handbook. Procurement under the project was carried out according to Bank guidelines, and with adequate record keeping and use of the Bank System for Systematic Tracking of Exchanges in Procurement, known as STEP. c. Unintended impacts (Positive or Negative) None reported. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory The M&E system as designed and implemented was sufficient to assess the achievement of project objectives. Also, M&E Quality of M&E Modest Substantial utilization extended beyond the project and benefited other lending operations and policy development. Quality of ICR --- Substantial 12. Lessons The ICR (pp. 50-51) offered several lessons and recommendations, including the following lessons restated by IEG: Effective service provision can be tailored to fragile and conflict areas to support access to quality health care. In the context of the insurgency in the North East, the project adjusted by providing mobile clinics, direct facility financing that guaranteed regular funding to facilities, and the introduction of psychosocial support in routine health care delivery. While the impact of such Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) approaches was not evaluated, the number of persons reached in insecure environments in the states of Adamawa, Yobe, and Borno continued to increase in spite of fragility, conflict, and violence. Rigorous counter-verification mechanisms are facilitated by enhanced separation of institutional functions. Under the initial parent project, quantitative verification was undertaken by the State Primary Health Care Development Agency, and qualitative verification was carried out by Primary Health Care Departments and Local Government Areas, complemented by community- based ex-post verification. This arrangement showed weaknesses under the original project. Hence, the Additional Financing of 2016 introduced independent non-state actors consisting of Independent Verification Agents and Contract Management and Verification Agencies. An early transition plan for financing may facilitate PBF financial sustainability. At project closing, there were concerns and challenges regarding the sustenance of PBF-related incentives. By contrast, the outlook at entry was favorable, as external contributions from donors represented a small proportion of total health financing in Nigeria, and a National Health Bill was expected to generate an additional US$3.14 per capita to the US$8 per capita public spending on health at that time. However, macroeconomic challenges that emerged during the decade revealed the need for concrete and realistic long-term financing options. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clearly written overall, results-oriented, and aligned to development objectives. It articulated the theory of change underlying the project's planned trajectory toward intended outcomes. The ICR provided a complete critique of the operation. The evidence provided was valid, and with plausible causality. The analysis was thorough and candid. It provided adequate information on fiduciary and safeguards compliance. It offered specific lessons derived from project experience. The ICR made noteworthy efforts in outlining the assumptions that underpinned project design and provided granular and quantitative information on the estimated negative impacts of COVID-19 on essential services (using GFF and DEC analysis in collaboration with MOH and HNP Teams). The ICR followed guidelines and was internally consistent, except in two areas: (i) there was some disconnect between M&E discussion and the rest of the review, and the ICR did not explain its conclusion on poor M&E in terms of M&E's primary purpose of assessing the achievement of development objectives and testing results chain links, while concurrently concluding that objectives were almost fully achieved; and (ii) the ICR was not concise and had a lengthy implementation narrative, resulting in a main text of 52 pages, thus moderately diluting the focus of its storyline. Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NG-States Health Prog. Invest. (FY12) (P120798) a. Quality of ICR Rating Substantial Page 22 of 22