Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) Report Number: ICRR0022094 1. Project Data Project ID Project Name P129472 SN-Health & Nutrition Financing (FY14) Country Practice Area(Lead) Senegal Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-53240,TF-15872,TF-16618,TF- 29-Jun-2018 36,520,012.65 A0565 Bank Approval Date Closing Date (Actual) 11-Dec-2013 30-Jun-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 20,000,000.00 21,278,000.00 Revised Commitment 40,978,405.81 20,978,405.81 Actual 36,559,847.42 18,128,610.15 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The objectives of the project were to increase utilization and quality of maternal, neonatal and child health and nutritional services, especially among the poorest households in targeted areas in the Recipient’s territory (Financing Agreement, 3/28/14, p. 4). The statements of objectives in the PAD and ICR were identical. Increases and revisions in associated outcome targets are explained in the M&E section, 9b. Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components 1. Results-based financing (RBF) for health and nutrition services and capacity building (Appraisal US$22.3 million; Revised US$15.7 million; Actual US$14.6 million). 1.1 Provision of a package of health services, comprising preventive and curative services, including maternal, neonatal and child health care through the provision of RBF grants in six selected regions in a phased manner. 1.2 Carrying out a program of activities aiming at: (a) building capacities of the Ministry of Health and Social Action (MOHSA) at central, regional and district levels to support the extension of the RBF mechanism in six regions, including technical assistance, training, and provision of goods; and (b) supporting monitoring and verification of activities financed under RBF through the engagement of community-based and independent third party verifiers. 2. Improving accessibility to maternal, nutrition and child health services (Appraisal US$14 million; Revised US$15.6 million, Actual US$15.6 million). The purpose of the component was to subsidize demand for healthcare and nutrition services through: 2.1 Universal health insurance scheme: the component was to provide activities that would strengthen capacities of national entities involved in the implementation of the scheme, including training on regulatory, financial and strategic aspects of the universal insurance; study tours, international and regional workshops; equipment; consulting services for the design of the Equity Fund that supports subsidies to the Community Based Health Insurance (CBHIs), also known as Mutuelles; and training and equipment for the creation of Mutuelles. 2.2 National nutrition strengthening program for improving demand for nutrition services by supporting relevant activities and fostering behavioral change for better nutrition outcomes. Activities were to build on the existing nutrition program in the six targeted regions, implemented locally by Community Execution Agencies that were NGOs working with, and on behalf of, local governments to implement a package of nutrition services and behavioral change communication at the community level, including growth promotion and monitoring for children between 0 and 24 months, behavior change communications, and detection and community care provision for malnutrition. Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) 2.3 Maternal health vouchers: The scheme was a pilot, to be implemented in the six project regions, covering half of their local collectivities, and aiming to incentivize two services: (i) completion of four antenatal care visits by poor pregnant women; and (ii) assisted deliveries. The scheme was to finance vouchers/cash transfers to pregnant women and the operational costs of the Community Execution Agencies/NGOs implementing the program, namely in managing the cash transfer, and in verification and communications. 3. Institutional strengthening and project implementation (Appraisal US$6 million; Revised US$9.5 million, Actual US$9.3 million). The component was to include the following: 3.1 Capacity building for better regulation of the hospital sector. The subcomponent would strengthen the existing hospital information system and, more precisely, roll out a Unified Hospital Information System in main hospitals, so as to better track production, revenues and costs. It would also support the design and piloting of a new hospital payment system and the development of an improved planning tool for hospital care. 3.2 Capacity building for better M&E of the overall health system in support of the National Statistical Agency for its “Senegal continuous survey," that consists of an annual survey, combining a household survey and a health facility survey. Expenditures were to consist of consulting services, training, workshops, and operating costs. 3.3 Capacity building for better general and financial management within the health system, notably to support the internal audit function within MOHSA. Project coverage: The project targeted six (out of 14) administrative regions in Senegal: Kaffrine, Kedougou, Kolda, Tambacounda, Sédhiou, and Ziguinchor, with a total population of about 3.5 million people or about 25% of the total population (PAD, p. 9, and ICR, p. 8). The six selected regions were among the poorest in the country, with the worst health results in terms of maternal and child mortality, and the lowest rates of service coverage for basic services. Component revisions A 2016 restructuring provided support for Ebola preparedness under Component 3 on institutional strengthening. Also, it added central-level entities as beneficiaries of RBF bonuses for motivational purposes, and added the adjacent health district of Gossas in Fatick Region. A 2018 restructuring reallocated funds among components, where institutional capacity building increased by US$3.5 million, while RBF allocations under component 1 were reduced by US$6.6 million. The total project funding envelope was reduced by US$1.5 million. Specifically, the 2018 restructuring added new activities in line with the Global Financing Facility for Every Woman Every Child (GFF) investment case (see section 7).to enhance the provision of quality health services, behavior change for adolescent health, and demand for reproductive health services. The ICR (p. 14) noted the following: Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) - Sub-component 1.3 was meant to support the procurement of obstetrical and maternal health equipment, training, a mobile midwife strategy, mechanisms for quality control of drugs, and geographic accessibility of essential medicines. - Sub-component 2.4 for adolescent health aimed at supporting behavioral change interventions and communication strategies to promote demand for reproductive health services targeting adolescents. - Additional capacity building activities were added for procurement, the Maternal Health Directorate, and the Planning, Research and Statistics Directorate. - Sub-component 3.4 was to support health financing to support studies, training and dissemination activities related to health financing reforms, including domestic resources mobilization and the role of public-private partnerships. - Under Component 2 for improving accessibility to health care services, allocations were increased from US$14 million to US$15.6 million, notably for nutrition. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost and financing. The original project cost was US$42.3 million (PAD, p. 14), consisting of an IDA Credit of US$20 million, a grant of US$20 million from the Multi-donor Trust Fund for Health Results Innovation, and a grant of US$2.3 million from the Senegal Health Results-Based Financing Trust Fund supported by USAID. Project allocations were revised and reduced to US$40.8 million in 2018, and the actual cost at project closing was US$36.6 million. No direct Borrower financing was envisaged or provided. Dates. Project appraisal was finalized on 11/15/13 and the project was approved on 12/11/13. It became effective on 5/14/14. The project was restructured on 5/4/16 to revise disbursement arrangements and financial management, and to provide support for preparedness in confronting Ebola. A Mid-Term Review was undertaken on 6/1/17. A second restructuring on 4/10/18 revised the results framework and expenditure categories, and extended the closing date by one year from 6/29/18 to 6/30/19, at which date the project closed. 3. Relevance of Objectives Rationale At appraisal, the project was responsive to Senegal's priorities for improving maternal & child health and nutrition, while promoting health care demand, access and quality of health services, and advancing the universal health coverage agenda. The objectives were consistent with the Country Partnership Strategy (CPS) FY2013-2017 and its strategic objectives to improve the delivery of health services and to improve maternal and child health services. CPS outcomes included the percentage of deliveries attended by skilled birth providers, percentage of children completing immunization, and children receiving micronutrients. Project objectives remained consistent with the strategic goals of Senegal in the health and Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) social sectors, as outlined in the National Sanitary and Social Development Plan (2019–2028) aiming to reinforce governance and financing in the health sector, develop the supply of health and social services, and promote social protection. At project closing, the objectives remained fully aligned with the Bank Group Country Partnership Framework (CPF) FY20-FY24, specifically under its first Focus Area to build human capital with a focus on the most vulnerable, including CPF Objective 1.1 to improve early years outcomes for children ages 0-5 in health, nutrition, and brain development; and CPF Objective 1.4 to empower women and adolescent girls to have more control over their childbearing and productivity. The objectives are also consistent with the twin goals of reducing poverty and promoting shared prosperity, and they contribute to the development of Senegal’s human capital in areas where the country still lags behind such as persistent regional imbalances in terms of access to basic and reproductive health services, and high fertility. In addition, the objectives remain consistent with the Sustainable Development Goals adopted by the United Nations, notably Goal 3 to “Ensure healthy lives and promote well-being for all at all ages," and its targets for reduction of maternal and under-five mortality, achieving universal access to sexual and reproductive health care services, and universal health coverage. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase utilization of maternal, neonatal and child health and nutritional services, especially among the poorest households in targeted areas Rationale It was reasonably expected that:  RBF interventions for maternal and child health services in six of the poorest regions;  the application of a quality check-list (see Objective 2) along with quantitative targets in determining payments and accountability for provided services;  behavior change communications promoting demand for reproductive health services targeting adolescents;  provision of nutrition services, behavior change communications, and support to local nutrition platforms;  supporting health insurance schemes and vouchers to promote financial accessibility to health services;  raising human resources skills, quality of drugs, and strengthening the regulation of the hospital sector; and Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472)  capacity building for MOHSA and its facilities; and provision of equipment and goods, and information tools, would plausibly lead to increased financing availability at the front line for maternal and child health and nutrition services, increased demand for health and nutrition services, and increased motivation and accountability. All of these expected outputs would plausibly contribute to increased utilization and quality of maternal & child health and nutrition services among poor households. Given the project's geographic targeting of six of the most disadvantaged regions of Senegal, and given that, in five of the six regions, the percentage of households who were poor (defined as those belonging to the lowest wealth quintile) was more than double the national average, it was plausible that the project would disproportionately benefit poor households in terms of increased utilization and quality of health services. Outputs and intermediate results Health personnel receiving training reached 5,414, exceeding the target of 3,144. The number of health facilities that had an RBF contract reached 373 facilities, exceeding the target of 318 facilities. The number of people receiving essential health, nutrition and population services reached 3.9 million in 2019, exceeding both the original target of 1.6 million and the revised target of 3 million people. The number of females included in these results was 2.3 million, exceeding the target of 1.8 million females. The number of women and children who have received basic nutrition services reached 3.1 million in 2019, exceeding both the original target of 0.96 million and the revised target of 2.3 million. The number of pregnant women receiving ante natal care during a visit to a health provider reached 0.53 million in 2019, exceeding the target of 0.47 million. The number of deliveries attended by skilled health personnel reached 292,572 in 2019, exceeding the target of 271,700 deliveries. As for the pilot scheme on maternal health vouchers, 39,164 vouchers were provided, short of the target of 50,200 vouchers. The pilot served in revealing difficulties in identifying beneficiary mothers. The number of children fully immunized reached 0.52 million in 2019, exceeding the target of 0.45 million children. The number of children aged 0-23 months who attended at least one growth monitoring and promotion service during the three preceding months reached a cumulative number of 0.63 million, exceeding the target of 0.59 million. The number of people effectively targeted by communication activities of the Universal Health Insurance reached 239,840 in 2019, exceeding the target of 165,000 people. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) The number of operational modules for the Integrated Management Information System for the Universal Health Insurance reached 5 modules in 2019, attaining the target of 5 modules. The number of Demographic and Health Survey (DHS) and Health Service Delivery Reports produced every year reached 8 in 2019, achieving the target. Overall implementation progress relevant to the objective: Disbursements were slow during the initial four years, but accelerated during the last two years of the project. According to the Restructuring Paper dated 4/27/2016 (p. 6), Component I (RBF) encountered delays that included disagreements on procedures and delayed recruitment of Independent Verification Agencies, while Component 2 (Improvement of accessibility to maternal, nutrition and children health services) and Component 3 (Institutional strengthening) progressed satisfactorily. According to the second Restructuring Paper of 4/10/2018 (p. v), contracts for verification were signed in January 2016, the first RBF verification was completed in May 2016, and the first RBF payments were made in October 2016. Hence, RBF was undertaken during 2016, 2017, and 2018, in lieu of the originally envisaged 4-year period, and was discontinued at the end of 2018, having disbursed 93% of its revised allocations (US$14.6 million out of US$15.7 million). Outcomes The number of new acceptors of modern contraceptive methods reached 128,648 in 2019, exceeding the target of 107,000 new acceptors. The number of births (deliveries) attended by skilled health personnel reached 292,572 in 2019, exceeding the target of 271,700 deliveries. The percentage of pregnant women having at least four antenatal care visits reached 44% in 2019, exceeding the target of 35%. The number of severely malnourished children who were referred and received at a health facility for all necessary visits reached 5,721 children in 2019, short of the target of 6,700 children. Utilization was facilitated by improved coverage, as the number of poor people reached by the project aggregated at 4.4 million people in 2019, exceeding the target of 2.8 million people, although a percentage would have been more informative than an absolute number. In addition to reliable data of the health information system (TTL clarifications, 3/17/2020), the ICR (p. 24) stated that other sources of information also corroborated the pattern of positive results. The 2018 DHS data for the great South region, covering five of the six project regions, indicated a positive trend on indicators measuring coverage in basic maternal and child health services. The favorable trends were also confirmed by MOHSA Evaluation Report of November 2019 in the context of the government’s own ICR. For nutrition outcomes, DHS surveys suggested that the greater South outperformed the rest of the country (ICR, p. 24), as the surveys indicated that, between 2012 and 2018, the stunting rate decreased from 26% to 17% in the project regions, while the rate remained stable at the national level at 19%. The percentage of women declaring that they faced significant barriers to access Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) health care (with the lack of money identified as the main factor) decreased from 51% in 2011 to 43% in 2018, and the reduction was larger for women in households belonging to the lowest quintile. The direct pro-poor orientation of the project was noteworthy, as reflected by the geographic targeting of the poorest regions and the alleviation of financial obstacles to seek health care. Also, according to the ICR (p. 29), the Universal Health Insurance allowed the poorest (beneficiaries of the Bourse de Sécurité Familiale or Family Safety Net Cash Transfer Program) to be affiliated to the CBHI-Mutuelles without paying any contribution, thus further protecting the poor from financial risks associated with illness. Per TTL clarifications (3/16/2020), there were no other relevant health operations in the Southern regions during the same period. Unlike health status improvements that may be explained by various underlying socio-economic determinants, substantial changes in utilization and quality normally require direct interventions, including the reduction of financial barriers to utilize services. Hence, it is reasonable to conclude that the project contributed to increasing utilization and quality of services in the project-targeted areas, especially among the poor. Rating Substantial OBJECTIVE 2 Objective Increase the quality of maternal, neonatal and child health and nutrition services, especially among the poorest households in targeted areas Rationale Rationale and outputs: The same as under Objective 1, above. As importantly, the project monitored the average score of a health quality index that consisted of a quality checklist to assess health facilities’ performance throughout RBF implementation. Quality aspects included: availability of essential equipment, hygiene, financial and drug management, monitoring, maternal care, immunizations, family planning, and infectious disease management. Both quantitative and quality aspects were considered in determining quarterly payments to health facilities during implementation. Outcomes In conjunction with the outcomes described under Objective 1 above, the index of quality of health care increased from a baseline of 0% in 2011 to 76.6% in 2018, exceeding both the original target of 65% and the revised target of 70%. Rating Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project contributed to increased utilization and quality of maternal, neonatal and child health and nutrition services, especially among the poorest households in the targeted regions, and most of its targets were exceeded. The aggregation of two almost fully achieved objectives is consistent with a substantial rating for efficacy. Overall Efficacy Rating Substantial 5. Efficiency In 2013, the PAD’s economic analysis estimated costs and benefits by considering the cost of RBF financing and nutrition activities without including institutional strengthening activities, which are less tangible. The time period for assessing benefits was 3.5 years or 7 semesters (PAD, p. 78). Based on LiST simulations (Lives Saved modeling software Tool), the analysis estimated that maternal mortality would decrease in the six targeted regions by 29%, and under-five mortality by 12%. The analysis compared a project scenario with a status quo scenario. A 3% discount rate was used. A sensitivity analysis was not undertaken. Discounted benefits and costs were estimated at US$35.6 million and US$18.2 million respectively, resulting in an estimated benefit-cost ratio of 1.95. The analysis assumed that the benefits would result from uniform mortality reductions over a short period of time with effective implementation progress according to plans. The ICR did not undertake a cost-benefit analysis at project closing. It stated that the interventions added by the second restructuring in 2018 consisted mainly of capacity building and institutional strengthening activities, which were difficult to include in simulation models (ICR, p. 27). But institutional strengthening, regardless of its size, should not have hindered the preparation of a cost-benefit analysis based on core health and nutrition interventions, and the analysis would have excluded institutional strengthening from its calculations, as was done by the PAD at entry. Significant shortcomings were observed in the efficiency of implementation related to RBF under component 1. The ICR stated that severe difficulties and delays were encountered in the implementation of RBF, procurement of Independent Verification Agencies, and in developing a web-based platform. Staff managing the National RBF Program lacked commitment, motivation and capacities, and were not responsive to feedback from the field. Financial mechanisms were not flexible to facilitate the rapid implementation of interventions, resulting in ineffective cash flow. Per TTL clarifications, there was a paradox in that front-line implementers, such as for nutrition, were eagerly waiting for project funds to be released by MOHSA so as to begin implementation. The ICR (p. 33) stated that management of finances was a bottleneck, compounded by inefficiencies and delays within the Ministry of Finance. However, financial management improved during the last two years of the project. There were inefficiencies resulting from bridging with the USAID-supported pilot, as standards differed, Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) and stakeholders involved in the pilot phase were resistant to change (ICR, p. 30). The ICR also reported inefficiencies at the project coordination level and noted that multiple responsibilities taken on by the coordinator may have been an obstacle to the timely resolution of implementation issues, penalizing the whole project (ICR, p. 33). There were recurring coordination and communication problems within the coordination team, lack of managerial leadership, and delays in addressing requests for processing and staffing issues. Technical issues remained unresolved and there was little, if any, communication between technical directorates of the ministry. Reallocations among components were based on absorptive capacity (Restructuring Paper, 4/10/2018, p. vii). In addition to the reduction in RBF funding, allocations for strengthening the hospital regulation system were also reduced, as related activities under the Hospital Directorate suffered major delays, and it became clear that procurement of information technology equipment and training activities could not be carried out before the end of the project. Despite stated political support to the program, the lack of real RBF ownership by MOHSA officials likely doomed the approach, according to the ICR (p. 32), as the RBF mechanism was not perceived as a key strategy to improve health sector performance, and the government decided to close the RBF program at the end of 2018. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated high, as the objectives were aligned with national strategies and with the Bank’s Country Partnership Framework at project closing. Efficacy is rated substantial, consistent with almost fully achieving the project objectives. Efficiency is rated modest in view of insufficient economic analysis and significant shortcomings in the efficiency of implementation. These findings are consistent with moderate shortcomings and an outcome rating of moderately satisfactory. a. Outcome Rating Moderately Satisfactory Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) 7. Risk to Development Outcome The project was part of a larger national strategy initiated in 2012 to expand universal health coverage (PAD, p. 4) that has continued political support. Based on the information provided by the ICR, it appears that the questionable commitment at the sectoral level was limited to the RBF mechanism itself rather than to the broader health development agenda, including project objectives. The government remains committed to improving health outcomes and fighting malnutrition as part of the second pillar of the Emerging Senegal Plan or PSE 2014 – 2035. In April 2019, the Universal Health Insurance Agency became an autonomous agency under the Ministry of Community Development and Social and Territorial Equity, institutionalizing a split between provision and financing of health care (ICR, p. 39). Institutional strengthening under the project was substantial. Continued external support to the health agenda is likely. The Global Financing Facility for Every Woman Every Child or GFF held discussions with partners and adopted an Investment Case in March 2018 for the development of a Reproductive, Maternal, Newborn, Child and Adolescent Health (RMNCAH) strategy, focusing on five priority areas related to an RMNCAH package, adolescent health, supply of healthcare services, and health system governance. The Investment Case targets five disadvantaged Southern regions that were already supported by this project, thus increasing the likelihood of coordinated support from donors. A follow-on project for “Investing in Maternal, Child and Adolescent Health” (P162042), supported by the Bank, was approved in September 2019, and is financed by an IDA Credit of US$140 million and a grant of US$10 million from GFF. The follow- on operation aims at scaling up successful approaches tested under this project to improve maternal and child health and nutrition (community nutrition platforms – with community execution agencies and NGOs, health insurance for the poorest, cash transfers to poor pregnant women, and availability of critical maternal and child health inputs) and to pilot innovative approaches on adolescent health and the quality of care. 8. Assessment of Bank Performance a. Quality-at-Entry In addition to its substantive merits in responding to complex issues, the project was considered an important conduit for Bank re-engagement in Senegal (TTL interview, 3/16/2020). Project preparation was informed by analytical work such as the Public Expenditure Review conducted in 2012-2013, and technical assistance on hospital reform. Preparation was informed by experience gained from other projects (PAD, p. 14 and ICR, p. 37), including a USAID-supported RBF pilot that had a positive experience. Main lessons pertained to verification arrangements and incentives. Project preparation planned an implementation period of four years in line with other RBF schemes that were adequately implemented in other West African countries, and in more difficult contexts. Preparation benefited from a Quality at Entry Review meeting in August 2013, including for limiting the number of PDO-level indicators and for maintaining flexibility to support the government’s evolving strategy on Universal Health Coverage (ICR, p. 37). Overall implementation arrangements were adequate and clearly defined (PAD, pp. 15-17). MOHSA, with its Directorate for Financial Management, was the main implementing agency. The Directorate for General Health was in charge of technical coordination. A Project Coordinator was to monitor day-to-day implementation, and a Steering Committee was to oversee the overall project. The Steering Committee Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) had wide representation, including from the Ministry of Economy, the Delegation for Social Protection and National Solidarity, the Unit Against Malnutrition, international development partners, local governments, and MOHSA. M&E arrangements were adequate overall. Fiduciary aspects were deemed to be acceptable subject to setting up financial management arrangements that would meet the Bank’s minimum requirements under OP/BP10.00, and once mitigation measures for financial management were implemented as recommended by a financial management assessment (PAD, pp. 21). However, there were some shortcomings related to the identification and assessment of risks associated with the project. Substantial capacity risks were underestimated and were deemed to be moderate by the PAD (p. 19). Technical assistance was viewed as sufficient to alleviate concerns about MOHSA’s limited experience in health reform and financial management. Stakeholder risks were also deemed to be moderate, and yet there was a lack of internal support and buy-in for RBF within MOHSA. Such risks turned out to have an important impact in delaying project implementation (ICR, pp. 37-38). Nevertheless, and as stated in the PAD (p. 19), RBF mechanisms and universal health insurance programs are always risky because they are transformational. Still, in the broader regional context, the project design was adequately informed by Bank experience in other Western African countries. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The supervision team provided very close support throughout project implementation with a stable task leadership that changed once in 2017 (ICR, p. 38). The incoming TTL had been part of the team since the preparation stage, thus favoring continuity. The ICR stated that 10 ISRs were produced and provided detailed and clear information on implementation issues that were faced, although updates for some indicators were delayed. The Mid-Term Review produced quality information to support informed decisions. Throughout implementation, the Bank provided strong technical support and led policy dialogue on health financing. The team provided substantial implementation support on procurement, financial management, and disbursements. The team was pro-active in pursuing development objectives, resolving arising challenges, and working closely and intensively with government counterparts (ICR, 38). In addition to two difficult restructurings, the team pursued in June 2017 the revision of technical and operational aspects of the RBF model, and agreed with MOHSA on an accelerated roadmap, which included the implementation of a revised manual and staff capacity strengthening. In 2016, the team facilitated the creation of sub-accounts for implementing entities to overcome slow financial mechanisms (Restructuring Paper dated 4/27/2016, p. 9). Overall implementation delays that were observed during implementation were beyond the team's control (ICR, p. 38). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Development objectives were clearly specified. The indicators were measurable and reflected project objectives. The theory of change was plausible and illustrated how key activities and outputs would lead to the desired outcomes. Intermediate results indicators were pertinent and appropriate in capturing activities and outputs. Data on most of the selected indicators were already produced by the existing health information system (PAD, p. 17). Health care utilization included four PDO-level indicators (acceptors of modern contraception, attended deliveries, antenatal care visits, and referral of severely malnourished children). The indicator on the proportion of poorest people required other sources of data. The Senegal Continuous Survey, combining a household survey or DHS and a health facility survey, was to be undertaken on a yearly basis. Overall M&E design and arrangements were adequate, and the existing information system was well embedded institutionally. The design planned for RBF-related verification by third-party Independent Verification Agencies with counter-verification by independent Community Based Organizations. The development of a web-based platform was planned to improve the speed and reliability of reporting. Training and capacity building included support to the “continuous survey” (see paragraph below) and an impact evaluation of the RBF intervention. b. M&E Implementation Following initial implementation delays, the first RBF verification was completed in May 2016, as stated in section 4. There was low capacity in compiling RBF data, resulting in errors. MOHSA's routine information system, known as DHIS2, became the primary source of data (ICR, p. 24). As such, overall M&E functions were likely to be sustained under the country systems. It was decided to drop the impact evaluation of RBF, as (according to the ICR, p. 21) it was unlikely to contribute to the learning agenda of the Health Results and Innovation Trust Fund. The restructuring of April 2018 increased three outcome targets, and lowered another (antenatal visits) to correct a mistake in the baseline, as the latter was lower than that of the PAD (ICR, p. 20). The restructuring also modified the definition of two indicators to cumulative numbers based on data availability, although a percentage would have been more informative than an absolute number, and such changes were unrelated to project ambition. The household surveys for DHS and health facility surveys were completed every year as planned (ICR, p. 25). c. M&E Utilization M&E findings were used to increase target values, adjust others, modify definitions, and provide evidence on the achievement of objectives. In addition to the health information system, alternative sources of information such as DHS surveys were used (ICR, p. 35) to corroborate or complement the findings. M&E was also used to inform subsequent interventions (see section 7). Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as environmental assessment category C in view of limited risks that would be addressed by the Medical Waste Management Plan. b. Fiduciary Compliance Financial management. The Fiduciary Directorate of MOHSA was responsible for financial management. Performance was initially very low. Slow fund flows negatively impacted implementation during the initial years. In addition to shortcomings at MOHSA’s level, there were delays at the Ministry of Finance regarding the management of the designated account, according to the ICR. Since late 2017, a moderately satisfactory rating for financial management was maintained. In 2018, supervision missions noted that important efforts for improvement and significant progress were made in financial management, and that Bank recommendations were followed (ICR, p. 36). The ICR did not report on audits, but the TTL clarified (3/16/2020) that there were no major audit qualifications, but rather remarks and recommendations for improvement. The Ministry of Health was responsive and adequately followed up on audit recommendations. Procurement. Procurement activities were undertaken through MOHSA’s existing mechanisms and according to guidelines. Procurement performance was rated moderately satisfactory for the duration of the project, as there were some shortcomings in approvals, payments, and contract management, and in updating procurement information in the Systematic Tracking of Exchanges in Procurement System known as STEP (online system for tracking procurement activities under Bank-financed projects). c. Unintended impacts (Positive or Negative) None reported. d. Other -- Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Both ICR and this ICR Review rated relevance of objectives as high. The ICR rated efficacy as modest, and this ICR Review rated efficacy as substantial because the objectives were almost fully achieved, and the Moderately project contributed to their Outcome Moderately Satisfactory Unsatisfactory substantial achievement. The ICR rated efficiency as substantial, and this ICR Review rated efficiency as modest because of insufficient economic analysis and significant shortcomings in the efficiency of implementation. Moderately Bank Performance Moderately Satisfactory Satisfactory M&E arrangements as designed and implemented were sufficient Quality of M&E Modest Substantial to assess the achievement of objectives. Also, M&E informed subsequent interventions. Quality of ICR --- Modest 12. Lessons The ICR (pp. 39-40) offered lessons and recommendations, including the following lessons, restated by IEG: Program ownership at all government levels is essential to ensure the success of RBF schemes. Under the project, the RBF program was accepted but not actually owned by the sector, resulting in slow implementation. Ensuring the success of a project requires carefully designed institutional arrangements, and a level of support proportionate to the capacity of entities involved. Under the project, institutional arrangements required striking a balance between using government systems and Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SN-Health & Nutrition Financing (FY14) (P129472) pursuing adaptations in financial mechanisms and staff incentives to facilitate effective implementation. Despite implementation challenges, an investment project can lay the foundation for a future partnership between the Bank and the Client, with agreed modes of support tailored to the Client’s needs. Although RBF implementation in Senegal was challenging, both parties collaborated to develop a follow-on operation, and used the learning experience of the project and some of its methods, such as quality check-lists and a web-based platform, and added focus on results in the follow-on project to improve the quality of health care and management practices. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was largely results-oriented and illustrated the theory of change that was aligned to development objectives. The results chain reflected how activities would lead to intermediate results and contribute to the desired outcomes. The quality of evidence was adequate and built on the existing information system and other data sources. The ICR offered useful lessons that were directly derived from project experience. But the ICR had multiple shortcomings. There were internal inconsistencies, such as in analyzing efficacy. The ICR had gaps in its reporting on fiduciary compliance and in its overall clarity, along with a very lengthy implementation narrative. The economic analysis was insufficient, and the reason provided (increased capacity building) for not preparing an economic analysis, as undertaken at entry, was not valid. a. Quality of ICR Rating Modest Page 16 of 16