Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) Report Number: ICRR0023715 1. Project Data Project ID Project Name P164953 CAR SENI Project Country Practice Area(Lead) Central African Republic Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-D3750,TF-A8196 31-Dec-2021 51,925,474.87 Bank Approval Date Closing Date (Actual) 27-Sep-2018 31-Jan-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 53,000,000.00 10,000,000.00 Revised Commitment 53,000,000.00 9,894,216.00 Actual 51,925,474.87 9,894,216.00 Prepared by Reviewed by ICR Review Coordinator Group Reantha Pillay Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project's Development Objective (PDO) as stated in the Project appraisal Document (PAD), Financing Agreement and Implementation Completion Results Report (ICR) were "to increase utilization and improve the quality of essential health services in targeted areas in the territory of the Recipient" (PAD, para 31; Schedule 1, Financing Agreement; ICR, para 17). Although the project was restructured three times, including changes to the results framework in June 2021, the PDO of improved utilization and quality of health services remained unchanged. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) Despite several outcome targets revised downward at restructuring, the project's achievement exceeded both the original and revised targets in almost all cases. As such, this ICR Review opted not to conduct a split evaluation. Further, the removal of project activities related to community-based services is not deemed a shrinkage of project scope or ambition but as an adaption to the evolving safety and security situation in Central African Republic (CAR) – a Fragile, Conflict, and Violence (FCV) country context (PAD, para 11). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-Jun-2021 c. Will a split evaluation be undertaken? No d. Components Component 1: Improving the quality and utilization of essential health services at facility and community levels through Performance-Based Financing (PBF) (US$36 million at appraisal- US$1 million from the Government of CAR, US$28 million from International Development Association (IDA), and US$7 million from the Global Financing Facility; US$35 million at closure - of which US$28 million from IDA, and US$7 million from the Global Financing Facility) Subcomponent 1: Performance Based Financing (PBF) and Delivery of Free Healthcare This subcomponent involved the continuation and expansion of PBF in 15 health districts (2 new), 20 district hospitals, and 372 health centers in Health regions 2, 3, 4, 5, and 6 in CAR, building on work done in the Health System Support Project (P119815). Payments to facilities were conditional on the quantity and quality of services provided. The PBF focused on a pre-determined package of cost-effective preventive and curative Maternal, Child and Neonatal Health (MNCH) services, along with treatment for HIV/AIDS, Tuberculosis, and family planning. This subcomponent included the provision of free services to pregnant women, children under five years, marginalized and poor households. In periods of violence, healthcare for all was to be fully subsidized. At appraisal, PBF was to include community outreach in health facilities' catchment areas, however, at restructuring in June 2021, activities related to delivery of community-based health services were removed because the government did not put in place a National Community Health Policy or Strategy. Subcomponent 1.2: PBF verification and technical assistance This subcomponent included i) strengthening purchasing mechanisms through rigorous quality and quantity verification systems both prior to and after making a payment, ii) technical support and capacity building at Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) all levels (including tertiary hospitals), and iii) support for procurement of essential drugs through a centralized approach to ensure that health facilities had the critical inputs needed to provide services. Component 2: Reinforcing the capacity of the Recipient’s health system and addressing Gender Based Violence (At both appraisal and closure: US$18 million, of which US$15 million from IDA and US$3 million from the Global Financing Facility) Subcomponent 2.1: Health system strengthening This subcomponent included the creation of a national health platform, and an Investment Case which is an operational and feasible costed plan focused on key priority interventions to reduce maternal and child mortality. These two activities facilitated the Ministry of Health taking a leading role in coordinating the various health-related partners. At appraisal this subcomponent included i) conducting resource and geographical mapping, ii) defining a national health financing strategy, iii) supporting the implementation of a national Human Resource for Health Strategy, iv) improving the pharmaceutical supply chain, and v) establishing a heath information management system. Subcomponent 2.2: Addressing Gender Based Violence (GBV) This subcomponent included i) the establishment of integrated centers providing free, holistic medical care, psychological and socioeconomic support for victims of GBV, and ii) awareness raising and communication at all levels on the topic. Activities were defined after appraisal based on a national gathering of a GBV focus group, led by the Ministry of Health, in collaboration with various United Nation (UN) agencies, and international Non-governmental Organizations (NGOs). Activities included the provision of post exposure prophylaxis kits, emergency contraception and treatment for sexually transmitted infections at health centers and district hospitals as well as training for healthcare providers. Subcomponent 2.3: Project implementation support This subcomponent included project implementation through a Project Implementation Unit housed within the Ministry of Health. The subcomponent included financing operating costs, training, and equipment; paying salaries of international and national consultants; audits and communications, as well as the implementation and monitoring of safeguards. Component 3: Contingent Emergency Response (CERC) (US$0 equivalent at appraisal and closure) This component was detailed in the PAD (PAD, para 57), however was omitted from the ICR as it was never utilized. The component was for the rapid reallocation of project funds in the event of a disaster or crisis, to mitigate major adverse economic and/or social impact. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost and Financing: The total cost of the project at appraisal was US$54 million, reduced to US$53 million at first restructuring in June 2021. IDA provided US$43 million (US$42,031,259) and the Global Financing Facility provided US$10 Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) million (US$9,894,216), 100 percent of which was disbursed by closure. Amounts in parenthesis are from the Bank's internal portal and subject to currency exchange rate variation. Counterpart Financing: It was expected that the government would contribute US$1 million to the project in counterpart funding. However, CAR was facing a dire fiscal context during project implementation. The legal covenant for counterpart funding was removed during the first restructuring as all parties agreed that the IDA and the Global Financing Facility funds would be sufficient to complete the project. Dates Project approval took place on 27-Sept-2018 and the project became effective 4 months later, on the 7-Feb- 2019. The project was restructured three times: 1. First restructuring took place on 24-Jun-2021 for a change in the results framework when the 2018/19 Multiple Indicator Cluster Surveys (MICS) data became available to recalculate targets, change in project closing date by 9 months from 31-Dec-2021 to 31-Sep-2022, change in Legal Covenants (no counterpart contribution), change in implementation schedule. (Amount disbursed prior to this US$23.92 million, 45 percent). 2. Second restructuring took place on 29-Sep-2022 for a change in closing date by 4 months to 31- Jan-2023 (Amount disbursed prior to this US$48.75 million, 92 percent). 3. Third Restructuring took place on 10-Dec-2022 for reallocation between disbursement categories (Amount disbursed prior to this US$48.58 million, 92 percent) The inconsistency in the US$ value between amount disbursed in Sep-2022 and Dec-2022 is due to currency fluctuations. This project closed on 31-Jan-2023, 13 months later than intended at appraisal. 3. Relevance of Objectives Rationale CAR has a long history of violence related to natural resource disputes and inter-ethnic tensions and is considered a Fragile, Violence and Conflict (FCV) country context (PAD, para 11). At appraisal CAR had the lowest human capital index in the world (PAD, para 6). The health system in CAR was not functioning well with low coverage of essential health services, lack of functional structures and qualified staff, difficulties in accessing health services and medical supplies, and a lack of monitoring and epidemiological surveillance capabilities. The country had the highest rates of maternal mortality (882 per 100,000 live births at appraisal) and under-five mortality (129 per 1,000 live births at appraisal). Malnutrition was an underlying cause of 48 percent of under-five deaths (PAD, para 6). Furthermore, the country’s ongoing conflict has resulted in an epidemic of GBV. The PAD noted 11,110 incidents in 2016 (PAD, para 7) and the ICR noted cases of rape, sexual slavery, physical assault, and kidnapping of women and girls between the ages of 10 and 75 in the country (ICR, para 6). Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) In light of the dire context, the PDO of improved utilization and increased quality of health services was well aligned with the government of CAR's priorities. The latter is evidenced by government policies such as the 2018 Presidential Decree for Targeted Free Healthcare which guaranteed access to basic care, including drugs, for all pregnant, post-partum women, children under-five, and survivors of GBV, the Ministry of Health's budget line in 2019 dedicated to the PBF program (PAD, para 88), and the National Health Development Plan III in 2022 which outlined the path to Universal Health Coverage by 2030. The PDO was well aligned to Bank strategies, notably, the CAR Country Partnership Framework (CPF) for 2021-2025.The first focus area of this CPF is Human Capital and Connectivity to Boost Stabilization, Inclusion and Resilience and includes strategic objective 1.3 Deliver Quality and Inclusive Health Services in recognition of the prerequisite of health to develop human capital. The project's focus towards investing in systems that supported strong governance and the delivery of basic social services to all areas, particularly those outside the capital city of Bangui, is highly relevant and consistent with the Bank's Africa Human Capital Plan which supports strengthened social service systems delivery, and empowerment of women and girls. It is also consistent with the FCV Strategy which strives to strengthen the capacity and legitimacy of core institutions, renewing the social contract and rebuilding trust. The Bank has successfully assisted many developing countries, including CAR, to develop and implement PBF and free healthcare programs (PAD, para 66) and this project builds on the successful implementation experience of the previous Health System Support Project (P119815) in country (PAD, para 76). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase utilization of essential health services in targeted areas in the territory of the Recipient. Rationale Theory of Change The theory of change developed by the appraisal team was sound (PAD, Figure 4). The project activities were inter-connected, and the theory of change is applicable to both objectives discussed in this section. It was reasonably expected that the project's activities would contribute to improving the health status for the population of CAR and building its human capital by: Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) (a) increasing healthcare utilization through mitigating i) supply-side issues by prioritizing delivery of essential health services and community based MCNH services, and ii) increasing demand for health services through reducing costs for the poor and vulnerable, thus promoting access. (b) improving the quality of delivered health services (and further increasing utilization) through i) improved management of health facilities at all levels through PBF and ii) strengthened stewardship by the Ministry of Health through institutional capacity building and technical assistance. One of the critical assumptions noted in the PAD is “poor quality of care was a critical bottleneck for improving utilization” which inherently links the two PDOs (PAD, Figure 4 p.32). Outputs and Intermediate Results The ICR offered evidence from the PBF database, the National Malaria Program database, the Immunization program database, and the National Health Management Information System database. The project achieved the following:  241,955 post-natal care visits were received by women, surpassing both the original target of 173,958 and the revised target of 89,979 post-natal care visits.  71,075 pregnant women received two doses of intermittent preventive treatment for malaria, falling short of the original target of 107,519 and just surpassing the revised target of 69,887. These data were not reported in the PBF data portal (as it was not part of the PBF package of services), and therefore could not be validated at the completion stage of the project.  2,928,737 people received health services free of charge, surpassing the original target of 372,351 which remained unchanged. The project greatly surpassed this target (787 percent) which should have been adjusted during the restructuring to account for it being under-estimated initially but was not.  5 districts hospitals now have integrated GBV Services, surpassing the original target of 3 which remained unchanged.  80 398 pregnant women received at least 4 ante-natal care visits which did not meet the original target of 569,000 or revised target of 251,613. The project did not collect these data until the third quarter of 2020, so the indicator only counts data for the last 7 quarters of the project (out of 13). The project team later added that this was a complex indicator which required tracking the number of visits of the same person whilst the PBF portal was only reporting number of ante-natal care visits, with no way to track the cumulative visits per individual. This was corrected in the PBF portal in 2020" (ICR Review Preparation Interview, 26-Oct-2023). At restructuring in June 2021, the following three intermediate results indicators were proactively removed by the Bank as they related to delivery of community-based health services and the government had not put in place a National Community Health Policy or Strategy.  The number of children 6-59 months who received vitamin A supplementation (twice per year) in the total catchment area: no data were collected.  The number of children 1-5 years old who received a deworming tablet in total catchment area: no data were collected. Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953)  The number of women and adolescents who received family planning services: 342,511 women and adolescents surpassing the original target of 203,331 despite the removal of this indicator. Outcomes  709,774 people received essential health, nutrition, and population services surpassing both the original target of 474,066 and the revised target of 450,640.  496,841 woman received essential health, nutrition, and population surpassing both the original target of 328,725 and the revised target of 315,448.  142,601 children were immunized, surpassing the original target of 98,186 which remained unchanged.  301,633 women and children received basic nutrition services, surpassing both the original target of 192,496 and the revised target of 179,658.  265,859 deliveries were attended by skilled health personnel, surpassing both the original target of 183,384 and the revised target of 172,796.  GBV related Outcomes: The ICR notes that the project was instrumental in increasing utilization of services among GBV survivors (ICR para 29), however, the results framework lacks indicators to support this with the only GBV related indicator being "number of district hospitals that have integrated GBV services". The project team later added that the results framework did not adequately capture the magnitude of funding that went into this component, nor did the indicator capture the positive impacts of this component. The project team emphasized this was a comprehensive health system strengthening project and cautioned that dedicating more indicators to this component risked having too many indicators (ICR Review Preparation Interview, 26-Oct-2023). The ICR goes beyond the results framework to highlight the following GBV associated results i) the development of a national strategy to fight GBV, harmful practices and child marriages, ii) provision of holistic, free care at 5 district hospitals and 21 satellite health facilities for 2,982 GBV survivors, including 1,658 survivors of rape, iii) small scale rehabilitation and training of doctors and psychosocial agents and equipping facilities with post-rape and dignity kits, ambulances and specialized medical equipment, iv) 461 NGO-led community activities which focused on raising awareness about violence, the benefits of health and equitable relationships, and the availability of new services at the district hospitals, v) establishment of a GBV Coordination Committee which conducted ten coordination meetings during the life of the project, two at national level and eight at decentralized levels, to ensure collaboration of all stakeholders. Attribution The ICR comments on attribution (ICR, para 28), noting that the project was the primary financier of health services in the 15 health districts the project covered and as such it can be reasonably assumed that without the project, the basic package of health services, including GBV services, would not have been accessible to many poor and vulnerable households. The ICR does not offer indication of trends in the form of comparing utilization with non-project health districts. The project team later added that it was a major feat to have data in the15 project districts and emphasized that the other areas were mainly supported through humanitarian interventions given that the country is coming out of severe conflict and that little to no data are available as a result (ICR Review Preparation Interview, 26-Oct-2023). Objective 1 is rated high as the project increased utilization of essential health services such as childhood immunization, basic nutrition, and population health services. This is even more impressive given that globally Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) health systems saw a drop in utilization due to COVID-19 which coincides with the project period. Despite several outcome targets revised downward at restructuring, the project's achievement exceeded both the original and revised targets in almost all cases. Rating High OBJECTIVE 2 Objective To improve the quality of essential health services in targeted areas in the territory of the Recipient. Rationale Theory of Change As for objective 1. One of the critical assumptions noted in the PAD is “poor quality of care was a critical bottleneck for improving utilization” (PAD, Figure 4 p.32). The ICR is fair to point out that several studies demonstrate that improved quality of care in facilities is a predictor of utilization of antenatal care and immunization, however, this ICR Review cautions that whilst there may be some correlation, we cannot infer that improved utilization resulted from improved quality, particularly in an FCV setting where there are other demand drivers for care such as extreme unmet need. Outputs and Intermediate Results The project achieved the following:  86 percent completeness of country health data in health management information system, surpassing the original target of 80 percent which remained unchanged.  91 percent availability of tracer drugs at health centers, up from a baseline of 84 percent, and surpassing both the original target of 90 percent and the revised target of 85 percent.  64 percent quality satisfaction score of beneficiaries for health services in district hospitals. However, the results are not comparable to baseline due to adaptations to the tool in the updated PBF manual and limited data collection due to COVID-19. Clear trends in patient satisfaction cannot be established.  No data were collected on percentage of facilities receiving performance payments on time. This was because the PBF portal was not designed in such a way to track the timeliness of payments. This is being addressed in the follow on project with improvements in the portal.  1,272 health professionals, including 250 midwives, were employed full-time in health facilities which did not meet the original target of 4,306 or the revised target of 1,506. At restructuring in June 2021, the following intermediate results indicator was removed as it related to delivery of community-based health services: Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953)  254,941 people referred to health facilities by community workers which did not meet the original target of 292,652 people referred. Outcomes Health facilities reported their quality against a scorecard based on beneficiary surveys, which were then verified by district health teams. 1. 51 percent quality score of health centers in targeted areas which did not meet the original, unchanged target of 80 percent. 2. 81 percent quality score of hospitals in targeted areas which did not meet the original, unchanged target of 85 percent. Data from the results framework are difficult to compare across time because a revised PBF manual was introduced in 2020, which altered the quality indicators that were being measured. Over the course of the project, a decreasing number of health facilities reported these data, making the quality scores unreliable. If one simply looks at the scorecard data there are no consistent trend upwards or downwards (ICR, para 31). The project financed critical inputs to care - including equipment, medicine, infrastructure, and health workers - all of which can reasonably be expected to have had a positive impact on quality, however, evidence for the translation of these inputs into the outcome of quality care is insufficient. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Objective 1: Objective 1 is rated high as the project increased utilization of essential health services such as childhood immunization, basic nutrition, and population health services, which is even more impressive given that globally health systems saw a drop in utilization due to COVID-19 which coincides with the project period. The ICR goes beyond the results framework to highlight the notable achievements of the project associated with GBV service utilization. Objective 2: Objective 2 is rated modest as while the project financed critical inputs to care - including equipment, medicine, infrastructure, and health workers - all of which can reasonably be expected to have had a positive impact on quality, evidence for the translation of these inputs into the outcome of quality care is insufficient. M&E challenges contributed to this and will be further discussed in section 9. Attribution: This project was the primary financier of health services in the 15 health districts the project covered. It can be reasonably assumed that without this project, the basic package of health services financed would be inaccessible to many households because they would have to pay out of pocket and health facilities would have been less capable of care delivery. Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) The overall efficacy is assessed as Substantial at a marginal level, with caveats given that objective two is rated with modest achievement. Overall Efficacy Rating Substantial 5. Efficiency Efficiency is rated Substantial with moderate shortcomings. No quantitative economic analysis was done at appraisal due to a lack of household survey data (last MICS outdated, from 2010), however, the appraisal team selected interventions that were cost effective to be financed by the PBF program, based on experience from low- and middle-income countries (PAD, Table 3). The economic analysis prepared for the ICR (ICR, Annex 4) estimates that the project had a benefit-cost ratio of 1.05-3.14 based on benefits quantified of component 1- averted maternal, neonatal and child (1-59 months) deaths and stillbirths through increased coverage of MNCH services and costs quantified at US$36 million assuming 100 percent disbursement of component 1's budget. The use of the World Health Organization’s (WHO) Choosing Interventions that are Cost-Effective “CHOICE” method of valuing the benefits has resulted in an over estimation of economic value. It should be noted that the use of one to three times gross domestic product per capita as a cost-effectiveness threshold has been widely criticized by experts and that WHO has moved away from it. Despite the lack of consensus, the literature suggests that cost effectiveness thresholds for low-middle-income countries is closer to half gross domestic product per capita and between one to two times gross domestic product per capita for high income countries. Applying half gross domestic product per capita per Disability Adjusted Life Year (DALY) derives a monetary value of US$18.84 million in the 2018 model (Benefit Cost Ratio 0.51) and US$10.95 million in the 2019 model (Benefit Cost Ratio 0.3). Using this approach to valuing life-years saved, suggests the project was not economically viable in either model. However, it should be noted that it is possible that any overestimation in economic benefits due to the use of one to three times gross domestic product per capita, may be offset by the underestimated total benefits underlying the economic value calculation (ICR, Annex 4, para 19). Gross domestic product per capita is incredibly unreliable given CAR’s socio-economic context. There are challenges in using any GDP related measure to assess economic value of lives saved. The ICR's economic evaluation was transparent and showed two different methodologies. Implementation Inefficiency Administrative costs accounted for 35 percent of PBF costs in CAR with verification accounting for 16 percent as the project had to rely on international NGOs for verification due to the low in-country capacity. Relatively high verification costs are commonly associated with PBF. Furthermore, the project was working outside of the capital where population density is low, and people are difficult to reach. The project team later added that whilst working in Bangui may have come with an efficiency saving, it would have risked the relevance of the project. Working in Bangui, the Bank would not have addressed those in most desperate need and would have contributed less to the peace process given that the literature suggests that in FCV settings, addressing neglected areas is a strong contributor to peace. The project experienced a 13-month delay attributed to COVID Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) and the conflict situation. The design of the project was undertaken at the height of PBF popularity and at the time it was standard practice to design PBFs to the international gold standard. Subsequently, there is an understanding that there are adaptions that need to be made to PBF in FCV settings. All procurement activities were completed. Financial management was consistently rated MS. The proceeds were essentially fully disbursed. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the PDOs of improved utilization and increased quality of health services was well aligned to both the government and Bank priorities and strategies in CAR and as a result are rated high. The overall efficacy rating is substantial at a marginal level, with caveats given the marked increase in health service utilization and acknowledging the modest achievement of quality objectives. The efficiency rating is substantial with moderate shortcomings. The outcome rating is moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The risk to development outcomes is high due to factors mentioned in the ICR such as i) the high administrative costs of PBF particularly verification and counter verification, ii) the complexity of PBF which included tracking indicators and monitoring quality and, iii) government lacking a clear strategy and vision on key health system pillars such as human resources, medical supply chain and community-based services. Further threats to development outcomes not mentioned in the ICR include i) consolidation of political power considering the referendum to allow the incumbent president to remain in power and, ii) Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) reliance on international NGOs for verification and provision of holistic GBV services which is threatened by cyclical NGO funding and may hinder capacity building of local expertise. It should be noted that efforts at institutional strengthening were made through the project implementation unit throughout the life of the project. Staff benefited from training related to procurement, financial management, environmental and social safeguards, and project management, however notable work still needs to be done (ICR, para 41). Activities to mitigate the risk to the development outcome include: i) making PBF more contextually appropriate by decreasing the number of indicators tracked, simplifying the quality tools, reducing the number of verification visits needed and increasing the quality of coaching during supervision visits, ii) working actively with the government to ensure that the follow on project relies more heavily on national expertise to reduce costs and ensure better integration into the health system, iii) continued support of the GBV Coordination Committee which is working with multiple stakeholders. The project team later added that there is political will and commitment from the Ministry of Health who describe this project and the follow on project as a locomotive force for development work in the country (ICR Review Preparation Interview, 26- Oct-2023). 8. Assessment of Bank Performance a. Quality-at-Entry The Bank's quality at entry rating has been rated as moderately satisfactory. Preparation was largely adequate with moderate shortcomings. The PBF design could have been more contextually adapted to the operational context, but it built on successful prior experience, including that of the PBF National Technical Unit (PAD, para 77). The design of GBV-related project activities was strategically relevant given the epidemic of GBV and contextually adapted as the experts brought in to assist with the design had contextual knowledge built on Eastern Congo and similar settings (ICR, para 56). Despite the PAD highlighting the importance of GBV, the results framework lacked indicators aside from "Number of districts hospitals with integrated GBV Services" to assess the efficacy of this activity. The design of the second component was better contextualized as it was left broad to allow the Ministry of Health to respond to the evolving, unstable situation and left space for core and urgent activities to be defined based on the country’s needs (PAD, para 37 and ICR, para 42). The design allowed flexibility, and further improvements were realized during implementation. Development partners (European Union, UN agencies, NGOs) were engaged during preparation, which built the foundation for the development of the Investment Case (PAD, para 35 and ICR, para 43). The Ministry of Health support was secured through several missions during preparation, building on the successful implementation experience of the Health System Support Project (P119815) (PAD, para 76). Notably there were no gaps or interruptions in health services between the two projects. Apart from moderate shortcomings, project risks were adequately assessed (ICR, para. 49). Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The lack of country-based Bank staff posed a challenge to supervision and became worse during the COVID-19 pandemic when travel was restricted (ICR, para 82). The project implementation unit was submitting technical terms of reference and budgets that needed in-depth quality review and revisions with substantial back and forth. This long, protracted process affected the ability to rapidly respond. The Bank was understaffed with the project team slow to respond to government requests (ICR, para 83). Supervision improved substantially once two Bangui based health economists were hired in 2021. The Bank's mid-term review for the project was delayed and took place between April 22 and May 3, 2021, which was seven months before the project was initially expected to close. This delay was because as all parties were occupied with the COVID-19 pandemic response. This delay resulted in late restructuring which exacerbated several errors in Bank supervision at restructuring, which include i) failure to adjust the indicator target on "number of people receiving free health services" to account for it being under-estimated at appraisal ii) failure to simplify the quality score card adequately despite the declining trend in facilities providing quality related data iii) failure to consider how the adaptations in the quality scorecard impacted comparability to baseline, and trends in the quality indicators. The initial design of PBF was adapted midway so that both equipment and medicines were procured by the project centrally through a memorandum of understanding with UNICEF and then distributed to facilities due to a lack of a local market. The adapted approach had success near the end of the project, with government, facility staff and Bank supervision missions noting marked improvements. Other adaptions to PBF that should have been considered included addressing the high administrative costs and long verification processes. Two accredited Bangui based staff oversaw financial management and procurement. However, they were managing all projects in the portfolio and were on a rotating schedule whereby they spent two weeks outside of Bangui every 6 weeks. The project faced numerous financial management and procurement challenges that further point to inadequate Bank supervision. These are detailed in section 10. Monitoring of the project’s social safeguard was stronger than environmental aspects, the latter being affected by challenges in medical waste management etc., as discussed in section 10. The inadequate staffing, errors at restructuring particularly with regards to changes made to quality indicators, lack of contextual adaptations to PBF during implementation, serious financial and procurement issues and inadequate reporting for environmental safeguards all point to a moderately unsatisfactory quality of Bank Supervision. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project had a straight-forward theory of change and results framework, with clearly defined PDOs, Intermediate Results Indicators, baseline figures and targets, and sources of data. The technical design of the project draws from the best practices in improving health outcomes, service delivery, quality of care, and health systems management in fragile contexts and targets both demand-side and supply-side barriers to health service utilization. The results framework did not estimate coverage, as there were no solid estimates of the target population. Notably, indicators for GBV related outputs and outcomes were missing with the only GBV related indicator being "number of districts hospitals that have integrated GBV services". As discussed in section 4, the project team later added that this was a comprehensive health system strengthening project and cautioned that dedicating more indicators to this component risked having too many indicators to track (ICR Review Preparation Interview, 26-Oct-2023). There is a need to keep the results framework lean and user friendly in FCV contexts. The M&E design focused on strengthening the national system and avoiding creating a parallel system. Data from all programs were migrated into the health information management system and monthly and quarterly data from the health information management system were aggregated as annual project indicators. b. M&E Implementation There were significant issues with implementation, particularly with verification and counter-verification. Agencies were recruited late (ICR, para 61) and the counter-verification only happened once, which limited the quality of the project’s data. At restructuring in 2021, the Bank team used estimates provided by the project implementation team and not the raw data downloaded from the PBF portal. Using the data provided, approximately half of the service delivery indicators were at risk of falling short of targets, and targets were reduced. Had the raw data from the PBF portal been reviewed the data would have shown that many indicators had met their original targets (ICR, para 61). It should be acknowledged that credibility of PBF data are also questionable as they only captured the verified data which are problematic given the delays of putting in place the verification and counter verification mechanism (ICR Review Preparation Interview, 26-Oct- 2023). The data on health service utilization were complete, but the project faced challenges in collecting data on quality indicators. The revisions to the quality scorecard were not sufficiently considered when the results framework was restructured, which rendered the quality data at the end of the project incomparable to data at the start of the project. Moreover, the patient satisfaction data were too cumbersome for facilities to collect which resulted in the number of health centers that reported quality data decreasing each quarter (ICR, Figure 5). It is unclear if this trend was noted during implementation or only at ICR preparation, and there is no evidence of actions taken by the Bank or project implementation team to address this decline in reporting. There was a misalignment in incentives as district teams responsible for entering quality data received the bulk of their payments irrespective of Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) reporting (ICR, para 63). COVID-19 further complicated quality score card data collection as follow-up in the community was compromised. Armed conflict also disrupted government supervision over this period. Support with data management was provided as the Bank hired and trained managers for the health information management system. Geo-enabled monitoring was used in this project and has now become best practice in CAR Bank projects (ICR para 64-65). c. M&E Utilization Little evidence of M&E utilization is provided in the ICR. The ICR notes that creating a culture of data use takes time. At the request of the Minister of Health, the follow-on project will work on communicating project results more broadly, which will help create greater demand and use of data. M&E Quality Rating Modest 10. Other Issues a. Safeguards Social: The project is rated Category B (Partial Assessment) with a moderate risk and one policy is triggered: OP 4.10 on Indigenous Peoples as the assessment confirms the presence of indigenous communities who meet the requirements of OP 4.10 in some of the zones targeted by the project (PAD, para 134). The project complied with the requirements of OP 4.10 by ensuring that free healthcare was provided to indigenous people within targeted districts. Furthermore, the project gained broad support from indigenous populations during the preparation of the Indigenous Peoples Policy Framework (ICR, para 70). The project implementation unit was staffed with a Social Safeguards Specialist and two GBV experts who routinely conducted supervision visits for project activities. The project implementation unit began safeguards missions as soon as the project was approved and set-up a functional grievance redress mechanism where beneficiaries could call in complaints. However, this was not accessible to all beneficiaries due to limited cell phone use in some of the targeted districts (ICR, para 71). At the time this project was prepared, a Sexual Exploitation Abuse or Harassment (SEA/H) risk assessment and action plan was not required. However, during implementation, the project proactively implemented measures to prevent SEA/H. These included codes of conduct for project implementation unit staff and the integration of clauses to prevent SEA/H into contracts signed with consultants and contractors. One reported GBV incident was linked to a contractor of the project. The incident was reported to the Bank and measures were taken to end work with the contractor and ensure that the survivor received appropriate services (ICR, para 72). Environmental: Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) The project is rated Category B (Partial Assessment) with a Moderate risk and one policy is triggered: OP/BP 4.01. The project was rated Satisfactory on both Environmental and Social until January 2022, at which the ratings were downgraded to Moderately Satisfactory (ICR, para 85). An Environmental Assessment was undertaken, and the project prepared a separate Environmental and Social Management Framework and a Medical Wastes Management Plan as required. An Environmental Safeguards Specialist was hired within the project implementation unit and there was close coordination with the Environmental Division at the Ministry of Health and the Ministry of Environment. Management of medical waste was a challenge throughout project implementation due to lack of incinerators at many facilities and hygiene and sterilization scores had a downward trend from 2019 when the project started (ICR, Figure 6). Monitoring of the project’s social safeguard was stronger than environmental aspects because, inter alia, the environmental specialist was based outside of Bangui until 2021. For the duration of the project, there were updates on the environmental and social safeguards in each supervision Aide Memories. However, there was only one entry in the Integrated Rating Tool in October 2022 for Environmental aspects, whereas there are routine reports for social aspects. b. Fiduciary Compliance The overall financial management risk was rated Substantial at appraisal (PAD, para 107). During implementation, risks were rated Moderate, and performance was consistently rated Moderately Satisfactory (ICR, para 73). Two accredited Bangui based staff oversaw financial management and procurement. However, they were managing all projects in the portfolio and were on a rotating schedule whereby they spent two weeks outside of Bangui every 6 weeks. The project faced numerous financial management and procurement challenges that further point to inadequate Bank supervision (ICR para 74, 77, 78). A financial management mission took place in November 2019, less than one year into project implementation, to assess performance of the project implementation unit and recommendations were made. In October 2021 an in-depth review identified multiple weaknesses and issues of non-compliance related to procurement. Notably, several procurement documents were missing. As it related specifically to an NGO under investigation, the review found that the procurement processes were not carried out as per the regulations and the payments made under these contracts, were not executed in accordance with the contracts’ terms and conditions. The mission further noted that the project implementation unit was not conducting sufficient site visits to verify the presence of the NGO staff and performance of activities financed by their contracts. The financial review and determination of conclusions and actions was ongoing at the time the project was closing (ICR, para 74). In the final months of the project, an urgent four-month extension of the closing date was needed because key procurement contracts related to medical product and equipment were not completed. At this time, it was clear that many contracts had been singed without the World Bank’s review and clearance, and the contracts in question had not been extended before they expired (ICR para 78). At the time of preparing this ICR Review, IEG followed up with the project team and the project team confirmed that only 0.24% of total project disbursement was affected by non-compliant procurement activities and processes and by a lack or absence of oversight over the contracts as indicated by the in- depth review conducted for the specific case. The project team was not aware of other reviews of a similar Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) nature or any decisions which resulted in ineligible expenses that demanded government reimbursement. There were other non-compliant procurement activities, but these did not conclude as ineligible expenses. According to the project team, the Bank made the request to the Borrower to reimburse ineligible project resources using the template approved in the package. The Borrower has not yet reimbursed the Bank despite follow up by the Bank (which, according to the project team, goes to treasury and bypasses sectoral level). c. Unintended impacts (Positive or Negative) An unintended positive impact of the project highlighted in the ICR is institutional change that supports a zero-tolerance approach to sexual exploitation, abuse, and harassment. The Ministry of Health is working to develop codes of conduct and other sector specific policies that will be used with future development partners. The ICR did not explore any unintended negative impacts of the project. d. Other Gender: The project's interventions benefitted women and girls. The high impact health services are linked to reducing maternal mortality, expanding access to family planning, and treating GBV survivors. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The following lessons selected are the most relevant of the six lessons presented in the ICR:  Complex and cumbersome quality tools can result in poor response rates and limit the reliability of quality related data. In this project, patient satisfaction data were too cumbersome for facilities to collect which resulted in the number of health centers that reported quality data decreasing each quarter. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953)  Inadequate contextualization of PBF to FCV can result in high administration costs that divert funds away from the basic inputs of care. In this project, administrative costs accounted for 35 percent of PBF costs as the project had to rely on international NGOs for verification due to the low in-country capacity.  In-country presence of World Bank staff can improve project performance. This project's performance improved whenever support was provided by technical, fiduciary and safeguards staff from the World Bank based in Bangui. IEG offers an additional lesson:  Projects in FCVs are often associated with high operating costs. This is even more apparent when project's work outside of the capital where population density is low, and people are difficult to reach. But the latter can bring potential peace process gains from addressing neglected areas, which must be factored in into coverage decisions. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain Yes. An assessment is recommended with the prospect of distilling lessons for achieving quality enhancements in healthcare services in FCV contexts given the operational challenges. 14. Comments on Quality of ICR The ICR is consistent with the ICR guidelines and results orientated. It offers high quality analysis on the relevance of objectives and efficacy. The ICR is notably candid in its assessment of the project experience, including its experience with financial management and procurement. The cost-effectiveness analysis undertaken was transparent, offered two methodological approaches and discussed its limitations at length. Some methodological concerns were raised in Section 4. The ICR offers high quality, valuable lessons based on project experience. A minor issue was that although no project resources were ultimately disbursed under the Contingent Emergency Response (CERC) component, the ICR should have noted this component. The ICR could have provided better triangulation of evidence and some indication of trends in non-project areas. With a main text at 36 pages, the ICR was not concise. a. Quality of ICR Rating Substantial Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CAR SENI Project (P164953) Page 19 of 19