Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00006313 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-58360, TF-A2561, TF-A2792) ON A CREDIT IN THE AMOUNT OF SDR 105.9 MILLION (US$150.0 MILLION EQUIVALENT) AND A GRANT FROM THE GLOBAL FINANCING FACILITY IN THE AMOUNT OF US$40.0 MILLION AND A GRANT FROM THE JAPAN POLICY AND HUMAN RESOURCES DEVELOPMENT FUND IN THE AMOUNT OF US$1.1 MILLION TO THE REPUBLIC OF KENYA FOR THE TRANSFORMING HEALTH SYSTEMS FOR UNIVERSAL CARE PROJECT May 29, 2024 Health, Nutrition & Population Global Practice Eastern And Southern Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective {Sep 30, 2023}) Currency Unit = Kenya Shilling (KES) KES 148.10 = US$1 US$1.31 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Victoria Kwakwa Country Director: Keith Hansen Regional Director: Daniel Dulitzky Practice Manager: Francisca Ayodeji Akala Task Team Leader(s): Toni Lee Kuguru, Jane Chuma ICR Main Contributor: Josine Umutoni Karangwa ABBREVIATIONS AND ACRONYMS ANC Antenatal Care ASAL Arid and Semi-Arid Land BEmONC Basic Emergency Obstetric and Newborn Care CE Citizen Engagement CEmONC Comprehensive Emergency Obstetric and Newborn Care CERC Contingency Emergency Response Component CHUs Community Health Units COD Cause of Death COVID-19 Coronavirus Disease 2019 CPS Country Partnership Strategy CRS Civil Registration Services CRVS Civil Registration and Vital Statistics DHIS District Health Information System 2 DQA Data Quality Assurance e-CHIS Electronic Community Health Information System e-JHIC Electronic Joint Health Inspection Checklist GFF Global Financing Facility for Women, Children, and Adolescents GoK Government of Kenya HFS Health Financing Strategy HRH Human Resources for Health IDA International Development Association IRI Intermediate Results Indicator KEMSA Kenya Medical Supplies Authority KHIS Kenya Health Information System KHP Kenya Health Policy KMTC Kenya Medical Training College KQMH Kenya Quality Model for Health mCPR Modern Contraceptive Prevalence Rate M&E Monitoring and Evaluation MPDSR Maternal and Perinatal Death Surveillance and Response MTR Midterm Review NEMA National Environment Management Authority NHIF National Health Insurance Fund PCNs Primary Care Networks PDO Project Development Objective PHC Primary Healthcare PHRD Policy and Human Resources Development Fund RBF Results-Based Financing RMNCAH Reproductive, Maternal, Newborn, Child, and Adolescent Health SoE Statements of Expenditure STEP Systematic Tracking of Exchanges in Procurement TF Trust Fund TWG Technical Working Group THS-UCP Transforming Health System for Universal Care UHC Universal Health Coverage VMG Vulnerable and Marginalized Group TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 12 II. OUTCOME .....................................................................................................................15 A. RELEVANCE OF PDOs ............................................................................................................ 15 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 16 C. EFFICIENCY ........................................................................................................................... 22 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 24 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 25 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME .................................26 A. KEY FACTORS DURING PREPARATION ................................................................................... 26 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 27 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ...27 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 27 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 28 C. BANK PERFORMANCE ........................................................................................................... 30 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 32 V. LESSONS AND RECOMMENDATIONS ..............................................................................32 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ............................................................34 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ..........................44 ANNEX 3. PROJECT COST BY COMPONENT ............................................................................47 ANNEX 4. BORROWER COMMENTS .......................................................................................48 The World Bank Transforming Health Systems for Universal Care (P152394) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P152394 Transforming Health Systems for Universal Care Country Financing Instrument Kenya Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Republic of Kenya Ministry of Health Project Development Objective (PDO) Original PDO The project development objective is to improve utilization and quality of primary health care services with a focus on reproductive, maternal, newborn, child, and adolescent health services. Revised PDO The project development objective is to improve utilization and quality of primary health care services with a focus on reproductive, maternal, newborn, child and adolescent health services and to provide immediate and effective response to an eligible crisis or emergency . Page 1 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 150,000,000 150,000,000 130,849,596 IDA-58360 40,000,000 40,000,000 39,244,696 TF-A2561 1,100,000 949,060 949,060 TF-A2792 Total 191,100,000 190,949,060 171,043,352 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 191,100,000 190,949,060 171,043,353 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 15-Jun-2016 04-Jul-2016 25-Mar-2019 30-Sep-2021 30-Sep-2023 Page 2 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 24-Jun-2019 57.43 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Change in Institutional Arrangements 31-Jan-2020 93.23 Change in Components and Cost Reallocation between Disbursement Categories Change in Disbursements Arrangements 20-Oct-2020 114.25 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories 26-Mar-2021 140.74 Change in Loan Closing Date(s) 15-Sep-2021 162.68 Change in Loan Closing Date(s) Reallocation between Disbursement Categories 26-May-2023 173.74 Reallocation between Disbursement Categories Other Change(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 09-Sep-2016 Satisfactory Satisfactory 0 02 17-Mar-2017 Satisfactory Satisfactory 17.50 03 11-Sep-2017 Satisfactory Moderately Satisfactory 17.50 04 21-Mar-2018 Moderately Satisfactory Moderately Satisfactory 23.56 05 27-Sep-2018 Moderately Satisfactory Moderately Satisfactory 39.59 06 20-Dec-2018 Moderately Satisfactory Moderately Satisfactory 40.11 07 23-Jun-2019 Moderately Satisfactory Moderately Unsatisfactory 57.43 Page 3 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) 08 19-Dec-2019 Moderately Satisfactory Moderately Unsatisfactory 83.31 09 12-Jun-2020 Moderately Satisfactory Moderately Unsatisfactory 105.00 10 26-Jan-2021 Moderately Satisfactory Moderately Unsatisfactory 135.21 11 26-Jul-2021 Moderately Satisfactory Moderately Satisfactory 162.25 12 08-Feb-2022 Moderately Satisfactory Moderately Satisfactory 176.55 13 19-Jul-2022 Moderately Satisfactory Moderately Satisfactory 176.70 14 13-Dec-2022 Moderately Satisfactory Moderately Satisfactory 174.12 15 14-Apr-2023 Moderately Satisfactory Moderately Satisfactory 173.74 16 29-Sep-2023 Moderately Satisfactory Moderately Satisfactory 181.72 SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Public Administration - Health 17 Health 83 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Disease Control 1 Pandemic Response 1 Health Systems and Policies 90 Health System Strengthening 30 Reproductive and Maternal Health 30 Child Health 30 Nutrition and Food Security 10 Nutrition 5 Food Security 5 Page 4 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Victoria Kwakwa Country Director: Diarietou Gaye Keith E. Hansen Director: Amit Dar Daniel Dulitzky Practice Manager: Magnus Lindelow Francisca Ayodeji Akala Task Team Leader(s): Yi-Kyoung Lee Toni Lee Kuguru, Jane Chuma ICR Contributing Author: Josine Umutoni Karangwa Page 5 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. The Transforming Health Systems for Universal Care Project (THS-UCP) was designed when Kenya had just embarked on a very ambitious and rapid devolution process. The 2010 Constitution of Kenya reflected the Kenyan people’s desire for equity, transparency, and accountability, including access to basic services and resulted in the fast- tracked devolution of roles and responsibilities, particularly in the health sector, to the 47 newly created counties. With a guaranteed unconditional transfer of national revenue, the county governments were expected to address local needs for devolved services, including health care. The building blocks for devolution were evolving, including intergovernmental structures and mechanisms for intergovernmental cooperation and transfer of resources to deliver on policy priorities. 2. The THS-UCP focus was also guided by the Second Medium Term Plan (MTR Plan 2013–2017) of the Government of Kenya (GoK), which had a strong focus on inclusive economic growth and the Sustainable Development Goals. Aligned to Vision 2030, the MTR Plan highlighted key policy actions, reforms, and programs that would enable Kenya to achieve accelerated and inclusive economic growth. Primary health care (PHC), maternal and child health services, access to clean water and sanitation, and education were priority areas for the Government. The MTR Plan also emphasized full implementation of the devolution process, as required in the Constitution, and prioritized developing the capacity of county governments and improving coordination between the two levels of government. 3. Devolution aimed to improve equity by moving resources closer to the people and promoting accountability by making counties accountable for results. However, at the time of project preparation, early evidence had shown that devolution might also erode recent achievements unless urgent attention was given to the management of the transition and the functionality of the devolved systems and structures. While roles and responsibilities for national and county governments were outlined in the Constitution, the Kenya Health Policy (KHP), and the County Government Act, these needed to be further clarified and capacity needs strengthened to implement their new mandates. 4. The health status of Kenya’s population had improved over the previous decade, but challenges, including considerable inequity, remained. Under-five mortality and infant mortality rates were halved between 2003 and 2014 due to the increased use of essential health services. However, neonatal mortality declined at a slower rate, with more than 42 percent of deaths under 5 years of age occurring in the first month of life. Despite improvements in the nutrition status since 2003, more than one in four children under five were still stunted. The total fertility rate reduced to 3.9 births per woman after a decade of stagnation, but the maternal mortality ratio remained unacceptably high at 362 per 100,000 live births in 2014. Also, teenage pregnancy remained high with 18 percent of girls between the ages of 15 and 19 having begun childbearing. 5. Utilization of essential health services had improved on average, but demand- and supply-side barriers persisted. On the demand side, socio-cultural beliefs and practices, low status of women, poverty, high cost of services (including transportation), long distance to health facilities especially in arid and semi-arid land (ASAL) counties, and poor health provider attitudes impeded the demand for essential services including reproductive, maternal, newborn, child, and adolescent health (RMNCAH) services. On the supply side, key health system barriers included: weak stewardship and Page 6 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) evolving governance structures, inadequate health information and civil registration and vital statistics (CRVS) systems, weak management of human resources for health (HRH), insufficient essential medicines and medical supplies, inadequate and inequitable health care financing, and poor quality of care. 6. Several health financing initiatives had been introduced to better reach the poor, but they were not well coordinated. The abolishment of user fees in all PHC facilities (Levels 2-3) and free maternal care in public health facilities demonstrated the Government’s commitment to universal health coverage (UHC), especially for women and children. With support from the World Bank Group, Kenya was piloting a health insurance subsidy program for the poor. A reproductive health output-based aid voucher program, supported by the German Development Bank (Kreditanstalt für Wiederaufbau, KfW), had been ongoing in selected counties since 2005 to address demand-side barriers and improve women’s access to health care. A results-based financing (RBF) program for a package of core PHC services was being scaled up in 21 ASAL counties under the Kenya Health Sector Support Project (KHSSP, P074091). The Government was also providing health insurance for the elderly and the severely disabled. However, these initiatives were not well coordinated resulting in fragmentation of health financing, inefficient service delivery, duplication, and high operational costs due to different implementation and reporting arrangements. 7. The leading institution implementing insurance schemes, the National Hospital Insurance Fund (NHIF), had been undergoing reforms, but important institutional weaknesses persisted. The Government was in the process of finalizing a Health Financing Strategy (HFS), which identified a prioritized set of policies to address existing health financing challenges. The HFS provided a framework that would enable Kenyans to benefit from their constitutional right to health and move towards UHC. 8. Improved development partner (DP) coordination was critical to ensuring the efficient delivery of PHC services, especially during this transition period. Many DPs, each using different tools, guidelines, and structures, were supporting the delivery of quality PHC with a focus on RMNCAH services, especially in underserved areas. The MoH was finalizing the Kenya Health Sector Partnership Coordination Framework to strengthen harmonization of planning, budgeting, and monitoring of results. This framework was meant to guide partnership coordination of the health sector among all stakeholders. 9. At the time of preparation of the THS-UCP, Kenya was one of the first countries to receive co-financing from the Global Financing Facility for Women, Children, and Adolescents (GFF), with a grant of US$40.0 million from the GFF Trust Fund (TF). An RMNCAH Investment Framework (2016), identifying prioritized bottlenecks and a set of smart evidence- based interventions for scale-up, had been prepared and costed, through an extensive, eight-month multi-stakeholder consultative process. The THS-UCP design and scope reflected priority strategies identified in the RMNCAH Investment Framework to address: (a) disparities and inequitable coverage through investments to underserved populations and areas; (b) prioritized bottlenecks that prevented the delivery and scale-up of proven, high-impact, evidence-based interventions to women, children and adolescents; (c) vital gaps in the health system to support an efficient and effective delivery of high-impact RMNCAH interventions optimizing existing, and mobilizing new public and private sector investments in the health sector; and (d) community engagement to generate demand, promote behavior change, and enhance social accountability. The RMNCAH Investment Framework built on the existing CRVS strategy, and the Government was developing a HFS. Furthermore, evidence-based and high-impact interventions identified in the RMNCAH Investment Framework were expected to inform the development of county annual work plans (AWPs) to address each county’s specific prioritized bottlenecks or areas where they were lagging behind. Under the project, International Development Association (IDA) and GFF funding were proposed to leverage other DP financing including Page 7 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) increased financing from domestic sources and the private sector. Several DPs had agreed to support and coordinate their financing in support of the RMNCAH Investment Framework. Most of the DP support was focused on the underserved counties, identified also as priority counties in the RMNCAH Investment Framework. 10. The Government of Japan provided co-financing to the THS-UCP through the ‘Performance and Results with Improved Monitoring and Evaluation’ window of the Policy and Human Resources Development Fund (PHRD) which aimed to enhance the use of country systems and promote evidence-based decision making by strengthening the monitoring and evaluation (M&E) systems of the country. The PHRD provided tailored support to priority countries to improve the M&E systems and build capacity for ensuring the availability of timely, reliable, and quality data to (a) inform policy actions and evidence-based decision making at the national and sub-national levels and (b) monitor the progress of health programs and projects. The THS-UCP, with support from a PHRD grant (US$1.1 million), aimed to address key GoK M&E system challenges. Theory of Change (Results Chain) 11. A revised Theory of Change (TOC) or results chain was developed ex-post, based on the revised PDO, component descriptions, activities, and Results Framework (RF). The project aimed “To improve utilization and quality of primary health care services with a focus on reproductive, maternal, newborn, child, and adolescent health services and to provide immediate and effective response to an eligible crisis or emergency”. This was achieved by addressing key health challenges, including low budget allocation to health, suboptimal financial protection for the most vulnerable, shortage of quality human resources for health, low data quality, and incomplete vital events registration. By investing in high impact evidence-based interventions the project contributed towards increasing demand for PHC services at community and facility levels, improving service readiness of existing health facilities, and improving institutional capacity and inter- governmental collaboration, for delivery of PHC services. Page 8 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) CHALLENGES ACTIVITIES OUTPUTS SHORT-TERM PROJECT OUTCOMES OUTCOMES Performance-based allocations to Low budget all 47 counties to scale up Improved standardization, allocation to health evidence-based key priority quality, appraisal, and interventions from their AWPs implementation of AWPs across all 47 counties, for improved Inequitable Training for evidence-based AWP primary healthcare results utilization of or formulation + development of access to PHC AWPs appraisal system services Capacity building on planning and budgeting for all 47 counties Increased demand for Increased number of PHC services at Training and equipment of functional CHUs community and facility community health units (CHUs) + levels Engagement of the community Increased number of Improved utilization of Lack of knowledge through CE mechanisms Vulnerable Marginalized PHC services with and information Groups (VMGs) reached focus on RMNCAH services, including for Suboptimal Dissemination of the HFS + A benefit package developed, Service readiness of VMGs financial protection analytical work to inform its costed, and disseminated existing health for the vulnerable implementation facilities Limited capacity of Rehabilitation + equipment of Improved health facility health facilities to health facilities and labs infrastructure + availability of Improved provision of provide essential essential equipment (including immediate and services for COVID-19) effective response to an eligible crisis or emergency Insufficient budget Procurement and distribution of FP Increased availability of FP for Family Planning commodities commodities commodities Shortage of quality Training of midwives through the Increased number of available skilled human HRH KMTC + contracting of health HRH workers Absence of Training and equipment of health Increased number of health systematic facilities inspectors facilities inspected and inspection of meeting safety standards facilities and providers Development and Dissemination of Increased number of health RMNCAH- related strategies, facilities with capacity and tools Incomplete KQMH guidelines, and tools, including for to conduct quality of care MPDSR assessments and MPDSR. Suboptimal regular use of M&E tools Increased number of Improved institutional Operationalization of the sector capacity and inter- Improved quality of operations research M&E framework and KQMH + governmental PHC services with completed training and supportive supervision collaboration, for focus on RMNCAH delivery of PHC services services Low data quality Capacity building on DHIS + DQA Increased number of reports protocols submitted in DHIS2 in a timely manner Incomplete vital Piloting of key CRVS interventions events registration Increased monitoring and timely reporting on vital events Poor knowledge Cross-county Knowledge sharing sharing across and capacity building activities Increased evidence-based counties decision making Page 9 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Project Development Objectives (PDOs) 12. Project Development Objective (from Project Appraisal Document): To improve utilization and quality of primary health care services with a focus on reproductive, maternal, newborn, child, and adolescent health services. 13. The project was expected to benefit the whole population, with the key beneficiaries being women of reproductive age (WRA) including adolescents, and children under five who utilize primary healthcare (PHC) services most. Key Expected Outcomes and Outcome Indicators 14. Key expected outcomes and their associated indicators aligned to each objective are detailed below. Objectives PDO level indicators (Outcomes) To improve utilization of primary 1. Children immunized with the third dose of Pentavalent (Percentage) health care services with a focus on 2. Pregnant women attending at least four antenatal care (ANC) visits (Percentage) reproductive, maternal, newborn, 3. Women between the ages of 15-49 years currently using a modern FP method child, and adolescent health services (Percentage) To improve quality of primary health 4. Births attended by skilled health personnel (Percentage) care services with a focus on 5. Inspected facilities meeting safety standards (Percentage) reproductive, maternal, newborn, 6. Pregnant women attending ANC supplemented with iron and folic acid (IFA) child, and adolescent health services (Percentage) Components 15. The project had three components. 16. Component 1. Improving Primary Health Care Results (US$141.0 million, of which US$106.0 million from IDA, US$35.0 million from GFF: Actual cost at closing US$131.22 million). This component aimed to improve the delivery, utilization, and quality of PHC services at the county level with a focus on RMNCAH services. This component had two parts comprised of support to (a) counties to implement key RMNCAH priorities identified in the county health AWPs; and (b) ensure availability of RMNCAH strategic commodities (i.e., family planning commodities). Each year counties that met the eligibility criteria (e.g increased allocation to health from the previous year) would receive an allocation informed by improved results and the county revenue allocation formula, which is the Government’s formula for sharing revenue amongst the counties. Counties would use these annual allocations to implement identified priorities from their AWPs. 17. More specifically, the component supported the implementation of evidence-based county appropriate supply and demand side interventions, for example: (a) improving the functionality of existing facilities to deliver quality essential PHC services, and (b) increasing demand for services at the community and facility levels by strengthening community units to deliver preventive and promotive health care and engage the community to improve accountability of PHC services through citizen engagement (CE) mechanisms (for example, community dialogue days). In addition to the county allocations, the component supported procurement of RMNCAH strategic commodities through the Kenya Medical Supplies Authority (KEMSA). To ensure sustainable financing for these commodities, while filling the immediate gap, the Page 10 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) annual allocation through the project was structured to taper over the life of the project with the expectation that the Government would gradually increase domestic financing as project funding decreased. 18. Component 2. Strengthening Institutional Capacity (US$28.7 million, of which US$22.6 million from IDA, US$5.0 million from GFF, and US$1.1 million from PHRD: Actual cost at closing US$28.41 million). This component aimed to strengthen institutional capacity to better deliver quality PHC services under Component 1. More specifically, this component focused on three areas under three subcomponents: 19. Subcomponent 2.1. Improving quality of care. This subcomponent aimed to support: (i) the Department of Health Standards, Quality Assurance and Regulations as well as the Health Regulatory Boards to strengthen routine inspections of public and private health facilities; and institutionalize quality assurance towards certification; (ii) the Division of Family Health (DFH) to develop and/or disseminate RMNCAH-related strategies and guidelines, including improving adolescent sexual and reproductive health (ASRH), newborn health and nutrition to address high teenage pregnancy, neonatal morbidities and stunting, and conduct operations research; (iii) the Kenya Medical Training College (KMTC) to strengthen midwifery training. This subcomponent was co-financed by the GFF grant. 20. Subcomponent 2.2. Strengthening Monitoring & Evaluation (M&E) and Civil Registration and Vital Statistics (CRVS). This subcomponent aimed to support the Division of M&E, Health Research Development and Health Informatics to: (i) operationalize the sector M&E framework; (ii) strengthen the Kenya Health Information System (KHIS); and (iii) pilot innovative approaches to improve coverage of vital events registration within the health sector in close collaboration with the Civil Registration Services (CRS). This subcomponent was co-financed by both the GFF grant and PHRD grant. 21. Subcomponent 2.3. Supporting Health Financing Reforms towards Universal Health Coverage (UHC). This subcomponent aimed to support the Division of Health Care Financing to: (i) disseminate the HFS to get buy-in from various stakeholders drawing from the recently completed stakeholder analysis; (ii) conduct analytical work to inform the implementation of HFS and health-financing reforms towards UHC; and (iii) build capacity for UHC leadership at the national and county levels. This sub-component was co-financed by the GFF grant. 22. Component 3. Cross-County and Intergovernmental Collaboration and Project Management (US$11.4 million from IDA: Actual cost at closing US$8.81 million). This component aimed to enhance cross-county and intergovernmental collaboration as well as facilitate and coordinate project implementation. More specifically, this component focused on two areas under two subcomponents: 23. Subcomponent 3.1. Cross-county and Intergovernmental Collaboration. This subcomponent aimed to finance activities that promote cross-county initiatives and intergovernmental collaboration to address common demand- and supply-side barriers. 24. Subcomponent 3.2. Project Management. The project supported project management staff at national and county levels of government, office equipment, operating costs, and logistical services for day-to-day project management. This also included (a) M&E activities; (b) fiduciary activities; and (c) safeguards activities as well as (d) TA and capacity-building activities to support the project sub-technical working group (TWG) under the Intergovernmental Forum for Health in carrying out their responsibilities. Page 11 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) 25. There were six Level II restructurings of the project during implementation which involved revision of the PDO, PDO indicators, and Components. Revised PDOs and Outcome Targets 26. The PDO was revised in October 2020 to: To improve utilization and quality of primary health care services with a focus on reproductive, maternal, newborn, child, and adolescent health services and to provide immediate and effective response to an eligible crisis or emergency. Revised PDO Indicators 27. The results framework was revised twice (in June 2019 and in October 2020).. During the post-midterm review (MTR) restructuring in June 2019, indicators under Objectives 1 and 2 were revised. In October 2020, indicators related to the new Objective 3 were added. Below is the table with the details of the changes made to the Results Framework. Indicator at Appraisal Final changes Objective 1: Improved utilization of primary health care services PDO level indicators Children younger than 1 year who Indicator was dropped and replaced with: Children immunized with the third were fully immunized (Percentage) dose of pentavalent (Percentage). Target was revised from 76% to 84%. Pregnant women attending at least Target was increased from 46% to 52%. four ANC visits (Percentage) Births attended by skilled health Target was increased from 64% to 67%. personnel (Percentage) Women between the age of 15–49 Target was increased from 45% to 52%. years currently using a modern FP method (Percentage) Intermediate Results Indicators Health facilities offering Basic Indicator was dropped due to data quality issues. Emergency Obstetric and Neonatal Care (BEmONC) Services (Percentage) Functional community health units Indicator definition was revised to: Average number of community units (Number) that report through the DHIS. Target was revised from 2,400 to 5,331. Objective 2: Improved quality of primary health care services PDO level indicators Inspected facilities meeting safety Indicator definition was revised to specify that public health facilities and standards (Percentage) correct the score to be 60 percent or more (instead of 61). Definition was revised to: Public health facilities (L2–L4) inspected which achieve at least Page 12 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) 60 or more percent using the JHIC. Pregnant women attending ANC This indicator was upgraded from an intermediate results indicator to a supplemented with IFA (Percentage) PDO-level indicator for quality of care as it reflects the completeness and quality of ANC visits (clinical quality), and/or the availability of iron folic acid at facilities (structural quality). Target was also increased from 40% to 73%. People who have received essential The indicator and its description were revised according to the new health, nutrition, and population guidelines for the corporate indicators. services (Number) [Core Sector Indicator] Intermediate Results Indicators Facilities inspected for The definition was revised to specify that the project will track inspections safety standards in public health facilities. The indicator was maintained but targets were revised from percentage to number to address data quality issues related to the list of facilities not being routinely updated. Facilities submitting complete DHIS Indicator was revised to: Reports submitted to the DHIS in a timely manner data in a timely manner (Percentage) (Percentage). Definition was revised to: Average percentage of forms 710 and 711 submitted to the DHIS by the 15th of the following month. Registration of births (Percentage) Indicator name was revised to: Births registered within 6 months of occurrence (Percentage). Definition was revised to: Births registered at registration office within 6 months of occurrence. RMNCAH-related operations research Indicator name was revised to: RMNCAH- and UHC-related research completed to inform policy/strategy completed to inform policy/strategy. The end target for this indicator was (Cumulative Number) reduced from 3 to 2. Implementing entities submitting the Indicator definition was revised to: Implementing entities submitting the annual Financial Management (FM) annual FM and technical report no later than 45 days after the end of each and technical report on time calendar quarter. Target was increased from 80% to 95%. (Percentage) The indicator “Lessons learned from UHC Phase I documented and disseminated (Yes/No)” was added. Objective 3: Provision of immediate and effective response to an eligible crisis or emergency PDO level indicators The indicator “Percentage of reported suspect COVID-19 cases investigated based on national guidelines” was added. Baseline was 0 and Target set at 80%. Intermediate Results Indicators The indicator “Number of designated laboratories with RT-PCR capacity to test for SARS-CoV-2” was added. Baseline was 2 and Target set at 8. The indicator “Percentage of vulnerable and marginalized communities reached in their indigenous language” was added. Baseline was 0 and Target set at 80%. Page 13 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Revised Components 28. Component 4. Contingent Emergency Response Component (CERC) (US$10.0 million from IDA: Actual cost at closing US$6.9 million). Starting in 2019, the task team provided technical assistance to the Government to strengthen health security and pandemic preparedness through the Kenya Health Systems Strengthening for Universal Health Coverage (P164023). This included the development of a governance framework for pandemic preparedness, a situation analysis of the national policies and frameworks for pandemic preparedness and a health security financing assessment. To enable the Government’s timely response to an emergency, the CERC component was added in January 2020 and activated in March 2020 to respond to the emergence of the COVID-19 pandemic. This component was meant to support four main areas: (a) medical supplies and equipment; (b) response, capacity building and training; (c) quarantine, isolation, and treatment centers; and (d) risk communication and CE during the COVID-19 outbreak. Most of the investment (89 percent) were meant to be for medical supplies and equipment to enhance capacity in testing through purchase of laboratory equipment, reagents, and supplies; and provide protection for health workers through purchase of personal protective equipment. Additionally, the CERC supported case finding and contact tracing by providing operations support for rapid response and contact tracing teams at national and county level. The CERC also funded the quarantine of exposed health workers, the purchase of tents to provide surge capacity for isolation, and communication to sensitize communities and promote behavior change. 29. The component costs were revised through three restructurings (in June 2019, January 2020, and October 2020). The table below details the component costs revisions, and actual costs at project closure. Project components Estimated 1st 2nd 3rd Actual Costs at Allocations at Restructuring Restructuring Restructuring project closure Approval (U$ Allocation Allocation Reallocation (US$ million)* million) of funds Component 1. Improving Primary 150.00 150.00 150.00 141.00 131.22 Health Care Results Component 2. Strengthening 15.10 29.10 28.70 28.70 28.41 Institutional Capacity Component 3. Cross-County and 26.00 12.00 11.40 11.40 8.81 Intergovernmental Collaboration and Project Management Component 4. Contingency 0.00 0.00 1.00 10.00 6.90 Emergency Response Component (CERC) *Approximately US$ 15.76 million has been cancelled. Other Changes 30. A restructuring dated March 26, 2021, included the extension of the closing date of the IDA credit from September 30, 2021 to September 30, 2023 to allow for adequate time to implement all planned project activities, particularly at county level where implementation was delayed by budget processes beyond the control of the project (i.e., consistent delayed enactment of the County Allocation Revenue Act (CARA), lengthy funds flow processes) and restrictions during the COVID-19 pandemic which hampered implementation (e.g., travel and group restrictions). Page 14 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) 31. A restructuring dated September 15, 2021, included an (a) extension of the closing dates of the PHRD grant (1 year) and GFF grant (2 years) to May 31, 2022, and June 30, 2023 respectively, to allow enough time to complete the associated activities, and (b) reallocation of funds across disbursement categories- US$585,000 was reallocated from disbursement category 1(b) to 2(a) (both under Component 2) to accommodate the increased scope of CRVS related activities and community health activities. 32. A restructuring dated May 26, 2023, included a change in the disbursement category allocations and percentages for disbursement categories 1(a) and 2(b) to ensure that the GFF grant would be fully utilized before the grant closing date of June 30, 2023, and to support the nation-wide scale up of the electronic community health information system (e-CHIS). Rationale for Changes and Their Implication on the Original Theory of Change 33. The PDO was revised to reflect the need to include and activate the CERC to respond to the emergence of the COVID-19 pandemic. As indicated above, the results framework was revised twice to make adjustments to indicators and to increase targets to better reflect intended results and the project timeframe. The project restructurings were informed by the evolving Government priorities, data availability, and the emergence of the COVID-19 pandemic. While there were some changes to original components, these were not significant and did not impact the original theory of change of the project. The priorities supported under this operation remained the same. The TOC presented above reflects COVID- related activities and their links to the revised PDO. II. OUTCOME A. RELEVANCE OF PDOs Rating: High Assessment of Relevance of PDOs and Rating 34. The PDO and design of the project were fully aligned with the Country Partnership Strategy (CPS) for Kenya (FY2014–FY2018) at the time of project Appraisal in 2016 and remain fully aligned with the Country Partnership Framework for Kenya (CPF, 2022–-2027). At the time of Appraisal, the project was aligned with (a) the second domain of the CPS engagement which aimed to “protect the vulnerable and help them develop their potential in order to promote shared prosperity”; and (b) the third domain of the CPS which focused on building consistency and equity that has devolution at its core. The project supported both domains by improving delivery, utilization, and quality of PHC services in underserved areas while strengthening equitable service delivery in a devolved setting. The CPS’s strong focus on results and accountability was also well rooted in the project. At the time of project closure, the project was aligned with Objective 4 of the CPF: Shrink disparities in learning and health outcomes. The focus of supporting the GoK’s goal of achieving UHC by 2030 and addressing disparities in access to PHC services and outcomes was aligned to the design of the project which, through a RBF mechanism, empowered counties to implement county specific interventions known to improve delivery of quality PHC services. Page 15 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) 35. The project scope was also consistent with national health policies including the Kenya Health Policy 2014–2030, the Kenya Health Sector Strategic and Investment Plan (KHSSIP) 2014–2018, and the Kenya RMNCAH Investment Framework. At project closure the project was also aligned with the Kenya UHC Policy 2020-2030 and the Kenya Health Financing Strategy 2020-2030, the Kenya Health Sector Strategic and Investment Plan: 2018-2023, the Kenya Vision 2030, and the Kenya Kwanza Manifesto 2022 – both of which prioritize PHC as the pathway towards achieving UHC. 36. Through the revised PDO and the activation of the CERC, the project remained consistent with World Bank Group’s response during the relief phase of support in Kenya, Kenya’s National COVID -19 Contingency Plan, as well as Objective 6 of the CPF “increase household resilience to, and national preparedness for shocks.” 37. The rating of the relevance of the PDO is therefore assessed as High. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome The overall Efficacy of the PDO is rated Substantial. 38. The PDO was assessed against three parts: (i) to improve utilization of PHC services; (ii) to improve quality of PHC services; and (iii) to provide immediate and effective response to an eligible crisis or emergency. The full project results framework is presented in Annex 4. Objective 1: Improved utilization of primary health care services Rating: High 39. This objective was measured using three PDO-level indicators and four IRIs. Progress against the PDO indicators (PDOI) is as follows: (a) PDOI 1 and 2 achieved/surpassed their targets; (b) PDOI 3 is considered substantially achieved. All IRIs achieved/surpassed their target. See Objective 1 results framework table below. Baseline End Target Actual % Value at Achievement project closing Objective 1 : Improved utilization of primary health care services PDO level indicators PDOI 1. Children immunized with the third dose of Pentavalent 79.5 84.0 85.4 131.1 (Percentage) PDOI 2. Pregnant women attending at least four ANC visits (Percentage) 39.7 52.0 53.8 114.6 PDOI 3. Women between the ages of 15-49 years currently using a 47.8 52.0 37.7 NA* modern FP method (Percentage) Intermediate results indicators IRI 1. Functional community units 302 5,331 9,400 180.9 Page 16 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) IRI 2. A benefit package developed, costed, and disseminated (Number) 0 1 1 100.0 IRI 3. People who have received essential health, nutrition, and 0.0 10,060,000 15,394,537 153.0 population (HNP) services (Number) Number of children immunized 0.0 5,400,000 7,639,386 Number of deliveries attended by skilled health personnel 0.0 4,660,000 7,755,151 IRI 4. Lessons learned from UHC Phase I documented and disseminated No Yes Yes 100.0 *Note: see explanation below 40. Prior to the MTR, the task team was made aware of the limitations of PDOI 3 definition in the KHIS and anticipated to use the KDHS data at project closing to confirm the achievement of this indicator. The project prioritized the use of Government information systems, despite their challenges, to ensure accountability and sustainability. Thus, for PDOI 3, the project relied on a proxy indicator for modern contraceptive prevalence rate that was generated from the KHIS. This indicator estimates coverage by counting the frequency of visits to a health facility to assess use of FP commodities. However, this approach does not adequately account for shifts in the national method mix towards long-acting methods which require fewer health facility visits. Efforts over the course of project implementation were made to highlight challenges with the methodology to proxy modern contraceptive prevalence rate (mCPR) with administrative data, and a key action agreed by MoH going forward is to include new metrics (e.g., estimated modern use), revise data collection tools, and strengthen validation procedures for data in the KHIS. Therefore, this ICR considered the data from the KDHS to assess the true achievement of this indicator during the lifetime of the project. The KDHS demonstrated an increase in the percentage of women between the ages of 15-49 years currently using a modern FP method from 39.1 percent (KDHS 2014) to 42.0 percent (KDHS 2022) for all women. This positive trend is also consistent with the trends observed in the Performance Monitoring for Action (PMA) survey where mCPR increased from 42.0 percent to 46.0 percent in the same period among all women of reproductive age. See additional data in table below. The project results framework had targeted a 4.2 percent increase for this indicator over the life of the project which is aligned to the 4.0 percent increase reported in the PMA survey and the 3.0 percent increase reported in KDHS over what can be considered a similar period of review. Therefore, while the data from KHIS may appear to signal that this indicator, was not achieved, based on the analysis of more reliable estimates of the same indicator, this ICR considers the PDOI 3 as substantially achieved. Indicator Source Baseline value Actual value (project (project design) closing) Women between the ages of 15-49 years KDHS 53 (2014) married 57 (2022) married currently using a modern FP method (Percentage) women women 42 (2022)- all 39.1 (2014) – all women women Women between the ages of 15-49 years PMA 42 (all women) 46 (all women) currently using a modern FP method (Percentage) 53 (married women) 61 (married women) 41. Key activities which contributed to improving utilization of PHC services (Objective 1) and the related outcomes: • Strong focus on results using a performance-based approach. Under Component 1, counties received annual performance-based allocations based on achievement of verified results to support priorities identified in their Page 17 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) AWPs that aimed to improve access to and utilization of PHC services. By shifting the focus from inputs to results, county governments were incentivized to implement high-impact RMNCAH interventions, closely monitor key RMNCAH coverage indicators, and course correct if needed to improve utilization of PHC services. Furthermore, this approach provided flexibility to counties to implement context-specific interventions recognizing the epidemiological and geographical variation across all 47 counties. Approximately US$111 million was disbursed to all 47 counties. • Improved quality of AWPs. The project contributed to (a) standardize and harmonize planning guidelines and tools including templates for the counties; (b) develop a quality assurance/appraisal system of AWPs; (c) build evidence-based planning and budgeting capacity for counties with support from the Kenya School of Government and DPs; and (d) coordinate DPs providing technical support in planning and budgeting to avoid duplication of support and ensure that all counties are covered. • Increased county budget allocation to health. To ensure project funds under Component 1 complemented domestic resources allocated to health and did not crowd out domestic financing, the project included an eligibility criterion whereby counties were mandated to increase every year their allocation to health, by up to 30.0 percent. Commitments to increasing domestic resources for health improved with nearly 80% of counties allocating at least 30.0 percent of their budget to health in FY 2022, a massive jump from only 19.0 percent in FY2015. These levels have now become the standard and reference threshold for county governments and continue to be reinforced through other programs supported by DPs. • Improved functionality of existing health facilities. The project support included: (a) equipment of 5 skilled labs in KMTC campus; (b) renovation of maternity wings; (c) equipment of regional blood transfusion centers; and (d) procurement and installation of waste treatment equipment. • Increased demand for services at the community and facility level. The project support included capacity building and equipment of CHUs, as well as organization of community dialogue days to engage the community to improve accountability of PHC services and healthcare seeking behaviors. • Increased availability of human resources for health with an equity lens. In order to address critical shortages of skilled health workers especially in the ASAL counties, the project targeted training of students from VMGs and overall trained a total of 784 enrolled community health nurses at the KMTC, all of which have passed the nursing council exams. Out of 399 graduates that were surveyed, 138 were fully employed in the health sector. Several VMG communities reported that having a nurse who understood their language and culture encouraged community members to attend clinics and respond to health messaging. The project also supported training of other existing healthcare workers. • Increased availability of RMNCAH strategic commodities. With co-financing from the GFF, a total of US$19.9 million was used to procure FP products and 99.0 percent of the value of these products were distributed to health facilities. Contraceptives representing more than 6.2 million couple years of protection were distributed to health facilities across all 47 counties by KEMSA. At project closure there was less than 3.0 percent of the total procurement in excess stock of combined oral contraceptive pills which were disposed of following standard procedures. To ensure sustainability, the annual project funds allocation was reduced over the lifetime of the project with the understanding that the GoK would gradually increase its domestic contributions. • Improved equitable utilization of health services. The project had an equity focus and supported the development and implementation of vulnerable and marginalized groups plans (VMGPs) which increased awareness of VMG priorities to county leadership. These plans Included key priority interventions aimed at improving utilization of health services for VMGs. At project closure, three activities from the VMGPs were not completed due to delays in finalizing the environmental safeguards documents and procurement processes. This resulted in the downgrading of the Overall Safeguards Rating from Moderately Satisfactory to Moderately Page 18 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Unsatisfactory. However, overall, the project supported implementation of 52 VMGPs with 99.0 percent of the activities were completed. Objective 2: Improved quality of primary health care services Rating: Substantial 42. This objective was measured using three PDO-level indicators and six IRIs. Progress against the PDOIs is as follows: (a) PDOI 4 and 6 surpassed their targets; (b) PDOI 5 is considered as partially achieved. All IRIs achieved/surpassed their targets. See Objective 2 results framework table below. Baseline End Actual Value % Target at project Achievement closing Objective 2: Improved quality of primary health care services PDO level indicators PDOI 4. Births attended by skilled health personnel (Percentage) 57.0 67.0 75.5 185.0 PDOI 5. Inspected facilities meeting safety standards (Percentage) 0.0 50.0 37.0 74.0 PDOI 6. Pregnant women attending ANC supplemented with IFA 31.0 73.0 74.0 102.4 (Percentage) Intermediate results indicators IRI 5. Facilities inspected for safety standards (Number) 0.0 1,635 6932 424.0 IRI 6. RMNCAH related operations research completed to inform 0 2 2 100 policy/strategy (Number) IRI 7. Reports submitted to DHIS in a timely manner (Percentage) 88.8 98.0 98.4 104.3 IRI 8. Births registered within 6 months of occurrence (Percentage) 65.9 80.0 80.6 104.2 IRI 9. Grievances registered related to delivery of project benefits that are 0.0 80.0 100.0 125.0 addressed (Percentage) IRI 10. Implementing entities submitting the annual FM and technical 0.0 95.0 100.0 105.3 report on time (Percentage) 43. The number of facilities inspected for safety standards (IRI 5) has increased throughout project implementation and surpassed the target set in the project RF. However, when measuring the percentage of inspected facilities meeting safety standards (PDOI 5), the target was not fully achieved due to reasons outside of the WB task team control. The project supported the development and scale-up of a standard health facilities inspection checklist and the transition from a manual checklist to an electronic joint health inspection checklist (e-JHIC) for more efficient implementation and real- time monitoring of inspection results. For sustainability, the project provided capacity building to the Kenya Health Professions Oversight Authority (KHPOA) staff on the e-JHIC to ensure the system remains functional beyond the lifetime of the project. At project closure, the transition from the manual checklist to the e-JHIC in all facilities was still ongoing due to the following challenges: (a) delays in scale-up and implementation of the facility inspections; (b) challenges in migrating the inspections data collection tool from paper-based to the electronic joint health inspection checklist (e-JHIC); (c) low uptake of inspections in most counties; (d) logistical support challenges for inspectors; (e) high turnover of joint health inspectors (JHIs) at county level; (f) under reporting due to usage of manual inspection tools in some facilities, and Page 19 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) (g) low number of re-inspections. It is highly probable that the data reported for PDOI 5 is likely an under-estimation of the impact of the project on the increase in percentage of inspected health facilities meeting the safety standards. It is also important to highlight, as detailed earlier under objective 1, that the project financed critical healthcare inputs including FP commodities, equipment, infrastructure, and HRH, which all had a positive impact on quality of care. Studies1 have demonstrated that improved quality of care is a predictor for utilization of antenatal care and immunization services. Thus, the improvements in PDOI 1 and 2 can also be interpreted as being correlated with improvements in quality of care. 44. Key activities which contributed to improving quality of care (Objective 2) and the related outcomes: • Improved inspections of health facilities. The project supported the scale-up of the e-JHIC and inspection of a total of 6932 facilities in all 47 counties. The project also supported (a) development of an e-JHIC data collection tools and dashboard; (b) biannual review meetings with JHIs, focal persons, and County Directors of Health; (c) training of JHIs; (d) procurement of inspection tablets; and (e) distribution of summary inspection books for quality-of-care improvement. • Improved capacity for Maternal and Perinatal Death Surveillance and Response (MPDSR). The project supported the dissemination of MPDSR tools to health facilities and implementation of the revised MPDSR guidelines through training of County MPDSR Committees and cascading the trainings to county and sub-county levels. • Institutionalization of KQMH. The project supported development of the Quality-of-Care Certification and Accreditation Framework for the Kenya Health Sector to underpin the provision of quality health services. The framework defines the process of ensuring delivery of quality health services to the Kenyan population including the compliance to regulatory, service delivery and quality improvement standards. • Increased development/dissemination of RMNCAH-related strategies and guidelines. The project supported two operations research: (a)The Kangaroo Mother Care Report, which demonstrated different capacities for KMC across levels of care, facility type (government and faith-based) and (b) the research on the Uptake of Iron and Folic Acid Supplements (IFAS) among women of reproductive age. • Health Financing reforms towards UHC. The project supported the development of three bills which came into law on November 2, 2023 – (a) Facility Improvement Fund Act which permits government health facilities to retain and use their own source revenue with county governments providing complementary funding, (b) Primary Healthcare (PHC) Act which provides a framework for financing PHC services through establishment of a PHC Fund, entrenching the role of community health providers and providing a management framework for the identification, composition and operationalization of primary care networks (PCNs) and community health units; and (c) the Social Health Insurance Bill. Additionally, the project supported the development of the National Health Insurance Fund (NHIF) Regulations which expanded coverage to the informal sector by providing a framework for their voluntary enrollment to the NHIF. Finally, the project supported workshops where counties shared experiences in implementing PHC reforms at sub-county level and reviewed key data points for monitoring establishment and functionality of PCNs. • Strengthened health information systems and M&E. Through the PHRD grant, the project supported (a) the reactivation and capacity building of the M&E TWG; (b) M&E capacity assessment to document national and county capacities in M&E and highlighting the gaps that may hinder the health sector to develop good performance measurements and track progress towards identified goals; (c) development of the health sector M&E Investment Case which served as a business plan to address the gaps identified in the assessment; and (d) annual joint M&E review meetings. The project also supported the transition from the tenth to the eleventh 1Audo M, Ferguson A, Njoroge P. Quality of health care and its effects in the utilization of maternal and child health services in Kenya. East Afr. Med. J. 2005;82(11):547. Page 20 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) version of the International Classification of Disease for recording of cause of death (COD): support to developing a training curriculum in COD certification and coding; and training of certifiers and International Classification of Diseases (ICD)-11 coders. These activities supported the institutionalization of M&E and evidence-based decision making at all levels, including increases in reporting rates and quality of data. • Improved coverage of vital events registration (CRVS). With co-financing from the GFF, the project supported the CRS in (a) sensitization of county health management team members (CHMTs) to accelerate registration of births; (b) supportive supervision visits to verify quality of data and level of performance within the health facilities; (c) consultative meetings for adoption and use of the international medical form of medical certificate of cause of death and community death notification; and (d) a mobile registration pilot in Narok and Tran Nzoia counties: a total of 8,100 birth certificates were issued. Overall, the project supported the training of 570 chiefs and assistant chiefs as well as 450 health personnel in birth and death registration processes. The overall outcomes achieved for CRVS were the increase in birth registration completeness from 65.9 to 80.6 percent between 2015 and 2022 and slight increase in death registration completeness from 45.4 to 47.6 percent within the same period. • Strengthened intergovernmental collaboration. The project strengthened collaboration between counties and the national government by supporting the implementation of five cross-county and intergovernmental collaborations proposals to address common demand- and supply-side barriers to provision of quality PHC services. These focused on improving (a) capacity of blood transfusion services (e.g procurement of equipment and supplies); (b) quality of adolescent sexual & reproductive health interventions; and (c) utilization of RMNCAH services. The reports and lessons learned were shared through the Intergovernmental Forum for Health to facilitate cross county learning. Objective 3: Provision of immediate and effective response to an eligible crisis or emergency Rating: High 45. This objective was measured using one PDO level indicator and two IRIs. PDOI 7 and all associated IRIs surpassed their target. See Objective 3 results framework table below. Baseline End Actual Value % Target at project Achievement closing Objective 3: Provision of immediate and effective response to an eligible crisis or emergency PDO level indicators PDOI 7. Reported suspected cases of COVID-19 cases investigated based on 0.0 80.0 100.0 125.0 national guidelines (Percentage) Intermediate results indicators IRI 11. Designated laboratories with COVID diagnostic equipment (Number) 2 8 412 650.0 IRI 12. Vulnerable and marginalized communities reached in their 0.0 80.0 100.0 125.0 indigenous language (Percentage) 2This IRI is mapped to the government's response to the COVID-19 pandemic. When the COVID-19 Health Emergency Response Project (CHERP, P173820) and CERC were designed, there were only 8 designated laboratories for COVID-19 testing. The number of laboratories equipped with diagnostic equipment has since increased to 41 laboratories. Page 21 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) 46. Key activities which contributed to the provision of immediate and effective response to an eligible crisis or emergency (Objective 3) and the related outcomes: • Increased availability of supplies. The project supported procurement of various medical equipment including 22 ventilators, assorted laboratory reagents and consumables, 11,430 personal protective equipment kits, 38,000 surgical non-sterile gowns, 28,000 N95 masks, and 60 rapid deployment quarantine tents. • Increased functionality of the emergency operations center and rapid response teams through procurement and distribution of 24-hours ambulance services, provision of fuel and communication tools at both national and county level, and other needed operational costs. • Improved risk communication through diffusion of health promotion messages via various communication tools including posters, banners, and posters. The extend to which the messages reached the community including VMGs were assessed through an annual knowledge, Attitude and Practice survey. 47. Based on the achievement of the three project objectives, the overall rating for Efficacy is Substantial given few minor shortcomings although very significant achievements were made under the operation. C. EFFICIENCY Assessment of Efficiency and Rating: Substantial Allocative efficiency and cost benefit analysis: Rating: High 48. A cost benefit analysis was conducted at Appraisal and demonstrated that the project was a sound economic investment. At Appraisal, the value of the project’s benefit was US$954.2 million, and the cost was US$174.9 million. The net benefit was US$779.2 million with a benefit-to-cost ratio of 5.46:1, meaning a return of US$5.46 for every dollar invested. Sensitivity analysis suggested that the project would still be economically viable even if it only achieved half of the benefits estimated. The project’s efficacy rating of Substantial suggest it is it reasonable to assume that the benefits of the project were attained. 49. The project contributed to a reduction in maternal morbidity and mortality and increased child survival by contributing to increases in key RMNCAH coverage indicators including ANC4, use of FP commodities, and skilled birth attendance. Additionally, the project supported the strengthening of the community health services which contributed to improving the uptake of these high impact interventions. Reduced maternal mortality and morbidity enhance current and future productivity and improve the quality of life for women and their families. Finally, the project contributed to an increase in immunization which has been well documented to be one of the most cost- effective public health interventions, averting an estimated 4.4 million deaths yearly. Investing US$1 in childhood vaccination can potentially yield a return on investment of US$20 in low- and middle-income countries between 2021 and 2030.3 50. The cost of activities such as the selection of COVID-19 interventions supported by the CERC was informed by evidence-based guidance tailored to the Kenyan context by relevant entities such as the National COVID-19 Task 3 https://data.unicef.org/topic/child-health/immunization/ Page 22 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Force, which likely ensured the value for money of these interventions. Cost-effectiveness analysis of providing personal protective equipment to healthcare workers against COVID-19 suggested 11.04 times return on investments.4 Analysis of providing advanced care equipment such as ventilators identified these as less cost effective when compared against investments in essential care services but clarified that the findings were more informative of the sequencing of investment rather than the actual benefits of advanced care.5 The direct costs of healthcare worker deaths in the first year of the pandemic, which advanced equipment could have contributed to averting, were estimated to be as high as US$5.22 million and contributing to a total economic cost of US$113.20 million – about 17.98 times Kenya’s GDP per capita.6 Implementation efficiency Rating: Modest 51. Project implementation was guided by the Project Operations Manual. At project closing 91.00 percent of the total project commitments were disbursed. This includes 88.60 percent disbursement of the IDA credit and 98.20 percent disbursement of the grants (PHRD and GFF grants). The project became effective in 2016 during the early stage of devolution and played a pivotal role in building the capacity of the newly created county governments which became responsible for the delivery of health services. To enhance government ownership and sustainability, the project strengthened nascent devolved country processes/systems rather than create parallel, project-specific processes. During preparation, task team was aware of the challenges in this context associated with implementing a project in all 47 counties with county level results-based financing design using Government systems. During implementation the task team took all the necessary measures within their control to strengthen capacity at the county level, provide the needed trainings, and apply flexibility measures within WB guidelines to ensure smooth project implementation. Overall project implementation efficiency was suboptimal due to several factors outside of the WB task team control: • Implementation across all 47 counties with variable needs and capacities. At the time of project preparation, Kenya had recently devolved resulting in 47 autonomous county governments with health being the most devolved sector. To ensure equitable access to project support, county governments and the Council of Governors firmly advocated that the project be implemented across all 47 counties despite recommendations of the World Bank task team and management to focus on high burden counties identified in the RMNAH Investment Framework. In practice, this resulted in implementing, coordinating, and overseeing 48 sub-projects: 1 MoH sub-project and 47 county sub-projects. This led to unsustainable high administrative, operational, and monitoring transaction costs for both the PMT and the World Bank task team providing implementation support and oversight. • High staff turnover requiring continuous training at county level. Throughout implementation, nearly half 4 Kazungu, J., Munge, K., Werner, K., Risko, N., Vecino-Ortiz, A. I., & Were, V. (2021). Examining the cost-effectiveness of personal protective equipment for formal healthcare workers in Kenya during the COVID-19 pandemic. BMC health services research, 21(1), 992. https://doi.org/10.1186/s12913-021-07015-w 5 Kairu A, Were V, Isaaka L, Agweyu A, Aketch S, Barasa E. Modelling the cost-effectiveness of essential and advanced critical care for COVID-19 patients in Kenya. BMJ Glob Health. 2021 Dec;6(12):e007168. doi: 10.1136/bmjgh-2021-007168. PMID: 34876459; PMCID: PMC8655343. 6 Wang, Huihui; Zeng, Wu; Munge Kabubei, Kenneth; Rasanathan, Jennifer; Kazungu, Jacob; Ginindza, Sandile; Mtshali, Sifiso; Salinas, Luis E.; McClelland, Amanda; Buissonniere, Marine; Lee, Christopher T.; Chuma, Jane; Veillard, Jeremy; Matsebula, Thulani; Chopra, Mickey. 2023. The Economic Burden of SARS-CoV-2 Infection Amongst Health Care Workers in the First Year of the Pandemic in Kenya, Colombia, Eswatini, and South Africa. © Washington, DC: World Bank. http://hdl.handle.net/10986/40070 License: CC BY-NC 3.0 IGO Page 23 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) of the counties (22) went through at least three changes in county procurement officers, resulting in the need to continuously train the county procurement officers in World Bank procurement guidelines and use of the Systematic Tracking of Exchanges in Procurement (STEP) which contributed to implementation delays. Similarly, eight counties had changed their accountants at least three times throughout implementation which adversely affected financial reporting and inconsistencies in accounting information. During the August 2022 elections, over 80.0 percent of counties changed staff members in their project implementation units with restructuring of the Health and Finance Departments. This led to delays in implementation. • Delayed transfer of the county allocations which resulted in a shortened implementation period within each financial year. Based on the experience of the RBF pilot and scale up implemented under the closed KHSSP, a condition was included at design to improve the flow of funds within county level. However, funds flow from national level to county level was affected by considerable delays which was a cross-cutting issue affecting all World Bank supported projects which were implemented at the devolved level. For the most part throughout project implementation, for any given fiscal year, project funds were transferred to the counties between December and January, meaning that there was less than a year for absorption of funds and implementation. This contributed to the suboptimal absorption of funds at the county level. • Delays in expenditure documentation at county level. Throughout project implementation, there were significant delays in documentation in client connection which affected future disbursements as the project used the Statements of Expenditure (SoE) method of disbursement. 52. At project closure, not all IDA and TFs project resources were fully utilized: • Unspent balances at the county level. At project closure, there was an estimated US$800,000 of unspent funds that will be returned to the World Bank. The August 2022 elections delayed budgetary processes at county level also contributed to this suboptimal absorption of funds at county level. • Cancellation of unutilized funds. A total of estimated US$15,014,164 IDA credit were cancelled. Similarly, US$755,303 from the GFF grant were cancelled. 53. On balance, the Rating for Efficiency is Substantial given the very high cost-benefit of the operation and modest implementation efficiency. The project was still able to disburse 91.00 percent and achieve its objectives. It is also important to note that these achievements were made during the height of the COVID pandemic and this is a testament to extraordinary efforts made by the implementing teams. D. JUSTIFICATION OF OVERALL OUTCOME RATING 54. Overall Outcome rating is Satisfactory, given the High rating for Relevance, the Substantial rating for Efficacy (with only very minor shortcomings), and the Substantial rating for Efficiency. Despite implementation challenges it is noteworthy to recognize that this did not prevent the project from reaching its goals and achieving most of its objectives within the original project timeframe. Page 24 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 55. The THS-UCP interventions benefited women and girls. The project contributed to improve access to and quality of health services for women: • The percentage of women attending at least four ANC visits increased from 39.7 percent (KHIS, 2015) to 51.9 percent (KHIS, 2023). • The percentage of married women between the ages of 15-49 years currently using a modern FP method increased from 53.0 percent (KDHS, 2014) to 57.0 percent (KDHS, 2022). • The percentage of pregnant women attending ANC supplemented with IFA increased from 31.0 percent (KHIS, 2015) to 80.3 percent (KHIS, 2023). • The percentage of births attended by a skilled health personnel increased from 57.0 percent (KHIS, 2015) to 75.5 percent (KHIS, 2023). 56. The project has also benefited women from vulnerable and marginalized communities. The project supported a total of 784 enrolled community health nurses at the KMTC, all of which have passed the nursing council exams, among which 464 were women. Institutional Strengthening 57. The PMT benefitted from training during the lifetime of the project. Staff received training related to procurement, financial management, and environmental and social safeguards. Similar trainings necessary for adequate project implementation and management were also provided at county level. 58. The project also contributed to strengthening institutional capacity in the areas of planning and budgeting, quality of care and safety standards assessments at the facility level, monitoring and reporting on vital events, health management information system data quality, and operationalization of the sector M&E framework. Poverty Reduction and Shared Prosperity 59. Evidence shows that increased access to health services is critical to improving living standards and economic growth. The project had an equity focus targeting VMGs and supported their key development priorities and raised the awareness of these to county leadership. The project also targeted students from VMG communities as Community Health Enrolled Nurses at the KMTC so that they could return to their respective underserved communities. Other Unintended Outcomes and Impacts 60. The THS-UCP, being implemented in all 47 counties, was one of the frontrunner projects in implementing a project at the decentralized level and influenced the design of other projects in Kenya. Lessons learned from implementation of Page 25 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) the THS-UCP continue to influence other WB projects in the Kenya portfolio including the design of the Building Resilient and Responsive Health Systems (BREHS) project which aims to improve utilization and quality of primary healthcare services and strengthen institutional capacity for service delivery. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 61. The GoK demonstrated strong commitment and leadership throughout preparation of the project. Project preparation was highly consultative involving engagement with both levels of Government and relevant departments, key partners, and other stakeholders. The project design leveraged the lessons learned from the KHSSP which closed on June 30, 2018. For example, design and implementation experience from the RBF pilot and scale up informed the design of Component 1 of THS-UCP. 62. The THS-UCP was one of multiple financing instruments for the RMNCAH Investment Case Framework, which was intended to be used to coordinate donor support. The THS-UCP was prepared in close collaboration with and was co- financed by the GFF and JICA which ensured complementarity in their support to the GoK even outside of the project. The scope and results framework of the project were largely aligned to the RMNCAH Investment Framework. The project results framework was informed by the availability of routine data from the KHIS. 63. During preparation, discussions were underway with key DPs including USAID, DFID and Danida to support implementation by providing technical assistance to the MoH and counties to strengthen capacity in planning and budgeting, appraising performance, M&E and supply chain management. A multi-donor TF (DFID and Danida) and single- donor TF (USAID) were established in 2017 to enhance effectiveness in achieving sustainable RMNCAH results by strengthening health systems and supporting progress towards UHC. The TFs were administered by the World Bank to operate alongside the project and provide implementation support. 64. The mandate of the CoG was to provide a mechanism for consultation amongst County Governments, share information on performance of the counties in execution of their functions, facilitate capacity building for Governors, and consider reports from other intergovernmental forums on national and county interests amongst other functions. The CoG served as the coordinating entity for the newly created county governments and coordinated dialogue. Therefore, during project preparation, it was agreed that the Component 1 Coordinator and Assistant Coordinators be based within the CoG headquarters to provide hands on coordination, implementation, and monitoring support to the county governments. Page 26 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) B. KEY FACTORS DURING IMPLEMENTATION Factors within Government control 65. Throughout project implementation, delayed transfer of the county allocations from national to county level resulted in a shortened implementation period within each financial year, and high staff turnover at county level required re-engagement and continuous training. The latter was especially the case after the 2 presidential elections which took place in year 2017 and 2022. Similarly, the start of project implementation was delayed due to a loan lapse in another project which put a freeze on the entire Kenya portfolio. Nonetheless, the MoH and county governments remained committed to the implementation of the project and the UHC agenda which aligned to the project objective. 66. Overall, the PMT was dedicated to the coordination and management of the project. Critical to the implementation of county activities, a Component 1 Coordinator and Assistant Coordinators were maintained at the CoG headquarters. Nonetheless, due to the scale and complexity of the project, particularly Component 1, the PMT could have benefited from additional fiduciary support. Factors subject to World Bank control 67. The World Bank task team was responsive and processed six restructurings throughout the lifetime of the project to course correct and make the necessary adjustments based on the country needs, priorities, and availability of data. It is noteworthy that core members of the preparation team remained as the World Bank task team throughout implementation and closing. Throughout most of implementation, the Task Team Leaders were based in country and able to provide timely support and guidance needed for such a complex project. Nonetheless, due to the scale and complexity of the project, particularly Component 1, the World Bank task team could have benefited from additional fiduciary support. It is also noteworthy that while THS-UCP was under implementation, the same World Bank task team prepared and provided implementation support to the Kenya COVID-19 Health Emergency Project (P173280) which included 2 additional financings. Factors outside of Government control 68. The COVID-19 pandemic necessitated restructuring to revise the project PDO and reallocate resources to the CERC to address the evolving country needs. This reallocation provided critical inputs without negatively affecting the achievement of the other project objectives. During the initial stage of the COVID-19 pandemic, implementation was affected by the mitigation measures introduced by the GoK (as was the case in most countries) to prevent and minimize the spread of COVID-19 (e.g., travel and group restrictions). IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 69. The M&E design is overall rated Substantial. Throughout project implementation, all key activities, outputs, and results related to all three project objectives were comprehensively documented in the AMs and sufficient information and data was available for preparation of this ICR. Shortcomings of the available routine data source, the KHIS, were also discussed with Government during project design and implementation and documented in the Aide Memoires (AMs) and were outside of the World Bank team’s control. Page 27 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) M&E Implementation 70. The M&E implementation is rated Substantial. The project results framework relied on routinely collected data from the KHIS (despite its limitations) and project reports. This ensured that data was available throughout project implementation for review at least every 6 months during implementation support missions and kept Government accountable for the results. Where KHIS fell short, and external data was also available, for example for modern contraceptive prevalence rate, additional data was provided in the AMs to show progress. At project closing, the KDHS 2022 also served as an additional data source to measure achievement of the PDO. M&E Utilization 71. The M&E utilization is rated High. Throughout project implementation, review of the results framework data was used to monitor progress towards achievement of the PDOs and course correct, as needed. RMNCAH coverage data was captured in the RMNCAH scorecards (which was informed by data from KHIS) which were used to monitor progress. The RMNCAH MDTF provided M&E TA to support counties to utilize the data during development of the annual work plans. At the project’s midterm review, the RF was revised to (a) ensure that it adequately reflects the PDO; (b) reflect updated baseline values that were revised after project preparation; (c) replace indicators with suboptimal data quality with more reliable indicators; and (d) review targets based on the achievement analysis. See details under Significant Changes During Implementation section. Justification of Overall Rating of Quality of M&E 72. The overall rating for Quality of M&E is Substantial. Despite a few shortcomings, the M&E plan allowed for adequate monitoring of the project PDOs and data and information were available to assess the true impact of the project. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Social Safeguards compliance 73. Throughout project implementation, overall safeguards rating was Moderately Satisfactory up until project closure where it was downgraded to Moderately Unsatisfactory due to incomplete implementation of three activities in the VMGPs. The project triggered Operational Policy (OP) 4.10 and a VMGF was prepared before appraisal. Drawing from the framework, 31 counties prepared VMGPs which identified barriers to VMGs accessing project services and activities that could address them. Counties were requested to include VMG activities in their AWPs. Under the project, each county with VMGs allocated at least 5.0 percent of budgets to VMG activities and 2.0 percent to grievance mechanism strengthening. 74. Most of the activities (99 percent) were completed, with considerable impact on the uptake of health services by these hard-to-reach communities. These impacts were shared during a VMG experience sharing workshop and a documentary that can be shared with other MoH and government programs. However, at project closure, three activities were still outstanding which led to the downgrading of the overall safeguards rating in the final project Implementation Status and Results Report. 75. The project strengthened the grievance redress mechanism (GRM) with particular emphasis on VMG areas with the county VMG focal point and VMG health staff acting as additional focal points for VMG grievances. Trainings were Page 28 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) carried out for county focal points and logbooks were printed and distributed for use and replication in health facilities. Awareness raising on the GRM was carried out during outreaches and by posting contact details and including in service charters in the health facilities. Several counties developed innovative approaches, linking to county grievance systems or using social media or toll-free lines to encourage and respond to feedback. The KHPOA has included complaints management in the professional conduct standards, and it is hoped that learning from the project will help strengthen health sector grievance management in future. Environmental safeguards compliance 76. Throughout project implementation, the environmental safeguards were rated Moderately Satisfactory. The THS-UCP triggered OP/BP 4.01 on Environmental Assessment and was assigned environmental category B. No significant and/or irreversible adverse environmental impacts were anticipated from the investments to be financed under the project. At project preparation, no civil works other than maintenance and minor renovations of existing facilities were anticipated. An Environmental and Social Management Plan (ESMP) was thus considered adequate and developed for adoption and implementation by healthcare facilities receiving support for rehabilitation and maintenance works. The project’s environmental and social (E&S) requirements such as the adoption and implementation of the ESMP by contractors were included in most works’ contracts. However, gaps were observed in some instances due to inadequate E&S support to county procurement staff preparing bid and contract documents for the works. Monitoring of ESMP implementation by the healthcare facilities was carried out by the environmental safeguards focal person in the PMT, with the support of County Public Health Officers (CPHOs). The monitoring of activities were largely successful, albeit with challenges due to a high turnover of CPHOs in the counties. 77. Provision of essential healthcare under the project was likely to lead to the generation of health care wastes ranging from non-infectious to highly infectious wastes, with potential to result in adverse impacts to the environment. To address this, MoH updated and publicly disclosed the Health Care Waste Management Plan (HCWMP). Counties were supported to adopt and implement the HCWMP, and training provided to healthcare waste handlers and staff on waste management and on infection prevention and control. The project also supported renovation of waste management facilities such as incinerators, burning chambers, ash pits and placenta pits, and the purchase of PPE, waste bins, and a few customized vehicles for the handling and transportation of healthcare waste. Although not envisaged at project preparation stage, five new waste treatment equipment (WTE) comprised of four Medical Waste Treatment Incinerators (MWTI) and a Medical Waste Treatment Microwave (MWTM) were procured and their sheds constructed. Environmental and Social Impact Assessments (ESIAs) for these facilities were carried out and the resulting ESMPs implemented during the construction and installation phase of the WTE. The facilities are operational, although none had obtained an operating license from NEMA by the time of preparing this ICR. On behalf of the counties, the MoH carried out an environmental and social audit (ESA) of the facilities as requested by NEMA to accompany the application for the operating licenses. The counties will be responsible for carrying out annual ESA of the facilities, and for ensuring their continued compliance with relevant environmental regulations. Procurement compliance 78. Procurement risk rating was High at the beginning of the project until November 2019 and thereafter reduced to Substantial during entire project implementation except for half a year period in 2021 when it was High. The procurement performance rating was Moderately Satisfactory during the entire period of project implementation. Page 29 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) While overall, there was compliance with selection methods and market approaches at national level and across all 47 counties, there were some cases of procurement non-compliance during implementation. The procurement post review (PPRs) and 2022 in-depth internal audit identified weaknesses and issues of non-compliance related to procurement, including: (a) missing records in procurement files at implementing agencies and equally not uploaded in STEP; (b) inadequate capacities to develop requisite procurement documents; and (c) inadequate contract review/award processes and contract management. 79. The delays in implementation of project activities partly attributed to the above procurement challenges and resulted in extensions of the project closing dates. 80. The PPR and in-depth internal audit of the THS-UCP recommended measures to enhance the procurement capacities of the implementing agencies for future projects in Kenya. These recommendations include: (i) ensuring that there is appropriate procurement support within the Project Management Team commensurate with the scale of implementation; (ii) ensuring that planned activities in the procurement plan are based on annual work plan and budget; (iii) constitution of quality review committee for all procurement documents including terms of reference and technical specifications; (iv) development and implementation of contract management mechanisms besides appointment of contract implementation team; and (v) establishing inspection and acceptance committee with relevant technical staff and end users to ensure that all deliverables are inspected and accepted before being paid for. Financial Management (FM) Compliance 81. Throughout project implementation, the FM rating was Moderately Satisfactory. Despite multiple capacity building activities carried out, there were still persistent FM weakness issues noted at Counties levels. Key issues noted included: i) inadequately supported project payments; ii) commingling of project funds in some counties; iii) inaccuracies in preparation of project cash book leading to alterations; iv) delays in surrender of imprests; v) weaknesses in maintenance of the imprest register; vi) delayed funds flow to the counties; and vii) low absorption of project funds. 82. Within the PMT, an adequate accounting team was maintained throughout the period of implementation. The interim financial reports were submitted to the World Bank within the stipulated timelines. The fiduciary oversight provided on the project was deemed sufficient as provided by the internal auditor at the ministry, the respective county internal audit department, the Independent Integrated Fiduciary Review Agent, the Office of the Auditor General, and finally by the PMT staff through missions. There were challenges with the status and irregularity of the undertakings of the county internal audit committees providing limited oversight on the implementation of the agreed actions. C. BANK PERFORMANCE Quality at Entry 83. The World Bank task team was for the most part based in country, except for the Task Team Leader based in Washington, D.C. who conducted multiple missions to ensure smooth project preparation. Subject matter experts also provided inputs to the design during preparation (e.g., the International Finance Corporation and the Development Economics team that piloted the JHIC). 84. The World Bank task team provided effective support to the Government and the final project design reflected the World Bank’s task team’s flexibility and deep understanding of the country context. The World Bank task team used Page 30 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) lessons learned from the recently closed KHSSP and introduced a results-based approach into the THS-UCP to ensure counties receive resources partly based on achieved results. Eligibility criteria was also added to encourage the GoK to increase domestic resources going towards health services overtime. Initially, the project design focused on high-burden counties identified in the RMNCAH Investment Framework. However, due to the recent devolution of the health sector, the GoK strongly advocated that the project provides support to all 47 counties. Nevertheless, the World Bank task team still ensured that a way to provide more support to marginalized counties was embedded in the design: the first year of disbursements to counties was based on need (worse performing counties received a larger allocation); similarly, in the following years, more allocation was given to low performing counties since they started with low baselines and allocation was based on the improvement in indicators – so counties which started with low baselines had larger improvement. In addition to improved performance, the allocation formula was informed by the County Revenue Allocation (CRA) formula used to calculate the county equitable share. 85. Development of the project design was highly consultative with the MoH, CoG, county governments and key DPs including bilaterals and UN agencies. The project was being prepared in parallel with the RMNCAH Investment Framework and therefore was informed by the analyses determining the high burden counties and “best-buy” interventions. Quality of Supervision 86. Overall, the quality of supervision was Satisfactory. The World Bank task team conducted implementation support missions every six months and as well as monthly meetings. As many task team members were based in country, the missions and meetings were conducted in person up until the beginning of closures due to the COVID-19 pandemic. In addition, the World Bank fiduciary and safeguards teams provided enhanced support and trainings to the PMT as needed to ensure adherence to World Bank procedures. Two core members of the health team were present from project preparation to closing- and both became the TTL and co-TTL. 87. During the COVID-19 pandemic, before the Kenya COVID-19 Emergency Response Project was approved, the World Bank task team swiftly responded to the Government’s request and restructured the THS-UCP to activate the CERC to support of the national COVID-19 efforts. Similarly, needed adjustments were made to the results framework to ensure monitoring of these interventions. 88. A total of six restructurings were conducted throughout project implementation to better align with the country’s priorities and course correct to ensure achievement of the project’s objectives. The restructurings are detailed in the significant changes during implementation section of the ICR. 89. The scale and complexity of this type of project would have required a larger team, especially increased support for safeguards, procurement, and financial management colleagues to ensure adherence to World Bank requirements. Although each mission was accompanied by field visits in different counties, due to the scale of the project, the team was not able to conduct field visits in all 47 counties within the lifetime of the project, also affected by the restrictions during the COVID-19 pandemic. Justification of Overall Rating of Bank Performance 90. The overall rating of the Bank performance is Satisfactory. Page 31 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) D. RISK TO DEVELOPMENT OUTCOME 91. The THS-UCP contributed to major advancements in improving utilization and quality of primary health care services with a focus on RMNCAH services. The project focused on sustainability by strengthening and building institutional capacity at both national and county levels. This was specifically relevant for counties which were newly formed and required significant technical assistance and capacity building to strengthen inter alia planning and budgeting processes, technical capacity and M&E capacity. The most notable risks to development outcome are the following: • Availability of FP commodities: The sustainability of the family planning program continues to be an area of concern given the reliance in recent years on commodities financed by the project and those donated by DPs. A sustained multi-partner dialogue has secured a budget allocation in the FY2023/24 national budget for FP commodities. This would be a substantial achievement, but continued advocacy will likely be necessary to ensure the release and execution of these funds. • Absorption of the graduates from KMTC: The project has supported the training of 784 enrolled community health nurses. Currently only one-third are fully employed in the health sector and the others were employed part-time at health facilities. Further documentation of the impacts of having VMG nurses within VMG communities and further communication to the counties to encourage their absorption should be considered. The recently approved BREHS project will continue encouraging counties to hire the VMG nurses trained through the THS-UCP as part of the support to HRH in the refugee hosting counties of Garissa and Turkana. • Sustained prioritization of health in county budget: There is a risk that now that the project has closed, counties may not continue allocating sufficient funds to health as new priorities emerge. To sustain prioritization of health in county budget, the World Bank teams should keep continuous and sustained engagement with county leadership to prioritize health in their budget and their decision making- since health is devolved. The recently approved BREHS project has maintained an eligibility criterion that will encourage counties to maintain or increase their allocation to health. V. LESSONS AND RECOMMENDATIONS 92. Future projects should consider balancing standardization of interventions for cross-cutting issues while also allowing for flexibility to address county specific gaps to allow for a simpler project design and manageable implementation supervision. The THS-UCP was largely implemented at the county level. County governments are autonomous and responsible for their own budgets and service delivery of health services. Given the results-based financing approach and recognizing the variation in priorities at county level and the autonomy of each county, the project was essentially implemented as 48 sub-projects, which put a strain on both the PMT and the World Bank task team. There are cross cutting issues that affect most if not all counties which can be addressed with a standard package of interventions; however, there are differences amongst the 47 counties which call for a flexible approach to support appropriate, context-specific, and efficient implementation of interventions. Having this balance in the future projects design would limit the complexity at implementation as well as monitoring implementation. 93. For projects implemented at the devolved level, a dedicated project team at the CoG should be maintained for the duration of project implementation to support and monitor county implementation and as much as possible limit activities that have been shown to negatively impact absorption (e.g., procurement heavy workplans). The THS-UCP Page 32 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) revealed weaknesses in fiduciary capacity at the county level. At county level, there should be a dedicated team, including accountant and procurement officer, that supports the project throughout the life of the project. 94. Future projects should ensure that mechanisms are in place for availability of quality data to allow for correct quantifications and monitoring of the results framework. Underlying data issues negatively affected certain project results related to (i) quantification of FP commodity needs, (ii) monitoring of PDOI 4 on women between the ages of 15- 49 years currently using a modern FP method, and (iii) monitoring PDOI 5 on the percentage of facilities inspected meeting quality standards. The FP procurement process relied too heavily on incumbent quantification assessment supported by partners which had methodological issues. In addition, the methodology for collecting data for women using modern contraceptives was measured each time a woman visits a health facility for FP commodities. This methodology is biased towards short-acting FP commodities and therefore underestimated coverage as preference shifted towards long-acting FP commodities. Finally, issues with the e-JHIC made it difficult to collect data on all the health facilities conducting inspections since many health facilities still used paper-based assessments. It is recommended that future projects ensures that counties put in place sustainable mechanisms to ensure availability of data for all indicators of the results framework. The outcome indicator was the ideal indicator to monitor, however shortcomings described in this ICR affected its achievement. 95. Future similar health projects should consider the lessons learned from this project regarding timely disbursement of county allocations. Delays in funds flow was a major contributor in the suboptimal absorption of funds at the county level and delays in the THS-UCP implementation. These delays were further compounded by protracted procurement processes. For the most part throughout project implementation, for any given fiscal year, project funds were transferred to the counties between December and January in any given fiscal year, meaning that there was less than a year for absorption of funds and implementation. Design of future health projects should be mindful of these potential delays and include necessary mitigation measures. . Page 33 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improved utilization and quality of primary health care services Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Children immunized with the Percentage 79.50 76.00 84.00 85.40 third dose of Pentavalent 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Pregnant women attending Percentage 39.70 46.00 52.00 53.80 at least four ANC visits 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Page 34 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Births attended by skilled Percentage 57.00 64.00 67.00 75.50 health personnel 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Women between the ages of Percentage 47.80 45.00 52.00 37.70 15-49 years currently using a modern FP method 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Inspected facilities meeting Percentage 0.00 50.00 37.00 safety standards 31-Dec-2015 15-Jun-2016 30-Sep-2023 Comments (achievements against targets): Page 35 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Pregnant women attending Percentage 31.00 40.00 73.00 74.00 ANC supplemented with IFA 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Objective/Outcome: Provision of immediate and effective response to an eligible crisis or emergency Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reported suspected cases of Percentage 0.00 80.00 100.00 COVID-19 cases investigated based on national guidelines 20-Oct-2020 20-Oct-2020 30-Sep-2023 Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: Improved primary health care results Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 36 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Target Completion People who have received Number 0.00 10,060,000.00 15,394,537.00 essential health, nutrition, and population (HNP) 29-Sep-2016 29-Sep-2016 30-Sep-2023 services Number of children Number 0.00 5,400,000.00 7,639,386.00 immunized 29-Sep-2016 29-Sep-2016 30-Sep-2023 Number of deliveries Number 0.00 4,660,000.00 7,755,151.00 attended by skilled health personnel 29-Sep-2016 29-Sep-2016 30-Sep-2023 Comments (achievements against targets): Component: Component 2: Strengthened institutional capacity Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Facilities inspected for safety Number 0.00 70.00 1,635.00 6,932.00 standards 11-Mar-2016 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Page 37 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reports submitted to DHIS in Percentage 88.80 85.00 98.00 98.40 a timely manner. 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Births registered within 6 Percentage 65.90 80.00 80.60 months of occurrence 31-Dec-2015 15-Jun-2016 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion RMNCAH related operations Number 0.00 3.00 2.00 2.00 research completed to inform policy/strategy 11-Mar-2016 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Page 38 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion A benefit package Number 0.00 1.00 1.00 developed, costed, and disseminated 11-Mar-2016 15-Jun-2016 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Lessons learned from UHC Yes/No No Yes Yes Phase I documented and disseminated 20-Jun-2019 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Component: Component 3: Cross-county/intergovernmental collaboration and project management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Functional community units Number 302.00 2,400.00 5,331.00 9,400.00 31-Dec-2015 15-Jun-2016 24-Jun-2019 30-Sep-2023 Page 39 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Grievances registered related Percentage 0.00 80.00 100.00 to delivery of project benefits that are addressed 31-Dec-2015 15-Jun-2016 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Implementing entities Percentage 0.00 80.00 95.00 100.00 submitting the annual FM and technical report on time 11-Mar-2016 15-Jun-2016 24-Jun-2019 30-Sep-2023 Comments (achievements against targets): Component: Component 4: Contingency Emergency Response Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 40 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Designated laboratories with Number 2.00 8.00 41.00 COVID diagnostic equipment 18-Mar-2020 20-Oct-2020 30-Sep-2023 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Vulnerable and marginalized Percentage 0.00 80.00 100.00 communities reached in their indigenous language 18-Mar-2020 20-Oct-2020 30-Sep-2023 Comments (achievements against targets): Page 41 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 1. Children immunized with the third dose of Pentavalent (Percentage) Outcome Indicators 2. Pregnant women attending at least four ANC visits (Percentage) 3. Women between the ages of 15-49 years currently using a modern FP method (Percentage) 1. Functional community units (Yes/No) 2. A benefit package developed, costed, and disseminated (Number) 3. People who have received essential health, nutrition, and Intermediate Results Indicators population (HNP) services (Number) 4. Lessons learned from UHC Phase I documented and disseminated (Yes/No) Objective/Outcome 2 1. Inspected facilities meeting safety standards (Percentage) 2. Pregnant women attending ANC supplemented with IFA Outcome Indicators (Percentage) 3. Births attended by skilled health personnel (Percentage) 1. Facilities inspected for safety standards (Number) 2. RMNCAH related operations research completed to inform policy/strategy (Number) Intermediate Results Indicators 3. Reports submitted to DHIS in a timely manner (Percentage) 4. Births registered within 6 months of occurrence (Percentage) 5. Grievances registered related to delivery of project benefits that are addressed (Percentage) Page 42 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) 6. Implementing entities submitting the annual FM and technical report on time (Percentage) Objective/Outcome 3 1. Reported suspected cases of COVID-19 cases investigated based on Outcome Indicators national guidelines (Percentage) 1. Designated laboratories with COVID diagnostic equipment (Number) Intermediate Results Indicators 2. Vulnerable and marginalized communities reached in their indigenous language (Percentage) Page 43 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Yi-Kyoung Lee Task Team Leader(s) Joel Buku Munyori Procurement Specialist(s) Henry Amena Amuguni Financial Management Specialist Evelyn Anna Kennedy Team Member Monica Gathoni Okwirry Team Member Gibwa A. Kajubi Social Specialist Kishor Uprety Counsel Eva K. Ngegba Team Member Gandham N.V. Ramana Team Member Son Nam Nguyen Team Member Edward Felix Dwumfour Social Specialist Elizabeth Laura Lule Team Member Samuel Lantei Mills Team Member Christiaan Johannes Nieuwoudt Team Member Joyce Cheruto Bett Team Member Jane Chuma Team Member Sophie Nelly Rabuku Team Member Page 44 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Yvonne Wangui Machira Team Member Stephen Diero Amayo Team Member Suzanne Ndunge Kiamba Social Specialist Racheal Njeri Gitau Team Member Edwin Wafula Barasa Team Member Toni Lee Kuguru Team Member Maina Ephantus Githinji Social Specialist Benjamin Kithome Kilaka Social Specialist Supervision/ICR Toni Lee Kuguru, Jane Chuma Task Team Leader(s) Boaz Okoth Akello Procurement Specialist(s) Henry Amena Amuguni Financial Management Specialist Meron Tadesse Techane Financial Management Specialist Winnie Achieng Adhoch Procurement Team Eunice Jemutai Cherutich Environmental Specialist Simon Francis Nguru Wandeto Team Member Ndiga Akech Odindo Team Member Kenneth Munge Kabubei Team Member Brendan Michael Hayes Team Member James Thumi Muturi Team Member Jacob Omondi Obongo Social Specialist Annastacia Waithera Wacheke Team Member Vanessa Sigrid Tilstone Social Specialist Maletela Tuoane Team Member Josine Umutoni Karangwa Team Member Nancy Makungu Gamusa Team Member Page 45 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) Tesfahiwot Dillnessa Zewdie Social Specialist Gladys Akurut Alupo Procurement Team Evarist F. Baimu Counsel Sujani Eli Team Member Mohammad Ilyas Butt Procurement Team B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY15 23.630 175,748.28 FY16 71.019 582,102.72 FY17 7.975 50,382.87 FY18 0 -4,205.61 FY19 2.622 61,992.86 FY20 .810 177,433.14 FY21 0 75,625.00 Total 106.06 1,119,079.26 Supervision/ICR FY16 1.200 6,443.04 FY17 53.931 286,885.44 FY18 44.725 300,334.04 FY19 58.028 691,485.54 FY20 124.443 3,262,558.45 FY21 59.982 1,547,787.18 FY22 76.559 2,770,765.04 Page 46 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) FY23 54.443 1,160,949.41 FY24 36.389 504,786.35 Total 509.70 10,531,994.49 ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (%) Improving Primary Health 150.00 131.22 87.50 Care Results Strengthening Institutional 15.10 28.41 188.15 Capacity Contingency Emergency 0.00 6.90 N/A Response Cross-county and Intergovernmental 26.00 8.81 33.90 Collaboration, and Project Management Total 191.10 175.34 91.75 Page 47 of 48 The World Bank Transforming Health Systems for Universal Care (P152394) ANNEX 4. BORROWER COMMENTS The borrower did not provide any comments. Though the Ministry of Health shared the Government ICR, which is consistent with the findings of this ICR. Page 48 of 48