Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) Report Number: ICRR0024083 1. Project Data Project ID Project Name P126130 TJ Health Services Improvement Project Country Practice Area(Lead) Tajikistan Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-56660,IDA-D0700,IDA-D5470,IDA- 31-Jan-2019 40,995,268.79 H8790,TF-14871,TF-B2817 Bank Approval Date Closing Date (Actual) 30-Jul-2013 30-Jun-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 15,000,000.00 6,800,000.00 Revised Commitment 41,799,983.34 6,800,000.00 Actual 40,995,268.79 6,800,000.00 Prepared by Reviewed by ICR Review Coordinator Group Denise A. Vaillancourt Jenny R. Gold Susan Ann Caceres IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As stated in the Financing Agreement between the Republic of Tajikistan and International Development Association (IDA), dated September 20, 2013 (Schedule 1, p. 5), the project’s development objective (PDO) was “…to contribute to the improvement of the coverage and quality of basic primary care services in rural health facilities in selected districts.” The design document presents the same statement, but specifies “… primary health care services…”(PAD, pp. i, 8). Page 1 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) Under the 2015 restructuring and approval of a first Additional Financing (AF1), the PDO was changed in to read as follows, “…to contribute to the improvement of the coverage and quality of basic primary health care services in selected districts,” eliminating the reference to “rural health facilities.” As stated in the (restructuring) Project Paper, dated May 21, 2015 (p. 12), the PDO was changed “…to include all PHC facilities, not just rural, because technical assistance on health financing under Component 1 of the proposed AF1 would serve both rural and urban PHC facilities. Together, these services form the core of first-level services and provide similar basic preventive and curative care, and some specialist outpatient services to both urban and rural populations.” Because the focus remains on primary health care services in selected districts, this change is not reason enough for a split rating. However, the dropping in 2019 of key outcome indicators (percentage of pregnant women receiving antenatal care four or more times from a skilled health provider; and the contraceptive prevalence rate of women in project districts), supported under the original project and under AF1, but no longer under AF2 (November 25, 2019 Project Paper), does warrant a split rating methodology. The raising of some outcome targets under the 2021 restructuring does not warrant a split rating, as these outcomes were fully achieved or surpassed. The ICR did not undertake a split rating. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 22-Jun-2015 c. Will a split evaluation be undertaken? Yes d. Components Original Components: Under the original design, project support was to be delivered through three components, for a total original cost estimated at $23.0 million, of which $15 million to be financed by IDA, $4.8 million to be financed by the Health Results Innovation Trust Fund, and $3.2 million to be financed by the Government of Tajikistan (GoT). Original estimates (by source) are provided for each component. Revised estimates under the two additional Financings and actual costs of these three components are provided in the following “Revisions to Components” section. Component 1: Performance-Based Financing (PBF) (original estimate of $12.8 million – of which $6.0 million financed by an IDA Grant, $4.8 million financed by a Health Results Innovation Trust Fund, and $2.0 million by the Government of Tajikistan): This component supported a PBF pilot at the PHC level in initially eight districts (rayons) in Khatlon and Sogd regions (oblasts), where rural health centers (RHCs) and their subsidiaries health houses (HHs) were eligible to receive quarterly performance-based payments based on the quality and quantity of maternal and child health care (MCH) and non-communicable diseases (NCD) services delivered. Reported outputs and quality were to be verified through quarterly visits by the state health and social protection supervision services (SHSPSS) and twice annually by an independent survey firm contracted by the United Nations Children’s Fund (UNICEF). The performance payments Page 2 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) supplemented the status quo in-kind appropriations from the state budget, and facilities, within simple spending rules, had discretion over their use, with up to 70 percent being eligible for salary bonuses. The component included a rigorous impact evaluation (IE) of the PBF intervention. Component 2: Primary Health Care (PHC) Strengthening (original estimate $6 million – all financed by IDA) to enhance the capacity for safe and effective essential services provision. Subcomponent 2.1: Quality Improvement. The subcomponent financed the training of PHC doctors and nurses from the project districts and selected comparison districts in a six-month financial management (FM) training program. Nurses and doctors from all PHC facilities in the project districts and selected comparison districts also participated in continuous medical education (CME) on clinical treatment protocols for MCH and selected NCD care. This subcomponent also supported the introduction of the Collaborative Quality Improvement and Citizen Score Cards (CSC) interventions, with and without PBF. Subcomponent 2.2: Physical Infrastructure Improvements. This subcomponent supported the improvement of PHC facility infrastructure in the project districts. This included the provision of basic medical equipment and supplies to all PHC facilities in the project districts, and some rehabilitation and reconstruction works for selected PHC facilities to ensure basic functionality. Twenty-four PHC facilities in the project districts were originally identified for works in accordance with agreed criteria. Component 3: Project Management, Coordination, and Monitoring and Evaluation (M&E) (original estimate of $4.2 million – of which $3.0 million financed by IDA, and 1.2 million provided by GoT in kind). This component financed the expenses associated with the implementation and management of the project at the central, regional, and district levels. It supported recurrent costs, office equipment and furniture, vehicles for project supervision, consultant salaries, travel expenses, study tours to enhance the knowledge of PBF schemes, training for the Project Coordination Group (PCG) members and project implementation staff at regional and district levels, M&E, and project audits. Revisions to Components Under the restructuring and AF1 approved in August 2015, a total of $10.07 million in IDA financing was added to the originally estimated project cost of $23.0 million, for a total revised cost of $33.07 million. The allocation of AF1 to each component and adjusted total project costs under the 2015 restructuring are presented below (Table A6.2, ICR p. 61): Component 1: Performance-Based Financing (AF1 added an additional $1.16 million to the initial $12.8 million for a total adjusted component cost of $13.96 million) to support:  Implementation of PBF in one additional district in Khatlon Region, and one district in the Districts under Republican Subordination (RRS) Region.  Additional technical assistance to support implementation of comprehensive PHC financing reforms, including fine-tuning the model for implementing PBF in district health centers (DHCs) and city health centers (CHCs).  Strengthened social accountability and improved outcomes by contracting local NGOs to facilitate discussions between communities and PHC providers of feedback received through citizen scorecards CSCs. Page 3 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) Component 2: PHC Strengthening (AF1 added an additional $8.3 million to the initial $6.0 million for a total adjusted component cost of $14.30 million to support:  Subcomponent 2.1: Quality Improvement o Scaling up of trainings to four new districts.  Subcomponent 2.2: Physical Infrastructure Improvements o Change from rehabilitation to reconstruction of several RHCs, and reconstruction and equipment of additional RHCs in the first eight and newly added four project districts. o Minor rehabilitation and provision of teaching equipment for the Khatlon and Sogd Family Medicine Training Centers. Component 3: Project Management, Coordination, and M&E (AF1 added an additional 0.61 million in IDA financing to the initial $4.2 million for a total adjusted component cost of $4.81 million) to support:  Expenses associated with project management and implementation at central, regional and district levels, including in the new project districts. Under the restructuring and AF2, approved in December 2019, a total of $12.0 million was added to the revised estimated cost of $33.07 million, culminating in an updated cost of $45.07 million. The $12.0 million increment comprised $10.0 million in IDA financing under AF2 and $2.0 million in co-financing provided by GAVI. An assumed additional in-kind contribution by GoT to project management brings up the total cost to $45.68. The allocation of these increments to each component and adjusted total component cost estimates and actual costs at closing are presented below (ICR, Table A6.2, ICR p. 61): Component 1: Performance-Based Financing (the 2019 restructuring/AF2 provided an additional $5.875 million – of which $4.37 million in IDA financing and $1.505 in GAVI co-financing – to the (2015) estimated component cost of $13.96, culminating in a revised component estimate of $19.83 million; actual cost at closing: $19.44 million) to support:  Scaling up of the intervention to six additional districts  Revision of PBF indicators/formula and reduced incentive payments to improve fiscal sustainability after project completion. In response to project IEs finding of limited impacts on coverage indicators, indicator revisions include addition of a new indicator incentivizing health workers to visit each household in their catchment area for patient engagement activities to raise awareness and uptake of PHC services.  Piloting of electronic patient registries in selected PBF districts starting with Spitamen (hardware, software, training, incentivization of citizen enrollment) for better planning, financing and management of PHC services.  Supporting the transition to government ownership and sustainability of PBF through the development of Ministry of Health and Social Protection of the Population (MoHSSP) and its technical working group and the development of its action plan for nationwide scaling of a PHC financing mechanism that integrates PBF with per capita financing (PCF). Component 2: PHC Strengthening (the 2019 restructuring/AF2 provided an additional $4.175 million – of which $3.68 million in IDA financing and $0.495 in GAVI co-financing – to the (2015) estimated Page 4 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) component cost of $13.96, culminating in a revised component estimate of $18.48 million; actual cost at closing: $18.87 million) to support:  Subcomponent 2.1: Quality Improvement o Scaling up of six-month Family Medicine training courses to six additional project districts o PHC management training for all heads of PHC facility networks and directors of RHCs across the country (roughly 1000) to develop skills and planning data analysis for performance, leadership and community outreach, and sessions on climate-related health issues and resiliency measures. o Support for two-year FM specialty training. o Support for the development, installation, testing of and training on specialized software to improve the quality and coverage of PHC services in PBF districts. o Revised CSC mechanism which includes all stakeholders jointly developing action plans with targets and follow-up mechanisms.  Subcomponent 2.2: Physical Infrastructure Improvements o Technical assistance for independent verification of PHC facility site survey to establish infrastructure and equipment needs in six new PBF districts. o Support of rehabilitation and repair works, including installation of solar panels for selected PHC facilities in the six new districts. o Provision of equipment and supplies, including basic medical equipment bags for PHC staff, doctors and nurses, and computers. The medical bags are to be distributed to: (1) FM doctors and nurses in GBAO Region where the level of equipment is poor; and (2) doctors and nurses newly trained in FM in the new project districts. Component 3: Project Management, Coordination, and M&E (the 2019 restructuring/AF2 provided an additional $2.56 million – of which $1.95 million in IDA financing and an assumed $0.61 million in GoT in-kind counterpart – to the (2015) estimated component cost of $4.81 million, culminating in a revised component estimate of $7.37 million; actual costs at closing: $6.08 million) to support: expenses associated with project management and implementation at central, regional and district levels, including in the six new project districts. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost. The total actual cost of the project at closing, was $44.49 million (ICR p. 60), almost double (193 percent) the original estimate. The actual costs of Components 1 (Performance-Based Financing) and 3 (Project Management, Coordination and M&E) were 50 percent over their original estimates, while the actual cost of Component 2 (Primary Health Care Strengthening) was more than triple (315 percent) the original estimate, reflecting an increasing emphasis on this component during the course of implementation and two Additional Financings (AFs). Financing and Borrower Contribution. Of the total actual cost: $34.19 million (or 77 percent) was financed by IDA grants and credits (AFs included); $6.8 million (or 15 percent) by Trust Funds (TF- 14871/Health Results Innovation and TF-B2817/GAVI); and $3.6 million (or 8 percent) by the Borrower. In SDR terms, all IDA credits and grants, totaling 24.6 million SDRs, were 100 percent disbursed, including: the original IDA grant (IDA-H8790 for 10 million SDRs); an IDA credit and grant under AF1 (IDA- Page 5 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) 56660 credit for 4.0 million SDRs; IDA-D0700 grant for 3.3 million SDRs); and an IDA grant under AF2 (IDA-D5470 for 7.3 million SDRs). In addition, both Trust Funds were fully utilized: TF-14871 (Health Results Innovation Trust Fund) in the amount of $4.8 million; and TF-B2817 (GAVI) in the amount of $2.0 million. Key Dates. The project was approved on July 30, 2013, and became effective four and one-half months later, on December 11, 2013. A mid-term review was conducted in March 2017. The actual closing date, June 30, 2023, was four years and five months beyond the original closing date of January 31, 2019. The project was restructured five times, including the two additional financings. The first (Leve 1) restructuring (approved on June 22, 2015) included AF1 in the amount of SDR 7.3 million ($10.0 equivalent), slightly revised the PDO statement (as detailed in Section 2.a), changed the results framework, but not at the PDO level, reallocated funds across components, extended the closing date and amended the implementation schedule. The second and third restructurings (approved on August 8, 2018 and September 12, 2019, respectively) extended the closing dates. The fourth (Level 2) restructuring (December 18, 2019) included AF2 in the amount of SDR 7.3 million ($10.0 million equivalent). It also introduced changes in the results framework at the PDO level (see Section 2.a) and reallocated funds across components. The fifth restructuring (December 24, 2021) changed the results framework at the PDO level, reallocated funds across components and disbursement categories, extended the closing date and amended the implementation schedule. 3. Relevance of Objectives Rationale The PDO is highly relevant to current country conditions, as described in the World Bank’s October 2, 2023 PAD on the follow-on Tajikistan Health (“Millati Solim”) project. Despite substantive progress over the last 20 years, Tajikistan trails other ECA countries on key indicators, such as life expectancy, child mortality, and stunting. Non-communicable diseases (NCD) and gender-based violence (GBV) are also significant and on the rise. Inadequate access to certain services and low service quality are the root causes of Tajikistan’s lagging health outcomes. All Tajiks are entitled to subsidized health services from public providers, with user fee exemptions for several demographic and socioeconomic groups, including the poor. While Tajikistan has achieved high utilization of key MCH services, access to NCD services is low, with only 13 percent of patients with hypertension and 20 percent of patients with diabetes under treatment. Regional and socioeconomic inequities are growing, even for services with high national coverage. Gaps in service quality are also pervasive and, together with low coverage, cause many deaths per year, which could have been averted. Service quality is undermined by inadequate physical facilities and human resources issues, including: low skill levels; poor management, distribution, and utilization of health personnel; an acute shortage of family medicine doctors to provide primary health care services and fulfill their role as gatekeepers for higher levels of care. Only a small share of family doctors and nurses successfully pass basic knowledge tests for the diagnosis and treatment of common non-communicable and childhood conditions, and over-prescription of antibiotics is common, all pointing to substantive gaps in clinical knowledge and practice. As a result, the population often bypasses cost-effective primary care and seeks more expensive specialized services instead. The need to increase domestic financing for the health sector and improve the quality of, and stimulate demand for, cost-effective PHC services is particularly Page 6 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) urgent in light of the rise in NCDs and rapid population growth. The COVID-19 pandemic also revealed the need for substantive investments to strengthen capacity for prevention and management of health emergencies. Over and above investments in infrastructure, equipment, and human resources, the health sector needs structural reforms to prepare the healthcare system for future challenges. Specifically, reforms are needed to address: health financing inequities and inefficiencies; fragmentation of financing due to lack of integration of vertical programs; limited digitization of documentation and data transmission; an inadequate per capita financing policy, which does not assign funding according to size and epidemiological and demographic profiles of catchment area populations; and low, ineffective and fragmented domestic funding of PHC and consequent excessive out-of-pocket expenses. The PDO is highly relevant to the current development priorities of the country. GoT’s “National Development Strategy of the Republic of Tajikistan (NDS) for 2030” prioritizes the strategic objective of human capital development, within which the goal of improving the health of the population through improved access, quality and efficiency of health services and systems, is prominent. NDS 2030, together with the fully costed Prioritized Investment Plan (PIP), articulates Development Directions to this end: (1) improving access, quality and responsiveness of PHC services; (2) enhancing sustainable sector financing; (3) strengthening health management information systems and digital capacity; and (4) ensuring sustainable human resources development; (5) improving the health of mothers, newborns, children, and adolescents and their access to sexual and reproductive health services and rights; and (6) strengthening prevention and management of NCDs and disabilities. The PIP serves as the main instrument for facilitating complementarity and coordination of various sources of financial and technical support (domestic and international) and is aligned with national sector-wide management processes, such as the Joint Annual Review of sector performance. Since the mid-2000s, the GoT has confirmed its commitment to a wide array of health system reforms in various policy documents. But reform implementation has been lagging, particularly in health financing. The COVID-19 pandemic created a sense of urgency and momentum among key stakeholders to hasten progress towards universal health coverage and galvanized support to modernize PHC to create a cost-effective, patient-centered, and flexible system for a better response to unexpected challenges. Moreover, the PDO is highly relevant to the World Bank’s current Country Partnership Framework for the Republic of Tajikistan for the period FY2019 to FY2023. Under Focus Area I (Human Capital Development) Objective 2 seeks to enhance health services through two main interventions: (1) activities to improve maternal and child health services, complemented by other critical services, such as clean water and sanitation, improved hygiene, and nutrition interventions; and (2) continued support to improvements in the coverage and quality of basic PHC services in selected districts, including: the strengthening of financing and physical infrastructure for PHC; improvements to sector-wide financing for greater efficiency and equity; and improvements to service quality, including the strengthening and more effective use of health worker capacity. The project is also fully aligned with the World Bank’s twin goals, its strategy for health, nutrition, and population, and global health commitments, to which it contributes. For over two decades, the WB had played an integral role in supporting Tajikistan’s health sector, rendering it well equipped to support a project of this nature. As noted in the PAD (paras. 16-19), the World Bank has been supporting GoT in strengthening the country’s health sector through a series of projects, which supported: key health sector policy reforms (PHC per capita financing, a basic benefits package, human resources development and sector-wide planning capacity); policy development and management capacity at central, regional and PHC levels; and strengthening of PHC services delivery Page 7 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) capacity (retraining of specialists as family medicine doctors and nurses, rehabilitation and construction of PHC infrastructure, and initiatives to improve the quality of PHC, as cited in the November 25, 2019 restructuring paper). The World Bank has also provided complementary support through its Programmatic Development Policy Grant series (supporting financing and management reforms) and its administration and oversight of grant funding (Japanese Social Development Fund and Global Food Price Crisis Response, supporting nutrition interventions targeted to women and children at the PHC and community levels). The Bank’s growing experience supporting performance-based financing in other countries was also an advantage. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Contribute to the improvement of the coverage of basic primary care services in rural health facilities in selected districts (2013) Rationale The original project design supported activities to (a) pilot performance-based financing (PBF) as a means of incentivizing improvements in the coverage and quality of PHC services, with a focus on MCH and NCDs, in selected facilities, in eight pilot districts, four each in the regions of Khatlon and Sogd; (b) strengthen service delivery capacity in PBF pilot facilities (through the training of clinical staff and the upgrading of PHC infrastructure and equipment); and (c) introduce/test, in a smaller subset of PBF facilities and in selected control facilities: (i) the Collaborative Quality Improvement (CQI) intervention, under which, healthcare workers are trained to apply flowcharts to manage clinical cases and receive supportive feedback on performance; and (ii) the Citizen Scorecard (CSC) intervention, which aimed to strengthen PHC provider accountability and responsiveness to communities, including communities’ rating of provider performance and enhanced dialogue for improvements. Under the first restructuring and AF1, concomitantly approved in August 2015, the PDO was slightly revised to remove the reference to “rural health facilities,” leaving open the possibility to roll out project activities to higher-level PHC facilities located in urban centers of project-supported districts, i.e., district health centers and city health centers. Under AF1, the PBF intervention was scaled up to include two additional districts: an additional one in Khatlon, and one in RRS; and the infrastructure and quality improvement interventions were scaled up to include four additional districts: the two aforementioned districts added for PBF (in Khatlon and RRS); and two districts in Gorno-Badakhshan Autonomous Oblast (GBAO). AF1 also covered the financing gap for construction/renovation of project-supported PHC facilities, due to price increases causing cost overruns. All changes under the 2015 restructuring and AF1, were consistent with the original PDO and did not affect the theory of change or expected outcomes. Changes to outcomes and targets under AF1 mainly Page 8 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) reflected the change in closing date and adjustments to baselines and targets linked to the addition of new project districts. These (original and 2015 adjusted) activities were expected to lead to a number of outputs and intermediate outcomes, including: the functioning of the PBF pilot, paying for progress against selected coverage indicators for priority PHC services (the number of eligible health facilities in which PBF is initiated and functioning; timely PBF payments to PBF facilities; the number of independent verification visits; districts in which PBF MIS is operational; improved outreach); the evaluation of the PBF pilot for the purposes of learning and accountability; enhanced service delivery capacity of PHC facilities, as a result of training and CQI activities and improvements in PHC infrastructure; and enhanced citizen engagement (the number of CSC sessions conducted and the number of action plans emanating from these sessions). These outputs and intermediate outcomes were, in turn, expected to contribute to improved coverage outcomes for critical MCH services (reproductive health, child health, and nutrition) and NCD services (treatment of hypertension) delivered at the PHC level. Under Original Objective 1, including the minor adjustments introduced under the 2015 restructuring and AF1 approval, (a) MCH coverage was to be measured by: share of pregnant women receiving antenatal care (ANC) four or more times from a skilled health provider; an increase in contraceptive prevalence rates (CPR); timely post-natal visits (mothers receiving at-home post-natal visits within first three days after discharge from maternity); and pregnant and lactating women and/or children under age five reached by basic nutrition services; and (b) NCD coverage was to be measured by: share of hypertensive adults currently receiving anti-hypertensive treatment. (The indicator tracking timely post-natal visits was classified as an IRI in the RF, but IEG considers it to be an outcome indicator, since it tracks coverage.) Outputs and intermediate results Functioning of the PBF pilot and its evaluation  Under the original objective and 2015 restructuring/AF1, PBF payments incentivized nine performance indicators prior to the December 2019 restructuring (with slight refinements introduced in 2017). These indicators targeted key services, including: Child health and nutrition: o Number of children under 13 months fully vaccinated; o Number of children under five with detected malnutrition (number of children aged 0-24 months, whose weight and height were measured according to the recommended growth chart and whose parents were advised on proper nutrition); o Number of children under five years suffering from moderate malnutrition who received consultation (number of children under five with detected malnutrition, whose parents received advice on proper nutrition and child care, and who are experiencing improvements in nutritional status); Reproductive health: o Number of pregnant women with first visit to antenatal care (ANC) during the first 12 weeks of pregnancy; o Number of pregnant women with at least four ANC visits to RHC/HH (the last visit falls into last three weeks of pregnancy)/(last visit falls into last two weeks of pregnancy); Page 9 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) o Number of postnatal visits during first week (number of mothers who received first postnatal patronage visit at home within the first three days after discharge from the Maternity House); o Number of women aged 15-49 years who are using modern contraception, as at the end of the month (subdivided into: new users and continued users of modern contraception); Non-communicable diseases: o Number of newly detected and registered patients with hypertension; o Number of patients with hypertension under the continuous treatment, at the end of the month. PBF Functioning o By the project’s end, PBF was initiated in a total of 720 eligible health facilities, exceeding the original target of 316 (228 percent achieved) and the two subsequent restructured targets -- the target (reset in 2015) of 413 (174 percent achieved) and the target (reset in 2019) of 700 (103 percent achieved), and fully achieving the final target (reset in December 2021) of 720. o By the project’s end 100 percent of PHC facilities eligible for PBF payments received timely PBF payments in the preceding quarter in all selected districts in three regions: Sogd, Katlon and RRS (added in August 2015). o By the project’s end, 14 independent verification visits were completed on schedule, exceeding the original target of eight (175 percent achievement) and the revised target of nine (156 percent achievement) reset in August 2015, and fully meeting the revised target of 14 reset in December 2019. o The ICR reports that the PBF MIS was operational in all six new districts, added under AF2, fully achieving the AF2 target of six. While it is assumed that the PBF MIS became operational in the original 10 districts (the eight districts under the original design and the two PBF districts added under AF1), the ICR does not explicitly state this. o By the project’s end, 97 percent of PBF facilities completed the household engagement exercise, exceeding the target of 70 percent introduced in 2019 (139 percent achievement) and the revised target of 45 reset in 2021 (216 percent achievement). While this indicator was introduced under the December 2019 restructuring to raise awareness about PHC services and to increase uptake, it is appropriate to report here as a contributor to original Objective 1. District-level data shows home visits per population to be 27 percent higher in PBF districts, compared to non-PBF districts; and home visits per health care worker increased by two-thirds in PBF districts, indicating enhanced productivity. Home visits were instrumental for ensuring continuity of access to PHC services, and for COVID prevention and control, including the roll-out of COVID-19 vaccines (ICR, para. 45). o In 2017, an evaluation of the PBF pilot experience was completed and an action plan for roll-out to the seven districts was prepared and implemented. Enhanced citizens’ engagement  By the project’s end, a total of 816 citizen scorecard sessions were conducted in project districts to enable a discussion between communities and PHC staff on citizens’ feedback in the 16 project districts (with the standard of holding one session per year for each of the 72 facilities in each of the project and control groups). This exceeded the original target, set in August 2015, of 232 sessions (352 percent achievement), and the revised target, set in December 2019, of 450 sessions (181 percent achievement), and fully met the final target set in December 2021, of 816 sessions (100 percent achievement). Page 10 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) o Women made up, on average, 80 percent of participants, exceeding the target by more than a factor of three. While this indicator was added under the December 2019 restructuring, it is relevant to report here.  Citizen scorecard sessions were meant to inform the development and implementation, by PHC facilities, of community action plans responsive to citizen feedback. By the project’s end, 99 percent of PHC facilities in 16 project and control districts had acted on community action plans. Enhanced service delivery capacity of PBF facilities  A total of 13,867 health personnel received training under the project, exceeding all targets set under the project: 500 (original target); 7,957 (set in August 2015); 11,289 (set in December 2019); and 13,700 (set in December 2021). Training data (Annex 1) is not disaggregated by region or type of training.  Improvements in PHC infrastructure were achieved through the rehabilitation and/or equipment of 481 health facilities, exceeding the original target of 24, the revised target, set in 2015, of 38 RHC facilities constructed and 350 PHC facilities equipped, and the revised target, set in December 2021, of 425 health facilities. The ICR (Annex 1) does not provide any disaggregation of the 481 facilities rehabilitated and/or equipped, by region, or by type of infrastructure improvement(s) provided.  PBF payments rewarded childhood vaccinations, child growth monitoring and nutrition services, ANC visits, PNC visits, new and continuous users of modern family planning, and treatment of hypertensive patients. Payments were used to provide staff bonuses and to cover other essential costs of operations.  By the project’s end, 97 percent of PBF facilities completed the household engagement exercise, exceeding the 2019 target of 70 percent (139 percent achievement), and the 2021 revised target of 45 percent (216 percent achievement). Incentives for patronage visits were introduced in 2021. Outcomes Coverage data provided here are cited in the ICR and drawn from project data collected by MoH at the facility and district levels. The absence of reliable denominators (populations/target groups residing in facility catchment areas), detailed in the M&E section, challenged the measurement of Objective 1 and, according to the ICR, likely resulted in the reporting of higher coverage rates than actually achieved. An Impact Evaluation (IE) conducted on the PBF and PHC strengthening interventions and their effect on service coverage is also cited. Issued in 2019, the IE’s reported results are confined to the early project period (2015-2018), more than four years before the June 2023 project closing. As such, the IE informed the 2019 project restructuring and AF2 for better results. Coverage of Child Health and Nutrition Services:  By the project’s end, a total of 262,577 children were immunized, exceeding the target of 180,000 set in December 2019 (146 percent achievement), and fully achieving the revised target set in December 2021 of 262,256. This is a standard corporate results indicator (CRI). There was a related performance indicator under the PBF pilot, which rewarded the share of children under 13 months of age who are fully immunized. However, the ICR (because of issues with the denominator) did not report on the share of children fully immunized. The IE did not find positive impacts for childhood vaccinations. Page 11 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130)  Under Original Objective 1, the project aimed to track and increase the number of pregnant/lactating women and/or children under age five reached by basic nutrition services (growth monitoring, nutritional counseling, or demonstration services), with no specific target set in the original design. Data was to be disaggregated by region and by patient type (pregnant/lactating women; children under 24 months; children 24 months to 5 years). The project PBF replaced this with the indicator mothers counseled on nutrition to better align with the PBF data collection, which was initially (August 2015) classified as an IRI, and subsequently (December 2019) classified as an outcome indicator: o From a baseline of 182,452, a total of 326,730 mothers were counseled on nutrition, exceeding the targets of 230,000, set under the December 2019 restructuring (142 percent achieved), and 290,512, set under the December 2021 restructuring (112 percent achieved.) The ICR does not provide disaggregated data, as specified in the results framework.  By the project’s end, 1,814,084 women and children had received basic nutrition services, exceeding the original target of 425,000 (427 percent achievement), and the revised target of 1.0 million (181 percent achievement), reset in 2019, and substantially achieving the revised target of 1,878,990 (97 percent achievement), reset in 2021. This was a CRI indicator classified as an IRI in the results framework. But IEG considers it to be an outcome indicator for Objective 1 (coverage). The absence of disaggregation by type of client and type of service leaves questions about the nature and coverage of these services.  The 2019 Project Paper for AF2 (p. 9) reports that 100 percent of children under 2 years of age received growth monitoring and counseling.  According to the IE, PBF and the concomitant trainings had statistically significant positive impacts for HHs, where service availability increased by 24 percentage points for growth monitoring, 12 percentage points for child nutrition services, and three percentage points for iron folate supplementation (ICR, para. 39). The ICR (para. 41) reports that the IE did not find significant effects on the timing and number of growth checks for children under two. Coverage of MCH services: Reproductive Health  Under the Original Objective 1, an original PDO indicator set targets for increases in the share of pregnant women receiving ANC four or more times from a skilled health provider. This indicator was dropped under the December 2019 restructuring and AF2 approval, “due to measurement challenges and because activities measured by it were no longer incentivized by the PBF scheme” (ICR Table 2, p. 11). Baselines (2012 DHS) and targets set for this outcome indicator were as follows: o an increase from 79.2 percent to 85 percent for Sogd (original PDO); o an increase from 39.2 percent to 45 percent for Khatlon (original PDO); and o an increase from 78.7 percent to 82 percent for RRS (new region added under August 2015 restructuring). No data is reported on actual trends during the time prior to the December 2019 restructuring, when PBF incentivized provision of this service. The ICR (para. 41) reported that the IE found no significant project effects on the timing and number of ANC checks. In the absence of sufficient evidence, progress against these targets is assessed modestly achieved.  Under the Original Objective 1, an original PDO indicator set targets for increases in the contraceptive prevalence rate (CPR). This indicator was dropped under the December 2019 restructuring, “due to Page 12 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) measurement challenges and because activities measured by it were no longer no longer incentivized by the PFB scheme” (ICR Table 2, p. 11). Baselines (2012 DHS) and targets set for this outcome indicator were as follows: o increase from 30.7 percent to 35 percent for Sogd; o increase from 22.9 percent to 27 percent for Khatlon; and o increase from 23 percent to 27 percent for RRS. No data is reported on actual trends in CPR during the time prior to the December 2019 restructuring, when PBF incentivized provision of this service. The ICR (para. 41) reports that, “No reliable data were available to assess impacts of the PBF on family planning.” In the absence of sufficient evidence, progress against these targets is assessed modestly achieved.  By the project’s end, 82,800 deliveries were attended by skilled health personnel in project facilities, substantially achieving the original target, set in December 2019, of 100,000 (83 percent achievement), and exceeding the revised target of 80,000 (104 percent achievement). This was an intermediate outcome in the project’s results framework and reported as such in the ICR. But IEG considers it to be an outcome indicator for Objective 1.  The share of mothers receiving timely postnatal care exceeded targets in the 10 districts supported under the original project and AF1 (approved in 2015). This indicator was introduced in the December 2019 restructuring, but it is relevant to PBF efforts, which incentivized post-natal care from the project’s outset. o From a baseline of 90 percent, 99.7 percent of mothers received timely postnatal counseling in the original 10 districts, exceeding the targets of 93 percent, set in December 2019, and 95 percent, reset in December 2021. The ICR (para. 40) reports that the IE found that the project increased timely use of post-natal care by 14 percentage points. The very high coverage rates are likely due to denominator issues (see Section 9). o From a baseline of 60 percent, 99.4 percent of mothers received timely postnatal counseling in the six new districts added under AF2, exceeding the targets of 65 percent, set in December 2019, and 93 percent, reset in December 2021. Coverage of NCD services  Under Original Objective 1, the project aimed to track and increase the share of hypertensive adults in project districts currently receiving antihypertensive treatment, setting a target (under the 2015 restructuring) of 30 percent. Data was to be disaggregated by region (Sogd, Khatlon, and RRS) and by age (18-39 years; and 40+ years). This indicator was dropped under the 2019 restructuring because hypertention was no longer to be incentivized under AF2. This indicator was categorized as an IRI in the ICR. But IEG considers it to be an outcome indicator, as it is a measure of coverage. The ICR (Annex 1) does not report on progress in achieving this target prior to December 2019 when the treatment of hypertension was incentivized under the PBF scheme. In the absence of sufficient evidence, achievement of this target is rated modest.  According to the IE, PBF and the concomitant trainings increased the availability of diabetes services offered by RHCs by 32 percentage points (ICR, para. 39).  The IE also reported that the project improved the share of adults who had their blood pressure measured by a health worker in the previous year by 8 percentage points and the share of adults self- reporting high blood pressure by 7 percentage points. In a context where two-thirds of hypertension cases go undetected, this finding indicates improved awareness of the condition. The IE also found Page 13 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) that the project raised the share of adults with self-reported high blood pressure who reported being prescribed medication by 4 percentage points. An indication of the baselines and actuals (rather than just the seemingly modest percentage point increases) in the ICR would have provided a better sense of the extent of progress achieved. General/composite coverage indicators:  By the end of the project, 2,176,204 people received essential health, nutrition and population (HNP) services, exceeding both the original target of 650,000 (335 percent achieved) and the target revised in 2019 of 1,350,831, and substantially achieving the target revised in 2021 of 2,230,685. o Of these people, 1,651,910 (or 76 percent) were female, exceeding the original target of 400, the target revised in 2019 of 1,216,908, and the target revised in 2021 of 1,364,483 (121 percent achievement).  According to the IE, PBF increased the share of adults over 40 years of age using any PHC by 6 percentage points and the share using any health services by 5 percentage points. Moreover, there was no crowding out of non-incentivized services. Facility records show no indication of reductions in rates of curative child and adult services not subject to bonus payments (ICR p. 43).  Rating is (barely) Substantial Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective Contribute to the improvement of the coverage of basic primary health care services in selected districts Revised Rationale The project’s theory of change and results framework logic, as presented under Original Objective 1, remained largely the same under the 2019 restructuring and AF2, but there were changes to outcome indicators. Two PDO indicators (share of pregnant women undertaking at least four ANC visits; and CPR) were dropped because of measurement challenges and replaced with two new PDO indicators (mothers receiving timely post-natal care services; and mothers counseled on nutrition, the latter classified as an IRI under the original design). Following suit, PBF indicators and payments no longer incentivized ANC or CPR, but continued to incentivize post-natal care and nutrition counseling. The only outcome indicator of NCD coverage (share of hypertensive adults currently receiving anti-hypertensive treatment) was dropped, but a quality indicator for the proper treatment of hypertensive patients was added (and assessed under Objective 2). In response to the IE’s finding of limited impact on coverage, a new intermediate outcome indicator was added to incentivize providers to visit all households in their catchment areas to encourage uptake of PHC services. Also under AF2, the formula for PBF was adjusted to reduce incentive payment amounts to improve prospects for post-project fiscal sustainability; more attention was paid to GoT ownership and integration of PBF with per capita financing; and PBF was scaled up to six additional districts. The December Page 14 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) 2021 restructuring adjusted for the (13-month) delay in AF2 effectiveness, and included support for a better transition to a new project and enhanced fiscal sustainability of PBF under GoT ownership. The revised design continued to support the same activities (PBF to incentivize coverage and quality of PHC services; strengthening of service delivery capacity; and Collaborative Quality Improvement (CQI) and Citizen Scorecard (CSC) implementation); outputs and intermediate outcomes (paying for progress against selected service delivery indicators for priority PHC services; enhanced service delivery capacity; and enhanced citizen engagement); and outcomes (improved coverage of priority services, delivered at the PHC level. Outputs and intermediate results Functioning of the PBF pilot and its evaluation Under the December 2019 restructuring and AF2, PBF payments incentivized seven performance indicators. Changes from the original nine supported up until the 2019 restructuring included: the dropping of three reproductive health indicators due to measurement issues, and the addition of an outreach indicator to incentivize increased uptake of services. These indicators, showing changes from the original nine, include: Child health and nutrition: o Number of children under 13 months fully vaccinated (retained) o Number of children aged 0-24 months, whose weight and height were measured according to the recommended growth chart and whose parents were advised on proper nutrition (retained); o Number of children under five with detected malnutrition, whose parents received advice on proper nutrition (revised, dropping previous references to advice on child care and improvements in nutritional status.); Reproductive health: o Number of pregnant women with first visit to antenatal care (ANC) during the first 12 weeks of pregnancy (dropped); o Number of pregnant women with at least four ANC visits to RHC/HH (the last visit falls into last three weeks of pregnancy)/(last visit falls into last two weeks of pregnancy) (dropped); o Number of mothers who received first postnatal patronage visit at home within the first three days after discharge from the Maternity House (retained); o Number of women aged 15-49 years newly provided with modern contraceptives during the verified period; and number of women aged 15-49 years who were previously provided modern contraceptives and continued to receive oral contraceptives or the next IC injection (dropped); Non-communicable diseases: o Number of newly detected and registered patients with hypertension (retained); o Number of patients with hypertension under the continuous treatment, at the end of the month (retained); Outreach: Page 15 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) o Number of household visits made by health personnel during the verification period (new) The high achievement of outputs and intermediate outcomes related to the functioning of the PBF pilot, reported under Original Objective 1, apply here, as well (facilities covered, timely payments, independent verification visits). The high achievement of outputs and intermediate outcomes related to the enhancement of citizen’s engagement, reported under Original Objective 1, apply here, as well (citizens scorecard sessions; women’s participation, and districts acting on community action plans). The high achievement of outputs and intermediate outcomes related to enhanced service delivery capacity of PBF facilities, reported under Original Objective 1, apply here, as well (health personnel training, upgrading and equipment of facilities, home visits). Outcomes Assessment of outcomes is the same as detailed under Original Objective 1, with the exception that Reproductive Health coverage is only assessed by one service indicator (postnatal care), and no longer by ANC coverage and CPR. In short:  Coverage of child health and nutrition services shows positive trends, with some caveats. Because of unreliable denominators, coverage is reported in terms of numbers of women and children receiving services, with targets exceeded. However, (1) details on the coverage and the exact services provided are lacking on those indicators; (2) the actual coverage reported on children under two receiving growth monitoring and counseling (100 percent) and children fully vaccinated (99 percent) are likely overstated (as noted in the ICR) because of unreliable denominators. The IE documents statistically significant positive impacts for HHs, where service availability increased by 24 percentage points for growth monitoring, 12 percentage points for child nutrition services, and three percentage points for iron folate supplementation. But the IE did not find significant effects on the timing and number of growth checks for children under two. Substantially achieved (based on positive trends).  Coverage of reproductive health services also shows positive trends, although likely overstated in terms of actual coverage levels. Virtually all – 99.7 percent of mothers in the original 10 project districts and 99.4 percent of mothers in the additional 6 districts under AF2 – are reported to have received timely postnatal care, showing an increase over the respective baselines of 90 percent and 60 percent. The IE found that the project increased the timely use of PNC by 14 percentage points. A total of 82,800 deliveries were attended by skilled health personnel in the project facilities, almost achieving the original target of 100,000 and exceeding the revised target of 80,000. Substantially achieved.  Coverage of NCD services. The ICR does not report on the percentage of hypertensive adults currently receiving anti-hypertensive treatment, an indicator introduced in August 2015, with a target of 30 percent. The December 2019 restructuring dropped this indicator because hypertention treatment was no longer incentivized under AF2. However, the task team, in a June 28, 2024 supplemental note to IEG, documents that the number of newly detected and registered patients with hypertension and the number of patients with hypertension under continuous treatment as of the end of the month were incentivized under PBF throughout project implementation. According to the IE, PBF and concomitant trainings increased the availability of diabetes services offered by RHCs by 32 percentage points. Substantially achieved. (The quality of NCD services is assessed under Objective 2.) Page 16 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130)  General/composite coverage indicators showed increases in the number of people receiving HNP services, exceeding the original, 2015, and 2019 targets, and substantially achieving the 2021 target. Within this group, the number of females receiving HNP services exceeded all targets. Substantially achieved. Revised Rating Substantial OBJECTIVE 2 Objective Contribute to the improvement of the quality of basic primary care services in rural health facilities in selected districts (2013) (with 2015 revised statement eliminating reference to “rural health facilities”, assessed by IEG as not reason enough to warrant a split rating) Rationale The project’s theory of change and results framework are essentially the same as those detailed under Original Objective 1, with the changes detailed under Revised Objective 1. The activities, outputs, and intermediate results supported were synergistic in their support of improving both the coverage and the quality of PHC services. Under the original design, quality was to be measured by two outcome indicators: the share of children under five with diarrhea treated with any Oral Rehydration Therapy (ORT); and the scores of a composite health facility quality index. In 2019 the first of these two indicators (treatment of diarrhea in children under five) was dropped due to measurement challenges (ICR p. 11). But, it was captured in the Child Health category of the health facility quality index (the quality of diagnostics and treatment of diarrhea among children < 5 years). Quality was also to be measured by the correct treatment of hypertension. Outputs and intermediate results The same outputs and intermediate results reported under Original Objective 1 also contributed to improvements in the quality of services. In short:  The functioning of the PBF pilot (facilities covered by PBF, timely payments for performance, independent verification visits) was highly achieved;  The enhancement of citizen’s engagement (citizens scorecard sessions, women’s participation, and districts acting on community action plans) was highly achieved; and  The enhancement of service delivery capacity of project facilities (health personnel training/skills development covering both clinical and management aspects, upgrading and equipment of facilities, and home visits for improved outreach) was highly achieved. Outcomes: A health facility quality index was used to track levels and trends in quality of care for project-supported facilities. The index used verified data from project facility records, based on indicators of infrastructure, Page 17 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) equipment, availability of medicines, administrative and managerial processes, and adequacy of clinical care. Under the original quality index design, each project facility was scored on each item in the index, creating a facility score that ranged between 0 and 300 for rural health centers (RHC) and 0 and 180 for health houses (HHs). The following results show average improvements in the project facilities, as follows:  Early 2015 quality index scores revealed an average baseline score of 201 out of 300 for RHCs (67 percent) and 125 out of 180 (69 percent) for HHs in the first eight districts included in the project. Targets were set at average scores of 83 percent by project completion. The ICR (para. 47) reports that already by the end of 2015, scores in project facilities in the first eight districts had already exceeded the 83 percent target (but does not cite those exact end-2015 scores). In 2017, the quality index was revised, placing increased emphasis on quality of services relative to other quality indicators, and a new manual was prepared. Revised scoring was organized around four quality indicator categories, which were applied during 2019-2022: (1) administration (administration, hygiene and sanitation, consulting room, laboratory services, and drugs and supplies management); (2) Child Health (medical records; identification and treatment of childhood malnutrition, diarrhea, and acute respiratory infections); (3) Maternal Health (medical records, quality of postnatal home visits within first three days after delivery); and (4) Non-Communicable diseases (medical records, hypertension treatment quality). Under this new, more refined quality index, perfect scores were changed to 1253 for RCHs and 583 for HHs. By the time of project completion:  From a baseline of a 60 percent average health facility quality of care score for RHCs, and a 51 percent average for HHs, the 10 participating districts supported under the original project and AF1 combined achieved an average score of 91 percent for RHCs (1144/1253) and 91 percent for HHs (532/583), exceeding all targets: those under the original design and the subsequent revisions to the quality index (set at 83 percent for RHCs and HHs), those set under the 2019 restructuring (83 percent for RHCs and 73 percent for HHs), and those set under the 2021 restructuring (88 percent for RHCs and 86 percent for HHs).  From a baseline of a 55 percent average health facility quality of care score for RHCs, and a 50 percent average for HHs, the six additional districts supported under AF2 achieved an average score of 84 percent for RHCs, and 81 percent for HHs, exceeding the targets set for these AF2 districts of 65 percent for RHCs and 55 percent for HHs. The ICR (paras. 48-52) presented results from the 2018 endline IE survey and other sources indicating the project’s impact on improvements to PHC infrastructure, the availability of medical equipment and medicines, and the knowledge and clinical practice, all components of enhanced quality of care.  According to the IE, the project led to increases of: 17 percentage points in access to piped water, 33 percentage points in the availability of heating in patient rooms, 59 percentage points in the availability of private consultation rooms for RHCs and HHs; and 24 percentage points in the availability of patient toilets. The project also increased the availability of infection prevention and control infrastructure and equipment in RHCs, including sharps disposal in consultation rooms.  The IE found significant improvements in the medical equipment availability score, based on 23 items for RHCs and HHs. For RHC labs, the availability of Glucometers and refrigerators for reagents increased by 20 and 14 percentage points, respectively; and there were large increases in the availability of cold storage equipment (e.g., a 47 percentage point increase in the availability of ice- lined refrigerators). The IE also found an estimated 20 or more percentage point increase in the Page 18 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) availability of five of the seven essential medicines. Impacts on Amoxicillin and HIV test kits availability showed very significant improvements of 52 percentage points and 57 percentage points, respectively.  The IE found evidence suggesting improved clinical knowledge and practice, but also some remaining gaps in clinical care quality. The share of providers able to correctly diagnose severe dehydration increased by 26 percentage points, and the share of providers recalling recommending exam procedures for cardiovascular assessment increased by 14 percentage points. Direct clinical observations indicated that healthcare workers in PBF facilities provided higher quality of care. Two examples include: an increase of 17 percentage points in the share of consultations with children under five, during which measurement of weight and height were registered; and an increase of 12 percentage points in the share of core physical exam activities carried out during adult consultations. Two gaps in clinical practice include: the mean share of correctly conducted adult examinations in PBF facilities survey standing at only 47 percent in the 2018 endline survey; and the share of core clinical history questions asked during adult consultations increasing by 17 percentage points, but remaining at 50 percent at endline.  The Collaborative Quality Improvement intervention led to an increase of 13 percentage points in the share of recommended physical examinations carried out during consultations of children under five. But there was no impact on the quality of adult consultations.  Household survey data showed significantly higher levels of patient satisfaction in PBF districts, with respect to staff competency, and the engagement of PHC providers with the communities. Adults above 40 were also significantly more likely to agree that PHC staff is welcoming and respectful. When asked about changes over the previous three years, adults in PBF districts were significantly more likely to report improved collaboration between the community and health facilities. Another measure of quality was the correct treatment of hypertension patients.  According to PBF verification records, from a baseline of 80 percent, 98 percent of hypertension patient charts showed treatment according to protocol, in the selected facilities in the 10 districts supported under the original project and AF1, exceeding the target of 85 percent set in December 2019, and fully achieving the target of 98 percent, set in 2021.  PBF verification records also reveal that selected facilities in the six new districts supported under AF2 also exceeded all targets set for this same indicator. From a baseline of 20 percent, virtually all (98.9 percent) of the hypertension patient charts showed treatment according to protocol, exceeding the targets of 30 percent set in December 2019, and 89 percent in 2021. Rating High OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall efficacy in achieving the original PDO is rated as Substantial, but barely so. Achievement of Original Objective 1 (contribute to the improvement of the coverage of basic primary care services in rural Page 19 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) health facilities in selected districts) is assessed as Modest, overall, despite positive trends documented for some of the indicators. The two main PDO indicators for Objective 1 (four or more ANC visits during pregnancy; and contraceptive prevalence rates) could not be measured because of inadequate denominators, and so were dropped in December 2019, six years into the project, and also consequently dropped from the PBF performance indicators, even though these remain important primary reproductive health services, needed in Tajikistan. The IE revealed no progress on ANC and no data on CPR. Nevertheless, other reproductive health indicators (timely post-natal care; and assisted deliveries) showed positive trends, along with indicators of child health and nutrition services coverage (immunizations, nutrition counseling) and general/composite coverage indicators for PHC. Most of these indicators are reported in terms of number of people served, but lack detail on services delivered and coverage. Those few that reported on the share of the target population covered showed very high coverage rates, likely (according to the ICR) over-estimated because of the unavailability of accurate denominators. Achievement of (unchanged) Objective 2 (contribute to the improvement of the quality of basic primary care services in rural health facilities in selected districts) is assessed as High. Health facility quality index scores exceeded targets, both for the facilities in the 10 districts supported under the original project and AF1, and for the facilities in the six additional districts supported under AF2. Virtually all hypertension patient charts inspected by PBF visits showed correct treatment, according to protocol, exceeding targets. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Overall efficacy in achieving the revised PDO and associated targets under the December 2019 restructuring is rated as Substantial. Revised Objective 1 was substantially achieved. Project data show positive trends in the uptake of child health services (immunizations, basic nutrition services, such as growth monitoring and counseling), and reproductive health services (assisted deliveries, timely postnatal care), as well as in general/composite indicators (number of people/women who received essential HNP services). However, there were some shortcomings in data quality, especially lack of disaggregation of data for more detail on what groups are receiving what services, and inaccurate denominators that culminated in the reporting of extremely high coverage of target populations, which may be (according to the ICR) overstated. Achievement of (unchanged) Objective 2 (contribute to the improvement of the quality of basic primary care services in rural health facilities in selected districts) is assessed as High for the same reasons as stated above. Overall Efficacy Revision 1 Rating Substantial Page 20 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) 5. Efficiency Economic Efficiency. Project spending was focused on interventions and priorities where return on investment was high, including: (a) health promotion, prevention, and PHC, which could prevent and address a majority of health issues, saving higher-level facilities from having to provide costly services (and reducing out-of-pocket expenses); (b) primary health care quality, an inherent weakness and a necessary step for enabling services to produce better health outcomes; and (c) cost-effective services for preventing and addressing critical health conditions which account for a major share of the disease burden, specifically: MCH services, nutrition and NCDs, especially the detection and treatment of hypertension. An end-of-project cost benefit analysis (CBA) considered the benefits arising from the delivery of MCH services in rural areas to improve access, which would in turn decrease maternal mortality and under-five mortality. It also considered the benefits from investing in the improvement of healthcare infrastructure and equipment and the skills development of healthcare professionals, resulting in improved access to high quality healthcare services. Based on a total disbursement of $40.99 million over 10 years, and using a 3 percent discount rate (consistent with similar contexts) to reflect the time preference and risk premium, the project benefits were estimated at $284.87 million and the overall internal rate of return (IRR) was calculated at 160 percent, with a net present value (NPV) of $192.78 million. This calculation was based on relatively conservative assumptions, and still the benefits outweigh the costs with a benefit-cost ratio of 6.5, an indication of high value for money. Implementation efficiency was substantial overall. Activities were implemented as planned and project funds were fully disbursed, including IDA financing and two trust funds (detailed in Section 2.e). The parent project moved swiftly from approval to effectiveness. Cost overruns for civil works were quickly addressed under AF1 so that all activities were implemented as planned, or with minor delays, during 2013-2019. A one-year delay in AF2 ratification was caused in part by the disruptions of the COVID-19 pandemic, and also by GoT’s concerns about the fiscal and technical scalability of the pilot and consequent disagreements over the allocation of AF2 between PBF and physical infrastructure. Bridge financing was secured to maintain project coordination staff for most of the lapse period, and AF2 proved to be an important part of the country’s COVID-19 response. The delay in ratification required a 12-month extension of the closing date and led to the cancellation of some AF2 activities, including the piloting of electronic patient registries. Despite challenges imposed by COVID-19, all main activities under AF2 (scaling of PBF to additional districts, trainings, and investments in infrastructure and equipment) were completed as planned. A shortcoming in project design and implementation, which undermined efficiency, was the failure to discuss with GoT and to assess more fully the prospects for sustainability and GoT ownership of the PBF scheme. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 21 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Overall Outcome Ratings Achievement against original Achievement against revised Ratings dimension objective objective Relevance of objective High Efficacy Obj 1: Contribute to improvement of the coverage of basic PHC services Modest Substantial in selected districts Obj 2: Contribute to improvement of the quality of basic PHC services in High High selected districts Overall Efficacy (barely) Substantial Substantial Efficiency Substantial Outcome Rating MS S Outcome Rating Value 4 5 Amount disbursed $28.99 million $15.40 million Disbursement (%) 65 percent 35 percent Weight Value 2.6 1.75 Total weights 4.35 (rounded to 4.0) Overall Outcome Rating Moderately Satisfactory (4.0) Outcome Rating: The relevance of the PDO is rated High, as it is responsive to current country conditions, the country’s current priorities, and the World Bank’s current strategy for Tajikistan. Efficacy in achieving Original Objective 1 (contribute to improved PHC coverage) is rated Modest because of lack of progress in ANC coverage and in the CPR and M&E issues, which undermined reporting on some other aspects of coverage, while efficacy in achieving Objective 2 (contribute to improved PHC quality) is rated High, with all quality targets exceeded, for an overall Efficacy rating of Substantial, but barely so because of M&E issues in measuring coverage. Efficiency is rated Substantial because the project focused on cost-effective interventions and was implemented efficiently overall. These subratings translate into an overall outcome rating of Moderately Page 22 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) Satisfactory under the original PDO, taking into account the Modest achievement of Objective 1 and data issues, undermining the accurate and full reporting of coverage outcomes. With the dropping of the ANC and CPR outcome indicators, efficacy in achieving revised Objective 1 is rated Substantial. While evidence shows positive trends overall in the number of people provided key MCH services, including child immunization and nutrition services, postnatal care, and general PHC services, and PBF’s effects on service availability, there is a lack of specificity in the data presented to assess the actual coverage of target populations with various services (when it is reported, actual coverage is overstated because of inaccurate denominators) and a lack of specificity in the exact services rendered (e.g., “number of children immunized” falls short of indicating the share of children of a specific age fully immunized). Efficacy in achieving (unchanged) Objective 2 is rated High, for an overall efficacy rating of Substantial. Efficiency is rated Substantial. These subratings translate into an overall outcome rating of Satisfactory under the revised PDO. Given that 65 percent of total project financing was disbursed under the original PDO, and 35 percent was disbursed under the revised PDO, the weighting of outcome ratings before and after the December 2019 restructuring culminates in an overall Moderately Satisfactory outcome. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Because of project design issues and a late start in transition planning (details in Section 8), GoT has deemed the PBF approach to be too fiscally and technically challenging for nationwide scale-up. The consequent loss of the high-powered incentives to improve the coverage and quality of PHC services at project closing poses a significant risk to PDO achievements. A qualitative study undertaken when PBF payments lapsed in 2020 because of delayed effectiveness of AF2 indicated that while facilities generally managed to maintain their infrastructure and quality enhancing processes introduced under PBF, staff foresaw challenges with facility maintenance and modernization going forward. The loss of PBF funding had already led to supply shortages (medicines, kits, reagents, office materials), which undermined quality of services (e.g., hypertension management and counseling). Health care managers observed that the removal of bonuses coincided with a decrease in staff interest and motivation to provide time-consuming preventive services. Towards the end of the project, when PBF payments were reduced, facilities had to rely more heavily on limited GoT funding to maintain project investments. Funding constraints led to reports of fuel shortages for ambulances and challenges maintaining new equipment, managing medical waste, and ensuring uninterrupted water and electricity supply. Nevertheless, GoT remains committed to health sector reform to maintain and expand this project’s achievements and to sustainable improvements in PHC financing and delivery. It is embarking on an ambitious reform of PHC financing, including the establishment of a national strategic purchaser and the piloting of strategic purchasing of PHC services in Sughd oblast and Dushanbe. The follow-on Tajikistan Millati Solim (Healthy Nation) project (P178831) will support the introduction of the national purchaser and accompanying institutional reforms to pave the way for replacing input-based financing with blended PHC financing, which may include some elements of PBF. The project will digitize PHC services in pilot regions to enhance evidence-based decision-making, transparency and accountability at all levels of the PHC Page 23 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) system. It will also support a new payment mechanism that will be budget neutral initially to enhance prospects for national scale-up, and continued investments in PHC infrastructure and equipment. 8. Assessment of Bank Performance a. Quality-at-Entry The project’s objectives were aligned with the country’s needs and development priorities, as well as with the Bank’s own strategies and priorities. The project’s theory of change and its results chain were logically laid out and actions, and outputs plausibly linked to intermediate outcomes and outcomes. They aimed to support the strengthening of high impact MCH and NCD services at the PHC level, with an emphasis on prevention and wellbeing, and to incentivize, through a PBF pilot, improvements in the coverage and quality of these services, delivered at lower cost to GoT and patients than specialist services. Uptake of these services was expected to culminate in the impact of better health of the populations covered. Technical, financial and economic aspects were grounded in experience and studies, gleaned from Tajikistan and other countries, making the design both logical and evidence-based. As detailed in ICR (para. 66), the project design drew on new evidence, built on a range of preparatory studies and analyses conducted in the project regions, and harnessed the relevant experience of other development partners. Project design also benefited from a year-long pre-pilot phase of PBF implementation in one district, before scaling up to an additional seven. Environmental and fiduciary aspects (as described in Section 10), and implementation arrangements were adequately factored into the design. There were, however, two significant shortcomings in quality at entry. First, because of inadequate denominators, the outcome indicators chosen to assess achievement of Objective 1 (coverage of key services) were not measurable. As detailed in Section 9, this was an issue that persisted throughout implementation. Second, the project lacked a clear empirical case for project continuation under GoT funding. During the design phase, the fiscal, technical, legal, and political challenges of introducing PBF over an input-based financing system (or its integration with a new capitation mechanism under per capita financing) were not sufficiently discussed or explored by the Bank with GoT with a view to outlining a mutual vision and feasible pathway for sustainable scaling during the project design. Indeed, such transition planning only began in earnest during the final project stage, too late to gain substantial traction. The initial (2015-2019), high-powered incentives paid to facilities for performance were found to be very high and not sustainable. Even after their reduction under the December 2019 restructuring (and AF2), they raised PHC worker salaries by an average of 70 percent, leading some stakeholders interviewed for the ICR to suggest that the design was driven more by a general interest in investigating possible impacts of PBF than by the fiscal realities of the country. Because the IE only evaluated impact of the more aggressive incentivization of the project’s early years, the impact of the reduced, more sustainable incentives, applied during 2020-2023, remain unclear. Stakeholder analyses undertaken during design (as described in ICR, para. 65) attempted to identify possible supporters and detractors, enable strategic engagement and address concerns to the extent possible. But this was not sufficient to engage GoT in the exploration of PBF or its feasibility and sustainability post-project. Page 24 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Project supervision was focused on development impact, facilitated by its tracking of the reporting – and independent verification – of performance against PBF indicators. Moreover, the timing of the IE, with its mid-course (2018) endline data set and early 2019 publication, facilitated its use not only as a mid-term check on project performance and progress against outcome indicators but also as a source of lessons for fine-tuning design for even greater impact. However, the M&E issues surrounding the tracking of coverage indicators persisted. Supervision was regular, with 20 ISRs issued over the course of 9.5 years. Aide- memoires routinely provided evidence of progress and incorporated professional advice from World Bank experts. The findings and ratings of the 2017 mid-term review and all ISRs were candid in their assessments of progress and their ratings were realistic. Supervision of fiduciary and safeguard aspects and the adequate mitigation of any issues were satisfactory (as detailed in Section 10). The World Bank was responsive to client needs and demonstrated flexibility in adapting to evolving priorities, including proactive restructurings, while still aimed at the goals of improved coverage and quality of priority PHC services. It adapted the PBF performance-based formula to strengthen weakly performing areas of clinical care, identified by the IE. Specifically, performance bonus calculation and quality improvement support were shifted towards clinical care quality enhancement and the level of PBF payments was reduced for greater sustainability of PBF, and greater possibility of transitioning to GoT funding after completion. AF2 also responded the COVID-19 pandemic, incorporating of COVID-19 prevention, counseling and screening into the newly incentivized home visits. The Bank was diligent in communicating with GoT, providing up-to-date information and analysis on the status and impact of activities, issues encountered, and exploring options to address issues, which arose in light of evolving needs. Stakeholders’ feedback during the ICR mission revealed GoT’s appreciation of the Bank’s technical skills and advice. Stakeholder interviews also revealed their appreciation of the Bank’s coordination with other development partners, particularly GIZ, World Health Organization and UNICEF, all with a view to optimize resources supporting PHC. Project supervision also promoted coordination among all government agencies participating in the project. Shortages and high turnover of healthcare workers and verifiers posed challenges to implementation, with well-trained workers’ opportunities to work in other Russian-speaking countries for better remuneration. For the same reasons, human resources challenges also undermined the work of the project coordination group. While it performed well, vacant positions proved hard to fill and new hires could require extensive training before functioning effectively. There were also some shortcomings. First, stakeholders raised the high turnover of international Bank staff as a particular challenge. Five TTLs led the project in the 10 years between project approval and closing. This frequent change in project leadership (and in other team members) created substantive transaction costs, especially given Tajikistan’s complex health system and political environment. Moreover, few of the international staff spoke Russian. Second, Bank and GoT resources committed to the per capita financing policy were insufficient to achieve the comprehensive institutional changes required for full-fledged capitation. With only one per capita financing staff and one international consultant dedicated to this policy, the conceptual, financial and political resources in the Bank and GoT dedicated to this activity did not match these substantive demands. PBF was discontinued at project closure because no fiscally and technically viable transition plan could be developed. Despite AF2 support of transition planning in the last two years of implementation, the financial, technical and administrative requirements of scaling up PBF were deemed too high by GoT, so the PBF pilot was discontinued at Page 25 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) project closing. Closer collaboration between the project coordination group and MoHSSP throughout the project might have increased the prospects of transitioning to GoT financing through improved GoT capacity and ownership (ICR, para. 73). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s objectives were clearly stated, and the theory of change was sound. While the results framework adequately captured the intended coverage and quality outcomes of priority MCH and NCD services delivered through the PHC facilities, the outcome indicators were not measurable. Because the project supported only a few selected facilities in a select number of districts, in a select number of regions, the challenge was to establish, at the outset, an accurate denominator for the population (and an estimate of the various target groups therein) living within the catchment area of each facility. Instead, DHS 2012 and 2017 were designated as baseline and endline data sources. But the denominators for populations (and target groups therein) available under these surveys were only disaggregated to the regional level. Since project facilities made up a small portion of all PHC facilities in the 16 project districts, the DHS region-level denominators were not representative of the project facilities and thus could not adequately capture the project effect. Moreover, DHS 2017 (timed to take place only two years into the rollout of PBF and less than a year after the first civil works had been completed) was premature for measuring endline project outcomes, even for the original closing date of January 31, 2019, and certainly for the actual closing date of June 30, 2023. The multiyear coverage of DHS data (e.g., for ANC coverage) also challenged a proper baseline. Because of DHS issues, the project drew on PBF MIS data from the project facilities to track the population coverage indicators, but unreliable data on the denominator generated unrealistically high service coverage rates (95-100 percent). A number of intermediate results indicators were misclassified, as they were actually measuring coverage outcomes. Some indicators made it difficult to assess real coverage (e.g., number of children immunized, instead of share of children belonging to a specific age group fully immunized (defining full immunization). On the other hand, the facility quality index provided a composite measure of service quality with scores based on systematic assessments of (1) administration (organization and management, hygiene and sanitation, consulting room, laboratory services, management of drugs and supplies); (2) child health (medical records, quality of nutrition, diarrhea, and acute respiratory infection services); (3) maternal health (medical records, quality of ANC); and (4) NCDs (medical records, quality of hypertension treatment). The M&E design also included an Impact Evaluation (IE) which would seek to ascertain (i) impact and cost- effectiveness of the PBF model implemented in Tajikistan; and (ii) whether PBF is more effective if Page 26 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) implemented in conjunction with additional low cost interventions that address potential constraints to PBF effectiveness: CQI and citizens report cards. PDO-level and intermediate outcome indicators were to be monitored using: (a) routine reporting through PBF MIS; (b) baseline and follow-up surveys; and (c) program reports. The M&E Coordinator of the MoH Coordination Group, with the support of a local consultant, was given responsibility for preparing progress reports, monitoring key performance indicators and results, and apprising the Bank of progress. Data collection and reporting were to be harmonized with country systems. The PBF MIS was to leverage ongoing efforts to upgrade the country’s HMIS and efforts were to be made to ensure compatibility of the PBF MIS software modules with the new District Health Information System II, supported by the EU. Facilities were responsible for self-reporting on PBF indicators through routine reporting and internal verification channels, with semi-annual independent verification of PBF reporting led by UNICEF. b. M&E Implementation The 2017 mid-term review flagged the measurement challenges and prompted the project to explore alternative ways to obtain better data. However, neither an additional round of household data under the IE, nor the computation of district-level indicators from the DHS could be implemented. Measurement challenges prompted the dropping of three PDO coverage indicators under the December 2019 restructuring and AF2 approval (ANC, CPR, and ORT). ANC and CPR were also dropped from the services incentivized by the PBF scheme. The new PDO indicators (timely PNC and the number of mothers receiving nutrition counseling), already included among PBF indicators, were strategic as they were also cost-effective interventions. Because of reliance on PBF MIS data, there were persistent concerns about the unreliability of the denominator for the PNC coverage indicator. It is likely that the high turnover of TTLs (5 TTLs over the 10 years between approval and closing) contributed to the lack of consistent attention to M&E. Outside the measurement issues for assessing coverage, M&E plans were successfully implemented. Tracking of the facility index score was successful, as was the tracking of output- oriented indicators. Data collection, reporting, and independent verification were undertaken largely as planned. The IE remedied some of the shortcomings of the original results framework, enabling the identification of causal effects of the project and providing some mid-project data (from seven of the 10 original districts supported under AF1) on ANC coverage and a few other coverage indicators (reported in Efficacy section), but not for CPR and ORT. Because of its timing and coverage of the early years of the project (2014-2018), IE results provided an opportunity for learning and adaption within the project and enabled important contributions to global knowledge on PBF in low- and middle-income countries. The project also commissioned two qualitative studies assessing: (1) the impact of the lapse in PBF payments in 2020, and (2) avenues to improve the uptake of PHC services. c. M&E Utilization M&E results were used to monitor implementation and take corrective action to enhance progress towards the intended outcomes. The facility quality index scores were used to identify and address performance gaps. An example of this was the assignment of higher weights in the performance bonus calculation to clinical care quality indicators when it was documented that they lacked progress compared to indicators of structural quality. Results from the IE study were also integrated into project Page 27 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) design refinements. An example of this was the introduction of incentives for patient home visits to increase service uptake after the IE revealed limited impacts on some indicators of PHC utilization under the original performance incentives. M&E is rated Modest because the design had significant shortcomings, which impeded the measurement of the coverage objective and were not fully resolved during implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was assigned an Environmental Assessment category “B” in light of planned rehabilitation of RHCs in the pilot districts. The Borrower had a satisfactory environmental and social performance track record throughout the project’s life. The Environmental Management Framework (EMF) was prepared for the original project and updated for AF1 and AF2 in light of the expanded scope. The EMF provided guidelines for site-specific Environmental Management Plans (EMP), including mitigation and monitoring measures. EMPs and related checklists were disclosed and publicly consulted in all participating villages and made effective through the signature of civil works contracts that included an EMP checklist in annex. EMP implementation was monitored by the PCG Environmental Specialist and by Supervising Construction Engineers at sites and reported to the PCG and the World Bank. The Bank’s Environmental Specialist also monitored EMP implementation, working jointly with the MoHSSP, PCG and contractors to assess compliance of civil works with site-specific EMP and with the Occupational Health and Safety Requirements. Environmental safeguards implementation support visits confirmed that EMP-related measures were monitored effectively and that there were no serious complaints from stakeholders or the public. Specific attention and recommendations were provided to improve solid waste management, dust suppression, and asbestos management at the sites. The ICR (para. 81) reports that, "The overall safeguards rating was Satisfactory during project implementation." Citizen Engagement. The project developed and implemented the Stakeholder Engagement Plan (SEP) and Grievance Redress Mechanism (GRM). The project-supported CSC mechanism supported community engagement and enhanced the accountability of health providers. The process involved (i) focus groups of citizens/users of health services to score performance and behavior; (ii) PHC staff self assessment; and (iii) a jointly developed action for each RHC and attached HHs approved by the district authority. Civil society facilitators ensured that the voices of vulnerable groups were heard, culminating in enhanced dialogue, mutual understanding, and motivation to resolve issues jointly. Under AF2, there was a dedicated grievance redress hotline and email-based service. Information boards with contact information at central and regional levels and “complaints and suggestion boxes” were placed at PHC facilities. Household engagement visits encouraged citizens to communicate their complaints and suggestions for service delivery improvements. Citizen feedback informed the action planning by respective PHC facilities. Page 28 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) b. Fiduciary Compliance Financial Management (FM) performance was assessed as Satisfactory and Moderately Satisfactory throughout implementation. FM arrangements at the PCG were satisfactory and acceptable to the Bank. MoHSPP updated its automated accounting system to accommodate the rigorous requirements for project accounting and reporting. Interim financial reports were submitted in a timely manner, providing reliable financial information. Audits by independent auditors were regularly undertaken, and their opinions were unqualified. The processing of transactions through the MoF Treasury, where the designated accounts were held, had its challenges. Converting US dollars to TJS took one to three working days, and settling payment orders took up to 10 days. Review and signature of withdrawal applications for designated account replenishment and direct payments by MoF required up to 14 days. Consequent delays in the disbursement of project funds hindered the timely completion of activities. Amid continuous local currency devaluation, MoHSSP incurred substantial foreign currency exchange losses, exacerbated by the fact that currency conversion from the US dollar designated account at the National Bank and the corresponding payment transactions in TJS from the Treasury account did not occur on the same dates. Under AF2, the designated accounts were relocated from the Treasury to a commercial bank. While this addressed the issue of losses from USD/TJS currency conversions, the project continued to face delays in MoF approval of withdrawal applications until the closing date. There were also delays in GoT’s payment of social tax obligations on performance-based incentives. In line with Agreed Minutes of Negotiations between GoT and IDA under the original project and AF2, the Recipient committed to providing a government contribution to cover social taxes on performance-based incentives. But the payments of the 25 percent social tax from the MoF to the Tax Committee faced considerable delays throughout AF2 implementation. GoT did not pay the fourth quarter social tax, leaving a Government debt to the Tax Committee at project closing in June 2023. MoF affirmed their commitment to pay the debt in various meetings with GoT agencies and the Bank, and with the ICR team. The debt of $28,595 remained outstanding as of the ICR publication. Procurement capacity of the PCG was adequate. PCG worked in close collaboration with MoHSSP procurement staff on the preparation of procurement documents and acceptance of deliverables. MoHSSP’s Procurement Department staff, Tender Evaluation Committee members and PGC staff attended several Bank procurement trainings, covering the Bank’s New Procurement Framework and its Guidelines on Prevention and Combating Fraud and Corruption. Procurement shortcomings included weaknesses in the contract administration system, delays in Systematic Tracking of Exchanges in Procurement (STEP) entries and contract award publication, and inadequate quality of procurement documents. Significant delays in civil works packages were due to the late preparation of PHC facilities design at the outset of the parent project. To address a key procurement risk of inadequate quality of civil works, the Bank highlighted the importance of independent third-party verification and recommended hiring an independent firm to this end. MoHSSP considered the option of regular physical inspections by the PCG engineers, under the supervision of MoHSSP Capital Construction department, to be sufficient, for this purpose. The Bank found that option to be sufficient, and the Bank’s technical expert visited sites regularly. This risk was well managed and the quality of construction was assessed as satisfactory. Page 29 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) c. Unintended impacts (Positive or Negative) The ICR (paras. 60-62) highlights in its Section II.E (Other Outcomes and Impacts): the project’s clear gender impact thanks to its focus on MCH services; the sector’s strengthened management and service delivery capacity, thanks to training and other supports; and its pro-poor design, which lifted some of the barriers that underserved rural populations face in accessing essential health services. All of these were intended impacts. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment IEG’s split ratings methodology assessed performance against Original Objective 1 as modest due to a lack of progress on ANC coverage, a lack of evidence on CPR coverage, and inadequate denominators for Outcome Satisfactory Moderately Satisfactory these and other coverage indicators. The 65 percent weighting of the MS outcome under the original PDO and the 35 percent weighting of the S outcome under the revised PDO result in an overall outcome of MS. Moderately Bank Performance Moderately Satisfactory Satisfactory Issues with inadequate approach to measurement of indicators related to ANC and CPR and other service coverage Quality of M&E Substantial Modest indicators (at design and during implementation) undermined the evaluation of Objective 1 (coverage) and are assessed as significant shortcomings. Quality of ICR --- Substantial 12. Lessons Page 30 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) The following lessons drawn from the ICR are slightly edited by IEG: When tracking and assessing coverage of priority PHC services, the failure to ensure data to estimate service coverage within the targeted population groups and catchment areas of the project can undermine the proper assessment of a project’s outcome. In the case of this project, DHS survey data were initially used, but they were only disaggregated at the level of the region, not at the level of selected health facilities supported under the project, or even at the district level, and so were not sufficiently representative. The project’s subsequent reliance on administrative data pointed to the importance of (a) capturing in the numerator all health service providers in a geographic area; and (b) the availability of reliable data on the denominator derived (or calculated) from the administrative data. Moreover, ensuring M&E capacity to measure all relevant service indicators is important early in a project. When a pilot of a complex reform like PBF is not closely aligned with a country’s financial and technical capacities and when exploring and crafting a feasible vision for scaling is not part of the pilot process from the outset, Government ownership and propensity to continue such a reform post-project is likely to fall short. The project addressed highly relevant challenges with an evidence-based design. But while the magnitude of PBF bonuses likely contributed to impacts, the bonuses posed a fiscal challenge and nationwide scaling remained beyond GoT capacity. Greater and earlier focus on fiscal sustainability, technical capacity building beyond the pilot areas, and deeper collaboration of PCG experts and MoHSSP may have increased the probability of scaling up under GoT financing. A further deepening of institutional integration and daily collaboration between Ministries of Finance and of Health and Social Protection of the Population is likely to enhance Government ownership and sustainability of health financing reforms post-project. The project was the first health operation in Tajikistan under which the Minister of Health and Social Protection of the Population headed the PCG. Still, coordination between the Ministry of Finance and MoHSSP, as a crucial actor for health financing reforms, remained limited in practice and contributed to lack of GoT ownership and meaningful dialogue about sustainability. The follow-on operation attempts to overcome these obstacles by: (a) closely coordinating design with MoF and MoHSSP at Minister-level; (b) an institutional arrangement making MoHSSP department heads leads for appropriate components of the project’s reform falling within their domain; and (c) an Intersectoral Committee chaired by the Prime Minister with Ministers of Finance and Health and Social Protection of the Population functioning as deputy chairs to coordinate health financing reforms. Paying more acute attention to the challenges of attracting and retaining healthcare workers is likely to inform and enhance the design of training and selection of candidates, as well as broader financing and human resource reforms in the health sector. The IE revealed that even the large salary incentives under PBF did not increase health worker retention rates, given the strong pull of better income opportunities for Russian-speaking specialists in surrounding countries like Russia. This challenge might be better assessed and addressed in future sector-wide reforms. 13. Assessment Recommended? Page 31 of 32 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review TJ Health Services Improvement Project (P126130) No 14. Comments on Quality of ICR The Quality of the Evidence provided in the ICR was somewhat sufficient, with some shortcomings (acknowledged in the ICR) because of M&E issues. What evidence was presented in the ICR was detailed and properly referenced. The ICR was candid in assessing the quality of the evidence it presented, especially how the lack of reliable denominators undermined the proper assessment of coverage of key PHC services. Annex 1.B provided even more detail on what was delivered by Component. The impact evaluation was used as a complementary source for assessing mid-project results. Quality of Analysis and Results Orientation were sufficient, overall, with two caveats. First, there could been a more systematic organization of the assessment of efficacy around the results framework, specifically the systematic presentation of performance-based incentives (and how they changed over time), and the linking of these incentives to outcomes (and how they changed over time). An email to IEG sent on Friday, June 28, 2024, provided this information fully and systematically, along with a fuller explanation of the health facility quality index scores. Second, a split rating would have highlighted the reduced outcome of Original Objective 1 prior to the 2019 restructuring, under which 65 percent of disbursements were made. Quality of Lessons was excellent. Lessons focused on the most salient experiences and challenges faced under the project. Moreover, they were candid and forthcoming and useful to other health projects undertaking similar reforms. Internal Consistency/adherence to guidelines. The ICR was internally consistent and largely adhered to guidelines. a. Quality of ICR Rating Substantial Page 32 of 32