Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) Report Number : ICRR0020140 1. Project Data Project ID Project Name P084977 HEALTH & SOC PROT Country Practice Area(Lead) Additional Financing Kyrgyz Republic Health, Nutrition & Population P112142,P125470,P125470 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-49570,IDA-H1970,IDA-H3910,IDA-H7000,TF- 30-Jun-2011 682,000,000.00 11265,TF-11531,TF-56540,TF-56799,TF-90119,TF- 91613,TF-97870,TF-98740 Bank Approval Date Closing Date (Actual) 15-Dec-2005 30-Jun-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 45,000,000.00 7,704,262.49 Revised Commitment 45,000,000.00 7,704,132.99 Actual 44,789,000.65 10,491,826.98 Sector(s) Health(50%):Compulsory health finance(22%):Central Government(18%):Other social services(10%) Theme(s) Health system performance(33%):Other social protection and risk management(17%):Child health(17%):Tuberculosis(17%):HIV/AIDS(16%) Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives Original objectives. The project’s original objectives, as stated in the Development Grant Agreement (DGA, p. 13), were “to improve the health status of the population of the Recipient through creation of an effective, comprehensive and integrated delivery system of individual and public health services, including increased responsibility of every citizen, family, society and public administration bodies for the health of each person and for the society as a whole." The Project Appraisal Document (PAD, p. 7) states the objectives differently and includes a social protection objective. The PAD states that the project would support implementation of the Manas Taalimi Health Reform Program through a Sector-wide Approach. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) Revised Objectives under the first restructuring. The first restructuring with additional financing (AF1), approved on 06/12/2008, retained the original PDO and added two more objectives as follows: The objectives of the Project are: “(i) to improve the health status of the population of the Recipient through creating an effective, comprehensive and integrated system for the delivery of individual and public health services, including increased responsibility of every citizen, family, society and public administration bodies for the health of each person and for the society as a whole; and (ii) improving and protecting the health and nutritional status of said populations against the instability of national food prices as well as assisting poor families to manage and mitigate the impact of rising food prices through scaling up and strengthening targeted cash transfers to said families and populations” (Financing Agreement, 6/25/2008, p. 4). The Project Paper of 5/27/08 describes the objectives as follows: The original project development objectives will be amended to include two additional specific objectives (objectives [d] and [e] below) under the proposed additional financing. The revised objectives will be to improve health status in the Kyrgyz Republic: (a) by improving access, financial protection, efficiency, equity and fiduciary performance in the Kyrgyz health sector; (b) to ensure sufficient and reliable financing for health; (c) to strengthen the targeting of social benefits by developing effective administration and information management systems to improve access to social services in general; (d) protecting and improving health and nutritional status of particularly vulnerable populations in the Kyrgyz Republic in the face of food price shocks, by providing nutritional supplements and nutrition education to pregnant/lactating women and infant/young children; and (e) helping poor Kyrgyz families manage and mitigate the impact of food price shocks and protect consumption (general and food consumption) by scaling up and strengthening targeted cash transfers. Revised objectives under the second restructuring. The second restructuring with additional financing (AF2), approved on 04/25/2011, added four new, distinct objectives under the following PDO statement: “The objectives of the Project are to assist the Recipient in (a) improving the performance of its health sector; (b) strengthening the targeting system in the delivery of social benefits; and (c) improving and sustaining the health and nutritional status of vulnerable populations affected by food price shocks" (objective 'c' actually consists of two objectives). The Financing Agreement of 2011 is consistent with the Project Paper of 5/11/2011. Following IEG/OPCS guidelines, this Review will assess achievement of objectives as stated in the financing agreements. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 06-Jun-2008 c. Components Component 1. Support for the Manas Taalimi Health Reform Program and Program of Work (appraisal: US$13 million; AF2: US$21.10 million; aggregating at an actual total IDA disbursement of US$34.10 million). Financial support was to be provided under a sector-wide approach (SWAp). The level and distribution of funds were to be adjusted annually based on sector performance, absorptive capacity, identified needs, and the availability of parallel financing for selected program expenditures. The decisions on the level of funding were based on the annual work programs, budget and procurement plans, progress against monitoring indicators, and results of biannual "Health Summits" between the government and donors. Component 2. Strengthening the Administrative System of the Ministry of Labor and Social Protection (appraisal: US$2 million, AF1 US$5.0 million; AF2: US$0.3 million; aggregating at an actual total IDA disbursement of US$7.30 million). This component was to facilitate the implementation of policy reforms and to strengthen the administrative systems enabling social protection and health systems to target subsidies to poor or disadvantaged households. Component 3. Protecting Health and Nutritional Status. AF1 added this component (US$1.0 million). Another US$2.6 million was added by AF2 to scale up activities supported under AF1, both aggregating at an IDA disbursement of US$3.60 million. The component was to provide nutritional supplements and nutrition education to pregnant and lactating women and children under five. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost Since the operation consisted of a SWAp, IDA funds were pooled with the state budget to finance a share of the annual program of work. Other partners listed below took a similar approach, while USAID supported specific parts of the program of work through parallel Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) financing. Under the SWAp, expenditures were not defined in detail ex ante as in a traditional investment. At appraisal, IDA financing amounted to US$15 million. With two additional financings in 2008 and 2011, the total estimated cost aggregated at US$45 million, with an actual total disbursement of US$44.79 million (with the difference due to exchange rate fluctuations). Financing The larger Manas Taalimi Program was appraised at US$652 million (2006-2011), but the actual cost reached US$1,678 million. Other financiers included U.K. Department for International Development, Swedish Agency for International Development, German Development Bank-KFW, Global Alliance for Vaccines and Immunization, Russian Federation, and Lichtenstein Development Service. The borrower’s initial contribution was US$448 million and increased to US$690 million during the period 2006-2011. The follow-on Program, the Den Sooluk Program (2012-2014), was estimated at US$1,304 million (ICR, p. 41), including a borrower’s contribution estimated at US$718 million. Dates The project became effective on 6/8/2006, underwent a mid-term review on 5/28/2008, and closed on 6/30/2015, four years beyond the original closing date of 6/30/2011, to accommodate two additional financings and one extension. The project was restructured on 06/12/2008, with an additional financing amounting to US$6 million. A second restructuring on 04/25/2011 brought a second additional financing of US$24 million (credit: US$13.20 million; and grant: US$10.80 million). A third restructuring on 03/12/2014 extended the project closing date to 6/30/2015 to complete the implementation of project activities. Disbursements under the three project phases: Disbursement reached 16% under the original objectives, 26% under the revised objectives/AF1, and 58% under the revised objectives/AF2. 3. Relevance of Objectives & Design a. Relevance of Objectives Relevance of the original objectives is rated Substantial. The project objectives were responsive to the Country Assistance Strategy for 2003-2006 which included stemming the deterioration in social services and key infrastructure. The objectives on improved health status were articulated in the national program and were carried over in the government's new health sector program (2012-2016). The project objectives continue to be relevant to the government’s National Sustainable Development Strategy (2013-2017) and to the current Country Partnership Strategy for the Kyrgyz Republic (FY14-17). Relevance of the revised objectives under AF1 is rated Substantial as both health and nutrition were country priorities, as was the need to assist the poor in managing and mitigating the impact of rising food prices. Relevance of the revised objectives under AF2 is rated Substantial in view of the importance of improving health sector performance, health and nutritional status of the vulnerable, and targeting in the delivery of social benefits. The Health Reform Program, Manas Taalimi, aimed at institutionalizing the reforms that were previously initiated under Manas I. The objectives for improved health sector performance were articulated in the national program and were carried over in the government's new health sector program (2012-2016). The objectives continue to be relevant to the government’s National Sustainable Development Strategy (2013-2017) and to the current Country Partnership Strategy for the Kyrgyz Republic (FY14-17). Rating Revised Rating Substantial Substantial b. Relevance of Design Relevance of Design under the original objectives is rated Substantial. The design of health interventions lays out a results chain that contributes to improved health status, keeping in mind that health status also depends on larger socio-economic determinants beyond the control of the project. The results chain was laid out as follows, and the logical links between them were clear: Planned health activities, including support to selected programs, were expected to generate enhanced involvement of communities in the promotion of their health, increased and more efficient financing of services, better delivery of public health services, improved health practices, a more efficient mix and utilization of inpatient and outpatient services, and improved management of resources. Relevance of design under the revised objectives/AF1 is rated Substantial as the results chain linking planned health activities to outputs and intermediate outcomes to improved health status was maintained, but with a lack of articulation of activities and their connection with the Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) desired outcomes for improved nutritional status of the population, and for assisting poor families in managing and mitigating the impact of rising food prices. Relevance of design under the revised objectives/AF2 is rated Modest. The results chain and logical links for improving health sector performance were plausible, including through improved equitable access, enhanced affordability, improved sector stewardship and management, improved efficiency of service delivery, and the provision of sufficient and more reliable financing to the sector. But it was not clear how the development of a beneficiary database and the provision of technical assistance would lead to a strengthened targeting system in the delivery of social benefits in the absence of key measures to fulfill eligibility policies. The activities and the results chain for improving and sustaining the health status of vulnerable populations were not articulated. Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Improve the health status of the population. Rationale Outputs: The project increased sector financing and provided financial support to selected priority health programs. It supported a countrywide inventory of assets and health centers in rural areas to better plan the reduction of access barriers. The project developed equipment specifications for health centers, family group practices, general practice centers, territorial hospitals, and oblast-merged hospitals, and provided medical supplies and equipment. It supported the establishment of 1,700 village health committees covering 84 percent of all villages in the country. The village health committees collaborated with primary health care facilities in health promotion, delivering health information, and identifying high-risk individuals at the community level and referring them to health facilities. Health promotion focused on healthy life styles and behavior change, including smoking cessation and alcohol abuse. The project supported expanded maternal & child health services, the expansion of pharmacy networks in rural areas, free health services under the state guaranteed benefit package, the additional drug benefit package in line with the integrated management of childhood illnesses, services for tuberculosis services, HIV/AIDS control, and cardio-vascular disease control, and human resources for health to improve staff supply in remote regions. The project provided training in strategic planning, financial management, and procurement. It supported studies and program-wide monitoring capacity. Outcomes: The objective was substantially attained as evidenced by improvements in the health status of the population. However, as stated above, it is understood that health status improvements are also dependent on broader socio-economic determinants, and that project interventions were important contributing factors. Life expectancy for men increased from 64.3 years in 2004 to 70.5 in 2014, exceeding the target of 65 years; and life expectancy for women increased from 72.2 years in 2004 to 74.6 years in 2014, exceeding the target of 73 years. Between 2004 and 2012 the infant mortality rate declined from 25.7 to 20.2, exceeding the target of 22.3. Under-five child mortality declined from 31.8 in 2004 to 23 in 2012, exceeding the target of 27.2, but with regional disparities. Vaccination coverage for children was maintained at 96 percent, and skilled attendance at delivery was maintained at 99.2 percent. The TB mortality rate declined from 11.2 in 2004 to 6.7 in 2014, exceeding the target of 9. Other health status indicators showed progress, but without attaining their targets. Maternal mortality decreased from 54.1 in 2006-2008 to 46.6 in 2011-2013 with a target set at 40. TB incidence fell from 113.6 in 2004 to 98.8 in 2014, with a target set at 91. Cardiovascular disease mortality rates among 30-39 year olds and among 40-59 year olds declined slightly from 50 to 48.8 and from 307 to 295, but fell short of the targets of 31 and 203, respectively. Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) Substantial Revised Objective Not applicable. Revised Rationale Not applicable Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Objective 2 Objective Improve and protect the nutritional status of the population against the instability of national food prices. Rationale The ICR did not provide sufficient information to assess progress towards achievement of this objective. The TTL explained that the nutritional status of important segments of the vulnerable population was addressed by the ICR (TTL, 4/28/2016) as shown under objective 7 below on pregnant/lactating women, infants and young children. The protection of the nutritional status of the population at large, in the context of food insecurity, spans many population groups, including adolescents. Relevant data from two rounds of demographic and health surveys indicating improvements was provided by the team (comments received from the Global Practice, 6/14/2016). Therefore, achievement of this objective is rated Substantial. Rating Substantial Revised Objective Not applicable Revised Rationale Not applicable Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) Objective 3 Objective Assist poor families to manage and mitigate the impact of rising food prices. Rationale Outputs: The project increased the Unified Monthly Benefit by US$1 per month for each beneficiary under the "Monthly Benefit for Low-Income Families and Children Program" as an increment to the original benefit of US$3.50 per month, over a 10-month period from October 2008 to July 2009. The top-up payments were financed by US$4.8 million from the AF1 grant. Outcomes: The objective is considered partially achieved because a significant proportion of poor families was not reached by the system on which the assistance was built. The program reached less than one-third of the poorest quintile according to the Public Expenditure Review of 2014 (ICR, p. 27). Rating Modest Revised Objective Not applicable Revised Rationale Not applicable Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Objective 4 Objective Improve the performance of the health sector. Rationale Outputs: The outputs are described above, under Objective 1. Outcomes: Progress towards this objective was variable with mixed results. The project has generated overall reliable financing for the sector, but with persistent fund transfer delays resulting in funds being available only near the end of the year. The targets that were set for government expenditure on health were met. Public spending on health as a percentage of total government spending rose from a baseline of 10.3 percent in 2005 to 13.1 percent in 2014, achieving the target of 13 percent. The per capita health spending was estimated at US$19 in 2005 and has risen to US$76 in 2014. Equitable access to health services has substantially improved under the project. Outpatient care utilization in rural areas increased across all quintiles. At the end of the project, the urban-rural utilization gap disappeared (ICR, figure 3-2). The increase in hospitalization was greater for rural areas than urban areas (ICR, Annex 2b). However, rather than attaining the desired improvements in affordability of health services and financial protection, these have deteriorated. Mean out-of-pocket payments have grown very rapidly between 2006 and 2014 for all income quintiles, with the highest rates of increase for the lowest quintiles, among the poor. Mean out-of-pocket payments have more than tripled during this period (from 714 to 2,761 soms) in nominal terms, and almost doubled (from 553 to 985 soms) in real terms. The mean spending for prescribed medicines almost Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) tripled between 2006 and 2014, and increased more than seven-fold for non-prescribed drugs during the same period. At the outpatient level, two thirds of reported out-of-pocket payments were for consultations, 22 percent for laboratory services, and the remainder for informal payments and miscellaneous costs. Reported inpatient payments included 44 percent for drugs, 17 percent as co-payments, 28 percent for personnel, and 9 percent for medical supplies and laboratory tests. By comparison, out of pocket expenditures as a share of household budgets was on the decline prior to the project. Minimal progress was made in improving efficiency of services (ICR, p. 22). The average number of days a hospital bed was occupied in a year declined from 318 days in 2006 to 300 in 2014, a level lower than the baseline, while unnecessary hospital admissions remain an unresolved issue (ICR, p. 22, and Annex 7). Inefficiencies in the mix and organization of services persisted. There was no evidence that efforts to encourage greater use of primary care and family practitioners have led to more efficient use of services. On the demand side, there was a persistent preference by the population to bypass primary care physicians and primary health care centers in favor of specialist and hospital care. On the supply side, enforcement of the proper use of the referral system was weak, with a lack of clarity of the complementarity and links across different levels of the referral systems (ICR, p. 22). Prevention, promotion, and early detection and treatment, while cost-effective, were neither emphasized at the primary health care level, nor actively sought by populations who could most benefit from them. HIV/AIDS program coverage did not attain its targets and little progress was made in managing cardio-vascular disease. There was a disincentive for primary health care providers to register high blood pressure patients because failure to follow up with the patients resulted in sanctions (ICR, p. 19). There was no compelling evidence of improved stewardship and management. Central MOH was overstretched with a consequent lack of support and oversight to the regions. There were concerns about MOH's ability to undertake strategic management (evidence-based policy analysis, planning, programming, budgeting, monitoring and evaluation) and fiduciary management. The undersized MOH at the oblast level rendered it incapable of strategic management and oversight (ICR, p. 33). Quality issues persisted at various levels, including human resources, physical infrastructure, and service delivery. There was no routine supervision of health facilities, and there were no functioning quality assurance mechanisms. The knowledge of health providers in the proper management of various medical conditions was variable, as was the adherence to clinical practice guidelines. Achievement of this objective is therefore rated Modest. Rating Modest Revised Objective Not applicable. Revised Rationale Not applicable. Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Objective 5 Objective Strengthen the targeting system in the delivery of social benefits. Rationale Outputs: The project supported the establishment of the social assistance beneficiary database under the Corporate Information System of Social Assistance through the provision of equipment and software, technical assistance, and training of 682 specialists from the ministry of social development and territorial departments. The project allowed data collection from 59 municipal and rayon departments, 28 medical-social expert commissions, 15 social residential facilities, and rehabilitation centers. The project built the database to enter applications and to Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) process individual cases for claimants. The database was also used to generate beneficiary financing requests by regional social protection departments. The project provided technical assistance and training, including study tours, to improve the ministry’s capacity in targeting. The outputs included the development of a social protection strategy for 2012-2014, a social protection program for 2015-2017, and analytical inputs on benefit targeting options and activation and graduation of beneficiaries to be piloted under the SWAp2 operation. Outcomes: While important building blocks were developed by the project, the objective was not achieved. There was little progress in improving targeting in the delivery of social benefits, and in consolidating existing social benefits so as to reduce leakages to the non-poor and to reduce duplications. The project has not been able to affect the overall situation in the sector (ICR, p. 27). The ICR states that progress would likely require more time and political will. Co-payment exemptions for various categories obtaining free medical care under the State Guaranteed Benefit Package were continued regardless of poverty status (TTL, 4/28/2016). The list of exempted categories included 28 social categories and 16 medical conditions. Furthermore, copayment exemptions were expanded to pregnant women and children under five. The policy of co-payment exemptions which was maintained did not favor the poor since 57 percent of beneficiaries were not in the bottom two income quintiles, and 53 percent of the bottom two quintiles were not exempted from co-payment (ICR, p. 23). As for the cash transfers under the Monthly Benefit for Low Income Families Program, there were no improvements in program targeting and coverage of the poorest (ICR, p. 27). Rating Modest Revised Objective Not applicable. Revised Rationale Not applicable. Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Objective 6 Objective Improve and sustain the health status of vulnerable populations affected by food price shocks. Rationale The ICR did not provide information about this objective, including health data on vulnerable populations affected by food price shocks. Infant and child mortality data (provided by the Region, 6/14/2016) is insufficient to explain and demonstrate achievement of the objective. Rating Modest Revised Objective Not applicable, no change. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) Revised Rationale Not applicable. Revised Rating Not Rated/Not Applicable PHEFFICACYTBL Objective 7 Objective Improve and sustain the nutritional status of vulnerable populations affected by food price shocks. Rationale Outputs: The project provided nutritional supplements and nutrition education to pregnant/lactating women and infants and young children. The distribution of nutrition supplements reached 83 percent of children countrywide. Related training was completed. All micronutrient and flour fortification packages financed under SWAp were distributed. Nutritional supplements and nutrition education were provided to 143,000 pregnant and lactating women and 500,000 children under five, including supplements for iron and folic acid, and premix for flour fortification and Vitamin A. Also, the project provided support to village health committees to undertake community-based nutrition campaigns. Outcomes: The percentage of pregnant women with anemia declined from 54 percent in 2008 to 35 percent in 2014. Declines were substantial in all regions. Between 2010 and 2014, the share of pregnant women who received folic acid increased by 16 percent, from 57.8 percent to 66.8 percent, short of the target of 80 percent. Most regions registered improved coverage. The proportion of pregnant women who received potassium iodide also increased between 2010 and 2014 from 67.8 percent to 83.6 percent. All regions and Bishkek and Osh cities registered increases except Osh, which remained at 70 percent. In 2009, 7.3 percent of children 6-24 months old had Vitamin A deficiency; a UNICEF assessment showed that, by 2014, there was a 27 percent reduction in anemia rates among children 6-24 months old. Rating Substantial Revised Objective Not applicable. Revised Rationale Not applicable. Revised Rating Not Rated/Not Applicable 5. Efficiency At appraisal, the economic analysis centered on the underlying rationale of the health reforms and investments supported under the SWAp (ICR, pp. 21-22), namely improving budget sustainability by mitigating allocative and technical inefficiencies, improving targeting of resources for Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) priority health programs, and reducing fragmentation in donor support and budgetary distortions. No economic rate of return was undertaken for this operation, but many interventions that were supported by the project, including for primary health care, maternal and child health, nutrition, TB, and HIV, are recognized internationally as technically sound. The principle of having pooled financing to facilitate the financing of priority activities that were not adequately addressed previously has not been well achieved during the project period. While some programs, such as maternal & child health and HIV/AIDS, received adequate allocations, other programs did not, such as cardio-vascular disease (which had negligible funding although its disease burden remains among the highest in the country). The Working Group on cardio-vascular disease has put forward a proposal to align incentives of primary health care physicians to screen for risk factors, but the proposal started to have traction only near the end of the project. Exempted categories in the State Guaranteed Benefit Package such as veterans, pensioners, children, and pregnant women culminated in large leakages. Some services under the State Guaranteed Benefit Package did not meet the criteria of public goods as identified at the beginning of the project. The delay in Bishkek joining the national pool resulted in a skewed allocation of the budget towards the capital. The program's work plans were not sufficiently prioritized, exceeding what the budget and pooled funds could afford. Both the timeframe of the SWAp and its financing doubled from an initial amount of US$652 million at appraisal to US$1,304 million in 2012-2014 (ICR, Annex 1, p. 41). On the other hand, recurrent costs for infrastructure, equipment, and vehicles were negligible and constituted 1.3 percent of the total government expenditures on health. The targeting of social benefits did not improve during the project period, and, by project closing, less than one-third of the poorest 20 percent were covered by the monthly benefit for low-income families with children program. The project did not effectively use the database that was developed to facilitate targeting accuracy and to promote cost-effectiveness of benefits. Also, the lack of effective utilization of the database delayed the ability of the ministry of social development to exploit it as a tool for problem analysis and planning. The project also had notable inefficiencies in its implementation. Transaction costs under the SWAp were high for both MOH and donors (ICR, p. 30). The increased transaction costs included donor coordination activities, including the preparation, execution, and follow-up of joint reviews, health summits, and joint planning activities. There were persistent delays in fund transfers to the ministry of health, resulting in fund bunching towards the end of the year, urge for rapid spending, and compressed implementation periods. There were delays in process tasks such as procurement. There were frequent administrative changes and staff turn-over, along with difficulties in retaining qualified staff. A third restructuring on 03/12/2014 extended the project closing date to 6/30/2015 to complete the implementation of project activities resulting in a 10- year implementation period. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the original objectives, which, stated briefly, were to improve health status of the population, relevance of objectives is rated Substantial. The objectives were responsive to country conditions, Bank strategy, government strategy, and the financing partners’ joint plans under the SWAp. Relevance of Design is rated Substantial as the planned activities 'contribute' to health status improvements. The objective to improve the health status of the population has been substantially achieved. Efficiency is rated Modest. The review findings are indicative of moderate shortcomings under the original objectives, and therefore an outcome rating of Moderately Satisfactory. Under the revised objectives/AF1, the project continued with the objective to improve the health status of the population, and added two additional objectives: to improve and protect the nutritional status of the population against the instability of national food prices, and to assist poor families to manage and mitigate the impact of food prices. Relevance of these objectives is rated Substantial as both health and nutrition Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) were country priorities. Relevance of design is rated Substantial as the results chain linking planned health activities to outputs and intermediate outcomes to improved health status was maintained, but with a weak articulation of activities and their connection with the desired outcomes for an improved nutritional status of the population, and for assisting poor families in managing and mitigating the impact of rising food prices. The objectives to improve the health status of the population and to improve and protect the nutritional status of the population were substantially achieved, but the objective to assist poor families to manage and mitigate the impact of rising food prices is rated Modest. Efficiency is rated Modest. The review findings are indicative of moderate shortcomings, and therefore an outcome rating of Moderately Satisfactory. Under the revised objectives/AF2, reformulation of the PDO meant that objectives now could be summarized as (1) improving the performance of the health sector, (2) strengthening the targeting system in the delivery of social benefits, (3) improving and sustaining the health status of vulnerable populations, and (4) improving and sustaining the nutritional status of vulnerable populations. For these, relevance of objectives is rated Substantial, and relevance of design is rated Modest. While the results chain for strengthening the performance of the health sector was plausible in general, it was not clear how a strengthened targeting system in the delivery of social benefits would be achieved in the absence of key measures to fulfill eligibility policies. The activities for improving and sustaining the health status of vulnerable populations were not articulated. Achievement of three objectives -- strengthening the targeting system in the delivery of social benefits, improving health sector performance, and improving and sustaining the health status of vulnerable populations affected by food price shocks -- is rated Modest. Achievement of the objective to improve and sustain the nutritional status of vulnerable populations is rated Substantial. Project efficiency is rated Modest as both the financing needs and timeframe were twice the initial estimates, and there were implementation inefficiencies. The review findings are indicative of significant shortcomings, and therefore an outcome rating of Moderately Unsatisfactory. According to IEG/OPCS guidelines, when a project’s objectives are revised, the final outcome is determined by the weight of Bank disbursements under each set of objectives: • Under the original objectives, the outcome is rated Moderately Satisfactory (4) with a weight value of 0.64 (4x16% in disbursement). • Under the revised objectives/AF1, the outcome is rated Moderately Satisfactory (4) with a weight value of 1.04 (4x26% disbursement). • Under the revised objectives/AF2, the outcome is rated Moderately Unsatisfactory (3) with a weight value of 1.74 (3x58% disbursement). All aggregate at 3.42 (rounded to 3) which is indicative of a Moderately Unsatisfactory rating. Taken together, these ratings indicate significant shortcomings in the project's preparation and implementation, and therefore an overall Outcome rating of Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating The risk to development outcomes achieved by the project is rated Substantial. The overall cost of the program exceeds available financing, including funding provided by parallel financiers. In particular, the funding gap of the State Guaranteed Benefit Package represents a challenge to financial sustainability. The 2006 expansion of co-payment exemptions to include all deliveries, children between one and five years old, and pensioners, and the 2011 salary increase for health workers have, together, stretched the limited resources available. There is considerable scope for reducing costs in the future, including a more affordable benefits package, better targeting of co-payment exemptions, and more judicious deployment of services. But there is political reluctance to support measures that may be poorly received by relevant constituencies, such as narrowing of targeted benefits and further consolidating the excess of hospital/higher-level services. Furthermore, the population itself may not be fully supportive of the overall program until it gains more confidence in the quality of care provided at the primary health care level and more appreciation of the reform program benefits. Financial viability has been bolstered in the short run with continued financial support by the Bank and other development partners under SWAp2. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) a. Quality-at-Entry The Bank’s performance in identifying the project, facilitating preparation, and appraising the operation was adequate overall, but with moderate shortcomings. Health strategies and interventions were grounded in sound technical work. Project preparation benefited from previous lessons learned during the original Manas Program, including rightsizing the health sector and pursuing a stronger delegation of management to healthcare providers. SWAp implementation arrangements envisaged the strengthening and use of country systems. Planned interventions were gender sensitive. Environmental aspects were well addressed. However, human resource capacity risks were not fully explored, and there was little recognition of the difficulties that may be encountered in attracting and retaining qualified staff since there is demand and better remuneration outside the civil service. The second smaller component on strengthening the administrative system of the Ministry of Labor and Social Protection with the view to facilitate the implementation of policy reforms was not well prepared. Social protection elements were not included in risk assessment and mitigation (ICR, p. 9). According to the ICR, the decision to include a social protection component in the SWAp was made largely for the sake of expediency, avoiding the administrative time and costs of preparing a standalone project (ICR, p. 10) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank Team provided regular supervision during the nine-year implementation period. The aides-memoire reveal that oversight meetings such as joint annual reviews, health summits, and thematic meetings were held regularly and were substantive in their content (ICR, p. 36). The annual reviews included the observations of joint financiers on issues constraining implementation. The ICR notes that, over time, both the health summits and joint annual reviews became increasingly evidence-based with a focus on the program objectives. The Bank took lead responsibility for supervising fiduciary aspects and engaging with the ministry of finance. Issues concerning inadequate program allocations and lack of prioritization were beyond the control of the Bank Team (Explanation provided by the Region, 6/14/2016). The Bank assumed the role of lead coordinating partner and the importance of a SWAp TTL selection was highlightedby the ICR. Support for the preparation of the follow-up operation was adequate. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The government showed ownership and commitment to health reforms, but much less to social protection strategies. Political instability and civil unrest, which were encountered during the nine-year implementation period, precipitated frequent changes in the government's leadership and frequent replacements of ministers. This culminated in attempts to change the path of reform and to abandon the SWAp (ICR, p. 37). This was also catalyzed by the parliament's reticence to embrace some of the more politically sensitive plans regarding eligibility, and which would require cutting back on benefits to the non-poor. There was reticense to embrace the rationalization of service delivery and access, which involves the consolidation of hospitals and specialized care in favor of enhanced primary care. There were persistent delays in budget formulations and approvals causing serious delays (ICR, p. 37) in fund transfers to the ministry of health, resulting in fund bunching and compressed implementation periods (as stated previously), inefficiencies, and inability of health facilities and professional associations to execute their full training commitments in so short a timeframe. The lack of driving and implementing eligibility policies was detrimental to the effective and efficient delivery of social benefits, and to their financial sustainability. Government Performance Rating Moderately Unsatisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) b. Implementing Agency Performance MOH was the lead implementing agency for health aspects. In general, its day-to-day performance was adequate. Key covenants were fulfilled. MOH embraced a participatory process involving a broad range of stakeholders, including NGO representation. However, there were shortcomings in the ministry's performance in process tasks, notably in procurement, and contract oversight and management. The Ministry of Social Development was the lead agency for social protection and performed adequately in developing the database for beneficiaries. It performed effectively in distributing the benefit supplement to vulnerable populations in response to the food crisis. Its low performance in improving targeting in the delivery of social benefits is attributable to other political levels beyond its control. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design M&E design built upon the design of the Manas Taalimi Program. Overall, the indicators reflected the objectives and measured improved health status and service performance. The indicators were measurable and the data collection methods were appropriate. Data sources included the Republican Medical Information Center for health and bio-statistical data, National Statistical Committee Household Surveys for financial protection and affordability data, special studies to assess reform impacts, and mid-term and end-of-program evaluations. These agencies were well embedded institutionally with strong stakeholder ownership. Under the SWAp arrangement, Joint Annual Reviews were undertaken to assess performance in implementing the previous year’s work plan and to inform the work plans of the following years. Shortcomings in the measures of program performance were in the areas of service quality and sector stewardship. The number of indicators was excessive; however, the TTL explained that the indicators were based on the program framework and joint annual reviews (TTL, 4/28/2016). b. M&E Implementation Collection and analysis of most data were undertaken as planned. The Republican Medical Information Center collected and vetted health and statistical data reported by health facilities. Three national household surveys were undertaken during the project period (March 2007, March 2010, and April 2015). The surveys measured health services affordability and financial protection across income quintiles. The Mandatory Health Insurance Fund collected and monitored health services performance data. Financial management reports were generated regularly. The project monitoring of priority programs strengthened program-specific M&E. The Joint Annual Reviews, which were routinely carried out throughout the life of the project, continue under the ongoing SWAp2. The expertise of the Health Policy Analysis Center complemented the capacity of MOH in data analysis. A mid-term review and end-of-program review of Manas Taalimi, conducted in 2008 and 2011, were rigorous and informed reform strategy and implementation. In addition, UNICEF was closely involved in assessing nutrition outcomes. c. M&E Utilization Information generated was factored into the joint annual reviews, and was effectively used in planning yearly work plans. The information was used for the mid-term and end-of-program reviews. The data informed policy dialogue between the government and development partners, and also informed the design of the new health sector program, Den Sooluk (2012-2016). While the generation and use of health information for decision-making was reasonably adequate, both the government and development partners recognized that improvements were still required, and therefore commissioned a Swiss-financed consultancy to assess M&E. This study concluded in November 2014 that there was a need for further improvements in the quality of the data, its triangulation, and its optimal use at the central and regional levels. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) M&E Quality Rating Substantial 11. Other Issues a. Safeguards The Environmental Assessment Safeguard (OP/BP/GP 4.01) was triggered under the project, which was environmental category B. The only civil works investment envisaged for pooled financing was the rehabilitation of the MOH administrative building in Bishkek. An environmental management plan was developed in consultation with a broad range of local stakeholders, including representation from the Ministry of Ecology and Emergency Situations, the Department of Sanitary Epidemiological Surveillance, and MOH's United Directorate of Construction Enterprises. The plan was publicly disclosed. Stakeholder meetings confirmed the adequacy of the legal and regulatory framework for activities that were envisaged in the environmental management plan. Provisions for environmental screening were included in the project operations manual. The environmental management plan included provisions to mitigate potential adverse environmental impacts in the sector, and not just those financed under the project. The plan outlined environmental standards for hospital waste management, drawing on work already in progress under the previous Bank-supported project and a Swiss-financed project. The “National Targeted Program on Hospital Acquired Infections and Health Care Waste Management” was previously approved by MOH in 2002 and was being piloted at the time of project preparation. It was scaled up and incorporated into the Manas Taalimi Program with Swiss support. Direct oversight of safeguard compliance was delegated to the Swiss development partner. The Bank's supervision reports consistently noted satisfactory adherence. A joint meeting with the Swiss and MOH staff responsible for hospital infection control and waste management corroborated the progress made in building MOH capacity in this area, and highlighted the importance of transitioning from parallel financing to sustainable budget allocations. The social protection component did not trigger any safeguards. b. Fiduciary Compliance Financial Management. Under the project, fiduciary responsibilities for both health and social protection were handled by MOH. During the preparation stage, a health sector fiduciary assessment was undertaken by the Bank and recommended a list of actions to address weaknesses prior to effectiveness, and these actions were complied with. A financial management supervision plan advised on monitoring compliance. Following a slow first-year start-up in the implementation of fiduciary arrangements, all implementation and dated covenants were complied with. MOH undertook annual sector reviews, prepared annual work plans and budget allocations. Reporting and accounting provisions were adequately executed. Financial Monitoring Reports were produced and submitted on a regular basis, and were consistent with underlying reports. Internal control (audit) units were established within MOH and the Mandatory Health Insurance Fund. Most of the external financial audits were submitted on time. Two audits for FY2007 and FY2011 were qualified, each citing internal control deficiencies. Once, for FY2013, the auditor issued a disclaimer of opinion. These were addressed through action plans prepared and executed by MOH and the insurance fund, and were overseen by the Bank. Internal reporting designated the project as a "problem project" when auditors had difficulty accessing books at the hospital-level in 2012-13. The issue was resolved after an action plan was prepared and executed. By project closing, all Bank and Trust Fund resources had been fully accounted for. There were no issues of corruption or misuse of funds. Some audit recommendations were addressed by project closure, and the rest were followed under the ongoing SWAp 2. KfW continued to provide capacity building support for financial management. Procurement. A procurement plan for 2006, the first year of the Manas Taalimi Program, was finalized at negotiations, and procurement procedures were spelled out in the operations manual. A capacity building plan was developed to strengthen sector systems. Annual independent procurement audits were also envisaged to carry out post reviews, which were published on the MOH website and were made available to the development partners. Prior review for a majority of procurement contracts was continued until sufficient capacity and performance was demonstrated. Both the Bank and joint financiers maintained a combination of prior reviews and ex post procurement audits to ensure adherence to procedures. There were procurement delays, which improved through a learning-by-doing process and with support from the Bank's procurement team. A procurement consultant was recruited to provide further support. The Bank and other partners provided technical expertise to ensure proper technical specifications in bidding documents. There were weaknesses in contract oversight and management, executing payments of multi-year contracts, and in monitoring warranties. Overall, Bank procurement guidelines were followed, with exceptions noted below. At the project start-up, MOH proceeded with Single Source Selection and contractual arrangements in the absence of the Bank’s no-objection. A KfW consultant was assigned to review all contracts to verify execution against the 60 percent advance that was provided. Also, the Bank alerted the Health Insurance Fund that undertaking centralized procurement of drugs from its revenues and according to public procurement practices was in violation of the SWAp terms of agreement under which all procurement for Manas Taalimi would be the responsibility of MOH, and to be carried out according to Bank procedures. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment The overall outcome rating is based on a split evaluation under the original and revised objectives as detailed in Section 6. There were improvements in the health and nutrition status, but minimal progress was made towards attaining social protection objectives. There Outcome Moderately Satisfactory Moderately Unsatisfactory were mixed results in improving health sector performance. Out of seven objectives, three were substantially achieved, and four were rated Modest. Efficiency was modest as both financing needs and implementation timeframe doubled, along with implementation inefficiencies. Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Commitment and ownership fluctuated, with attempts to change the path of reform and to abandon the SWAp. There was reluctance to drive and implement eligibility criteria for social benefits and pro-poor targeting to the detriment of effectiveness and efficiency in Borrower Performance Moderately Satisfactory Moderately Unsatisfactory the delivery of social benefits. There was reticense to embrace the rationalization of health service delivery. Delays in budget formulations, approvals, and fund transfers resulted in compressed implementation periods and difficulties in fulfilling implementation commitments. Quality of ICR Substantial --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HEALTH & SOC PROT(P084977) Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Selected lessons drawn from the ICR (pages 38-39), adapted by IEG: • In-depth technical analysis and evidence-based dialogue have the potential to promote political understanding and ownership of reforms. • Even with the benefits of training and technical assistance, the development of human resource capacity and its retention are not likely to be fully achieved or sustained in the absence of other determinants, including a clear definition of responsibilities, proper remuneration, selective and strategic contracting out, and a stronger local role. • The Bank's choice of a Task Team Leader is critical for a SWAp operation. A Bank TTL who leads a SWAp can bring out and fully utilize the comparative advantages of development partners. • A reliable and timely flow of funds matters not just for ensuring high expenditure levels, but also for ensuring the effectiveness of expenditures. Avoiding late bunching at the end of the year facilitates efficient spending. 14. Assessment Recommended? Yes Please explain Differences in views on ratings. 15. Comments on Quality of ICR The ICR is results-oriented and highly analytical. The report provides a candid account of the project's performance and identifies useful lessons derived from project experience. The ICR's thorough analysis of financial sustainability issues is noteworthy. However, the ICR should have given more consideration of the objectives as delineated in the formal Development Grant Agreement and subsequent Financing Agreements. Also, the main text of the ICR, reaching 40 pages, is unnecessarily lengthy and could have been more concise. a. Quality of ICR Rating Substantial