Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) Report Number : ICRR0020376 1. Project Data Project ID Project Name P101206 BO-Exp. Access to Reduc Hlth Ineq (APL3) Country Practice Area(Lead) Bolivia Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-43820 31-Jan-2014 16,462,005.00 Bank Approval Date Closing Date (Actual) 24-Jan-2008 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 18,500,000.00 0.00 Revised Commitment 11,174,075.00 0.00 Actual 10,285,040.89 0.00 Prepared by Reviewed by ICR Review Coordinator Group Maria De Las Mercedes Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Vellez 2. Project Objectives and Components a. Objectives This project was the third and final operation of a series of Adaptable Program Loans (APL) that pursued the reduction of infant, child, and maternal mortality rates in Bolivia. This third phase builds on the three main achievements of Phases I and II: a) a focus on performance agreements and results, which increases accountability across the sector; b) the expansion of the health insurance system to reach more of the poor; and c) the implementation of a National Program for the Expansion of Coverage (EXTENSA), which provides health care to Bolivians in geographically remote areas. Page 1 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) The specific objectives of this APL III, as stated in the Loan Agreement and the Project Appraisal Document (PAD), were to: (i) reduce occurrence of critical risk factors affecting maternal and infant health in the Target Areas so that current gaps between regions are reduced; (ii) reduce chronic malnutrition among children under 2 years of age in the Target Areas; (iii) increase health insurance coverage in the Target Areas; and (iv) upgrade the National Health Information System (Sistema Nacional de Informacion en Salud - SNIS) so that it will be integrated with Bolivia’s new health insurance program. This ICR review undertakes a split evaluation in three phases: (i) approval to December 2012, (ii) December 2012 to July 2014, and (iii) July 2014 to closing, since the results framework has been revised in December 2012 and July 2014. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The project combined interventions at the national level with others more focused on 88 most vulnerable areas of Bolivia (82 municipalities and 6 peri-urban areas). The government selected target areas based on risk of food insecurity, number of institutional births, access to drinking water, literacy levels, connection to sewer systems, school attendance rates, and availability of electric power (PAD p. 42). The project had four components as stated in the PAD. Activities that were cancelled during the December 2012 restructuring are marked below as "eliminated." Component 1. Stewardship Role of Health Authorities – Essential Functions in Public Health (IDA US$4 million at appraisal; US$1.4 million actual): This component aimed to strengthen capacity of national, regional, and local health authorities to perform Essential Functions in Public Health (EFPH) – which range from surveillance and disease control to social participation, regulation, and M&E. It was divided into 3 sub-components: 1.1) Strengthening the National Health Information System (SNIS): this sub-component was to finance equipment, technical assistance, software tools, training, and communication services to support a structural change in the way data is captured: 1.1.1) Design and adaptation of technical standards and processes of the SNIS in epidemiological surveillance; 1.1.2) Monitoring and evaluation, related to oversight of the Government’s Sector Development Plan for Page 2 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) 2006-2010 at the national level (eliminated); 1.1.3) Development of a National System for Health Research (eliminated); 1.1.4) Financing for the Demographic and Health National Survey 2011 (eliminated). 1.2) Regulation, results-based management (RBM), and culture of accountability: this sub-component was to support strengthening the capacity of the Ministry of Health and Sports (MSD) and departmental and local health authorities: 1.2.1) Provision of technical assistance to develop standard practices and management tools; 1.2.2) Strengthening the coordination of international and multilateral donors (eliminated); 1.2.3) Carrying out of project impact evaluation (eliminated); 1.2.4) Dissemination of health results achieved through community meetings, workshops and publications; 1.2.5) Strengthening of the MSD Human Resources Policy. 1.3) Development and implementation of a National Program of Quality: this sub-component was to support licensing, certification, and monitoring of health facilities, including hospitals, clinical labs, blood banks, etc.; incorporation of quality managers into the MSD and referral networks; training and diffusion in the use of quality standards; and preparation and validation of standards, processes, and methodologies. Component 2. Family, Community and Intercultural Health (IDA US$9.9 million at appraisal; US$8.2 million actual): This component aimed to improve access to maternal and infant health services in the project’s target areas. Activities were to support the development of Intercultural Maternal and Infant Health Referral Networks, complementing the existing EXTENSA health program. These networks were to promote demand for maternal and infant health care by focusing on the following three goals: 1) increasing the number of safe institutional childbirths; 2) increasing the number of referrals of obstetric emergencies directly from the community; and 3) providing access to a referral system for children with acute respiratory and digestive diseases. It was divided into two sub-components: 2.1) Development and Strengthening of Intercultural Maternal and Infant Health Referral Network, including support for: 2.1.1) Analysis of current referral networks (diagnosis of the status of resources and existing capacities of health networks in target areas, expected to reduce 52 unconnected networks into a smaller number of more structured networks with increased capacity); 2.1.2) Renovation of health facilities, public housing and purchase of medical equipment (renovation of 300 facilities, purchase of new primary care equipment, construction and rehabilitation of housing for health workers, conversion of seven primary care centers into secondary care hospitals); 2.1.3) Financing expanded human resources (creation of new brigades in rural networks; improving medical residences; strengthening formal training in pediatrics, internal medicine, ob/gyn, and surgery; incorporation of nutritionists into networks; support for rotating specialized personnel to work temporarily in rural areas) (eliminated); 2.1.4) Professional training (program that establishes a network training day to link referring doctors and patients with secondary care doctors); operational research (support for research on outbreaks) (eliminated); 2.1.5) Regional Equipment Maintenance Centers (support for the creation of three regional centers for equipment maintenance) (eliminated). 2.2) Strengthening of Local Management and Community Participation, including support for: 2.2.1) Participatory planning (workshops focusing on local epidemiological profiles and disease Page 3 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) surveillance, with emphasis on maternal and child health and nutrition) (eliminated); 2.2.2) Support for Health Information Analysis Committees (training for health workers and community leaders in methodology to gather and analyze data) (eliminated); 2.2.3) Public awareness (dissemination of community inter-sectoral health strategies and results) (eliminated); 2.2.4) Development of health promotion materials with community participation (eliminated); 2.2.5) Training and support for local/municipal health boards (DILOS) to increase capacity to manage local health issues (eliminated). Activities under component 2 were designed in coordination with a Social Protection Program (providing conditional cash transfers [CCTs] to families with infants for health and nutritional care) and the Zero Malnutrition Program (which encourages consumption of fortified complementary food to meet nutritional needs of children and pregnant women, as well as expanded access to clean drinking water and improved sanitation). Both programs aimed at stimulating demand for health care services in the same areas. Component 3. Health Insurance Program (IDA US$3.2 million at appraisal; US$0.56 million actual): This component aimed at supporting the implementation of a new health insurance system (SUS) through three sub-components: 3.1) Strengthening the SUS enrollment system: 3.1.1) Training on the use of the enrollment system (eliminated); 3.1.2) Implementation of a communication and social marketing strategy for health insurance enrollment (eliminated); 3.1.3) Printing and distribution of enrollment forms nationwide (eliminated). 3.2) Strengthening capacity to plan, manage and carry out monitoring and evaluation of the Single Health System (SUS) at the national, departmental, and municipal levels: 3.2.1) Incorporation of statistician, public health specialist and economist positions into the Departmental Units for Technical and Financial Management (UGTFDs); 3.2.2) Provision of office and computer equipment to the UGTFDs; 3.2.3) Provision of technical assistance for auditing and management of incentives, and financing of supervisory visits of the National Unit for Technical and Financial Management (UGTFN) to UGTFDs and networks. 3.3) Development of the SUS monitoring and evaluation system. Component 4. Project Administration (IDA US$0.9 million at appraisal; US$0.6 million actual): The fourth component was to support project administration with equipment, technical assistance, training, and operating costs to finance the administration of the project, as well as financial and procurement audits. Revised components During the 2012 restructuring, project components were revised and many activities were cancelled. Component 1: Most of the activities under sub-component 1 and 2 designed to strengthen the SNIS, the MSD, and local health authorities were eliminated (see above). New activities were added: development of an index to monitor the progress of mothers and children during the first two years of life; and support to the Cumbre de Salud (a bottom-up, participatory consultation process in the health sector). Page 4 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) Component 2: Activities for expansion and training of human resources, development of regional maintenance centers, and strengthening of local management and community participation (original sub- component 2.2) were dropped (see above). Component 3: Activities related to the unified health insurance scheme were dropped, and instead the project contributed to increased population registration in the Family Health Record (Carpeta Familiar) and trained key personnel and community leaders on the use of this enrollment system. In July 2014, a second restructuring added a new activity for renovation (construction and equipment) of the Onco-Hematologic Unit of La Paz Children’s Hospital. Its purpose was to increase child survival by addressing the needs of children with cancer (especially leukemia) and hematological health problems. Overall, the project restructurings eliminated most of the activities originally defined under Component 3 and assigned a greater proportion of funds to support investments in health facility infrastructure. As a result of the restructurings, the proportion of project funds allocated to infrastructure investments increased from 24.8% (US$4.6 million) to 62.7% (US$6.8 million). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost, Financing, Borrower Contribution: The original total project cost was estimated at US$26.2 million, from which US$18.5 million was to be financed through an IDA Credit, and US$7.7 million through the borrower’s contribution. After three years of project life, about 44% of the total credit was cancelled, and the total IDA disbursement was US$10.28 million. Dates: The APL III was restructured four times over the course of its implementation. The PDOs remained unchanged. 1) In December 2012, the project underwent a full level 2 restructuring that: (a) reduced the scope of project components; (b) eliminated two key performance indicators (KPIs); (c) reduced the credit amount; and (d) reallocated funds among categories consistent with the reduced scope of the project and reduced credit amount. The total amount disbursed at this point was US$2.81 million. 2) In January 2014, the project's closing date was extended for six months to January 31, 2014. 3) In July 2014, another level 2 restructuring included: (i) a revision and update of the Results Framework, changing some targets and dropping some indicators; (ii) a reallocation of funds among categories; (iii) a change in disbursement estimates; (iv) an extension of the closing date by 15 months from July 31, 2014 to October 31, 2015; and (v) the inclusion of a new activity consisting of the renovation of the Onco- Hematologic Unit of La Paz Children’s Hospital. The total amount disbursed at this point was US$6.42 million. 4) In August 2015, the project’s closing date was extended by two more months to allow for the Page 5 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) completion of civil works at the Onco-Hematologic Unit of La Paz’s Children Hospital. 3. Relevance of Objectives & Design a. Relevance of Objectives The project objectives had high relevance to the country context, government strategy, and Bank strategy during the entire project life. Despite recent progress in human development indicators, Bolivia remains at the bottom of regional rankings on maternal and child health outcomes. Thus maternal and child mortality, malnutrition, and access to appropriate health care services were and still remain key development challenges. Project objectives 1 and 2 were directly aligned with long-term goals of the APL series, and objectives 3 and 4 were indirectly linked: expanding health insurance coverage is expected to result in higher access to affordable health services, which in turn would impact the health status of the target population. Objective 4, upgrading the National Health Information System, aimed at facilitating the implementation of the expanded health insurance system, and was less related to access to health services and health status. Overall, the project objectives were grounded in the government’s commitment to improve the health of the Bolivian population and aimed at consolidating achievements of the previous two phases of the APL program. Moreover, they were aligned with the goals of the parallel Zero Malnutrition and Social Protection programs. The project objectives were consistent with the Bank’s 2007 Health, Nutrition, and Population strategy, which has a focus on improving health outcomes particularly for the poor and vulnerable, and with the Country Partnership Strategy (CPS) for the fiscal period 2012-2015 by supporting the human development and access to basic services results areas. Rating Revised Rating High High b. Relevance of Design The project's design and proposed activities were logically linked to the expected outcomes. The original project design was ambitious and planned to increase the availability and quality of health care services by investing in health facilities, quality assurance, and human resource capacity. It also proposed health promotion activities through outreach to local populations, community participation, and public awareness campaigns. Both quality improvements and demand-side interventions were expected to increase utilization of maternal and child health care services and thus reduce risk factors affecting maternal and infant health and nutrition. The project coordinated with other two programs that were contributing to achievement of the first two objectives: the national Zero Malnutrition program provided fortified complementary foods for young Page 6 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) children and pregnant and lactating women (also in project areas), and the IDA Social Protection project aimed at supporting the same families with conditional cash transfers for using basic health services like ante- and post-natal checkups. The project’s scope was reduced at the 2012 restructuring both in terms of target areas (beneficiaries) and number of activities. The elimination of activities (particularly in the areas of community participation and public campaigns, support of the expansion of health insurance, and expansion and training of human resources) weakened the causal relationship between project activities and objectives. The creation of new brigades in rural networks was also cancelled, even though these brigades were the indigenous peoples' preferred way to receive health care, according to the social assessment at appraisal (PAD Annex 10), because of the cultural closeness with the brigades’ health workers. At the 2014 restructuring, the project included an additional activity to support the construction and rehabilitation of the Hematology-Oncology Center of the Children's Hospital in La Paz. This activity was explicitly requested by the President Evo Morales during the Mid-Term Review in August 2013. The task team clarified that this activity was aligned with the project objective of reducing the occurrence of critical risk factors affecting infant health (despite the ICR acknowledged that the project’s causal chain had weakened). The lack of specialized facilities to do bone marrow transplants prevented children with leukemia, particularly from poor socioeconomic backgrounds, to be diagnosed early and therefore to receive effective treatment. According to data provided by the task team (Annex 5- complementary information about new activities), between 2010 and 2010, leukemia moved from the 30th to the 25th cause of mortality in Bolivia, and most of leukemia deaths were associated with children (especially poor children), The relevance of design for Phase III is rated Substantial. Relevance of Design Rating approval - December 2012: Substantial Relevance of Design Rating December 2012 - July 2014: Modest Relevance of Design Rating July 2014 - closing: Substantial Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Reduce occurrence of critical risk factors affecting maternal and infant health in the targeted areas so that current gaps between regions are reduced. Rationale Page 7 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) Outputs: Quality assurance processes: • 308 facilities (80%) in target areas initiated a self-evaluation process based on norms. • 16 facilities were accredited, and 34 facilities were in the process of accreditation at project closing. • 240 action plans were developed. • 135 health workers were trained in quality management methods. Investments in infrastructure and equipment of health facilities: • 16 rural health facilities in La Paz were equipped. • 8 health facilities in the Department of Beni were equipped. • 74 equipment and infrastructure projects were defined, and 16 equipment and infrastructure projects were approved by department health services. • 49 health facilities were fully equipped and functioning by project closing. Diagnostics of the internal capacity of networks: • A unified system for the registration of health facilities was developed. • 24 health facilities located in 8 municipalities across the Departments of Beni and La Paz were identified and equipped. • A georeferenced database of health facilities and networks was created at the national level. All intermediate outcome indicator targets related to access and quality dimensions were achieved: • 94% of health units applied sector regulation norms and the National Program of Quality of Health (PRONACS) norms, exceeding the original and revised targets of 80% and 90%, respectively. • 98% of health units in the target areas followed norms for primary and secondary health care facilities, exceeding the 2014 target of 20%. • 100% of civil works were finalized in the health facilities in the areas of intervention, achieving the 2014 target. • 100% of equipment was installed in health facilities in the areas of intervention, achieving the 2014 target. • 77% of the user population receiving services was satisfied with the quality of services in the project areas, exceeding the original target of 30%. Outcomes: The gap in access to essential maternal health care services between project target areas and the rest of the country was reduced: • The ratio between the percentage of pregnant women receiving four pre-natal care check-ups in the target areas and the rest of the country reached 0.89 at the end of the project life, exceeding the original target of 0.85. • The ratio between the percentage of institutional deliveries in the target areas and the rest of the country Page 8 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) also exceeded the original target of 0.85, reaching almost parity (0.98) in 2015. However, the indicators’ ratio structure does not provide information on absolute health service coverage. Access inequalities between target areas and the rest of the country could be reduced without access being improved in target areas (i.e. a decrease in the denominator would also increase the ratio). To complement these data, Annex 10 of the ICR shows that institutional delivery rates increased faster in target municipalities (from 45% to 55%) than nationwide (from 65% to 71%) between 2006 and 2012. Increased service utilization per se may not result in a reduction of risk factors. It is the good quality of prenatal care and institutional delivery services that would enable early detection of risk factors like high blood pressure, diabetes, and low birth-weight, as well as development of strategies for preventing pregnancy-related complications. That said, the project supported improvements in the quality of services that may have contributed to increase utilization and reduce the risk factors. Seventy-seven percent of the patients receiving services were satisfied with the quality of services provided in the project areas. Rating Objective 1 original (phase I and II): Substantial Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective This objective did not change during the project life, but target values were revised upwards. Revised Rationale Outputs: The restructured project supported the strengthening of the Children's Hospital. According to the ICR, at the time of closing the onco-hematological unit was 50% constructed, and equipment items have been purchased or solicited. During the interview with IEG, the task team confirmed that the Oncologic Unit of the Children Hospital was functioning. Outcomes: In July 2014, targets were revised (and made more ambitious) to account for higher baseline values, reflecting the project’s effective implementation date in 2009 and the reduction of project target areas (scope). Page 9 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) • The ratio between the percentage of pregnant women accessing four pre-natal care check-ups in the target areas and the rest of the country did not achieve the revised target of 0.95. In fact, the gap increased by three percentage points from the 0.91 revised baseline in 2009. • The ratio between the percentage of institutional deliveries in the target areas and the rest of the country, by contrast, increased from 0.72 in 2009 to 0.98 at project closing, exceeding the revised target of 0.89. This objective is rated as modest for phase III because achievements slightly worsened with respect to the previous phase, and, the still the ratio structure of the KPIs does not inform about the percentage of women receiving maternal services in project areas. The ICR does not provide additional information on institutional delivery rates after 2012. Revised Rating Modest PHEFFICACYTBL Objective 2 Objective Reduce chronic malnutrition among children under 2 years of age in the Target Areas. Rationale Outputs: In addition to renovation of health facilities and equipment investments mentioned under objective 1, the project implemented a system for community nutrition surveillance by forming committees for community surveillance in intervention target areas. Outcomes: At the end of the project, there was no outcome-level KPI to assess the reduction of chronic malnutrition. The original KPIs (measuring the percentage of children receiving exclusive breast feeding at 6 months in the target areas and percentage of children 2 years old taller than -2Z scores in the target areas) were dropped in the 2014 restructuring. The restructuring paper stated that the National Health Information System did not collect the basic information to measure these indicators (restructuring paper Report No: RES15248 p.4). According to the ICR (p. 15), these outcome indicators were dropped to avoid duplication, as they were being collected for the “Bono Juana Azurduy” (i.e. a government program promoting the use of maternal and child health services). The original KPIs were replaced by an intermediate outcome indicator: • The percentage of 2-year-old children covered with height growth control (CRED) in the intervention areas of the project increased from 35% in 2009 to 89% in 2015, exceeding the target of 40%. The ICR (p. 19) suggested that the 40% target defined in 2014 was underestimated given that Page 10 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) progress along this indicator was well under way and that the baseline was 35% in 2009. The ICR (p. 15) also acknowledged that the relationship of this indicator to the goal of reducing malnutrition is indirect, and therefore the indicator does not provide strong evidence on the achievement of objective 2. Moreover, given the presence of two parallel programs in the same project areas (“Bono Juana Azurduy” Project and the Program for Zero Malnutrition), the attribution of outcomes to the project’s intervention is less plausible. The ICR acknowledges the contribution of external factors to the achievement of project targets. The additional data presented in the ICR's Annex 10 show that at national level the percentage of children under 2 years old with weight-for-age less than 2 standard deviations decreased from about 5.5% in 2007 to 4.3% in 2012, but there is no evidence that malnutrition has in fact decreased in project target areas. Since malnutrition indicators did not update targets over time, but were replaced by one intermediate indicator at the 2014 restructuring, the assessment of this objective 2 does not apply split rating. Rating Objective 2 original (entire project): Modest Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Increase health insurance coverage in the targeted areas Rationale Outputs: The project carried on the following activities during Phase I aimed at supporting the implementation of a new universal health insurance (SUS) program, which was expected to result in higher health insurance coverage rates: • Agreement letters between the municipal governments and the MSD to allocate 15% of funds to finance SUS implementation were finalized; • Development and implementation of software for the administrative and financial management of public health insurance schemes at the municipal level (Health Information and Financial Control System – SICOF); • Training on the financial administration of public health insurance systems: 337 municipalities received training on Maternal and Infant Health Insurance (SUMI); 104 municipalities received training on insurance for elderly populations (SSPAM); • Training of 674 municipal health staff (two per municipality) on administrative and financial management of the public health insurance systems; • Purchase and distribution of 25 sets of information technology equipment for health units in 9 health Page 11 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) regions (SEDES). After the 2012 restructuring, the project supported: • A social public consultation (Cumbre de Salud) about the SUS, which implementation was delayed because of a disagreement between the executive and legislative branches regarding its institutional framework. • The development of a Family Health Records (Carpeta Familiar), which was implemented in nine Departmental Health Services. Carpeta Familiar is an instrument that registers the most relevant health events of patients and allows for the identification of population subgroups according to age and health risks/conditions (e.g. children under 5 years old, people with tuberculosis, pregnant women, etc.) within a community to guarantee access to services under the future SUS. It uses a system of colored markers to map population profiles into a community map/sketch (http://snis.minsalud.gob.bo/52-boletin-carpeta- familiar). According to the task team, the project supported the development of forms and instruments of the Carpeta Familiar, and the training of health workers to map the families in the health centers’ catchment area. The task team clarified that the Law 475 to establish the SUS was indeed passed in December 2013 (this is a correction from the ICR). According to the text of the Law 475 published in Official Bulletin on December 30 2013, financial protection and access to health care services will be provided free of charge to the eligible population (i.e. pregnant women, children under 5 years old, people older than 60 years old, women in reproductive age, and people with disabilities not covered by Social Security Insurance (Seguro Social Obligatorio de Corto Plazo)). At the same time the law eliminated the SUMI (Maternal-Child Universal Insurance) created in 2002 and the SSPAM (Senior Adult Health Insurance) created in 2006. https://www.minsalud.gob.bo/images/Documentacion/normativa/L475.pdf That said, there is no evidence that the SUS was fully implemented by the end of the project in 2016. Outcome: The original KPI (i.e. percentage of population enrolled in health insurance in target areas) was dropped at the 2012 restructuring, leaving the project with no indicator to measure progress until a new KPI was introduced in the 2014 restructuring. The ICR provides additional information on health insurance coverage across departments (p.57). According to national household survey data, the percentage of the population covered by any health insurance increased throughout all departments in Bolivia in the period 2009-2012. However, the department of Tarija was the only one where coverage achieved the 80% target set in the original KPI. Among the rest of departments, health insurance coverage reached a maximum of 67% in 2012. In particular, the departments of Chuquisaca and Potosi, where the project targeted 24% and 28% of their municipalities, respectively, reached 43% and 34% health insurance coverage in 2012, far below the target of 80%. New evidence provided by the task team in support of objective 3 during Phase I, which the ICR did not Page 12 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) specifically focus on, shows that: (i) the share of mothers and children enrolled in the SUMI increased from 56% to 92% between 2008 and 2012, and (ii) the number of services provided to the beneficiaries by the SUMI was expanded from 9.7 million to 19.2 million, in the same period, according the Ministry of Health administrative registers. The available evidence is not sufficient to ascertain that health insurance coverage reached 80% in the most vulnerable municipalities of Bolivia targeted by the project. No further evidence applies to the Phase II (2012- 2014). Rating Objective 3 original (Phase I and II): Modest Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 3 Revision 1 Revised Objective This objective did not change during the project's lifetime. A split rating is justified for Objective 3 because the project 2014 restructuring lower the bar both in terms of the indicator's nature and the specified target. Revised Rationale Outputs: The project helped the Government with the passage of the health insurance law 475 and helped put in place the necessary mechanisms for future implementation of the law. All previously mentioned outputs apply to Phase III as well. Outcomes: The 2014 restructuring introduced a new KPI, which target was revised downward compared to the original health insurance coverage indicator. • By the end of the project, 80% of target population was registered in Family Health Records (Carpeta Familiar), exceeding the 2014 target of 60%. But the new KPI does not reflect insurance coverage, but rather a registration system to inform the health insurance system once implemented (as acknowledged by the ICR in p.16). As explained above, the registration system of the Carpeta Familiar allows for the identification of the beneficiaries of the Law 475. The new KPI is an imperfect proxy of health insurance coverage because Law beneficiaries belong to specific population subgroups. So having 80% of population registered in Carpeta Familiar, is not equivalent as to have 80% of population covered by the health insurance. The task team stressed that as result of the Law 475 and based on work carried out to register families into the health insurance registry, the number of monthly health services provided to the insured population Page 13 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) increased from a monthly average of 2.06 million in 2014 to 2.58 million in 2015. The rating for Phase III is considered Modest, it reflects the achievement of objectives, not specific indicators. Revised Rating Objective 3 (phase III): Modest Revised Rating Modest PHEFFICACYTBL Objective 4 Objective Upgrade the National Health Information System (SNIS) so that it will be integrated with Bolivia’s new health insurance program. Rationale Outputs: The project contributed to improving information systems for better management and decision making: • 2154 facilities developed and implemented software for primary care; • 80 secondary and tertiary public hospitals developed and implemented clinical and statistical information systems; • 9 Department Health Services developed and implemented software for vital registration and for human resources for health. Outcomes: The original KPI (i.e. percentage of health insurance reports generated by software systems including information about production of services) was dropped twice at the 2012 and the 2014 restructurings, according to both restructuring papers, because the SUS was not implemented within the project's scope (Report No: RES15248, p.4). In the 2014 restructuring, an intermediate outcome indicator was moved as the PDO indicator. Given that the task team stressed that there was always an indicator throughout the project duration to measure objective 4, this objective will not have a split rating. The KPI for this objective show that: • All modules included in the SNIS were fully implemented in the SEDES and in all the heads of the referral networks in the intervention areas, exceeding the target of 60%. The project made considerable progress in delivering outputs to improve Bolivia’s health information systems. The project team states that such improvements, although they could not be applied to the SUS, benefited social security entities since the project financed the development of the SICOFS – Sistema de Control Financiero de la Salud – used to: (i) register basic financial data and monitor health insurance contributions and to apply and calculate the impact of changes in health insurance premiums; (ii) control Page 14 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) payments to healthcare providers according Law 475; and (iii) register and follow up the beneficiaries of the health insurance program. Despite the wide applicability of the upgraded SNIS, the objective specifically calls for the development of the health information system to be used by the new health insurance program. The objective kept an overambitious objective that should have been revised accordingly when it became clear that the new health insurance system specifically cited in the objective was not going to be implemented during the project's lifetime. For this reason, the achievement of this objective is considered modest. Rating Objective 4 original (entire project): Modest Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 5 Objective Overarching goal of the APL series: Reduction of infant, child, and maternal mortality rates. Rationale The development objectives of the APL series were defined as long term impacts on maternal and child health outcomes: • Infant mortality rates decreased from 67 to 33 per 1,000 live births between 1999 and 2013, surpassing the APL target of 48. • Under-5 mortality rates decreased from 92 to 45 per 1,000 live births between 1999 and 2011, surpassing the APL target of 57. • Maternal mortality rates decreased from 390 to 160 per 100,000 live births between 1999 and 2011 according to government estimates (and to 200 per 100,000 live births in 2013 according to the Pan- American Health Organization), surpassing or partially achieving the APL target (164 per 100,000 live births). In the absence of an impact evaluation, the substantive reduction in mortality rates cannot be directly attributed to the program interventions because of the multi-sectoral nature of the determinants of infant, child and maternal mortality. Nevertheless, the interventions proposed in this APLIII were aligned towards the improvements in maternal and child health. Rating Not Rated/Not Applicable Page 15 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) PHREVDELTBL PHREVISEDTBL 5. Efficiency Cost-effectiveness: Cost-benefit analyses (CBA) were conducted at project appraisal and completion. Both CBA considered increased lifetime earnings as project benefits. These earnings would result from infants and children saved from malnutrition, and from maternal, infant, and child deaths avoided, although the multi- sectoral nature of the determinants of mortality was recognized and therefore outcomes could not be directly attributed to project interventions. International literature findings were used to estimate direct project effects on the above outcomes (e.g. children from mothers who had received prenatal care were 30% less likely to die; 20% of maternal deaths are avoidable through prenatal and community-based interventions; 50% of maternal deaths can be prevented by ensuring access to essential obstetric care, etc.). Ex-post CBA estimated fewer maternal and child deaths avoided, presumably because of the reduction in the project’s scope and coverage, but no clarification is provided. On the cost side, both CBA included total project costs and education expenditures resulting from children whose lives have been saved. Again, cost estimates at completion were lower (probably reflecting the change in project scope and cancellation of some of the credit, as well as lower education expenditures as a consequence of the lower number of lives saved). Neither ex- ante nor ex-post CBA provided an estimate of the number of project beneficiaries. Considering a time horizon of 30 years, the IRR was estimated at 9.3% at appraisal and 10% at completion. Reporting is not sufficiently clear to allow for an assessment of the robustness of results. Qualitative efficiency: According to project documents, overall implementation progress was unsatisfactory during the first three years, since only a few proposed activities were undertaken due to slow progress with health reforms. By 2012 the project had disbursed only 15% of the loan amount, and given the unlikely disbursement of the remaining funds before the closing date, the project cancelled about half of total funds (restructuring paper, December 2012). Implementation delays were related to new organizational arrangements within the MSD: contrary to previous phases, the implementation of this APLIII was planned to be integrated into regular MSD functions instead of through a separate program implementing unit. Also, changes in administration across the government and within the MSD led to delays in project management and establishment of leadership to begin implementation. External factors also affected efficient project implementation, as limited availability of construction firms delayed procurement and planned civil works. Positively, the agency in charge of implementing civil works managed to execute the increased proportion of project funds reallocated to infrastructure investments from the other components after the two restructurings. Project costs: There is no evidence that the design interventions were the least cost alternative to achieve desired outcomes, although the project coordinated activities with other programs oriented to encourage demand for health services, which avoided a duplication of efforts. While it is true that the project cancelled about 44% of the total credit, freeing scarce Bank’s lending resources, it had also reduced the project’s scope of about 50% (from 88 locations (82 municipalities + 6 areas) to 44 municipalities), thus reducing its impact on vulnerable population. Moreover, the project was extended in three occasions for a total time of 23 months (about a 44% longer that the original project life). As the ICR notes (ICR p 23), project delays involved high project supervision costs from the bank budget (about US$ 364,000 from mid FY14-FY16 in staff time and Page 16 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) travel costs according to ICR Annex 4 p.43) According to IEG guidelines, there is no split rating for efficiency. This, however, does not prevent the application of the overall efficiency rating into different project phases to calculate the overall outcome rating. The overall efficiency of the project is considered Modest. The improved implementation efficiency that may have followed the initial delays and resulted from the PIU ability to execute a substantial proportion of ‘fewer’ project resources, it does not appear to be sufficiently strong to counterbalance the abovementioned shortcomings and merit an overall efficiency rating of Substantial. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  9.30 Not Applicable 100.00 ICR Estimate  10.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Phase I (Approval-Dec 2012): Moderately Unsatisfactory. Relevance of objectives and design are high and substantial, respectively. The project’s efficacy is in the ‘modest range’, with a substantial achievement of objective 1, and modest achievement of objectives 2, 3, and 4. Efficiency is rated Modest. Phase II (Dec 2012-Jul 2014): Moderately Unsatisfactory. While relevance of objectives remained high, relevance of design was modest because of the elimination of activities that weakened the results framework. Achievement of objectives remains as in the previous phase. The efficiency rating remains Modest. Phase III (Jul 2014-closing): Moderately Unsatisfactory. Relevance of objectives remain high, and design is considered again substantial with the incorporation of the Children’s Hospital activities. Achievement of objective 1 slightly worsened with respect to previous phases and is rated modest. Achievement of the second objective remains modest. Objectives 3 and 4 were modestly achieved. Efficiency remains modest. The resulting combined outcome rating is Moderately Unsatisfactory, indicative of significant shortcomings in the project's preparation and implementation. Restructuring Phase I (Approval-Dec Phase II (Dec 2012-Jul Phase III (Jul 2014- Page 17 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) Periods 2012) 2014) closing) Relevance Objectives high Design substantial modest substantial Efficacy Objective 1: reduce substantial modest risk factors for MMR and IMR Objective 2: reduce modest malnutrition Objective 3: expand modest modest modest health insurance coverage Objective 4: upgrade modest National Health Info System Efficiency modest Outcome Rating Moderately Moderately Moderately Unsatisfactory Unsatisfactory Unsatisfactory Outcome Rating 3 3 3 Value Amount Disbursed 2.81 3.60 3.87 (US$ million) Disbursement (%) 27.3% 35.1% 37.6% a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating The risk to development outcome is considered modest. The project objectives are of high priority for the country and are likely to continue to be supported by current as well as future governments. The fact that some key reforms were supported by an established legal framework makes them unlikely to be reversed. The project team has clarified that, in terms of human resources, personnel from the project implementing unit have been hired as permanent MSD employees. However, the effective integration of the implementing unit into current functions of the MSD is not clear. In addition, infrastructure investments made by the project will require considerable maintenance expenditures, and these could be threatened by slow economic growth and reduced revenues from hydrocarbon resources. Page 18 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The program’s objective was highly relevant to the needs of the country and supported Bolivia’s health system reform. The programmatic nature of the selected lending instrument allowed for the targeting of long- term impacts such as maternal and child mortality rates. The original proposed activities in this third phase of the APL were critical for the achievement of intended outcomes. Activities were selected to tackle the remaining key bottlenecks to the continued implementation of the reform and achievement of its objectives. Project design took into account other parallel programs to exploit synergies and avoid duplication of efforts. However, the planned activities proved to be overly ambitious, covering a broad spectrum of interventions – ranging from system governance and service delivery to financing -- that required stronger implementation capacity and leadership than was available. Difficulties in establishing an implementation team and executing a diverse set of activities, including the implementation of the health insurance component, were not envisioned in the risk assessment and mitigation plans at preparation. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank allocated a large budget and staff resources (three Bank staff assumed Task Team Leader responsibility during project implementation) to supervise, monitor, and report on project progress, and the Bank team was continuously and proactively engaged with the implementing agencies at all stages of the Project. While project restructuring clearly responded to changes in government needs, the first restructuring took place after three years of project implementation and two years of continuous unsatisfactory project ratings. Moreover, objective 4 should have been revised when it became clear that the new health insurance system specifically cited in the objective was not going to be implemented during the project's lifetime. The replacement of key performance indicators with some output-level measures was intended for more precise measurement, but it limited the possibility of tracking improvements over the entire duration of the project. Quality of Supervision Rating Page 19 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance The goals of the APL program were grounded in the government’s commitment to improve health of the Bolivian population, and they were aligned with other programs running in parallel (as already mentioned). However, the failure to pass a law that would enable the introduction of the SUS was a significant shortcoming. It resulted in delays and cancellation of several project activities that reduced the scope of the project interventions, both in terms of target areas and number of activities supported. Additionally, the Government’s decision-making process was slowed down due to organizational restructuring, frequent changes in managerial levels, and delays in allocating skilled resources to project management and implementation. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The implementing agencies were the Ministry of Health and Sports (MSD) and the National Social Investment and Productive Fund (FPS). Unlike in the previous APLs I and II, this APLIII was not planned to be executed by an independent implementation unit. Instead, project activities (execution, supervision, monitoring, and evaluation) were integrated into regular MSD functions with the purpose of enhancing sustainability and allowing better coordination with other programs. There was a need to build the MSD’s capacity to efficiently implement and administer its health programs and strategies, and it was expected that Project Implementation Unit (PIU) staff from previous phases of the APL would be converted into permanent MSD employees to ensure continuity and institutional memory. While the ICR does not provide evidence of such incorporation, the project team states that most of the key PIU staff was absorbed by the MSD to work in the area of planning and/or information systems (SNIS). There were delays in the formation of the implementation team that negatively affected project execution, resulting in the need for extension of project closing dates and partially contributing to high project supervision costs. On the other hand, despite initial delays, once the leadership and team were established the agency managed to execute a substantial proportion of project resources, as these were reassigned to infrastructure investments from the other components following the two restructurings. Implementing Agency Performance Rating Moderately Satisfactory Page 20 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The results framework at appraisal included 6 KPIs and 12 intermediate outcome indicators with complete baseline and target values. Arrangements for results monitoring in Annex 3 of the PAD clearly delineate frequency, instruments, and responsibilities for data collection and M&E reporting. Indicators were constructed to measure progress in project areas vis-a-vis the rest of the country. KPIs associated to the objective of reducing risk factors of maternal and child health actually measured inequalities in health care utilization, as they were phrased as ratios. The gap between health care utilization and reduction of risk factors in maternal/child health was partially filled by the intermediate indicator on consumers’ perceptions about the quality of service received. The project design envisaged an impact assessment which was be done with the participation of an external consulting firm (PAD p. 11), but there is no mention of this study in the ICR. b. M&E Implementation The majority of the indicators were tracked regularly for the purpose of supervision reports; however, changes in project activities and the resulting modifications of the results framework brought challenges in terms of data collection and reporting. The results framework was revised twice. The 2012 restructuring dropped KPIs associated to objectives 3 and 4 because most of the related project activities were canceled. The former was not replaced, which left the project lacking a measure of progress during Phase II. KPI introduced at the 2014 restructuring did not measure health insurance coverage itself, but rather a registration system to inform the health insurance system once implemented. The 2014 restructuring also dropped the two KPIs aiming at measuring progress towards the achievement of objective 2 because the SNIS (original source proposed) was not collecting such information. These indicators were replaced by an output-level indicator less linked to the expected outcome. The ICR (p. 19) also acknowledges that the revised targets may have been under-ambitious compared to the current levels of achievement at the time of the 2014 restructuring, when progress towards the achievement of these targets was well under way. Many indicators of the revised results framework were not adequately linked to the objectives they intended to measure. c. M&E Utilization The ICR (p. 11) reports on the numerous activities the project undertook to support health management and information systems (software in primary health facilities, monitoring of clinical statistics in secondary care facilities, etc.), but it does not provide information on the effective use of the project’s M&E. The project team Page 21 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) stated that M&E activities started to be used to drive project decisions in a more effective way after 2013. M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was Environmental Screening Category B (partial environmental assessment) and triggered OP/BP 4.01 (Environmental Assessment), due to the likely increase in production of health care waste and the impacts of civil works. This assessment recommended a close follow up of physical rehabilitation impacts by an internal unit within MSD. An additional environmental assessment was conducted and approved during the third restructuring when the project added the construction of the onco-hematologic unit of La Paz Children’s Hospital. The project also triggered the Indigenous Peoples (OP/BP 4.10) safeguard policy since it focused on intercultural health for the maternal and infant population, as well as the fact that most beneficiaries self- identified as indigenous. The project was considered an Indigenous Project, and hence no indigenous people’s plan was necessary. The ICR did not include a specific statement about safeguards compliance, but the TTL subsequently indicated that the project was considered to be in compliance. b. Fiduciary Compliance Financial management: Financial management performance was considered Moderately Satisfactory during project implementation. Despite low project execution during the first three years, no major shortcomings were identified regarding financial management in either of the two implementing agencies. The FPS (but not the MSD) presented some financial audits and financial transactions documentation with some delay. Procurement: Procurement was rated Satisfactory over the life of the project. No serious procurement issues were identified by post review missions and procurement audits. The FPS delays in contracting civil works are explained by limited supply of construction firms available to pursue the job. c. Unintended impacts (Positive or Negative) None reported. Page 22 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) d. Other -- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment The ICR Review based the assessment of efficacy on the sub-objectives exactly as Moderately Moderately stated in the legal agreement. Outcome Satisfactory Unsatisfactory For some objectives, there was insufficient evidence of achievement of intended outcomes. Risk to Development Modest Modest --- Outcome Following OPCS/IEG harmonized guidelines, the Moderately Moderately Bank Performance Bank Performance rating in Satisfactory Unsatisfactory this case is determined by the Outcome rating. Following OPCS/IEG harmonized guidelines, the Moderately Moderately Borrower Performance Borrower Performance rating Satisfactory Unsatisfactory in this case is determined by the Outcome rating. Quality of ICR Substantial -- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR offers the following relevant lessons from the preparation and implementation of this project: • The implementation of project activities cannot depend on the promise of approving a law. The design of this APLIII included activities towards the increase of insurance coverage under a new unified health insurance scheme (SUS), which in the end was not established. By anticipating that risk, the project design could have included support for passing the law or for other financial protection coverage strategies. • Carefully estimating times required to complete transitions is essential to project design and implementation. Page 23 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) The shift of the implementing unit into the MSD organizational structure took more time than originally estimated and delayed project execution at the beginning of the project. Efforts to mitigate these risks to implementation at project design could have allowed for more timely implementation. • The implementation of information systems requires efforts along two separate fronts: in the collaborative design of the systems, and in the creation of the demand for the information generated by the systems. In its support of the design and implementation of the information systems, the project made great efforts to ensure that systems were designed to collect data that was valuable to managers and policy makers by conducting careful analyses of needs. The project also made sure to build the demand for the use of newly available data by conducting manager and decision-maker training workshops. This ICRR highlights an additional lesson in terms of assessing attribution: A project designed to coordinate with other government programs with similar goals can enhance efficiency by minimizing duplication of efforts. If the collaborating programs do not design their M&E arrangements to specifically collect information that will enable them to separate out their respective contributions, however, such collaboration limits both programs' ability to attribute any achievement of development outcomes to the activities of either program. M&E design that supports attribution of collaborating partner programs is possible, but a framework must be in place as collaboration begins, and must be supported by adequate data collection and analysis. This APL III was originally designed to complement the Zero Malnutrition and Social Protection programs, both aiming to increase demand for health care services and reduce malnutrition. Without a clear counterfactual, it is difficult to disentangle effects of this APL project and other program effects. 14. Assessment Recommended? Yes Please explain The ICR Review considers that for some objectives there was insufficient evidence of achievement of project outcomes, and therefore the outcome rating was downgraded. Also, a full project evaluation would shed light on the effectiveness, consolidation, and sustainability of the whole APL program investments on maternal and child health outcomes. 15. Comments on Quality of ICR The ICR provides an adequate description of the project's objectives, outputs, and outcomes. It is candid in highlighting weaknesses in the causal pathway between activities and development objectives when project components changed. The ICR also evaluates the adequacy of the KPIs, recognizing that some selected indicators were weak proxies of the projects expected outcomes. Page 24 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BO-Exp. Access to Reduc Hlth Ineq (APL3) (P101206) The ICR provides additional information on immunization and health insurance coverages, comparing target and non-target areas and department level figures to support achievement of objective 1 and 3. The ICR could have acknowledged the contribution of other parallel programs in achieving expected or actual outcomes. Some information provided later on by the task team was not reflected, or sometimes contradict, in the ICR document (e.g. the approval of the Law underlying the creation of the health insurance program). a. Quality of ICR Rating Substantial Page 25 of 25