Document of The World Bank Report No: ICR00003720 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA 43820-BO) ON A CREDIT IN THE AMOUNT OF SDR 11.7 MILLION (US$ 18.5 MILLION EQUIVALENT) TO THE PLURINATIONAL STATE OF BOLIVIA FOR A EXPANDING ACCESS TO REDUCE HEALTH INEQUITIES- APL III (HEALTH SECTOR REFORM PROJECT, THIRD PHASE) June 30, 2016 Health, Nutrition, and Population Global Practice Bolivia, Chile, Ecuador, Peru and Venezuela Country Management Unit Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective May 2016) Currency Unit = Bolivian Bolivianos (Bs) Bs 1 = US$ 0.14 US$ 1.00 = Bs 6.91 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS APL Adaptable Program Loan CDC Centers for Disease Control and Prevention CPS Country Partnership Strategy CRED Control de Crecimiento y Desarrollo - Height Growth Control DO Development Objective EPHF Essential Public Health Functions EXTENSA Programa Nacional para la Extension de Cobertura de Seguros – National Program for the Expansion of Insurance Fondo Nacional de Inversión Productiva y Social – National Social Investment FPS and Productive Fund GoB Government of Bolivia HSRP Health Sector Reform Program ICR Implementation Completion Report IDA International Development Association IEG Independent Evaluation Group IMCI Integrated Management of Childhood Illness IMR Infant Mortality Rate INE Instituto Nacional de Estadística – National Institute of Statistics IP Implementation Progress IRR Internal Rate of Return ISN Interim Strategy Note ISR Implementation Status and Results Reports M&E Monitoring and Evaluation MMR Maternal Mortality Rate MS Ministerio de Salud – Ministry of Health PAD Project Appraisal Document PAI Programa Ampliado de Inmunizaciones - Expanded Immunization Program PDI Project Development Indicator PDO Project Development Objective PRONACS Proyecto Nacional de Calidad de Salud - National Program of Quality of Health RF Results Framework SBS Seguro Básico de Salud – Basic Health Insurance ii SEDES Servicio Departamental de Salud – Departamental Health Service SNIS Sistema Nacional de Información en Salud – National Health Information System SUMI Seguro Universal Materno-Infantil – Maternity and Childhood Insurance SUS Sistema Único de Salud – Single Health System U5MR Under Five Mortality Rate UGTFD Unidad de Gestion Técnica y Financiera Departamental – Departamental Unit for Technical and Financial Management UGTFN Unidad de Gestion Técnica y Financiera Nacional – National Unit for Technical and Financial Management WDI World Development Indicators Country Director Alberto Rodriguez Senior Global Practice Director: Timothy Grant Evans Practice Manager: Daniel Dulitzky Task Team Leader: Andre C. Medici ICR Team Leader/Author: Roberto F. Iunes iii BOLIVIA Expanding Access to Reduce Health Inequities- APL III (Health Sector Reform Project, Third Phase) TABLE OF CONTENTS Data Sheet A. Basic Information....................................................................................................... v B. Key Dates ................................................................................................................... v C. Ratings Summary ....................................................................................................... v D. Sector and Theme Codes .......................................................................................... vi E. Bank Staff .................................................................................................................. vi F. Results Framework Analysis .................................................................................... vii G. Ratings of Project Performance in ISRs .................................................................... x H. Restructurings ............................................................................................................ x I. Disbursement Profile ................................................................................................. xi 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 9 3. Assessment of Outcomes .......................................................................................... 13 4. Assessment of Risk to Development Outcome......................................................... 21 5. Assessment of Bank and Borrower Performance ..................................................... 21 6. Lessons Learned ....................................................................................................... 23 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 24 Annex 1. Project Costs and Financing .......................................................................... 25 Annex 2. Outputs by Component ................................................................................. 27 Annex 3. Economic and Financial Analysis ................................................................. 38 Annex 4. Bank Lending and Implementation Support/Supervision Processes............. 42 Annex 5. Beneficiary Survey Results ........................................................................... 44 Annex 6. Stakeholder Workshop Report and Results................................................... 45 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 46 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 51 Annex 9. List of Supporting Documents ...................................................................... 52 Annex 10. Assessment of Project Achievement towards Outcomes ............................ 53 MAP .............................................................................................................................. 58 iv Data Sheet A. Basic Information Expanding Access to Reduce Health Inequities Project (APL III)-- Country: Bolivia Project Name: Former Health Sector Reform - Third Phase (APL III) Project ID: P101206 L/C/TF Number(s): IDA-43820 ICR Date: 04/10/2016 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: APL Borrower: BOLIVIA Original Total XDR 11.70M Disbursed Amount: XDR 6.80M Commitment: Revised Amount: XDR 6.85M Environmental Category: B Implementing Agencies: Ministry of Health Fondo Nacional de Inversión Productiva y Social (FPS) Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/05/2007 Effectiveness: 06/19/2009 06/19/2009 12/10/2012 01/10/2014 Appraisal: 12/10/2007 Restructuring(s): 07/14/2014 08/28/2015 Approval: 01/24/2008 Mid-term Review: 11/30/2011 08/19/2013 Closing: 01/31/2014 12/31/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory v C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Unsatisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Supervision Yes None time (Yes/No): (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 15 24 Compulsory health finance 9 1 Health 62 75 Sub-national government administration 14 Theme Code (as % of total Bank financing) Child health 20 50 Health system performance 40 20 Nutrition and food security 20 10 Population and reproductive health 20 20 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Calderon Pamela Cox Country Director: Alberto Rodriguez Carlos Felipe Jaramillo Practice Manager: Daniel Dulitzky Keith E. Hansen Project Team Leader: Andre C. Medici Carlos Marcelo Bortman ICR Team Leader: Roberto F. Iunes ICR Primary Author: Roberto F. Iunes vi F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project Development Objectives for APL III were: (i) to reduce occurrence of critical risk factors affecting maternal and infant health in the targeted areas so that current gaps between regions are reduced; (ii) to reduce chronic malnutrition among children under 2 years of age in the targeted areas; (iii) to increase health insurance coverage in the targeted areas; and (iv) to upgrade the National Health Information System (Sistema Nacional de Información en Salud -SNIS) so that it will be integrated with Bolivia’s new health insurance program. (a) Overarching Program Indicator(s) Original Target Values Actual1 Value Achieved 1 Indicator Baseline Value (from approval at Completion or Target documents) Years Indicator 1 : Infant Mortality Rate per 1,000 live births Value quantitative or 67 48 32.8 Qualitative) Date achieved 1999 2015 2013 Comments (incl. % Surpassed (180%). achievement) Indicator 2 : Under 5 Mortality Rate per 1,000 live births Value quantitative or 92 57 44.9 Qualitative) Date achieved 1999 2015 2011 Comments (incl. % Surpassed (134%). achievement) Indicator 3 : Maternal Mortality Rate per 100,000 live births Value quantitative or 390 164 200/160* Qualitative) Date achieved 1999 2015 2013/2011* Comments (incl. % Partially Achieved (84%) and achieved* (101.7%). achievement) (b) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Ratio between the percentages of pregnant women receiving four pre-natal care in the Indicator 1 : areas of project intervention and the rest of the country. Value quantitative or 0.91 0.85 0.95 0.89 Qualitative) Date achieved 12/31/2009 12/31/2014 12/31/2015 12/18/2015 1 Estimated Values as reported by The Pan-American Health Organization vii Comments (incl. % Achieved (105% against original, 94% against revised). Revised July, 2014. achievement) Ratio between the percentage of institutional deliveries in the areas of Project Indicator 2 : intervention and the rest of the country. Value quantitative or 0.72 0.85 0.89 0.98 Qualitative) Date achieved 12/31/2009 12/31/2014 12/31/2015 12/31/2015 Comments (incl. % Surpassed (115% against original, 110% against revised). Revised July, 2014. achievement) Percentage of 2 year old children covered with height growth control (CRED) in the Indicator 3 : intervention areas of the Project. Value quantitative or 35.00 40.00 89.00 Qualitative) Date achieved 12/31/2009 12/31/2015 12/31/2015 Comments (incl. % Surpassed (222%). achievement) Percentage of Target Population Enrolled in the Family Register Forms ( Carpeta Indicator 4 : Familiar) in the Project Target Areas. Value quantitative or 0.00 80.00 60.00 80.00 Qualitative) Date achieved 12/31/2009 12/31/2014 12/31/2015 12/31/2015 Comments (incl. % Achieved (100%) against original indicator, surpassed (133%) against revised. achievement) The modules included in the SNIS are fully implemented in the SEDES, and in all the Indicator 5 : heads of the referral networks in the intervention areas. Value quantitative or 0.00 100% 60% 100% Qualitative) Date achieved 12/31/2009 12/31/2014 12/31/2015 12/31/2015 Comments Achieved (100%) against original indicator, Surpassed (167%) against revised. Changed (incl. % from Intermediate Indicator to PDI July, 2014. achievement) viii (c) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised Target approval Completion or Values2 documents) Target Years Percentage of health units applying the sector regulations norms and the PRONACS Indicator 1 : norms Value (quantitative 0.00 80 40 94 or Qualitative) Date achieved 12/31/2009 12/31/2014 12/31/2015 12/31/2015 Comments Surpassed (118% against original, 235% against revised). Revised July, 2014 (incl. % achievement) Percentage of health units in the target areas following approved normative to Indicator 2 : characterize health units at first and second level Value (quantitative 0.00 20 98 or Qualitative) Date achieved 12/31/2009 12/31/2015 12/31/2014 Comments (incl. % Surpassed (490%). Added July, 2014. achievement) Indicator 3 : Percentage of civil works concluded in the health facilities in the areas of intervention Value (quantitative 0.00 100 100 or Qualitative) Date achieved 12/31/2009 12/31/2015 12/31/2014 Comments (incl. % Achieved (100%). Added July, 2014. achievement) Indicator 4 : Percentage of equipment installed in the health facilities in the areas of intervention Value (quantitative 0.00 100 100 or Qualitative) Date achieved 12/31/2009 12/31/2015 12/31/2014 Comments (incl. % Achieved (100%). Added July, 2014. achievement) Percentage of user's population receiving services satisfied with the quality of the Indicator 5 : services received in the Project areas Value (quantitative 0.00 30 76.50 or Qualitative) 2 All indicators in the Project’s results framework were revised in the restructuring paper of July, 2014 to reflect changes to project activities. ix Date achieved 12/31/2009 10/31/2015 12/31/2014 Comments Surpassed (255%). Added July, 2014. Survey was designed and implemented by the (incl. % SNIS and the Project Execution Unit achievement) G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 05/17/2008 Satisfactory Satisfactory 0.00 2 10/20/2008 Moderately Unsatisfactory Satisfactory 0.00 3 06/19/2009 Unsatisfactory Unsatisfactory 0.00 4 07/29/2009 Unsatisfactory Unsatisfactory 0.00 5 12/09/2009 Unsatisfactory Unsatisfactory 0.00 6 01/09/2010 Unsatisfactory Unsatisfactory 0.00 7 05/06/2010 Unsatisfactory Unsatisfactory 0.30 8 06/30/2010 Moderately Satisfactory Moderately Satisfactory 0.30 9 02/22/2011 Moderately Unsatisfactory Moderately Unsatisfactory 1.05 10 10/15/2011 Moderately Unsatisfactory Moderately Unsatisfactory 1.77 11 06/19/2012 Unsatisfactory Unsatisfactory 2.24 12 02/27/2013 Moderately Unsatisfactory Moderately Satisfactory 3.22 13 10/20/2013 Moderately Satisfactory Moderately Satisfactory 4.56 14 05/31/2014 Moderately Satisfactory Moderately Satisfactory 6.42 15 12/10/2014 Moderately Satisfactory Satisfactory 7.99 16 06/10/2015 Moderately Satisfactory Moderately Satisfactory 9.67 17 12/30/2015 Moderately Satisfactory Moderately Satisfactory 10.20 H. Restructurings ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Approved Reason for Restructuring & Key Restructuring Date(s) PDO Changes Made DO IP in USD Change millions 1. Reduce Project scope (components) without change of PDOs; 2. Reduction of credit amount; 3. Adjust results framework; 4. Reallocation of funds 12/10/2012 N U U 2.81 between categories. Main reason: delay in the approval of the Health Insurance Law and consequent impact on implementation of select Project activities. 1. The closing date was extended by 01/10/2014 N MS MS 5.27 5 months to July 31, 2015. x ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Approved Reason for Restructuring & Key Restructuring Date(s) PDO Changes Made DO IP in USD Change millions 1. Revision of results framework; 2. Reallocation of funds between categories; 3. Change disbursement 07/14/2014 N MS MS 6.42 estimates; 4. Extend closing date by 15 months. Main reason: Co-financing of the Onco-hematologic Block of La Paz’s Children Hospital 1. Extension of closing date by 2 months. Main reason: to allow for 08/28/2015 MS MS 9.82 completion of Onco-hematologic Block of La Paz’s Children Hospital. I. Disbursement Profile xi 1. Project Context, Development Objectives and Design 1. In 1999, the Government of Bolivia (GoB), launched a major Health Sector Reform Program (HSRP) with the overall goal of reducing the country’s infant mortality rate. The HSRP was structured around two main strategies designed to: (i) increase coverage and quality of health services and related programs that would improve the health of the population, and to empower communities to improve their health status; and (ii) strengthen local capacity to respond to health needs. The Reform was to be implemented in the following priority areas: (i) the development of an unified model of attention based on an integrated care for children; a package of services for women and newborns; and the establishment of a new, broader, and sustainable program of immunizations; (ii) the establishment of a basic health insurance program targeting the poor; (iii) the introduction of new management instruments, such as performance agreements, to guarantee the progressive and systematic development of efficient supervision and accountability; (iv) the development of new mechanisms for resource allocation, with a demand-based approach, which will promote greater community participation and empowerment; and (v) the deepening of the decentralization process through the establishment of service networks within health districts related to municipal governments. 2. The “Expanding Access to Reduce Health Inequities Project,” the object of this ICR, is the third and final operation of a series of Adaptable Program Loans (APLs) designed to support the HSRP. 3. The overarching development objective of the APL series or Program was to help the GoB reduce the country’s infant mortality rates by complementing other interventions in education, rural productivity and water and sanitation. As stated in the APL I Project Appraisal Document (PAD), the APL was the appropriate financing instrument to support the stated goal of the HSRP, because: (i) reducing infant mortality rates requires changes that can only be achieved over the long-term; (ii) the introduction of the Reform’s new management instruments requires experimentation, learning-by doing and adjustment over time; and (iii) the Bank’s presence in the sector would help secure priority to the objectives of the Reform over successive government administrations. 1 The three APLs financed a total of approximately US$71.3 million, 95 percent of the originally planned US$75 million total credit amount. 4. The objective of the first APL (APL I) was broadly defined to help reduce infant mortality rates. The APL I would also follow the two main strategies of the HSRP mentioned above. The Project was approved on June 15, 1999 with a total credit amount of US$17.8 million, and closed on December 31, 2003. According to its ICR, the APL I successfully supported the creation and implementation of two health insurance programs directed at mothers and children: the Maternity and Childhood Insurance (Seguro Universal Materno-Infantil, SUMI) and the Basic Health Insurance (Seguro Básico de Salud, SBS). The APL I also strengthened Bolivia’s immunization program, implemented performance agreements between the central government 1 Although not explicitly discussed in the main body of the APL I PAD, reducing maternal mortality is also presented as an APL series goal in its Annex 1. In the third APL, a target for reducing under-five mortality rate is also presented as a goal of the series (see also Section 1.2, below). 1 and regional departments and improved health services by upgrading the infrastructure and equipment of health units. During this first APL phase, human resources were trained, protocols were designed and a process for monitoring quality of care was developed. The Project’s outcomes were rated as Satisfactory by the Implementation Completion Report (ICR) and the Independent Evaluation Group (IEG). 5. The development objectives of the second APL (APL II) also supported the two main strategies of the HSRP. The APL II was approved on June 28, 2001 with a total credit amount of US$35.0 million, and closed on June 30, 2008, after an extension of the closing date by 24 months, and after changes to indicator definitions and targets which were approved in a project restructuring. This second phase expanded the health insurance programs implemented in the first APL, by launching the National Program for the Expansion of Insurance Coverage (Programa Nacional para la Extensión de Cobertura de Seguros, EXTENSA) insurance. The APL II also continued to support the national immunization program, implementing a national program to improve quality of health services and supporting new strategies and tools to improve management practices at departmental and local levels as well as improving the leadership and steering role of the Ministry of Health and Sports (Ministerio de Salud, MS). The Project’s ICR and IEG outcome rating was Moderately Satisfactory. 6. In summary, an important set of policies and interventions were developed under APLs I and II to strengthen the performance of public health services. These aimed at: improving child and maternal health status; expanding the insurance system; financing investments in primary health care, and the strengthening of cost-effective health interventions such as the Expanded Immunization Program (PAI), the Integrated Management of Childhood Illness (IMCI), and a mother-baby package of services. 7. The objectives of the third phase of the APL series (APL III) were broadly aligned with the two tenets of the HSRP. These are: (i) to increase coverage and quality of health services and related programs that would improve the health of the population, and to empower communities to improve their health status; and (ii) to strengthen local capacity to respond to health needs. The APL III was approved on January 24, 2008 and provided an original credit of SDR11.7 million (approximately US$18.5 million) to support an expanded number of activities (see Section 1.2, below, for a more detailed description of the development objectives of the APL III). 1.1 Context at Appraisal 8. In 1999, at the time the APL series was originally designed, Bolivia’s human development indicators were among the lowest in Latin America. Life expectancy at birth, for instance, was near the bottom of regional rankings, only above Haiti.2 A sizable proportion of the Bolivian population lived in poverty (36 percent lived below the national poverty line), without access to safe water (23 percent had no access to a safe water source), adequate food supplies (the depth of the food deficit was 239 kcal) and with little or no access to basic health services (56 percent of births were attended by skilled health staff).3 Efforts to control communicable diseases had not shown significant improvements and malnutrition rates among children remained stagnant 2 World Development Indicators (WDI) data. 3 WDI data. 2 and high.4 By 2007, when the APL III was designed, the Bolivian economy was growing steadily (GDP growth was 4.6 percent), but poverty rates remained high (30 percent of the population was below the national poverty line) and access to services in rural areas remained relatively low (16 percent of the rural population lacked access to an improved water source). 5 9. The election of President Evo Morales in 2005 represented a major political and ideological change. Morales was elected by an absolute majority of the popular vote and obtained a majority in the lower house of Congress. With growing revenues from the hydrocarbon sector, the Morales administration took office in 2006 promising sweeping reforms aimed at improving the health status of the population and reducing health inequities. The measures proposed by the Government included: (i) strengthening health system networks; (ii) increase coverage, access to and demand for health services, especially for the most vulnerable populations; (iii) increase the focus on the promotion of good health practices and of disease prevention; (iv) increase the use of intercultural health practices; (v) prevent chronic malnutrition; (vi) upgrade the health insurance system; (vii) strengthen the MS’s capacity to implement the Essential Public Health Functions (EPHF).6 The Government’s priorities focused on achieving better health results in maternal and infant mortality and nutrition. 10. Main Health Sector Issues. The data available for the period that traversed from the first to the third APL, continued to show large variation in health indicators and persistent inequities across regions, income quintiles and sub-systems. While the percentage of children under the age of 5 suffering from chronic malnutrition was 32 percent in the Highlands, it was half of this (16 percent) in the Plains (2003 data). Similarly in 2003, maternal mortality ratio (per 100,000 live births) was 320 in the Highlands, 147 in the Valleys and 206 in the Plains.7 Health expenditures were nearly five times higher among the highest income quintile (US$116.34 per capita per year) when compared to the lowest income quintile (US$24.43 per capita, 1999 data). In 2007, the MS provided health services to nearly 70 percent of the population, but its expenses represented only 25-28 percent of the national expenditures in health. Meanwhile the sub-systems responsible for the remainder 30 percent of the population (primarily social security and, to a much lesser extent, the private insurance sector), accounted for 42-50 percent of expenditures. Health expenditure per capita had increased from US$57 to US$70 between 2000 and 2007 and life expectancy had 4 1999 and the 2003 Health and Nutrition Surveys showed the percentage of children with chronic malnutrition at 27 percent. 5 WDI data. 6 The Essential Public Health Functions are: (1) monitor health status to identify and solve community health problems; (2) diagnose and investigate health problems and health hazards in the community; (3) inform, educate, and empower people about health issues; (4) mobilize community partnerships and action to identify and solve health problems; (5) develop policies and plans that support individual and community health efforts; (6) enforce laws and regulations that protect health and ensure safety; (7) link people to needed personal health services and assure the provision of health care when otherwise unavailable; (8) assure competent public and personal health care workforce; (9) evaluate effectiveness, accessibility, and quality of personal and population-based health services; (10) research for new insights and innovative solutions to health problems (source: United Sates Centers for Disease Control and Prevention, CDC). 7 National Institute of Statistics (Instituto Nacional de Estadística, INE) 2003-2004. 3 increased by four years from 61 to 65 years over this same period of time.8 By 2007, inequalities remained, but trends pointed to improvements in the health sector overall (see Annex 10). 11. Country Partnership Strategy and Rationale for Bank involvement. The Bank provided important support for Bolivia’s health sector for almost 10 years prior to the preparation of the APL III. The Bank’s funds, through the APL program, provided technical assistance, facilitated donor coordination, and provided training and financing; achieving significant results in expanding health insurance coverage and increasing access to health services to previously underserved populations.9 The first two phases were consistent with Bolivia’s Interim Strategy Notes (ISN). The APL III was designed with the aim of consolidating progress achieved during the previous two phases of the APL program and was grounded on the Morales administration’s commitment to improve health of the Bolivian population. In fact, in 2006 the Bank and the GoB agreed on an ISN to move forward on projects and discussions within the context of three objectives: (i) enhancing good governance and transparency, (ii) fostering jobs through growth, and (iii) providing better services to the poor. Furthermore, the APL III was aligned with three of the four tenets of the Bank’s Strategy for Health, Nutrit ion, and Population Results, approved earlier in 2007. The APL III continued to be relevant under the Country Partnership Strategy (CPS) for the fiscal period 2012-2015 by supporting the human development and access to basic services results area. Finally, the Bank’s continued support to the Health Sector Reform program, its international experience supporting such reforms and its important coordinating role in the health sector set it as an important partner for the continued implementation of the reform through the APL III. 1.2 Original Project Development Objectives (PDO) and Key Indicators 12. The APL III aimed to continue supporting the GoB in its efforts to improve the lowering trend in infant mortality rates (IMR) and accelerate the reduction in the rates of maternal mortality (MMR). Accordingly, the PDO identified in the APL III PAD and Project’s Legal Agreement were: (PDO 1) to reduce the occurrence of critical risk factors affecting maternal and infant health in the target areas so that current gaps between regions are reduced; (PDO 2) to reduce chronic malnutrition among children under 2 years of age in the target areas; (PDO 3) to increase health insurance coverage in the target areas; and (PDO 4) to upgrade the National Health Information System (Sistema Nacional de Información en Salud, SNIS) so that it becomes integrated with Bolivia’s new health insurance program. 8 WDI data. 9 Implementation Completion Report on Health Sector Project, Second Phase, World Bank 2009. 4 13. These objectives were to be measured by the following original outcome indicators : Table 1: APL III Original Outcome Indicators PDO Original Outcome Indicator - Ratio between the percentage of pregnant women receiving four pre-natal care check-ups in the target areas and the rest of the country 1 - Ratio between the percentage of institutional deliveries in the target areas and the rest of the country - Percentage of children receiving exclusive breast feeding at 6 months in the target 2 areas - Percentage of children 2 years old over -2Z scores in the target areas 3 - Percentage of population enrolled in health insurance in target areas - Health insurance reports generated by software system include information about 4 production of services 14. The three APLs were also subject to an overarching development objective: to reduce the country’s IMR by approximately 30 percent ( from 67 to 48 per 1,000 live births) at the end of the APL Program. Additionally, the PADs of the first and third APLs also indicate a Program development objective of reducing the country’s MMR: the original target set in the APL I was to reduce Bolivia’s MMR from 390 to 290 per 100,000 live births.10 In this third APL, the final target for the reduction in MMR is set at 164 per 100,000 live births. The APL III also adds an explicit Program target for reducing under-five mortality rate (U5MR) from 92 to 57 per 1,000 live births. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 15. The APL III was restructured four times over the course of its implementation, however, its PDOs remained unchanged. The Results Framework (RF) was revised twice (in 2012 and 2014) to remain consistent with the on-going reform process. After the first three years of implementation, the Project had only completed a small number of the planned activities, as the MSD delayed the establishment of the team responsible for its implementation and the Bolivian health reforms did not progress as expected, leading to an overall unsatisfactory rating of its implementation progress. 16. The initial challenges to Project implementation led the Government and the Project team to revise the Project framework and to cancel some of the credit resources (about 40 percent) in a full Level 2 Restructuring in December 2012. The new RF responded to the government needs, by allowing greater flexibility to undertake activities and adopt measures to guarantee the eventual implementation of health insurance (Component 3) while tracking Project progress with revised intermediate indicators, targets and baselines. The new activities that aimed to extend health insurance coverage (registering families at the health units under a national household registration process--Carpeta Familiar) were expected to provide the database for the implementation of the 10 In the APL I, the Program target of reducing maternal mortality does not appear in the main body of the PAD, but in its Annex 1. 5 health insurance, when approved by Congress. It should be noted, however, that even though the restructuring did not change any of the PDOs, the indicators related to PDOs 3 and 4 were dropped, which meant that the Project was left without any definition of how to measure progress towards the fulfillment of two development objectives. 17. In January of 2014, the Bank approved a six month extension of the project’s closing date. The operation was extended from January 31, 2014 to July 31, 2014, to allow enough time to complete a full Level 2 Project Restructuring, as requested by the Government of Bolivia in November 2013. There were no changes to PDOs, PDIs, components or activities. It is important to note that the original restructuring request letter from the Government asked for a 18 months extension of the Project closing date (and not a six months extension) and requested the inclusion of a new activity under Component 2 – Renovation of the Onco-Hematologic Unit of the Children’s Hospital in La Paz. This restructuring was, hence approved to allow for the preparation of a third restructuring that could allow for the inclusion of new Project activities. 18. The third Project Restructuring, another full Level 2, was approved in July 2014, again revising the RF of the Project. Indicators, baseline data and targets were revised to increase clarity, improve the accuracy of indicator definitions and data, and to incorporate indicators that would be more adequate to measure progress towards the PDO. The revision of the RF was comprised of: (a) an updated version of the PDIs and simplification of intermediate indicators; (b) the adaptation of some indicators originally linked to the Single Health System (Sistema Único de Salud, SUS), which was not implemented by the Government; (c) the definition of targets and protocols for some Project indicators that were not defined in the PAD; (d) a revision of end of Project target values to reflect expected progress based on the revised baselines and areas of intervention; and (e) replacement of indicators to avoid duplication. The Project restructuring also included the addition of new activities in support of the renovation of the Onco-Hematologic Unit of La Paz Children’s Hospital, but these did not lead to any specific changes in Project objectives or indicators, as the renovation will support the improvement of children’s health related to onco- hematologic disorders (such as leukemia) and avoid early mortality for this group. The restructuring also extended the closing date of the Project for 15 additional months from July 31, 2014 to October 31, 2015. 19. On August 2015, a fourth Project Restructuring was approved. This extended the closing date of the Project to December 2015 in order to complete civil works at the Onco- Hematologic Unit of the Children’s Hospital, as requested by the Government of Bolivia. This restructuring did not adjust the PDO nor the PDIs. 20. These changes define three Project phases. The first one reflects the original design and extends from effectiveness until the first full Level 2 restructuring in December of 2012, i.e. between 2009 and 2012. The second phase embodies the changes introduced by this first restructuring and encompasses the period until the changes in the Results Framework introduced by the full Level 2 restructuring of July of 2014, thus spanning the period between 2012 and 2014. The third and final phase captures the changes of this third restructuring and lasts until the closing of the Project in December of 2015, i.e. from July of 2014 to December of 2015. 6 1.4 Main Beneficiaries 21. In addition to the focus on improving health outcomes of mothers and children, this APL also targeted the most vulnerable municipalities. The first and second APLs were designed to benefit all Bolivian children under five years of age and all mothers using public health services, a total population of about two million. This third APL continued to focus on these sub- populations and, despite its national scope, had a greater emphasis on 44 municipalities that had been identified by the Government as being the most vulnerable. 1.5 Original Components 22. The APL III originally had four main components as described below (see Annex 2 for more details): Component 1: Stewardship Role of Health Authorities – Essential Functions in Public Health (US$7.7 million: US$4 million IDA + US$3.7 million Government): this component aimed at strengthening the capacity of national, regional and local health authorities to effectively perform the critical EFPH. Component 2: Family, Community and Intercultural Health (US$11.6 million: US$9.9 million IDA + US$1.7 million Government): this component aimed at improving access to maternal and infant health services in the Project’s target areas. Component 3: Health Insurance (US$4.2 million: US$3.2 million IDA + US$1 million Government): to support the Government of Bolivia’s implementation of the new “SU SALUD” health insurance program. Component 4: Project Administration (US$2.2 million: US$0.9 million IDA + US$1.3 million Government): to support Project administration with equipment, technical assistance, training and operating costs to finance the administration of the Project, and financial and procurement audits. 1.6 Revised Components 23. Project components were revised with the Level 2 Project Restructuring of December 2012. Component 1 was revised to include two new activities: (i) the development of an index to follow the progress of mothers and children 2 years after birth, and (ii) the support to the organization and consolidation of the Cumbre de Salud.11 The restructuring cancelled activities related to: (i) monitoring and evaluation of the Government’s Sector Development Plan for 2006 - 2010 which was not implemented by the MS; (ii) the establishment of a National Health Research System; (iii) the Demographic and Health National Survey 2011; (iv) activities to strengthen the coordination with bilateral and multilateral donors; and (v) Project evaluation activities.12 Five 11 The Cumbre de Salud was the first participatory consultation process in the health sector promoted by the MSD. The Cumbre de Salud was organized as a bottom-up process, where the Departments will organize regional conferences, with broad participation of the municipal health authorities, providers and citizens on the definition of local priorities. These instances will select the delegates to participate in the Cumbre de Salud, which took place on March 2013. 12 A government agency (UDAPE) was leading an impact evaluation of this and other Bank- financed projects. 7 activities were cancelled in Component 2: (i) expansion of human resources; (ii) development of continuing education programs; (iii) implementing operational research of disease outbreaks and suspected outbreaks; (iv) creation of regional maintenance centers; and (v) strengthening of local management and community participation. Activities for the implementation of a national program of quality, planned under Component 1, were integrated into Component 2. Component 3 was revised to cancel activities related to: (i) the strengthening of the SU SALUD insurance program enrollment system; and (ii) the strengthening of the monitoring and evaluation management capacity of the National Unit for Technical and Financial Management ( Unidad de Gestión Técnica y Financiera Nacional, UGTFN) and Departmental Units (UGTFDs). A new sub- component to support the rollout of the Universal Health Insurance Program record system was included to ensure training of key personnel on the forms required to enroll the population in the Family Health Record (Carpeta Familiar). 24. Project components were revised again with the Level 2 Project Restructuring of July of 2014. In particular: (i) Component 2 was revised to include activities for the renovation of the Onco-Hematologic Unit of La Paz Children’s Hospital to address the needs of children with cancer (especially leukemia) and hematological health problems. The inclusion of these activities were justified on the on the basis of their life saving and increase in survival of children, particularly the rising number of poorest children identified with these problems by the primary health services in El Alto, La Paz and other poor areas of the country; (ii) Component 3 was left to reflect only the previous expenditures corresponding to the registration of families benefitted by the current health insurance schemes before the restructuring. 25. In summary, as a result of the various rounds of restructuring, the Project eliminated most of the activities originally defined under Component 3 and assigned a greater proportion of its funds to support investment in health facility infrastructure. The failure to approve a universal health insurance law within the time of Project implementation led to a reduction in the reach of Component 3 from a comprehensive support to the implementation of a new health insurance system (SUS) to the design and implementation of a new Family Health Record System for insurance enrollment. The original Project design had envisioned investments in infrastructure for SDR 2.9 million (24.8 percent of Project funds) while Project restructuring redefined these amounts to SDR 4.295 million (62.7 percent of Project funds). 1.7 Other significant changes 26. In addition to the changes in the Project components, sub-components and activities noted above, the following changes were also made to the Project. The first Restructuring (December 2012) led to a partial credit cancellation of SDR 4.85 million (approximately 41.5 percent of the originally approved amount) and the reallocation of credit proceeds. A second Restructuring in January of 2014, extended the Project’s Closing Date (from January 31, 2014 to July 31, 2014), to allow enough time to complete a full Level 2 Project Restructuring, as requested by the Government of Bolivia in November 2013. There were no changes to PDOs, PDIs, components or activities. The third Restructuring (July of 2014) led to the change in loan closing date by 15 months and to the reallocation of loan proceeds (see Annex 1). On August 2015, a fourth Restructuring extended the closing date of the Project to December 2015 in order to complete civil works at the Onco-Hematologic Unit of the Children’s Hospital, as requested by the Government of Bolivia. This restructuring did not adjust the PDO or the PDIs. 8 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 27. The Project design considered a variety of available information sources to assess the status of and inequities in maternal and child health indicators in Bolivia. The Project outlined progress in selected maternal and child health, health service and health financing indicators across time, income groups and subnational regions. The Project also considered the organization of the health sector in the country, providing an analysis of the division of functions assigned to the different levels of government, an important element for the implementation of reform plans. Additionally, the Project design also benefited from an analysis of lessons learned from the implementation of the two previous Project phases. However, the Project design did not include an analysis of indicators related to maternal and child health, nutrition or insurance coverage, specific to the 88 locations selected for Project targeting. 28. The Project’s design was comprehensive in its support to the ongoing Health Sector Reform in Bolivia. The third phase of the APL Program was also appropriate to the long-term nature of the reform and the Bank’s continued commitment to its support. The Project design was in line with the main sector strategies defined by the Morales administration and was consistent with the goals of reducing IMR and MMR and their risk factors, increasing financial coverage for maternal and child health services and strengthening the capacity of the MS and that of departmental and local authorities. The Project was designed after the full achievement of five of the six APL II triggers, allowing for much confidence for further Project implementation. The Project components were designed to provide assistance to the implementation of the reform through a number of complimentary strategies that supported advancements in a number of specific fronts. The varied support to reform activities defined under the APL III was ambitious. The RF of the Project tried to bridge the proposed Project activities with the Development Objectives and overall programmatic objectives. The Project design, however, failed to define the critical risk factors affecting maternal and infant health, which were the basis for the first PDO, leaving much room for interpretation and definition of related activities. Given the resulting breadth of activities proposed by the Project, a number of these (such as the strengthening of information systems and measuring child growth rates) had only an indirect effect on the Project outcome indicators. 29. Risk identification and mitigation measures were described with moderate adequacy, but the extent of these risks was underestimated. Given the complexity of the Project’s design, the implementation capacity was more likely to be a substantial risk. The assessment of the risks related to implementation capacity and sustainability was rather optimistic in assessing as a moderate risk, the capacity of the MS in absorbing the responsibility for the large number of activities included in the Project design and the required changes in implementation. Similarly, the original risk assessment did not consider the possible difficulties in establishing the legal framework necessary for the implementation of the unified health insurance system, a moderate risk at the time. Finally, the risk analysis failed to take into account the complexity of the political changes that were occurring in the country and the potential impact that they could have on Project implementation. 9 2.2 Implementation 30. The Project incurred a delay in implementation. The Project became effective on June 19, 2009, and closed on December 31, 2015, 23 months later than the original date of January 31, 2014, with two extensions already noted. The Project, therefore, suffered from underutilization of funds throughout its implementation. The changes in Project scope led to a 41.5 percent cancellation of the originally approved amount and a full disbursement of the (restructured) credit amount at the time of closing. 31. Institutional and implementation arrangements were an initial bottleneck to Project implementation. The Project was the first of the three APL phases to implement activities within the MS rather than through a Project implementation unit. Delays in the establishment of organizational arrangements to lead and coordinate the implementation of Project activities made it nearly impossible to begin the implementation of the Project; for example, delays in defining a person or group within the MS to undertake procurement processes, made it impossible to begin contracting technical support and infrastructure activities. At the time of the first Level 2 restructuring of December 2012, i.e. almost five years after Project approval and three and a half years after achieving effectiveness, only 15 percent of Project resources had been disbursed out of the total original loan amount of SDR 11.7 million. Although the pace of Project implementation and Project disbursements were very slow at the beginning, the measures taken by the Bank and the Government in December 2012 allowed for a recovery in the Project’s implementation (see also the Relevance of Implementation section, below). With this restructuring, the Bank provided supervision and technical assistance to the Government in an effort to strengthen managerial capacity and fiduciary procedures to implement the Project activities. The Government resolved fiduciary and administrative bottlenecks, reforming the management structure of the implementation team in the MS, allowing a better performance and the possibility to speed up disbursements and achievements. Following this restructuring the Project performance was upgraded to moderately satisfactory and increased its disbursement from 26 percent in November 2012 to 71percent13 in July 2014. The restructuring was effective in providing a big push in both disbursement and achievement of Project indicators. 32. Factors outside the control of the GoB. The limited availability of construction firms with the required capacities and experience to undertake activities related to the refurbishment and upgrading of health facilities was outside the control of the GoB. Delays in bidding and contracting of these firms affected Project implementation of the Onco-Hematologic Unit of the La Paz Children’s Hospital by ultimately postponing the conclusion of these activities. 33. Two factors under Government control had adverse effects on Project implementation. These were: (i) changes in administration across the Government and within the MS led to delays in Project management and in the establishment of the necessary leadership to begin Project implementation; and (ii) large delays in the initial definition of a Project implementation team within the MS delayed the initiation of Project activities. 13 Both percentages were calculated in reference of the new loan amount of 6.85 million SDRs after the cancelation of 41% of the total loan amount. 10 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 34. Design. The PAD included a clear M&E framework with indicators adequately defined, baselines set and target values indicated. The original PDO indicators and intermediate indicators generally captured the essence of the Project’s development objectives. PDO indicators were also well aligned with the overall APL program objectives of reducing IMR and MMR, increasing access to financial coverage and strengthening Government capacity. Intermediate outcome indicators – related to data collected by SNIS in Component 1 (Strengthening Government Capacity); related to service delivery to women and children in Component 2 (Strengthening the Maternal and Infant Health Network); and related to insurance enrollment in Component 3 (Increase in Health Insurance) – were appropriate in measuring critical areas of Project implementation towards its objectives. While some of the indicator targets were ambitious, the significantly positive progress of the reform and its implementation through the previous two program phases, as well as the GoB’s strong commitment to the Reform, justified their selection at appraisal. In brief, PDO and intermediate outcome indicators at the design stage were adequately selected to measure progress in implementation of the Project’s components and to measure progress in achieving the PDO. 35. Implementation. While the majority of the indicators were tracked regularly for the purpose of the Implementation Status and Results Reports (ISRs), there were a number of challenges in data collection and reporting that can be summarized as follows: (i) the initial delay in establishing a team within the MS that would be dedicated to the monitoring of the Project activities made such task very difficult in the early stages of the Project; (ii) changes to Project activities required changes to Project indicators, making the tracking of progress more difficult; and (iii) problems in undertaking the national census that was to provide the sampling frame for data collection through the National Health Survey delayed the availability of health specific data for the monitoring of Project outcomes. 36. Revisions as part of the restructuring. As already noted, both PDI and intermediate outcome indicators were revised, some were dropped and some were added as part of the first and third restructurings. The first restructuring dropped two PDI, while the third restructuring changed the definition of two PDIs, dropped three and added two new PDIs. Documentations contained the details of the revised M&E framework including information on reporting frequency, data sources and responsible units and justification for revision, cancellation, progress to date and likelihood of achievement. M&E for the Project continued to be rated moderately satisfactory; even though the availability of the data (i.e. annual statistics) was not always timely, due to delays in publishing and/or dissemination of statistical reports. Taking into account the restructuring processes that the project went through, it was challenging to track the progress of indicators, their measurement strategies and targets in a consistent manner through time. Furthermore, as noted before, the first restructuring dropped the indicators related to PDOs 3 and 4, which meant that the Project was left without any definition of how to measure progress towards the fulfillment of two development objectives throughout its second phase. 37. Utilization. A large number of activities undertaken by the Project aimed to create a robust management and information system for the health sector. By the end of the Project, the Bolivian health sector had instituted software for primary health services in 72 percent of the country’s primary health facilities, had implemented a system for the monitoring of clinical statistics in over two thirds of secondary and tertiary health facilities in the country, had developed a system for the 11 management of human resources, had established a unified system for the registration of health facilities and had undertaken a beneficiary satisfaction survey whose results were shared with each facility. In addition, a financial control system in health was implemented and strengthened. The system is now fully digitized, facilitating the generation of action plans by the MS and the local authorities. The full benefits of these systems and the information they generate will largely be reaped in the coming years. 2.4 Safeguard and Fiduciary Compliance 38. The original Project triggered the Environmental Assessment and Indigenous Peoples safeguard policies. The environmental assessment was triggered due to the likely increase in production of health care waste and the impacts associated with the physical rehabilitation of health centers and other construction activities. The assessment recommended the identification of a unit internal to the MS that would be responsible for the up keeping of these laws throughout the Project’s implementation. During the third restructuring, when the activity of construction and equipment of the Onco-Hematologic Unit of La Paz Children’s Hospital was included in the Project, an additional environmental assessment was conducted and approved. The Indigenous People’s safeguard policy was triggered because the Project focused on intercultural health for the maternal and infant population, as well as the fact that most beneficiaries self-identified as autochthonous. The Project was considered an Indigenous Project and hence, no indigenous people’s plan was necessary. 39. Procurement. Procurement was rated satisfactory over the life of the Project. Post review missions and procurement audits found no serious issues with procurement processes in either of the implementing entities, and most of the delays in contracting processes for civil works and consultants by the FPS are explained by the limited availability of construction firms discussed in paragraph 32. 40. Financial Management. Financial Management was rated as moderately satisfactory over the life of the Project. Financial audits and documentation of Project financial transactions were presented to the Bank with some delay by the MS, but not by the FPS. However, major problems with financial management were not identified in any of the implementing entities (the MS and the FPS). 2.5 Post-completion Operation/Next Phase 41. The PDO continues to be relevant to the health sector of Bolivia, but no further operations are planned. 12 3. Assessment of Outcomes (see Table 4 in Section 3.4 for a summary of the ratings) 3.1 Relevance of Objectives, Design and Implementation Overall Relevance Rating: Phase 1 – Modest; Phase 2 – Modest; Phase 3 – Substantial 42. Relevance of the Project’s objectives: High in all three phases of the Project. Throughout the three phases of the Project, its objectives were in full support of the country’s reform objectives, health sector challenges, and identified government priorities. Infant and maternal mortality, malnutrition and access to appropriate health care services were and still remain key development challenges. The Project was also consistent with the Interim Strategy Note developed in 2006 between the GoB and the Bank (Report No. 36095-BO), where it was agreed that the Bank and the Country would pursue projects that supported enhancing good governance and transparency, fostering jobs through growth and providing better services to the poor. Finally, the targeting strategy defined under the Project was in line with the Government’s commitment to improving the health and availability of quality health services for the poor and allowed for close coordination with the Bono Juana Azurduy and the Zero Malnutrition Program strategies. 43. Relevance of the Project’s design: at the time of appraisal, the relevance of design is rated as Substantial. The Project was originally designed to support key elements of the overall Health Sector Reform: the improvement of service delivery and an increase in financial protection (through the expansion of health insurance) for women and children especially those in underprivileged locations in Bolivia. The Project aimed to achieve these goals, by undertaking a variety of activities that would tackle bottlenecks to the efficient functioning of the Bolivian health system. Increasing capacity in the national, regional and local health institutions would improve the overall government ability to perform the EFPH, especially during the process of political decentralization that was ongoing at the time of Project design. Efforts to improve maternal and infant referral networks directly tackled system inefficiencies in service delivery and efforts to increase health insurance coverage were meant to tackle inefficiencies caused by low levels of financial pooling. Despite this logical consistency with the objectives, the initial design proposed the implementation of too many activities. 44. The relevance of the Project design after the first restructuring, i.e. during its second phase, is rated as Modest. The cancellation of a large number of activities that had originally been defined to support the achievement of the PDOs weakened the causal relationship between Project activities and its objectives. Namely, the cancellation of Project activities in support of the expansion of health insurance and those related to the quality of delivery of malnutrition care for children (increase and improvement of human resources, for example) further undermined the Project’s direct responsibility for the achievement of the second and third PDOs. Nonetheless, remaining and newly included Project activities, in the second phase of the Project, were more closely related to one another as well as newly defined Project indicators, providing for a more targeted approach to Project implementation and the achievement of its objectives. 45. The relevance of the Project design during its third phase is again rated as Modest. The inclusion of activities in support of the construction and refurbishment of the Onco- hematologic unit of La Paz’s Children’s Hospital further weakened the causal pathway between 13 Project activities and its development objectives. However, all other activities continued to support the achievement of defined development indicator targets. 46. Overall, the Project interventions were, to a large extent, related to institutional capacity building efforts and were relevant in supporting the reform process. Component 1 institutional capacity building efforts focused particularly on health system stewardship: the strengthening of information systems; the strengthening of regulatory processes, management analysis and communication; and the strengthening of human resources for the management of referral networks. Component 2 supported the development of institutional capacity for maternal and child health referral networks and communities, mainly by improving analytical capcity and training local health managers. Component 3 aimed at improving institutional capacity by strengthening M&E information systems. The cancellation of a number of activities during the first Project restructuring did not reduce the Project’s emphasis on institutional capacity building; this, in fact, remained a major emphasis of Project activities.. 47. Relevance of the Project’s implementation: Modest in Phases 1 and 2 and Substantial in Phase 3. The relevance of the Project’s implementation throughout the first p hase of the Project is rated as Modest, as delays in the organization of the appropriate support structure for the units of the MS, defined in the institutional arrangements of the Project, as well as continuing issues with the definition of the responsible units within the MS for Project implementation, led to several negative (Unsatisfactory and Moderately Unsatisfactory) IP ratings from Project effectiveness until the restructuring of December 10, 2012. After the first Project restructuring (Phase 2), the relevance of the Project’s implementation continues to be rated as Modest: at this time, the Bank and MS implementation teams worked together to identify implementation challenges and redefine the Project scope within the new reform and political circumstances. The Bank team changed the Project operational approach, while the MS introduced changes to the Project’s management structure. However, implementation progress towards the achievement of the health insurance development objective continued to stall. There were difficulties in contracting of firms to complete facility infrastructure construction and refurbishment activities during the second phase that led to continued delays in Project implementation. The recovery in implementation capacity and increases in the pace of disbursement led to the upgrade of Project implementation ratings to Substantial in the third phase of the Project: during this last period, the Project concluded its activities and was additionally able to introduce and conclude the construction of the Onco- hematologic unit of La Paz’s Children’s Hospital. 3.2 Achievement of Project Development Objectives (see also section (d) of Annex 2 for more detailed information and tables comparing revised PDIs) Overall Efficacy Rating: Phase 1 – Modest; Phase 2 – Substantial; Phase 3 – Substantial 48. PDO 1 (to reduce in the gap of critical maternal and infant health risk factors between the people living in the municipalities prioritized by maternal and infant health interventions and the rest of the Bolivian population). This PDO is rated as High for the first and second phases of the Project, and Substantial for the third phase. The Project’s coordinated efforts to increase facility adherence to quality of care norms and regulations, as well as its investments in facility infrastructure and equipment throughout the target areas led to a near achievement of the targets set for this PDO. This PDO had two indicators associated with it: (i) the ratio between the 14 percentage of pregnant women accessing four prenatal care visits in the areas of Project intervention and the rest of the country and (ii) the ratio between the percentage of institutional deliveries in the areas of Project intervention and the rest of the country. There was a high level of achievement of this objective throughout the Project, the data available at the end of the Project indicate that it achieved a ratio of 0.89 for the first indicator (105 and 94 percent fulfillment of its original and revised targets, respectively) and a ratio of 0.98 for the second indicator (155 and 110 percent fulfillment of its original and revised targets, respectively). These are the two PDO indicators that least changed by the restructurings. Furthermore, the relationship between the PDIs and the PDO is direct. The target of the second PDO indicator was revised upward in July, 2014, but to a level that had already been achieved at the time. Despite this, the fact remains that the Project goals for this indicator were achieved. See also Annex 10 for additional data and analysis. Rating PDO 1: High. 49. Key Project outputs and inputs that supported the achievement of these objectives (results chain) were: (i) diagnosis for quality assurance: 80 percent of facilities in target areas initiated a self-evaluation process based on norms; 16 facilities were accredited; 34 facilities are in the process of accreditation; 240 action plans were developed; 135 health workers were trained in quality management methods; (ii) diagnostics of the internal capacity of networks: development of unified system for the registration of health facilities; identification and equipment of 24 facilities in 8 municipalities in Beni and La Paz departments; georeferenced database of health facilities and networks at national level; (iii) equipment and renovation of health facilities: 16 rural health facilities in La Paz equipped; 8 health facilities in Beni equipped; 74 equipment and infrastructure projects defined; 16 equipment and infrastructure projects approved by department health services; 49 health facilities fully equipped and functioning. 50. PDO 2 (to reduce chronic malnutrition among children younger than 2 years of age in the rural areas of the 82 municipalities targeted by the Project). This PDO is rated as Modest throughout the three phases of the Project. The Project’s investment in facility infrastructure and equipment together with its close collaboration with the “Bono Juana Azurduy” Project and the Program for Zero Malnutrition led to the final achievement of the set targets for this objective. The indicator associated with PDO 2 was defined as the percentage of 2-year-old children covered with height growth control (CRED) in the intervention areas of the Project. At the time of Project completion the Project had achieved 89 percent coverage (222 percent achievement of the target). At the time of the July, 2014 restructuring, the two PDIs previously defined for tracking the achievement of this objective were dropped to avoid duplication, as they were being collected for the “Bono Juana Azurduy.” The two PDIs were replaced by one indicator whose relationship with the measurement of the goal of reducing malnutrition is indirect and the target defined was already nearly achieved (98 percent) at the time of the restructuring. As a result, the target for this indicator was substantially surpassed at the end of the Project. The data presented in Annex 10 shows that the region of the Valleys was the only one to experience declines in weight-for-age malnutrition between 2007 and 2013. Rating PDO2: Modest. 51. Key Project outputs and inputs that supported the achievement of these objectives (results chain): (i) equipment and renovation of health facilities: 16 rural health facilities in La Paz equipped; 8 health facilities in Beni equipped; 74 equipment and infrastructure Projects defined; 16 equipment and infrastructure Projects approved by department health services; 49 15 health facilities fully equipped and functioning; (ii) implementation of the system for community nutrition surveillance: committees for community surveillance formed in intervention target areas. 52. PDO 3 (overall increase in health insurance coverage in the target areas). This PDO is rated as Negligible to Low for the first phase of the Project and Substantial thereafter. The Project’s investments to develop and roll out the Family Registry Forms contributed to the achievement of the indicator associated with this objective: percentage of target population enrolled in Family Registry Forms (Carpeta Familiar). Due to the lack of progress in the implementation of the health insurance program during the first phase of the Project, the achievement of this PDO is rated as Negligible to Low. The first Project restructuring in 2012 led to advancements in the achievement of this PDO by adjusting to the circumstances and emphasizing efforts to enroll the target population in the Family Registry Forms. As the original indicator was dropped and not replaced by any other in this first restructuring, the retrofitting to this phase of the new indicator introduced by the restructuring of July 2014, shows a 67 percent achievement (see the table in Section E of Annex 2. Even though a new and lower target was set in the restructuring of July of 2014 to reflect the delayed start of Project implementation, the Project achieved its original goal. At the time of Project completion, the Project had achieved an 80 percent enrollment rate in the target areas, surpassing the target of 60 percent (100 and 133 percent achievement of original and revised targets, respectively). However, this new PDO indicator, does not measure health insurance coverage in itself, but rather reflects registration in a system that is to inform a health insurance system, when implemented. See also Annex 10 for additional data and analysis. Rating PDO3: Substantial. 53. Key Project outputs and inputs supported the achievement of these objectives (results chain). These are: (i) training for the financial administration of public health insurance systems: 337 municipalities received training on SUMI; 104 municipalities received training on Insurance for elderly populations; (ii) agreements between municipal governments and MS: for the use of 15 percent of municipal government funds for the financing of the SUS; (iii) development of instrument and tools for the health insurance registration system ( Carpeta Familiar), developed and implemented in 9 Department Health Services. 54. PDO 4 (the SNIS incorporates a new information strategy and production information in the intervention areas is generated from the health insurance management module). This PDO rating increased from Negligible to Low for the first phase, to Substantial in the second phase, and finally to High for the third phase of the Project. The non-implementation of the health insurance program explains the Negligible to Low rating for this PDO in the first phase. However, large efforts and investments in the improvement of Bolivia’s health information systems led to the achievement of this objective during the second and third phases of Project implementation. The indicator for this objective was defined as “the modules included in the SNIS are fully implemented in the SEDES and in all the heads of the referral netw orks in the intervention areas.” At the time of Project completion the modules were implemented in the SEDES and all heads of the referral networks, thus achieving a 100 percent completion or 100 and 167 percent achievement of the original and revised targets, respectively, as the target value was reduced during the third restructuring. Rating PDO4: Substantial.14 14 It should be noted that this project outcome (generation of an information strategy and production data) should not be considered as a development outcome, per se, but rather as an intermediate outcome. 16 55. Key Project outputs and inputs supported the achievement of the Project’s objectives (results chain). These were: (i) development of new information systems: software for primary care developed and implemented in 2154 facilities; clinical and statistical information system developed and implemented in 80 secondary and tertiary public hospitals; software for vital registration developed and implemented in nine Department Health Services; software for human resources for health developed and implemented in nine Department Health Services; development of instrument and tools for the health insurance registration system (Carpeta Familiar), developed and implemented in nine Department Health Services. Table 2: Results Framework Summary PDO indicators Phase 1 Phase 2 Phase 3 (2009-2012) (2012-2014) (2014-2015) Indicators on Original Indicators December indicators15 (Feb 2014)16 201517 Surpassed 2 2 4 Achieved 0 0 0 Partially achieved 1 1 1 Not achieved 3 1 0 Total 6 4 5 Total surpassed, achieved and partially 3 (MS) 3 (S) 5 (HS) achieved 50% 75% 100% IO indicators Surpassed 0 1 3 Achieved 1 1 2 Partially achieved 0 4 0 Not achieved 11 6 0 Total 12 12 5 Total surpassed, achieved and partially 1 (HU) 6 (MS) 5 (HS) achieved 8% 50% (100%) Weighted Average(2) 45.8% (MU) 72.5% (S) 100% (HS) Notes: (1) HU:0-14%, U:15-29%, MU:30-49%, MS: 50-69%, S:70-84%, HS:85-100%. (2) Weighting Criteria for Average rating: PDO indicators (90%) + IO (10%) 56. Overarching/Program development objectives. As noted in Section 1.2, the APL series defined, at its onset, a set of longer term targets to be achieved at the end of the three APLs. The data available indicates that the infant and child (under-5) mortality rate goals have been surpassed while the goal for the reduction in maternal mortality was partially achieved (84 percent) if data from the Pan-American Health Organization is used, however, estimates published by the Government in 2016 provide estimates MMR of 160 per 100,000 live births, which would lead to an achievement of 102.5%: It must be noted, however, that the methodology used for these 15 ISR Seq 12 – Feb 2013 16 ISR Seq 15 – May 2-14 17 Last ISR 17 estimates relies on the verbal recovery of information in 2014-2015 of deaths occurred in 2011 and in data of 2000, 2003 and 2012. Table 3: APL Program Series Overarching Development Objectives Target at Baseline Program Development Objective End of APL Result (Year) (1999) Series 1 Infant mortality rate (per 1,000 live births) 67 48 32.80 (2013) Under-5 mortality rate (per 1,000 live 92 57 44.90 (2011) births)2 Maternal mortality rate (per 100,000 live 390 164 200/160* births)3 (2013/2011*) Notes: 1Estimated value; 2Estimated value; 3Estimated value. Estimated range: 130-310 per 100,000 live births. Source (results data): Pan-American Health Organization; *Ministerio de Salud, Estudio Nacional de Mortalidad Materna (2016). 57. Attribution. The fact that a logical causal relationship can be established between the PDIs and the Project’s interventions implies that, conceptually, the development indicators achieved can be conceived as related to the Bank’s actions, even though direct attribution of PDOs cannot be asserted due to the lack of an impact evaluation. Furthermore, considering that the majority of the Project’s activities were centered on capacity development and infrastructure, and the multi- sectoral nature of the determinants of infant, child and maternal mortality (recognized in the PADs) it is difficult to credit to the Bank’s intervention for the substantive reductions in mortality observed.18 3.3 Efficiency Efficiency rating: Phase 1 – Modest; Phases 2 and 3: Substantial 58. Cost Effectiveness. The economic and financial analysis presented in Annex 3 replicates the cost-benefit analysis conducted during Project preparation to present an estimated impact of the Project’s implementation. The Project’s cost-benefit analysis is based on the Project’s overall costs and its measurable economic benefits. The Project generated social and economic benefits through its impact on childhood and maternal mortality, as well as on nutrition. Due to limited available data, it has been impossible to attribute all changes in mortality and nutrition status in the Project target areas to the Project’s activities. The analysis includes all Project costs but considers only the benefits directly derived from the implementation of activities within Component 2: “Family, Community and Intercultural Health” of the Project. The Project’s internal rate of return (IRR) would be higher if benefits resulting from the other Project components had been included in the analysis. 18 Two other related projects were, in fact, implemented by the Government in the selected municipalities at the time of the APL III implementation: the “Bono Juana Azurduy” and the Program for Zero Malnutrition. The “ Bono Juana Azurduy” promoted the use of maternal and child health services of the target population wh ile the Program for Zero Malnutrition used multi-sectoral approaches (including facility level interventions) to reduce causes of malnutrition. These two projects and the APL III were originally designed to complement each other and hence, any attribution of achievement of outcomes ought to include the three projects in tandem. 18 59. As per justification in the original PAD, the economic analysis for the Project consists of an evaluation of 3 Project outcomes: (i) infant mortality, (ii) maternal mortality, (iii) infant malnutrition. The Project’s benefits were estimated in terms of projected income flows for averted deaths and malnutrition and the Project costs included the total Project disbursements and the foreseen costs of educating children whose lives were saved by the Project. The analysis estimates an IRR of 10 percent at 30 years after the start of the Project. The breakeven point was seen to occur 21 years after the Project was approved. 60. Implementation efficiency. As already noted, the Project showed a slow disbursement (15 percent) during the first five years after its approval (leading to a rating of Modest), until changes to the Project’s operational approach and to its management structure were introduced by the Bank and the Government in the first Project restructuring (leading to a rating of Substantial efficiency thereafter). 3.4 Justification of Overall Outcome Rating Overall Outcome Rating: Phase 1 – Moderately Unsatisfactory; Phase 2 – Moderately Satisfactory; Phase 3 – Satisfactory 61. The Project’s overall outcome rating is Moderately Satisfactory. The Project has introduced important and positive changes to Bolivia’s health sector, thus contributing to the Government’s efforts to reform the sec tor. The overall Relevance rating was Substantial, in phases one and three of the Project and Modest in the second phase, as the changes in components diluted the relationship between activities and Project goals. It should be noted, however, that infant and maternal mortality, malnutrition and access to appropriate health care services were and still remain key development challenges and the Project’s goals were aligned with the Country’s and the Bank’s priorities and strategies. Furthermore, despite the implementation problems observed with the Project, the Bank and the MS teams have acted to introduce the necessary changes to the Project to allow for the recovery of its implementation and the progress towards the achievement of its objectives. The overall Efficacy rating was Modest during the first phase of the Project (before the first restructuring) but given improvements in progress towards Project objectives, it was rated Substantial thereafter. It has to be noted that while four of the five PDO indicator targets were finally achieved, the high rates of overachievement for several indicators, suggest that targets may have been underestimated, as the progress towards the achievement of these indicators was well under way when baselines and targets of PDO indicators were revised for the restructuring of July of 2014 (at the time, the Project closing date was set to October of 2015). It is for these reasons that the Overall Efficacy ratings given for each of the three Project phases are slightly below the ratings related to the achievement of PDOs produced by the numerical calculations performed in Table 2. The fact that a logical causal relationship can be established between the PDIs and the Project’s interventions implies that, at least conceptually, the development indicators achieved can be conceived as related to the Bank’s actions, but it is difficult to credit to the Bank’s intervention the substantive reductions in mortality observed. Finally, Efficiency received a Modest rating during the first phase of the Project mainly due to the slow improvements in implementation efficiency but the rating increased to Substantial during phase two and three of the Project, as great improvements in Project execution were observed. 19 Table 4: Overall Outcome Ratings Phase 1 (2009-2012) Phase 2 (2012-2014) Phase 3 (2014-2015) Relevance Objectives High High High Design Substantial Modest Modest Implementation Modest Modest Substantial Overall Relevance Substantial Modest Substantial Efficacy PDO1 High High Substantial PDO2 Modest Modest Modest PDO3 Negligible to Low Substantial Substantial PDO4 Negligible to Low Substantial High Overall Efficacy Modest Substantial Substantial Efficiency Modest Substantial Substantial Moderately Outcome Rating Unsatisfactory Moderately Satisfactory Satisfactory Rating value 3 4 5 % disbursed 26% 35% 39% Weighted value 0.78 1.40 1.90 Cumulative 0.78 2.18 4.08 (rounding to 4) 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 62. The Project’s goals related to infant, child and maternal mortality and financial protection implies a substantial social impact. The risks and determinants associated with these objectives disproportionately impact the poor and vulnerable. The increasing number of poor children identified with cancer and hematological problems in several of the poorest areas of the country also justified the poverty impact of the activities related to the renovation of the Onco- Hematologic Unit of La Paz Children's Hospital. (b) Institutional Change/Strengthening 63. The Project positively contributed to the strengthening of the MS, of the Social Investment and Productive Fund (Fondo Nacional de Inversión Productiva y Social, FPS) and of local governments. The Project not only helped develop and implement a number of information systems for the improved management and decision making in these institutions but also financed the training of nearly 600 administrative personnel of the MS on aspects of public financial management, over 100 managers on health service provision management and over 130 personnel on quality assurance management. (c) Other Unintended Outcomes and Impacts 64. The Project had one important unintended outcome. Although the Project focused primarily in the strengthening of health service delivery systems, its efforts to improve quality and 20 to include protocols for service delivery that targeted primarily indigenous peoples, it generated a growth in the demand for health services as well as an increased community and neighborhood involvement in the management and demand for quality of health services. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 65. A satisfaction survey was conducted in 2014. From a total of 58 health facilities targeted by the Project’s activities, the satisfaction rates were: 69 percent in Cochabamba, 78 percent in La Paz, 81 percent in Tarija, and 78 percent in Oruro. 4. Assessment of Risk to Development Outcome 66. There is a Moderate risk that Project development outcomes will not be maintained. Key reforms are currently preserved in a solid legal framework by law and therefore unlikely to be reversed. The fact that the Project implementation was directly conducted by the MS adds to its sustainability, furthermore, all Project activities were fully integrated into the programs and responsibilities of the MS. Finally, the risks to development outcome are mitigated by the fact that the Project’s goals continue to be a priority to the country and any government. However, the Projects’ investment in facility infrastructure will require continued local and National Government investment for maintenance: this investment is at risk from a slowing economic growth and decreased hydrocarbon revenues. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 67. The main strengths in ensuring quality at entry were three: (i) the programmatic nature of the Project; (ii) the APL design that supported Bolivia’s health system reform; and (iii) a clear rationale for Bank’s involvement. Planned interventions were framed within the ongoing and previously supported Health Reform Program and APL projects and focused on ensuring continued support to the implementation of reform efforts. Activities were selected to tackle the remaining key bottlenecks to the continued implementation of the Reform and achievement of its objectives. The Bank was able to contribute to the ongoing efforts not only financially but also by bringing its international expertise in the support of health reforms and institutional strengthening. 68. Shortcomings: The activities envisaged at Project design covered a broad spectrum of interventions that required strong implementation capacity and leadership. The Project aimed to cover a large number of needs ranging from system governance, service delivery to financing in order to improve specific outcomes related to maternal and infant health. While the definition of each sub-component was detailed and had a visible connection with the results, many of the activities planned under the Project would only have an indirect effect on outcomes. Risk assessment and mitigation plans did not envision the difficulties of establishing a Project 21 coordination team within the MS, the difficulty of implementation of the diverse set of activities, and the difficulties that would arise in the implementation of the health insurance component. (b) Quality of Supervision Rating: Moderately Satisfactory 69. Bank’s performance during Project implementation is rated Moderately Satisfactory. Task teams focused on the Project’s development impact throughout implementation. A large budget and staff resources, including fiduciary and safeguards were allocated and the Project was adequately supervised, closely monitored, and reported on. However, shortcomings in M&E supervision and the long delay in restructuring of the Project’s activities undermined Project implementation, the measurement of its achievements and delayed its completion. 70. The Bank teams engaged continuously and proactively with the MS and the FPS, the agency responsible for the implementation of civil works, on a wide variety of operational and technical issues at all stages of the Project. Three Bank staff assumed TTL responsibility during Project implementation with handover missions. The transition was smooth as the local specialists continued providing daily support throughout the duration of the Project. The supervision activities were performed regularly and substantively. Aide-memoires and ISRs were systematic in their focus on the overall program and Project implementation. They were of good quality, and performance ratings were evidence-based. The Bank teams provided timely and precise support to the implementing units in an environment where the decision-making processes of the MS were slowed down due to organizational restructuring and frequent changes in managerial levels. 71. However, although the Bank management and team responded to the changing needs of the government through Project restructuring, this did not originally happen on a timely and practical manner. The first Project restructuring took place after three years of Project implementation and two years of continuous unsatisfactory Project ratings. Although the three subsequent Project restructurings clearly responded to changing government and Project circumstances, the number and frequency of these restructurings could also suggest the Bank’s difficulty in foreseeing further changes in Project implementation. Additionally, some shortcomings on M&E supervision are noted. With the two initial Project restructurings, there were a number of changes to indicator definitions that were intended for more precise measures but limited the possibility of tracking improvements over the entire duration of the Project. (c) Justification of Rating for Overall Bank Performance 72. The overall Bank performance is judged to be Moderately Satisfactory on the basis of a rating of Moderately Satisfactory for quality at entry and Moderately Satisfactory for supervision. 22 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 73. With regards to the institutional framework, the inability of the Government to pass the law necessary for the establishment of the unified health insurance system was a major drawback. This affected Project implementation resulting in the restructuring of the Project and the cancellation of a large proportion of activities. Additionally, the Government was slow in assigning the appropriate number of skilled personnel necessary to conduct the Project’s management and implementation. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 74. The performance of the MS and of the FPS as implementing agencies is rated moderately satisfactory. The MS commitment to the health reform and ownership of the Project was high only after the Project implementation team was defined. The Project’s initial delays in implementation can be attributed to frequent changes in leadership in the Ministry and the consequent difficulties in establishing an implementation team. These delays were the main causal factor for delays in implementation, the need for extensions to the Project closing dates and partially contributing to the high Project supervision expenditures. Nonetheless, given the limited Project implementation budget that had to be stretched over a greater number of years than originally planned, the MS and the FPS were able to undertake a large number of diverse activities and together with the Bank have acted to introduce the necessary changes to the Project to allow for the recovery of its implementation and eventually achieve Project targets. It should also be noted that the FPS managed to execute a substantial proportion of Project resources, as these were reassigned to infrastructure investments from the other components after the two restructurings. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 75. Given the challenges presented above, the overall borrower performance is rated as moderately satisfactory. 6. Lessons Learned 76. Four important lessons were learned from the implementation of this Project: (i) A project’s design should not rely on the promise of the passing of a law necessary for the implementation of project activities. The APL III was designed to implement activities that would increase insurance coverage under a new and unified health insurance scheme. The insurance scheme could not be implemented because the law underlying its creation was not passed during the time of Project implementation. Foreseeing this risk could have allowed for the definition of Project activities in support of the passing of the law or in support of other financial protection coverage strategies; 23 (ii) Carefully estimating times required to complete transitions is essential to project design and implementation. The APL III was designed within a framework of project implementation defined by the previous two phases. At the time of Project onset, however, the government administration had undergone dramatic changes (changes in ideals, priorities and leadership) that led to initially frequent changes in the MS and subsequently influenced Project ownership. Similarly, the Project had envisioned a rapid transition from an implementation model based on an externally managed Project implementation unit, to a team that was fully incorporated in the Government’s administration, the MS. These transitions took much more time than originally envisioned and were a bottleneck for Project implementation. Efforts to mitigate these risks to implementation at Project design could have allowed for a timely implementation; (iii) The careful management of the political economy in decentralized health systems is essential to the implementation of Project activities at the local level. The Bank, the MS and FPS all made important efforts to work with local partners, understanding their needs, capacity and times, to ensure the implementation of Project activities; and (iv) The implementation of information systems requires efforts in two separate fronts: in the collaborative design of the systems and in the creation of the demand for the information generated by the systems. In its support of the design and implementation of the information systems, the Project made great efforts to ensure that systems were designed to collect data that was valuable to managers and policy makers by conducting careful analyses of needs. The Project also made sure to build the demand for the use of newly available data by conducting manager and decision-maker training workshops. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 77. Most of the comments received from the MS and all the comments received from the FPS have been incorporated. The observations received did not lead to a change in overall rating as requested by the MS and/or in conclusions. See Annex 7 for the full comments and the comments/responses from the ICR team (in Spanish). (b) Cofinanciers The Project was not co-financed. (c) Other partners and stakeholders No other partners and stakeholders were consulted. 24 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (SDR (SDR millions) Appraisal millions) Component 1: Stewardship Role of 2.50 0.91 36.4% Health Authorities Component 2: Family, Community 6.30 5.17 82.1% and Intercultural Health Component 3: Health Insurance 2.00 0.35 1.75% Project Administration 0.60 0.40 66.7% Unallocated 0.30 0.00 0.0% Total Baseline Cost 11.70 6.83 58.4% Physical Contingencies 0.00 0.00 0.00 Price Contingencies 0.00 0.00 0.00 Total Project Costs 0.00 0.00 Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 11.7 6.83 58.4% (b) Financing Appraisal Actual/Latest Type of Percentage of Source of Funds Estimate Estimate Cofinancing Appraisal (USD millions) (USD millions) Borrower 6.00 0.00 .00 International Development Association 10.46 0.00 .00 (IDA) (c) Reallocation of Loan Proceeds Expenditure Category Original Revised: 1st Revised: 3rd Amount of Restructuring Restructuring the Loan Amount of Amount of Allocated the Loan the Loan Original Revised: 1st Revised: 3rd (Expressed Allocated Allocated Restructuring Restructuring in SDR) (Expressed in (Expressed in And % SDR) SDR) And % And % Component 1 – Component 1 – Component 1 – 2,500,000 1,510,000 1,185,621.67 Goods, consultant Goods, consultant Goods, consultant 21.4% 22.0% 17.3% services, Training services, Training and services, Training and MS Operating MS Operating Costs and MS Operating Costs Costs 25 Component 2 - Component 2 - Component 2 - (a) Goods, (a) Goods, consultant (a) Goods, 3,200,000 400,000 400,000 consultant services, training consultant 27.4% 5.8% 5.8% services, and MS Operating services, training and Costs for training and MS Operating Component 2 of MS Operating Costs for Part the Project except Costs for 2 of the for Component Component 2 Project except 2(i) (b); of the Project for Part 2(a) (b) Works, goods and (b) Works, goods (ii); consultant and consultant 2,900,000 4,100,000 4,294,627 (b) Works, goods services services 24.8% 59.9% 62.7% and (including audits) (including consultant under Component audits) under services 2(i) (b) of the Component 2 (including Project and; (a) of the audits) under Project and; Part 2(a) (ii) (c) FPS Operating (c) FPS Operating of Project Costs under Costs under 200,000 200,000 394,627 and; Component 2 Component 2 1.7% 2.9% 5.8% (c) FPS Operating Costs under Part 2 of the Project Component 3 – Component 3 – Component 3 – 2,000,000 127,761 127,761 Goods, Consultant Goods, Consultant Goods, Consultant 17.1% 1.9% 1.9% Services, Training Services, Training and Services, Training and MS Operating MS Operating Costs and MS Operating Costs for Part 3 of for Component 3 of Costs for the Project the Project Component 3 of the Project Project Project Management Project 600,000 512,239 447,363.33 Management – – Goods, Consultant Management – 5.1% 7.5% 6.5% Goods, Consultant Services (including Goods, Consultant Services audits) and MS Services (including (including audits) Operating Costs for audits) and MS and MS Operating Project Management Operating Costs for Costs for Part 4 of Project the Project Management Unallocated 300,000 0 0 2.5% 0.0% 0.0% Total Amount 11,700,000 6,850,000 6,850,000 26 Annex 2. Outputs by Component (a) Original Project Components The APL III originally had four main components as described below: Component 1: Stewardship Role of Health Authorities – Essential Functions in Public Health (US$7.7 million: US$4 million IDA + US$3.7 million Government): this component aimed at strengthening the capacity of national, regional and local health authorities to effectively perform the critical EFPH. This component included three sub-components: (1A) Strengthening the National Health Information System (Sistema Nacional de Información en Salud, SNIS): support to the areas of data gathering, analysis, reporting, monitoring, supervision, evaluation, management and research; (1B) Regulation, results-based management and culture of accountability: to strengthen the capacity of the MS and of the departmental and local health authorities in order to apply modern management methodologies and instruments, and to design priority-based programming; and (1C) Development and implementation of a national program of quality: support the process of licensing, certification and monitoring of Bolivia’s health facilities to assure the delivery of quality health care services. Component 2: Family, Community and Intercultural Health (US$11.6 million: US$9.9 million IDA + US$1.7 million Government): this component aimed at improving access to maternal and infant health services in the Project’s target areas through two sub-components: (2A) Development and strengthening of intercultural maternal and infant health referral network: (i) analyze and determine the capacities of the networks in the target areas; (ii) renovate health facilities, public housing and purchase of medical equipment; (iii) expand human resources; (iv) develop programs of continuing education; (v) conduct operational research of disease outbreaks and suspected outbreaks; and (vi) create regional maintenance centers; and (2B) Strengthening of local management and community participation: to strengthen community participation in the management of local health activities, including promotion of health lifestyles, and the demand for maternal and child health services. Component 3: Health Insurance (US$4.2 million: US$3.2 million IDA + US$1 million Government): to support the Government of Bolivia’s implementation of the new “SU SALUD” health insurance program. This component consisted of three sub-components: (3A) Strengthening of the SU SALUD enrollment system: to support the rollout of a national SU SALUD record system for enrollment of families and individuals, permitting the classification of members and coverage according to geographic and demographic characteristics; 27 (3B) Development and strengthening of monitoring and evaluation (M&E) management capacity in the National Unit for Technical and Financial Management (Unidad de Gestión Técnica y Financiera Nacional, UGTFN) and Departmental Units (UGTFDs): to strengthen the capacity to plan, manage, and carry out M&E of SU SALUD at the national, departmental and municipal levels; and (3C) Development of a M&E system for SU SALUD: to support the processes of evaluating SU SALUD’s public policies, transparency, and accountability as well as an analysis of SU SALUD’s local and departmental effectiveness. Component 4: Project Administration (US$2.2 million: US$0.9 million IDA + US$1.3 million Government): to support Project administration with equipment, technical assistance, training and operating costs to finance the administration of the Project, and financial and procurement audits. 28 (b) Outputs by Project Component Table 1: Outputs by Project Component Sub-Component Planned outputs at Appraisal Actual outputs/outcomes at ICR A Component 1: Stewardship Role of Health Authorities – Essential Functions in Public Health: to strengthen the capacity of national, regional and local health authorities so all can effectively perform the critical EFPH. (Estimated cost at appraisal: SDR$2.5M; Final cost: SDR$0.91M) A.1 Strengthening the Support the areas of data gathering, analysis, reporting, Development of new information systems: National Health monitoring, supervision, evaluation, management and x Software for primary care (SOAPS): developed and Information System research. The SNIS would be enhanced to support the new implemented in 2154 facilities. (Sistema Nacional de Ministry of Health and Sports priorities (Health x Clinical and Statistical Information System (SICE): Información en Salud Insurance; Zero Malnutrition Program; Family, developed and implemented in 80 secondary and tertiary - SNIS). Community and Intercultural Health program) with public hospitals. appropriate data analysis and reporting. x Software for vital registration (SIAHV): developed and implemented in 9 Department Health Services. x Software for human resources for health (SOREHH): developed and implemented in 9 Department Health Services. x Development of instrument and tools for the health insurance registration system (Carpeta Familiar): developed and implemented in 9 Department Health Services. A.2 Regulation, Results- Strengthen the capacity of the Ministry of Health and Implementation of the Demographic and health survey: based Management Sports and of the departmental and local health x Three departmental level analysis documents of 2008 and Culture of authorities in order to apply modern management survey. Accountability. methodologies and instruments, and to design Capacity building for high and medium level central, departmental programming based on priority and goal selection. and municipal level government staff: x On Law 1178: 597 persons trained x On health management: 115 persons trained Strengthening of international and multilateral donors and the generation of a coordinated planning process: x Mapping of Donors x Reports on financial management accountability of the Ministry of Health, 2010 Human resources Management: x System for the registration of human resources: developed and implemented in 9 Department Health Services. 29 Sub-Component Planned outputs at Appraisal Actual outputs/outcomes at ICR x Studies of Human Resources for Health in Tertiary level Hospitals: 3 studies conducted Support to the development of the National Health Summit: x Departmental Summits: La Paz, Chuquisaca and Beni A.3 Development and Support the process of licensing, certification and Diagnosis of the management of departmental units for quality Implementation of a monitoring of Bolivia’s health facilities to assure the assurance: National Program of delivery of quality health care services. Institutions x 308 facilities (80 percent) in target areas initiated a self- Quality. covered included hospitals, ambulatory health centers, evaluation process based on norms. clinical laboratories, blood banks, clinics and o 16 facilities were accredited rehabilitative services, and other health service units. o 34 facilities are in the process of accreditation o 240 action plans were developed o 135 health workers were trained in quality management methods B Component 2: Family, Community and Intercultural: to improve access to maternal and infant health services in the Project’s target areas. (Estimated cost at appraisal: SDR$6.3M; Final cost: SDR$5.17) B.1 Development and Strengthen networks by (i) analyzing and determining Diagnostics of the internal capacity of networks: Strengthening of capacities of the networks in the target areas, (ii) x Development of unified system for the registration of Intercultural Maternal renovating health facilities, public housing and medical health facilities and Infant Health equipment, (iii) expanding human resources of the x Identification and equipment of 24 facilities in 8 Referral Network. networks, (iv) developing a program of continuing municipalities in Beni and La Paz departments. education for each referral network, (v) conducting x Georeferenced database of health facilities and networks at operational research of outbreaks and suspected national level. outbreaks, (vi) creating regional maintenance centers. Legal framework for the establishment of unified system of facility registration: x Framework for the definition of primary care facilities: implemented and training materials developed x Framework for the definition of secondary level Hospitals: implemented and training materials developed x Framework for the definition of referral and counter- referral systems: implemented and training materials developed x Instruments for the registration of traditional medicine providers developed and implemented Equipment and renovation of health facilities: x 16 Rural health facilities in La Paz equipped x 8 health facilities in Beni equipped x 74 equipment and infrastructure Projects defined 30 Sub-Component Planned outputs at Appraisal Actual outputs/outcomes at ICR x 16 equipment and infrastructure Projects approved by department health services x 49 health facilities fully equipped and functioning x Onco-hematological unit 50% constructed and equipment items have been purchased or solicited B.2 Strengthening of Strengthen community participation in the management Implementation of the system for community nutrition surveillance Local Management of local health activities, including promotion of health (SVIN): and Community lifestyles, and the demand for maternal and child health x Committees for community surveillance formed in Participation. services. intervention target areas. C Component 3: Health Insurance: to support the Government of Bolivia’s implementation of the new SU Salud health insurance program. (Estimated cost at appraisal: SDR$2.0M; Final cost: SDR$0.35M) C.1 Strengthening of the Support the rollout of a national SU SALUD record Reports generated by the information system SU SALUD system for enrollment of families and individuals, enrollment system permitting the classification of members and coverage according to geographic and demographic characteristics. C.2 Development and Strengthen the capacity to plan, manage, and carry out System for Financial Control in Health Strengthening of monitoring and evaluation of SU SALUD at the x Designed and implemented in 339 municipalities M&E Management national, departmental and municipal levels. Training for the financial administration of public health insurance Capacity in the systems National Unit for x 337 municipalities received training on Maternal and Infant Technical and Health Insurance (SUMI) Financed x 104 municipalities received training on Insurance for Management Elderly populations (SSPAM) (Unidades de Gestión Agreements between municipal governments and Ministry of Health Técnica y Financiera x For the use of 15 percent of municipal government funds Nacional – UGTFN) for the financing of the Unified Health Insurance (SUS) and Departmental Implementation of Law 475 units (UGTFDs). x Targeted beneficiaries know about law C.3 Development of a Support the processes of evaluating SU SALUD’s public NOT IMPLEMENTED Monitoring and policies, transparency, and accountability as well as an Evaluation System analysis of SU SALUD’s local and departmental for SU SALUD. effectiveness. 31 Sub-Component Planned outputs at Appraisal Actual outputs/outcomes at ICR D Component 4: Project Administration: to support Project administration with equipment, technical assistance, training and operating costs to finance the administration of the Project, and financial and procurement audits. (Estimated cost at appraisal: SDR$0.60M; Final cost: SDR$0.40M) 32 (c) Intermediate Results Table 2: Intermediate Result Indicators Intermediate Outcome Original Status After Available data New Restructurings Component 1 - Strengthening Government Capacity 2011 - implementation of first registry system of human resources for health 2012 - implementation of clinical and statistical The SNIS includes new information system Original Dropped information modules 2012 - implementation of vital information systems module 2013 - implementation of software for primary health care 2011 - implementation of human resources for health module in 9 SEDES The modules included in the 2012 - implementation of "Carpeta Familiar " SNIS are fully implemented in in 9 SEDES the 9 Departmental Health Became 2012 - implementation of vital information Service (Servicio Departamental development systems module in 9 SEDES Original de Salud, SEDES), the 9 outcome 2012 - implementation of clinical and statistical Departments and in all the heads indicator information system in 80% of level II and III of the referral networks in the hospitals intervention areas 2013 - implementation of software for primary health care nationally 2015 -100% 2014 - Implementation of norm to define secondary level care providers at the national level 2014-2015 - Facilities begin self-evaluation The health units apply the sector using PRONACS norms regulation norms and the Original Maintained 2014-2015 - Implementation of norm to define PRONACS norms primary care providers at the national level 2015 - 94% of health units in the target areas following approved normative to characterize health units at first and second level 2015 - 2154 facilities report monitoring data Monitoring and Evaluation of the through primary health care software system Project done with regular reports 2015 – 60% of level II and III hospitals report from information system data through clinical and statistical information providing the data (reports Original Dropped system should have the data 2015 - 9 SEDES report data through vital disaggregated by area, region, information systems module municipality, indigenous and 2015 - 9 SEDES report data through registry non-indigenous, rural and urban) system of human resources for health Component 2 - Strengthening the Maternal and Infant Health Network % of referral networks evaluated 2015 - 80% of health facilities in intervention as satisfactory in the yearly Original Dropped area undertook a quality assessment, 5% were quality assessment accredited. % of pregnant women receiving pre-natal care within the first 20 Original Dropped No information weeks of pregnancy in the areas of intervention 33 Intermediate Outcome Original Status After Available data New Restructurings % of mothers receiving post- natal care within 10 days of Original Dropped No information delivery in the areas of intervention Percentage of children under 2 Became years old who participate in the Original outcome 2015 - 89% growth monitoring sessions in indicator the areas of intervention Percentage of indigenous people satisfied with the delivery Original Dropped No information services received Percentage of civil works concluded in the health facilities New New indicator 2015 - 100% in the areas of intervention Percentage of health units in the target areas following approved New New indicator 2014 - 98% normative to characterize health units at first and second level Percentage of equipment installed in the health facilities in New New indicator 2014 - 100% the areas of intervention Component 3 - Increase in Health Insurance Percentage of the target population enrolled at the Original Dropped 2012 - 40.8% (household survey data) national level Percentage of municipalities achieving 80% of the Original Dropped No information management performance tracers at the national level Percentage of the population receiving services satisfied with New New indicator 2014- 76.5% Satisfaction the quality of the services 34 (d) Achievement of Project Development Objectives PDO 1: reduction in the gap of critical maternal and infant health risk factors between the people living in the municipalities prioritized by maternal and infant health interventions and the rest of the Bolivian population. Two Project Development Indicators were associated with this objective. PDI 1.1 Original Outcome Indicator Revised Outcome Indicator Definition Baseline Target Description Baseline Target (Date) (Date) (Date) (Date) Ratio between the percentage of Ratio between the percentage pregnant women receiving four of pregnant women accessing 0.66 0.85 0.91 0.95 prenatal care visits in the areas four prenatal care visits in the (2006) (2014) (2009) (2015) of Project intervention and the areas of Project intervention rest of the country and the rest of the country Final indicator 0.74 Percentage achieved 78% Note: indicator was rephrased (change highlighted). - The original baseline for this indicator, as described in the appraisal document, was 0.66 in 2006. The third Project restructuring officially revised the baseline for this indicator to reflect the date of Project effectiveness, 2009, and set the ratio at 0.91. The data available at the end of the Project indicate a 78% fulfillment of this target (ratio of 0.74). There were, however, concerns that the number of pregnant women might be overestimated due to the use of outdated census information that does not reflect recent declines in fertility rates. PDI 1.2 Original Outcome Indicator Revised Outcome Indicator Definition Baseline Target Description Baseline Target (Date) (Date) (Date) (Date) Ratio between the percentage of Ratio between the percentage institutional deliveries in the 0.68 0.85 of institutional deliveries in the 0.72 0.89 areas of Project intervention (2006) (2014) areas of Project intervention (2009) (2015) and the rest of the country and the rest of the country Final indicator 0.98 Percentage achieved 110% - The original baseline for this indicator as described in the appraisal document was 0.68 in 2006. At the time of the third restructuring, the baseline and target values were officially revised to reflect the 2009 date of Project effectiveness and a change in the Project’s intervention area, to a ratio of 0.72. The target ratio for 2015 was revised slightly upward from the original target and set to 0.89. Data show an achievement rate of 110% (0.98) as of Project completion. 35 PDO 2: To reduce chronic malnutrition among children younger than 2 years of age in the rural areas of the 82 municipalities targeted by the Project. One development indicator was associated with this objective: PDI 2.1 Original Outcome Indicator Revised Outcome Indicator Definition Baseline Target Description Baseline Target (Date) (Date) (Date) (Date) Percentage of 2-year-old children covered with height 35% 40% growth control (CRED) in the (2009) (2015) intervention areas of the Project Final indicator 89% Percentage achieved 222% Note: new indicator. - PDI dropped: Percentage of 2-year-old children with height over -2Z scores in the intervention areas of the Project; - PDI dropped: Percentage of children receiving exclusive breast-feeding at 6 months in the Project areas. Indicator was dropped at time of restructuring. PDO 3: Overall increase in health insurance coverage in the target areas. One indicator was associated with this objective: PDI 3.1 Original Outcome Indicator Revised Outcome Indicator Definition Baseline Target Description Baseline Target (Date) (Date) (Date) (Date) Percentage of target population Percentage of target population enrolled in the Project areas 0% 80% enrolled in Family Registry 0% 60% (2006) (2014) Forms (“Carpeta Familiar”) in (2009) (2015) the Project areas Final indicator 80% Percentage achieved 133% - The original indicator could not be tracked because the Single Health Insurance (SUS) was not implemented within the scope of the Project; - The project achieved the original 80 percent target for enrollment (133 percent of the revised target). 36 PDO 4: The SNIS incorporated a new information strategy, and production information in the intervention areas is generated from the health insurance management module. One indicator was associated with this objective PDI 4.1 Original Outcome Indicator Revised Outcome Indicator Definition Baseline Target Description Baseline Target (Date) (Date) (Date) (Date) Health insurance management The modules included in the module reports include SNIS are fully implemented in 0% 100% 0% 60% information about production the SEDES and in all the heads (2007) (2014) (2009) (2015) of the referral networks in the intervention areas. Final indicator 100% Percentage achieved 167% Note: new indicator. The table below summarizes and incorporates the information related to the achievement of PDOs into the three phases of the Project. 37 Table 3: Summary Table of PDO Achievements by Project Phase Phase 1 Phase 2 Phase 3 June 2009 - December 2012 December 2012 - July 2014 July 2014 - December 2015 PDO PDI Baseline Target Observed Baseline Target Observed Baseline Target Observed % Achieved Obs. % Achieved Obs. % Achieved Obs. (Date) (Date) (Date) (Date) (Date) (Date) (Date) (Date) (Date) Ratio between the percentage of pregnant women receiving four 0.66 0.85 1.03 0.66 0.85 0.94 1.1 121% 111% Rephrased prenatal care visits in the (2006) (2014) (2011) (2006) (2014) (2013) areas of project intervention To reduce in the gap of and the rest of the country critical maternal and infant Ratio between the health risk factors between percentage of pregnant the people living in the 1 1.1 women accessing four 0.91 0.95 0.89 municipalities prioritized by 94% Revised prenatal care visits in the (2009) (2015) (2015) maternal and infant health areas of project intervention interventions and the rest of and the rest of the country the Bolivian population Ratio between the percentage of institutional 0.68 0.85 0.94 0.68 0.85 0.89 0.72 0.89 0.98 1.2 deliveries in the areas of 111% 105% 110% (2006) (2014) (2011) (2006) (2014) (2013) (2009) (2015) (2015) project intervention and the rest of the country Percentage of 2-year-old Dropped in children with height over -2Z 37.6 22.0 32.4 37.6 22.0 32.1 2.1 33% 35% 3rd scores in the intervention (2006) (2014) (2011) (2006) (2014) (2012) Restructuring areas of the project To reduce chronic Percentage of children malnutrition among children Dropped in receiving exclusive breast 51.0 65.0 51.0 65.0 younger than 2 years of age 2.2 N.A. N.A. N.A. N.A. 3rd 2 feeding at 6 months in the (2006) (2014) (2006) (2014) in the rural areas of the 82 Restructuring project areas municipalities targeted by Percentage of 2-year-old the project children covered with height 35.0 40.0 39.0 Retrofitted 35.0 40.0 89.0 Added in 3rd 2.1 New growth control (CRED) in the 98% 223% (2009) (2015) (2013) from Phase 3 (2009) (2015) (2015) Restructuring intervention areas of the project Percentage of target Dropped in 0.0 80.0 0.0 3.1 population enrolled in the 0% 1st (2006) (2014) (2012) project areas Restructuring Overall increase in health Percentage of target 3 insurance coverage in the population enrolled in Family target areas 0.0 60.0 40.0 Retrofitted 0.0 60.0 80.0 Added in 3rd 3.1 New Registry Forms (“Carpeta 67% 133% (2009) (2015) (2013) from Phase 3 (2009) (2015) (2015) Restructuring Familiar”) in the project areas Health insurance The SNIS incorporates a new Dropped in management module reports 0.0 100.0 information strategy, and 4.1 N.A. N.A. 1st include information about (2006) (2014) production information in Restructuring production 4 the intervention areas is The modules included in the generated from the health SNIS are fully implemented in 0.0 60.0 40.0 Retrofitted 0.0 60.0 100.0 Added in 3rd insurance management 4.1 New 67% 167% the SEDES and all the heads of (2009) (2015) (2013) from Phase 3 (2009) (2015) (2015) Restructuring module the referral networks in the % Disbursed 26% 35% 39% Note: N.A. - Not available 38 Annex 3. Economic and Financial Analysis The Project’s cost-benefit analysis is based on the Project’s overall costs and its measurable economic benefits. The Project generated social and economic benefits through its impact on childhood and maternal mortality as well as on nutrition. Due to limited available data it has been impossible to attribute all changes in mortality and nutrition status in the Project target areas, to the Project’s activities. This analysis, hence, aims to present an estimated impact of the Project’s implementation. In a similar fashion to the cost-benefit analysis conducted during Project preparation, this analysis includes all Project costs but considers only the benefits directly derived from the implementation of activities within Component 2: “Family, Community and Intercultural Health” of the Project. The Project’s IRR would be higher if benefits resulting from the other 3 Project components had been included in the analysis. Three Project outcomes were included in the analysis: (i) a reduction in maternal deaths, (ii) a reduction in deaths of children under the age of 2, and (iii) a reduction in malnutrition (weight for age -2Z) among children under the age of 2. Necessary data were not available to allow for the assessment of benefits arising from decreases in unnecessary hospital stays that would have also resulted from improvements in management and infrastructure of health facilities. The IRR of the Project would have likely been higher had we been able to include these benefits in our analysis. Estimates of the reduction in maternal and infant mortality and infant malnutrition relied on the 2012 Census data for population estimates, data from the 2012 Household Survey for estimates of availability of institutional deliveries and the World Development Indicator database for data on maternal and childhood mortality in Bolivia. The number of maternal deaths averted was calculated using the estimated number of births per year in Bolivia, the percentage of women giving birth in a facility for each Project year and the estimated changes in overall maternal mortality through the Project’s implementation period. We assumed that deliveries in a health facility led to a 50% reduction in mortality among mothers (Jowett, 2000). The number of infant deaths averted was calculated using the estimated number of births in a year, the percentage of women receiving prenatal care and estimated neonatal mortality rates for each year of Project implementation. We assumed that children born to mothers who had received any prenatal care visits had a 30% reduction in probability of dying in the first 6 months of life (Lara and Pullum, 2005). Finally, to calculate the number of children under the age of 2 who were saved from malnutrition, we used population estimates and malnutrition rates for each year the Project was implemented. We assumed that children saved from malnutrition had a 50% reduced chance of death. Deaths averted from a reduction in malnutrition were added to those averted from the availability of prenatal care (Pelletier, 1995). We assumed that mortality and malnutrition rates would have remained the same if the Project had not been implemented. The number of deaths and cases of malnutrition were calculated as the difference in number between the year 2007 when the Project began, and the estimated number of 39 deaths or cases, based on our assumptions. Given the targeted nature of the Project and the availability only of country-wide data, we assumed the Project’s impact only on one fifth of the population. Table 3 provides details on the estimated number of deaths and cases of malnutrition that were averted through Project implementation. Table 4: Project Estimated Impact on Mortality and Malnutrition 2008 2009 2010 2011 2012 2013 2014 2015 Total Maternal deaths averted 5 10 16 22 24 29 32 35 171 Infant deaths averted 60 121 173 219 264 300 338 372 1,847 Infants saved from malnutrition 167 334 501 667 834 1,001 1,168 1,335 6,006 We used our estimates of the number of averted deaths and cases of malnutrition to estimate the income flow resulting from women and children whose lives were saved by the Project’s implementation. We used the income profiles and probabilities of employment for the entire rural population of Bolivia and for rural women, available from the 2012 Household survey for our estimates. Figures 1 and 2 below, show earning and employment profiles used in this analysis. To keep our estimates conservative we adjusted income flow for reported labor market participation; we did not assume any income for persons outside of the formal labor market. We used the average age of mothers (27.6 years) as the starting point for our income estimates arising from maternal deaths averted. A time lag in earnings for infant child deaths averted was used where employment- adjusted incomes were calculated only once children saved turned 15 years of age. We also assumed that children saved from malnutrition would have an earning premium of 10 percent (Hoddinott 2003; Quisumbing 2003; Gillespie and Haddad 2003; Alderman, Hoddinott and Kinsey 2002; Ross and Horton 2003). 40 Figure 1: Age-earning Profiles in Rural Figure 2: Age-working Profiles in Rural Areas (2012) Areas (2012) 35000 1 30000 0.9 0.8 Labor income (Bs) Percent employed 25000 0.7 20000 0.6 0.5 15000 0.4 10000 0.3 5000 0.2 0.1 0 0 Age Mean Income(all) Mean Income(women) Age percent employed (all rural) percent employed (female rural) This analysis considers two elements of Project related costs: (i) investment costs of the World Bank financed Project and (ii) the cost to educate children whose deaths were averted. As mentioned earlier, the entire Project costs were included in the analysis as were disbursed through time. The cost to education was calculated using the WDI database data on GDP per capita through time and the percentage of GDP per capita spent on children’s primary and secondary education. For conservative estimates we assumed children attended 5 years of primary school and 5 years of secondary school. A lag was introduced in our models to account for an age at entry at school at 6 years of age. Table 4 shows the Project’s costs and benefits through time. Both costs and benefits were discounted to 2007 for the purpose of comparison with a rate of 3%. Our estimates resulted in an IRR of 10% 30 years after the Project start with a breakeven point at 21 years after Project commencement. Table 5: Project Costs, Benefits and Internal Rate of Return (US$ '000) Direct Project Other Costs Year Social Revenue Cost (Education) Difference Cumulative 2007-2010 180.42 1,052.68 (872.26) (872.26) 2011-2015 1,156.08 10,137.06 115.00 (9,095.99) (9,968.25) 2016-2020 1,727.59 334.05 1,393.54 (8,574.71) 2021-2025 5,949.35 262.21 5,687.15 (2,887.56) 2026-2030 15,348.15 63.59 15,284.55 12,396.99 2031-2035 19,609.77 19,609.77 32,006.76 2036-2040 18,540.05 18,540.05 50,546.81 2041-2045 18,207.01 18,207.01 68,753.82 2046-2050 12,391.79 12,391.79 81,145.61 IRR 25 years 7.4% IRR 35 years 11.0% IRR 30 years 10.0% IRR 40 years 11.4% 41 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Lending Marcelo Bortman Task Team Leader - Senior Public Health Specialist GHN04 Keith Hansen Vice President GGHVP Daniel Cotlear Lead Economist GHN04 Fabiola Altimari Montiel Senior Counsel LEGLE Xiomara Morel Lead Financial Management Specialist GGO22 Patricia De la Fuente Hoyes Senior Financial Management Specialist GGO22 Lourdes Consuelo Linares Senior Financial Management Specialist GGO22 Luis M. Schwarz Senior Finance Officer WFALA Luz Zeron Senior Financial Management Specialist GGO22 Maria Lucy Giraldo Senior Procurement Specialist LCSPT Ximena B. Traa-Valarezo Social Evaluation Specialist GWADR Jorge Villena Chavez Environment Specialist LCSHD Patricia Alvarez Senior Operations Officer GED04 Jose Pablo Gomez-Meza Sr. Economist -Health LCSHH Julio Velasco Research Analyst GMF04 Julio Loayza Senior Economist GMF04 Maria Alejandra Velasco Operations Analyst LCCBO Patricia Orna Language Program Assistant GSP04 Miriam Cespedes Program Assistant GGODR Monica Claros Team Assistant LCSHD Teddy Ernesto Landaeta Consultant LCCBO Javier Jahnsen Consultant LCSHD Cecilia Lorena Brady Consultant LCSHH Supervision/ICR Marcelo Bortman Task Team Leader - Senior Public Health Specialist GHN04 Amparo Gordillo-Tobar Task Team Leader - Senior Economist – Health GHN04 Andre Medici Task Team Leader Senior Economist – Health GHN04 Patricia Alvarez Senior Operations Officer GED04 Lourdes Consuelo Linares Senior Financial Management Specialist GGO22 Patricia De la Fuente Hoyes Senior Financial Management Specialist GGO22 Tatiana Perez Financial Specialist Consultant GGO22 Alvaro Larrea Lead Procurement Specialist GGO04 Jose Rasmussen Senior Procurement Specialist GGO04 Julio Sanjines Senior Procurement Specialist GGO04 Monica Tambucho Senior Finance Officer WFALN Maria Virginia Hormazabal Finance Officer WFALN Renata Pantoja Financial Analyst WFALN 42 Elena Segura Senior Counsel LEGLE Maria Alejandra Velasco Operations Analyst LCCBO Susana Perez Executive Assistant LCCBO Patricia Orna Language Program Assistant GSP04 Gabriela Moreno Program Assistant GHN04 Miriam Cespedes Program Assistant GGODR Monica Claros Program Assistant LCCBO Carla Jerez Team Assistant LCCBO Patricia Velasco Team Assistant LCCBO Cristian Pereira Consultant GEDDR Tatiana Duran Consultant GEDDR Maria Lucy Giraldo Consultant GGODR Victor Villegas Consultant GHNDR Luis Eduardo Santalla Consultant LCSFM (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY07 17.27 101.19 FY08 18.94 120.43 Total: 36.21 221.62 Supervision/ICR FY08 6.08 13.01 FY09 27.81 29.70 FY10 23.73 96.13 FY11 28.25 121.52 FY12 30.80 149.78 FY13 29.18 123.24 FY14 33.56 182.04 FY15 32.52 158.79 FY16 21.26 114.18 Total: 233.19 988.39 43 Annex 5. Beneficiary Survey Results N/A 44 Annex 6. Stakeholder Workshop Report and Results N/A 45 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Comments from the Ministry of Health: COMENTARIOS AL INFORME DE CIERRE DEL PROYECTO APL III A continuación detallamos los comentarios al informe de cierre de acuerdo al informe remitido (a) Indicador(es) General(es) del Programa Indicador 3: Tasa de Mortalidad Materna por 100.000 nacidos vivos Aclaración.- De acuerdo al Estudio de Mortalidad Materna realizado por el Ministerio de Salud _ INE la gestión 2015 la tasa de mortalidad materna se ha reducido a 160 adjunto documento. En ese sentido, agradeceremos ajustar el dato o incluir el comentario Equipo ICR: Incorporado. (b) Indicador(es) del PDO Indicador 1: Razón entre el porcentaje de mujeres embarazadas que han recibido cuatro consultas prenatales en las áreas de intervención del proyecto y el resto del país . Aclaración.-Para la gestión 2015, el MS realizó un ajuste a este indicador por dos razones; x La información con la que se procesó el indicador con corte al 18 de Diciembre de 2012 no tenía la información cerrada de la gestión 2015. x El dato del denominador (Número de Mujeres Embarazadas) estaba sobre estimado en su proyección que había considerado los datos del Censo 2001.A la fecha el INE presento los nuevos datos de población que permitieron modificar los resultados del indicador, con un denominador ajustado, dando un resultado de 89%. Solicitamos ajustar el dato o incluir el comentario en el documento Equipo ICR: Incorporado. x H. Restructuraciones Primera Restructuración 12/10/2012 No se modificó el marco de los resultados esperados. Equipo ICR: Los indicadores relacionados a los PDOs 3 y 4 fueron cancelados en la primera restructuración. 1.3 PDO Revisado (aprobado por la autoridad de aprobación original) e Indicadores Claves, y razones/justificación 14 El APL III fue reestructurado cuatro veces en el transcurso de su ejecución, pero no cambiaron los PDO El comentario no corresponde: E l APL en su tercera fase no tenía establecido contar con una Unidad de implementación de proyecto a diferencia de los anteriores APL s. La implementación estaba prevista a través de la estructura del Ministerio para la parte técnica, los consultores del Proyecto se incorporaron a las diferentes unidades del Ministerio y la parte financiera se la realizo a traves de la Dirección General Administrativa/UGESPRO Equipo ICR: Incorporado. 46 Parrafo 15. Aclaración que, en la restructuración 2012 no se modificaron resultados, se mantuvieron los indicadores de resultado y los intermedios. Si se cancelaron actividades que no se habían iniciado o no tenían gran avance Favor corregir comentario en el texto del documento. Equipo ICR: Se cancelaron los indicadores relacionados a los objetivos 3 y 4. Aclaración.- El último párrafo no corresponde ya que a pesar de haber reducido el presupuesto al componente tres y haber establecido la carpetización en este componente, se lograron cambios importantes en los seguros de salud, como su manejo administrativo ,la unificación en un solo seguro de los que fue SUMI SSPAM . A través de la promulgación de la Ley 475, que representa la implementación del Seguro vigente, fue impulsada por el Proyecto en las áreas objetivo y en el resto del país. Este nuevo seguro que, si bien no se amplió a toda la población, incremento su cobertura en grupos poblacionales y en prestaciones Equipo ICR: Aún que hayan ocurrido avances en las políticas asociadas al aseguramiento en salud, no ocurrió la implementación del seguro originalmente previsto por el proyecto, y por lo tanto no se alcanzó este objetivo. 1.4 Principales Beneficiarios Solicitamos aclarar comentario Si bien el Proyecto tenía un área de intervención del proyecto en la que se trabajó con mayor énfasis (44 municipios); sin embargo, todas las acciones fueron implementadas a nivel nacional y esto se lo puede verificar por la normativa que respalda cada una de estas acciones y por lo arriba comentado en sentido de que las actividades fueron desarrolladas como acciones del MS para el nivel nacional. Equipo ICR: Incorporado. 2.2 Ejecución 30. Los arreglos institucionales y de ejecución fueron un cuello de botella inicial para la Ejecución del Proyecto Aclaración 31 Cuando mencionamos capacidad limitada para construcción y remodelación de establecimientos de salud por favor solicitamos especificar que fue el caso del Bloque de Oncohematologia del Hospital de Niño, en el caso de los demás proyectos no tuvimos esta limitante. Equipo ICR: Cambiado. 35. Revisiones como parte de la reestructuración Aclaración.-Las tres últimas líneas no corresponden. Como mencionamos en el inciso • H. Restructuraciones en la primera restructuración y también se menciona en este párrafo, líneas arriba que no se cambiaron los PDI favor aclarar comentario. Equipo ICR: Como se ha explicado, en la primera restructuración se cancelaron los indicadores relacionados a los objetivos 3 y 4. 36. Uso Aclaración.- Favor complementar que también se implementó el sistema de control financiero en salud, sistema de consolidación de la carpeta familiar. Este sistema de información fue totalmente digitalizado y fortalecido; lo cual representa un beneficio mayor ya que dicho sistema y la información provista, permita al MS y a los niveles locales generar planes de acción más concretos actualmente. Por ejemplo el sistema de consolidación de información de la carpeta familiar permite a los gobiernos municipales la planificación de sus Plan operativo anual de acuerdo a las determinantes de salud de su población. Equipo ICR: Cambiado. 47 38. Adquisiciones. Aclaración :-No se puede calificar moderadamente satisfactoria por el retraso en los procesos de adquisiciones ya que de acuerdo al PAD menciona que será evaluado satisfactoriamente si ambas unidades ejecutoras realizan los procesos de acuerdo a las normas establecidas por el Banco Mundial como ser a la “Guía: Adquisiciones bajo Prestamos IBRD y Créditos AIF” del Banco Mundial con fecha mayo 2004 y la “Guía: Selección y Empleo de Consultores por Prestatarios del Banco Mundial” con fecha mayo 2004 (ambas versiones fueron actualizadas en octubre del 2006 . Otro aspecto para la calificación que consideramos importante es el resultado de las auditorias de Adquisiciones. Equipo ICR: De acuerdo, cambiado para satisfactorio. 39. Gestión financiera Aclaración Consideramos que el retraso en el reporte de los estados financieros y las auditorias, antes de la reestructuración del proyecto no debería definir su desempeño como moderadamente satisfactorio. Si revisamos el PAD, se establecieron varios aspectos para su evaluación que fueron cumplidos, y consideramos que deben ser tomados en cuenta. Finalmente para la calificación debe ser tomado en cuenta el resultado de las auditorias que refleja el desempeño del Proyecto. Equipo ICR: Sin embargo, los retrasos reducen un poco la calificación. 43. La relevancia del diseño del proyecto después de la primera reestructuración, es decir durante la segunda etapa, es calificada como Modesta. Aclaración como MS No consideramos modesta la relevancia del proyecto posterior a la restructuración. Si bien por ejemplo el proyecto no tuvo acciones directas sobre los objetivos uno y dos ,tomando el mismo ejemplo que no se realizó el incremento de recursos humanos a las áreas de intervención para alcanzar los objetivos uno y dos, el proyecto implemento un sistema de información en recursos Humanos que registro y reporto las brechas de recursos humanos permitiendo al Estado a través del MS por primera vez incorporar 2000 médicos al primer nivel de atención totalmente sostenibles en el tiempo y con mayor impacto en la población que se refleja en la disminución de los indicadores de desnutrición y mortalidad materno infantil. Equipo ICR: Los criterios de calificación en este caso son apenas cuatro: 1. despreciable a baja; 2. modesta; 3. significativa y 4. alta. No hay moderadamente satisfactorio. 3.2 Logro de los Objetivos de Desarrollo del Proyecto (véase también sección (d) del Anexo 2 para más detalles y cuadros con una comparación de los PDI revisados) En el inciso b s del presente documento realizamos la aclaración y solicitud de modificación del dato el resultado del indicador de resultado uno Equipo ICR: Incorporado. 5.2 Desempeño del Prestatario (a) Desempeño del Gobierno Aclaración.- Solicitamos tomar en cuenta el desempeño del proyecto en su conjunto y no solo se lo evalué en función a un solo componente. Equipo ICR: Se enfatizaron dos aspectos: los problemas relacionados con el aseguramiento y las dificultades asociadas a la definición del equipo de implementación del proyecto al interior del MS que llevaron a problemas de ejecución y a la cancelación de casi mitad de los recursos originalmente asignados. 73. El desempeño del MS y el Fondo Nacional de Inversión Productiva y Social como Las agencias ejecutoras se califican como moderadamente satisfactorio. 48 La evaluación se limita a los procesos lentos antes de la restructuración del proyecto, y solo evalúa un aspecto del retraso en la implementación en su fase inicial y no así el logro de las metas y la importante ejecución post-restructuración. Solicitamos revisar los términos. Equipo ICR: Se ha matizado el texto. Conclusiones y Recomendaciones Consideramos que la calificación asignada al proyecto como moderadamente satisfactoria no corresponde por que no se puede evaluar el desempeño del proyecto por la modificación en las actividades de un solo componente, y teniendo mayor énfasis en la etapa previa a la restructuración , sin tomar en cuenta el cumplimiento de los acuerdos legales para la implementación ,los resultados de los informes de las auditorias financieras y de adquisiciones. Finalmente que no se tome en cuenta los resultados alcanzados en los 5 indicadores establecidos en las enmiendas del proyecto. Por lo que recomendamos revisar la calificación asignada a los aspectos de la evaluación y considerar una nueva calificación al proyecto. Equipo ICR: La calificación es consistente con el último ISR. Como se explica en el texto, la calificación final refleja la subestimación de las metas y en las limitaciones de los indicadores y resulta también de los criterios definidos por el Banco para su estimación. 49 Additional Information from FPS: En el componente de Infraestructura a cargo del FPS se logró la conclusión de una cartera de proyectos conformada y distribuida geográficamente de la siguiente manera: Cartera de Proyectos: Pre Inversión 16 Proyectos Inversión Infraestructura y Equipo 23 Proyectos Equipamiento 29 Proyectos Total proyectos 68 Proyectos Área de Intervención: Equipamiento médico Beni, Cochabamba, la Paz Oruro, Santa Cruz Total de 19 municipios Infraestructura La Paz, Cochabamba, Oruro Total 5 municipios Al anterior detalle de proyectos se incluyó el proyecto de la Unidad de Oncohematología del Hospital del Niño en La Paz. En términos de ejecución financiera el FPS logro consolidar una ejecución acumulada al 30 de abril de 2015 de USD 6.969.648 (DEG 4.689.254) con recursos del crédito y USD 3.247.024 con recursos de contraparte local. Este nivel de ejecución que representa cerca del 70 % del monto total del Programa se consolidó después de dos reasignaciones de fondos que le fueron asignando al FPS mayores responsabilidades en la ejecución según se muestra en el cuadro siguiente: Monto Original Primera Segunda Incremento Asignado al Reasignación de Reasignación de en la Subcomponente 2 Fondos Fondos Asignación [DEG] [DEG] [DEG] 3.100.000 4.300.000 4.689.254 51.26 % 50 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N/A 51 Annex 9. List of Supporting Documents Alderman H, Hoddinott J, Kinsey B. Long-Term Consequences of Early... 2003 [cited 2016 May 2]; Available from: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.1.4304 Gillespie S, Haddad L. The double burden of malnutrition in Asia: causes, consequences, and solutions [Internet]. Sage Publications India; 2003 [cited 2016 May 2]. Available from: https://books.google.com/books?hl=en&lr=&id=o8- GAwAAQBAJ&oi=fnd&pg=PP1&dq=Gillespie+and+Haddad+2003&ots=MPHhmRasGw&sig =2sWlMD1KJGy284Y2vPsDYKaw2Hs Hoddinott J. Pathways from poverty in sub-Saharan Africa. In: Pre-workshop meeting of the BASIS CRSP Policy Conference on Combating Persistent Poverty in Africa, Cornell University [Internet]. 2003 [cited 2016 May 2]. Available from: http://eldis.org/vfile/upload/1/document/0708/DOC15179.pdf Horton S, Ross J. The economics of iron deficiency. Food policy. 2003;28(1):51–75. Jowett M. Safe motherhood interventions in low-income countries: an economic justification and evidence of cost effectiveness. Health Policy. 2000 Oct 1;53(3):201 –28. Lara GJ, Pullum TW. Infant mortality in Egypt: Exploring the role of prenatal care and implications for public policy. In [Unpublished] 2005. Presented at the 2005 Annual Meeting of the Population Association of America Philadelphia Pennsylvania March 31-April 2 2005.; 2005 [cited 2016 May 2]. Available from: http://www.popline.org/node/275530 Pelletier DL, Frongillo Jr EA, Schroeder DG, Habicht J-P. The effects of malnutrition on child mortality in developing countries. Bulletin of the World Health Organization. 1995;73(4):443. Quisumbing AR. Food aid and child nutrition in rural Ethiopia. World Development. 2003;31(7):1309–1324. 52 Annex 10. Assessment of Project Achievement towards Outcomes We used other available data to complement the assessment of Project achievement of its original objectives. PDO 1: To reduce the occurrence of critical risk factors affecting maternal and infant health in the targeted areas so that current gaps between regions are reduced. Using Government data on vaccination coverage and institutional deliveries, we find that differences across departments increase for vaccination coverage and institutional deliveries, while differences between target and non-target municipalities increase for vaccine coverage but not for institutional deliveries, between the year of the APL III Project’s approval and 2012-2013 (the year of the latest available data). While pentavalent vaccine coverage decreases slightly on average for Bolivia between 2006 and 2013, coverage decreases dramatically in target municipalities decreasing from 80% to nearly 67% in 2013. Figure 3: Pentavalent Vaccine Coverage in Target and non-target municipalities 90.0 85.0 Vaccine Coverage 80.0 BOLIVIA TARGET MUNICIPALITIES 75.0 OTHER MUNICIPALITIES 70.0 65.0 2006 2007 2008 2009 2010 2011 2012(p) 2013(p) Year Source: National Institute of Statistics Between 2005 and 2013, the differences in pentavalent vaccine coverage across departments widens; the departments of Beni and Pando see increases in coverage while the departments of Potosi and Chuquisaca see decreases in coverage. 53 Figure 4: Pentavalent Vaccine Coverage across Departments in Bolivia 110.0 105.0 100.0 BOLIVIA 95.0 Percent Coverage CHUQUISACA 90.0 LA PAZ COCHABAMBA 85.0 POTOSÍ TARIJA 80.0 SANTA CRUZ 75.0 BENI PANDO 70.0 65.0 2005 2006 2007 2008 2009 2010 2011 2012(p) 2013(p) Year Source: National Institute of Statistics The percentage of institutional births increased between 2006 and 2012 in Bolivia from 65 to 71 percent. Project target municipalities saw large increases in this indicator, however, with institutional births increasing from approximately 45 to 55 percent in the 6-year period. Although the target municipalities continued to lag behind other municipalities in the county, the difference decreased from nearly 15 percentage points to nearly 6 percentage points. Figure 5: Institutional Deliveries in Target and non-Target Municipalities 75.0 70.0 65.0 Percentage of Children 60.0 BOLIVIA 55.0 TARGET MUNICIPALITIES OTHER MUNICIPALITIES 50.0 45.0 40.0 2006 2007 2008 2009 2010 2011 2012(p) Year Source: National Institute of Statistics Although differences in institutional deliveries decreased when comparing between target and non- target municipalities, differences across departments increased between 2006 and 2012. While the 54 percentage of institutional deliveries increased relatively equally in most departments, the department of Beni and Pando saw increases in this indicator of nearly 10 percentage points in this 6 year period. Figure 6: Institutional Deliveries across Regions in Bolivia 110.0 100.0 90.0 BOLIVIA Percentage of children CHUQUISACA 80.0 LA PAZ COCHABAMBA 70.0 POTOSÍ TARIJA SANTA CRUZ 60.0 BENI PANDO 50.0 40.0 2006 2007 2008 2009 2010 2011 2012(p) Year Source: National Institute of Statistics PDO 2: To reduce chronic malnutrition among children under 2 years of age in the targeted areas From available health survey data, we can see that malnutrition of children under 2 years of age decreased in Bolivia overall between 2007 and 2012, with decreased prevalence in the Valleys region, no change in the Highlands region and a large increase in the Plains region of the country. 55 Figure 7: Percentage of children ages 6-23 months with weight for age less than 2 standard deviations from the mean 7 6 Percent of children 6-23 months 5 4 National Highlands 3 Vallyes Plains 2 1 0 2007 (LB) 2012 (ESNUT) Year (Survey) Source: LB and ESNUT PDO 3: To increase health insurance coverage in the targeted areas Available national household survey data show that health insurance coverage increased throughout all departments in Bolivia from 2009 to 2012. The departments of Tarija and Beni were able to achieve high levels of insurance coverage during this period of time, with the increase in coverage in the department of Pando was much less dramatic. 56 Figure 8: Coverage with any insurance (public/private/other) by department 100 90 80 National 70 Percentage of population Beni 60 Chuquisaca Cochababma 50 La Paz 40 Oruro Pando 30 Potosi 20 Santa Cruz Tarija 10 0 2009 2012 Year Source: 2010 and 2013 National Household Surveys 57 MAP 58