Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) Report Number : ICRR0020351 1. Project Data Project ID Project Name P095563 PE- (APL2) Health Reform Program Country Practice Area(Lead) Peru Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-76430 31-Jan-2015 162,400,000.00 Bank Approval Date Closing Date (Actual) 17-Feb-2009 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 15,000,000.00 0.00 Revised Commitment 11,984,637.36 0.00 Actual 11,984,637.36 0.00 Prepared by Reviewed by ICR Review Coordinator Group Erik A. Bloom Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives Framed within the long-term objective of the Health Reform Program (Adaptable Program Loan in all its phases) to reduce maternal and infant mortality rates in the nine poorest regions of the country (Amazonas, Huanuco, Huancavelica, Ayacucho, Apurimac, Cusco, Cajamarca, Ucayali and Puno), the development objectives of this second phase of the APL were to: (i) improve family care practices for women (during pregnancy, delivery and breast-feeding), and children under the age of three; (ii) strengthen health services networks with capacity to solve obstetric, neonatal, and infant emergencies and to provide comprehensive health services to women (during pregnancy, delivery, and breast-feeding) and children under the age of three; and (iii) support the Ministry of Health's (MINSA's) governance functions of regulation, quality, 3/20/2017 Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) efficiency, and equity for improving the new health delivery model of maternal and child health care in a decentralized environment (Loan Agreement, p. 5). A 2014 revised targets for some intermediate outcome indicators, but as these were not key outcome indicators, a split assessment is not warranted. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? --- d. Components Component 1. Improving health practices at the household level for women (during pregnancy, delivery and breastfeeding) and children under the age of three in rural areas of selected Regions (demand-side interventions), by: a) design, implementation, and monitoring of a behavioral change communication and education program to promote healthy practices at the household level, including increased demand for health services; and b) promotion of comprehensive health insurance (SIS) enrollment rights and identity rights of the targeted population. Component 2. Increasing the capacity to provide better maternal and child health services for the poor (supply-side interventions); through: a) the improvement of the quality of services in health facilities of the nine regions; and b) the provision of support for the integrated health delivery model and the development of support systems to raise the efficiency and effectiveness of health networks. Component 3. Strengthening government capacities to offer more equitable and efficient health systems in a decentralized environment (governance and financing) by: (a) supporting a regulatory framework and increasing quality in the provision of health services, (b) expanding SIS enrollment; (c) strengthening data monitoring and accountability in the system; and (d) supporting the decentralization of health services. Component 4. Project coordination and Monitoring and Evaluation (M&E), through the provision of technical assistance, financing of incremental operating costs, and external and concurrent audits. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was appraised in December 2008 and was approved in February 2009. It became effective in December 2009. The project was closed in December 2015, approximately one year after originally planned. The project was restructured twice. The first restructuring, in 2011, changed the funding allocation to reflect better how the government was providing its financing to the project. The second, in 2014, extended the project, modified the results framework, and introduced changes in procurement procedures. The project was largely financed by the government ($132.4 million) with planned support from IBRD ($15 million) and the Inter-American Development Bank (IDB; $15 million), for a total of $162.4 3/20/2017 Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) million. Although the government provided most of the financing, it wanted the World Bank and IDB's participation to provide technical support and to improve the quality of procurement and financial management. Allocation of costs by component Components US$ Million Percent Planned Actual amount amount Component 1. Improving health practices at the household 6.0 6.4 107% level for women (during pregnancy, delivery, and breastfeeding) and children under age three in rural areas of the nine targeted Regions Component 2. Increasing the capacity to provide better 142.3 138.5 97% maternal and child health services for the poor Component 3. Strengthening government capacities to offer 5.2 3.3 62% more equitable and efficient health system in a decentralized environment Component 4. Project coordination, and Monitoring & 8.9 16.9 102% Evaluation Total 162.4 165.1 102% Allocation of financing by source. Data by component and source are not available. Source US$ million Percentage Planned Actual Government 132.4 138.2 104% IBRD 15.0 12.0 80% IDB 15.0 15.0 100% Total 162.4 165.2 102% 3. Relevance of Objectives & Design a. Relevance of Objectives The project focused on a major issue facing Peru, namely the high level of health inequality affecting a number of isolated regions. This inequality was particularly pronounced in maternal and infant health and reflected the lack of knowledge and the low level of coverage by health facilities in poorer areas of the country, combined with high financial costs to access hospital care if necessary. The project was well aligned with the Country Partnership Strategy (CPS, 2012-2016), which is still in force at the time of this review. The CPS clearly identifies maternal and child health as priorities and identifies the project as the cornerstone for its support to health (Pillar 1.2 of CPS). The CPS also identifies inequality as a 3/20/2017 Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) major development challenge, particularly in rural areas. The objectives, with their focus on poorer areas with low levels of coverage (particularly maternal services), are very much in line with the relevant CPS. The project was also well aligned with the government's strategy towards health, in particular with its goals to expand services to areas with low coverage. In addition, the project's objectives supported the government's efforts to strengthen decentralization and provide greater access by expanding the public health insurance system. Rating High b. Relevance of Design The project's objectives centered around improving family care practices, health services networks, and overall system governance to benefit women and young children. The project's design had a direct focus on women and children, with specific health-related activities. While behavior change communication has the potential to improve outcomes (particularly those related to the first objective), they are unlikely to have great impact on their own. The PAD refers to IADB support for community initiatives, which may also contribute to achievement of the project's objectives (pp. 13-14 of the PAD). The PAD does not provide much guidance on the specifics, although the initiatives appear to be focused on training. Component 1 also included support for efforts to enroll women and children in the public insurance scheme. This activity was to focus on ensuring that beneficiaries had the necessary identification documents to receive coverage. While this is not a typical activity for a health project, the lack of identification was identified as a major constraint in enrollment. Focusing on identification therefore had the potential to play an important role in ensuring effective coverage. The third objective focused on improving the capacity of the Ministry of Health. Component 3 was to provide support by improving the regulatory framework to improve service quality. This would not have had a specific focus on women and young children, but it was designed to provide support to many aspects of the health system. Component 3 also was to include support to strengthen the public insurance system (known as SIS), the Ministry’s monitoring capacity, and support to local government capacity. A major element of the project’s design involved the construction and renovation of infrastructure, at the secondary level (Component 2, p. 14 of PAD). While this does support the objectives to improve family care practices for women and children and to strengthen health services networks for women and children, it is not the most effective way to improve child and maternal health, especially among vulnerable populations in isolated areas. In particular, it ignores health providers at the community level that play a "frontline" role in promoting maternal and child health, particularly in remote areas where access to urban centers is difficult. The PAD identifies the need for more community level services, in particular “maternal houses," which were not supported by the project. 3/20/2017 Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Improve family care practices for women (during pregnancy, delivery, and breastfeeding) and children under the age of three. Rationale Family care practices refer here to interventions and actions that are likely to improve women’s health during pregnancy and delivery (including the use of breastfeeding) and the health of young children. Given the nature of this objective, it is often difficult to distinguish between outputs and outcomes. Outputs The ICR reports that 1,178 health workers were trained in behavior change communication (target of 758). The project also supported the distribution of dissemination equipment to 1,423 health centers and provided programming for 18 television stations and 45 radio stations (p. 30 of the ICR). While none of these indicators give a sense of coverage, they represent an important contribution. However, coverage of SIS is not reported in the results framework. The ICR reports that around 667,000 young children and 1,889,000 women received birth certificates and SIS affiliation (p. 30 of the ICR). The project made a clear contribution to this increase in coverage. Although the results framework of the ICR does not indicate what this coverage implies in terms of percentage, it clearly represents a significant gain in the proportion of the population covered. Outcomes The ICR reports on a number of outputs relevant to the objectives related to children. During the project, the percent of SIS-affiliated children who received growth controls increased from 34 to 57 percent (target: 66 percent). For women beneficiaries, the percentage of SIS-affiliated pregnant women receiving a number of key laboratory tests increased from 37 to 69 percent (original target 80 percent; revised target 69 percent). Similarly, the percentage of SIS-affiliated pregnant women receiving nutritional supplements increased from 37 percent to 55 percent (original target: 80 percent; revised target: 60 percent). The main indicators are expressed in terms of the increase in coverage of the population covered by SIS and do not give a sense for the total population that received lab tests or nutritional supplements. There was an increase in the proportion of pregnant women who have had at least one antenatal visit from 20 to 69 percent (original target: 45 percent). During the project period, the proportion of children who were exclusively breastfed increased from 64 to 80 percent (original target: 80 percent).The project’s 3/20/2017 Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) support for dissemination, training, and intercultural communication is likely to have partially contributed to this gain (pages 50 & 57 of the ICR). The same cannot be said of the reduction of anemia among pregnant women from 42 to 36 percent (original target: 35 percent) and the reduction of anemia among young children from 70 percent to 57 percent (original target: 60 percent). While these are important gains, they are not necessarily related to the project and reflect broader national trends. In addition, there is evidence that there has been little long-term progress on anemia and that the decrease may reflect random fluctuation rather than a real reduction; thus the results are less stable than the initial and target numbers suggest (p. 15 of the ICR; external evidence). Based on the above evidence, achievement of this objective is rated substantial, reflecting the project’s contributions to a number of important primary health indicators through its support for training and information campaigns as well as its contribution to increasing SIS coverage. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Strengthen the capacity of health services networks to solve obstetric, neonatal, and infant emergencies and to provide comprehensive health services to women (during pregnancies, delivery, and breast-feeding) and children under the age of three. Rationale This objective refers to efforts to strengthen the formal health system to address maternal and child health issues. Outputs The project supported the construction and rehabilitation of infrastructure and the purchase of equipment in 104 health facilities. The project provided technical support to several hospitals, operated an internship program, and provided training on a number of topics, including intercultural communication and the vertical delivery of health services (ICR, pp. 31-34). The project also supported the development of new protocols related to child and maternal health. In addition, the project financed an internship program, which had 647 participants and focused on emergency obstetrics and neonatal care. This program included staff from primary and secondary level providers (including hospitals) as well as some community-based staff (ICR, p. 32; PAD, p. 56). Outcomes The ICR identifies that the proportion of SIS households that had out-of-pocket expenditures for medicine decreased from 67 to 56 percent (original target 25 percent; revised target 55 percent). There is no clear causality between the project and this outcome. Likewise, the decline was significantly less than 3/20/2017 Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) was originally planned. The ICR also reports that the hospital mortality rate for neonates (children under the age of one month) dropped from 9.5 to 5 percent (original target 5 percent). The project is likely to have contributed to this reduction, through its support for equipment and infrastructure. The project support for training through internships is likely to have contributed to this reduction as well, as this was a major theme of the training. While the ICR does report that the infant mortality decreased nationally, it does not provide data on a regional level to allow comparisons between project and non-project regions. Annex 3 provides data comparing health progress among regions (pages 42 to 44). In all cases, the baseline for the indicators in the project regions are "worse" than those for the non-project regions. The ICR reports that, in the period from 2009 to 2014, child malnutrition dropped by 41 percent in project regions compared to 39 percent in non-project regions. Given the nature of project interventions, it is unlikely that the project would have had much of an impact. Likewise, the number of maternal death between 2007 to 2015 declined by 28 percent in project regions compared to a 38 decline in non-project regions. It is quite likely that the project's support for training (through internships) as well as strengthening hospital services did make a contribution. Likewise, the number of institutional births increased by 13 percent in project regions and by 16 percent in non-project regions. Based on the above, achievement of this objective is rated substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Support the Ministry of Health's governance function of regulation, quality, efficiency, and equity for improving the new health delivery mode of maternal and child health care in a decentralized environment. Rationale Although the project was primarily focused on supporting the health system in the poorest regions, it also provided support to the national Ministry of Health to improve its performance. In many cases, this support was expected to have a positive impact on the other two objectives. Output and Outcomes The project provided support to develop 32 technical norms and regulations in a range of areas including health insurance and medical procedures, among others (pp. 34-35 of the ICR). This was complemented by a wide range of technical assistance supporting SIS, particularly as it decentralized its responsibilities (pp. 36-37 of the ICR). The project also supported a number of training programs for health staff to focus on administration. It also financed 20 studies on a variety of topics. The objective focuses on providing support to improve the governance and the administration capacity of the Ministry. The ICR argues that this support lead to stronger management capacity at the regional level. This 3/20/2017 Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) is evidenced by a reduction in disbursement delays in the budget execution for maternal and child health in the targeted regions. Likewise, the ICR argues that the training and technical assistance contributed to cultural and institutional change within the regional offices. This was based on observations from the ICR team as well as qualitative evaluations carried out by the project (pp. 18-19 of the ICR). The project's capacity building and technical assistance for SIS was timely and facilitated the expansion of SIS, in particularly in promoting identity rights (p. 21 of ICR). Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective Reduce maternal and infant mortality rates in the nine poorest regions of the country Rationale The overall APL Program objective was to reduce maternal and infant mortality rates in the project regions. The ICR reports significant improvement in infant mortality from 2005 to 2015, as well as improvement in institutional deliveries (a proxy for maternal mortality). However, as outlined above it is unclear whether this project's interventions would have made a significant impact on these indicators. (Note: Following normal practice for an APL, this review comments above on contribution of this project toward the overall APL program objective and does not give a rating for this objective.) Rating Not Rated/Not Applicable PHREVDELTBL PHREVISEDTBL 5. Efficiency There were significant delays in designing the project, resulting from political changes. The ICR reports that once the government agreed to support the project, it was quickly approved. The project was implemented in the expected time (five years). However, effectiveness was delayed by 10 months, which lead to an 11-month extension. By the end of the project, 20 percent of World Bank funds were not disbursed. The ICR explains that this was due to delays in procurement and civil works that were never resolved (p. 20 of the ICR). The project’s support for the expansion of SIS appears to have been cost-effective. The intervention has the potential to improve the well-being of beneficiaries by increasing access and reducing potential income shocks. The cost of this intervention appeared to be low, primarily involving the issuing birth certificates and 3/20/2017 Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) identify cards for poor beneficiaries. Investments in maternal and child health tend to be among the most cost-effective in the health sector, as measured by years of life saved. However, the project provided substantial support to infrastructure at the secondary level. While component 1 does have a strong focus on community needs, component 2 does not. The PAD cites the importance of increasing coverage for the poorest (p. 34 of the PAD) and likewise on the importance of primary health clinics (p. 38 of the PAD). It is therefore not likely that the project was particularly cost-effective when it comes to providing child and maternal health services. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated High based on the objectives' link to country conditions, country strategy, and Bank strategy. Relevance of design is rated Substantial, as the project's planned activities were largely linked to achievement of intended outcomes, though the focus on infrastructure at the secondary level was not the most effective way to improve family care practices for vulnerable women and children. Achievement of the objective to improve family care practices for women and young children is rated Substantial, based on the contribution of training, information campaigns, and increases in SIS coverage to improvements along key indicators. Achievement of the objective to strengthen health services networks for women and young children is rated Substantial, as the project tis likely to have contributed to many of the outcomes. Achievement of the objective to support MINSA's governance is rated Substantial, with project support reportedly leading to enhanced management capacity related to maternal and child health. Efficiency is rated Modest, as infrastructure provision at the secondary care level is not the most cost-effective way to achieve the project's objectives related to maternal and young child health. This review gave the relevance of objective a high rating and the relevance of design, a substantial rating. The three sub-objectives in the efficacy section were rated substantial. Finally, efficiency was rated modest. Based on this, the project is rated moderately satisfactory. 3/20/2017 Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Although there were delays in designing and approving the project due to political concerns, the project and its objectives are broadly accepted in Peru and are consistent with current national health policies. The government’s follow-on project was built upon PARSALUD, with coverage expanded throughout the entire country and providing a similar set of interventions. While the new program does not have the same focus on the poor, it is likely to sustain the investments made under PARSALUD. The ICR reports that it has remained difficult to retain staff in project regions (page 24). While this is not a new phenomenon, it may put at risk the project's support for training unless the training process continues under government systems or the new project. The government has continued its support for SIS, which continues to expand and now covers more than half of the population. Strengthening SIS was one of the project’s main goals, through the project's support to the poor to gain the necessary identification papers and by strengthening the administration of SIS (p. 22 of the ICR).. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry Project preparation was delayed due to changes in government policy. While this was outside of the control of the Bank, the ICR comments that the Bank did not sufficiently take advantage of this hiatus to develop new interventions. Likewise, the Bank could have taken advantage of the extra time to have worked on the project's results framework, given many concerns about the monitoring and evaluation capacity of the implementing agency. It could have also worked to improve the relevance of the indicators (pp. 23-24 of the ICR). Despite the delays for political reasons, the team was able to work with the government to develop a project that was largely financed by the government and had a wide set of interventions. This required complicated negotiations with the Ministries of Health and Finance. While the project was primarily focused on infrastructure, it did include a wide range of soft interventions that aimed to provide behavior change and encourage enrollment in SIS. During the project preparation period, the Bank was able to address the two safeguard policies that the project triggered. For both the Environmental Assessment and Indigenous Peoples safeguards, the Project developed plans that were subsequently and clearly incorporated into project design. Likewise, the PAD 3/20/2017 Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) identified a number of fiduciary weaknesses. In addition to identifying these risks, the Bank proposed a comprehensive approach to improving financial management (pp. 66-79 of the PAD). While the PAD contained an economic analysis, it provided limited guidance beyond discussing two options using cost- effectiveness analysis (pp. 88-90 of the PAD). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The World Bank was engaged with the project, carrying out regular supervision missions every six months. In addition, the Bank carried out a number of stand-alone financial management and safeguard supervision missions to ensure compliance. As a result, safeguards were monitored throughout the project period. These missions were useful in ensuring that the executing agency took necessary actions. The World Bank restructured the project in 2011 (two years after approval) to reallocate resources. The second restructuring in 2014 made adjustments to the indicators along with a number of administrative changes. The ICR reports that the Bank was generally proactive in reporting available results, although some of the PDO-level indicators were not available until late in the project implementation period. The Bank did not take advantage of the first restructuring to make necessary changes in the results framework. The rigidities in Peru’s institutional structure made major changes difficult to arrange. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Throughout preparation and implementation, the project had uncertain political support. While the government always supported the project’s goals in the abstract, it was slow to accept the project’s specific design. However, when approved, the government did provide necessary funds and approve regulatory changes. The Ministry of Economy and Finance was rigid in modifying the project. This was particularly the case for modifications of softer interventions. Likewise, the Ministry of Economy and Finance did not provide additional counterpart financing for component 2 or for the project staff at the time of closing. Government Performance Rating Moderately Unsatisfactory 3/20/2017 Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) b. Implementing Agency Performance The Ministry of Health created a stand-alone Unit to administer the project. This Unit was adequately staffed and was relatively stable, with a low degree of turnover. Since it had been several years since the first phase of the project closed, the Unit had to be reconstituted. The Unit had good communication with both the World Bank and IADB, including a willingness to work with electronic documents (p. 8 of the ICR). The ICR reports that the Unit was proactive with financial management and cooperated with Bank counterparts. This is exemplified by timely audit reports and efforts to improve internal expenditure reports. Likewise, the Unit was active in monitoring for collusion and cooperated with the Bank’s anti-corruption unit (INT) in an investigation. The Unit was also quite active in implementing safeguards. This included steps to monitor implementation, both at the input and output level. The Unit cooperated with the different safeguard monitoring missions (pp. 25-26 of the ICR). The project was quite slow in disbursing funds for civil works, and much of this activity was delayed until after the mid-term review. By the closing date, all commitment funds were disbursed. However, 20 percent of project funds were not disbursed. This was caused by legal problems in finalizing a contract for a consulting firm. The Unit tried to solve the issue and believed resolution was possible up to three months before the project closed (p. 20 of the ICR). It is possible that the Unit could have solved this issue or alternatively advised the Bank that it would not be possible to fully disburse the project. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design Many of the project’s indicators are not well aligned with its objectives. For example, the results framework includes few indicators that measure strengthening health systems (objective 2) or improving health system governance (objective 3). There is only one PDO-level indicator that is relevant for objective 2. There are a number of indicators at the intermediate outcome level, especially for objective 3. At the intermediate outcome level, the indicators are generally well designed and appropriate for monitoring the objective. The results framework at the project level is largely composed of higher-level outcomes, such as the level of anemia and similar measurements. While these are important and are the result of the project's logical model, several of them are outside of scope of the project's stated objectives. For objective 1, several PDO indicators refer to higher-level outcomes that are the logical result of achieving the sub-objective. For example, reducing anemia is a consequence of improving health practices rather than an indicator of improved health practices. At the time of project preparation, most of the indicators could be monitored using existing data sources or by specialized instruments. In this manner, the project avoided the need to set up a parallel reporting system. Baseline data was available at the time of preparation. 3/20/2017 Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) b. M&E Implementation As the results framework used readily available data, most of the indicators were tracked as expected, using the proposed data sources. Although five indicators were dropped during the course of implementation, two were reported in the ICR using available data. The five dropped indicators largely focused on objective 2 and particularly objective 3. The results framework was monitored as part of supervision. During a restructuring (2011) and the mid-term review (2013), the results framework was modified to reflect the availability of data and to adjust target values for some intermediate outcome indicators based on changing circumstance. c. M&E Utilization The utilization of monitoring and evaluation in the project was mixed. At the national level, policy makers did take advantage of some of the results as they related to implementing different work streams. However, there is little evidence that data were used at the local level, given the delays in implementing parts of the information system. M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project triggered two safeguard policies: Environmental Assessment (OP4.01) and Indigenous Peoples (OP 4.10). The Environmental Assessment was category B. The ICR reports that there were not issues with implementing either of the triggered safeguards. In 2013, the World Bank carried out a number of targeted missions focusing on the implementation of safeguards. All of the participating regions had significant indigenous populations from different ethnic and language groups. To develop the Indigenous Peoples Plan, the Bank team conducted a series of workshops in different regions. The Plan included specific indicators based on the project's results framework. The plan proposed a number of interventions including greater use of indigenous language and culturally appropriate interventions (pp. 91 to 93 of the PAD). The Environment Assessment safeguard was triggered due to construction to be carried out with project financing as well as potential impacts from solid and liquid waste. Peru's existing legislation was expected to provide good protection and to mitigate environmental risks (pp. 95 to 96 of the PAD). The ICR confirms that there were several missions to review the progress of the safeguards. Based on the findings of these missions, the World Bank supported adjustments to the project (p. 11 of the ICR). Component 1 included many actions from the Indigenous Peoples Plan, including developing radio programs in a variety of languages and training health personnel in cross-cultural communication (pp. 30-31 3/20/2017 Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) of the ICR). Likewise, the project supported a number of activities to support the environmental assessment, including technical assistance and training to support construction. b. Fiduciary Compliance The project did not face any significant fiduciary issues. During the project, the World Bank generally rated financial management satisfactory in most of its supervision reports. The ICR reports that both the financial reports and audit reports were provided on time. The ICR reports that there were some issues with procurement, which was often delayed. At the end of the project, this lead to the cancellation of around of 20 percent of the project’s financing. The ICR attributed this to poor planning (p. 11 of the ICR). c. Unintended impacts (Positive or Negative) Following the ICR, one major impact was on the Identification Rights Movement. While the project focused on birth certificates as a route to expanding the coverage of the public insurance scheme, this support helped fuel the larger movement. The support was also useful for beneficiaries to gain access to other government programs. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Risk to Development Modest Modest --- Outcome Moderately Moderately Bank Performance --- Satisfactory Satisfactory Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as 3/20/2017 Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) appropriate. 13. Lessons The ICR (pp. 26-27) identifies a number of interesting lessons, including: 1. The health sector can successfully interact with other agencies to push forward the agenda of both agencies. The project was an example of how the health sector supported the issuance of birth certificates. While there are many initiatives to encourage the unregistered to get identification papers, tying the initiative to a useful benefit (health insurance) appears to have to been successful. 2 . While having good baseline data and generally reliable data sources is important, a good results framework also requires realistic targets and outcomes that are related to the PDO. While the project had a reasonably strong results framework, the lack of focus on several key outcomes as well as the lack of data for some indicators reduced the utility of the results framework. The project could have identified indicators that were better aligned with the project's underlying model. Likewise, the project could have better assessed what data were currently available and what data were likely to be difficult to collect; this was not an issue for the key indicators but proved a challenge for a few indicators. 3 . Politics can play a major role in designing and implementing investment projects. There is much discussion about political economy and project implementation, and this project can serve as an example of how this can affect design and implementation in practice. After initial delays, the project team was able to work with the new government to implement a project that was consistent with their goals. 14. Assessment Recommended? Yes Please explain An assessment is recommended for several reasons. First, the project was focused on supporting basic health services (in particular, maternal care and care for young children), with equity targeting. Second, the project was largely financed by the government. This is an interesting example of the Bank providing value-added beyond financing. Third, while the project was active, there were a number of complementary Bank programs and projects. 15. Comments on Quality of ICR The ICR provides a candid and useful overview of the project. It presents a good discussion of many of the project's shortcomings along with a review of its accomplishments. This is particularly the case with the 3/20/2017 Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE- (APL2) Health Reform Program (P095563) discussion of Bank and Borrower performance. In the annexes, the discussion of project activities is strong, as is the economic analysis. While the ICR goes beyond just presenting the results of the indicators, it would have benefited from a more analytical approach to attribution and the appropriateness of the indicators. Annex 3 contains data that could have been used to discuss attribution for the second objective. a. Quality of ICR Rating Substantial 3/20/2017 Page 16 of 16