ICRR 13593 Report Number : ICRR13593 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/30/2011 PROJ ID : P078991 Appraisal Actual Project Name : Health Services US$M ): Project Costs (US$M): 82.2 not available Extension And Modernization (2nd Apl) Country : Nicaragua Loan /Credit (US$M): Loan/ US$M ): 11.0 10.3 Sector Board : Cofinancing (US$M ): US$M): 67.1 64.5 Sector (s): Health (93%) Central government administration (5%) Sub-national government administration (2%) Theme (s): Population and reproductive health (25% - P) Health system performance (25% - P) Child health (24% - P) Nutrition and food security (13% - S) Decentralization (13% - S) L/C Number : C4050 Board Approval Date : 04/05/2005 Partners involved : Finland, Sweden, Closing Date : 11/30/2009 06/30/2010 IDB, Netherlands Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz John R. Eriksson IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA, page 23), the objective of the project was to improve maternal and child health in the Borrower's poorest municipalities by: (i) ensuring that PCs [participating communities] have access to a package of basic health and nutrition services; (ii) strengthening the Borrower's health service facilities network; and (iii) providing institutional strengthening and capacity building to the MOH . According to the Project Appraisal Document (PAD, page 14), the overall objective of the project was to improve maternal and child health in the poorest municipalities of Nicaragua , to be achieved through (i) the extension of coverage of a package of basic health services to the poorest areas of the country, via the new integrated model of health care; (ii) the physical and functional strengthening of the health service facilities network, focusing on maternal-infant care, so as to provide a seamless path of essential services in the target areas; and (iii) institutional strengthening and capacity building , concentrated specifically on developing the purchasing, monitoring and evaluation capacities of the Ministry of health (MOH), its leadership role to enhance donors' alignment and harmonization, and the management capacity of MOH decentralized units and Autonomous Governments of the Atlantic Coast. The DCA statement is used as the basis for this Review since it is more monitorable . This project was the second part of a two-phase Adaptable Program Loan (APL). The objective of the overall APL was to improve health outcomes in Nicaragua, particularly among the poor, by raising the efficiency, effectiveness, equity and sustainability of the Nicaraguan health system . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Component 1: Extension of the Package of Basic Health Services (PBHS) PBHS ) to Vulnerable Rural Populations at the US$36. Community -Level (Appraisal : US$36 36 .5 million; Actual : not available ): This component financed capitation transfers to Local Systems of Integrated Health Care (Sistema Local de Atencion Integral de Salud - SILAIS) to deliver a standardized package of basic health services to participating communities, as well as technical assistance to develop institutional capacity in the SILAIS . The standard package had a strong focus on maternal and child health and included growth monitoring, integrated care for women of reproductive age, and care of prevalent diseases. The MOH entered into management agreements with each SILAIS, which in turn purchased services from public and/or private providers. The participating communities came from 79 municipalities and two indigenous territories based on specific targeting criteria . Component 2: Strengthening of the Network of Services in Targeted Areas to Support Implementation of the US$ 27. PBHS (Appraisal : US$27 27 .8 million; Actual : not available ): This component aimed to strengthen the MOH's health services network through reinforcing physical and functional aspects of health facilities . Activities included rehabilitation and equipping of facilities; expansion of the Casas Maternas (Maternal Homes) network; and strengthening of family planning services . Component 3: Improvement of Stewardship, Institutional Strengthening and Decentralization US$ 17. 17 .8 (Appraisal : US$17 million; Actual : not available ): This component aimed to develop the MOH's leadership and capacity to improve health service delivery. Activities included the development of capacity to evaluate the performance of the national health system and coordinate external assistance; technical assistance to monitor population policy; and development of purchasing and fiduciary functions . The activities also supported MOH decentralization through the transfer of financial resources to the SILAIS and local health care providers, including in the two autonomous regions (indigenous) of the Atlantic Coast. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost and Financing The ICR does not provide a figure for the actual total cost of the project, although Annex 1 provides actual costs for each of the four co-financiers (IDB, Sweden, Netherlands, and Finland ). The total of these is US$74.8 million. According to the project team, as the project was designed to support a sector -wide program which included government funds as well, it is not possible to determine the actual cost . At the time of appraisal, a number of development partners including Finland, Sweden, the Netherlands, Inter-American Development Bank (IDB) and IDA agreed to pool funds to create a single fund (called FONSALUD) to finance the Five-Year Implementation Plan (FYIP) of the national health strategy . However, pooling of IDA funds was later deemed to be legally unfeasible due to the differences in fiduciary procedures between IDA and the other donors . IDB was faced with a similar situation. Instead, IDA (and IDB) agreed to maintain a separate project with objectives identical to the FYIP . US$714,000 of the IDA credit was cancelled due to misprocurement (see Section 11 for details). Borrower contribution The planned Borrower contribution was US$ 4.1 million. The ICR does not provide an actual figure for the Borrower contribution, as the project supported a sector -wide program and thus it is not possible to determine the exact Borrower share. Closing date The closing date was extended from November 30, 2009 to June 30, 2010, to allow for the completion of minor rehabilitation works, to finalize procurement of medical equipment for indigenous communities, and to complete the financial management assessment of the capitation transfers . 3. Relevance of Objectives & Design: High The 2008 Country Partnership Strategy identifies The relevance of the project objectives is rated High. investment in the delivery of basic services, particularly in poor rural areas, as a key pillar . The objectives are also directly relevant to the Bank's corporate focus on achieving the Millennium Development Goals on maternal and child health. The Government's current health strategy, the National Human Development Plan for 2009-2001, prioritizes health promotion, combating maternal and child mortality, increasing services to the poor, adapting appropriate care for indigenous groups, and making better use of existing resources . The relevance of the project design is rated Substantial . The project design drew on successful models of the delivery and financing of primary health care, namely the provision of a standard package of basic health services through capitation transfers . Clear targeting criteria were established to identify the poorest municipalities . The project was also designed in the context of a Sector -Wide Approach (SWAp), given the multiplicity of donor-funded projects in health which were all consistent with the government's health strategy . Although the original plan to create a single donor fund was untenable, the ICR reports that the SWAp remained a highly effective mechanism for donor coordination. According to the project team, the APL approach was beneficial in ensuring that sector objectives are sustained amidst political transitions . 4. Achievement of Objectives (Efficacy): To improve maternal and child health in the poorest municipalities of Nicaragua . Substantial . Intermediate outcomes cited below (under each of the three activities "by" which the project objective was to be achieved ) show likely Substantial achievement of the overall objective . The ICR (page 24) reports the amount of project funding that each of the targeted SILAIS (Sistema Local de Atencion Integral de Salud ) received - the SILAIS were selected from among the poorest municipalities in the country according to pre -determined criteria. However, it is not clear whether the amounts received were equivalent to the amounts planned . While the maternal, infant, and under -5 mortality rate showed declines, these figures are reported for the country as a whole so changes specifically in the targeted areas are not known. In addition, the baseline data represented 2001, while the project implementation period began in 2006. Outputs: (See outputs listed below) Intermediate Outcomes: The proportion of institutional deliveries increased from 36.2% in 2005 to 78.7% in 2009 in the targeted areas. Coverage of early prenatal care increased from 34.4% in 2005 to 48.1% in 2009 in the targeted areas. Coverage of complete prenatal care increased from 49.5% in 2005 to 58.4% in 2009 in the targeted areas. Outcomes: The Maternal Mortality Rate (MMR) decreased from 96/100,000 in 2001 to 61/100,000 in 2009, surpassing the target of 65/100,000. The Infant Mortality Rate (IMR) decreased from 31/1,000 in 2001 to 23/1,000 in 2008, surpassing the target of 27/1,000. The Under-5 Morality Rate (U5MR) decreased from 41/1,000 in 2001 to 27/1,000 in 2008, surpassing the target of 33/1,000. The percentage of the population with chronic malnutrition remaining on a similar level from 17.8% in 2001 to 16.9% in 2006, falling short of the target of 12.8%. The overall objective was to be achieved "by": Extension of coverage of a package of basic health services to the poorest areas of the country . Outputs: US$5.2 million in capitation transfers to the 12 targeted SILAIS to finance a standardized package of basic health services (BPHS). Outcomes: The number of beneficiaries of the BPHS increased from 129,698 to 529,652, surpassing the target of 470,000. The proportion of institutional deliveries in the 12 targeted SILAIS increased from 36.2% in 2005 to 78.7% in 2009. Immunization coverage (pentavalent vaccine) in infants under one year old increased nationwide from 78.0% in 2005 to 98.3% in 2009, surpassing the target of 97.5%. The increase in project -specific areas is not reported . Coverage of early prenatal care increased in targeted municipalities from 34.4% (28,656 women) in 2005 to 48.1% (32,533 women) in 2009, surpassing the target of 37.3%. Coverage of complete prenatal care increased in targeted municipalities from 49.5% in 2005 to 58.4% in 2009, surpassing the target of 55.0%. Ph ysical and functional strengthening of the health service facilities network, focusing on maternal Physical -infant care Outputs: Rehabilitation of two health posts and three health centers . Provision of medical equipment and emergency units in 12 hospitals. Outcomes: Utilization of maternal homes in project areas by pregnant women increased, as measured by the increase in the number of discharges from 2,400 in 2005 to 9,749 in 2009, surpassing the target of 6,000. Hospital discharge rates in the 12 targeted hospitals increased from 4.02% in 2005 to 6.3% in 2009, surpassing the target of 5.25%. 3 hospitals in the project areas with critical path services were in compliance with MOH certification standards, falling short of the target of 9 hospitals. 60% of public health providers in project areas were in compliance with management agreement targets in 2009, falling short of the target of 100%. 75% of maternal deaths were audited in project areas in 2009, falling short of the target of 100%. Beneficiary surveys conducted in 2009 and 2010 showed that 77% of beneficiaries had a "positive perception that the time used by staff to provide services were adequate ". However, over 50% of beneficiaries concurred that cleanliness of facilities and waiting times were the most important issues that needed improvement . Institutional strengthening and capacity building of the MOH, Outputs: Completion of the 2006 National Health and Demographic survey . Technical assistance in the areas of procurement, financial management, and auditing . Outcomes: 100% of the MOH budget in the Autonomous Atlantic Regions was administered by the respective MOH budget executing units. The project likely contributed to the overall program objective . 5. Efficiency (not applicable to DPLs): Efficiency of the project is rated Modest . The PAD presented an economic analysis for the project, which included contributions from other donors for a total cost of US$82.2 million disbursed over five years . The Internal Rate of Return was calculated as 54% over ten years. The ICR provided a cost-benefit analysis of the APL project, estimating that 30 to 40% of the benefits from the program as a whole could be attributed to the Bank investment . The PAD did not provide a comparable estimate of the share of benefits. Benefits were derived from significant reductions in maternal and infant mortality, with moderate reductions in malnutrition, as well as savings from improved health care practices . A low estimation of benefits resulted in a slightly negative rate of return, while a high estimation of benefits resulted in a rate of return of 6.0%. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 6% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project objectives were highly relevant and design was substantially relevant . The overall objective to improve maternal and child health in the poorest municipalities was rated substantial due to improvements in intermediate outcomes such as antenatal coverage and proportion of institutional deliveries . Efficiency was rated modest. Overall outcome is therefore rated Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The PAD did not provide an overall project risk rating, although several risk factors were identified, all of which were rated either Moderate or Low. Financing will be continued through a follow -up operation which was approved by the Bank in December 2010. The ICR notes that the establishment of project coordination responsibilities in an existing MOH directorate (General Division of Planning and Development - DGPD) rather than in a stand-alone Project Implementation unit (PIU) has been an important factor in ensuring the project's sustainability . However, as noted in the ICR, the country is highly prone to external shocks such as natural disasters, which could significantly impact any advances in health status . In addition, the heavy dependence on external financing entails the risk of relying on aid that could be withdrawn or reduced significantly . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: at -Entry : The project drew on successful models of the delivery and financing of primary health care, Quality -at- and the standardized package was developed using inputs from a comprehensive evaluation of the 14 primary health care delivery projects that were being implemented in the public sector at that time . Lessons learned from the APL1, including the successful pilot model of the Casa Maternas and the decentralization of administrative responsibilities to the SILAIS, also informed the project design . Risks and mitigation measures were adequately addressed. However, shortcomings in M&E design included the inadequate selection of key indicators . The ICR also notes that covenants of the Indigenous Peoples' Plan (IPP) were poorly defined. Supervision : Although the efforts to establish a pooled donor fund had not succeeded, the FONSALUD group remained an effective mechanism for donor coordination in which the Bank was an active participant . The ICR reports that members of FONSALUD "welcomed IDA's strong commitment to meet regularly with donors, to coordinate actions, to discuss and receive feedback on activities directly related to aspects financed under the Credit, and to follow up on agreed common indicators ." The ICR reports that procurement and financial management staff were heavily utilized and that technical staff provided frequent monitoring and follow up . However, there were two cases of misprocurement (see Section 11), and the IPP was not being adequately implemented until close to the end of the project period . The IPP implementation was rated Moderately Unsatisfactory for nearly two years, until the Government prepared an action plan in mid -2009 to address the shortcomings and implement corrective actions . The IPP rating was subsequently changed to Moderately Satisfactory. at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government performance : The Government demonstrated strong commitment to the project objectives, providing effective leadership through the SWAp . Although a change of administration took place in 2007 and a new National Health Plan was introduced in 2008, the Government remained committed to the overall objectives . Implementing Agency performance : The SILAIS received capitation transfers and monitored progress at the local levels through management agreements, which were shown to be an effective mechanism for holding SILAIS accountable for results . Although the PIU from the previous APL 1 project had been eliminated in the APL2, former PIU staff were absorbed under various divisions of the MOH and helped sustain implementation momentum and effectiveness. However, "frequent" turnover in fiduciary positions led to difficulties in procurement, including two cases of misprocurement (see Section 11), and the IPP was not being adequately implemented for most of the project period, although it was rated "moderately satisfactory" after mid-2009. There was "frequent" turnover in procurement staff as the MOH had difficulties retaining highly qualified staff that were knowledgeable of the Bank's procurement procedures . Misprocurement was declared for two contracts . In one case, national, instead of international, bidding procedures were used; in the other case, only two rather than three proposals were requested under local shopping procedure . US$714,760 was cancelled due to the misprocurement and the Government repaid the amounts in November 2009. Toward the end of the project, two procurement staff were financed to minimize the problems that arose from frequent staff turnover . The ICR does not indicate if procurement problems were resolved . a. Government Performance :Satisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : Key indicators (maternal and child mortality, malnutrition) were based on those agreed upon collectively by the Government and development partners as part of the SWAp . However, these were not the appropriate indicators given their long -term nature. According to the project team, arrangements were not made to track them in project-specific areas due to the sector -wide design; instead, technical audits were conducted to ensure that project funds were being used properly to achieve objectives . Donors had agreed to use the Nicaraguan Demographic and Health Survey carried out every five years as the source of data, but data from the 2006 survey were not available in a timely manner for establishing baseline figures and the next survey is not scheduled until 2011. Specific arrangements to support M&E implementation or develop capacity were not provided . The intermediate outcome indicators, rather than the PDO indicators, were more relevant and appropriate for monitoring the project's results (i.e. proportion of institutional deliveries, immunization coverage ). M&E Implementation : The ICR reports that the monthly and quarterly reports from SILAIS, which were responsible for monitoring progress at the local level, were generally received in a timely manner . As health outcome indicators were not tracked for the project areas specifically, the Bank (and IDB) financed annual technical audits to validate information that was being collected by the SILAIS . The quality of results was audited based on a sample of municipalities. and the most recent audit report found that information related to the intermediate outcome indicators "provided an acceptable basis for determining achievement of results ", although "improvements were needed at the primary health care centers regarding accuracy and completeness of basic data gathered by them ." Also, with the change in Administration in 2007, the practice of transferring MOH budget to health facilities based on a system of services rendered was ended (replaced with budget transfers linked to performance agreements ) and thus one of the original indicators to track the percentage of total budget transferred to the targeted SILAIS was no longer monitored . The indicator measuring per capita public health expenditure was also not tracked as it was not included in the M&E system. M&E Utilization : Although the ICR reports that the M&E system is functioning adequately and "providing information on sector-wide progress", there are no specific details provided . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : The project was assigned an Environmental Category "B" rating, and therefore an Environmental Assessment was prepared, including a health care waste management plan . The ICR reports that compliance with environmental safeguards was satisfactory and that most hospital rehabilitation programs included the redesign of waste management facilities and practices . The project also required an Indigenous Peoples Plan which provided recommendations on including traditional medicine practices, strengthening local networks of midwives, and providing social control mechanisms . The ICR reports that a 2008 assessment of the plan's implementation found that "little progress was made on reaching remote areas, establishing new sub -regional health centers, training of additional doctors and midwives, and strengthening social control mechanisms ... largely due to the exclusion of key stakeholders from the central MOH during supervision in the dialogue with representatives from the two autonomous regions, which led to tension between the two parties ." The IPP implementation was rated Moderately Unsatisfactory for nearly two years, until the Government prepared an action plan in mid -2009 to address the shortcomings and implement corrective actions. The IPP rating was subsequently changed to Moderately Satisfactory . Financial Management : The ICR reports that overall, the MOH complied with financial covenants and presented the annual financial and external audit reports with no significant delays . No major financial management problems were reported, although a 2008 audit identified some questionable costs related to the capitation transfers to the SILAIS and health centers. A financial management assessment was conducted and concluded that for the period under review, the correct accounting and internal controls were in place and capitation transfers were well managed . Procurement : There was "frequent" turnover in procurement staff as the MOH had difficulties retaining highly qualified staff that were knowledgeable of the Bank's procurement procedures . Misprocurement was declared for two contracts. In one case, national, instead of international, bidding procedures were used; in the other case, only two rather than three proposals were requested under local shopping procedure . US$714,760 was cancelled due to the misprocurement and the Government repaid the amounts in November 2009. Toward the end of the project, two procurement staff were financed to minimize the problems that arose from frequent staff turnover . The ICR does not indicate if procurement problems were resolved . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The use of common national outcome targets can be helpful for donor harmonization; however, by themselves, they may not allow for monitoring achievement of the objective (s) of the specific project being financed. 14. Assessment Recommended? Yes No Why? To verify the ratings and document lessons learned . 15. Comments on Quality of ICR: The ICR was overall satisfactory with a concise presentation of information . However, there were no comments reported from co-financiers, and Annex 1 on project costs reports IDA financing only, rather than for the project as a whole. A more detailed discussion of how the performance agreements were used to ensure delivery of services (i.e. the BPHS) would have been informative. The ICR should also have reported on the extent to which the procurement problems were resolved. a.Quality of ICR Rating : Satisfactory