Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) Report Number : ICRR0020427 1. Project Data Project ID Project Name P106619 DO (APL2) Health Ref II Country Practice Area(Lead) Dominican Republic Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-77770 30-Oct-2015 30,500,000.00 Bank Approval Date Closing Date (Actual) 17-Sep-2009 29-Feb-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 30,500,000.00 0.00 Revised Commitment 29,559,124.47 0.00 Actual 29,559,124.47 0.00 Sector(s) Public Administration - Health(95%):Health(5%) Theme(s) Health system performance(100%) Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Loan Agreement of October 30, 2009 (p. 5) and the Project Appraisal Document (PAD) of August 20, 2009 (p. 5), the objectives of the project were “to improve: (i) the capacity of the Borrower’s Regional Health Services (RHSs) to deliver in a timely fashion quality services known to improve the health of mothers, children and people with chronic conditions by public providers at the first level of care; and (ii) health system responsiveness, defined as the institutional capacity of public sector health organizations to conduct strategic purchasing of health care services and goods and to respond to public health emergencies.” This project was the second phase of a 12-year Adaptable Program Loan to be implemented in three phases. The overall objectives of the APL were: i) improved mother and child health, and poverty reduction; ii) implementation of new health sector reform laws, including Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) strengthening the stewardship role of the Ministry of Health; and iii) consolidating universal health insurance as the public stewardship organization for the health sector, development of regional health care networks, and health insurance mechanisms. This assessment will separate the achievement of "delivery of services in a timely fashion" from the delivery of "quality services." Also, during a June 2014 restructuring, region VII was added to the project, resulting in a modification of outcome targets. Therefore, this project validation will assess project outcomes before and after the restructuring. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 27-Jun-2014 c. Components The project consisted of four components: Component 1: Introducing results-based financing mechanism for the first level of care in Regional Health Services (appraisal estimate US$21.49 million, actual US$8.25 million, 38.4% of appraisal estimate): This component was to finance results-adjusted capitation mechanisms to better align incentives to improve the delivery of health services at the first level of care in selected regional health services (regions VI and VIII). Component 2: Strengthening the capacity of the Ministry of Health (MoH) to improve and monitor the health system responsiveness while fostering transparency and accountability (appraisal estimate US$10.00 million, actual US$7.09 million, 70.9% of appraisal estimate): This component was to finance goods, technical assistance, non-consulting services, training and operational costs to strengthen the capacity of the MoH to deliver public health goods and services while fostering transparency and accountability, and to strengthen existing information systems for the adequate functioning of results-adjusted capitations. Component 3: Improving the quality of public spending on health care goods and services (appraisal estimate US$12.6 million, actual US$18.2 million, 144.5% of appraisal estimate): This component was to finance goods, technical assistance, non-consulting services, training and operational costs to strengthen the coordination and institutional capacity of public sector organizations for more sustainable financing, planning and purchasing of health services, and other key inputs needed to improve the quality of health services with an emphasis on primary health care. Component 4: Support for response to public health emergencies (appraisal estimate US$0.10 million, actual US$0.07 million, 70% of appraisal estimate): This component was to finance consulting and non-consulting services, technical assistance and goods including medicines, laboratory and protective equipment, laboratory reagents and other medical supplies, development and dissemination of materials for information campaigns, and training for emergency preparedness to support the response to public health emergencies of local or international nature. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project costs: The project was estimated to cost US$44.24 million. Actual cost was US$33.66 million. Financing: The project was to be financed by a US$30.5 million credit by the International Bank for Reconstruction and Development (IBRD). Actual financing was US$29.56 million, 96.9% of the appraisal estimate. Borrower contribution: The borrower was to contribute US$13.74 million. The actual contribution was US$4.1 million, 29.8% of the appraisal estimate, due to the government's fiscal challenges. Dates: The project was restructured three times: On June 27, 2014 the project was restructured to: i) align indicator targets to take into account the expansion of the project from regions VI and VIII to include region VII; ii) use project funds to cover the remaining financing needs of component 1; iii) reallocate funds to key Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) activities that would further contribute to the attainment of the PDOs; and iv) adjust disbursement estimates to reflect expenditure projections based on an updated implementation plan. On October 28, 2015 the project was restructured to extend the closing date of the project by four months from October 30, 2015 to February 29, 2016 to allow for the completion of essential activities. On December 4, 2015 the project was restructured to: i) modify the project’s financing plan to increase the share of Bank financing retroactively; ii) reallocate funds between disbursement categories; iii) update disbursement estimates; and iv) adjust the cost of components. 3. Relevance of Objectives & Design a. Relevance of Objectives The objectives of the project were highly relevant at project appraisal and closing. Even though the Dominican Republic had been achieving improvements in its health outcomes over the past few decades, the country was still facing worse mother and child health outcomes than other countries in the region with similar or lower levels of economic development. Additional challenges included the steadily growing impact of non-communicable diseases (NCDs), inefficiency and low public health service quality, and weaknesses in health system management and in public spending quality and control. The project supported the government’s efforts to sustain growth and make growth more inclusive, as pointed out in the government’s 2013-2016 plan that aims to expand access to public health insurance and upgrade the quality of health services. Also, the government’s national development strategy 2030 aims to guarantee health care to the entire population and promote poverty reduction and social and territorial inequality. The project was in line with the Bank’s Country Partnership Strategy at appraisal (FY06- 09), which aimed to achieve greater social equity through human development and increased coverage and quality of basic public services. At project closing the project was in line with two of the strategic development challenges identified in the Bank’s Country Partnership Strategy (FY15-18), to promote equitable, efficient, transparent and sustainable management of public resources, and to strengthen social service delivery. Rating Revised Rating High High b. Relevance of Design The planned activities were logically and plausibly linked to the achievement of the project objectives. Activities to improve the health of mothers, children and people with chronic conditions included the provision of health services at the first level of care through Results Based Financing, including a capitation payment mechanism. Health care providers were to receive a first installment, which was 50 percent of the total amount of the capitation transfer, paid on a monthly basis upon verification of a registry of eligible beneficiaries by the national health insurance (SENASA). The second installment, which could be up to 50 percent, was paid every four months. Its calculation was based on how many of the ten monitoring results indicators had been achieved by the regional health services. Each indicator counted for five percent of the overall capitation. Planned activities to improve the health system’s responsiveness included technical assistance and trainings to strengthen the capacity of the Ministry of Health (MoH) and the coordination and institutional capacity of public sector organizations. However, the project design gave limited consideration to the exogenous risk of lacking counterpart financing, and implementation delays occurred. Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) Objective 1 Objective Improve the capacity of the Borrower’s Regional Health Services (RHSs) to deliver in a timely fashion quality services known to improve the health of mothers, children and people with chronic conditions by public providers at the first level of care: DELIVERY OF SERVICES IN A TIMELY FASHION Rationale Outputs: • 366,236 individuals received health insurance. • Studies on risk factors and burden of diseases were conducted, disseminated, and used by the corresponding Ministry of Health units to develop annual operational plans for health promotion and prevention, achieving the target. • 17 consultancies, 26 workshops and 21 training courses focusing on strengthening the stewardship role of the Ministry of Health, separating the purchase and provider functions, were conducted. • At project closing, the health management system of the Ministry of Health contained a new automated/electronic module for monitoring & information (M&I) with the information needed for quarterly monitoring of results related to the first level of care, with data available for regional health services, the Ministry of Health at the central level, and the national health insurance (SENASA), achieving the target. Outcomes: • Between 2011 and 2016, the percentage of pregnant women from the target population who had completed a risk evaluation before the 15th week of pregnancy increased from 0.43% to 50%, achieving the original target of 50% and surpassing the revised target of 40%. • Pregnant women in the project regions had on average 3.43 consultations per pregnancy, versus 1.8 consultations in non-project regions. • In the project regions the percentage of pregnant women who only had one consultation was 46.1%, while in non-project regions it was 63.9%. The percentage of women who had more than three consultations was 23% in project regions, versus 10.7% in non-project regions. The percentage of women who had a consultation before the 12th week of pregnancy was 38.3% in project regions, versus 23.8% in non-project regions. • 68% of children from the target population under the age of 15 months completed the vaccination scheme according to national protocols, surpassing the target of 60%. 63% of children over 14 months and under 24 months completed the vaccination scheme according to national protocols, surpassing the original target of 60% and the revised target of 45%. These indicators lacked a baseline. Rating Substantial Revised Objective The objective was not revised, but some outcome targets were changed when an additional region was added in June 2014. Revised Rationale The project achieved all revised outcome targets, as stated above. Revised Rating Substantial PHEFFICACYTBL Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) Objective 2 Objective Improve the capacity of the Borrower’s Regional Health Services (RHSs) to deliver in a timely fashion quality services known to improve the health of mothers, children and people with chronic conditions by public providers at the first level of care: DELIVERY OF QUALITY SERVICES Rationale Outputs: In addition to outputs reported above: • 90% of clinical files had completed information for external audit to verify RHS-reported monitoring results indicators, achieving the original target of 85%. Outcomes: • 45% of individuals symptomatic of tuberculosis with results from sputum smears were included in clinical files of the corresponding health center, surpassing both the original target of 40% and the revised target of 35%. • Between 2011 and 2016 the percentage of individuals from the target population who were screened for hypertension according to national protocols increased from 0.89% to 64% in 2016, surpassing the original target of 60% and the revised target of 40%. The percentage of individuals from the target population diagnosed with hypertension who were under treatment according to national protocols increased from 3.54% to 38%, not achieving the original target of 50% but surpassing the revised target of 35%. • Between 2011 and 2016 the percentage of individuals from the target population who were screened for diabetes according to national protocols increased from 0.07% to 43%, surpassing the original target of 40% and the revised target of 35%. The percentage of individuals diagnosed with diabetes type II under treatment increased from 0.79% to 34%, surpassing the original target of 40% and the revised target of 30%. • Between 2011 and 2016 the percentage of post-partum women referred from hospitals to their corresponding first level of care units with proper documentation included in their clinical files increased from 0.44% to 63%, surpassing the target of 50%. • Between 2011 and 2016 the percentage of children under 24 months with growth and development monitor controls according to national protocols increased from 0.27% to 62%, surpassing the original target of 60% and the revised target of 50%. Rating Substantial Revised Objective The objective was not revised, but some outcome targets were changed when an additional region was added in June 2014. Revised Rationale The project achieved all revised outcome targets, as stated above. Revised Rating Substantial PHEFFICACYTBL Objective 3 Objective Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) Improve health system responsiveness, defined as the institutional capacity of public sector health organizations to conduct STRATEGIC PURCHASING OF HEALTH CARE SERVICES AND GOODS. Rationale Outputs: In addition to outputs noted above: • Equipment for the Program for the Supply of Essential Medications and Office for Logistical Support (PROMESE/CAL) was purchased. • A conceptual design of the “Information System of Medicines and Inputs in the National Health System Framework of the Republic” was developed. • A tariff system and mechanism for the hiring of service providers of SENASA was developed. • The General Health Information System (SIGS) was launched and modifications such as designing electronic family health files and enhancing the inter-operability of information modules and systems were made. • Three consultancies to develop strategies for Information Education Communication to affiliates to improve access and healthy habits were conducted. • Subsidized regime affiliate cards were prepared and printed in the project’s regions. • Three consultancies to assess the costs and sustainability of the subsidized regime were conducted. • The information system and management model for a chain of medicine and input provision including an automatic electronic prescription system in regions VI, VII and VIII was implemented. Outcomes: In addition to outcomes noted above: • All accredited health centers from participating regional health services that did not have a pharmacy supplied by the Program for the Supply of Essential Medications and Office for Logistical Support (PROMESE/CAL) regularly received medicines procured and distributed by PROMESE/CAL Rating Substantial Revised Objective Indicator targets related to this objective were not revised. Revised Rationale Indicator targets related to this objective were not revised. Revised Rating Substantial PHEFFICACYTBL Objective 4 Objective Improve health system responsiveness, defined as the institutional capacity of public sector health organizations to RESPOND TO PUBLIC HEALTH EMERGENCIES. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) Rationale Outputs/Outcomes: • The capacity in the Ministry of Health was strengthened to improve the detection, investigation, and treatment of possible Ebola cases. Medical equipment and a vehicle were purchased, and the attendance of trainings in Cuba and Colombia was financed. • An information risk management system for SENASA was developed. As no public health emergencies arose during the project period, the capacity of public sector health organizations directly to respond to emergencies was not demonstrated, but institutional capacity was strengthened. Rating Substantial Revised Objective Indicator targets related to this objective were not revised. Revised Rationale Indicator targets related to this objective were not revised. Revised Rating Substantial PHEFFICACYTBL Objective 5 Objective The program objectives of the overall 12-year Adaptable Program Loan were: i) improved mother and child health, and poverty reduction; ii) implementation of new health sector reform laws, including strengthening the stewardship role of the Ministry of Health; and iii) consolidating universal health insurance as the public stewardship organization for the health sector, development of regional health care networks, and health insurance mechanisms. Rationale Give the outputs and outcomes listed above, this project, which constituted the second phase of the APL, furthered achievement of each of the overall program objectives, with the caveat that clear links between the achievements of the project (outputs, outcomes) and reduction of poverty were not established. Rating Not Rated/Not Applicable Revised Objective The overall program objectives were not revised. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) Revised Rationale The overall program objectives were not revised. Revised Rating Not Rated/Not Applicable 5. Efficiency The PAD (p. 19) conducted a cost-benefit analysis. It assumed that the benefits derived from the project were quantifiable from the reduction of deaths of pregnant women, puerperal mothers, and infants and the decrease in mortality and disabilities from hypertension and diabetes. The analysis was based on the expected future income over the lifetime of individuals targeted in the project. The estimation used a discount rate of 8%, representing the social investment opportunity cost of the Dominican Republic. In addition, a range of discount rates was used, and the results were included in a sensitivity analysis of the Net Present Value (NPV) of the project. At a discount rate of 8%, and when the rates of reduction of infant and maternal mortality and disabilities from chronic diseases were at the middle of their respective ranges, the net present value of the project was estimated at US$ 936,759, and the corresponding internal rate of return was 8.35%. The analysis concluded that the largest contribution to the net present value of the project comes from a decrease in infant mortality, followed by a decrease in maternal mortality. The ICR did not conduct a cost-benefit analysis but compared health outcomes of project regions versus non-project regions between mid- 2011 and 2014. The comparative analysis shows that consultations per person were generally higher in project regions than in non-project regions.The ICR concluded that the decreases in infant mortality, maternal mortality and disabilities from chronic diseases were positive but lower than was originally estimated in the PAD. The RBF mechanism's indicators and targets were efficiently designed and implemented, covering a broad segment of the population and focusing on the country's basic package of health services. Despite delays due to a slow restructuring of the project, implementation bottlenecks were overcome effectively. The mid-term review found that while the Project Implementation Unit (PIU) was efficient in managing operating costs, the Project Coordinating Unit (PCU) was less efficient, especially when it came to staff and vehicles used. However, the PCU was effective in reducing costs after the mid-term review (see Section 9b). Based on the above, efficiency is rated Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  8.35 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) 6. Outcome Relevance of the objectives was High before and after the restructuring, as the objectives were responsive to country conditions, Bank strategy, and government strategy. Relevance of design was Substantial before and after the restructuring, as planned activities were plausibly and logically connected to intended outcomes, but the project's design did not take into account adequately the exogenous risk of insufficient counterpart financing. Achievement of all four project objectives was Substantial, meeting or exceeding outcome targets. Efficiency was rated Substantial due to likely high rates of return and strong implementation efficiency. Taken together, these ratings are indicative of minor shortcomings in the project's preparation and implementation under both the original and revised outcome targets, and therefore an overall Outcome rating of Satisfactory. a. Outcome Rating Satisfactory 7. Rationale for Risk to Development Outcome Rating Modest: The project built capacity within the MoH's different units that is likely to be sustained. The National Health Insurance scheme is planning to manage and expand the results-based financing mechanism as a national program, adopting the same performance agreements that were used for component 1 in the project regions, paving the way for the mechanism to continue being used in the regions. However, the project is currently not being followed by another Bank operation, and activities implemented under the project do not receive any other funding. The government continues to allocate insufficient funding to the health sector, presenting challenges to sustaining project achievements. While the Inter-American Development Bank is currently implementing a similar project, modeled after this one, it includes different regions than those covered here. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The project built on lessons learned from APL 1 and previous experience of Bank operations in the Dominican Republic. The Bank conducted an intensive policy dialogue with the government and key health sector authorities during the six months of project preparation. The Bank chose an APL as the financing instrument, which was appropriate given the long-term plan for changes in the Dominican health sector. The Bank performed a risk analysis and identified relevant risk factors such as personnel changes at key public organizations and management positions slowing down implementation, challenges related to implementing a new two-installment capitation payment mechanism, and issues related to several implementation entities with different levels of capacity. Mitigation efforts included the signing of a memorandum of understanding to obtain political support, ensuring that the project design maximized the use of existing institutions and management structures, and strengthening administrative and financial management systems. However, a critical risk related to the availability of counterpart financing was excluded from the PAD due to previous positive experience in this area. Also, there was a lack of clarity regarding the two implementing agencies’ responsibility and autonomy in the areas of operations, procurement and fiduciary management, leading to implementation challenges. Quality-at-Entry Rating Moderately Satisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) b. Quality of supervision The Bank team included experienced and technically diverse experts. It provided continuous technical and operational support to strengthen capacity and conducted policy dialogues with all different government units involved in the implementation of the project. Also, the Bank team proactively addressed implementation challenges such as the lack of counterpart financing. Furthermore, the Bank team supervised and supported the implementation of technical and financial audits and commissioned an external review of different project areas such as staffing, internal controls, and costs in order to identify shortcomings and provide solutions. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The government was committed to achieving the development objectives. During project implementation, the government was proactive in overcoming enrollment and information system challenges. When the project experienced low enrollment levels and disbursement was less than one percent of the annual operating plan budget during the initial 18 months, the government adjusted the tool that was being used by the Regional Health Services to identify the eligible population. Also, the government made effective use of the clinical management system (CMS), which was important for monitoring indicators at the primary care level, making it mandatory for all health staff. However, due to a tight fiscal situation, the government could not provide a significant amount of the agreed counterpart financing. The government delayed addressing this issue with the Bank and officially asking for an increase in Bank financing, leading to implementation delays. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The project was implemented by two Implementing Agencies. The Project Implementation Unit (PIU) was located within the Ministry of Health and was responsible for the implementation of components 2 and 4. The Project Coordination Unit (PCU) was located within the Executive Commission for Health Sector Reform (CERSS) and was responsible for the implementation of components 1 and 3. The two agencies coordinated implementation effectively, carried out regular visits to the Regional Health Services (RHS), organized review and implementation workshops to promote knowledge exchange, and conducted studies and surveys. However, there were some moderate shortcomings. Project implementation experienced delays due to changes in CERSS’ key staff including the technical coordinator, the manager of the financial management unit, and the Results Based Financing component coordinator. Furthermore, some technical and audit reports and consultancies were not delivered on time. The Mid-Term Review found that while the PIU was conservative in managing operating costs, the PCU did not contain operating costs effectively, especially in terms of number of staff hired and vehicles used. These issues were successfully resolved in the latter part of the project period. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) The objectives of the project were clearly specified and well reflected in the indicators in the project’s Results Framework. The Results Framework included five Project Development Indicators and 11 Intermediate Outcome Indicators. The majority of indicators had a baseline, and all indicators were measurable in terms of numbers, timing, and location. Some indicators were used for project evaluation and for implementing the Results Based Financing mechanism. The M&E design included internal oversight and external concurrent audits to be conducted by an external firm. A new unit in charge of national coordination of monitoring, information and evaluation systems within the MOH was set up to be in charge of developing the information system required for results-adjusted capitation. b. M&E Implementation During the project restructuring in June 2014, the Results Framework was revised to reflect the expansion of the project to region VII. Also, targets for some indicators based on the target population expansion had to be aligned. Indicators were measured on a regular basis. Due to delays in the development of key applications of the information system, the baseline data for the package of health service indicators was not established until 2011. The M&E was owned and utilized by various stakeholders. The Results Based Financing mechanism used ten indicators of the Results Framework against which the variable capitation payment had to be paid. The amount of the capitation was based on the achievement of the annual targets, which were set by the MoH. The Clinical Management System was used as a main source of data for several indicators, especially for tracking the performance of component 1. In order to encourage doctors to enter patient health information electronically during consultations, the MoH made use of the system mandatory and provided incentives, training, and logistical and technical support. c. M&E Utilization The Regional Health Services used the Results Framework to inform decision-making. Establishing the M&E system as the basis of primary care management fostered a culture of M&E. The Results Based Financing mechanism had a positive impact on the utilization of the Results Framework due to the required ongoing indicator data reporting. M&E Quality Rating High 11. Other Issues a. Safeguards According to the PAD (Annex 10), the project was originally classified as category B, but then was reclassified as category C because no civil works were to be financed and no adverse environmental impact was anticipated. The PAD (p. 82) states that the project triggered safeguard policy OP/BP 4.01 (Environmental Assessment), but the ICR (p. 16) states that OP/BP 4.01 was not triggered. The project team stressed that OP/BP 4.01 was triggered only due to the project's proposed support to the government to develop a strategic approach to address biomedical waste management in the health sector, and the ICR confirms that an environmental assessment was conducted for this purpose. Technical assistance was provided and training activities were conducted to support the differentiated management of biomedical and infectious waste generated in health centers in Santo Domingo. According to the ICR (p. 16), the implementation of safeguard activities was satisfactory. b. Fiduciary Compliance Financial Management Financial Management was a critical function for this project due to the Results Based Financing mechanism of component 1. A separate Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) concurrent audit of component 1 verified that its funds were disbursed in a timely fashion, used for project purposes, and complied with the project’s operation manual. The borrower submitted audit reports on time, and they were generally unqualified. Internal controls and flows of funds experienced some weaknesses because counterpart financing was insufficient. The Bank conducted a detailed review of project expenditures and commitments, and based on its findings increased its share of project financing and lowered the share of counterpart financing at the government’s request. Financial Management was rated Moderately Satisfactory beginning in mid-2011 due to one region's delay in implementing the required Financial Management information system and failure to submit audit reports on time. Procurement At the initial stage of the project, procurement plans were submitted in a timely manner and procurement specialists with experience in Bank operations were hired in both implementing agencies. The procurement rating was downgraded from Satisfactory to Moderately Satisfactory in July 2012 when a post procurement review identified procurement issues related to terms of reference, technical specifications, and some shortcomings in evaluation processes. No procurement assessment under supervision was completed. The ICR (p. 16) states that, overall, there was compliance with the Bank's fiduciary guidelines. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Outcome Modest Modest --- A critical risk related to the availability of counterpart financing was not adequately considered during preparation. Also, there was a lack of clarity regarding the two Bank Performance Satisfactory Moderately Satisfactory implementing agencies’ responsibility and autonomy in the areas of operations, procurement, and fiduciary management, leading to implementation challenges. Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Modest --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DO (APL2) Health Ref II (P106619) The ICR (pp. 29-31) presents lessons learned, including the following (adapted by IEG): • During project preparation, the feasibility of financial contributions by the counterpart needs to be carefully evaluated. In this project, the government could not make the contribution that was originally promised, leading to implementation delays until the Bank increased its share of project financing. • Results Based Financing is a powerful mechanism to improve the quality of health care service provision. However, it can also create resistance, since the mechanism puts more responsibility on health staff. Therefore, communication and availability of information on potential benefits is critical. In this project, resistance towards the new financing mechanism diminished once the benefits materialized. • When using two different implementing agencies, it is critical to ensure close coordination and take advantage of synergies to avoid duplication of activities and hiring of new staff resulting in higher operational costs. In this project, the PCU faced high operating costs due to staffing and vehicles used. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR provides a good overview of project preparation and implementation and is relatively concise. Unlike the PAD, the ICR does not include a traditional economic analysis but instead provides a comparative analysis of outcomes. Also, the ICR does not include detailed information on what outputs the project produced. Although the ICR presents many lessons learned, most of these lessons come across as quite generic and do not provide adequate project-specific information to allow for a deeper understanding of the emergence of learning and know-how from the project's preparation and implementation experience. a. Quality of ICR Rating Modest