Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) Report Number : ICRR0022205 1. Project Data Project ID Project Name P144871 Improving Quality and Efficiency of Heal Country Practice Area(Lead) Croatia Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-83650 30-Jun-2018 72,732,451.75 Bank Approval Date Closing Date (Actual) 08-May-2014 31-Oct-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 103,500,000.00 0.00 Revised Commitment 88,378,650.00 0.00 Actual 72,732,451.75 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Loan Agreement (p. 5), the project's objectives were "to improve the quality of health care and efficiency of health services in the Republic of Croatia". b. Were the project objectives/key associated outcome targets revised during implementation? No Page 1 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) c. Components The project employed a Program-for-Results (PforR) instrument with ten disbursement-linked indicators (DLIs). Its planned activities were organized around five priority areas (drawn from the eight priorities of the 2012-2020 National Health Care Strategy; see Section 3a): Strengthening management capacity in health care (estimated cost in the government program, 2012- 2017: EUR 10 million) and reorganizing the structure and activities of health care institutions (estimated cost: EUR 125 million): Implementing a hospital master plan; implementing hospital reforms and governance and management changes; promoting group practices for primary health care doctors; expanding secondary-level outpatient services, including high-resolution outpatient centers; and redefining long-term health care services and palliative care. Fostering quality in health care (estimated cost: EUR 25 million) and strengthening preventive activities (estimated cost: EUR 14 million): Implementing hospital accreditation; implementing Health Technology Assessment (HTA) of selected new health technologies; building a body of clinical protocols and care pathways; detecting and proper recording of specific "sentinel events for quality"; implementing technical audits and payment mechanisms to incentivize the use of clinical guidelines; and using the existing e-prescription system for quality control purposes. Preserving financial stability of health care (estimated cost: EUR 6 million): Developing centralized procurement of medical and non-medical supplies; rationalizing non-medical services; strengthening the performance-linked component in payments to hospitals and outpatient services; and strengthening the Ministry of Health's (MOH's) capacity to develop and present proposals to be financed by European Union (EU) structural funds. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates The estimated cost to achieve program objectives over the 2012-2017 period was EUR 180 million, with the project to cover EUR 75 million (approximately US$ 103.5 million), and the government the balance of EUR 105 million. The government planned to seek EU funds, if necessary, to partially finance its program; although this presented a financing risk, it was perceived that the project constituted an opportunity to help the government fulfill the ex-ante conditions to absorb EU funds and support the efficient use of these resources (Project Appraisal Document, PAD, p. 8). The total amount of loan proceeds was to be divided into ten equal allocations to the ten DLIs, with each allocation broken into two equal sub- allocations to be disbursed upon verified achievement of two consecutive targets for each DLI. In addition, to support fast start-up of activities, an advance equivalent to one DLI (EUR 7.5 million) was to be disbursed once the program was declared effective. EUR 64 million (US$ 72.7 million) was actually disbursed. EUR 10.96 million was cancelled due to the dropping of one DLI and partial reduction of another (see Section 8b). The program was approved on May 8, 2014 and became effective on September 8, 2014. It underwent a mid-term review on March 1, 2017. The closing date was extended from June 30, 2018 to October 31, 2019 at a June 27, 2018 restructuring, which also amended the results framework and cancelled part of the loan. Page 2 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) 3. Relevance of Objectives Rationale The project's objectives were highly relevant to health sector needs. At the time of appraisal, population aging in Croatia was decreasing available public financing for the health sector due to shrinkage of the working-age population while simultaneously increasing demand for health services. The country's recent entry into the EU in 2013 created obligations to align its health care strategy to fit with EU norms and regulations, but also necessitated fiscal tightening in the health sector. Its health system had been producing reasonable outcomes but at high costs that were considered unsustainable (PAD, p. 1). Hospitals continued to provide services that would be better and more cost-effectively delivered in an outpatient setting, and primary care was not acting as an effective gatekeeper. Hospital arrears put substantial financial pressure on the system, accounting for nine percent of Croatia Health Insurance Fund (HZZO) revenue in 2014. Although some progress had been made in implementing improved quality standards, many norms and protocols aligned with best international practice had not been adopted. The program built on a previous investment project financing operation, the Development of Emergency Medical Services and Investment Planning project (2008-2013, US$ 132.8 million), that developed a hospital rationalization plan, assessed options for rationalization of medical and non-medical hospital services, implemented a communications campaign to inform the public of the benefits of health sector reform, designed business process re-engineering in the HZZO, and implemented a geographic information system to improve data availability. The program supported the public finance pillar of the Country Partnership Strategy at appraisal (2014- 2017), which specifically envisaged a results-based engagement in the health sector to support reform efforts that would sustain good health outcomes at lower cost. It remains highly relevant to the Country Partnership Framework (2019-2024), with priority areas specifically related to improving the quality and efficiency of service delivery, including (in the health sector) promoting primary care practice, day hospitals, and hospital functional integration to create more cost-effective organizational settings and rapid-access care for patients. The objectives also directly supported Croatia's 2013 Economic Program, which planned for rationalization of hospitals as part of a first phase of health system reform. They were highly aligned with the 2012-2020 National Health Care Strategy, which identified priority issues: poor connectivity and insufficient continuity of health care across the primary, secondary, and tertiary levels; uneven or unknown quality of care; inadequate efficiency and effectiveness; poor or uneven availability of health services across regions; and relatively poor health indicators, especially those related to risk factors and health behaviors. Of the eight action areas intended to address these issues, five were explicitly addressed by the project: strengthening management capacity, reorganizing the structure and activities of health care institutions, fostering quality of care, strengthening preventive activities, and preserving financial stability of the sector. Rating High Page 3 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Improve the quality of health care. Rationale The program's interventions to expand secondary-level outpatient services, including high-resolution ambulatory centers, could be expected to increase the provision of non-invasive diagnostic and treatment procedures. Implementation of hospital accreditation, use of HTA to select new heath technologies, building a body of clinical protocols and care pathways, detection and recording of specific "sentinel events for quality," implementation of technical and clinical audits and payment mechanisms to incentivize the use of clinical guidelines, and use of the existing e-prescription system for quality control purposes could all be expected to strengthen quality control mechanisms and to increase the use of clinical guidelines for diagnosis and treatment. Improved quality control and implementation of clinical guidelines would reasonably lead to improved quality of health care. Outputs: A quality control mechanism was established with protocols and tools for technical audits of hospitals. A sentinel event surveillance system was established in hospitals with surgery wards. The HZZO Service for Control Directorate now conducts regular inspections of hospitals as well as specialized reviews based on risk assessments or significant complaints about health services. Based on technical audits in the preceding calendar year, there is public disclosure on the MOH website of the best-performing hospitals (including their specific results). However, political shifts, including four different health ministers over the course of program implementation, produced instability in key policies affecting the program. In 2015, the government decided to abandon the idea of implementing national accreditation for hospitals and instead move to international accreditation standards, and then in 2018 shifted back to a voluntary domestic accreditation procedure. As a result, there was "lack of awareness of hospitals of the process and benefits of accreditation, and thus, no perceived incentive for hospitals to apply for a voluntary and long process of accreditation" (ICR, pp. 12-13). Following public consultations in August 2019, new ordinances on accreditation were adopted and published. In addition, from 2016-2018, shares of performance- and volume-based payments to hospitals decreased, leaving hospitals with reduced incentives to provide more care to patients or to improve performance indicators. Intermediate outcomes: 99.43% of primary health care group practices had achieved set performance indicators and therefore accessed payment incentives in 2019, exceeding the target of 60%. The number of counties with specialized services for palliative care increased from one in 2012 to 19 in 2019, exceeding the target of 15. All secondary and tertiary hospitals had put nosocomial infection surveillance systems in place in 2019, surpassing the target of 30% of hospitals. Page 4 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) 82% of hospitals with surgery wards that had established safety-related sentinel surveillance schemes were reporting rates of specific events in 2019, surpassing the target of 60% (DLI 10). The percentage of doctors who were identified as overprescribing drugs and who engaged in a discussion with HZZO on a corrective course of action over the preceding six months increased from 20% in 2012 to 93.1% in 2019, exceeding the target of 90% (DLI 7; this DLI also would have contributed to the efficiency objective). Patient satisfaction, as measured by perception of increased responsiveness, remained stagnant at 38% in 2012 and 38.6% in 2017. There was no survey in 2018, and results from a follow-up survey in 2019 were not available at the time of ICR preparation. These patient satisfaction data may be explained by growing wait times, since hospitals were not reimbursed on the costs if more patients received care and treatment from 2016-2018 (ICR, p. 25). The percentage of best-performing rationalized hospitals that were publicly disclosed, including results, based on technical audits over the preceding 12 months improved from zero in 2012 to 90% (28 of 31 hospitals) in 2019, exceeding the target of 40% (DLI 5). Outcomes: An external hospital audit conducted in May-July 2019 found that all 33 acute care hospitals assessed in December 2018 met or exceeded 80% compliance with mandatory quality standards, exceeding the original target of 50% and revised target of 80% (DLI 6). This achievement was characterized by the ICR (p. 49) as having put "foundations in place for accelerated implementation of hospital accreditation." The ICR (p. 18) provided hospital-specific data suggesting that, as a result of these improvements, 13 of the 15 hospitals with the worst in-hospital mortality rates were able to substantially reduce mortality between 2015 and 2018, with mortality reductions ranging between -4% and -52% (ICR, p. 18). Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Improve the efficiency of health services. Rationale The program's interventions to implement a hospital master plan, facilitate hospital reform and governance and management changes, promote group practices for primary care physicians, and redefine long-term health care services and palliative care could be expected to increase the gatekeeper function of primary health care, differentiate acute care services from long-term care inpatient services, and rationalize health facilities through reprofiling and merger schemes. Improved development of centralized procurement, rationalization of non-medical services, and strengthening of performance-linked components in payments to hospitals and ambulatory services could be expected to lead to reductions in health expenditure while Page 5 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) preserving access and quality of care. This overall rationalization of the structure of health services delivery and managerial improvements would reasonably lead to improved efficiency of health services, measured as reduced public expenditures in health as a proportion of total public expenditures. Outputs: A hospital rationalization plan was completed and approved by MOH. Feasibility and pre- investment studies for "hospital reshaping schemes" were completed and delivered in June of 2019. Higher fee-for-service payments were established for outpatient-based procedures and surgeries, compared to a fixed pre-payment for the analogous inpatient services. The centralized procurement system was expanded. Internal financial audit units were improved in tertiary hospitals. HZZO now carries out annual calculation of prices of medicines and public tenders for determining the prices of medicines for basic and supplementary medicines lists. HZZO's departments for analytics and development of health services continue to analyze and review diagnosis-related group and diagnostic and therapeutic procedure costing. However, in 2016, the government cancelled a 2013 Law on Hospital Rehabilitation, reversing momentum for hospital optimization and especially for addressing arrears (the 2013 Law had transferred hospital ownership from counties to the MOH, mainly to improve management of arrears; the cancellation of the law turned ownership back to the counties). As a result, DLI 3 (an original outcome indicator) for percentage of rationalized hospitals without arrears incurred during the preceding calendar year was dropped. Intermediate outcomes: Two "hospital reshaping scheme" projects were implemented, achieving the target (DLI 2). The percentage of all surgeries included on the elective surgeries list performed as outpatient surgeries increased from 5% in 2012 to 60.94% in 2019, meeting the target (DLI 4). A value of 63.34% was achieved in mid-2018, but due to reconstruction activities to equip premises for day hospitalization and day surgeries, conducted by a majority of hospitals since late 2016 (with EU funding), the value dropped to 55.8% in December 2018 and then increased again to meet the target by the end of 2019. The ratio of primary care to secondary outpatient care to hospital inpatient care services improved from 58/18/1 in 2012 to 83/22/1 in 2019, not quite meeting the target of 90/19/1. The ICR did not clarify what this indicator measured. The project team explained that these ratios referred to numbers of services; for example, for every one hospital inpatient care service performed in 2019, 22 secondary outpatient care services and 83 primary care services were performed. 70.33% of primary care doctors were working in group practices by 2019, exceeding the target of 50% (DLI 9). However, the ICR (p. 20) noted that participation of doctors in group practices did not help improve the desired gatekeeping role of primary care, as an unspecified increase in referring patients from primary care and utilization of specialist services was observed. The percentage of total public spending on medical consumables, drugs, and medical devices for hospital services made through centralized procurement reached 44.53% in 2019, not meeting the original target of 60%, but exceeding the revised target of 32.34% (DLI 8). Outcomes: The number of hospital beds in rationalized hospitals classified as acute care beds decreased from 15,930 in 2012 to 12,315 in 2019, exceeding the target of a reduction to 12,800 beds (DLI 1). The project's original outcome indicator on reduction of hospitals with arrears was dropped. Page 6 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale Overall Efficacy Rating --- 5. Efficiency --- Efficiency Rating --- a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The program was highly relevant to country context at appraisal and to Bank and government strategy across its lifetime. The DLIs were substantially relevant, as they were well-defined and reasonably covered achievement of the program's objectives, though achievement of at least one depended on factors largely outside the program's control, and total financing may have been inadequate to incentivize desired outcomes. The objectives to improve quality and efficiency of health services were substantially achieved, but with moderate shortcomings related to incomplete development of accreditation processes, inadequate performance of primary Page 7 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) care physicians as gatekeepers, and persistent hospital arrears. Overall, these findings reflect moderate shortcomings in the program's preparation and implementation, producing an Outcome rating of moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Political risk is presumably low, as the MOH, HZZO, and EU are currently committed to improving the quality and efficiency of health services, though the ICR did not address the potential for political shifts to impact related health sector policies. The project team later confirmed that a new administration, with a four- year term, is committed to continuing the reforms established by the program, as evidenced in a recent Letter of Development Policy accompanying a 2020 Emergency Development Policy Operation. The MOH has continued to reduce unnecessary acute bed capacity, though the number of beds per capita remains above EU averages. Interviews with MOH staff for the ICR confirmed commitment to the hospital accreditation process (ICR, p. 30). The ICR, however, raised questions about the adequacy of MOH technical capacity, especially in the area of centralized procurement. The program mobilized 156 projects for support from the EU, all aligned with the objectives of the National Health Care Strategy, and the project team highlighted that the EU's Enabling Conditions focus on hospital network optimization and build on the approach of the PforR. 8. Assessment of Bank Performance a. Quality-at-Entry The program was innovative, the first of its type in the ECA region and the first PforR in the Bank's health portfolio. The choice of a PforR instrument was appropriate given government commitment at appraisal to a focus on results, the existence of a defined reform program (the 2012-2020 National Health Care Strategy), strong ownership and implementation readiness among key client institutions, the need to align the incentives of the Ministries of Health and Finance, and the role strengthening of country systems would play in enabling Croatia to meet conditionalities for EU funding mechanisms. Inputs and institutional capacity were not the main constraints to achieving results in the health sector (so investment lending was not appropriate), and the necessary legislation was in place, having already been supported for five years by a series of development policy operations. A cost-benefit analysis performed at appraisal was positive, estimating an internal rate of return of 4.85 to 9.83 percent for the first program phase (2012-2017), and 17.38 to 24.52 percent for the full program duration (2012-2020). There were moderate shortcomings. DLI allocations did not provide sufficient incentives to stimulate efforts toward achieving set targets (ICR, p. 17). As discussed in Section 8a, the program's outcome and intermediate results indicators were largely output-focused, though it should be acknowledged that PforRs are commonly focused on institutional changes, outputs, and early outcomes. In addition, while the overall Page 8 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) risk of the operation was assessed as moderate at appraisal (PAD, pp. 18-19 and Annex 7), given strong country systems and implementation capacity but the wide range and complexity of proposed reforms, it was assumed that the existing reform agenda would be continued. The risk of changes in MOH leadership (which materialized) was not anticipated, and there was no mitigation strategy for political volatility. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Implementation support was adequate but inconsistent. The mid-term review was conducted seven months later than scheduled (in March of 2017), and its observations and recommendations were not acted upon in a timely manner by the MOH. Implementation performance was downgraded to moderately unsatisfactory in October of 2017, but project restructuring was not finalized until June 2018, a delay the ICR (p. 29) attributed primarily to low levels of participation and commitment at that point from the MOH team. According to the ICR (p. 29), the Bank team's monitoring documents were candid about the program's progress, providing realistic signals to the MOH when progress faltered. Task team leadership changed three times over the course of the program; the ICR did not indicate whether these changes had an impact on implementation. The project team later confirmed that the handovers were smooth, facilitated by continuity in the technical teams. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The results framework at appraisal included four indicators that were labeled as outcome indicators, though both these and the designated intermediate outcome indicators focused primarily on outputs. Four of the ten DLIs were intended to demonstrate progress toward those four designated outcome indicators, while the other six DLIs were to serve solely as intermediate results indicators (PAD, pp. 27-31). The outcome and intermediate results indicators corresponded well with the DLIs and were mutually reinforcing. Measurement and verification of progress toward achievement of objectives were to be based on the country's existing M&E systems. The HZZO was to collect monitoring data and verify documentation for most of the intermediate results indicators and report aggregate data to the MOH on a semiannual basis. The MOH, through its Directorate for Health Protection, was to assemble all data and documentation necessary for monitoring, verification, and evaluation purposes. Page 9 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) b. M&E Implementation Over the program's first two years, monitoring was consistent and effective. With political changes and shifts in MOH leadership beginning in 2016, there were delays and gaps in reporting. At the 2018 restructuring, two outcome indicators (related to hospital arrears and hospital accreditation) were dropped because they were not likely to be achieved (ICR, pp. 11-12). Two intermediate results indicators (related to adoption of clinical protocols) were also dropped because they were not likely to be achieved, and the target for the intermediate results indicator on centralized procurement was lowered. Concurrently, the DLI on hospital arrears was dropped; the DLI on hospital accreditation was lowered from achievement of accreditation to "foundations are in place for accelerated implementation" of accreditation; and the target value for the DLI on centralized procurement was lowered. After the restructuring, M&E improved, with the MOH submitting monthly progress reports to the Bank and the Bank team holding regular video-conference sessions to discuss progress, issues, and any necessary corrective actions (ICR, p. 27). c. M&E Utilization The ICR (p. 27) stated that "good practices in systematic utilization of the M&E information of the program were observed." Status updates on key activities fed into the ongoing design of contracting arrangements between HZZO and service providers. M&E Quality Rating Substantial 10. Other Issues a. Safeguards At appraisal, an Environmental and Social System Assessment took into consideration the requirements of OP 9.00 (Program-for-Results Financing) to identify any adverse environmental or social impacts the program could generate (PAD, p. 17 and Annex 6). The program's social system was assessed as adequate, without substantial negative impacts on society. The initial environmental screening suggested that activities financed through the program would generate positive cumulative environmental impacts with improved overall health status of the majority of citizens. Potential negative impacts due to reconstruction works were not considered to be significant, given that Croatia's regulatory and institutional framework was sufficient to guarantee environmentally sound construction management. The ICR (p. 27) reported that safeguards performance was satisfactory until April of 2017, when the rating was downgraded to moderately unsatisfactory due to slow progress in implementing items in the Program Action Plan related to environmental safeguards. In July 2018, following the program's restructuring, the rating improved to moderately satisfactory based on progress in activities to assess compliance of hospitals with mandatory quality standards and environmental requirements. The rating remained moderately satisfactory until program closure, and all environmental actions in the post-restructuring Program Action Plan were completed. Page 10 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) b. Fiduciary Compliance An integrated fiduciary assessment conducted at appraisal found that the overall fiduciary and governance framework of the program was adequate to support implementation. The ICR (p. 28) reported that fiduciary performance was rated satisfactory throughout implementation. Disbursement was carried out upon verification of DLIs. All financial management reports were submitted on time, and no issues were identified (ICR, p. 28). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory There were moderate shortcomings in Quality at Moderately Bank Performance Satisfactory Entry related to risk Satisfactory assessment and DLI allocations. Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR (pp. 30-31) presented a series of insightful lessons that should prove useful to the preparation and implementation of other PforR operations, including: Priority areas selected for PforR will underperform if not matched to proposed financing, with adequate financial incentives for achieving expected results. In this case, the support provided through the PforR amounted to less than ten percent of the total cost of the National Health Care Strategy while supporting five of its eight priorities. Spreading the financing so thin limited the financial incentives to reach ambitious targets. Page 11 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Improving Quality and Efficiency of Heal (P144871) Political economy analysis that assesses potential changes in the political arena and crafts flexible mitigation strategies is an important element of project preparation. In this case, unanticipated changes in MOH leadership produced shifts in policy that undermined attempts to establish a hospital accreditation system and address a backlog of hospital arrears. 13. Assessment Recommended? Yes Please explain This project merits further assessment as the first PforR in the Bank's health portfolio and the first in the ECA region. 14. Comments on Quality of ICR The ICR was candid, clearly written, internally consistent, reasonably concise (31 pages), and sufficiently evidence-based. It usefully connected the evolution of the overall health policy environment in the country to the course of program implementation, including a summary table of key changes to the health sector during the program period (Table 3, p. 14). The ICR's presentation of the program's theory of change (p. 8) focused on the 2012-2020 health strategy's priority areas, rather than the program's objectives, as desired outcomes. Its discussion of the Relevance of DLIs (pp. 16-17) was thin; for example, it referred to "small shortcomings" with the definitions of some indicators without providing examples or detail on those shortcomings. a. Quality of ICR Rating Substantial Page 12 of 12