FOR OFFICIAL USE ONLY Report No: ICR00004883 INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT AND/OR INTERNATIONAL DEVELOPMENT ASSOCIATION IMPLEMENTATION COMPLETION AND RESULTS REPORT IBRD 83650 ON A LOAN IN THE AMOUNT OF EUR75.0 MILLION (US$103.5 MILLION EQUIVALENT) TO THE REPUBLIC OF CROATIA FOR THE IMPROVING QUALITY AND EFFICIENCY OF HEALTH SERVICES PROGRAM-FOR-RESULTS June 17, 2020 (This Implementation Completion and Results Report replaced the version published in the Board Operations System on June 18, 2020. Ratings in datasheet have been changed to match the ratings in the main report text.) Health, Nutrition & Population Global Practice Europe And Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 3, 2020) Currency Unit = EUR EUR 0.89 = US$1 US$ 1.12 = EUR 1 FISCAL YEAR January 1 – December 31 Regional Vice President: Cyril E Muller Country Director: Arup Banerji Regional Director: Fadia M. Saadah Practice Manager: Tania Dmytraczenko Task Team Leader(s): Huihui Wang, Baktybek Zhumadil ICR Main Contributor: Olena Doroshenko ABBREVIATIONS AND ACRONYMS AQAHS The Agency for Quality and Accreditation in Health and Social Care AMI Acute Myocardial Infarction CPS Country Partnership Strategy DLI Disbursement-linked indicator E&S Environmental and social ESSA Environmental and Social Systems Assessment EU European Union GDP Gross domestic product HTA Health technology assessment HZZO Croatia Health Insurance Fund ICR Implementation Completion and Results report KPI Key performance indicators M&E Monitoring and evaluation MoF Ministry of Finance MoH Ministry of Health NCD Non-communicable diseases OECD Organisation for Economic Co-operation and Development PAD Program appraisal document PAP Program Action Plan PDO Program development objectives PforR Program for Results TA Technical assistance TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROGRAM CONTEXT AND DEVELOPMENT OBJECTIVES.................................................... 4 A. CONTEXT AT APPRAISAL AND THEORY OF CHANGE .................................................................4 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION .............................................................. 11 II. OUTCOME .................................................................................................................... 16 A. RELEVANCE .......................................................................................................................... 16 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 17 C. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 21 D. OTHER OUTCOMES AND IMPACTS ........................................................................................ 22 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 24 A. KEY FACTORS DURING PREPARATION ................................................................................... 24 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 25 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 26 A. QUALITY OF MONITORING AND EVALUATION ....................................................................... 26 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 27 C. BANK PERFORMANCE ........................................................................................................... 28 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 29 V. LESSONS AND RECOMMENDATIONS ............................................................................. 30 ANNEX 1. RESULTS FRAMEWORK, DISBURSEMENT LINKED INDICATORS, AND PROGRAM ACTION PLAN ...................................................................................................................... 32 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 56 ANNEX 3. BORROWER’S COMMENTS ................................................................................... 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) DATA SHEET BASIC INFORMATION Product Information Program ID Program Name Financing Instrument Improving Quality and Efficiency of P144871 Program-for-Results Financing Health Services Program for Results Country IPF Component Croatia No Organizations Borrower Implementing Agency Government of Croatia Ministry of Health Program Development Objective (PDO) Original PDO The objective of the Program is to improve the quality of health care and efficiency of health services in the Republic of Croatia. PDO as stated in the legal agreement The objective of the Program is to improve the quality of health care and efficiency of health services in the Republic of Croatia. FINANCING FINANC E_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Administered Financing 103,500,000 88,378,650 72,732,452 IBRD-83650 Total 103,500,000 88,378,650 72,732,452 Total Program Cost 103,500,000 88,378,650 72,732,452 Page 1 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) KEY DATES FIN_TABLE_DAT Program Approval Effectiveness MTR Review Original Closing Actual Closing P144871 08-May-2014 08-Sep-2014 01-Mar-2017 30-Jun-2018 31-Oct-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 27-Jun-2018 47.71 Change in Results Framework Cancellation of Financing Reallocation between and/or Change in DLI Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROGRAM PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 17-Aug-2014 Satisfactory Satisfactory 0 02 05-Oct-2014 Satisfactory Satisfactory 0 03 15-Dec-2014 Satisfactory Satisfactory 9.39 04 08-Jun-2015 Satisfactory Satisfactory 9.39 05 24-Oct-2015 Satisfactory Satisfactory 9.39 06 19-Apr-2016 Satisfactory Satisfactory 9.39 07 06-Oct-2016 Satisfactory Satisfactory 47.71 08 17-Apr-2017 Moderately Satisfactory Moderately Satisfactory 47.71 Moderately 09 20-Oct-2017 Moderately Unsatisfactory 47.71 Unsatisfactory Moderately 10 24-Apr-2018 Moderately Unsatisfactory 47.71 Unsatisfactory Page 2 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 11 30-Nov-2018 Moderately Satisfactory Moderately Satisfactory 47.71 12 25-Jun-2019 Moderately Satisfactory Moderately Satisfactory 56.26 13 30-Oct-2019 Moderately Satisfactory Moderately Satisfactory 56.26 SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Public Administration - Health 10 Health 90 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 10 Public Private Partnerships 10 Human Development and Gender 100 Health Systems and Policies 100 Health System Strengthening 100 ADM STAFF Role At Approval At ICR Regional Vice President: Laura Tuck Cyril E Muller Country Director: Mamta Murthi Arup Banerji Director: Alberto Rodriguez Fadia M. Saadah Practice Manager: Daniel Dulitzky Tania Dmytraczenko Huihui Wang, Baktybek Task Team Leader(s): Carlos Marcelo Bortman Zhumadil ICR Contributing Author: Olena Doroshenko Page 3 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) I. PROGRAM CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL AND THEORY OF CHANGE Context 1. The Health System Quality and Efficiency Program for Results (“Program”) was launched in 2014, after Croatia joined the European Union (EU) in July 2013 and started recovery from the economic recession of 2008- 2013. The country experienced declines in personal consumption, exports, and investments, negative annual gross domestic product (GDP-see Figure 1), and unemployment rates of 17-18 percent in 2013-2014, the highest among the new EU member states. The country has experienced shortages of resources and was keen to engage funds for social and economic development and efficiency optimization. Figure 1: GDP Growth Rate and GDP Per Capita (purchasing power parity, PPP) in Croatia in 2008-2018 30000 6 25000 4 2 20000 0 15000 -2 10000 -4 5000 -6 0 -8 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 GDP per capita, PPP (current international $) GDP growth (annual %) 2. The country was spending about 7-8 percent of GDP on health in 2010-2014, significantly more than countries with similar GDP per capita in the region. Such high spending allowed the Government to keep the out-of-pocket expenditures at around 15 percent, despite the post-crisis decreasing per capita spending on health in US$ current equivalent (see Figure 2). The main operator of health financing funds in Croatia – Croatia Health Insurance Fund (HZZO) – is responsible for both collecting mandatory health insurance contributions since 2015 and contracting providers. Page 4 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Figure 2: Health Spending in Croatia in 2008-2016 18.0 1,259 15.7 15.5 15.4 1,400 1,188 1,132 16.0 1,126 1,200 1,030 14.0 931 958 14.5 838 884 1,000 12.0 13.7 14.0 10.0 12.8 800 12.1 11.5 8.0 600 6.0 7.7 8.2 8.1 7.8 7.8 6.8 7.1 7.2 7.2 400 4.0 2.0 200 0.0 - 2008 2009 2010 2011 2012 2013 2014 2015 2016 Current health expenditure per capita (current US$) Current health expenditure (% of GDP) Out-of-pocket expenditure (% of current health expenditure) 3. Health spending structure and organization of service delivery in Croatia was not optimal. Before the implementation of the Health System Quality and Efficiency Program for Results, Croatia had more hospital beds (5.2 beds in Croatia compared to EU average of less than 4 beds per 1,000 population in 2012). Primary care physicians were mainly working in solo practices without payment incentives linked to performance indicators. Approximately 80 percent of the total health expenditure was spent on curative care (50 percent) and medical goods (28 percent). The spending on curative care gradually increased between 2005 and 2016. Relative to other EU countries, Croatia spent less on primary care (at less than 20 percent, it ranks the seventh lowest) and more on hospital care (at 42 percent, it ranks as the fourth highest), and very little on long-term care (at less than 1 percent, it ranks as the second lowest). 4. One of the key issues facing the health financing has continued to be a problem of arrears. Hospital arrears presented a substantial financial pressure on the system, accounting for about 9 percent of total HZZO revenue in 2014 (and 10 percent in 2018) or about 0.8 percent of GDP. More than 85 percent of hospital arrears are for drugs and medical supplies, almost two thirds of which (about 60 percent) had incurred by tertiary hospitals and about one third by general hospitals. The arrears first appeared as an issue after the Croatian Homeland War in the mid-1990s, as the result of decreased financing of the health sector. The arrears further increased after opening the market for drugs following joining the World Trade Organization in 2000. From 1995, injections from the state budget to cover health care arrears were made in 14 years during the 20 subsequent years. The chronic debts generated by hospitals for drugs negatively affect the prices of drugs, as drug suppliers include the financial risk of receiving delayed payments in the prices they charge for drugs. 5. Croatia entered EU in 2013 with many health outcomes indicators below the EU member states. Life expectancy at birth of 77.3 years in 2013, although about 3 years lower than in the EU, was growing steadily in Croatia. Mortality rates from cardiovascular disease were almost twice as high as the EU average and mortality rates from lung, breast and colon cancer were among the highest in the EU, pointing to shortcomings in health care delivery and public health interventions.1 1 State of Health in the EU: Croatia. Country Health Profile 2017. OECD and World Health Organization. https://www.oecd- ilibrary.org/social-issues-migration-health/croatia-country-health-profile-2017_9789264283312-en Page 5 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 6. The quality and efficacy of care delivery in Croatia had a lot of potential for improvement. The amenable mortality rate for Croatia (216.4 in 2015) was almost twice the average rate for the EU and almost 3 times the rate for top performing countries such as Denmark, Norway, and Switzerland. One of the reasons for the existing gap was the quality of hospital care and the space to further improve the outcomes of acute care for acute myocardial infarction (AMI), strokes2, and cancers. 7. At the time of Program preparation, the leadership was eager to demonstrate improvements in the Croatian health system before and after the EU accession. The Bank’s counterparts on the Government side were keen to implement quick and effective measures to reduce the gap between Croatia and the EU in terms of health outcomes and health efficiency indicators. The implementation capacity and coordination between all involved agencies (Ministry of Finance [MoF], Ministry of Health [MoH] and HZZO) were strong and forward- looking. 8. To address the issues of efficiency and quality in health, the Government of Croatia adopted the National Health Care Strategy 2012–2020. It identified the strategic challenges and reform priorities for the health care sector, including (a) poor connectivity and insufficient continuity of health care across levels (primary, secondary and tertiary) in the health system; (b) uneven or unknown quality of care; (c) inadequate efficiency and effectiveness of the health care system; (d) poor or uneven availability of health care across regions; and (e) relatively poor health indicators, particularly those related to risk factors and health behaviors. 9. The National Health Care Strategy defined eight main priorities, five of which were supported by the Program. The priorities of the National Health Care Strategy and the financing of the Program are presented in Table 1; it also provides the boundaries for the Program in terms of the supported priorities of the strategy. The implementation of the National Health Care Strategy was not formally budgeted at the time of the Program preparation, but the Government provided an estimated cost of the Program at the total of EUR 409 million. The implementation of the Program was supposed to be financed mainly from the Government funds, with the support from the Bank’s loan and EU funds. Table 1: Priorities of the National Health Care Strategy and Boundaries of the Program In EUR million Priorities of the National health Care Strategy 2012- Government Actual No. WB 2020 financing in financing for financing 2012-2020 2012-2019 i. Developing a Health Information System and eHealth 45.0 955.0 ii. Strengthening and better using human resources in 12.0 health care iii. Strengthening management capacity in health care 14.0 64.03 iv. Reorganizing the structure and activities of health 260.0 care institutions 2 The standardized 30-day hospital mortality rate for acute myocardial infarction (AMI): the rate in Croatia (11.2 in 2016) is several times the rates reported in selected top-performing countries such as Denmark (3.6), Norway (3.8), Sweden (4.2), and Slovenia (5.2). A similar trend to that of AMI mortality can be observed for stroke mortality in Croatia with the rate being several times higher than that in selected top performing countries such as Denmark, Sweden, and Norway. 3 Initial allocation of the Program was EUR 75 million, reduced at restructuring to EUR 64 million Page 6 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) v. Fostering quality of care 40.0 vi. Strengthening preventive activities 24.0 vii. Preserving the financial stability of health care 10.0 viii. Improving cooperation with other sectors and society 4.0 in general  Total 409.0 64.0 955.0 Theory of Change (Results Chain) 10. A results chain from the Program Appraisal Document (PAD) is presented in Figure 3. The Program supported the objectives and expected results of the National Health Strategy 2012-2020. To improve two critical areas of the health services (quality and efficiency) and, considering the objectives of the Country Partnership Strategy (CPS) for FY14-17, it was agreed with the Government that the Program would include 5 out of the 8 priorities of the Strategy. These priorities were considered key to rationalize the health facility network, improve quality of health care services, and promote financial sustainability of the health sector. 11. The five priorities within the Government's Strategy that the Program supported are (using their numbering): iii. Strengthening management capacity in health care. The specific areas of focus include education and differentiation of management, data analysis, planning and researching the health care system, and strengthening the management authority of community health centers; iv. Reorganizing the structure and activities of health care institutions . Improving integration and cooperation in primary health care and public health, developing and implementing a hospital master plan to rationalize and modernize hospital services, increasing the continuity of care between hospital and out- of-hospital services, structural modifications to hospitals, and increasing centralized (joint) procurement for hospitals; v. Fostering quality in health care through (a) improving quality of monitoring, health worker education, and better public information for users; (b) developing, implementing, and monitoring clinical guidelines and accreditation; (c) introducing performance-based contracting and performance-based payments, with a specific emphasis on pay-for-quality initiatives; and (d) developing and implementing a formal Health Technology Assessment (HTA), including strengthening capacity to implement; vi. Strengthening preventive activities by increasing the budgetary share of preventive activities in the health budget, improving management of preventive activities and programs including the introduction of performance-based contracting for prevention and strengthening preventive care at the primary care level; strengthening systems to monitor harmful environmental factors and early warning/response systems; and vii. Preserving financial stability of health care by focusing on strengthening the voluntary health insurance market, improving financial discipline in the health care system through greater accountability, improving the strategic allocation of health resources, and reducing corruption and informal payments. Page 7 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Figure 3: Program Results Chain Page 8 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Rationale for Program Support, and Program Scope and Boundaries 12. This Program supported the priorities of the World Bank’s CPS for FY14-174. The CPS focused on the renewed and sustained growth, with emphasis on expenditure rationalization, including through rationalized health spending. Pillar 1 (Fiscal adjustment through reforms at the sector level) envisaged a results-based engagement in the health sector to support the Government's sectoral reform efforts in order to sustain good health outcomes at a lower cost. Specifically, objective 3 of the pillar 1, Introducing cost rationalization and efficiency measures in the health sector, focused on monitoring of the total reduction in public accrued health spending and was linked to the implementation of the hospital masterplan, supported by the Program. According to the Completion and Learning Review of the CPS5, the objective was successfully achieved by reducing the total number of acute care beds by 23 percent and decreasing the public accrual spending from 7.4 percent in 2015 to 6.6 percent in 2016. 13. The Program was also aligned with the CPS objective of improving competitiveness and financial sustainability, by involving the private sector through the rationalization of medical and nonmedical services and by increasing the efficiency of the public health care system, and the objective of maximizing the benefits of EU membership by supporting the absorption of EU funds. The Program was also expected to contribute to the World Bank's Twin Goals6 by increasing access to higher quality health care services, especially for the poorest segment of the population, and by supporting a more equitable allocation of available resources for health . 14. The Program continued to support the priorities of the Government outlined in the new country engagement document, the Country Partnership Framework for the period of FY19-24 7, after the CPS’ closing date was extended until October 31, 2019. The objectives of the Program remain in line with the Government’s priorities identified in the Systematic Country Diagnostic8, such as closing institutional gaps in the health care system, including those related to the efficiency of both financing and delivery of services. The Program supported government efforts to improve the quality and efficiency of service delivery. This included promoting primary care practice, day-hospitals and hospital functional integration to create more cost-effective organizational settings and rapid-access care for patients. Centralized procurement of drugs and medical supplies was also intended to reduce unnecessary cost in service delivery, minimize waste, and ensure equal care countrywide. 15. The Program was built on the work of the previous successfully completed Investment Project Financing operation in Croatia. With the support of the World Bank, the Development of Emergency Medical Services and Investment Planning Project9 (which closed on December 31, 2013), the MoH and HZZO have: (a) conducted a hospital rationalization analysis to develop a hospital rationalization plan; (b) developed EU proposals to mobilize resources to finance investments in support of the hospital rationalization process; (c) assessed options for rationalization of medical and non-medical hospital services; (d) implemented a communication campaign to inform the public about the benefits of the health sector reform being implemented; (e) designed the Business Process Reengineering in the HZZO; and (f) implemented a Geographic Information System that improved data availability and allowed proper monitoring and evaluation, which required a reliable and disaggregated information system. 4Croatia - Country partnership strategy for the period FY14-17 (Report Number 77630) http://documents.worldbank.org/curated/en/668871468026684484/pdf/776300CAS0P143020Box377322B00OUO090.pdf Page 9 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Choice of instrument 16. The Program-for-Results (PforR) instrument was an appropriate mechanism to support country priorities in health system development. The PforR instrument was selected for the proposed operation because: (a) it was aligned with the National Health Care Strategy 2012-2020; (b) the country was able to mobilize sufficient resources for investment and demonstrated high institutional capacity for implementation of the lean operation; and (c) the Program was intended to align the incentives of the MoH and MoF to achieve program results. Strengthening the country systems aimed at enabling Croatia to achieve the ex-ante conditionalities for EU funding mechanisms and formulate strong EU grant proposals to get access to an important future source of funding for the health sector. 17. The boundaries of the PforR Program were clearly set in terms of: (a) the priorities of the National Health Care Strategy supported; (b) the actual scope and duration of the Program; and (c) required engagement with the EU funding opportunities. The Bank’s contribution of EUR75 million (EUR64 million after restructuring) was only partially covering the country’s effort to implement the priorities of the National Health Care Strategy 2012- 2020 (total allocated financing for the Strategy amounts to EUR409 million, excluding the support provided by the EU). Program Development Objective (PDO) 18. The PDO stated in the PAD and Legal Agreement was to improve the quality of health care and efficiency of health services in Croatia. Key Expected Outcomes and Outcome Indicators 19. PDO level results indicators (PDOI) were designed to capture quality and efficiency domains of the PDO statement. Two PDOIs with sub-targets were also defined as disbursement-linked indicators (DLIs): - PDOI 1: First phase of the hospital master plan achieved all of the following milestones: (a) the total number of acute care beds reduced by 20 percent, from 15,930 to 12,800 (DLI 1); and (b) 80 percent of rationalized hospitals without arrears incurred during the preceding calendar year (DLI 3). - PDOI 2: Quality control procedures in place including: (a) at least 40 percent of best-performing rationalized hospitals are publicly disclosed (including results) based on the technical audit in the preceding 12 months (DLI 5); and (b) at least 50 percent of rationalized hospitals accredited by the Agency for Quality and Accreditation in Health and Social Care (AQAHS) through the Acceptable Accreditation Process 8 (DLI 6). Program Results Areas and DLIs 20. The Program was appraised with 10 DLIs. For the purposes of the Implementation Completion and Results (ICR) report, the DLIs have been attributed to the two areas identified in the PDO statement: quality and efficiency in Table 2. It explains how the DLIs can be categorized and associated with each of the key areas: DLIs 5 Croatia - Completion and Learning Review for the Period of FY14-FY17 (Report Number 136266) 6 The World Bank Group’s twin goals are to end extreme poverty by 2030 and boost the incomes of the bottom 40 percent (including in high-income economies), referred to as shared prosperity. 7 Croatia - Country Partnership Framework for the Period of FY19-FY24 (Report Number 130706) http://documents.worldbank.org/curated/en/501721557239562800/pdf/Croatia-Country-Partnership-Framework-for-the-Period- of-FY19-FY24.pdf 8 Croatia - Systematic Country Diagnostic (Report Number 125443) http://documents.worldbank.org/curated/en/452231526559636808/Croatia-Systematic-Country-Diagnostic 9 Croatia - Development of Emergency Medical Services and Investment Planning Project (Loan No. IBRD-75980) http://documents.worldbank.org/curated/en/480991468026678288/Croatia-Development-of-Emergency-Medical-Services-and- Investment-Planning-Project Page 10 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 1, 3, and 8 are efficiency indicators; DLIs 5, 6, and 10 are quality indicators; and DLIs 4, 7, and 9 contribute to both quality and efficiency improvements expected from the program. Table 2: Program development indicators and Disbursement-linked indicators by areas of the PDO PDO Changes at Results Disbursement-linked indicators (DLIs) indicator restructuring area DLI 1. Total number of hospital beds in Rationalized Hospitals Yes Efficiency classified as acute care beds. DLI 2. Number of "Hospital Reshaping Scheme" projects Efficiency implemented. Quality DLI 3. Percentage of rationalized hospitals without arrears incurred Yes Dropped Efficiency during the preceding calendar year. DLI 4. Percentage of all surgeries included in the elective surgeries Efficiency list performed as outpatient surgeries in the preceding six months. Quality DLI 5. Percentage of best-performing rationalized hospitals which are publicly disclosed (including results) based on the technical audit Yes Quality in the preceding 12 months. DLI 6. Percentage of rationalized hospitals accredited by AQAHS through the Acceptable Accreditation Process. Definition Yes Quality (Revised as Foundations are in place for accelerated implementation revised of hospital accreditation) DLI 7. Percentage of identified doctors with whom corrective course Efficiency of action has been discussed on a person-to-person basis in the Quality preceding six months. DLI 8. Percentage of total public spending per fiscal year on medical consumables, drugs, and devices for hospital (inpatient and Target Efficiency outpatient) services made through centralized procurement/ revised framework contracts and disclosed on the MoH website. DLI 9. Percentage of primary health care doctors in the Republic of Efficiency Croatia working in group practices. Quality DLI 10. Percentage of hospitals with surgery wards that have established quality- and safety-related sentinel surveillance schemes Quality that are reporting the rates of specific events B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDOs Outcome Targets, Result Areas, and DLIs 21. The only restructuring of the Program was completed on June 27, 2018. It was a Level 2 restructuring that included the following modifications: PDO indicators a. PDO Indicator 1 (first phase of the hospital master plan achieved all the milestones) (b) (80 percent of rationalized hospitals without arrears incurred during the preceding calendar year) was dropped. At the time of the mid-term review (MTR) and restructuring, hospitals arrears kept increasing due to the lack of progress in key reforms addressing this issue and the fact that new arrears generated by giving priority to Page 11 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) addressing accumulated older arrears. The assessment at the time was that the Program wouldn’t be able to resolve the problem of arrears by the closing date and the achievement of the PDO 1(b) was not feasible. b. PDO Indicator 2 (quality control procedures in place) (b) (at least 50 percent of rationalized hospitals accredited by AQAHS through the Acceptable Accreditation Process) was modified to read as follows: “Percentage of acute hospitals meeting mandatory quality standards, as defined in the Ordinance of Quality Standards and their Implementation from 2011, confirmed through a proxy-accreditation process.” At the time of the restructuring, PDO indicator 2 (b) was not likely to be achieved even with the Program extension. Efforts to establish a national accreditation system had been diluted by the Government’s changing position on this matter. The proposed change in the indicator intended to calibrate activities that served as an important steppingstone for national accreditation. Intermediate results indicators a. Intermediate results indicators 4 (growing body of clinical guidelines with specific protocols for most prevalent non-communicable diseases [NCDs], including care pathways) and 5 (number of primary health care doctors re-trained in updated clinical protocols) were dropped, because they were found less relevant to be monitored during the restructuring. Due to persisting absence of compulsory clinical guidelines, as well as legally binding protocols for practices in the same or similar situations, not much progress had been made in these indicators. b. The target for intermediate results indicator 9 (percentage of total public spending per fiscal year on medical consumables, drugs, and devices for hospital (inpatient and outpatient) services made through centralized procurement/ framework contracts and disclosed on the MoH website) was lowered from 60% to 32.34% to be consistent with the change of the linked DLI 8. c. Intermediate results indicator 15 (ordinance approving accreditation standards and procedure issued) was added, and concurrently became a DLI 6.2. Change in DLIs a. DLI 2 (number of Hospital Reshaping Scheme projects implemented). The baseline and final target values for DLI 2 remained unchanged (0 and 2, respectively). But due to delay of the achievement of interim and final target values, the full amount of EUR 7,500,000 for this DLI was agreed to be disbursed as one-off payment, with no scalability applied, and upon verified achievement of the respective final target value. b. DLI 3 (hospital arrears) was dropped and the full allocation in the amount of EUR 7,500,000 was cancelled from the loan amount. This DLI was originally linked to PDO indicator 1(b), and for the reasons explained above, even within the proposed Program extension period, was considered unlikely to be achieved. c. DLI 5 (percentage of best-performing rationalized hospitals which are publicly disclosed (including results) based on the technical audit in the preceding 12 months). The baseline and final target values for DLI 5 remained unchanged (0% and 40%, respectively). But, similarly to DLI 2, due to the delays in achievement of interim and final target values, the full amount of EUR 7,500,000 for this DLI was set be disbursed as one-off payment, with no scalability applied, and upon verified achievement of the respective final target value. d. DLI 6 (percentage of rationalized hospitals accredited by AQAHS through the acceptable accreditation process) was modified. At the time of the restructuring actions to accredit hospitals were delayed due to: (i) delays in establishing the enabling legal framework and the absence of a legislative basis for mandatory accreditation; (ii) lack of awareness of hospitals of the process and benefits of accreditation and, thus, no Page 12 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) perceived incentive for hospitals to apply for a voluntary and long process of accreditation; and most importantly; and (iii) a government decision taken in 2015 to abandon the idea of implementing national accreditation for hospitals and instead move to the use of international accreditation standards, thus diverting the efforts in another direction. Therefore, this DLI became unlikely to be achieved based on the old definition. Consequently, the DLI 6 was redefined to read as “Foundations are in place for accelerated implementation of hospital accreditation”, and was further divided into two new sub-DLIs: DLI 6.1 (percentage of acute hospitals meeting mandatory quality standards, as defined in the Ordinance of Quality Standards and their Implementation from 2011, confirmed through a proxy-accreditation process), and DLI 6.2 (ordinance approving accreditation standards and procedure issued). Both sub-DLIs were allocated EUR 3,750,000, respectively. e. DLI 8 (percentage of total public spending per fiscal year on medical consumables, drugs, and devices for hospital (inpatient and outpatient) services made through centralized procurement/framework contracts and disclosed on the MoH website).The target values for the DLI were reduced to a single target value and lowered to 32.34% (compared to the original two target values of 30% and 60%). Consequently, an amount of EUR 3,457,500 was cancelled from the original allocation for this DLI, based on the agreed during the PforR preparation formula. The revised allocation of EUR 4,042,500 for this DLI was planned to be disbursed as one-off payment, with no scalability applied, and upon verified achievement of the revised target value. Program Action Plan a. Action 17 (TA to AQAHS in foreseen implementation of hospitals’ accreditation (assistance in preparation of accreditation scheme legislation, accreditation, environmental guidance, trainings, etc.) was revised to read as follows: “Implement mandatory environmental quality standards. Prepare environmental standards proposal (containing full sets of environmental indicators envisaged by the ESSA” through AQAHS to be subsequently included in a comprehensive accreditations system” b. Action 18 (TA to AQAHS in establishment of monitoring system of the hospitals’ compliance with adopted mandatory quality standards) was reformulated as follows: Conduct a self-assessment and external results verification of mandatory standards and propose/outline (recommendations for) monitoring system for the hospitals’ compliance with full accreditation (including comprehensive environmental indicators). 22. Cancellation. Due to the afore mentioned dropping of DLI 3 and partial reduction of DLI 8, the amount of EUR 10,957,500 was cancelled from the loan. Consequently, the total loan amount decreased from EUR 75,000,000 to EUR 64,042,500. 23. Extension of the closing date. To allow additional time for the achievement of a few DLIs within the extended period, such as DLI 2 (hospital reshaping schemes), DLI 6 (hospital accreditation), and DLI 8 (centralized procurement), and, therefore, contribution to the quality and efficiency of health services in Croatia, the Program closing date was extended by 16 months, from June 30, 2018 to October 31, 2019. 24. Other changes. The Program encountered a number of changes which are summarized in Table 3, with more detail provided in Section II, Key Factors Affecting Implementation and Outcome. Page 13 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Table 3: Summary of Key Changes in the Health Sector During Program Implementation in 2013-2019 Change of Year Ministers and Key policy changes Changes influencing Program implementation hospital managers 2013 Law on  Hospital centralization under the MoH authority Rehabilitation  Hospital Sanation/Management boards approved  Joint procurements (since October 2012)  New contracting scheme for primary care: mixed payment (fixed part, capitation, fee-for-service (around 200 services introduced), group practices, prevention, 5-star extra services, Key Performance Indicators and Quality Indicators) 2014 Changed  Direct reporting of hospitals to the MoH Provider payment  Fee-for-service in ambulatory/out-patient settings (around 3000 changes services introduced)  More favorable prices for outpatient surgery and day care Quality Control  Extended responsibilities of the AQAHS 2015 National Hospital  Initiation of hospital integration Development Plan  New contracting scheme for hospitals (80% advance + 20% for 2014-2016 retrospective payment system based on diagnosis-related groups + performance payments of + 5% above the budget) HZZO  HZZO started collecting mandatory and voluntary contributions administration of (previously done by treasury) contributions 2016 Changed twice Cancellation of the  Hospital decentralization Law on  Revised contracting/payment scheme for hospitals (mostly Rehabilitation advance-based with only 10% payments based on diagnosis- related groups, and 3% performance payments) 2018 The National  Functional integration is set as a tool for hospital restructuring and Hospital quality improvement Development Plan for 2018-2020 2019 The Health Law  Health contribution rate has increased by 1%  The AQAHS became a part of the MoH The Health Care Quality Act Rationale for Changes and their Implication on the Original Theory of Change 25. The Program went through different periods of acceleration and slowdowns . During its first years (2015- 2017), 5 (DLI 1, 4, 7, 9 and 10) of 10 DLIs were achieved in full. The results were seen in the efficiency and quality tiers of the Program objectives. The country has achieved good progress with the reduction of beds in the hospitals, and the rapid growth of the outpatient-based elective surgeries, increase of the share of general practitioners working in group practices. In hospitals, incidence reporting was established in the form of quality- and safety-related sentinel surveillance, and doctors deviating from the goods practices of drugs prescription were identified and coached. Page 14 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 26. The delays in the implementation of the other Program activities and revised scope for the quality and efficiency agenda in the health sector resulted in its restructuring in 2018. The MoH revised the initial scope of accreditation from introducing international accreditation standards to a voluntary-based domestic accreditation procedure. After the abolishment of the hospital rehabilitation law, the appetite for hospital optimization decreased, the MoH lost the leverage to tackle hospital arrears, and implementation of hospital integration became a more challenging task. The MoH has decided that the initially arranged joint procurement for hospitals will be reorganized to become MoH-led centralized procurement for select medical categories. 27. After the restructuring, the initial objectives of the project were reduced. Two of the remaining five DLIs (DLI 2 and 5) required additional time to be implemented (implementation of hospital reshaping schemes and technical audits in hospitals). DLIs 6 and 8 were revised in view of changes in the expected scope of the accreditation process and joint procurement that could be achieved during the life of the program. As the Program did not directly support the activities that could help prevent or manage arrears and therefore the program could not directly influence the achievement of the target, DLI 3 on cleaning up hospitals from arrears was dropped. 28. The revised theory of change with modifications after restructuring was constructed based on the scope of the restructuring and consultations with the MoH. The modified theory of change is presented in Table 4. Table 4: Revised Program’s results chain Long-term Inputs Activities Outputs Outcomes goals Changes in Technical Results for Actions in Program deliverables within the control of the MoH and system assistance, medium and long- the Program HZZO performance and training, etc. term impact health outcomes Unchanged Unchanged Priorities iii and iv: Strengthened • Increased gatekeeper function of primary health management care capacity • Increased provision of outpatient diagnostic and Reorganized treatment services structures and • Differentiated acute care services from long-term activities of care, reduced number of acute care beds health care • Rationalized health facilities through reshaping institutions schemes Priorities v and vi: Improved • Additional clinical guidelines adopted quality of health • HTA of selected new health technologies services implemented • Technical and clinical audits implemented Strengthened • Sentinel events surveillance system implemented preventive • System of feedback for prescriptions established activities Page 15 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Priority vii: Preserved  Reduced • Performance-based payments for primary and financial stability incidence and hospital care amendable • Day care and outpatient services incentivized mortality from NCDs • Drug risk-sharing schemes for expensive drugs  Reduced implemented rates of • Savings achieved from centralized procurement hospital and framework contracts infections • Projects supported by EU structural funds  Reduced unit implemented costs of health services  Perceived increased patients’ satisfaction  Decreased regional health disparities II. OUTCOME A. RELEVANCE Relevance of PDO: High 29. The Program's development objectives were highly relevant. The objectives fully reflected the country’s health sector development priorities focused on quality and efficient health services contributing to sustainability of the health system and economic growth, which were declared in the National Health Care Strategy for 2012-2020. These priorities were also reflected in the National Hospital Plans for 2014-2016 and 2018-2020. 30. The Program’s objectives also retained their high relevance in the context of the World Bank’s newly adopted Country Partnership Framework for FY19-24, which under Objective 1 aims at Improving efficiency of public administration to implement strategy and deliver services. The Program design nested in the National Health Care Strategy and the MoH-led implementation arrangements reflecting the core Program features had a high relevance in the Croatian context in terms of achieving results and building institutional capacity. Relevance of DLIs: High 31. The DLIs were fully aligned with the PDO of the Program. The DLIs were well-designed to help the country develop its health care system towards the objectives of the National Health Care Strategy for 2012-2020. The National Health Care Strategy set ambitious goals for the Croatian health system to better align with the health indicators of European countries after the accession to the EU. The activities and results supported by the Program were linked to the strategic priorities of the Strategy and were helping the country to strengthen preventive and primary health care, while improving quality and efficiency of hospital services. Page 16 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 32. The proposed DLI indicators were effectively capturing the needed effort to improve efficiency and quality of health services. As presented in table 2, the proposed DLIs for the Program were appropriately linked to the PDO indicators (four DLIs are also PDO indicators), and a good balance was secured between quality and efficiency indicators, with some of them contributing to both domains of the PDO. Small shortcomings are related to issues with definitions in some DLIs. Each DLI was assigned with amounts of EUR 7.5 million, which was potentially providing not enough incentive to achieve set targets. Rating of Overall Relevance: High 33. On the basis of the foregoing, the overall relevance of the Program is High. No major changes to the strategic priorities of the health system development were introduced during the life of the National Health Care Strategy, and the PDO and DLIs remained relevant for the Program duration. All implementation status reports for the program confirmed the relevance of the program. The stakeholders interviewed after Program completion also confirmed the continued relevance of the Program for the Croatian health system development. Several DLIs were changed during restructuring to keep DLIs fully relevant. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective or Outcome 34. Achievement of Program’s objectives was reviewed for quality and efficiency domains of health care, consistent with the objectives of the Program to improve the quality of health care and efficiency of health services in the Republic of Croatia. Some of the Program’s results have potentially contributed to both areas of quality and efficiency as described above and explained in the following sections. Quality improvements 35. The Program contributed to improving the quality of health care through strengthening the system-level quality interventions. Key quality interventions included implementation of hospital audits and hospital benchmarking; setting up the foundation for implementation of accreditation of providers; and implementation of the quality- and safety-related sentinel surveillance and incidence reporting. 36. Program activities related to technical audits and hospital benchmarking were supported by PDOI 2(a)/ DLI 5 and confirmed as achieved. The Program supported development of in-house capacity for conducting hospital benchmarking. The MoH has built internal capacity for development of tools for technical audits10, data collection, analysis and interpretation. The first round of technical audits was completed in 28 acute care hospitals in 2017, discussed with hospital managers in September 2018 and collected information disclosed in the form of a national hospital benchmarking exercise. 37. Activities supporting the move towards hospital accreditation were linked to PDOI 2(b)/ DLI 6. Initially the target was ambitiously set as at least 50 percent of rationalized hospitals completed accreditation through the Acceptable Accreditation Process. However, accreditation was not mandatory and not seen by hospital managers as meriting the effort. During restructuring, the indicator was revised to approve the ordinance for the accreditation standards and procedure. To advance preparations for the accreditation, 33 acute hospitals conducted self-assessments at the end of 2018 and went through external technical audit during on-site visits in 2019. The tool for self-assessment was developed based on nine groups of standards and three standard 10The tools for technical audits were developed by the MoH group using Croatian clinical guidelines for the diagnosis and treatment of acute coronary conditions, materials of the Croatian Cardiology Society (European Cardiac Society Guidelines, 2012), guidelines for the diagnosis and treatment of brain stroke (Demarin et al., 2002) and guidelines for antimicrobial prophylaxis in surgery (Francetic et al., 2010) and other materials from their relevant literature review. Page 17 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) requirements related to environmental management. The process helped develop tools, processes and methodologies to establish a legal framework on accreditation standards and procedures. The ordinances on defining accreditation were approved in September 2019 after several rounds of public consultations, and achievement of the DLI 6 confirmed by the World Bank on October 31, 2019. 38. The Program’s quality improvement activities helped establish hospital sentinel surveillance systems with a focus on nosocomial infections and patient safety (DLI 10). Incidence reporting was introduced for hospitals with surgery wards. Such reporting included records on wrong patient surgery or wrong site surgery, postoperative pulmonary embolism and deep vein thrombosis, and non-traumatic diabetes-related lower-limb amputations. A web-based platform and emails were used for regular annual reporting by hospitals. The first target of 30 percent of relevant hospitals with established incidence reporting was achieved in 2016, and the second target of 60 percent was achieved and verified in 2017. The further scale up of the sentinel surveillance system was complete in 2018 with all hospitals with surgery wards reporting sentinel medical error events and cases of nosocomial infection. Thus, the Program helped established a system of incidence reporting, which is a basic block for quality improvement management in health facilities. 39. In-hospital quality improvement activities have helped to reduce in-hospital mortality . Available data provided by HZZO suggest that top 15 hospitals that had worst in-hospital mortality rates in 2015 were able to substantially improve this indicator by 2018 (with exceptions for Karlovac and Ogulin general hospitals). Major reductions in the in-hospital mortality rates were observed between 2017 and 2018, potentially as outcome of the hospital audits conducted in 2017 and repeated in 2018. Table 4 In-hospital mortality rate per 1,000 hospitalized in 2015-2018 for highest-ranked 15 general-profile hospitals Decrease (-), Hospital name 2015 2016 2017 2018 increase (+) O.B.SISAK 71.28 69.89 70.15 49.75 -30% O.B.ZABOK I BOL.HRVATSKIH VETERANA 65.26 65.27 65.41 57.63 -12% O.B.GOSPIĆ 59.14 50.94 54.63 53.66 -9% O.B.ZADAR 51.61 50.98 52.43 40.81 -21% K.B.DUBRAVA 49.95 49.48 50.53 24.20 -52% O.B.PULA 49.66 46.80 44.26 38.90 -22% O.B.SL.BROD 48.52 45.77 37.32 37.99 -22% O.B.BJELOVAR 48.13 41.71 52.74 46.25 -4% O.B.VIROVITICA 47.75 45.78 42.67 43.34 -9% O.B.KARLOVAC 46.91 42.60 46.12 52.08 +11% O.B.POŽEGA 45.83 47.59 46.61 41.07 -10% O.B.KOPRIVNICA 45.70 40.85 43.02 31.41 -31% O.B.ŠIBENIK 45.63 51.18 46.92 45.78 0% K.B.C.SPLIT 43.34 40.13 38.45 25.17 -42% O.B.OGULIN BOL.BRANITELJA DOM.RATA OGULIN 42.87 41.48 59.62 58.38 +36% Note: mortality rates provided for acute cases only, excluding same-day admission and discharge Page 18 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Efficiency improvements 40. The efficiency improvements in the health system of Croatia were expected at the primary care level and in hospitals. Efficiency gains at the primary care level were mainly seen from the scale-up of group practices and improved prescription patterns. At the hospital level, the efficiency was expected to increase from the reduction of acute beds, support of the functional integration of hospitals, shift of elective surgeries from inpatient to outpatient care, and savings from the joint procurements of pharmaceuticals and medical goods. 41. The rightsizing of hospital beds supported by PDOI 1(a)/DLI 1 was achieved during the second year of implementation in 2016. Croatia differentiated acute care services, distinguishing them from long-term care. A certain number of acute care beds was re-profiled in chronic care, long-term care, palliative care and day-care beds. The first target of 15,000, with an actual value of 13,572 was achieved in 2015 and the final target of 12,800, with an actual value of 12,146, was achieved in 2016, slightly increasing after the separation of 2 hospitals, to 12,315 in 2019. In total, the number of acute care beds was reduced by almost a quarter (23 percent). The total number of inpatient beds has also decreased from over 25,000 to slightly above 23,000. 42. The reduction of unnecessary acute beds capacity has been further supplemented by shifting elective hospital care to day-care and outpatient settings. DLI 4 supported the increase in the share of select outpatient- based elective surgeries from the low baseline11. The providers received incentives for provision of outpatient- based procedures and surgeries in the format of higher fee-for-service payments for these types of care, compared to a fixed pre-payment for the inpatient services. The progress in the achievement of this indicator was prominent, and the final target of 60 percent was achieved in the beginning of 2016 with an actual value of 61.1 percent. The HZZO further increased the prices of services that were provided as same-day surgery procedures, the numbers of same-day surgeries and outpatient procedures was steadily growing, covering a total of 69 procedures in 2019. Figure 5 Increase in day surgeries in Croatia between 2012 and 2016 compared to the EU Note: Chart was copied from the State of Health in the EU: Croatia. Country Health Profile 2019 43. Improvements in efficiency and quality of service delivery were expected from achievement of DLI 2: implementation of hospital reshaping/functional integration schemes. Optimization of hospital structures and functional integration of hospitals was one of the key expectations in the National Hospital Development Plan of 11Elective surgeries tracked in the DLI4 included: abdominal hernia surgery (5.8% performed as outpatient surgeries in 2012), cataract surgery (4.3% in 2012), local excision and removal of internal fixation, except on hip and femur (3.5% in 2012), palmar fasciectomy for dupuytren's contracture (2.1% in 2012), and tonsillectomy (0% in 2012). Page 19 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 2014-2016 and 2018-2020. The outcome of the integration was seen in the reallocation and concentration of hospital services to improve utilization of resources among the functionally integrated hospitals. The MoH intensified the process of piloting functional integration after the restructuring in 2018 and hired consulting firm to provide assessment and recommendations for functional integration in 6 pairs of hospitals within EU-supported “Structural Report Support Programme”. During the life of the Program the result was monitored and confirmed as achieved, with functional integration of two pairs of hospitals (general hospitals Ogulin and Karlovac, and Osijek and Našice). These hospitals reshaped several inpatient hospital activities retaining four core activities (internal medicine, surgery, pediatrics, and gynecology and obstetrics) while adapting other specialty services to current medical needs (increasing day hospitals/day surgeries, palliative care and telemedicine services, and decreasing acute beds). The larger hospitals in pairs concentrated more acute care in the combined catchment area for both hospitals and shared their staff to provide outpatient consultations in smaller hospitals on a regular basis. The pilot allowed the country to develop necessary legislation and procedures to enable scale up of functional integration at the county level. 44. Another area of expected gains in efficiency was supported by increased share of centralized procurement and framework contracts in DLI 8. Initially 10 institutions (9 state owned hospitals and HZZO) were assigned by MoH decision in 2012 to implement joint procurement for various procurement categories. The decisions on procurement of drugs and consumables for the majority of the county, and general hospitals stayed with local governments/counties, and only some of the county hospitals participated in joint procurement. The first target for this DLI of 30 percent of all hospital spending on medical consumables, drugs, and devices during the fiscal year for participating hospitals made through joint procurement was achieved in 2014, but then joint procurement was entirely suspended for 8 months in 2014-2015 in view of increased prices received in procurement. The share of goods procured through centralized procurement had only reached 16 percent in 2016 after the joint procurement was renewed and centralized at the MoH. During the restructuring, in view of setting up the centralized procurement from scratch in the MoH, the initial end target for this DLI of 60 percent was reduced to 32.34 percent based on a realistic estimate. However, the revised end target was overachieved. With almost 45 percent of the total amount of actual spending by the 10 hospitals whose owner is the Republic of Croatia made through centralized procurement in 2019. However, the savings received by the MoH in centralized procurements are marginal: in 2019 savings were around 1 percent of total contracted value compared to estimated costs based on historic numbers. 45. Efficiency improvements at the primary care level were due to increased participation of primary care doctors in group practices (DLI 9) and decreased inappropriate prescribing behaviors by doctors (DLI 7). The end target of 50 percent for participation of primary health care doctors in group practices was achieved in 2015 with an actual value of 65.1 percent. The share of doctors in group practices kept rising to 69 percent by the end of 2017 and stayed at the level of 70 percent in 2019. The implementation of the performance indicators linked to payments was expected to strengthen and improve quality of primary care, and participation of primary care doctors in group practices was included as one of the performance indicators. However, participation of doctors in group practices did not help improve the gatekeeping role of primary care: an increase in referring patients from primary health care and the utilization of specialist services was observed. 46. Rationalized prescription of drugs was included in key performance indicators (KPIs) for primary care. From the start of Program implementation, a system was established by HZZO to monitor outlier doctors for whom the total monetary value of drugs prescribed in the preceding six months was above a predefined average spending limit that accounts for number, gender, and age of their patient population. Such doctors were identified and visited by a HZZO inspector to discuss possible reasons for such overspending on a case-by-case basis and for cases where overspending was not justified, doctors received a warning and were mandated to attend an educational session on rational drug prescribing, organized by HZZO. It was expected that 90 percent of doctors Page 20 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) identified as outliers would be visited to discuss and correct prescribing behavior, and it was achieved soon after Program’s start in 2015 with an actual value of 93.11 percent. Prescription practices were continuously monitored as part of KPIs in contracts with providers of primary care. The percentage of primary care group practices achieving performance indicator on prescriptions12 was reaching 96 percent in 2015 and slightly deteriorated to 94 percent in 2019. Rating of Overall Efficacy: Substantial 47. Efficacy of the PforR-supported Program is rated Substantial. Both PDO indicators, consisting of three sub- indicators, approved as part of the June 2018 restructuring have reached and exceeded their end-targets, showing achievements in improving quality of health care and efficiency of health services in Croatia. C. JUSTIFICATION OF OVERALL OUTCOME RATING 48. The overall outcome rating derived by combining the assessments of relevance and efficacy of the Program is Moderately Satisfactory. While the relevance of the Program was confirmed high, the efficacy results were rated substantial due to shortcomings in the program implementation and reduced targets of the PDO after Program restructuring. 49. The overall rating for progress towards the achievement of PDO at the Program closing was maintained as Moderately Satisfactory due to several shortcomings during Program implementation. This is primarily due to major delays in the implementation and lack of the political will to continue substantial hospital sector reforms, which resulted in the following changes made as part of the restructuring: (a) cancellation of the PDOI 1(b)/ DLI 3 (reduction in hospital arrears), (b) reduction in the level of ambition of PDOI 2(b)/ DLI (hospital accreditation), and (c) reduction in the target of the intermediate results indicator (IRI) 9/ DLI 8 (centralized procurement). 50. The implementation progress of the Program was Moderately Satisfactory at the closing. During implementation, substantial delays in implementing Program activities due to lagging DLIs were observed. These delays were aggravating in the period between the Program MTR (February-March 2017) and Program restructuring (June 2018) due to the political volatility. In the period after approval, the Program received strong ownership from the Government and reform-oriented leadership in the MoH, with the important Law on Rehabilitation of Hospitals adopted in April 2013. That measure made it possible to achieve certain results, with 5 of 10 DLIs fully achieved during the 2015-2017 period, and 55.1 percent of the loan amount disbursed. However, the progress stalled once the Law on rehabilitation of hospitals was abolished in 2016, and the secondary hospitals were again de-centralized to the counties for management. This resulted in the recurrence of increases in hospital arrears, exemption of county-owned secondary hospitals from mandatory participation in centralized procurement, and substantial extension of time and efforts needed to reach an agreement with the counties on hospital functional integration due to political economy issues. Implementation of a hospital accreditation system was also challenged by non-mandatory status of accreditation and organizational changes involving merging of the previously separate AQAHS as a MoH department. 51. The program implementation intensified after restructuring in 2018. All remaining activities and DLIs were achieved by the closing date with the disbursement of 100 percent of funds. Overall, 11 out of 13 intermediate results indicators have achieved or exceeded their end-targets towards enhancing quality and efficiency of health services. Similarly, all the 18 actions in the post-restructuring Program Action Plan (PAP) have been successfully accomplished by Program closing. 12Non-performing practices include those that for three or more months in the year are overprescribing medications (above KPI benchmark for prescribing) Page 21 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) D. OTHER OUTCOMES AND IMPACTS Institutional Strengthening 52. The Program had a clear focus on institutional strengthening, since it was supporting the implementation of the Country’s National Health Care Strategy. The PforR instrument relies on country capacities to implement strategic priorities and achieve expected results, and this was the case in Croatia. Despite challenges with Program implementation derived from the frequent changes in the leadership of the MoH, the Program was able to achieve ample results in the priority areas of the National Health Care Strategy supported by the Program. Within the lifetime of the Program, major results were achieved as part of institutional development: the MoH established process for centralized procurement (by defining the policy for mandatory hospitals and selecting mandatory categories), developed hospital integration plans and hospital network documents, issued accreditation ordinance, established process for technical audits, etc. 53. The MoH was able to mobilize its capacity and strengthen implementation arrangements after a major slowdown in implementation of Program activities in 2017-2018 . The main challenges with the achievement of the results in hospital integration, joint procurements and hospital accreditation remained after the 2018 Program restructuring. The MoH has revitalized work on functional integration and led the pilot with functional integration in 6 pairs of hospitals (total 12 hospitals participating in the pilot). The hospital integration was closely coordinated by the MoH team and monthly meetings with hospital managers initiated by the MoH were among the key factors to accomplishing the initial stage of hospital functional integration and push forward the perspective plans to expand the integration process in the future. The major change with shift of the joint procurements run by hospitals towards their centralized at the MoH level for 9 country-owned hospitals was another institutional challenge for the MoH. Initially the centralized procurements were taking about 18 months from initiation to the actual completion of the tendering process in view of challenges with estimating needs, drafting specifications, interaction with potential bidders and media, etc. However, after the process was set the duration of the procurement was reduced to about 3-6 months on average. Finally, the AQAHS became a part of the MoH with reduced staff and functionalities. The MoH was still able to support the implementation of technical audits in 33 acute care hospitals using the core staff and coordinating engagement of external experts to achieve the agreed results. 54. The Program helped mobilize resources of the EU to support and supplement its activities . One of the positive impacts of the program that was expected is that the MoH would be able to develop and implement at least 7 projects supported by the EU to mobilize resources to finance investments in support of the hospital rationalization process, including feasibility and pre-investments studies. Before the Program started, the MoH had no experience in implementing EU projects (structural funds), and the expected maximum target of projects accepted for implementation was at the level of eight. However, a total of 61 projects within the Operational Programme "Competitiveness and Cohesion" and 95 projects within Operational Programme "Efficient Human Resources" were supported for implementation by the EU during the life of the Program. Projects supported by the EU were aligned with the objectives of the National Health Care Strategy and thus were supporting priorities of the Program. These projects included, among other priorities: (a) strengthening of primary care capacity by improving diagnostics and therapeutic capacity of primary care providers in 17 counties, primarily in remote and deprived areas; (b) establishment of unified emergency hospital admissions departments in 6 hospitals and improving efficiency and access to day hospitals and day surgeries in 28 locations; (c) implementing 4 projects to improve access to hospital care for vulnerable groups (children, patients with mental illnesses, palliative patients, patients with spinal injuries); and (d) various trainings to improve preventive, family medicine, emergency and other skills of medical professionals. Technical assistance project financed by the EU helped assess potential areas for hospital integration and thus contributed to implementation of hospital integrations schemes DLI of the Program. Page 22 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Gender 55. By addressing quality and efficiency of health care, the Program had potential gender impact. The Program supported activities in pursuit of Sustainable Development Goal 5: Gender equality (including providing women and men with good quality health care). Gender gap in life expectancy was 6.7 years in 2012, reducing to 6.1 in 2017, but still higher compared to 5.3 years in the EU in 201713. The improvements in the mortality rate among men in Croatia during 2013-2017 is higher than in the EU and OECD. However, the prevalence of NCDs is on the rise, with an increase of approximately 25-30 percent between 2012 and 2017, which is reflective of the trends observed in other EU countries. Looking into preventable and treatable mortality (figure 5), Croatia gained significant progress in addressing preventable mortality among male population, but improvements in treatable mortality are modest, and mainly observed for female population. Such changes can potentially be explained by an improved preventive role of the health sector and its primary care level, but less progress with the curative medicine. A longer observation period and in-depth study of the presented indicators could help understand sustainability of the changes and contributing factors. Figure 6 Change in preventable and treatment mortality rate in Croatia and EU in 2012-2016 (Eurostat) Change in preventable mortality rate per 100,000 in 2012- Change in treatable mortality rate per 100,000 population in 2016 2012-2016 -1.0 600.00 -19.1 200.00 -2.2 -4.8 -5.8 -3.9 400.00 -13.6 -11.2 -4.0 -7.7 -2.6 -5.0 100.00 200.00 0.00 0.00 EU all EU EU Croatia Croatia Croatia EU all EU males EU Croatia Croatia Croatia Males Females all males females females all males females 2012 2013 2014 2015 2016 2012 2013 2014 2015 2016 Poverty Reduction and Shared Prosperity 56. The Program has contributed to reduction of regional disparities and potentially improved accessibility of care for lower income and vulnerable groups of patients. The activities related to strengthening primary care might have influenced more people to seek and receive care at primary levels. The differentiation of acute versus long- term care helped reshape capacities in hospitals and increase rehabilitation and palliative care services which were minimal before the start of the Program. Hospital integration helped bring specialist clinicians closer to people living around smaller hospitals by organizing shifts of skilled professionals and telemedicine exchange between hospitals participating in functional integration pilots. The reported savings from the centralized procurement of select drugs and suppliers (generics, select materials for nuclear medicine) helped reallocate savings to procure other medical goods and provide better access to special medicine groups. Other outcomes 57. The Program had plausibly helped country realize climate co-benefits. The rehabilitation and upgrades of provider facilities were using energy efficient technologies. The hospital integration has helped reduce travel of patients to and from facilities as a shift of specialists were organized closer to their living places, and teleconsultations replaced in-person visits or transportation of patients between facilities with virtual sessions (e.g., increase from 22 telemedicine services in 2016 to 62 in 2018 and 55 sessions in the first half of 2019 for Osijek/Našice pair of hospitals, and from 280 in 2016 to 529 sessions in 2018 in Karlovac/Ogulin pair). Consolidation 13 Eurostat data https://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=demo_mlexpec&lang=en Page 23 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) of acute care episodes in larger hospitals could have had positive impacts with more rationalized waste management for services related to emergency care. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 58. The Program was prepared with ambitious perspectives. The Program was innovative: it was first of this type in ECA region and fist PforR in the health portfolio of the Bank. At the same time, however, the context was advantageous as Croatia had demonstrated effective implementation of previous World Bank projects, had ambitious goals of harmonizing its health system with that of the EU in view of the Country’s accession to the Union, and had a recently prepared strategic document with set objectives for development of the health sector (National Health Care Strategy 2012-2020) that were aligned with the Country Partnership Framework. 59. The major reform agenda was realistic in view of the adopted hospital rehabilitation act. The Law on Rehabilitation of 2013 allowed the MoH and HZZO to initiate consolidation of hospitals, set direct requirements to hospital managers to better control arears, and push for ambitious targets of improving efficiency and quality of service delivery. The cancellation of the Law in 2016 was seen as the main obstacle to achieving targets with clean- up of hospital arears and downgraded the ambition to pursue external hospital accreditation based on international standards. 60. The Program benefitted from well-designed DLIs and a detailed action plan in support of their implementation. The National Health Care Strategy 2012-2020 did not provide measurable and traceable indicators that would allow monitoring of its achievements. The Program has offered the framework to measure achievements of the Strategy in 5 out of its 8 priorities and a transparent results framework to measure achievement of important results. 61. The Program benefitted from the client’s readiness for implementation. Strong ownership and active engagement form MoH, HZZO, AQAHS, and MoF were evident during the Program preparation and appraisal stages. The readiness for implementation and commitments to the results was confirmed by early achievement of 5 out of total 10 DLIs in the first 2 years of the Program implementation. Page 24 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) B. KEY FACTORS DURING IMPLEMENTATION Factors subject to government and implementing agencies’ control 62. There was a high level of commitment to the Program from the Government despite frequent changes of the leadership teams. The Program was implemented using internal capacities of the MoH and without setting up a dedicated implementation unit supported by consultants. During the early years of the Program implementation, the activities of the Program were well managed and fully aligned with the aspirations of the MoH leadership. There were four health ministers, with different visions of implementation of health reforms, from the Program design phase till its closing date. The change of the Ministry leadership at the time of the Program’s MTR included a change of political party (from SDP, Social Democratic Party of Croatia, to HDZ - Croatian Democratic Union). There were also changes in the leadership of health care providers: many hospital managers changed in 2014, and all managers changed again in 2016, sometimes even twice during 2016. After changes in the Government, the implementation progress slowed down which was one of the main reasons for the delayed restructuring completed more than 15 months after the MTR. The Program implementation has significantly intensified in the last 2 years before extended closing date in October 2019. 63. The reforms of the hospital management were reversed during the Program implementation. The Law on Rehabilitation came into force on January 1, 2013, and, as part of it, rehabilitation was initiated in 31 health care facilities (29 hospital facilities and 2 health centers). The law transferred the ownership of these facilities from counties to the MoH, mainly to improve management of arrears. At that time, sanation (supervisory) boards were established with participation from the MoF, MoH and select professionals that were approving all procurements, employments and reviewing reports on management of the institutions. In 2016, ownership of all the institutions included in the rehabilitation plan was given back to counties, and appointments of hospital management changed from direct appointment by the Minister of Health to public competition. After 2016, the accumulation of arrears continued, and without introducing proactive management of causes for accumulation of arrears, the MoH and the MoF were able to secure several allocations of funds to decrease the growing amount of debts. 64. Health financing and provider payment changes. In 2014, the revised payment schemes were offered to care providers to incentivize performance and efficiency. However, in 2016-2018, shares of performance and volume-based payments to hospitals decreased leaving hospitals with minimal incentives to provide more care to patients or further improve performance indicators. The lack of volume-based relation between inpatient care provided and payments from HZZO to hospitals can potentially explain growing waiting times and continued accumulation of arrears, since hospitals were not reimbursed on the costs if more patients received care and treatment. 65. Several changes took place in the last year of Program implementation. In 2019, several changes were introduced through the Law of Healthcare offering some adjustments in the organization of the primary and emergency care. The Law also provided brief guidelines for restructuring and simplifying the hospital system by integrating hospitals. The previously autonomous AQAHS became a part of the Directorate for Health Tourism and Quality of Health Services with reduced staffing and functionalities. 66. The technical assistance element of the Program was relatively small and fell short of providing adequate support to the implementation of the intended Program activities. Because of the PforR nature, the successful implementation of the Program was not seen as a financial incentive by the MoH. The MoH team only received EUR 7.5 million for the technical assistance at the beginning of the Program implementation and thus had restrictions in terms of the scope of technical activities it could support to accelerate implementation of the Page 25 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Program’s results. Given the nature of the PforR instrument, at the time of the preparation a designated program implementation unit was not established, and all project activities were coordinated by the MoH staff. While the MoH staff demonstrated strong dedication throughout the life of the Program, the absence of technical assistance resources and consultants’ support might have resulted in some delays and additional bottlenecks to implementation progress. 67. The Program was able to coordinate its activities and support provided by EU to amplify achievement of its results. Close dialogue and cooperation with the EU increased significantly during the Program timeframe. As mentioned earlier, the MoH has implemented over 150 joint projects using structural funds and technical assistance available in the EU portfolio. Factors subject to control by the Bank 68. The Bank team demonstrated appropriate supervision and monitoring during Program implementation. These elements are described in more detail in Section IV.C “Quality of Supervision”. Factors outside the control of government, implementing agencies and WB 69. The macroeconomic and political context for the Program was initially challenging. Significant fiscal pressures after economic recession challenged economic growth. High levels of unemployment, reaching 18 percent, limited space for health insurance contributions by the Croatia population. The period after accession of the country to the EU has been naturally characterized by outflow of qualified workforce and decreased ambitiousness of reforms agenda compared to pre-accession period. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION Design 70. The Program benefitted from a straightforward monitoring and evaluation (M&E) framework. The PDO and intermediate results indicators corresponded well with the agreed DLIs and were mutually reinforcing. Substantive parts of the results framework were under the direct control of the MoH, and other indicators were embedded in the HZZO data management systems. The proposed targets were practical and mostly well defined. 71. A shortcoming of the results framework was that it was initially lacking a focus on outcomes. The results framework was designed to allow frequent monitoring of the Program’s progress, allow course correction, and demonstrate quantifiable results. However, major results were mainly described at the output level (rationalized hospital beds, approved hospital reshaping schemes, percentage of hospitals conducted audits, percentage of primary doctors working in group practices, etc.) and were not appropriate for measuring the changes at the level of outcomes. At the same time, it included ambitious targets of achieving hospital restructuring and a reduction in arrears, which were depending on political will and other factors beyond the team’s control in changing political environment. 72. One of the DLIs was challenging in terms of ambiguity of its definition. Specifically, for DLI 2, it was not clear how the success of the functional integration would be measured and how the implementation of the hospital reshaping schemes can be verified. It was therefore challenging to reach a full agreement of the results of the functional integration in the monitored hospitals, and the World Bank and MoH team had to develop and define requirements that would provide substantial evidence for achievement of the target. 73. The assessment of the beneficiary impact was not strong. The assessment of patient satisfaction by perception of increased responsiveness of service delivery was expected to be conducted annually. The responsibility Page 26 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) for the survey rested with HZZO. However, the surveys were only conducted in 2017 and 2019, but the results of the last survey are not available to the Bank team at the time of ICR preparation. It was therefore difficult to review the actual progress in the achievement of the beneficiary impacts. Implementation 74. The ups and downs in the Program implementation affected its monitoring arrangements. M&E and program monitoring was effective in the first two years of the Program implementation, when achievement of all Program indicators was on track. After the period of delays and gaps in reporting, program monitoring has been substantially strengthened in the post-restructuring period. Specifically, in addition to the MoH demonstrated commitment to continue submitting monthly updates on progress of the Program (including DLIs, results framework, PAP) as well as on sectoral reforms that the Program supports, the MoH and the Bank regularly held monthly pre-scheduled video-conference sessions to discuss progress, issues, and actions to address the issues following the MoH’s each monthly progress update. Given the importance of the health sector reform and the MoH’s commitment to a successful completion of the Program, these agreed arrangements had been diligently followed by both the MoH and the Bank up until the Program closing. Utilization 75. The good practices in systematic utilization of the M&E information of the Program were observed. The reports on implementation of the national reform program for 2017, 2018 and 2019 included activities supported by the PforR and provided timelines and adequate status updates for key activities. There was a strong alignment of several indicators with contracting arrangements between HZZO and service providers, specifically at the primary level in terms of controlling prescription behaviors of primary care doctors, their participation in group practices, and at the secondary level in terms of controlling surgeries and procedures performed at outpatient level, implementation and adherence to incidence reporting, etc. Justification of Overall Rating of M&E Quality 76. The overall rating of M&E quality is Substantial. The Program’s M&E framework was sufficient to assess achievement of Program objectives, and to inform the direction of the Program. However, the lacking focus on the outcomes and shortcomings with M&E reporting and utilization explain the proposed rating. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social Rating: Moderately Satisfactory 77. At appraisal, an Environment and Social Systems Assessment (ESSA) was carried out. The general ESSA finding was that the Program, with its focus on the needs of patients, removal of inefficiencies, and improvement of the quality of provided healthcare services, was not a threat, but rather an opportunity to improve the existing environmental and social performance of the Croatian healthcare system, serving as a framework for thorough integration of environmental and social considerations into the urgent and unavoidable reform processes. 78. Performance of environmental and social (E&S) safeguards continued to be satisfactory until April 2017. At that time the E&S safeguards rating was downgraded to Moderately Unsatisfactory due to slow progress with implementing items in the PAP related to environment safeguards. At that time, only one of the four environment-related actions identified in the PAP were addressed, resulting in the environmental and social risks being downgraded from moderate to substantial. In July 2018, after the Program restructuring, the E&S Page 27 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) improved to Moderately Satisfactory based on the progress in activities to assess compliance of hospitals with Mandatory Quality Standards and environmental requirements as well as parallel efforts to move towards full hospital accreditation based on international standards for quality of care, including related environmental standards. The E&S performance continued to be moderately satisfactory until the end of the Program, and all environmental actions in the post-restructuring PAP were completed. 79. Three environment-related requirements were used for audits. Environmental quality standards included in the mandatory quality standards and new Ordinance on Accreditation Standards for Hospital Health Institutions are that the: (a) health care institution must have procedures for proper collection and temporary storage of waste; (b) health care institution must have a hazardous medical waste storage built according to the regulations in force; and (c) medical health institution must conclude a contract with authorized persons for the activity of collecting, processing, using and/or disposing of medical waste. 80. All audited hospitals received high scores and passed the required 80 percent compliance threshold to successfully pass the audit. Fiduciary Rating: Satisfactory 81. At appraisal, an integrated fiduciary assessment had been carried out. It assessed the fiduciary arrangements relevant to the Croatia PforR to determine whether they could provide reasonable assurance that the Program funds would be used for their intended purpose. The integrated fiduciary assessment comprised separate assessments of the fiduciary risks relating to the Program’s (a) procurement; (b) financial management; and (c) governance. The objective of the assessment was to provide references that could be used to monitor the fiduciary system performance during implementation, as well as to identify actions, as relevant, to enhance the performance of the systems. Findings from the assessment concluded that the overall fiduciary and governance framework was adequate to support the implementation of the Program and included summary of the key risks and the corresponding mitigation actions identified. 82. The Program’s fiduciary performance was rated Satisfactory throughout the Program’s implementation. A total amount of EUR 64,042,500 (US$72,732,451.75 equivalent) was disbursed by April 2020 upon verification of remaining DLIs. All Financial Management Reports were submitted on time and no issues identified during the implementation. C. BANK PERFORMANCE Quality at Entry 83. The team was effectively coordinating with the MoH at the identification and preparation stage. The joint work of the World Bank and the government helped identify priorities of the National Health Care Strategy 2012- 2020 that were supported by the PforR. The World Bank, together with the MoH, prepared and agreed on the Results Framework that included measurable targets for the Program, which were missing in the strategic documents. The Bank and the government have agreed to use an innovative PforR instrument to support strategic priorities in the country that has secured country commitments to implement the Program. 84. The shortcomings of WBG performance during the design phase relate to the lack of outcome indicators in the original results framework. The output-focused nature of the PDO and intermediate indicators has already Page 28 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) been discussed. At the same time the framework offered operational and feasible results that were supported by clear actions and offered measurable and clear targets. 85. The assessment of risks related to the implementation of the program was neither full nor sufficiently detailed. The design of the PforR relied strongly on the continuation of the reforms’ agenda and the same level of commitment to the reforms as at the design stage. Changes in the MoH leadership were not foreseen and the mitigation strategies for political volatility were not prepared in order to hand over and sustain the same level of ambition and implementation support. This could be explained by the long-term perspective of the National Health Care Strategy that was setting a timeline for implementation of the priorities for the period of 2012-2020. However, changes in the government between 2016 and 2018 were one of the key reasons for delays and required a lot of the team’s effort to revisit and revitalize implementation of the PforR. The absence of an appropriate technical assistance component in the Program did not let to adjust and strengthen implementation after the MoH experienced gaps in its capacity. Quality of Supervision 86. Program monitoring and supervision was adequate. The MTR and Program Implementation Status Reports were filed appropriately, with the required details on the Program implementation progress and challenges. Program monitoring documents were candid about the Program’s progress, providing realistic signals to the MoH when the implementation was unsatisfactory. Team leadership changed three times during implementation, with different was task team leaders leading Program preparation, MTR, and restructuring to closing. Despite these changes, the MoH appreciated the level of support and extended supervision provided by the World Bank team that was tailored to the needs at different stages of the Program implementation. 87. One of the suggested gaps in implementation support was the delay between the MTR and Program restructuring. The MTR was conducted in March 2017 – later than the original mid-term point of August 2016 – and signaled the deteriorating progress towards achieving the PDO. At that time the implementation performance, previously consistently rated as ‘Satisfactory’, was downgraded to ‘Moderately Satisfactory’, and further downgraded to ‘Moderately Unsatisfactory’ in October 2017 after the MoH failed to take timely actions on the proposed restructuring. Lack of government action since the MTR to address implementation bottlenecks contributed to further delays, but the restructuring was only finalized in June 2018, more than a year after the team diagnosed implementation problems. This delay, however, can be explained by the low level of participation of the MoH team during 2017 in the implementation, and lowered levels of commitment to the objectives of the PforR. Justification of Overall Rating of Bank Performance 88. Bank’s performance is assessed to be Satisfactory. The proposed rating is a composite result from the assessment of the key factors that affected Program preparation, implementation and outcomes. D. RISK TO DEVELOPMENT OUTCOME 89. The risk that the achieved PDO indicators will not be maintained is Moderate. Political risk is low since the MoH, HZZO, and EU as one of the key donors in the health sector are committed to continuing the Program’s agenda of improving quality and efficiency of health services. The MoH has been reducing the unnecessary acute beds capacity, but the total number of beds in the country is still above the EU averages (5.5 beds per 1000 people in Croatia compared to 5 beds in the EU)14. The commitment to continue hospital integration may help the country achieve further improvements in rightsizing the capacity of its hospital sector. The commitment to continue the 14OECD/European Observatory on Health Systems and Policies (2019), Croatia: Country Health Profile 2019, State of Health in the EU, OECD Publishing, Paris, https://doi.org/10.1787/b63e8c9f-en. Page 29 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) hospital accreditation process was confirmed by the MoH during the ICR interviews. Development of an accreditation system is seen as an enabling factor for improved safety of patients, staff, and environment, efficiency of providers and increased levels of public trust in the health system. However, the non-mandatory status of the accreditation and modest plans of the MoH to conduct accreditation in 2020 in one to three hospitals may be a signal of modest levels of commitment to the accreditation agenda. 90. There are technical risks associated with MoH commitment and capacity to manage continued implementation of hospital integration and centralized procurements. The risk of MoH not having adequate technical capacity is moderate, because the MoH demonstrated its commitments to continue both directions and explained planned actions to sustain and expand both activities. For hospital integration, the MoH reported that changes have been introduced in the legislation that enable further activities in the process of functional integration. For centralized procurements, despite challenges in the technical capacity and risks associated with lengthy procedures that provide marginal savings, the MoH maintained its commitment to continue centralized procurements potentially accompanied by revising the number of procurement categories and setting up a dedicated procurement team. V. LESSONS AND RECOMMENDATIONS 91. This Program shows that PforR can support long-term and system-wide changes in the health sector. Croatia was the first country in the region that borrowed using the PforR instrument for health system development. The PforR was fully aligned with an existing Government program – National Health Care Strategy – which was key to the successful implementation of the operation. This Program had a very promising start because of the ownership of the reforms agenda, and presence of the instruments to quickly achieve significant results. The first two years of implementation was outstanding, with many indicators achieved and the disbursement rate reaching 55 percent (EUR 47.7 million). However, notwithstanding the sustained priorities of the National Health Care Strategy that the PforR supported, the changes in the political arena and MoH leadership resulted in significant delays. It took a lot of effort from the implementing team to reconfirm the PDO and to restructure the Program, including a partial cancellation of loan funds. An alternative approach could have been to use the restructuring for more radical revision of Program’s scope and remaining DLI targets. By the time of the ICR the Program had disbursed an additional EUR8.7 million, while the disbursement of the remining EUR 21 million is expected to take place within grace period after Program closing. The important lesson is to use the MTR and restructuring to adequately assess the achievements and issues and use such instruments to realistically adjust program scope and implementation arrangements. 92. For innovative instruments of lending, such as the PforR, the World Bank may need to actively support and align activities of different national partners. Because of the design of the PforR, MoH only received less than 10 percent of the Program amount for technical assistance to support its activities. Therefore, the MoH, its quality unit, and the HZZO were not financially motivated to achieve targets. The potential need for strengthening of the technical aspects of the MoH, allocating more resources for technical assistance support, or setting up a dedicated program implementation unit at the time of the restructuring, was not considered. The important lesson is to ensure a well-designed system of incentives in the Program for the implementing partner, and to technically support the Program activities, as well as routinely revisit such incentives and support, making necessary adjustments during MTR and restructuring of the PforR. 93. The results framework should be realistic and practical, but targets for PforR should be a combination of the output and outcome levels. The Program and its results framework were supporting ambitious targets, which were possible to achieve only with favorable political situation (e.g. hospital Page 30 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) restructuring, reducing hospital arrears), but were not adequately supported by technical assistance, a program implementation unit or financial allocations for achievement of such targets. The results framework proposed during Program design was somewhat short of outcome indicators. The PDO indicators and DLIs were mostly output-level targets. Implementation of the activities to achieve outcome level targets (e.g. increase of outpatient-based procedures and same day surgeries) was effective and allowed to see results that prompted improvements in the efficiency and quality agenda). The presence of measurable and practical indicators, as well as regular reporting from the MoH, its quality unit and HZZO to the World Bank and accountability of the MoH helped several actions to materialize. However, the Program did not pay enough attention to communicate its activities to external partners and beneficiaries (there was no dedicated public information campaign) and could not appropriately measure changes at the level of patients. The important lesson is, therefore, the need to pay enough attention to not only have well-defined and operationally feasible targets, but also make sure that the indicators include realistic output and outcome level targets and the procedures for their measurement and verification are well defined. 94. The PforR was supporting ambitious agenda but was not sizeable enough to be financially attractive to incentivize action for results. The initial estimate of the expenditures associated with the implementation of the National Health Care Strategy 2012-2020 at EUR 409 million was two times lower than the actual expenditures of EUR 955 million associated with its implementation in 2012-2019. The support provided through the PforR was offering less than 10 percent of the total cost of the National Health Care Strategy while supporting 5 out of 8 priorities, and therefore could not provide a sufficient financial incentive to reach the ambitious targets it had set. The lesson would be to select priorities in the PforR that are proportional to the proposed financing, and to financially incentivize the achievement of expected results . . Page 31 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) ANNEX 1. RESULTS FRAMEWORK, DISBURSEMENT LINKED INDICATORS, AND PROGRAM ACTION PLAN Annex 1A. RESULTS FRAMEWORK (i) PDO Indicators Objective/Outcome: Improved quality of health care in Croatia Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Quality control procedures in Percentage 0.00 40.00 40.00 90.00 place: Percentage of best- performing rationalized 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 hospitals which are publicly disclosed (including results) based on the technical audit in the preceding 12 months Comments (achievements against targets): Achieved. The Bank confirmed achievement of the final target of this PDO-level indicator (40%) on November 12, 2018. The MoH plans to conduct a new round of technical audits of hospitals in 2020. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Quality control procedures in Percentage 0.00 50.00 80.00 100.00 Page 32 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) place: Percentage of acute 30-Jun-2018 31-Dec-2017 31-Oct-2019 30-Oct-2019 hospitals meeting mandatory quality standards, as defined in the ordinance of Quality Standards and their Implementation from 2011. Comments (achievements against targets): Following the external hospital audits conducted in May-July 2019, the expert committee established by the MoH has confirmed that all the 33 acute care hospitals assessed in December 2018 met a minimum of 80% compliance with the mandatory quality standards. Objective/Outcome: Improved efficiency of health services in Croatia Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion First phase of the hospital Number 15930.00 12800.00 12800.00 12315.00 master plan achieved: Total number of hospital beds in 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 Rationalized Hospitals classified as acute care beds Comments (achievements against targets): Achieved. The Bank confirmed achievement of the full target (12,800) for this PDO-level indicator at an actual value of 12,161 on February 15, 2017. Based on data from HZZO, the latest reported value for this indicator is 12,315 beds, with the increase of 154 beds explained by the separation of two previously merged acute care hospitals (General hospital Nova Gradiška and General hospital Pakrac). Page 33 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) (ii) Intermediate Results Indicators Results Area: Quality of health care Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 5. Percentage of Primary Percentage 0.00 60.00 60.00 99.43 Health Care group practices achieving performance 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 indicators and accessing to payment incentives. Comments (achievements against targets): Surpassed. The reported value is dated August 31, 2019. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 8. Number of counties Number 1.00 15.00 15.00 19.00 implementing specialized services on palliative care. 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 Comments (achievements against targets): Surpassed. Data reported is dated June 30, 2019. Page 34 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 1. Varaždin County 2. Požega Slavonija County 3. Osijek-Baranja County 4. Međimurje County 5. Istria County 6. Karlovac County 7. Vukovar-Srijem County 8. Dubrovnik-Neretva County 9. Zadar County 10. Šibenik-Knin County 11. City of Zagreb 12. Sisak-Moslavina County 13. Primorje-Gorski Kotar County 14. Krapina-Zagorje County 15. Virovitica-Podravina County 16. Brod-Posavina County Page 35 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 17. Zagreb County 18. Vukovar-Srijem County 19. Koprivnica-Križevci County Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 10. Percentage of Percentage 0.00 30.00 30.00 100.00 Nosocomial Infections Surveillance system in place 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 in tertiary hospitals and secondary hospitals. Comments (achievements against targets): Surpassed. Data for 2018 from Expert Committee for Hospital Infections. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 11. Percentage of hospitals Percentage 0.00 60.00 60.00 82.00 with surgery wards that have established quality-and 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 safety-related sentinel surveillance schemes that are reporting the rates of Page 36 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) specific events. Comments (achievements against targets): Surpassed. The value reported for this indicator in 2019 is 100% for three sentinel events that are collected through the online information system (wrong patient surgery, wrong site surgery and postoperative pulmonary embolism). For the forth sentinel event (non-traumatic diabetes-related lower limb amputation), the data is not collected automatically and will be available in December 2019 once hospitals send them to the MoH in excel sheets by email. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 12. Percentage of patients’ Percentage 38.00 75.00 75.00 38.60 satisfaction by a perception of increased responsiveness. 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 Comments (achievements against targets): This indicator is behind target values. Data for 2017. There was no survey in 2018. A 2019 follow-up survey was conducted, but results are not available yet at the time of ICR preparation. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 13. Ordinance approving Yes/No N Y Y accreditation standards and procedure issued. 30-Jun-2018 31-Oct-2019 30-Oct-2019 Page 37 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Comments (achievements against targets): Achieved. Following public consultations held in August 2019, both ordinances (Ordinance on Accreditation Standards for Hospital Health Institutions OG 92/2019 and Ordinance on Methods, Conditions and Procedures for Accreditation of Health Care Provider OG 92/2019) were adopted and published in the Official Gazette on September 30, 2019. https://narodne- novine.nn.hr/clanci/sluzbeni/2019_09_92_1825.html https://narodne- novine.nn.hr/clanci/sluzbeni/2019_09_92_1826.html Results Area: Efficiency of health services Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. Ratio between primary Text 58 / 18 / 1 90 / 19 / 1 90 / 19 / 1 83 / 22 / 1 health care/secondary outpatient care/ and hospital 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 inpatient care services Comments (achievements against targets): Substantially achieved. CHIF (HZZO) data as of June 30, 2019. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Number of “hospital Number 0.00 2.00 2.00 2.00 reshaping scheme” projects implemented 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 Page 38 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Comments (achievements against targets): The MoH submitted to the Bank a formal notification with supporting evidence on October 15, 2019 claiming achievement of the target. The achievement was confirmed formally by the Bank following a review of submitted evidence. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 3. Percentage of all surgeries Percentage 5.00 60.00 60.00 60.94 included in the elective surgeries list performed as 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 outpatient surgeries in the preceding six months. Comments (achievements against targets): Surpassed. This indicator was achieved in February 2016. Since then, the maximum value of 63.34% for this indicator was reported based on HZZO data of June 30, 2018. This value dropped to 55.80% by December 2018 and increased again to 60.94% as of June 2019. These fluctuations have been due to intensive reconstruction activities conducted by majority of the hospitals since late 2016, with European Union funding received under the Call for Proposals "Improving cost-effectiveness and access to day hospitals and / or day surgeries” for equipping of the premises for the day hospital and day surgery activities, as well as the construction, adaptation and reconstruction of the existing premises for performing day surgery activities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 4. Percentage of primary care Percentage 0.00 50.00 50.00 70.33 Page 39 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) doctors working in group 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 practices. Comments (achievements against targets): Surpassed. This indicator was achieved in June 2015. Based on HZZO data, the latest reported value for this indicator is 70.33% as of August 31, 2019. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 6. Percentage of identified Percentage 20.00 90.00 90.00 93.10 doctors with whom a corrective course of action 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 has been discussed on a person-to-person basis in preceding six months. Comments (achievements against targets): Achieved. This indicator was achieved in June 2015. Since then, HZZO monitors doctors' prescription practices as part of Key Performance Indicators (KPIs) on a monthly basis. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 7. Percentage of total public Percentage 0.00 60.00 32.34 44.53 spending per fiscal year on medical consumables, drugs, 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 Page 40 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) and devices for hospital (inpatient and outpatient) services made through centralized procurement/ framework cntrc Comments (achievements against targets): The MoH submitted to the Bank a formal notification with supporting evidence on October 15, 2019 claiming achievement of the target and following the Bank's review achievement was confirmed formally on April 8, 2020. In addition to three tenders finalized in late 2018, in 2019, the MoH Public Procurement Unit concluded framework agreements following centralized procurement procedures for special groups of drugs and disposable materials for nuclear medicine. With these, the share of total annual value of framework agreements concluded by the MoH through centralized procurement in the total amount of actual spending by the 10 hospitals whose founder is the Republic of Croatia reached 44.53%. This exceeded the planned target value of 32.34% for this indicator. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 9. Number of proposals for Number 0.00 8.00 8.00 137.00 EU structural funds accepted by the relevant authority 31-Dec-2012 31-Dec-2017 31-Oct-2019 30-Oct-2019 after being correctly submitted. Comments (achievements against targets): Surpassed with a total of 137 proposals for EU structural funds accepted. 27 – Day hospitals (construction and medical equipment) Page 41 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) 6 - EMS departments (construction and medical equipment) 4 – Vulnerable groups (construction and medical equipment) 17 – Primary health care (construction and medical equipment) 1 – Živjeti zdravo (Living Healthy) implemented by the Croatian Public Health Institute 1 – Continuous professional training of family medicine practitioners 1 – Continuous professional training of EMS practitioners 66 – Specialist training of medical practitioners 13 – Health promotion and prevention 1- Establishment of emergency naval medical service by speedboats ANNEX 1B. DISBURSEMENT LINKED INDICATORS DLI IN00704669 TABLE DLI 1: 1. Total number of hospital beds in Rationalized Hospitals classified as Acute Care Beds. (Number) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 15,930.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 15,000.00 0.00 0.00 12,800.00 12,800.00 Page 42 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 3,750,000.0 0.00 Disbursed amount ($) 0.00 7,500,000.00 0 0 Comments (achievements against targets): The Bank confirmed achievement of the full target (12,800) for this PDO-level DLI at an actual value of 12,161 on February 15, 2017, with subsequent disbursement of the remaining balance of EUR1,315,909 allocated to this DLI. Based on data from HZZO, the latest reported value for this indicator is 12,315 beds, with the increase of 154 beds explained by the separation of two previously merged acute care hospitals (General hospital Nova Gradiška and General hospital Pakrac). DLI IN00704670 TABLE DLI 2: 2. Number of “hospital reshaping scheme” projects implemented (Number) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 0.00 1.00 0.00 0.00 2.00 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 3,750,000.0 Disbursed amount ($) 0.00 7,500,000.00 0 0 Comments (achievements against targets): The MoH submitted to the Bank a formal notification with supporting evidence on October 15, 2019 claiming achievement of the target for this DLI. The achievement was confirmed formally by the Bank following a review of submitted evidence. Page 43 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) DLI IN00704671 TABLE DLI 3: 3. Percentage of rationalized hospitals without arrears incurred during preceding calendar year. (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 0.00 0.00 0.00 40.00 80.00 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 Disbursed amount ($) 0.00 3,750,000.00 0 Comments (achievements against targets): This PDO level DLI was dropped as part of the Restructuring completed in June 2018 and associated Loan amount of EUR 7.5 million was cancelled. DLI IN00704672 TABLE DLI 4: 4. Percentage of all surgeries included in the elective surgeries list performed as outpatient surgeries in the preceding six months. (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 5.80 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 30.00 0.00 0.00 0.00 60.00 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 0.00 3,750,000.0 Disbursed amount ($) 0.00 7,500,000.00 0 0 Page 44 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Comments (achievements against targets): Surpassed. This DLI was achieved in February 2016. Since then, the maximum value of 63.34% for this indicator was reported based on HZZO data of June 30, 2018. This value dropped to 55.80% by December 2018 and increased again to 60.94% as of June 2019. These fluctuations have been due to intensive reconstruction activities conducted by majority of the hospitals since late 2016, with European Union funding received under the Call for Proposals "Improving cost-effectiveness and access to day hospitals and / or day surgeries” for equipping of the premises for the day hospital and day surgery activities, as well as the construction, adaptation and reconstruction of the existing premises for performing day surgery activities. DLI IN00704673 TABLE DLI 5: 5. Percentage of best-performing rationalized hospitals which are publicly disclosed (including results) based on the technical audit in the preceding 12 months. (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 0.00 20.00 0.00 0.00 40.00 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 3,750,000.0 Disbursed amount ($) 0.00 7,500,000.00 0 0 Comments (achievements against targets): This PDO-level DLI has been achieved: top-ranking rationalized hospitals were determined based on technical audit results, and the MoH disclosed the results of technical audits for 90 percent of rationalized hospitals (28 out of 31) on its website on November 6, 2018. The full disbursement of DLI allocation (EUR 7.5 million) was made on December 10, 2018. DLI IN00704674 TABLE Page 45 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) DLI 6: 6. Foundations are in place for accelerated implementation of hospital accreditation (Yes/No) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values No Actual values Yes Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 3,750,000.0 Disbursed amount ($) 0.00 7,500,000.00 0 0 Comments (achievements against targets): Achieved. Following the external hospital audits conducted in May-July 2019, the expert committee established by the MoH has confirmed that all the 33 hospitals assessed in December 2018 met a minimum of 80% compliance with the mandatory quality standards. DLI IN00704675 TABLE DLI 6.1: 6.1: Percentage of acute hospitals meeting mandatory quality standards, as defined in the Ordinance of Quality Standards and their Implementation from 2011, confirmed through prxy-accrdtion process. (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 0.00 0.00 0.00 0.00 0.00 100.00 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 Disbursed amount ($) 0.00 3,750,000.00 0 Page 46 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) DLI IN00704676 TABLE DLI 6.2: 6.2: Ordinance approving accreditation standards and procedure issued. (Yes/No) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values No Actual values Yes Yes Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 Disbursed amount ($) 0.00 3,750,000.00 0 DLI IN00704677 TABLE DLI 7: 7. Percentage of identified doctors with whom corrective course of action has been discussed on a person-to-person basis in the preceding six months. (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 20.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 75.00 0.00 0.00 0.00 90.00 93.10 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 0.00 3,750,000.0 0.00 Disbursed amount ($) 7,500,000.00 0 0 Comments (achievements against targets): Achieved. The Bank confirmed simultaneous achievement of the first (75%) and final (90%) targets for this DLI at an actual value of 93.10% on June 17, 2015, and one-off disbursement of the full allocation of EUR 7,500,000 for this DLI was made against this achievement. Page 47 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) DLI IN00704678 TABLE DLI 8: 8. Percentage of total public spending per fiscal year on medical consumables, drugs, and devices for hospital (in and outpatient) services made through centralized procurement and disclosed (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 30.00 0.00 0.00 0.00 0.00 44.53 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4,042,500.0 Disbursed amount ($) 0.00 4,042,500.00 0 Comments (achievements against targets): In addition to three tenders finalized in late 2018, in 2019, the MoH Public Procurement Unit has concluded framework agreements following centralized procurement procedures for special groups of drugs for HRK 903,257,538.87 without VAT and disposable materials for nuclear medicine for HRK 21,870,318.98 without VAT. With these, the share of total annual of value framework agreements concluded by the MoH through centralized procurement in the total amount of actual spending by the 10 hospitals whose founder is the Republic of Croatia reached 44.53%. The MoH submitted to the Bank a formal notification with supporting evidence on October 15, 2019 claiming achievement of the target and the achievement was confirmed formally by the Bank following a review of submitted evidence on April 8, 2020. DLI IN00704679 TABLE DLI 9: 9. Percentage of primary health care doctors in the Republic of Croatia working in group practices (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Page 48 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Actual values 0.00 30.00 0.00 0.00 50.00 70.33 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 3,750,000.0 0.00 Disbursed amount ($) 0.00 7,500,000.00 0 0 Comments (achievements against targets): Surpassed. The Bank confirmed simultaneous achievement of the first (30%) and final (50%) targets for this DLI at an actual value of 65.11% on June 17, 2015, and one-off disbursement of the full allocation of EUR 7,500,000 for this DLI was made against this achievement. Based on HZZO data, the latest reported value for this indicator is 70.33% as of August 31, 2019. DLI IN00704680 TABLE DLI 10: 10. Percentage of hospitals with surgery wards that have established quality- and safety-related sentinel surveillance schemes that are reporting the rates of specific events (Percentage) Baseline Year 2014 Year 2015 Year 2016 Year 2017 Year 2018 Year 2019 Total Original values 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Actual values 30.00 0.00 0.00 0.00 60.00 82.00 Allocated amount ($) 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3,750,000.0 0.00 0.00 0.00 3,750,000.0 0.00 Disbursed amount ($) 7,500,000.00 0 0 Comments (achievements against targets): Achievement of the full target (60%) for this indicator at an actual value of 64.7% was confirmed by the Bank on June 7, 2017. Based on data from the MoH, the latest reported value for this indicator is 100% for three sentinel events that are collected through the online information system (wrong patient surgery, wrong site surgery and postoperative pulmonary embolism). For the forth sentinel event (non-traumatic diabetes-related lower limb amputation), Page 49 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) the data is not collected automatically, and hospitals send them in excel sheets by email. These data will be available in December 2019, and the final aggregate value for all the four sentinel events will be included in the ICR. ANNEX 1C. PROGRAM ACTION PLAN PAP_TBL Achieved Action Timing Completion Measurement (Yes/No) Expansion of the centralized procurement Due Date 31-Oct-2019 Yes 32.34% of total public spending on medical system consumables, drugs, and devices for hospital services in the preceding fiscal year to be made through centralized procurement/framework contracts and disclosed on the Ministry of Health website. Comments: Completed; 44.53% at Program closure and projected to further increase with finalization of remaining tenders. Establishment of a quality control Due Date 31-Oct-2019 Yes Protocol and tools available for technical audit. mechanism and define related quality and efficiency control protocol for technical audit of hospital Comments: Page 50 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Completed. The established quality control mechanism and defined protocol for technical audits have been applied conducting technical audits in all 28 rationalized hospitals, which were completed on December 1, 2017. Establishment of a sentinel event Due Date 31-Oct-2019 Yes Surveillance system in place and first reporting surveillance system available. Comments: As of November 2018, quality and safety related surveillance system in place in 82% of hospitals with surgery wards. Data for 2019 is expected to become available in December 2019. Internal financial audit units functioning in Recurrent Yearly Yes Audit reports. tertiary hospitals (increased in coverage by 20% each year over 5 years of the Program) Comments: Completed. The health institutions are covered with internal audit function to a high extent (either having their own internal audit department or benefiting from an internal audit department that is joint for a couple of healthcare institutions). Completion of the hospital rationalization Due Date 31-Oct-2019 Yes Final report approved by the MoH. plan Comments: The second new National Hospital Development Plan for the period 2018-2020 (based on the National Health Care Strategy for 2012-2020), was formally adopted by the Government on September 20, 2018. Page 51 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Feasibility and pre-investment studies for Due Date 31-Oct-2019 Yes Feasibility and pre-investments studies for two “hospital reshaping schemes” “hospital reshaping schemes” completed. Comments: The MoH hired independent consultants, funded by European Commission, to assess the functionality of hospitals that were integrated. The final report from the consultants was delivered on June 05, 2019. Training in preparation of applications for Recurrent Continuous Yes Training report. EU funds, including Energy Efficiency proposals Comments: Completed. Employees of the MoH and health institutions continuously attended training workshops on preparation of applications for EU funds, including on energy efficiency, organized by the Croatian Ministry of Construction and Spatial Planning. Technical and project documentation and Recurrent Continuous Yes Sub-project proposals. preparation of applications for EU funds Comments: 137 proposals for EU structural funds developed by the MoH and health institutions have been accepted to date by the relevant authority after being correctly prepared and submitted. HZZO Service for Control Directorate has: Recurrent Continuous Yes Inspection and supervision reports. (a) conducted regular inspections of Page 52 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) hospitals under the Program; and (b) conducted specialized reviews based on risk assessments or significant complaints regarding health services provided. Comments: All the planned full controls of the operation and execution of contractual obligations in hospital health institutions, and specialized controls in hospitals have been conducted regularly by HZZO. Establishment and implementation of Due Date 31-Oct-2019 Yes Program report. Program for improvement of Energy efficiency in the Healthcare sector and preparation of projects for EU funds absorption Comments: Between Nov.2018 and Aug.2019, 35 public buildings in the health sector were selected for support in implementation of energy efficiency measures through the EU ‘Energy Efficiency Use of Renewable Energy in Public Sector Buildings’ funding program. Establishment and implementation of the Due Date 31-Oct-2019 Yes Program report. Program for education and training for radiological and radiotherapy safety Comments: There is evidence of progress, but mostly outside of the Program's influence. Page 53 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Public disclosure of the best-performing Recurrent Yearly Yes Disclosed reports hospitals based on technical audit in the preceding calendar year Comments: Results of the technical audits were presented on September 27, 2018 and best-performing hospitals (including their results) were disclosed on the MoH website on November 6, 2018. Public awareness and communication Due Date 31-Oct-2019 Yes Communication campaign evaluation report campaign about the health sector reform Comments: The initially planned communication campaign was implemented in November-December 2015. The MoH has continued to use communications to successfully address concerns and create consensus on various reform initiatives. Rationalization of expenditures for Recurrent Yearly Yes Study report. pharmaceuticals Comments: HZZO regularly carries out annual calculation of prices of medicines and public tenders for determining the prices of medicines for the basic and supplementary list of medicines. 17 new medicines were added to the HZZO reimbursement lists. Diagnosis-Related Group (DRG) system. Due Date 31-Oct-2019 Yes Review report. Review of the costing of each group based on some pilot costing system. Comments: HZZO department for analytics and development of health services (DTP and DRG) continuously works on the analysis and improvement of DRG and DTP prices. Current DRG and DTP prices are published in the Official Gazette. Hospital management training Due Date 31-Oct-2019 Yes Training Report. Comments: MoH and hospitals’ managerial and other staff are regularly trained in different subject areas (from postgraduate education in health Page 54 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) management to trainings related to the absorption of EU funds, etc.). Implement mandatory environmental Due Date 31-Oct-2019 Yes Environmental standards. Report on quality standards. Prepare environmental implementation of mandatory environmental standards proposal (containing full sets of quality standards. environmental indicators envisaged by the ESSA) through AQAHS to be subsequently included in a comprehensive accreditation system. Comments: The recently adopted Ordinance on Accreditation Standards for Hospital Health Institutions incorporates and organizes environmental management within those standards based on existing national regulation requirements. Conduct a self-assessment and external Due Date 31-Oct-2019 Yes Report on self-assessment and external results results verification of mandatory verification of mandatory standards. standards, propose / outline Recommendations for monitoring system for the (recommendations for) monitoring system hospitals’ compliance with full accreditation. for the hospitals’ compliance with full accreditation (including comprehensive environmental indicators). Comments: Process of verifying acute hospitals' compliance with mandatory quality standards was completed on September 25, 2019. Monitoring system is specified in new Ordinance on Methods, Conditions and Procedures for Accreditation of Health Care Provider. Page 55 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Huihui Wang, Baktybek Zhumadil Task Team Leader(s) Antonia G. Viyachka Procurement Specialist(s) Lamija Marijanovic Financial Management Specialist Luis M. Schwarz Team Member Vera Dugandzic Social Specialist Daria Goldstein Counsel Jasna Mestnik Team Member Anna Koziel Team Member Mohirjon Ahmedov Team Member Natasa Vetma Environmental Specialist Adrien Arnoux Dozol Team Member Maya Razat Team Member Ma Dessirie Kalinski Team Member Olena Doroshenko Team Member Zinaida Korableva Team Member Ruzica Jugovic Team Member Marina Mijatovic Team Member B. STAFF TIME AND COST Page 56 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY13 14.400 113,202.32 FY14 32.051 204,325.73 FY15 0 75.52 FY16 0 11,576.76 FY17 0 33,164.61 FY18 0 1,523.34 Total 46.45 363,868.28 Supervision/ICR FY15 27.920 140,278.57 FY16 17.902 93,819.76 FY17 25.269 97,255.72 FY18 18.873 116,586.59 FY19 31.369 170,569.28 FY20 22.141 115,978.99 Total 143.47 734,488.91 Page 57 of 58 The World Bank Improving Quality and Efficiency of Health Services (P144871) ANNEX 3. BORROWER’S COMMENTS The draft report was shared with the Borrower on May 12, 2020. At Borrower’s request, several paragraphs of the ICR were revised. The Borrower did not agree with the statement that not engaging consultants in the project implementation unit to support the implementation of the project has created bottlenecks in its implementation. On the contrary, the Borrower believes that if the PforR activities would have been implemented by the external staff, complications and implementation delays would have been even larger. Page 58 of 58