Document of The World Bank Report No: 25735 IMPLEMENTATION COMPLETION REPORT (IDA-28890; PPFI-P9830) ON A LOAN/CREDIT/GRANT IN THE AMOUNT OF US$ MILLION TO THE FORMER YUGOSLAV REPUBLIC OF MACEDONIA FOR A Health Sector Transition April 10, 2003 CURRENCY EQUIVALENTS (Exchange Rate Effective April 11, 2003) Currency Unit = Macedonian denar 61.2 denar = US$ 1 US$ 0.0161 = 1 denar FISCAL YEAR 1996 2002 ABBREVIATIONS AND ACRONYMS CME - Continuous Medical Education HIF - Health Insurance Fund HIL - Health Insurance Law HSTP - Health Sector Transition Project ICB - International Competitive Bidding IPH - Institutes of Public Health IPU - International Project Unit MIS - Management Information System MOH - Ministry of Health MTR - Mid Term Review NTP - National Tuberculosis Control Program PEIR - Public Expenditure and Institutional Review PIC - Public Information Campaign PR - Public Realtions PHC - Primary Health Care PHCS - Primary Health Care School PSMAC - Public Sector Management Adjustment Credit SSAC - Social Sector Adjustment Credit TA - Technical Assistance TB - Tuberculosis WBI - World Bank Institute WHO - World Health Organization Vice President: Johannes F. Linn Country Manager/Director: Orsalia Kalantzopoulos Sector Manager/Director: Annette Dixon Task Team Leader/Task Manager: Jean-Jacques de St. Antoine MACEDONIA, FORMER YUGOSLAV REPUBLIC OF Health Sector Transition CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 1 4. Achievement of Objective and Outputs 7 5. Major Factors Affecting Implementation and Outcome 17 6. Sustainability 18 7. Bank and Borrower Performance 19 8. Lessons Learned 20 9. Partner Comments 22 10. Additional Information 23 Annex 1. Key Performance Indicators/Log Frame Matrix 24 Annex 2. Project Costs and Financing 25 Annex 3. Economic Costs and Benefits 27 Annex 4. Bank Inputs 28 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 30 Annex 6. Ratings of Bank and Borrower Performance 31 Annex 7. List of Supporting Documents 32 Project ID: P036089 Project Name: Health Sector Transition Team Leader: Jean J. De St Antoine TL Unit: ECSHD ICR Type: Core ICR Report Date: June 24, 2003 1. Project Data Name: Health Sector Transition L/C/TF Number: IDA-28890; PPFI-P9830 Country/Department: FORMER YUGOSLAV REPUBLIC OF Region: Europe and Central Asia MACEDONIA Region Sector/subsector: Health (54%); Central government administration (34%); Health insurance (12%) KEY DATES Original Revised/Actual PCD: 09/01/1994 Effective: 12/23/1996 Appraisal: 04/05/1996 MTR: 03/24/1999 Approval: 06/20/1996 Closing: 03/31/2000 03/31/2002 Borrower/Implementing Agency: GOVT. OF FYRM/MOH Other Partners: STAFF Current At Appraisal Vice President: Johannes F. Linn Johannes F. Linn Country Director: Orsalia Kalantzopoulos Kenneth G. Lay Sector Manager: Annette Dixon Ralph W. Harbison Team Leader at ICR: Jean J. De St Antoine Ellen A. Goldstein ICR Primary Author(s): Silviu Radulescu 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome:S Sustainability:L Institutional Development Impact:M Bank Performance:S Borrower Performance:S QAG (if available) ICR Quality at Entry: U Project at Risk at Any Time: Yes 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The objectives of the Health Sector Transition Project (HSTP) stated in the SAR were to: (i) improve the health of the population by enhancing the quality of basic health services; and (ii) support an initial phase of policy reforms to increase cost effectiveness, fiscal sustainability and patient choice within the health system. The specific objectives of the project components were the following: (i) health finance and management component to build capacity for policy-making and health system management; (ii) basic health services component to strengthen primary health care and health promotion, particularly in rural areas; and (iii) supply and distribution of pharmaceuticals component to help reduce the cost of essential drugs through reforms creating a more competitive pharmaceutical market. 3.2 Revised Objective: Although the project was restructured and simplified at the mid-term review (MTR), the development objectives were not revised. During restructuring, it was acknowledged that the project was excessively ambitious to tackle the complex problems in health finance and in the pharmaceuticals sector in a comprehensive way within the scope of this project. 3.3 Original Components: The initial project design included the following components and subcomponents, with their respective costs estimated at appraisal: I. Health Finance and Management (estimated cost of US$ 4.2 million, of which IDA US$ 4.0 million). This component was to provide support for technical studies, strengthening of management skills in the Ministry of Health (MOH), Health Insurance Fund (HIF) and health facilities, and the development of management information systems. The following subcomponents were included: A. Health Financing Reforms (estimated cost of US$ 0.57 million). This subcomponent was to provide technical support for preparation and implementation of health financing reforms including: (i) definition of a fiscally-sustainable benefits package, to be covered by mandatory national health insurance, as well as supplemental premiums and packages; (ii) revision of copayment and user fee policies and penalties for non-contribution to health insurance; (iii) adoption of a capitation payment mechanism for public and private sector primary health care; and (iv) adoption of an integrated medical and financial information system for the public sector health network, as a prerequisite to defining a new payment mechanism for hospital care. It was planned to support a resident Health Financing Advisor responsible for coordinating short-term teams of local and foreign consultants to carry out policy studies related to each reform area. B. Strengthening of Health Insurance Fund (estimated cost US$ 1.13 million). This subcomponent was designed to increase the capability of the Health insurance Fund (HIF) personnel to analyze and implement policy options, through fellowships and in-country training workshops in health economics, principles of insurance, health system planning and public advocacy techniques, as well as a study tour of comparative health financing systems. Capacity building for the HIF was to include technical and financial support to develop trainers and training seminars for staff of district HIF offices. - 2 - C. Strengthening of Hospital Management (estimated cost of US$ 0.39 million). This subcomponent would support the creation of a Macedonia Hospital Association for the sharing of information and training for hospital administrators to utilize the information system. Envisaged activities were a study tour, training seminars and development of a continuing education program in facilities management for hospital administrators and medical directors. D. Public Information Campaign (estimated cost of US$ 0.38 million). Activities planned under this subcomponent were to support public information and advocacy campaigns using social marketing techniques, to enhance understanding and acceptance of health sector reforms by policy-makers, interest groups and the general public. E. Financing Accounting System (estimated cost US$ 1.68 million). This component was designed to provide the tools for effective management, and allow for the future introduction of a case-mix based hospital reimbursement system. It was planned to implement the first phase of a medium-term automation plan for an integrated medical and financial management information system for public sector health facilities. Planned activities were: (i) to test the system in two pilot districts, receiving an automation package for the participating Medical Centers (general hospital and polyclinic), two primary health care centers, district Institutes of Public Health (IPH) and district HIF offices; (ii) to provide information technology to the HIF and National IPH offices; and (iii) to establish the necessary electronic linkage for data transfer and processing within the HIF network. II. Basic Health Services (estimated cost of US$ 7.8 million, of which IDA US$ 6.5 million). This component was to equip rural primary health care centers, upgrade skills of primary care personnel and supply essential materials to five high-priority disease prevention programs. The following subcomponents were included. II.1. Priority Prevention Program (estimated cost of US$ 4.4 million). Five programs were selected on the basis of their potential impact in reducing the national burden of disease and on the cost-effectiveness of the proposed interventions. A. Immunization (estimated cost of US$ 0.05 million). The project was to supply hepatitis B vaccines, while UNICEF would provide parallel financing for most vaccines needed to combat childhood diseases and control epidemics, aiming at maintaining coverage rates of at least 95 percent. B. Tuberculosis (TB) Prevention (estimated cost of US$ 0.97 million). The project aimed to improve prevention of tuberculosis, through better screening capacity by providing spare parts for x-ray equipment, photographic film and replacement of two outdated mobile clinics with one new mobile van. C. AIDS Prevention (estimated cost of US$ 0.42 million). Test kits were to be provided for HIV screening. D. Blood Donation (estimated cost of US$ 0.35 million). The subcomponent aimed to increase blood supply and restore donation levels, by financing promotional materials in local languages. E. Maternal and Child Care (estimated cost of US$ 2.12 million). Activities under this subcomponent were to provide material support for reproductive health services, for drugs and equipment for prevention and treatment of disease in neonates and infants and to support restructuring and expansion of an existing cervical cancer screening program. - 3 - F. Laboratory Diagnostics (estimated cost of US$ 0.32 million). Equipment and materials to strengthen laboratory diagnostics for the prevention and treatment of infectious diseases were to be provided. II.2. Primary Health Care (PHC) (estimated cost US$ 3.4 million) including: A. Equipment for PHC (estimated cost of US$ 2.9 million) This subcomponent would provide basic equipment kits to all rural PHC centers in the country (an estimated 150 PHC centers with one or more full-time physicians, 50 PHC centers to which a full-time physician had to be assigned, and more limited equipment kits for 50 centers without a full-time physician). B. Capacity Building (estimated cost of US$ 0.47 million). Activities planned under this subcomponent were to: (i) provide technical and financial support for workshops and study tours to help the MOH and professional associations develop their own permanent programs for continuing education of primary care personnel, broadening the skills of primary care physicians and identifying and enforcing quality assurance standards for primary care; and (ii) provide short-term fellowships to develop capacity within the MOH for planning and implementation of primary care strategies. III. Supply and Distribution of Pharmaceuticals (estimated cost of US$ 6.4 million, of which IDA US$ 5.4 million). This component was to finance the procurement of essential drugs for primary care, and to provide technical advice, training and equipment to improve public procurement, reimbursement and quality control. The following subcomponents were included. A. Drug Procurement and Capacity Building (estimated cost of US$ 4.06 million). Planned activities under this subcomponent were: (i) technical support to revise the Drug Bill and accompanying regulations with respect to registration, pricing and generic substitution, and establish essential drug lists for primary and hospital care; (ii) building of local capacity for competitive procurement and for implementation of a reference price reimbursement system for HIF, through external internships in international procurement offices, technical assistance, studies to establish reference prices and support to carry out initial international tenders for drugs; and (iii) procurement of essential drugs and supplies from an agreed list, that were to ease HIF resource constraints and provide practical experience in international competitive bidding for pharmaceutical products. B. Drug Control Laboratory (estimated cost US$ 1.7 million). The plan was to support the provision of advanced drug testing equipment for quality control laboratories, short-term fellowships, in-country training and technical assistance to upgrade the drug quality assurance system. C. Pre-clinical and Clinical Pharmacology (estimated cost of US$ 0.38 million). This would help establish a system to test the possible adverse side effects of drugs. D. Drug Information and Rational Use (estimated cost of US$ 0.26 million). Activities were to: (i) support the establishment of a drug information center and a small center for adverse drug reaction monitoring; and (ii) promote more rational use of drugs through information campaigns directed at prescribers, dispensers and the public. A condition of project effectiveness was for the government to sign a contract with a procurement agency to ensure the procurement and distribution of pharmaceutical products, medical equipment and supplies under the project. In addition, the enactment of a Drug Law, satisfactory to IDA, regulating the registration, trade and distribution of drugs in Macedonia and all provisions for implementation of this legislation completed was a condition of disbursement of this component. - 4 - IV. Project management ­ International Project Unit (IPU) (estimated cost of US$ 1 million, of which IDA US$ 0.9 million). This component was to support an IPU in the MOH, responsible for the coordination, management and monitoring of the HSTP. The IPU was to be headed by a Director, supported by a core full-time staff funded under the Project consisting of a Procurement Officer, Disbursement Officer/Accountant, and an Office Administrator/Interpreter. The component was also designed to facilitate the work of the IPU by allowing to contract short-term foreign and local consultancy services in project management and procurement, as well as policy-related areas such as health economics and private sector development. 3.4 Revised Components: Following a slow implementation, and very limited disbursements during the initial project years, the project was simplified and restructured in October 1999 following the MTR. It was clear that the project was overly ambitious and complex and the local capacity needed substantial strengthening to implement the project. It was agreed that Credit funds would be reallocated to the two high-impact sub-components which were performing most successfully, the Continuous Medical Education (CME) program for primary health, and maternal and child health. Seven of the 17 sub-components were cancelled, including: I.C. Strengthening of Hospital Management, II.1.A. Immunization, II.1.C. AIDS Prevention, II.1.D. Blood Donation II.1.F. Laboratory Diagnostics III.B. Drug Control Laboratory, III.C. Pre-clinical and Clinical Pharmacology. Four of the remaining 10 subcomponents were scaled back. There was a significant reduction of pharmaceuticals procurement The revised components were as follows: Component Cost (US$) Rating Basic Health Services - Prevention 5,070,000 HS Programs Basic Health Services - Primary 4,818,000 S Health Care Health Finance and Management 2,666,000 S Pharmaceutical Policy and Supply 1,550,000 U Project Management 1,288,000 S 3.5 Quality at Entry: A QAG or QER review was not available at the time of project preparation. However, this ICR rates the quality as unsatisfactory for the following reasons. First, although the project strategy and concept were generally valid, the design was too complex for the implementation capacity existing at project launch. Second, the project time frame of three years and a half was too short and, as a result, two extensions totalling two years were required to allow the simplified project to be implemented. Third, the project was linked to an excessively ambitious sector reform agenda, but technical preparation of some of the components was not substantial enough, and there was insufficient agreement of major stakeholders. Fourth, the monitoring and evaluation framework was weak. The project was intended to be an instrument for achieving one of the aims for investment operations stated in the 1995 Macedonia CAS (Report 14379-MK), i.e. "maintaining an adequate social safety net through institutional strengthening of cash benefit programs, development of employment services, and - 5 - restructuring of health and education systems." According to the CAS, lending in the health sector was contingent specifically upon a "a medium-term strategy for structural reform of the health system." This strategy was prepared by five Macedonian technical groups as part of project preparation with support from the Bank and the World Health Organization (WHO). In view of the difficulties and controversies that occurred during project implementation (e.g. regarding model for development of primary care services, based on family doctor), evidence has accumulated to sustain the view that, although technically sound, the health reform strategy developed in 1995-96 was not shared by significant shareholder groups in the health sector. The project design sought to balance policy reforms and capacity building with investments, though without consistent consideration of whether the beneficiaries of the investments were in fact those institutions and actors who would bear the costs and challenges of reform. The risks related to the many different emerging groups of winners and losers could have been mitigated if there had been a strong champion to provide leadership and mediate between stakeholders. Also, the use of a strong participatory process would have helped. These prerequisites did exist during part of the preparation phase, but disappeared during implementation. This came about because of frequent changes of the MOH leadership, polarization of people and institutions inside versus outside the project. During the first half of the project, even working groups, that were potential instruments for consultation and dissipation of tensions, became a factor of increased negative reaction towards the project, as they were influenced by stakeholders left outside the process. Some of the project subcomponents were not well linked to the development objectives, either conceptually (e.g. laboratory equipment) or as incentives for change (e.g. TB prevention, as designed originally). Some of those subcomponents lacked the adequate input mix (i.e. physical investments, technical assistance (TA) and training) to support the policy agenda. The overall impression given by the original design is that of a large number of small investment components grafted on a complex sector reform policy agenda, more likely to be found in an adjustment operation. Annex 1 of the Staff Appraisal Report (SAR) presents an elaborate matrix with "project benchmarks". Most of these benchmarks were legislative and regulatory changes, in addition to some restructuring measures, which is unusual for an investment operation. The policy changes covered in this matrix exceeded the borrower's implementing capacity, and the leverage that the Bank would have through this project was overestimated. In the event, reform implementation was supported subsequently by an effective coordination with two subsequent adjustment operations - the Social Sector Adjustment Credit (SSAC) and the Public Sector Management Adjustment Credit (PSMAC). The design of the prevention programs subcomponent was particularly complex: resources were spread thinly across many objectives and programs, each of which required separate strategy development, project planning, stakeholder and implementation agency arrangements. There was inadequate technical preparation regarding strategy and project planning. Resources were allocated to support only some of the necessary inputs to achieve development impact. The design focused excessively on financial levers for reform (payment systems, performance of insurance administration), and neglected organizational aspects, especially at the micro level. A number of issues were not properly taken into account, notably: (i) the lack of appropriate governance, accountability, management autonomy and incentives for public sector healthcare providers (there seemed to be an assumption that public sector providers would respond to payment arrangements in the way that competitive private organizations would do); and (ii) the excessive specialization of providers in primary health care (separate physicians/services dealing with adults, children, emergencies, obstetrics and gynecology in PHC, preventive care ­ immunizations, home care) and the relations of obstetrics and gynecology, neonatology, and pediatrics in the provision of prenatal and perinatal care. The lack of an early agreement regarding the strategic direction to take in these areas resulted in tensions and delays in the - 6 - implementation of the project, exposure of the project to controversies and turf battles by different interest groups, as well as an incomplete use of the potential of consultants' capacity. Technical assistance was excessively focused on the development of contracting and payment mechanisms, and on the clinical training of staff, and not enough on addressing organizational and human resources issues. Further gaps in the design of implementation arrangements were the lack of provision for technical specialists in the IPU, the underestimation of capacity required for the expected volume of procurement activities. These had to be corrected later through the recruitment of health specialists and the contracting of an external procurement adviser. Both measures have substantially improved project implementation. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: 1. Improve the health of the population by enhancing the quality of basic health services. DO rating: Satisfactory. The Basic Health Component of the project has contributed to an impressive 21 percent reduction of perinatal mortality. The perinatal mortality rate is the number of fetal (stillbirth) and early neonatal deaths (within the first week after birth) per 1000 total births. There is evidence of improvement in the quality of perinatal and primary care offered by Continuous Medical Education (CME) graduates. In perinatal care evidence based practices in maternity and pediatric units caring for neonates have been widely adopted, as a result of the training started at the Royal Alfred Hospital in Australia under the project, and continued in the newly established National CME Center for Perinatology and Neonatology. General practitioners have improved their capacity to identify and manage cases of diabetes and hypertension in primary care. The increasing trend of TB incidence registered in the early nineties has been reversed, reaching the lowest level ever in 1999 (27.6 new cases per 10,000). The increase in 2000 (31.6 per 100,000), is a cause for concern, but has to be interpreted in the context of an inflow of 400,000 refugees during the Kosovo crisis. 2. The second project objective ­ to support an initial phase of policy reforms to increase cost effectiveness, fiscal sustainability and patient choice within the health system ­ is presented in the outcome indicator matrix in three different sub-objectives, as follows: Enhance cost-effectiveness of health services. DO rating: Satisfactory. Decreasing average length of stay in hospitals, increased number of consultations per physician in outpatient care and a decreasing trend of the excessively high ratio of primary care physicians to population suggest improvements in allocative efficiency, through a change in the balance of inpatient versus PHC services in favor of the latter, as well as higher technical efficiency in both hospital and PHC services. Although these changes are difficult to be attributed to project activities alone, the improved skills of about 40 percent of all PHC physicians in Macedonia, associated with the increased availability of basic diagnostic and treatment equipment at PHC level, have improved the capacity of general practitioners to deal with current conditions manageable at this level. The CME sub-component and the pharmaceutical component also made a modest impact on improving the cost-effectiveness of prescribing. 3. Ensure fiscal sustainability. DO rating: Satisfactory. The Health Insurance Law, developed with the support of the project, was passed in Parliament on March 29, 2000 and satisfied one of the conditions of the SSAC. Bylaws on mixed-capitation payment to physicians, copayment by users, and priority prevention programs (covering uninsured as well as insured) have been adopted. They were designed to improve fiscal control in payments to PHC and increase revenue from users while protecting the poor and chronically ill. The revised positive list for drugs, and new information systems have improved the HIF's capacity to control costs. - 7 - Arrears of the HIF to health service providers have not been eliminated, as projected in the SAR. Since the restructuring at the MTR, not all of the factors which affect the fiscal sustainability objective continued to be tackled by the project. This goal has been picked up in other parts of the Bank's work program in Macedonia - the Public Expenditure and Institutional Review (PEIR) and PSMAC that focused on the problems of financial management and control in the HIF. The PSMAC conditions eliminated the cause of arrears to public sector healthcare providers by replacing a poorly implemented "points" system for paying hospitals with prospective global budget contracts. Arrears of the HIF and public providers to commercial suppliers (notably of pharmaceuticals) remain a cause for concern. The government has agreed to include conditions related to the stabilization of these arrears in the second and third phases of the Public Sector Management Adjustment Loan (PSMAL) program. 4. Increase patient choice. DO rating: Satisfactory. PHC reform has increased patient choice through patient enrolment and capitation-based payment to private family medicine practices. Progress was made in this area with the adoption of by-laws on primary care capitation payment, copayment and priority prevention programs. Five hundred private physicians (representing more than 25 percent of PHC providers) have signed capitation contracts with the HIF. A public information campaign has been implemented to inform patients about their rights, obligations and options under the new arrangements. An important aspect of patient choice introduced via contracts with private practitioners is that doctors from all regions are well represented among those who have contracts with the HIF. Overall, the project helped to lay the foundation for several areas of health reform, including: (i) primary health care reform, and CME for primary care doctors; (ii) perinatal health services development, which has reduced perinatal mortality through intensive training and investment in equipment; (iii) training of staff in HIF, MOH and other health sector institutions; (iv) development of hospital contracting, financial management and information systems in the HIF; (v) development of a national drug policy, the drug information center, and training of doctors in rational prescribing; and (vi) public information campaign to promote health reform. 4.2 Outputs by components: I. Health Finance and Management Component (actual cost of US$ 2.42 million, of which credit US$ 2.37 million). Rating: Satisfactory To achieve project objectives, interventions (training and equipment) were designed to build capacity in health finance related organizations (hospitals, HIF, and MOH) that would develop and implement health finance policy change in Macedonia. Some of these interventions and policy changes were to be tested in pilot areas before national implementation. Limitations in staffing and technical skills in the MOH, HIF and hospitals were noted, and therefore, training, study tours, and longer-term training at all levels were included in the project to enhance technical capacity. The project also intended to build capacity through the purchase of information system hardware and software, and supplement technical expertise through foreign and local technical assistance. Enhanced capacity would help design and implement policy change (participation policy, basic benefits package, capitation payment for primary care, and the foundation for a resource-based payment for hospital care). Policy changes were to be implemented through changes in laws and regulations. 1.A Health Finance Reforms. Foundations for the proposed policy reforms ­ clearer public - 8 - accountability, participation (copayment) policy, basic benefits package, and new forms of provider reimbursement, including primary care capitation pilots ­ have been provided by technical assistance under a Japanese Grant for project preparation, followed by a contract financed by the Credit. Using this information and other Bank technical assistance, and reinforcing the need for legislative change through coordination with adjustment lending (SSAC and PSMAC conditions), the MOH prepared a Health Insurance Law (HIL) designed to implement public accountability measures, reform of provider payment, a new participation policy as well as a basic benefits package. The HIL was adopted by the Parliament in 2000, and emphasized the separation of the HIF functions from the MOH. This separation is a rather standard institutional arrangement in European countries with social health insurance, but has been controversial with the MOH in Macedonia, and has been associated with lack of cooperation and conflict at times between the MOH and HIF. The HIL envisages that the MOH should hold the HIF accountable and gives it a range of powers to approve laws and bylaws affecting patient access, to monitor and analyze HIF performance, and to approve key resource allocation and planning documents of the HIF. However, lack of MOH capacity has undermined that the effectiveness of the statutory powers of the Ministry to hold the HIF accountable and to exercise strategic policy leadership. The MOH has complained of powerlessness though in practice, even prior to the HIL, the HIF always exercised de facto autonomy and greater power than the MOH through its control over resource allocation. Attempts to build the MOH's capacity to carry out its statutory responsibility more effectively in association with PSMAC have been undermined by hiring and salary freezes in the public sector which have been requirements of IMF program/standby arrangements throughout the period of the project, and by the instability associated with political appointments at several tiers of organization in the MOH. Part of the results of technical studies, carried out over the first three years of the project, and focusing on baseline surveys in pilot areas (the towns of Prilep and Ohrid) for the introduction of primary care capitation, were not further used. This was replaced by a more realistic approach adopted during the second half of the project, of providing for a gradual phasing in of PHC finance reform and capitation, first among private sector doctors and then among public sector doctors. In the last two years of the project, technical assistance and support to working groups was provided for the preparation of bylaws required for the implementation of the HIL, such as those regarding Government financed prevention programs, copayments and contracting and payment arrangements for PHC doctors. The latter provided for a mixed payment system, consisting of capitation and an incentive part prioritizing preventive activities. Technical support was also provided to help prepare proposals for amending the Health Care Law, that was presented to Parliament in Spring 2002. These amendments include changes in the organization of public provider organizations, separation of PHC from inpatient services, a shift from publicly owned and managed PHC centers towards privately operated PHC units, as well as licensing procedures of the health care workers in PHC. The law was not passed and this issue will need to be addressed again. During the last year of the project, a strategic document about the educational, financial and organizational reforms in PHC was issued after a long public debate and two workshops. The broad strategic recommendations of the report have been accepted through a Memorandum of Understanding between the MOH, the HIF, and the Faculty of Medicine, but a number of important details still need to be worked out. This issue will be addressed again under the proposed second project. A plan for redistribution of doctors and dentists was carried out in the early years of the project, but this has not been implemented because of limited central levers over local management and staff deployment. - 9 - The intention of Ministers has been to use the extension of capitation-based payment to public sector doctors as the lever to drive redistribution. However, legislation required to underpin the necessary organizational change of public sector primary care has not yet been passed, and has been controversial with labor organizations and professional associations. No evidence has been gathered on changes in patient access associated with the extension of capitation contracts to a larger number of private doctors. I.B Strengthening of the Health Insurance Fund. Training of experts from the HIF, MOH and other health institutions in the areas of health policy, financing and management has been provided mainly through participation in short courses, workshops and study tours. Fifteen participants (from the MOH, HIF, and IPU) attended the Flagship training program in Health Sector Reform organized by the World Bank Institute (WBI) in Washington (1998, 1999, and 2000) and in Budapest (2001). A local version of the Flagship Course was held in Macedonia in December 2001, attended by 44 participants. The local course was highly appreciated by participants, as a learning experience, but also as a rare opportunity for critical examination of reform alternatives in Macedonia by a diverse group of stakeholders. One doctor from the Republican Institute for Health Protection has successfully completed her Masters studies at Aberdeen University, U.K, financed by the project. The project planned to select a small number of candidates for masters-level training in health economics and health management to develop their potential to be the next generation of leaders in the HIF. The plan was put into practice only for one staff member, who attended a one-year training program in international health policy at the London School of Economics. This staff has held a senior position in the HIF for about two years and has been responsible for leading the reform program required to meet the key HIF conditions of the PSMAC. A recurrent constraint for implementing this subcomponent was the difficulty of releasing staff from their duties to attend training. I.C Hospital Management. Fifteen medical center managers attended short term health care management courses in Krakow, Poland in 1998 and 1999; and London, U.K in 1999. This initiative proved to be premature: it took place ahead of changes in the financing and management of medical centers. As well, management turn-over associated with the 1998 change of government, limited the scope to follow up the first phase of training with subsequent modules. As a consequence, this subcomponent was cancelled at the MTR, and funds reallocated. I.D Public Information. Public information activities were designed to seek support and enhance understanding and acceptance of health sector reforms among policy makers, interest groups and the general public. The first public information campaign (PIC) was implemented by a Slovenian based public relations (PR) firm, contracted to create and execute the media campaign, to help the MOH set up a public relations unit, educate journalists, facilitate their better understanding of the health sector reforms, and disseminate information on the progress of health reforms. In the lead up to and aftermath of the 1998 election, delays in the design and implementation of the reforms made it impossible to make full use of the PIC services contracted. The planned information campaign was not implemented and the contract was terminated. In 2001, when essential by-laws for implementation of the reforms were finalized and started to be enforced, the project supported additional public information activities. The PIC used a public opinion survey, polling, and focus groups to lay the foundation for the communication strategy for health reform. This was followed by a PR campaign with the objective to inform the public about the planned and adopted reforms in the health sector, to build public confidence in the reform, and to explain how these reforms affect individuals. The campaign included the strengthening of the MOH PR capacity through the employment of a local communications specialist, several TV and radio commercials (with an estimated 1.5 - 10 - million of the population reached), the distribution of 160,000 leaflets, as well as billboards and posters to health care institutions in 30 locations in Macedonia. The campaign also developed two training seminars for journalists who specialize in health care issues, media training workshop for the officials dealing with the health reform and included a study tour for journalists on Estonia's health care reforms. Overall, the PR campaign achieved most of its objectives and was well organized. However, the analysis of the effectiveness of the campaign revealed that the at-times strained relationship between the management of the MOH and the HIF became publicly visible and impacted the image of the reform efforts. In addition, because of the political developments triggered by the Kosovo crisis, the public at large paid relatively less attention to the government's communications efforts beyond political issues and military activities in Macedonia. There was also insufficient capacity in the MOH to coordinate and provide enough input and guidance for a proper PR campaign. These issues will need to be further addressed under the next project. The latter should also incorporate a beneficiary assessment tool that would continue to measure the level of public satisfaction with the country's health system with special emphasis on the issue of access to health services and quality. Overall, the public information activities conducted under the project significantly curtailed the initial negative media coverage of the health reform, contributed to increase the level of the society's understanding of the need for the reform, and generated more positive attitudes towards the proposed changes among key stakeholders. It is essential that the MOH maintains the momentum of the public information effort in the future. I.E Information Systems. The project included the development of a management information system and a cost accounting system to underpin a national set of hospital accounts. This simple system would allow for the tracking of costs, patient outcomes, and information that eventually could establish a resource-based provider payment or budgeting mechanism. Health centers in two pilot districts were to be linked electronically to the HIF's regional branches to allow transmittal of appropriate data. The procurement of the management information system (MIS) has been a difficult and lengthy process. The first tender was unsuccessful as no responsive bids were received. Then, re-tendering was abruptly stopped by the MOH prior to bid opening. This was because, after the 1998 elections, there was a new team in the MOH and HIF and it expressed concerns about the need for a two-stage process for MIS development. That would consist of a first stage of system design, followed by a second stage of actual procurement of hardware and software. After advice from the Bank, additional external technical assistance (TA) and work of the local experts, the specifications for the system were prepared. An additional extension of the September 2001 closing date of the project was granted to allow for completion of the Bank financed elements of the MIS. The US$ 1.34 million contract for the supply of the MIS for the HIF, was funded with US$ 1.13 million from the credit and US$ 0.23 million from the government. By the closing date of the HSTP, the credit funds for this contract were fully disbursed, covering 100 percent of the price of MIS sub-system 1 (all hardware and operating systems), 80 percent of the price of sub-system 2 (core business applications that have been already developed and installed, such as registration of insured, contribution records, administration of contracts and payments for providers, general accounting, assets management and payroll), and 20 percent of the prices of the sub-systems 3, 4 and 5 (other application software). The remaining payments for sub-systems 2, 3, 4 and 5, and the costs of training were to be paid by the government who undertook to provide financing until the completion date of the MIS, i.e. December 31, 2002. The new system will help link costs to inputs employed and outputs provided and will be used to establish - 11 - efficiency benchmarks for health expenditures. The applications developed will also allow to process the new methods of payment to primary care providers and support automated monitoring and enhanced analysis of new hospital payment systems currently under development with TA under PSMAC. II. Basic Health Services (actual cost of US$ 11.57 million, of which credit US$ 9.37 million). Rating: Satisfactory This component was by far the most successful of this project. It was designed to address inefficiency in organization of health care through the establishment of sustainable cost-effective, preventive and curative primary health care services. The success of this component is all the more noteworthy because most of the activities under this component were actually implemented during the last three years of the project. This was due to high levels of commitment from the relevant MOH staff and particularly through the extraordinary efforts exhibited by the IPU staff to help implement this component. II.1 Priority Prevention Programs A. Immunization. As Hepatitis B vaccines had not been procured by the time of the MTR, this sub-component was cancelled and funds reallocated. B. Tuberculosis prevention. A National Tuberculosis Control Program (NTP) for 2000-2005 was prepared starting from a draft produced in 1999 by Macedonian experts, improved with the support of external TA, and endorsed by the MOH. The TA, provided by the project, financed an expert identified by WHO. In collaboration with the National Institute for Lung Diseases and Tuberculosis, the expert assessed TB control in Macedonia and investigated the situation in existing TB health facilities. This led to the design of the National TB Program based on the DOTS Strategy. The program is sound in terms of objectives and takes into account international best practice. It includes the strengthening of tuberculosis reporting and surveillance, case management, case detection, case definition and equipment needs. The replacement of the two old mobile X-ray units equipped with mass-miniature radiography (MMR) with one new mobile unit, as described in the SAR, did not occur. This resulted from the recommendations by a Bank supervision mission in October 1998 concerning the appropriateness of the initially designed intervention from a public health perspective, cost-effectiveness, and recurrent cost implications. MMR does not give a sufficient incremental yield of identified cases to justify an investment in such a form of active case detection. Instead, active case detection of high risk groups coupled with improvement of passive case detection through strengthening of the PHC and diagnostic system of the existing health service was recommended as the most cost-effective approach for Macedonia. Based on the NTP and expert recommendations, new X-ray equipment, consumables and supplies were purchased and delivered to the National Institute for Lung Diseases and Tuberculosis and TB dispensaries located in six areas with a high number of cases (Skopje, Prilep, Kumanovo, Tetovo, Kicevo and Bitola). Laboratory facilities were upgraded in 15 project sites. Equipment includes micro-bacteria analyzer for TB Institute, laminar-flow cabinets, microscopes, incubators, film processors, and sterilizers. Program monitoring capacity has been strengthened through provision of computers and a vehicle. All sites that received X-ray equipment were renovated and functionally adapted, with civil works cofinanced by WHO and the government. Improved diagnostic and management have resulted in a decrease of the average length of stay (ALOS) in TB hospitals. The average length of stay in the TB wards only in two general hospitals Bitola and Shtip - 12 - decreased form 22.0 days in 1996 to 17.9 in 2000. In the two special TB hospitals, the ALOS decreased from 38.2 to 32.0 days on average from 1997 to 2000. Similar trends were found in each of the tertiary hospitals (for adults and children). The cooperation between the Bank, the government and WHO has been quite positive. The technical assistance and training provided by WHO to the National TB control program have been essential for the successful implementation of this subcomponent. C. AIDS Prevention. Following the MTR, this subcomponent was cancelled and funds reallocated. AIDS incidence remains low, but the risk of an epidemic is increasing because of the number of injecting drug users. This issue will be revisited during the preparation of the second project. D. Blood Donation Programs. Activities in the Macedonian Institute of Transfusion, such as design of promotional materials and ID cards, that were planned to be financed by the project have been carried out without use of resources from the Credit. Computers and software for transfusion services have been donated to the Institute by a local source. Following the MTR this subcomponent was scaled back and no further investments were financed. E. Maternal and Child Care. This was the most successful subcomponent of the project. It provided support to the National Perinatal Program, designed with significant input from a foreign consultant, in close cooperation with a local adviser and several local working groups, and implemented principally during 2000 and 2001. The program provided for a new organizational framework and functioning of perinatal services, based on regionalizing neonatal care, defined the range of services at three levels of care, introduced protocols for transfer of high risk mothers to Skopje, and established a national transport service for sick neonates. The program is guided by principles of evidence-based education and practice. Training focused on thermoregulation of babies, breast-feeding, infection control, and treatment of jaundice. The implementation of the program included two series of training (16-24 weeks course) at the Royal Prince Alfred Hospital in Sydney, Australia for an initial group of 15 doctors and 10 nurses. The trained medical staff then trained local staff working in neonatalogy services. The local training was conducted in the CME Center established at the Special Hospital for Obstetrics and Gynecology - Cair (a district of Skopje), which is fully equipped with modern educational technology. Trainees surveyed were strongly positive about the educational curriculum, teaching and assessment methods. There are now 115 certified doctors and nurses who have successfully completed four modules (eight weeks in total) of training. They represent about 50 percent of all doctors and 25 percent of all nurses working in the neonatology services in Macedonia. US$4 million worth of equipment for perinatal care was procured and installed under this program, including: l Neonatal/pediatric ventilators l Neonatal monitors, blood gas/pH analyzers, bilirubinometers, oximeter l Infant incubators, mobile and for transport, infant warmers l Lamps for phototherapy, infusion and syringe pumps, humidifiers l Ultrasound scanners, mobile X-ray unit l Special ambulance for emergency neonatal transport l Computers for neonatal departments with over 1000 births, with specific application software (neonatal data base) The rehabilitation of civil works, amounting to US$335,000, was a joint financing effort between the government and UNICEF. It involved three units in Skopje: the Pediatric Clinic, the Clinic for Gynecology - 13 - and Obstetrics, and the Neonatal Intensive Care Unit in the Pediatric Clinic. At the same time, under its Baby-friendly hospital initiative, UNICEF, reconstructed the neonatal wards in all district hospitals in Macedonia. The combined result of improved knowledge and skills, changes in care practices, improved physical environment and availability of state-of-the-art equipment was a dramatic reduction in perinatal mortality. A decrease from 27.4 to 21.5 per 1000 births, from 1997-1999 to 2000-2001 respectively, was registered in departments providing care to 93 percent of all newborn in 16 Macedonian hospitals. A 36 percent reduction in early neonatal deaths was also achieved during the same period. These are consistent with the shift in the process of care for neonates, using evidence based practice, that was observed in an evaluation carried out in early 2002 by consultants in 12 maternity units and two pediatric units. F. Laboratory Diagnostics. No laboratory equipment and supplies have been purchased. Following the MTR, this subcomponent was cancelled and funds reallocated, as the expected development impact was unclear in the absence of any coherent strategy or project plan for this area of service, and no progress had been made in implementation. II.2 Primary Health Care. One of the main achievements of the project was the success of the CME program for primary care practitioners. The CME began as a pilot program in 1998 with the objective of establishing a deep vocational training commitment by primary care doctors to the concept of life long learning. The goal was to produce a team of educators who would be able to provide CME to all PHC doctors in Macedonia. Under the pilot program, a CME strategy was designed, two training centers were established (in Prilep and Veles), seven educators were trained and 247 participants completed the CME basic retraining course. During the second phase that began in 2000, the program was expanded to two additional centers (in Strumica and Gostivar). As a result of this program 24 educators were trained, and another 431 physicians completed the CME basic retraining course. An additional 408 participants attended courses during the second year of CME program, including newly developed modules in healthcare management, sexually-transmitted diseases, women's health, Internet course for healthcare professionals. Eight educators are now able to design and develop new teaching modules independently. In addition, 15 guidelines and short guides were prepared and published, including the guidelines developed for the management of most common diseases at PHC level. A website was created. Overall, 40 percent of all PHC doctors in Macedonia have attended CME courses supported by the project. To complement the training program, 678 CME graduates of the main CME course received individual equipment and 110 PHC equipment sets were provided to clinics throughout Macedonia (totaling US$3 million). This training program supported small-group, peer-based, post-graduate refresher courses that helped to prepare doctors for forthcoming challenges in PHC. As a result of their experience, many PHC doctors left the public sector to establish quality, effective privately operated PHC clinics. Most of the latter have publicly funded contracts with the HIF. Other trainers and former participants led the way to establish internationally accredited training in pre-hospital emergency medicine. There are signs that the primary care sector has become self-starting in its development, which is positive for its future. There is also a shift to more effective, evidence-based practice in primary care, with dramatic reductions in ampular therapy (which has been excessive in Macedonia), improvements in management of asthma, high blood pressure and ischemic heart disease, better diagnostic and follow-up of diabetic patients (reflected in a reported prevalence closer to the expected level). The basis for improved patient-physician relationships has been established, PHC doctors in Macedonia providing now patient leaflets of their own design to inform patients of their rights and help them manage their health concerns more effectively. One of the most important results of this program is its sustainability. Due to its great success which - 14 - exceeded all expectations, the program has now been institutionalized under the form of a Primary Health Care School within the Medical Faculty. Funding for the school has been secured from the Health Insurance Fund (2 percent of the PHC capitation budget). Ownership and administration issues of the CME centers have been also dealt with, allowing further operation and adequate use of the training equipment and facilities. The future strategy and the institutional arrangements for implementing it are described in several important documents finalized in the last year of the project: a Strategy for CME for Primary Care Doctors, a Memorandum of Understanding between the MOH, the medical faculty, the HIF, and a professional association regarding the development of primary care standards and the program for primary care specialist training approved by the Faculty of Medicine. The specialization program will allow training of new physicians moving into primary care according to standards compatible with requirements in European countries with well established PHC services. A strategy and new standards for the accreditation and licensing of PHC doctors were also prepared during this second phase, and the by-laws of the Chamber of Physicians were revised. Accreditation and licensing standards developed under the project would be implemented with the participation of professional associations, as provided by the draft of the new Health Care Law, presented to Parliament in July 2002. The law was not passed because it was linked to privatization, a much more controversial subject. The issue of licensing and accreditation remains an important issue on the government's agenda and this topic will be addressed again in the context of the second project. III. Pharmaceutical Supply and Distribution (actual cost of US$ 1.51 million, of which credit US$ 1.36 million). Rating: Unsatisfactory [The rating compares the project's achievements with the original design, not with the redesigned component after the MTR] A. Drug Procurement and Capacity Building. The development and passage of new drug legislation was a condition of disbursement of the pharmaceutical component. As significant delays occurred, in part due to changes of Minister of Health,. the passage of new drug legislation was then made a condition of the first tranche of the SSAC and was adopted in 1998. The new Drug Law allowed for: (i) streamlined registration of drugs: (ii) reference pricing based on generic products; (iii) adoption of essential drug lists for public sector reimbursement; and (iv) competitive bidding procedures for public sector drug procurement. Drug procurement capacity building and procurement of drugs under this subcomponent were originally oriented around emergency procurement of essential drugs to address acute shortages in the sector in the early to mid-1990s. By the time of the MTR, however, these components had not been launched and there was no more emergency regarding pharmaceutical supply. At the MTR and in follow-up discussions with a new Minister of Health later in 1999, it was agreed to scale down the pharmaceutical procurement and use it primarily as a training exercise for staff in the MOH and HIF. Approximately half the budgeted resources for procurement of drugs were reallocated to component II of the Project. One MOH and two HIF procurement staff were sent to an international procurement training course. A foreign expert in pharmaceutical policy and procurement was contracted in 2000 to provide on-the-job training and advice on an international tender for a small list of essential drugs. MOH and HIF staff participated in all stages of the tender and in the evaluation committee, thus received on-the-job training in international pharmaceutical procurement. The tender achieved price reductions of up to 60 percent compared to prices paid at the time by the HIF for these drugs, and exceeded expectations. Because of this success, a second tender was conducted by the same team of trained local staff with foreign expert advice, and also achieved substantial savings. Some problems emerged with the distribution of a limited number of drugs due to overestimation of the local needs. - 15 - On the basis of the two international competitive bidding cycles (ICBs), the IPU developed a template of bidding documents for the procurement of pharmaceuticals in accordance with the local law on public procurement. Despite this progress and the success of the tenders conducted within the aegis of the project, the HIF and MOH have not used international tendering for pharmaceuticals in their own procurements. The positive experience of the tenders remains useful for possible future advocacy for improvement of procurement practices. B. Drug Control Laboratory and C. Pre-clinical and Clinical Pharmacology. These subcomponents had not made any progress by the time of the MTR. There were no plans for development of these laboratories, nor of the Adverse Drug Reactions Center. Also, there appeared to be unresolved issues about the duplication of roles of laboratories at the Republican Institute of Public Health and the Faculty of Pharmacology, pointing to a case for rationalization. In view of this lack of progress and the lack of evidence of expected development impact, it was agreed at the MTR that these subcomponents would be cancelled, and the budgeted resources transferred to Component II. D. Drug Information and Rational Use. In 1998, a Drug Information Center was established in the Pharmacy Department of the University of Skopje, but by the time of the MTR, it was serving only few inquiries. In 1999, the WHO Humanitarian Affairs Office initiated a pharmaceutical project. This provided an opportunity for collaboration and created conditions for the HSTP to increase the effectiveness of its support to the Drug Information Center, develop a national drug policy, and improve prescribing practices. The WHO project supported a resident foreign pharmaceutical expert and local experts who helped develop policy, consult stakeholders, and strengthen local institutions. The HSTP financed equipment and the publication of drug bulletins and their dissemination by the Drug Information Center. In May 2001, a national experts' workshop on National Drug Policy was conducted. It was the result of 12 workshops of four working groups, supported by the project and WHO technical assistance. After the workshop, the government adopted a National Drug Policy document. There is an ongoing process of preparation of the Implementation Plan for the National Drug Policy, and that will require additional technical assistance. Two courses for Rational Drug Prescribing in PHC were attended by 159 PHC doctors. Using the "Personal Formulary" concept and different educational formats, participants were taught principles of drug selection and rational drug therapy of common medical problems. The training was evaluated as extremely positive by the participants. IV. Project management ­ IPU (actual cost of US$ 1.64 million, of which credit US$ 1.42 million). Rating: Satisfactory The successful implementation of this project during the last two-three years is due to the outstanding efforts and technical expertise of the IPU staff. In addition, important internal changes in IPU management were made in 1999-2000 which greatly improved the functioning and the internal controls of the unit. Changes included the replacement of the IPU Director and former project disbursement officer, the hiring of a health specialist, a new disbursement officer, a procurement assistant, as well as an external management and procurement advisor to the IPU. The rules of IPU management and accountability were improved. Together, these changes allowed the IPU to complete all the project activities in the work program, disbursing roughly $13 million of the $16.9 million since early 2000. This is a remarkable progress and the IPU team is commended for its dedication and commitment. - 16 - 4.3 Net Present Value/Economic rate of return: NA 4.4 Financial rate of return: NA 4.5 Institutional development impact: The project has had a significant institutional development impact on health professionals and policy makers. Forty percent of all PHC doctors have received CME under the project. Their performance has improved and this should have an impact beyond the project. Quality of care should further improve as further doctors are expected to be trained by the Primary Health Care School established within the Medical Faculty under the project and for which financing has been secured. The training received by 50 percent of doctors and 25 percent of nurses working in neonatology services should have an impact on reductions in neonatal and perinatal mortality for many years after the project closure as the new evidence-based protocols of care continue being used. The Flagship training program in health sector reform has been attended by 44 participants with important positions in the health sector. The experience from other countries shows that, because these courses foster a better understanding of health policy and finance, they usually result in a more fruitful dialogue with the Bank and may lead to innovations in the sector. Project implementing capacity has been developed. A core group of Macedonian staff in the health sector has now the necessary skills to carry out project activities in compliance with Bank project monitoring, procurement and financial management requirements. Also, key staff in central administration institutions (MOH, HIF, and Faculty of Medicine) have become familiar with the requirements of Bank project implementation and are in a better position to act as effective partners for future projects. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: 1. The Kosovo crisis of 1999 broke just after the project was restructured at the MTR. This increased the difficulty of following up on agreed actions. Some activities, especially training, TA, and installation of equipment, were delayed because of security issues affecting the travel of local and international consultants, trainees, and IPU staff. During the internal crisis in 2001, project implementation was similarly affected. 2. Initial resistance from the medical community and professional associations, mainly related to lack of consensus on major reform directions, such as development of family medicine/ primary care practice or introduction of accreditation mechanisms for health care providers and training programs. 5.2 Factors generally subject to government control: 1. Successive changes in Ministers of Health and senior appointees in the MOH and HIF since project start-up have eliminated continuity; reduced political commitment and project ownership, and delayed the project. 2. Weak capacities in the MOH and HIF and weak inter-institutional collaboration and support e.g those described in section 4.4 regarding experience of implementation of the public information campaigns and of the management information system. - 17 - 5.3 Factors generally subject to implementing agency control: 1. The MOH was the lead implementing agency for this project, supported by the IPU for daily project operations. During the first three years of the project, the IPU went through many staffing changes, had weak financial management and project management functions, and a limited capacity in procurement compared to the volume of activities. IPU staff would require to improve their skills for technical inputs, such as developing specifications and terms of reference. One important weakness of the original IPU was the absence of the position of health specialist. There were also frequent delays in administrative and procurement-related decisions. All this affected implementation efficiency, causing unnecessary delays. 2. Delays in implementing the training and education activities foreseen under the project have affected the upgrading of skills and knowledge needed to implement both the policy agenda and the investment program. 5.4 Costs and financing: The original project cost estimate was US$19.4 million, of which US$16.9 million equivalent would come from the IDA Credit. At the close of the project, the total project cost was US$17.1 million, of which US$14.5 million was financed by the credit, and US$2.6 million was provided by government through financing of all civil works and tax obligations. Only US$0.8 million was cancelled from the credit, the remaining US$1.6 million being the result of changes in the SDR/US$ exchange rate. 6. Sustainability 6.1 Rationale for sustainability rating: The sustainability of projects benefits is rated as likely. Key achievements that will ensure sustainability are the substantial quality improvements of PHC and perinatal care, through improved staff skills and upgraded equipment that is now part of the daily operation of those services. The CME capacity developed by the project (infrastructure, the available curricula, competent and enthusiastic trainers) is likely to keep the momentum for training activities. Physicians and nurses who have not yet received this training are likely to press for the continuation of the CME program. The financing needs for further PHC training could be largely covered from the 2 percent of capitation payments that the HIF has agreed to allocate to that effect. Despite the continuous change of health ministers during the six years of project implementation (with 6 Ministers replaced), the project has helped improve the quality of the policy dialogue, maintain the momentum for health reform, and the degree of stakeholder acceptance of reform. Health reform components have been included in other projects in Macedonia (e.g. development of hospital payment mechanisms is supported as part of PSMAC) and the Government has sought Bank support for a further project. In spite of traditionally strained relations between the MOH and HIF, the two have achieved to coordinate their actions and reach agreement on key by-laws related to health sector reform, training and accreditation of PHC physicians and public information activities. Overall, the HIF's commitment to develop and implement reforms has considerably increased during the project and is continuing after project closure. As mentioned in Section IV (Project management), a core group of Macedonian staff in the health sector has acquired the skills to implement projects in accordance with Bank requirements and are potential effective partners for future projects. The follow-up Health Sector Development Project under preparation - 18 - will build upon the achievements of this project, help build more institutional capacity, which should further ensure sustainability. 6.2 Transition arrangement to regular operations: The operation of most of the equipment procured under the project is financed from the current expenditure budget of recipient institutions. As the project was approaching closure, the Bank underlined to the MOH the need to ensure that managers of health care institutions continue to allocate proper resources, especially for consumables needed for the use of medical equipment. The MOH has decided on the future operation of the regional centers for CME that have been created. Agreement has been reached by the Dean of the medical faculty, the Minister of Health, and the HIF Director regarding the strategy for the Primary Health Care School (PHCS), including its implementation plan, organization, functions and funding. The priorities currently being addressed by the PHCS team are the CME business plan, status of the educators, CME roll out to continue in September. It was expected that in Autumn 2002 the selection process of educators, tutors and clinics for PHCS would start, the Governance Board appointed and twinning with an established PHC training institutions in Europe would be initiated. The HIF has agreed to complete the implementation of all of the health information system and start using it for day-to-day operations. 7. Bank and Borrower Performance Bank 7.1 Lending: Bank performance is rated unsatisfactory for the reasons presented below. 1. The project design was too complex, with too many components. It also involved too much policy reform for a first-time borrower, a former Yugoslav republic where political instability was likely to be high in the 1990s. The final design looked fragmented. It was difficult to link some subcomponents with the overall project objectives and it appears that their inclusion may have responded to the pressure from specific interest groups or institutions. 2. The Bank did not pay enough attention to the setting up of a project unit with staff having the right technical profile or with enough capacity in relation to the expected volume of activities. 3. The monitoring and evaluation framework was weak. It only defined outcome and output indicators, without providing a complete baseline or making provisions for putting it together at an early stage. There was no elaboration on the system of data collection and reporting. Some of the outcome indicators (e.g. infant mortality) were too general, insufficiently sensitive or specific for evaluating the impact of actual project activities. 7.2 Supervision: The Bank performance is rated highly satisfactory. The supervision of the project was steady and regular during the first years of the project, then became quite intense during the second part of the project life. The Bank team was changed in 1999, and together with substantial support from Bank management, it made a special effort to improve the implementation of the project. Intensity and depth of supervision activities dramatically increased after the MTR, bringing in more realism, giving priority to high impact activities for which the political environment and the available capacity in the health sector were conducive to successful implementation. It creatively linked the health policy agenda with other instruments available to the Bank in Macedonia such as adjustments operations and analytic work on public expenditure. The technical side of supervision was significantly strengthened. The Bank team kept a daily contact with the government counterparts and IPU staff, and provided continuous support and policy advice to move the project forward. The team also made a serious effort to improve project management by the IPU, including project - 19 - financial management which were of serious concern. This performance is commendable because the project, as restructured during the MTR, had to be implemented during a civil conflict in Macedonia that started in 1999 with the Kosovo crisis (resulting in large numbers of refugees in Macedonia) and continued with ethnic violence that lasted up to the Framework Agreement of 2001. The scheduling of Bank missions as well as that of consultants was not always possible, but when missions had to be postponed for security reasons, regular contact was maintained with the IPU and the latter made special efforts to implement the project under difficult circumstances. 7.3 Overall Bank performance: Given the above ratings, the overall Bank performance is rated satisfactory. Borrower 7.4 Preparation: The Borrower's performance during preparation is rated satisfactory. The Minister of Health involved in project preparation provided strong leadership for reform. A consultative process, involving several working groups, provided inputs for the preparation of the project. 7.5 Government implementation performance: The Government's implementation performance is rated satisfactory (details below). However, counterpart funds were often provided after delays and significant efforts by the IPU and MOH. 7.6 Implementing Agency: The performance of the MOH, as well as that of the IPU it has established and maintained for project implementation, is rated satisfactory. The basis for this judgement is the same as for the Bank's performance: weak at the beginning, but very strong after the MTR. The project was extended twice for a total of two years compared to the original closing date. Project implementation has been highly successful during the last two years of the project, with a total of roughly US$13 million disbursed during that time i.e. 90 percent the total funds disbursed (US$14.5 million). This is despite the fact that several Ministers changed during the last 3 years (a total of 6 Ministers changed during the project) and despite the most recent political and military crisis in Macedonia which could have delayed the project further Much of the recent success of the project is due to numerous factors, including : (i) strong commitment made by the Health Minister in 1999 to move the project forward; (ii) the replacement of the previous IPU Director and disbursement officer, and the hiring of new staff who were very dedicated to the project; (iii) revamping of internal IPU guidelines and substantial improvement of the financing accounting system; (iv) substantial external management and procurement support to the IPU during the last three years; (v) very dedicated and committed team of specialists working in steering committees; and (vi) in depth Bank supervision. 7.7 Overall Borrower performance: Based on the above ratings, the overall Borrower performance is rated satisfactory. 8. Lessons Learned A realistic assessment of the environment for project implementation is crucial. Political instability, leading - 20 - to frequent changes in leadership, the overall low level of technical skills of key institutions in the sector (MOH, HIF), and the low level of motivation and pay proved to be obstacles for the progress of the project, especially in the first two years. What seemed to work better were large investments in equipment and in training in the case of services addressing major health issues where significant gaps existed in terms of staff skills and technical level of facilities. Although they required considerable implementation effort , these investments had a high impact. High professional quality and commitment of training providers can get results even under most difficult circumstances. This was the case of neonatal care and to some extent of PHC development. What barely worked, with huge difficulties in implementation and had more modest impact were complex policy changes or investments in complex technical systems (e.g. MIS), requiring detailed technical preparation and consensus of many agencies. These tend to take longer time than planned (sometimes beyond project life), sustained TA, long term development of skills of staff of local institutions. Consensus on the professional and political level is of greatest significance for the success of these activities and explicit mechanisms for achieving that consensus need to be built into project design, starting with the proper involvement of the main stakeholders in the preparatory activities. In addition to quality technical assistance, customized surveys and empirical data from routine sources are necessary inputs for success in policy reform. The implementation of changes has to be facilitated by properly timed and designed public information activities. A prerequisite for successful PR is good communication and coordination of key institutions at the top. Only late in the project have the MOH and HIF shown joint interest. Even then, public information activities insufficiently used the results of public opinion research, and the campaign was more a commercial than an educational or promotional product. What does not seem to work using the means available in an investment operation are changes of policy and regulatory environment in areas where vested interests are significant, there is collusion of interests of players within the sector against changes, and the potential for corruption high. A typical case is the pharmaceutical sector. More involvement of local experts and end users would have helped to better match the characteristics of the goods and services provided to the specific needs and conditions in Macedonia. Future projects could use more local input for: (i) defining the technical specifications and performances of the goods, quantities, lists of the end-users, working conditions of the end-users: (ii) providing expertise in drafting TORs for foreign consultants; and (iii) building support for the implementation stage through increased ownership from the initiation of procurement and selection activities. More involvement of local experts would also require an adequate planning of resources for local consulting services. However, during the early years of the project, contracting of individual consultants was affected at times by a lack of precision in the definition of outputs and in some cases by a lack of transparency in the selection processes. When this occurred, it led to difficulties in management of consultant work, and has affected the project's reputation with some stakeholders who felt excluded. During the second part of the project, both the Borrower and the Bank addressed this problem upfront. As a result performance significantly improved and good experience has been gained through a strengthening of selection and consultants' work monitoring procedures. The procurement of equipment and consumables from UNICEF has not been a good experience. Under the project, this method was contemplated for the procurement of US$2.2 million of goods, but this was - 21 - reduced to only US$365,000 because of complaints from end users. The main sources of grief were: (i) the low quality of the medical equipment and consumables; (ii) unacceptable long delivery times, 3-4 months after the deposit of funds; (iii) too close expiry dates of some of the medical consumables; and (iv) rigid and atypical commercial conditions (100 percent advance payment, no warranty, no say by the client in the technical specifications, origin of the goods or manufacturer). Complaints on quality had two origins: (i) those referring to the more "sophisticated" items such as sphygmomanometers, stethoscopes, and thermometers, where the end-users' opinion was that those items were weak and perishable; and (ii) complaints referring to instruments such as forceps, probes, and scissors, where the material of the instruments was considered to be easily corrosive. Much time and effort were spent in the procurement of goods because there were numerous packages for ICB, international shopping (IS), and national shopping (NS). To improve economy and efficiency in the procurement process, better procurement planning and packaging should be undertaken at the project preparation phase. Both the Bank and the Borrower paid insufficient attention to the setup, maintenance and systematic use of a monitoring and evaluation framework. A better selection of relevant indicators and the compiling of a baseline would improve the quality of future projects. It will also be important to plan adequate human and financial resources for periodic reporting, supervision and decision-making by the implementing agency. 9. Partner Comments (a) Borrower/implementing agency: Macedonia Health Sector Transition Project ­ IDA Cr.# 2889-MK Implementation Completion Report (ICR) Borrower's Part Certain steps forward have been made in the health sector transition, however, it is a fact that this sector is still facing serious problems. Given the importance and necessity to promote the health sector operations, a lot of work needs to be done in order to achieve acceptable quality and efficiency of the health services necessary for the population , both in the public and the private sector. Project implementation was directed towards the following major objectives: health financing, improvement of expert skills of employees in the health sector, strengthening primary health care and improvement of pharmaceutical policy and medicines procurement, to the end of reducing prices of essential medicines and creating a more competitive medicine market in the Republic of Macedonia. The achievements of determined objectives in the above-mentioned health areas is essential for the improvement of quality of life in the Republic of Macedonia, especially in the rural areas. However, having in mind the significance and willingness to improve the general conditions in health in general, the project realization began too optimistically, as well as the estimates of time required to overcome the very complex problems present in the health sector. A pleasing outcome is the significant progress in primary health care. The organized training of the medical - 22 - personnel (especially in the rural areas) has already resulted in an improvement in the quality of primary health care. There is, and it is also expected in the period to come, reduction of natal and perinatal mortality, which is also a sign of improved quality of primary health care services. With respect to the progress in the area of health sector management and financing, we should take into account that the development of this component was affected by the frequent changes in the management personnel, stemming from the political instability in our country. In addition, low salaries also reduce the motivation of the employees, thus adversely affecting the service quality. All this led to discontinuity, resulting in incomplete project implementation. Concerning pharmaceutical policy in the area of medicine procurement and given the difficulties in this area during project implementation, as well as the need to change the project activities concept, we should be content since significant contribution has been achieved in this area. Essential and positive lists of medicines related activities have been completed, doctors were trained in rational prescription of medicines, and a pharmaceutical policy document has been prepared. Several areas remain where activities should be further undertaken in the future. These are: implementation of priority preventive programs (tuberculosis prevention, AIDS prevention, promotion of blood donations etc.), equipping and promotion of laboratory operations, procurement of additional equipment needed for the primary health care etc. Large amounts of funds were invested in the health sector, both for training of personnel and for equipment, thereby better results are expected in the future. Hence, we believe that certain effects were accomplished during the project realization, however, not to the extent foreseen. (b) Cofinanciers: (c) Other partners (NGOs/private sector): 10. Additional Information - 23 - Annex 1. Key Performance Indicators/Log Frame Matrix Outcome / Impact Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate Baseline (1995) 2000 if not otherwise specified Infant mortality rate reduced. 22.7 per 1000 live births 11.8 per 1000 live births Maternal mortality rate reduced. 21.8 per 100000 live births 13.7 per 100000 live births Incidence of tuberculosis (per 100,000). 40 31.6 34.1 (2001) Average immunization coverage maintained 95.4% 95.2%, 91.4% (2001) above 95 percent. Consultations per day per primary care 3204 per year 3795 per year physician increased. Average length of stay in hospitals 14.3 11.4 decreased. Market price/world price of tracer drugs decreased (averaged ratio of all tracers). Arrears accumulation by NHIF/facilities 3500 million MKD (1998) 1692 million MKD arrears of NHIF to health eliminated (amount in MKD). care providers, 2120 million MKD arrears of health care providers to their suppliers (December 2001) Number of licensed private providers 340 484 contracted by HIF (May 2002) increased. Output Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate 1End of project - 24 - Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) Appraisal Actual/Latest Percentage of Estimate Estimate Appraisal Component US$ million US$ million 1.Health Finance and Management 3.80 2.40 63.2 2.Basic Health Services 7.00 11.60 165.7 3.Supply and Distribution of Pharmaceuticals 5.70 1.50 26.3 4. Project management ­ IPU 0.90 1.60 177.8 Total Baseline Cost 17.40 17.10 Physical Contingencies 1.40 Price Contingencies 0.60 Total Project Costs 19.40 17.10 Total Financing Required 19.40 17.10 Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) 1 Procurement Method Expenditure Category ICB NCB 2 N.B.F. Total Cost Other 1. Works 0.00 0.00 0.00 0.80 0.80 (0.00) (0.00) (0.00) (0.00) (0.00) 2. Goods 9.40 0.00 3.40 0.60 13.40 (9.40) (0.00) (3.40) (0.00) (12.80) 3. Services 0.00 0.00 3.90 0.80 4.70 Consultancy and Training (0.00) (0.00) (3.90) (0.00) (3.90) 4. Incremental Recurrent 0.00 0.00 0.00 0.30 0.30 Cost (0.00) (0.00) (0.00) (0.00) (0.00) 5. PPF Refinancing 0.00 0.00 0.20 0.00 0.20 (0.00) (0.00) (0.20) (0.00) (0.20) 6. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 9.40 0.00 7.50 2.50 19.40 (9.40) (0.00) (7.50) (0.00) (16.90) Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) 1 Procurement Method Expenditure Category ICB NCB 2 N.B.F. Total Cost Other 1. Works 0.00 0.00 0.00 0.83 0.83 (0.00) (0.00) (0.00) (0.00) (0.00) 2. Goods 7.66 0.00 1.40 1.50 10.56 (5.75) (0.00) (0.73) (0.00) (6.48) 3. Services 1.13 0.00 4.07 0.03 5.23 - 25 - Consultancy and Training (0.00) (0.00) (2.11) (0.00) (2.11) 4. Incremental Recurrent 0.00 0.00 0.12 0.32 0.44 Cost (0.00) (0.00) (0.00) (0.00) (0.00) 5. PPF Refinancing 0.00 0.00 0.13 0.00 0.13 (0.00) (0.00) (0.00) (0.00) (0.00) 6. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 8.79 0.00 5.72 2.68 17.19 (5.75) (0.00) (2.85) (0.00) (8.60) 1/Figures in parenthesis are the amounts to be financed by the IDA Credit. All costs include contingencies. 2/Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. Project Financing by Component (in US$ million equivalent) Percentage of Appraisal Component Appraisal Estimate Actual/Latest Estimate Bank Govt. CoF. Bank Govt. CoF. Bank Govt. CoF. 1.Health Finance and 4.00 0.20 2.37 0.05 59.3 25.0 Management 2.Basic Health Services 6.50 1.30 9.37 2.22 144.2 170.8 3.Supply and Distribution 5.40 1.00 1.36 0.16 25.2 16.0 of Pharmaceuticals 4. Project management ­ 0.90 0.10 1.41 0.25 156.7 250.0 IPU TOTAL 16.90 2.50 14.51 2.68 85.9 107.2 - 26 - Annex 3. Economic Costs and Benefits - 27 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation Jan.-Sept. 1995 Appraisal/Negotiation October 1995 Supervision 05/23/1997 6 SR INFO TECH SPECIALIS (1); S S MISSION LEADER (1); HEALTH POLICY ADVISOR (1); OPERATIONS OFFICER (1); SR. HEALTH SPECIALIST (1); PROJECT OFFICER (1) 02/13/1998 5 MISSION LEADER (1); U U HEALTH ECONOMIST (1); INFO SYSTEM SPECIALIST (1); IMPLEMENTATION SPEC. (1); PROJECTS OFFICER (1) 10/08/1998 6 TEAM LEADER (1); HEALTH S S ECONOMIST (1); IMPLEMENTATION/PROCMT (1); RM PROJECTS OFFICER (1); PHARMACEUTICAL SPEC. (1); MEDICAL EQUIPMENT (1) 03/24/2000 3 TASK TEAM LEADER (1); S S OPERATIONS ANALYST (1); PROJECT OFFICER (1) 07/14/2000 5 TASK TEAM LEADER (1); S S PROCUREMENT ADVISOR (1); OPERATIONS ANALYST (1); PROJECT OFFICER (1); PHC CONSULTANT (1) ICR 02/21/2001 1 TEAM LEADER (1) S (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation Appraisal/Negotiation - 28 - Supervision ICR Total - 29 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA - 30 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU - 31 - Annex 7. List of Supporting Documents - 32 - - 33 -