Document of The World Bank FOR OFFICIAL USE ONLY Report No: 20576-RO IMPLEMENTATION COMPLETION REPORT (34090) ON A LOAN IN THE AMOUNT OF US$150 MILLION TO ROMANIA FOR A HEALTH SERVICES REHABILITATION PROJECT JUNE 28, 2000 HUMAN DEVELOPMENT UNIT EUROPE AND CENTRAL ASIA REGIONAL OFFICE This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective June 26, 2000 Currency Unit = Leu 1 Leu = US$ 0.0000472 US$ 1.00 = Lei 21,182.50 FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS AIDS Acquired Immune Deficiency Syndrome CME Continuing medical education DHA District Health Authority ECA Europe and Central Asia EU European Union FPSEU Family Planning and Sex Education Unit GMP Good manufacturing practice GOR Government of Romania GP General practitioner HIH Health Insurance House HIV Human Immunodeficiency Virus HMIS Health management information system HSRP Health Services Rehabilitation Project IBRD International Bank for Reconstruction and Development ILI Intensive Learning Implementation Completion Report IOMC Institutul de Ocrotire a Mamei si Copiluli (Institute for the Protection of Mother and Child) LUMNY London School of Hygiene and Tropical Medicine, University of London; Universite de Montreal, Department de Gestion de Services de Sante; and New York University, Wagner School of Public Administration Studies (a consortium of universities) MOF Ministry of Finance MOH Ministry of Health MTR Mid-term review NCHS National Center for Health Statistics NHIH National Health Insurance House NGO Non Governmental Organization Vice President: Johannes F. Linn Country Director Andrew N. Vorkink Sector Manager Armin Fidler Task Manager Richard Florescu FOR OFFICALI USE ONLY NIHSM National Institute for Health Services Management (Institutul de Management al Serviciilor de Sanatate) PCU Project Coordination Unit PHRI) Population and Human Resources Development (Japanese Grant) PHC Primary Health Care SAR Staff Appraisal Report STD Sexually Transmitted Disease TA Technical assistance TB Tuberculosis TOR Terms of Reference UNAIDS Joint United Nations Program on HIV/AIDs UNFPA United Nations Population Fund UNICEF United Nations Children's Fund USAID UJnited States Agency for International Development WB World Bank WHO World Health Organization ii This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not be otherwise disclosed without World Bank authorization. IMPLEMENTATION COMPLETION REPORT ROMANIA: HEALTH SERVICES REHABILITATION TABLE OF CONTENTS Page No. 1. Project Data I 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 4 5. Major Factors Affecting Implementation and Outcome 15 6. Sustainability 17 7. Bank and Borrower Performance 19 8. Lessons Learned 20 9. Partner Comments 22 Annexes Annex I Key Performance Indicators/Log Frame Matrix 23 Annex 2 Project Costs and Financing 25 Annex 3 Economic Costs and Benefits 28 Annex 4 Bank Inputs 30 Annex 5 Ratings for Achievement of Objectives/Outputs of Components 32 Annex 6 Ratings of Bank and Borrower Performance 33 Annex 7 List of Supporting Documents 34 Annex 8 Beneficiary Survey Results 35 Annex 9 Stakeholder Workshop Results 42 Annex 10 Borrower Contribution to the ICR 48 Map 28710 iii Project ID: P008759 Project Name: HEALTH SERVICES REHABILITATION Team Leader: Richard Florescu TL Unit: ECSHD ICR 7Tpe: Intensive Learning Model (ILM) of ICR Report Date: 06/28/00 1. Project Data Name: HEALTH SERVICES REHABILITATION L/C/TF Number: 34090 Country/Department. ROMANIA Region: Europe and Central Asia Region Sector/subsector: HB - Basic Health KEY DATES Original Revised/Actual PCD: 12/19/90 Effective: 01/31/92 01/31/92 Appraisal: 04/09/91 MTR: 01/31/94 03/15/94 Approval: 10/01/91 Closing: 06/30/96 06/30/99 Borrower/lmplementing .4gency: MINISTRY OF FINANCE/MINISTRY OF HEALTH Other Partners: STAFF Current At Appraisal Vice President: Johannes F. Linn Countiy Manager Andrew N. Vorkink R. Cheetham Sector Manager: Armin Fidler Ralph Harbison Team Leader at ICR: Richard Florescu Julian Schweitzer ICR Primnaty Auithor: Loraine Hawkins The ICR team comprised: Richard Florescu (Operations Officer, Task Manager, responsible for commissioning ICR inputs and collating findings), Loraine Hawkins (Snr Health Specialist), Silviu Radulescu (Public Health Specialist), Dana Burduja (NIHSM graduate, emergency medical services component), Simona Buse (NIHSM graduate, data analysis), Roddy Neame (Health information systems specialist), Ruth Levine (Peer reviewer, HNP thematic group), Imelda Mueller (Program assistant). 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU- Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability. L Institutional Development Impact: L Bank Performance: S Borrower Performiance: S QAG (if available) ICR Quality at Entry: Not assessed S Project at Risk at Any Time: SU 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 O1riginal Objective: Because the transition process was expected to result in a changing environment, the original objectives presented in the Staff Appraisal Report (SAR) dated September 1991 were broadly defined in order to respond to the emergency needs of the early years of transition and to allow flexibility during the project implementation. These objectives were to: (i) rehabilitate and upgrade the primary health care delivery system which is collapsing through want of equipment, spare parts, drugs, medical supplies and transfer of knowledge to health providers; and (ii) support the first steps of a major restructuring of health sector financing and management to ensure a sustainable and cost-effective health care system in the medium term. 3.2 Revised Objective: There was no formal revision of the original objectives during the course of the project implementation, inspite of its long life (seven years instead of the four years initially planned). There were, instead, large reallocations of funds among the project components, as a result of: (i) significant savings obtained in the procurement process; and (ii) cancellation of some activities. 3.3 Original Components: In order to meet the above objectives, the components were designed as follows (cost estimates at appraisal are indicated, together with planned allocation of the US$150 million loan funds): A. Upgrade Rural Dispensaries (base costs US$25.4 million, of which IBRD US$16.9 million). Rehabilitate of 420 rural dispensaries countrywide, in selected sites, in order to attract physicians to work there; B. Improve Reproductive Health (base costs US$35.6 million, of which IBRD US$32.5 million). Establish a network of 230 Family Planning Local Centers; provide equipment, educational materials and supplies to upgrade ten reference reproductive health centers (in University Hospitals); establish a Family Planning and Sex Education Unit at the level of the Ministry of Health (MOH); provide medical equipment for 50 maternity/neonatal units to improve the diagnostic and curative capacity; and provide a wide range of contraceptive supplies in sufficient quantities and at affordable prices. C. Train Health Practitioners (base costs US$0.9 million, of which IBRD US$0.6 million). Support the development of the Post-Basic Nursing School in Bucharest and improve the 41 new nursing schools in each of the country districts. Improve continuing education in selected medical specialties for 650 general practitioners (GP) and nurses working at the upgraded dispensaries.. D. Procure & Distribute Drugs and Consumables (base cost US$93.2 million, of which IBRD US$83.2 million). Finance the procurement of essential drugs to cover the gap between Romania's minimum needs and supplies from domestic production and projected donor assistance; prepare a strategy for restructuring and rehabilitation of the domestic pharmaceutical industry; provide financing for the procurement of equipment to ensure continued production of vaccines at the Cantacuzino Institute from Bucharest (the sole domestic producer of vaccines); and supply laboratory equipment, reagents and consumables for the 41 district blood transfusion centers. E. Improve Management of Emergencies (base costs US$8.9 million, of which IBRD US$3.6 million). Provide technical assistance (TA) to review the emergency transport system, and to supply ambulances and related medical and communication system. 2 F. Health Promotion & People's Participation (base costs US$4.8 million, of which IBRD US$4.5 million). Strengthen national planning, coordination and management of health promotion by developing a National Center for Health Promotion and Education; develop and provide training to the staff working in the district health education laboratories attached to the District Health Authority (DHA), and supply equipment to produce promotion materials; establish a health promotion fund to finance programs designed by various providers, including non-governmental organizations (NGOs). G. Develop a National Health Strategy (base costs US$3.1 million, of which IBRD US$2.8 million). Finance studies to evaluate various aspects of the health system in Romania, and make recommendation for reforms; support Government efforts to decentralize management and control of the health system in order to improve resource utilization, increase the system responsiveness to the local needs, and encourage local decision making; and conduct a two-year pilot decentralization program in four selected districts. I. Develop a Health Information System (base costs US$3.2 million, of which IBRD US$2.8 million). Provide TA to design the system, train staff, and provide equipment to the MOH, National Health Statistics Center (NHSC) and DHAs to support the first phase development of a national health information system. J. School of Health Services & Management (base costs US$3.2 million, of which IBRD US$2.9 million). Support the establishment of the School of Health Services and Management in order to provide future trained health care managers and policy analysts and improve the policy-making, management and evaluation in the health care institutions. 3.4 Revised Conmponents: Due to various administrative and legislative changes and overall development of the Romanian economy during the project implementation, some components were adjusted (by diminishing or increasing their magnitude) or cancelled, or new ones were designed. The following activities were cancelled: The 41 district nursing schools, belonging to the MOH at the time of project appraisal, were transferred to the authority of the Ministry of Education. As a result of this administrative change the nurse training sub-component was cancelled, and funds re-allocated. The SAR earmarked almost US$9.3 million for procurement of contraceptive products. As a result of rapid changes in the domestic market for supply of contraceptives, and of the significant savings obtained in the procurement process, almost US$7 million were re-allocated to other project areas. Due to lack of appropriate legislation, as well as weak interest in health promotion and education among decision-makers in the early years of project life, the envisaged Health Promotion Fund was never established, and the loan funds earmarked for this fund were re-allocated. Important savings were obtained in other subcomponents as a result of the competitive procurement organized under various components of the project. Loan funds saved from the above cancelled activities and from cost savings in other components were used to support selected national health programs (TB and HIV/AIDS) or to increase the size of other project components (Emergency Medical Services and Health Management Information System): - The TB Control Program was designed by the MOH with the TA support from the WHO. The Program imp lementation was jointly financed jointly by the Government of Romania (GOR), WHO, World Bank and NGO (Soros Foundation for an Open Society). The Bank provided US$5.0 million for the procurement of medical equipment. - The HIV/AIDS program, designed by the MOH with the support of an international NGO, provided for the establishment of day care centers for children with HIV/AIDS. The Bank financed medical equipment and consumables for such centers in four selected cities. 3 - The comprehensively redesigned Emergency Medical Services aimed to establish integrated emergency health services in remote rural areas and link them with the district hospitals. The component provided US$25.5 million for the acquisition of 250 ambulances, telecommunication and medical equipment for ambulances and emergency rooms, and emergency diagnostic and treatment equipment for 96 Rural Health Centers and 34 District Hospitals. 3.5 Quality at Entrv: The Project did not undergo QER or a formal QAG assessment of quality at entry. An informal review of quality at entry, conducted in the context of a training exercise, questioned the complexity of project structure, and commented on the risk of insufficient absorption capacity for the technical assistance and training activities proposed under the project. 4. Achievement of Objective and Outputs 4. 0 Brieif Vote on the History of Project Implenientation The project was designed in 1990-1991 at the beginning of transition, when the country was facing severe economic and social crisis. The previous totalitarian communist regime left a serious legacy of poor population health and health system problems, including high maternal mortality and morbidity, a decline in life expectancy, a rise in infant mortality and relatively high prevalence of preventable or treatable infectious diseases. Drugs and consumables were in short supply and of poor quality. Much medical equipment were obsolete or idle for lack of spare parts. Personnel skills and institutional capacity were weak, through years of isolation from external development. Faced with this huge agenda of problems to be tackled, the task team adopted an ambitious, complex design (US$150 million with 9 components and 29 subcomponents). The planned project duration of four years was also very ambitious. Consequently, it took seven years to complete implementation. This was the first World Bank-financed project implemented by a Government agency (MOH) other than the Ministry of Finance (MOF), and the second project in the project portfolio of Romania after a ten-year break in the relationship between the Bank and the GOR. The MOH staff lacked the project management, implementation and coordination skills. As a result, project take-off was slow, with delays compounded by changes of Minister and other senior decision-makers in the first year of implementation, creating a need to reaffirmn commitment to project objectives and activities. Project performance at Mid-Term Review (MTR) was rated as unsatisfactory. Although progress in some areas were encouraging (e.g. imports for drugs and contraceptives through UNICEF and UNFPA, a countrywide network family planning offices, establishment of the Health Management School, upgrading of vaccine production facilities), the Bank's MTR report raised serious concerns regarding the performance in the first two years of implementation. The report proposed changes to project activities and cancellation of loan funds amounting to almost US$20 million. In order to bring the project back on track, the Bank increased resources for supervision, and engaged staff based in the Resident Mission to provide the Borrower with advice on day-to-day management of the project, at the suggestion of the Government. At the same time, the Government put pressure on the MOH to speed up the decision-making process and to improve its administrative capacity. The engagement of both technical staff and senior management of the MOH in the project gradually increased, creating solid ground for improvement of project performance. A Bank mission, conducted in 1995 as a follow up to the MTR, "re-launched" the project after recognizing the GOR's efforts both in health reforms and in project implementation. The first of four extensions to the project closing date was approved following this mission. 4 Further delays in project implementation were encountered in 1997 when the Bank was about to declare misprocurement in two large procurement packages (ambulances and management information systems), although finally the MOH managed to comply with the Bank procedures. The last two years of project implementation prior to its final closing date of June 30, 1999 saw a period of stability in the Project Coordination Unit (PCU), improved implementation capacity, as well as major steps in health reform. 4. 1 Ozitconze/achievenzent of objective: By project closing, achievement of development objectives is considered to be satisfactory in general, although some subcomponents provided only limited contribution to the achievement of objectives. (i) rehabilitate and upgrade the primary health care delivery system which was collapsing through lack of equipment, spare parts, drugs, medical supplies and transfer of knowledge to health providers This objective was achieved through the following: 1. Upgrading and equipping 419 rural dispensaries across all districts and providing adequate housing facilities to rural general practitioners (GPs). As a result, a wider range of services are provided in these dispensaries, and there is evidence of patient perception of higher quality and better staff attitudes in rehabilitated dispensaries compared to non-rehabilitated dispensaries. There is some evidence that provision of housing improved continuity of doctor availability - from once or twice weekly visits to full-time presence. (See beneficiary survey findings in Annex 8 of this report.) 2. Equipping 50 maternity units and the Mother and Child Protection Institute, and training the medical staff in these units. Related health indicators improved over the period of project implementation (maternal mortality dropped from 1.7 per 1000 live births in 1989 to 0.4 per 1000 in 1998; maternal deaths due to obstetric risk dropped from 263 maternal deaths in 1989 to 96 in 1998; infant mortality dropped from 26.9 per 1000 live births in 1990 to 18.6 per 1000 in 1998), although the contribution of this Project to the improvement cannot be quantified and these indicators remain too high, reflected poor living standards as well as health system performance. 3. Developing a network of 240 localfamily planning units, eleven reference centers and afamily planning and sex education unit in the MOH, combined with the provision of a wide range of contraceptives over the period ofproject implementation. These interventions, in the context of liberalization of legislation governing abortion and contraception, and complementary activities by NGOs, led to important changes in the behavior of the population, resulting in the reduction in abortion rate to one third of its 1990 level by 1998, and a 100 percent increase in usage of modem contraceptive methods doubled between 1993 and 1999. Some project investments have made a relatively limited contribution to these outcomes: the 240 family planning units account for a relatively modest share of utilization of family planning services, with commercial suppliers and hospitals being the two most frequently used service providers. Supply of subsidized contraceptives has not been sustained: while increasing numbers of women are using modern contraception and are willing to pay for this out of pocket, cost is commonly cited as a reason for not switching to a preferred method of contraception, and contraceptive prevalence remains lower among the poor, less educated, rural and Roma. (See Reproductive Health Survey Romania 1999, on project files.) 4. Upgrading ofvaccines production in the Cantacuzino Institute. This doubled or even tripled domestic production of the main vaccines used by the MOH in the mass immunization programs and reduced unit costs. The Institute also upgraded to meet Good Manufacturing Practice (GMP) standards to raise the quality of products. Research capacity was also significantly improved, 5 enabling the Institute to become a reliable, scientific partner and resume its traditional research relationship with the Pasteur Institute of France. 5. Upgrading of ambulance services and emergency medical care through procurement of 250 equipped ambulances, training of ambulance staff and equipping of 95 rural health centers and 34 district hospital emergency departments (out of 40 districts plus Bucharest). An estimated 60 percent or more of functioning, equipped emergency ambulances in Romania today were procured through the Project. Ambulances procured under the project are used more intensively than pre-existing stock of ambulances, and their use is concentrated on major emergency cases. Equipment in some rural health centers are not being used, where recruitment of suitable staff and staff training in these facilities have not been carried out. Data is not available to assess any health impact or impact on service delivery. (See findings of survey and stakeholder workshop in Annexes 8 and 9 of this report.) 6. Upgrading diagnostic equipment for TB, as part of support to the TB control program also supported by WHO and Soros. Although this equipment is in use, and is technically more effective and safe than the existing stock of x-ray equipment, the TB control program to date has not yet been able to reverse the rising trend in TB cases. 7. The very large procurement of drugs and consumables carried out under the Project was intended to contribute to this first development objective. This procurement provided temporary improvement in the affordability of essential drugs in 1993 when domestic production had fallen following transition and the sector was experiencing shortages of drugs. By 1995, commercial supply of drugs was plentiful. However, perverse financial incentives made it unattractive for pharmacies to use the low cost drugs procured through the project and distributed through public sector warehouses, rather than much more expensive private supplies of the same products. Ministerial direction was needed to force the sector to use drugs procured through the project, although it was too late to prevent expiry of some of the drugs procured. This procurement produced very little sustainable development impact. Some lasting benefits were achieved on a small scale as a by-product of the procurement: the procurement of syringes facilitated a change in MOH's policy to favor single-use syringes, and hence improved infection control. The procurement was also used to introduce a national immunization program for hepatitis B. Some technical assistance provided under this component has helped to motivate privatization of the pharmaceutical industry. (ii) support the first steps of a major restructuring of health sector financing and management to ensure a sustainable and cost-effective health care system in the medium term This broad objective was achieved via the specific objectives listed below: 1. Strengthen planning, coordination and management of health promotion. This objective was only partially achieved. A Center for Health Promotion and Education was established and became operational, and some of its staff received masters-level training abroad under the Project. However, the Center's effectiveness was hampered for a long time by its weak administrative and managerial capacity, and by limited political support and guidance from the MOH. Development of a coherent national framework for the health promotion activities in Romania was very slow. The position and work of the Center has improved in the last two years, and a national health promotion strategy has been produced by the Center since the Project closed, with the support of WHO. The Center's production capacity for educational materials is still inadequate. 2. Develop a national health strategy. An initial assessment of health sector problems and development of options and recommendations for reform was carried out by consultants from the United Kingdom Kings Fund College (report available in project files). The report did not result in immediate policy decisions but contributed to informned debate among stakeholders in the sector, which culminated in the adoption of a new Health Insurance Law in 1997. A Health Financing Study was carried out by the Health Insurance Commission of 6 ,Australia in 1997-1998. The results of this study were used as a basis for amendment of the Law, in particular to increase the level of redistribution of funds between districts to improve equity. The decentralization pilot, carried out with foreign technical assistance for design and evaluation, focused on the following objectives: ensuring that the population has efficient and universal access to primary medical care; improving the quality of medical services; offering patients free choice of a family physician; improving the status of the family physician; and introducing a physicians' payment system based on a mix of per capita (60%) and fee-for-service payments (40%). Piloting took place in eight districts and continued until 1997. In 1995 there was a formal evaluation of the experiment, performed both by internal (Institute of Hygiene, Public Health and Health Services Management) and by external (Institute of Health Sector Development, London, UK) evaluators. (See report by Sue Jenkins et al. "Evaluation of Decentralization of Primary Care in Eight Pilot Districts" 1995, in project files). A summary of the results of the evaluation is included in Annex 8. In brief, the pilot districts achieved improvements in doctor productivity, in doctor incomes, in attitudes of doctors towards patients, and in referrals; however, hospital admissions increased, rural-urban differences persisted, and it was difficult to encourage participation in continuing medical education or group practice. National implementation of the pilot reforms was delayed until legislation was passed and administrative changes in health sector financing were put in place. However, an improved version of the service delivery and payment arrangements for family doctors, based on the same principles developed and tested in the pilot districts was introduced nationally in 1999 as one of the first provider payment measures implemented under the new Health Insurance Law. 3. Develop a health information system This objective was not achieved, principally because outputs were not fully delivered. Limited improvement was made in accounting systems for the MOH and DHAs, and in some parts of clinical systems in the Neamt pilot district, but at very high cost. Romania did not adopt a national health information strategy or policy in response to the foreign technical assistance carried out under the Project, although improved data collection for WHO was implemented. 4. Develop capacity for health services management and policy. This objective has been achieved through the successful establishment of the National Institute for Health Services Management and Policy, training of a critical mass of teaching staff for the Institute and a cohort of masters-level graduates from the School. There was a smooth handover from the LUMNY consortium of three Universities (London School of Hygiene and Tropical Medicine - University of London, Universite de Montreal - Department de Gestion de Services de Sante and New York University - Wagner School of Public Administration Studies) which helped to establish the Institute and provide initial training to permanent local staff trained under the Project. The Institute has attracted sustainable financing from diversified sources, has continued to develop new training programs since the Project closed, and has produced a wide range of published research and policy papers. The Institute provides local technical assistance to a range of donor-financed and locally-financed programs in the health sector. 4.2 Outputs by components: I. REHABILITATE AND UPGRADE THE PRIMARY HEALTH CARE DELIVERY SYTEM a. Upgrade Rural Dispensaries Four hundred and nineteen rural dispensaries were selected (about 10 in each county) for physical rehabilitation and upgrading of equipment. The following activities were carried out in the selected dispensaries: physical rehabilitation, funded 95 per cent from government sources; provision of basic medical equipment, dental units and devices procured by international bidding, financed from the loan 7 (standard equipment list supplied, described in Government contribution to ICR); and provision of basic laboratory equipment, financed by EU/PHARE under a parallel project. The initial costs of this project component were overestimated. Original plans for equipment were extended to include provision of refrigerators for the cold chain for immunization products. The remaining funds were redirected to other project components. Implementation of this component was delayed initially due to lack of consistent support for this policy from some in the succession of Ministers of Health, with considerable time lost to redirect the Ministry to the original project objectives. Also, the MOH's approach to reform was perceived as over-centralized and non-transparent, and the MOH lacked information on the rehabilitation status of dispensaries and the number of medical staff to be trained. After delays in 1992-1996, outputs were fully achieved, made possible by a period of stability in the Ministry and better monitoring of implementation at district level by the PCU and the Department of Medical Assistance in the MOH B. Improve Reproductive Health The program was successfully completed, faster than other project components. This may be partly due to the fact that many other agencies were involved in the preparation and implementation of this sub- component. Technical and financial support was also receive from UNFPA, WHO, USAID and specialized NGOs, in a well coordinated program of assistance. A Family Planning and Sex Education Unit (FPSEU) was established in the MOH, with TA financed by UNFPA, and played a major role in the preparation of the MOH family planning strategy and in the evaluation and monitoring of the programs implemented in the area of reproductive health. It was also actively involved in the donors and NGOs coordination. Together with foreign TA hired under the Project, the FPSEU assessed national contraceptive requirements, prepared contraceptive marketing studies and distribution plans. Also with support from UNFPA, training modules in the areas of counseling, sexual education and management were prepared, and later on delivered to physicians interested in getting family planning competency. A fellowship program was successfully conducted for the FPSEU and staff of reference centers. A countrywide network of 230 local family planning units, and 1 1 reference centers placed in the university hospitals countrywide, was established. Physicians and nurses working in the above mentioned local units received adequate training provided by reference centers. A wide range of contraceptive devices (condoms, oral contraceptive cycles, IUDs), in sufficient quantities and at very affordable prices, were purchased and distributed through the network, hospitals and pharmacies. For selected categories of users, the contraceptives were distributed free of charge. UNFPA acted as specialized procurement agent for the MOH for contraceptives, which brought significant savings. Since health financing reform, however, some family planning units have struggled to attract financing specifically for these services and have ceased to provide service. Health Insurance House contracts with GPs working in these units do not specifically remunerate family planning services, in part because of an unresolved debate within the College of Physicians on whether GPs (rather than specialists) should provide these services. Similarly, an initiative by donors to establish a revolving fund to provide ongoing finance for free contraceptives has not succeeded. Contraceptives are not reimbursable under current Health Insurance House policies. A program was implemented in the Institute for Mother, Child and Adolescent Protection to make it a referral facility for high risk pregnancies and a training unit for obstetricians and midwives. Investments made for medical equipment, training and fellowships almost certainly had an impact on health indicators 8 in this area. Project funds were used to replace outdated medical equipment in 50 maternity/neonatal referral units nationwide. C. Train Health Practitioners The implementation of this subcomponent was rated as unsatisfactory. Plans to strengthen the 42 district nursing schools were cancelled and funds reallocated to other project components. Some US$42,000 were used for the purchase of teaching equipment for the Post-Basic Nursing School in Bucharest and US$217,000 for financing a training-for-trainers programs for nurses and fellowship programs for nurses and GPs. D. Procure & Distribute Drugs and Consumables In accordance to the Loan Agreement, UNICEF was contracted to procure on behalf of the MOH all drugs and consumables under the project, and did so in the amount of US$3 1.5 million for essential drugs for primary health care and for hospitals and policlinics. The UNICEF procurement brought significant savings to the Government which allowed larger quantities of drugs to be purchased and released large amounts of funds to other project activities. The drugs were distributed through 17 MOH regional drug warehouses which sold to provider units in the health sector and to pharmacies with regulated low mark- ups. The revenues from mark-ups were consolidated in a dedicated fund, and used by the MOH to repay part of the loan to the MOF. The MOF and regional drug warehouse staff were trained in international procurement of drugs, under the same contract with UNICEF. The subcomponent also financed the procurement of hepatitis B and polio vaccines (US$4.7 million), consumables and disposables (US$15,9 million), blood collection systems (US$1.3 million) and laboratory equipment (US$0.8 million). After a delay of about a year, this subcomponent began to deliver drugs and consumables to alleviate shortages which persisted. Implementation was difficult and delayed as a result of changes in trade legislation, the tax system and registration procedures on the Romanian side, as well as frequent turnover of country-assigned staff and complex bureaucratic procedures on the UNICEF side. By the latter part of 1994, however, the period of acute shortages ended so that the need for this emergency procurement diminished, and commercial incentives which were operating in the pharmaceuticals market undermined continued implementation. In particular, the low mark-up on very low-priced drugs procured under the project provided pharmacists with little incentive to buy and sell these products; rather, higher priced products available through commercial supply earned the pharmacist larger percentage mark-ups. This public procurement and distribution also attracted criticism from private sector competitors. As a result, there were reported cases when the UNICEF drugs remained on the shelves of the warehouses for excessive periods of time and some (a minority) of drugs passed their expiry dates. To overcome this problem, it was necessary for the Govemment to put pressure on provider units and warehouses to use drugs supplied through the project. This method of procurement and distribution of low-cost drugs thus did not appear sustainable without wider changes in pharmaceuticals policy. On the drug policy area, the MOH managed to develop a national drug formulary with some delay, but no notable progress was achieved on the drug pricing policy, in spite of the efforts made by the Bank over the entire project life to persuade the Government to reimburse the compensated drugs at the level of generic drugs prices. The National Institute for Drug Control and the National Authority for the Control of Biological Products of Hluman Use were strengthened by: (i) the endowment with complex laboratory equipment; and (ii) the training of staff. As a result, the number of tests performed increased significantly up to 50,000 per year in 1998, while and the average registration time was reduced to a maximum of 18 months. A study on pharmaceutical sector restructuring was carried out in 1993 by the PS Consulting Group, the main beneficiary being the Ministry of Industry (available in project files). The study pointed out that the 9 only possible source of financing for the rationalization and modernization of the pharmaceutical industry would come from the private sector, and it was a useful tool for the Government in the dialogue with the interested foreign investors. E. Improve Management of Emergencies Ninety five rural health centers (RHCs) were established, two to three per district, each serving a catchment of around 10 rural communes. Diagnosis and emergency medical treatment equipment were purchased for these RHCs and for emergency rooms in 34 district hospitals, as well as 250 ambulances with related medical and telecommunications equipment. The total amount of loan funds disbursed under this program was close to US$25.3 million, of which US$9.8 million was for ambulances, US$15.1 million for medical and laboratory equipment and US$0.5 million for telecommunications equipment. Local authorities provided financing for the physical rehabilitation of infrastructure facilities of these RHCs. The Swiss Government co-sponsored the program by providing grant financing for the procurement of medical equipment for ambulances, and for emergency departments of the hospitals from the seven university cities (Iasi, Mures, Cluj, Dolj, Constanta, Timisoara and Bucharest), as well as technical assistance for the development of medical emergency management software. Difficulties were encountered with implementation of RHCs policy. A number of RHC sites were not physically rehabilitated to an adequate standard; in less successful RIHCs, equipment were redistributed to other providers (typically hospitals which lacked equipment of this standard); some are not adequately staffed to operate the equipment supplied, and the service is not functioning as intended. An in-depth analysis of the implementation of the rural health centers program was carried out, and the findings of this evaluation were discussed within a stakeholder workshop in February, 2000. These findings are summarized in Annex 9 of this report. Implementation of the ambulance system upgrade was significantly hampered by procurement delays, partly due to frequent changes in the trade legislation in the country. An assessment of the utilization of ambulances procured through the project was unable to obtain useful performance data, although the reporting system has improved in the ambulance stations of the six medical university centers, which are part of an EMS system improvement program financed by the Swiss Government. It is clear from available data, however, that ambulances procured through the project are being used quite intensively and appropriately. However, the ambulance system as a whole continues to show unacceptably long response times (from around 20-60 minutes), with excessive numbers of cars and drivers, but insufficient numbers of trained medical staff and equipped vehicles. F. Health Promotion & Public Participation The implementation of this subcomponent was rated as unsatisfactory and most of the loan funds earmnarked for it were reallocated to other project components. For most of the life of the project, there was a lack of understanding and lack of political support for the development and delivery of health promotion activities, which undermined implementation. The Center for Health Promotion and Education was established and became operational, as planned, although its effectiveness was hampered for a long time by its weak administrative and managerial capacity, and by lack of guidance and support from the MOH. Development of a coherent national framework for the health promotion activities in Romania was very slow, and the Center's production capacity for educational materials is still inadequate. Approximately US$95,000 was spent for the production of heath promotion materials and a further US$70,000 for minor equipment for the Center and district health promotion laboratories. The proposed Health Promotion Fund was never established, again due to the lack of commitment of the Government and the MOH to public health activities and because of reluctance of the Government in 10 general to spend public money through NGOs. The US$874,000 earmarked for this fund were reallocated to other components. The program provided also for medical research laboratory equipment for the Endocrinology Institute in Bucharest (US$938,000) and provided some fellowships programs for the Institute's researchers. II. SUPPORT THE FIRST STEPS OF HEALTH SECTOR RESTRUCTURING G. Develop a National Health Strategy This sub-component provided for both the preparation of a series of reports with detailed reform proposals and assistance in implementing the first phase of reforms. As part of the envisaged reform activities, the initial design included support for pilot decentralization experiments in four districts. The design of this component, outlined in the SAR, proved to be over-optimistic regarding timeframe for reform and decision-making. Studies and pilots completed in the first year of the project were meant to be a basis for actual implementation and evaluation in the second year. In reality, it took more than two years to have the plans for decentralization completed and actual implementation of pilot projects started in 8 districts towards the end of the third year. The reform committee, which comprised of representatives of MOH, MOF and specialized institutes and was supposed to lead the reform process, was not effective. A draft strategy paper called "A Healthy Future" was prepared for the Ministry of Health from early 1992 to early 1993 by a team of foreign consultants (coordinated by King's Fund College, London and Nuffield Institute for Health Services, Leeds) and the National Institute for Health Services and Management. The main objectives of reform proposed by the report were to: (i) shift towards independent providers, both in primary and secondary care, and develop new payment mechanisms for these providers; (ii) establish a national health insurance fund, as an autonomous body outside the MoH in charge of raising funds through insurance premiums paid by employers and employees, and distribute them to contracted providers; and (iii) develop an accreditation system for all types of providers of health care. Although the MoH never formally reacted to this plan, some of the outlined solutions were included in the 1993 Program and White Paper of the Government and in many of the reform measures taken since. However, this Program did not get translated into action, and it was not until 1997 that key legislation for health sector reform was passed. Without adequate legislation, the health sector reform process was stalled in the mid-1990's, except for the decentralization pilot described below. Only in 1997 and 1998, with further support from the World Bank through a PHRD grant and EU PHARE funds, was a more open process of consultations regarding health sector reform put in place, which provided inputs in the development of a new health sector reform strategy approved by the MoH in 1999. This technical work and the new MOH strategy supported significant changes in the regulatory framework needed for reform (e.g. transformation of family doctors and ambulatory specialists into private operators under contract with health insurance funds; increased autonomy of hospitals in administrating their resources; transfer from the state budget to the health insurance funds of the main role of financing the health sector; and involvement of local authorities, employers and employees representatives in health sector governance). H. Pilot Decentralization Program The pilot project for decentralizing medical services and reforming primary health care finance was designed in 1993 with technical assistance coordinated with British Council and provided by British, Danish and Swedish experts. The actual pilot started in 8 districts in the autumn of 1994, was evaluated in 1995,and continued to operate until 1997. The delay in start-up was partly due to the need for legislation: the Government passed a decision (370/1994) that allowed, for a limited time, new service II delivery and funding arrangements to be tested in selected districts. The design and results of the pilot are summarized above and in Annex 8 of this report. Although in the approach implemented by Government from 1994 to early 1997, there was less scope for experimenting than initially proposed, it is worth noting that piloting of reforms is quite unusual in the Romanian context. Instead of attempting an overnight change for the entire country using rigid rules established in advance (which used to be the "local" approach to management of change), the Government agreed to test changes on a smaller scale (8 district out of 41) and to be open from the beginning to adjustments and improvements. This approach allowed difficulties to be handled at a smaller scale, instead ofjeopardizing many of the positive changes had they been introduced abruptly on a national scale. This component also supported a Health Sector Financing Study, carried out in 1998 by the Health Insurance Commission of Australia. The study was added to the original scope of this subcomponent to respond to perceived limitations in the technical preparation of health insurance legislation, a key element of the reform process pursued by the Romanian Government. The objectives of the study were to: (i) develop a Romanian Health Financing Model, which would aid decision making and act as a mechanism for the coordination of health policy analysis; and (ii) provide policy advice and conduct analysis on issues relating to the to implementation of the Health Insurance Law, especially during 1998 and 1999. Customized software for modeling Romanian health sector finance was developed and used for: analyzing revenues available to the health sector at national and district levels; for producing projections of health services demand and expenditure; and modeling options for resource allocation to districts and alternative several scenarios related to implementation of specific policy options regarding doctors' incomes, financing of public health activities and introduction of co-payments. One of the key findings and recommendations of the study related to the insufficient resources provided by the Insurance Law (7% of revenues) for redistribution between districts, undermining the solidarity principle embedded in the law. The latter finding was a major topic of the health policy debate in Romania in 1998 which consequently led to amendment of the Health Insurance Law in November 1998 to provide a much greater proportion of revenues (25%) to be redistributed. i. Develop a Health Information System This component was implemented in two distinct phases: (i) 1992-1995 - the preparatory phase; and (ii) 1995-1999 - the procurement of a turn-key Health Management Information System. In the first phase of project implementation, the MOH contracted consulting services from the American company, Public Administration Services (PAS), to assist the National Center for Health Statistics (NCHS) in designing the health indicators system (1992). As a result of the objections raised by the Court of Accounts to some of the payments due from the MOH to the PAS company, the technical assistance contract was suspended in 1995, before all the initially agreed activities were completed. However, the main objective of the contract was achieved, and the MOH approved a new set of health indicators in line with the international standards recommended by the WHO. The training program initially planned was successfully completed, as well as the early procurement of basic IT equipment for emergency needs. Based on the results of the activities conducted in the first phase of the program, the MOH, NCHS and the Health Management Institute prepared in 1995, in the second phase of the program, an overall health automation strategy. The Bank endorsed this strategy and agreed to finance the first steps of its implementation. As a result of savings and reallocations from other project components, funds for this component were substantially increased. This allowed the launching of the procurement of a large turn- key IT system, covering the hardware and software needs of the MOH, NCHS, the 42 District Public 12 Health Authorities, as well as piloting an integrated health management information system on a full scale basis in a pilot district (Neamt). Procurement of integrated HMIS was initiated in May 1996; however, the contract with the selected supplier was signed only in February 1998, due to procurement delays and a further change of MOH leadership. The tight implementation schedule put significant pressure on both contract parties, leading to an excessive focus on the hardware and software procurement, at the expense of user participation, and management and training aspects of implementation. This had a negative impact on the overall functional performance of the newly procured system. An in-depth evaluation of the status of HMIS was carried out as part of the ICR Mission, based on interviews with beneficiaries in the MOH, NCHS and the pilot district. (The report of the consultant, Dr Roddy Neame is available in project files; stakeholder workshop discussion about HMIS, facilitated by Dr Neame, is contained in Annex 9 of this report.) Even one year after the project closing date, only the simpler aspects of the project have been completed: installation of hardware and software in the MOH, NCHS and District Health Authorities, which utilize two software systems (SAP and Comshare) on a stand-alone basis, and have electronic mail and internet access. In the Neamt pilot district, all contracted hardware were installed, but in an inappropriate configuration (equipment allocated to the wrong places, inability to network between various health organizations in the district). Software has mostly been installed, but with some critical gaps. The capability to exchange data between organizations or between medical and management information systems to achieve an integrated system has not been achieved. As a result, the HMIS is not being used for management purposes, with the exception of limited use of some accounting and management functions in SAP, and a successful test of the system in one small hospital (Bicaz hospital). There is user dissatisfaction with ambulance stations software, which is no longer used. Only the integrated blood transfusion system may be considered successfully implemented. A training center was established but is non-functional - due to lack of a server, reallocated to another location. These problems should have been foreseen and could be resolved by simple executive decisions by either prime contractor or the client. J. School of Health Services & Management The project supported long-term training overseas for the core staff of the National Institute for Health Services and Management (NIHSM): 12 young professionals graduated in health services and public health MA and MSc programs in well established Universities in the United Kingdom and Spain. Another 181 professionals, working in health services administration, participated during the life of the project in short term courses outside Romania. The project also provided support for renovating and equipping a building to host NIHSM, providing adequate facilities for training, research and technical assistance activities. A TA contract to support development of the School was signed in late 1993 with LUMNY, a consortium of Universities (London School of Hygiene and Tropical Medicine - University of London, Universite de Montreal - Department de Gestion de Services de Sante and New York University - Wagner School of Public Administration Studies). As part of this contract: (i) the curricula for two training programs in health services management and policy were developed (one master's type program of advanced training and one continuous education program); (ii) Romanian staff selected to take over responsibility for teaching undertook a one-term visiting assignment in the three partner institutions; and (iii) delivery of advanced training and continuous education in Romania. From 1994 until completion of contracted activities in 1996, two cohorts of trainees, 62 persons in total, graduated from the advanced training program (covering 16 subjects in a total of 24 weeks of full-time teaching), and three cohorts of trainees, 98 persons in total, graduated from the continuous education program (covering 11 subjects in a total of 13 12 weeks of full-time teaching). Towards the end of the contract, the input of Romanian trainers in delivery of the courses increased and after completion of the TA provided by LUMNY, training continued to be provided, with similar format and content, by Romanian staff. Between 1997 and RHRP closure in 1999, another three cohorts, of more than 1 10 persons in total, graduated from the continuous education program. The program currently continues to be offered. In the autumn of 1998, an advanced training program in Management of Health and Social Services was restarted by the Institute of Health Services Management, in collaboration with the University of Bucharest, leading to a Master's degree accreditation by the Ministry of Education. At present, IHSM continues to deliver advanced training and continuous education programs, as well as other short term courses in management, health policy and health promotion. It also provides technical assistance to the MoH, local health authorities and health insurance houses. IHSM has been also involved in implementing several studies and development projects jointly with EU-PHARE, USAID, US- Department of Health and Human Services, UNICEF, UNFPA, the Soros Foundation and training and research institutions in Hungary, Poland, the Czech Republic, Moldova, United States and the United Kingdom. K. The TB Control Program. The TB control program was designed by the Romanian specialists, following WHO recommendations and taking into account the local needs and culture, assisted by experts from the WHO, WB and NGO community. The program aimed to: (i) strengthen the institutional capacity of monitoring the spread status of the disease, and prepare and implement specific prevention and curative programs; and (ii) improve the investigation and treatment capacity of the specialty medical units countrywide. The second objective was achieved through the financing of the medical equipment under the WB-financed project. Implementation was collaborative. MOH provided for overall program coordination, maintaining the program as a priority, and providing logistical support at national and local levels. WHO provided technical and financial support for project monitoring and supervision of printing materials in the pilot district. The Health Rehabilitation Project financed the medical equipment. NGOs financed training of medical staff and support staff from the pilot district. Although some specific health indicators have improved in the last two years, it is too early assess the program impact on the overall health status of the population. 4.3 Net Presentt Value/Economic rate oJ return: Economic rate of return could only be estimated for one subcomponent of the project - investment in improving vaccines production at the Cantacuzino Institute in Bucharest. In addition to a range of non- quantifiable benefits, the investment is estimated to have yielded an internal rate of return of 7 percent (See details of the evaluation in Annex 3). Some non-quantified observations can be made on the benefits relative to costs of investment in a number of other project components. The largest component of the project, the procurement of drugs and consumables amounting to US$69.8 million (excluding investment in the Cantacuzino Institute, or 31% of project costs) has generated very little sustainable benefit. The US$24.1 million investment in health management information systems, amounting to 11 percent of project costs), has not been implemented successfully and is producing negligible benefits. The very large investment in new ambulances and equipment for the ambulance system and emergency rooms, amounting to US$58 million (26% of project costs), is not wasted expenditure, and may well be producing some improvement in survival of 14 emergency cases. However, this component was implemented in a way which did not achieve any organizational change in the highly inefficient ambulance services (very large fleets of unequipped vehicles and transport vehicles have not been reduced) and does not appear to improve the response time of ambulances to attend to emergencies to acceptable levels. It is not clear why this large investment was not used to achieve efficiency gains and better outcomes. It appears that the majority of the development benefits achieved by the Project were achieved in the remaining compcnents, which together account for less than 30 percent of project expenditures. 4.4 Financial rawe of return: N/A 4.5 lsistitiutionial development impact: The Project has successfully created a critical mass of Romanians trained in health policy and health management, and a successful Institute of Health Services Management which is playing a role in training, research and technical assistance to a range of health sector development projects. A Center for Health Promotion and Education has also been established, and although it has faced significant obstacles during the life of the Project, it appears to have continued to consolidate and expand its role since project close, assisted by other agencies such as WHO. The Project has had less impact on the institutional capacity of the MOH and Health Insurance House, for a range of reasons. EU/Phare projects in the past two years have been directed at capacity building in these organizations, but it is acknowledged that weaknesses in capacity remain at central level, while peripheral organizations in the health system face increased responsibilities under recent decentralizing health reforms, with improvement in their human resource base or management information systems. Nonetheless, by project close, the PCU had developed reasonable familiarity with Bank requirements and had completed an intensive program of procurements relatively successfully in the last two years of the Project. The PCU, provided that it remains stable, therefore provides a stronger starting point for implementation of a second health project. The Project undertook relatively modest investments in the training and development of health service providers, and in organizational development to support health reform and modernization of service delivery; this, in turn, limited the development impact of some of the investments in infrastructure and equipment (such as investment in rural health centers and in the ambulance system). 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implenmenting agency: External factors contributed to some of the factors listed in Section 5.2 below. The Project became effective early in the transition period, with the result that early post-transition elections changed the leacdership of the MOH, and required renewal of dialogue between the Bank and GOR regarding the project objectives and activities. The economic difficulties faced by Romania in the early years of transition contributed to the initial difficulties with availability of counterpart funds. Similarly, the nature of the transition meant that aspects of the general legislative and administrative framework (such as procurement law, trade law, tax law) affecting the Project were in need of review, which understandably took some time to carry out, and then required adjustments in project implementation. 15 5.2 Faclors generally sulbjeci to government control: The Court of Accounts (CoA). According to the Romanian legislation, spending of the public funds is audited by the Court of Accounts, which is accountable to the Parliament. This audit was exercised both ex-ante and ex-post by the CoA representatives in the MOH during the entire period of project implementation. The detailed control performed by the CoA, combined with restrictive and conflicting interpretations of both Romanian legislation and Bank procedures, slowed down procurement significantly and increased supervision costs. The role and approach the CoA affected the implementation of the entire Bank portfolio in Romania, and it was a constant subject for discussion between the Bank's management and the Romanian Authorities over the past five years. Recently the Parliament approved, at the Government's initiative, an amendment to the Law of the CoA, removing its ex-ante audit prerogatives, and transferring them to the Ministry of Finance, thus bringing the relevant Romanian legal framework closer to the international norms. High turnover of Ministers and senior managers. In the seven years of the project, the MOH faced a high turnover of senior decision makers, having ten Ministers and twice as many deputy Ministers. This unusual situation led to delays in decision-making while new managers familiarized themselves with the issues, and at times resulted in reversals of earlier decisions, which had obviously a negative impact on the project implementation. Lack of timely availability of counterpart funds. The initial estimated of GOR contribution to project financing was US$ 57.5 million. Though, in the end the Government provided more than this (US$75.57 million) as a result of the changes in the tax legislation, a longer implementation period (7 years instead of 5 initially planned) and increased civil works, there were periods in the earlier years of project implementation where counterpart funds were not available. This delayed the rehabilitation of rural dispensaries, in particular, and hence the timetable for equipping these dispensaries. 5.3 Factors generally subject to implementing agencv control: Slow and centralized decision making processes in the MOH and health system delayed critical decisions related to the Project and increased supervision costs. The MOH had a highly centralized approach to decision-making, and was reluctant to delegate responsibility internally, or to decentralize decision-making and day-to-day project implementation to subsidiary health sector organizations. The centralized approach also placed at risk the effectiveness of the project in meeting the needs of a wide range of health care providers in districts (rural dispensaries, health centers, district hospitals, etc) in a flexible way based on differences in local needs. High turnover of the staff in the MOH compromised the ability to build up a critical mass of experienced staff in the PCU, and also undermined the participation of line-staff in some areas of project implementation. This turnover was in part due to low salaries and inadequate working environment, but also due to replacement of staff following changes of Ministers and senior management. The MOH and MOF were generally reluctant to use consultants or contract out project administration to enable payment of market-based remuneration. Lack of trained staff in Bank specific procedures: procurement, disbursement, and project management. The problem of turnover was compounded by inability to recruit staff with appropriate training. Both factors also created difficulty in ensuring that the PCU staff at any time were trained in specific Bank procedures. The period of stability in staffing in the last two years of the Project finally allowed this problem to be overcome to a significant extent. 16 5.4 Costs andjifinacinig: Actual total project costs of US$ 224.53 million (excluding parallel finance of US I7.6 million) exceeded estimates at appraisal of US$207.5 million by 8.2%. The increase in expenditure was principally in two areas: (i) civil works, financed from counterpart funds, due to higher than anticipated rehabilitation requirements in rural dispensaries; and (ii) in the area of equipment, financed from donor sources not anticipated at appraisal. In addition, actual spending on training and technical assistance was lower than estimated at appraisal and funds from these categories were reallocated to additional equipment procurement. Sumyr osts bv Financier At Appraisal Actual at Project Close US$ millions US$ millions GOR 57.32 75.57 Loan 150 149.03 Donors (parallel finance) 0 17.60 TOTAL 207.5 242.20 Summar o Total Costs b Category of Expenditure At Appraisal Actual at Project Close US$ millions US$ millions Works 6.7 15.4 Goods 185.3 201.4 Services 11.6 6.4 Recurrent costs 3.9 1.4 By comparison with estimates at appraisal, there was a six to seven fold increase in spending on emergency medical services and health management information systems; a significant reduction in expenditure on training, health promotion and national health strategy; and expenditure on new components of TB control and HIV/AIDs. For details see cost and financing tables in Annex 2. 6. Sustainability 6.1 Rationale for sustainiability rating: The Project is rated as likely to be sustainable in aggregate, although some subcomponents do not appear to be sustainable. The following general factors are critical for sustainability of project benefits: (i) Stable or growing public health revenue; (ii) Stability in health sector strategy and sector management; (iii) Government's willingness to infuse health system with increased incentives for efficiency, quality and affordable access; and (iv) Government and population's willingness and ability to allocated adequate health expenditure to maintain existing investments and to increase expenditure in the areas of greatest need: primary care, emergency services and essential secondary care and public health. 17 Public health revenues have increased in 1999, following increases in contribution rates and improvement in collections. The Health Insurance Houses have implemented changes to financing for primary health care which protect allocations to this sub-sector and improve incentives for efficiency and for high priority services. The GOR budget for 2000 has significantly increased the MOH allocation for public health programs. While it is not possible to avoid political changes and changes in management, the decentralization of health sector financing and management which took place during the life of the Project, together with the participatory approach to development of a health strategy in recent years, helps to increase the prospects of stable commitment to the policies and programs supported by the Project. Some areas in the Project are expected to be the target of follow-up development in the second health project (rural primary health care, emergency medical services, priority public health programs) which should also help to improve the prospects for sustaining and building on the impact of the first project. On the other hand, the results of the beneficiary survey and stakeholder workshops point to evidence that some rural dispensaries and rural health centers are deteriorating and equipment is not used or not maintained as at project close. In addition, although the improvement in use of contraception and in maternal and child health indicators clearly appears to be sustainable, some of the family planning units established under the project (in particular, those staffed by GPs who have privatized recently) are no longer providing services due to lack of financial incentives and lack of ongoing supply of subsidized contraceptives. There is also evidence that some rural health centers are not adequately staffed and functioning, and these centers in general face difficulty in obtaining secure financing. The evaluation of the HMIS investment under the project indicate very poor prospects for sustainability or even complete implementation of this investment. This finding of variable sustainability points to the need for more systematic monitoring and follow-up of each participating provider organization. Sustainabilitv ratings by component: (HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, NA+ Not Applicable) A. Rural dispensaries L B. Reproductive health L C. Training (component cancelled) NA D. Drugs and consumables UN E. Management of Emergencies L F. Health promotion UN G. Health reform strategy L H. Health information system UN I. Health management school L J. TB and HIV/AIDS L 6.2 Transition arrrangement to regular operations: In most cases, the institutions benefiting from Project investment have successfully taken up responsibility for ongoing operation of programs, institutions and operation and maintenance of equipment. The environment of more stable revenue and greater autonomy that is now in place, generally appears adequate to ensure smooth transition to regular operations. As part of the ICR Mission, specific recommendations were agreed with the MOH about transition arrangements to achieve completion of implementation and ongoing operation of the HMIS piloted in Neamt district. Some further steps have been taken on the basis of these recommendations, but further monitoring and follow-up is required by the MOH. 18 In the case of project investments in dispensaries, health centers, ambulance centers and district hospitals, some of the activities of the second health project will be used as a basis for follow-up and monitoring. Health service plans to be developed in each district will review the status and functioning of services and the need for follow-up action in the areas of staff development, maintenance of equipment and requirements for further investment. An integrated emergency medical services strategy, to be rolled out nationwide under the second project, will ensure optimal use of existing ambulances and other EMS equipment, rational priorities for new investment, and clear responsibility for maintenance and other operating costs. 7. Bank and Borrower Performance Bank 7.1 Lending: The Bank correctly highlighted sector policy issues, and sought to strike a balance between longer term investments in capacity development and policy-related activities, more immediate improvements in service delivery and health outcomes, and emergency relief. With hindsight, the relatively large share of loan proceeds (55 percent) allocated to procurement of drugs and consumables for short-term emergency relief, without an objective of sustainable development impact, appears questionable. Perhaps reflecting the emergency nature of the situation, the SAR did not include a logframe with clear conceptual distinction between development objectives and outputs. This lack of clarity in the framework of the SAR has made it difficult to adopt a clear delineation between discussion of outcomes and outputs in this report. Monitoring indicators were not clearly specified for all components at the outset. As a result, for a number of project components, monitoring data is not available to permit assessment of impact. Moreover, the Bank underestimated the complexity of the Project as a whole and the constraints on implementation capacity. 7.2 Supervision: The Bank fielded supervision teams that provided an appropriate range of skills to advise on content and implementation of the Project. Substantial turnover in membership of the Bank team in the early years of implementation, however, added to the difficulties arising from turnover on the counterpart side. Instituting supervision from the Bucharest Bank Office appears to have been a successful strategy for achieving a flexible and timely approach to supervision. Supervision appears to have been carried out flexibly and took constructive advantage of opportunities to restructure the project to address high priority health problems (TB and HIV/AIDS), and to collaborate with other donors to increase the total project resources substantially and increase impact in a number of project components. 7.3 Overall Bank performance: Satisfactory. Borrower 7.4 Preparation: A capable and motivated counterpart team, with credible public health and clinical credentials, participated in project preparation in the immediate post-transition period, with evident commitment to project content. Unavoidably, given the post-transition environment, the counterpart team lacked implementation and management experience. However, the MOH team involved in preparation was replaced following the 1992 election, creating a necessity for a new team of counterparts to form a view on the objectives and activities of the Project and determine their level of commitment. 7.5 Government implementation peiformance: Turnover in Ministerial and senior managerial positions, shifting and conflicting approaches to the health system, inability to develop and adopt a stable health strategy, and lack of coordination among different players within Government created major problems for Project implementation. Donor coordination was also weak in the health sector. The final years of the project have seen an increase in stability, and greater alignment around major elements of a government health strategy. 19 7.6 If plementing Agency: Initial problems in the PCU were due to lack of experience with Bank projects or project management in general, lack of training in Bank-specific procedures, staff turnover, and difficulty in attracting suitably qualified staff under prevailing low public sector salaries. In the latter years of the project, the PCU succeeded in increasing the efficiency of procurement and the pace of disbursement. 7. 7 Overall Borrouwer performance: Unsatisfactory at mid-term review; satisfactory by project close. 8. Lessons Learned The Health Services Rehabilitation Project bears out some of the general lessons learned in ECA region regarding the design and implementation of health projects in countries undergoing political, economic and social transition. The lessons include the following: (a) under the best of circumstances, health sector reform remains a long and politicized process; (b) the most technically sound strategies cannot be implemented in the absence of strong, capable institutions and political will; and (c) the impact of the political economy and the sensitivity surrounding even seemingly simple health sector reforms are significant and must be directly addressed from the outset through comprehensive public information and communication strategies targeted at the full range of stakeholders. The experience of this Project shows that, despite the best efforts of Bank staff and counterparts, expectations were too high, the project was too complex, the initial project structure was too rigid and required repeated modification, and the implementation timeframes were not realistic in a constantly changing political and managerial environment. The Project illustrates the need for flexibility in design and during implementation. This could have been achieved through clearer and more detailed specification of project goals (since goals of improved health status, sustainable financing and efficiency remain constant), but less detailed specification ex ante of project inputs and the mode of implementation. Some of the problems have been ameliorated by recent changes in legislation. It must be noted that all of the foregoing were exacerbated by the very complex project structure. In addition, the Project highlights the need for continued investment in developing local capacity for sector policy, planning and management. Specific lessons learned under major components which were subject to intensive learning methods are listed below: Upgrading Rural Dispensaries. Using a uniform, top-down approach for deciding and implementing physical rehabilitation and-equipment upgrade of primary care facilities results in some inefficient interventions. Investments in physical infrastructure are at risk of under-utilization or even degradation, without an adequate commitment and participation from the staff operating the rehabilitated facilities. Some prior test of commitment by the staff/community should be used before a final decision is made about location of an investment (e.g. participation in training, co-financing of works). Complementary measures (envisaged in the original project design) such as changes in the incentives system or review of delivery arrangements and roles are needed to ensure the efficiency and sustainability of physical investments. Range of equipment and even decision whether to invest in certain facilities has to be based on stable and shared strategies which are developed and agreed for more than just project requirements. Not only legitimate health service needs should be taken into account, but also ongoing and likely changes in organization of delivery and funding arrangements. Otherwise, the risk exists to equip facilities which will lose their staff (e.g. dental care has shifted early to private delivery with large share of out of pocket 20 payment; therefore, many dentists have given up employment in rural dispensaries and virtually none were willing to fill positions in rural areas, in spite of better equipment becoming available). Management from a central unit of implementation of a component spread over all 41 districts of Romania has proven difficult, even if staff of District Health Authorities have been involved. Focusing on a manageable number of districts within any given implementation period is desirable. Health Reform Strategy. Ownership of reform initiatives (real or perceived) has to be shared between major stakeholders, otherwise competition between institutions or leaders delays implementation of measures accepted from the technical point of view. An appropriate balance has to be achieved with respect to implementation responsibilities and benefits between organizations with formal authority (usually branches of Government) and those with expert authority (technical or knowledge bodies). Although it was more rewarding and effective for providers of TA to work closely with the latter type of institutions, insufficient effort to have the former fully on board has resulted in decreased impact of technical recommendations on the decision-making process. Most of the long term impact on health policy development has been achieved through investments in training of people. Even if there are difficulties regarding short term recognition and efficient use of staff with new or improved health policy and management skills, in the long term, graduates of training programs, both in-country and external, have opportunities of shaping the reform process. Pilot approaches for testing new arrangements of service delivery and finance have been very difficult to accept in the Romanian policy environment, but have provided a very good base for further steps of reform. Running pilots would have been less challenged if those in charge of implementation would have given more attention to communication and evaluation aspects. Health sector reform involves a political and communication exercise at least of the same importance with developing and using technically sound proposals. Consultants' focus on deliverables, defined mainly as reports, combined with low capacity of senior decision-makers to make efficient use of technical assistance, reduced or delayed the impact of sound pieces of technical work. The situation was somewhat better in the cases where more extensive consultations with stakeholders were used in the process of TA. The more productive and successful experiences of technical assistance had at least a core team of experts based for longer time (at least 2-3 months) in Romania, using additional short-time experts for well identified tasks. Such teams were able to establish stronger relations with relevant counterparts, allowing better understanding of institutions and processes and access to substantive information, and better transfer of knowledge and skills. HMIS. Projects in this field may be better led by health managers or experts with a broad system perspective, rather than by IS/IT specialists. Projects of this nature need to involve the users of information in the design and implementation phases. Consultation with and participation of various system stakeholders is essential, in order to end up with a system which covers health sector automation information needs in a way which is coordinated, but decentralized. In the absence of these variables, HMIS developments will only cover the needs of a small part of the health sector and may be used only by a few IS/IT specialists and enthusiasts. The lack of sufficient and adequately trained staff, combined with divided responsibility at national level, and a lack of decision making authority at the local level, prevented the District Public Health Authority of the pilot district from functioning as a reliable counterparts to the contractors. Projects of such complexity should be designed and financed at the beginning of the project life, in order to give enough time for the proper completion, and follow-up monitoring, and to avoid excessive focus on rapid procurement of hardware and software. by the donors themselves. 21 9. Partner Comments (a) Borrower/ilmplementing agency: The GOR's contribution to this ICR is attached as Annex 10 to this report. The Borrower received the ICR and commented that in general the report reflects the implementation process and achievements of the Project. The Minister of Health also conveyed technical comments prepared by the PCU, which have been reflected in amendments to the final text of the ICR. The PCU's own final report on the HSRP was presented to the Romanian Cabinet and represents the official view of the Ministry of Health on the Project. The PCU noted that the Bank's assessment has many similarities with its own, and also adds the following comments (summarized below, the full correspondence dated June 22, 2000 is available in project files): Bank and Borrower Performance: Unsatisfactory performance at MTR was in part due to the Bank's inflexibility reallocation of funds between components in the early years of the project, in spite of the fact that initial allocations proved to differ significantly from real necessities. In the later years of the Project the Bank is perceived as being more flexible and open to new proposals. During the initial years of the project the PCU lacked skilled staff but used external technical assistance in procurement. In spite of this, procurement delays occurred. In the last two years of the Project, trained PCU staff were in place and managed procurement processes as required by the Bank without technical assistance. No complaints were received regarding these procurements. (b) Cofinanciers: N/A (c) Other partners (NGOs/private sector): Private sector participants in health management information systems, and NGO participants in emergency medical services, health services planning, policy and training fields participated in stakeholder workshops. Their comments are included in the summary of the workshops given in Annex 9. 22 Annex 1. Key Performance Indicators/Log Frame Matrix Outcome /Impe Indicators:_ _ Indicator/Matrix 1990 Level or Project Target Level 1998 or Latest Estimate Upgrade Dispensaries Yearly number of visits in upgraded Not available 3,037,656 (1998) dispensaries Yearly number of laboratory Nil 120,493 (1998) examinations Yearly number of EKGs Nil 42,739 (1998) Yearly number of dental treatments Nil 348,266 (1998) Improve reproductive health services Yearly number of abortions 992,265 259,888 Abortions rate 3152.6 per 1000 live births 1107.8 per 1000 live births Maternal mortality related to abortions 0.57 per 1000 live births 0.19 per 1000 live births Infant mortality 26.9 per 1000 live births 18.6 per 1000 live births Maternal mortality 1.7 per 1000 live births 0.42 per 1000 live births Prevalence of modern contraception 14% (1993) 30% (1999) Prevalence of contraception (total) 57% (1993) 64% (1999) (married women aged 15-49) Vaccination Measles new cases 10,211 101 in 1999, Q1 (exhibits 4 yearly cycle) (annual ave 1990-1994) 16 in 2000, QI Institute for Drug Control Average drug registration time 18 months (maximum) Health Reform Strategy Strategy adopted King's Fund report recommendations included in the Gov. Strategy Output :ndicatDrs: IndicatordMatrix 1990 Level or Project Target Level 1998 or Latest Estimate Upgrade Dispensaries Number of Physicians and dentists Not available 984 practicing in upgraded dispensaries Number of nurses working in upgraded Not available 1740 dispensaries 23 lndtitrlMatrix _ 41990 Level or project Targei Leer La Et Improve reproductive health Prepare contraceptive marketing Completed services plan Assess next year's contraceptive Completed annually requirements Vaccines Produced at Cantacuzino Institute Measles 400,000 doses 600,000 doses (I M capacity) Influenza 250,000 doses 500,000 doses (800,000 capacity) Institute for Drug Control 12,000 (1992) 50,000 Number of tests performed Strategy To Develop Implementation of a restructuring Study conducted and submitted to Pharmaceuticals Industry plan for pharmaceutical industry the Ministry of Industry, for use in the privatization process for the pharma. Industry Health Information System Prepare regular sets of health Health indicators in line with indicators WHO requirements prepared since 1995 Institute of Health Services 14 Romanian experts have been Management - Training for Trainers trained as trainers, by the LUNMY Consortium Number of graduates from Mgmt Master Progr. 62 graduates and Prog. other 33 currently in training Number and type of trainees certified 208 graduates of short-course in in-service courses management training; further short courses ongoing. Project Management - Strengthen 5 MOH staff trained in Procurement Services international procurement in Turin 5 MOH staff trained in procurement of drugs in UNICEF, Copenhagen 24 Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) __ _ _ Appraisal Actual/Latest Percentage of Estimate Estimate Appraisal Proiect Cost By Component US$ million US$ million Upgrade Rural Dispensaries 25.40 22.20 87.4 Improve Reproductive Health 35.60 24.10 67.7 T'rain Health Practitioners 1.40 0.40 28.6 Procure & Distribute Drugs and Consumables 93.20 77.15 82.8 Improve Manag:ement of Emergencies 8.90 58.40 656 Health Piromotion & People's Participation 4.80 2.80 20.8 Develop a National Health Strategy 3.10 1.20 37.5 Develop a Health Information System 3.20 24.10 753 School of Health Services & Management 3.10 2.30 74.2 National Health Programs (TB Control + 7.10 HlV/AIDS).._ Miscellaneous 4.80 Total Baseline Cost 178.70 224.55 Physical Contingencies 17.80 Price Contingencies 11.00 Total Project Costs (excluding 207.50 224.55 parallel finance from donors) Total Financing Required 207.50 224.55 25 Project Costs by Procurement Arrangements (Appraisal Estimate) US$ million equivalent) ________ __________ t0000 i00 0 Procurement Method' E~.ndItur. ategory 1G NCB Ote2 N.B.F. Total Cost 1. Works 0.00 0.00 0.00 6.70 6.70 (0.00) (0.00) (0.00) (0.00) (0.00) 2. Goods 122.40 0.00 61.80 1.10 185.30 (94.10) (0.00) (46.00) (0.00) (140.10) 3. Services 0.00 0.00 11.60 0.00 11.60 (0.00) (0.00) (9.90) (0.00) (9.90) 4. lncr. Recurrent 0.00 0.00 3.90 0.00 3.90 (0.00) (0.00) (0.00) (0.00) (0.00) 5. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 122.40 0.00 77.30 7.80 207.50 (94.10) (0.00) (55.90) (0.00) (150.00) Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) ____________________ Procur m ent M ethod'_ _ _ _ _ _ lOB z1 [(NC Oh2r N.BF. Total Cost 1. Works 0.00 0.00 15.50 0.00 15.03 (0.00) (0.00) (0.00) (0.00) (0.00) 2. Goods 95.00 0.00 106.50 0.00 201.50 (72.37) (0.00) (70.98) (0.00) (143.35) 3. Services 0.00 0.00 6.50 0.00 6.50 (0.00) (0.00) (5.76) (0.00) (5.76) 4. Incr. Recurrent 0.00 0.00 1.50 0.00 1.50 (0.00) (0.00) (0.00) (0.00) (0.00) 5. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 95.00 0.00 130.00 0.00 224.53 (72.37) (0.00) (77.63) (0.00) (149.03) l' Figures in parenthesis are the amounts financed by the Bank Loan. All costs include contingencies. 2/ Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. Procurement of drugs, consumable, contraceptives, equipment conducted through UNICEF (US$ 53. Imillion) and UNFPA (US$1.9 million) 26 Project Financing by Component (in US$ million equivalent) Project Financing by Component (in US$ million equivalent) Component Appraisal Estimate ActuallLatest Estimate Percentage of Bank Govt. CoFJI Bank Govt CoFJI Bank Govt Rural Dispensaries 16.70 12.74 3.45 18.75 2.20 20.7 147.2 Reproductive Health 32.15 9.48 20.94 3.16 1.10 65.1 33.3 Training 0.57 0.45 0.26 0.14 45.6 31.1 Drugs & Consumables 83.21 24.66 66.18 10.89 80.4 44.2 Mgmt of Emergencies 3.63 6.83 25.34 33.06 11.00 698.1 484.0 Health Promotion 4.53 1.01 1.25 1.56 27.4 154.5 Health Reform Strategy 2.82 0.78 1.11 0.01 39.0 1.3 Health Mgmt Information 2.82 0.89 18.48 5.62 0.30 655.3 631.5 System Health Mgmt School 3.10 0.48 2.14 0.16 69.0 33.3 TB Control + HIV/AIDS 5.93 1.14 3.00 0.0 0.0 Miscellaneous 3.21 0.0 0.0 PCU 0.74 1.08 Total 150.00 57.32 149.03 75.57 17.60 99.7 132 1. Note: there were no cofinanciers: parallel finance from donors is listed under this heading. 27 Annex 3: Economic Costs and Benefits Sub-Component: Upgrade of Vaccines Production at Cantacuzino Institute The Project provided financing for the replacement of equipment and spares to ensure continued production of vaccines at the only domestic producer of vaccines, the Cantacuzino Institute in Bucharest. In aggregate, the Project disbursed US$ 8,014,136 for equipment under this component. Almost all the equipment purchased is installed, currently functioning, and being operated by properly trained staff. Year Procurement Goods procured Amount. 1992 ICB Equipment for manufacturing biological products 3,967,497.50 for human use 1993 ICB Equipment to ensure GMP standards in vaccine 806,556.36 production and quality control 1994 International Laboratory equipment (automated dispenser, 148,228.80 Shopping spark tester, steam sterilizing unit) 1995 ICB Equipment to ensure GMP standards in vaccine 1,128,919.62 .production 1996 International Equipment to develop the Cantacuzino Institute's 123,801.53 Shopping cold chain for storage of vaccines and other therapeutic and diagnostic products 1996 International Antistatic, fireproof, chemical resistant treatment 247,843.00 Shopping x 2 of walls, floors and ceilings of vaccine laboratories; sterile air circulation equipment 1997 ICB Animal breeding unit 1,591,290.00 TOAL AM=NT 8o14436.81 USO The utilization of the newly purchased equipment allowed a significant improvement in the financial performance of the institute: increased production and productivity, lower prices and improved quality which met international standards, enabling wider marketing of the Institute's produces. The Institute's turnover increased from US$3.88 million in 1992, before the program start-up, to US$5.29 million in 1999, when the Project was closed. The vaccines production increased significantly from 1998 allowing the MOH to conduct large scale immunization programs with reducing need to import vaccines. The investment is estimated to generate an internal rate of return of 7 percent on monetized benefits. The most important non-quantified benefits are presented below: Measles Vaccine * production capacity increased from 400,000 doses/year to 1,000,000 doses/year; * the entire vaccine amount needed for children age of 9-12 months was delivered in small dosage ampoules which facilitated immunization field work and reduced waste; Influenza Vaccine * production capacity increased from 250,000 doses/year to 800,000 doses/year, by improving the efficiency of the multiplication of the influenza virus vaccine strains. BCG Vaccine * the new equipment used (freeze-drier, autoclave, thermostat, dispenser, etc.) enabled the expansion of freeze-dried batches by 23%; 28 Quality assurance and control * new department for Internal Quality Control; plans for equipment control, procedures for calibrating/standardizing/maintenance and operation according to GMP standards; * advanced production equipment, safe storage facilities and quality control equipment to meet the most stringent international standards for product quality, in line with the WHO requirements; * the Institute gained significant potential for participating in international tenders for supply of vaccines for large immunization programs, either as a stand alone producer or in joint venture with Pasteur Institute from France with which the Cantacuzino Institute has developed a solid partnership. 29 Annex 4. Bank Inputs (a) Missions: d9X:E:h'~~____ Count. Specalt Identification/Preparation I health specialist (team leader), 1 public health specialist, I pharmaceuticals specialist, I health economist, 1 family planning expert, I senior implementation specialist, I operations officer Appraisal/Negotiation I health specialist (team leader), 2 public health specialists, I pharmaceuticals specialist, 1 health economist, I family planning expert (UNFPA representative), I senior implementation specialist, I operations officer, ladministrative assistant Supervision I public health specialist Core team (program team leader), I health finance/ management specialist, I health information systems specialist, I medical equipment specialist, I program assistant, I operations officer (task team leader), I operations analyst MTR 03/94 1 senior implementation U U specialist I PAS I HNP specialist (mission leader), 1 senior implementation specialist and PAS, I pharmaceutical specialist, 30 Monthl No. of Persons and Specialty lmplementation Development Stage of Project Cycle Year (e.g. 2 Economists, I FMS, etc.) Progress Objective .____.___.____:__ Countt Specialty I program assistant, 1 health policy specialist, I health promotion specialist, 1 operations officer ICR 02/2000 1 health economist S S (program team leader), 1 public health specialist, I peer reviewer from HNP thematic group, 2 Support staff with medical and health management qualifications, 1 operations officer (task team leader) 1 Health information systems expert, 1 Health Services Management Expert (+ input from other staff of Institute of Health Services Management and surveyors) (b) Staff __Wof'prolectuAVLatest Estimate No. Staff weeksW (U$VS'O00 Identification/Preparation+ Appraisa_Negotiation 452.376 Supervision 1,109.209 ICR 18 45.769 Total 1,607.623 I/Totals include Trust Fund financing of US$3,315 2/ Staff weeks not available for years prior to 1999; 22.7 staff weeks supervision recorded in 1999. 31 Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Development bjectives:: Rating (i) rehabilitate and upgrade the primary health care delivery system which is M collapsing through want of equipment, spare parts, drugs, medical supplies and transfer of knowledge to health providers; M (ii) support the first steps of a major restructuring of health sector financing and management to ensure a sustainable and cost-effective health care system in the medium term. Qutputs qf Components: A. Rural Dispensaries AM B. Reproductive Health A C. Training (component cancelled) NA D. Drugs & Consumables AM E. Mgmt of Emergencies M F. Health Promotion M G. Health Reform Strategy SU H. Health Mgmt Information System N I. Health Mgmt School SU J. TB Control + HIV/AIDS Al K. Project Management M Social: Some components were targeted effectively at pro-poor services (rural primary health care, TB control and HIV/AIDS services). Other components were targeted at high priority health needs for women reproductive health services. Improvement in reproductive health services has occurred more markedly among better educated, urban women from higher socioeconomic groups. Procurement of drugs and consumables in the initial period of the project appeared to improve availability of low-cost drugs during a period of national shortages, but there is no data available on the distributional impact. For some components, a more participatory and less centralized approach to implementation would have improved effectiveness and sustainability. This was highlighted by the beneficiary survey for the upgrading of rural dispensaries, by one of the stakeholder workshops for emergency medical services, and by the stakeholder workshop for health management information systems. 32 Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating At MTR U At project close S 6. 2 Borrower performance Rating At MTR U At project close U 33 Annex 7. List of Supporting Documents The following documents are available in project files: King's Fund College, London and Nuffield Institute for Health Services, Leeds, "A Healthy Future" (a health sector strategy report prepared for the Romanian Ministry of Health), 1993 Sue Jenkins, John James and Catriona Waddington, "Evaluation of the Health Reform in Eight Pilot Districts in Romania" October/November 1995 PS Consulting Group, Romanian Pharmaceutical Industry Restructuring and Strategic Plan, December 1993 Jillian Clare Cohen, A Review of the Pharmaceutical Sector in Romania, September 1996, based on the findings of June 12-22 Mission Health Insurance Commission of Australia, Economic Analysis of Health Sector Reforms: Health Finance Policy Development in Romania ", and " Model User Guide (HEROM"), 1998 InterHealth Institute, Maryland, USA, Romania Health Sector Reform Report, June 5, 1998 Florina Serbanescu, Jay Friedman and Leo Morris, Reproductive Health Survey, Romania 1999, January 2000 Dr Roddy Neame, Appraisal of Health Sector Rehabilitation Project, Health Management Information Systems Component, February 2000. Institute of Health Services Management, Bucharest Appraisal of Health Sector Rehabilitation Project, World Bank Loan RO-3409, Primary Health Care Rehabilitation Component, May 2000 34 Annex 8. Beneficiary Survey Results Beneficiary surveys were carried out as part of the Intensive Learning Model ICR for two components of the project for which there are large numbers of disbursed beneficiaries, and for which further investment is contemplated in the planned second health project. This Annex also includes a summary of the findings of an earlier survey and evaluation of piloting of decentralization and primary health care financing reform in eight districts, which was carried out as part of the Project. 1. Upgrading of Rural Dispensaries: Comparative Survey of Beneficiaries for Sample of Upgraded and Non-Upgraded Rural Dispensaries A survey and in-depth evaluation of the upgrading of rural dispensaries under the project was carried out in April-May 2000 by the Institute of Health Services Management, Bucharest. The full report and data set are available in project files. A summary of findings is presented here. Methodology: Two objectives were set for the survey and evaluation: 1. Assessment of Project implementation (medical service provision) in rehabilitated vs. non- rehabilitated health care units. The following were assessed: * Use of medical equipment * Type of medical service provided * Level of activity X Participation of local authorities in the activities 2. Assessment of results in rehabilitated vs non-rehabilitated health care units. The following were assessed: * User satisfaction - population, physicians, nurses, local authorities * Medical staff attraction to rehabilitated dispensaries * New facilities made available to the population The assessment included two rehabilitated dispensaries and one non-rehabilitated dispensary per county in a number of ten counties selected on the criterion of proportional distribution among the country's historical provinces. At county level, dispensaries were randomly selected for assessment. The following tools were used: A. QUALITATIVE SURVEY For the main objective of the survey to be attained, a thorough-going interview and observation were used for data collection. For each of the 30 dispensaries visited a report was created and four interviews conducted of which three with health care staff (medical and nursing) and one with local officials (mayor, secretary of mayor's office). These techniques helped gather information on health care staff satisfaction with their profession, their social satisfaction and their material satisfaction. 35 The following variables were monitored: I . Professional career 2. The impact of equipment and condition of the building on activities 3. Services provided 4. Perception of medical service quality 5. Local authorities and community support to the dispensary 6. Compensation Summary of Results and Conclusions The qualitative component of the study suggests that it takes not only an opportunity such as the rehabilitation program, but also a motivated medical team, willing to seize that opportunity, in order to make a difference. Whenever the two ingredients were there, rural dispensaries provided the same (or sometimes better) standards of services compared to their urban counterparts. However, when a motivated team was missing, very little use was made of the additional equipment and the status of the building was degrading already 2-4 years after renovations were completed. The changes brought by rehabilitation reduced the tensions generated by physicians' tendency to seek jobs in more attractive urban areas, but were not always strong enough to motivate long term stability in one unit. Without significant involvement and detailed information from the staff meant to use the upgraded facilities, the rehabilitation has been assessed by those beneficiaries as an action that stopped half-way. Some investments and renovations, regarded by the staff in the dispensaries as higher priority, were either of inappropriate quality or not done at all (water supply, appropriate solutions for heating, sanitation, transport for home visits, etc.) Although the medical equipment of rehabilitated dispensaries significantly improved medical services and affected the physician's decision to stay in rural area, the study also found that part of the medical equipment was inadequate (e.g. rudimentary microscopes) or obsolete already at the date of purchase (e.g. laboratory equipment, out of date and very labor intensive in the use, financed by PHARE program, but still associated with the World Bank project; glass syringes provided at a time when the Ministry of Health, supported by another component of the World Bank project, was attempting to generalize use of disposable syringes). The investment in superfluous or obsolete equipment were regarded by interviewed staff as obvious waste of scarce resources, as well as eliciting psychological effects of frustration and exclusion of physicians supposed to use such equipment. In some of the surveyed locations the study found defective medical equipment and disrepair of the renovated premises and utilities (water supply, sanitation, heating, etc.) which can partly be blamed on carelessness of dispensary staff. Such attitudes can be explained by the fact that either the staff didn't go through an appropriate selection and formally assigned responsibility at the time of rehabilitation, or that they were not involved decisions regarding the rehabilitation process, hence the lack of informal "ownership" or willingness to assume responsibility. Relations between dispensaries and the mayors' offices have not improved, as might have been expected as a result of common involvement in the rehabilitation exercise. Although in many cases there are good interpersonal relations between the persons involved, the project usually strained the relations of the health care provider with the local authority, possibly due to lack of 36 partnership. Too often the medical staff were not given a voice in the process and had no choice but to put their signatures on the report of acceptance of the works, even if quality was below expectations. In spite of some of the more critical views of the professionals and representatives of the local administration involved in the study, in tertns of primary care users' satisfaction the rehabilitation of rural dispensaries can be regarded as successful. The opinion of users on all researched aspects of the rehabilitated provider units was better compared to the non-rehabilitated units. The main improvements, as perceived by the users, could be summed up as follows: better overall quality of medical services; improved behavior of the medical staff; better nursing staff distribution in the territory; better medical equipment; better quality furniture; cleaner premises; better amenities enjoyed by users. B. QUANTITATIVE SURVEY - INTERPRETATION OF USER QUESTIONNAIRES 442 persons were asked questions about the 30 dispensaries (20 rehabilitated and 10 non- rehabilitated) included in the assessment, of which 298 were surveyed about rehabilitated dispensaries and 144 about non-rehabilitated ones. The questionnaire was designed for users of family doctors' services in the rural communes surveyed. A sample of 15 users per dispensary was randomly selected from the rural commune physicians' lists of patients. The questionnaire was designed for users of family doctors' services in the rural communities surveyed. It asked 25 questions. The first four questions tried to identify the health care unit in greatest demand, the reason for this and the frequency of demand. The next nine were concerned with the assessment of modern amenities and medical staff behavior as viewed by users. Another four were targeted to medical services, another four concerned under-the-counter payoffs, and the concluding four -sample composition. Summary of Results and Conclusions: Unlike non-rehabilitated dispensaries, rehabilitated dispensaries have basic equipment and their premises have acceptable physical conditions required for provision of adequate primary care. The differences regarding physical infrastructure are reflected in the opinions of users regarding quality and availability of services. The overall quality of medical services is rated as very good by 59.6% users of rehabilitated units compared to only 36.1% in non-rehabilitated units. The difference is even higher when comparing users' opinions regarding availability of equipment (very good or good in 53.7% of rehabilitated dispensaries vs. 13.2% in non-rehabilitated). Reported user satisfaction with the health care provider depends to a great extent on the providers' potential to perform testing and treatment, if necessary. On all service items surveyed, services were more frequently provided by rehabilitated than non-rehabilitated dispensaries, indicating that the first category is providing better services to their users than the second, as detailed in Table 1.1 below. Table 1.1 - Medical services, rehabilitated vs. non-rehabilitated dispensaries Medksal service Rehabilita ed ispensaries Non-reabilitated dispensaries Total Number Percentaae Number Percenta2e Total 276 132 Medical consultation 270 97.8 131 99.2 Blood pressure measurement 236 85.5 112 84.8 37 Oscillometry 2 0.7 0 0 Electrocardiogram 61 22.1 2 1.5 Proctological exam 2 0.7 0 0 Gynecological exam 24 8.7 5 3.8 Cytohormonal smear test 0 0 0 0 PAP smear test 0 0 0 0 Visual test 22 8 2 1.5 Hearing test 17 6.2 2 1.5 Otoscopy 5 1.8 1 0.8 Cerumen removal 14 5.1 4 3 Front rhinoscopy 3 1.1 0 0 Nasal aspiration 1 0.4 0 0 Bladder catheterisation 3 1.1 2 1.5 Gastric lavages 3 1.1 0 0 Foreign body extraction 19 6.9 4 3 Minor surgery 49 17.8 26 19.7 In addition, the survey found that rates of periodical check-ups of children, pregnant women and workers were higher in rehabilitated dispensaries, which is consistent with the physician being a resident in every case. Family planning services, more frequent for rehabilitated dispensaries, may be a sign of the users' growing confidence in the family doctor (11.2 per cent vs. 8.3 per cent). As to under-the-counter payments, the percentage was similar in both groups (29.8 and 31, respectively) and were basically in food (47.8 and 48.9, respectively). The main reason for them was the patient's satisfaction with medical service quality (51.1 per cent vs. 59.9 per cent). In two cases the medical service was made conditional on the payoff. This suggests that equipped dispensaries have not attempted to exploit their improved ability to provide services by soliciting higher patient payments. 2. Emergency Medical Services: Survey of Ambulance Services in All Districts Benefiting from Provision of Equipped Ambulances and Training of Ambulance Staff The stakeholder workshop on Emergency Medical Services (described in Annex 9) developed a survey of districts to assess the utilization and effectiveness of equipped ambulances provided under the Project. The.survey requested data from all District Ambulance Services regarding: ambulance numbers, utilization of ambulances, and a range of timeliness and clinical performance indicators. It asked for data to be broken down by equipped ambulances procured under the Project, other equipped ambulances, unequipped ambulances and medical transport vehicles. A full list of data requested and full data sets received are available on project files. Response Rate: Twenty nine out of 41 districts (71%) were capable of providing some data, however all but nine were only able to provide data on vehicle numbers by type. Of the nine districts able to provide data on utilization and other performance indicators, only three to five were able to differentiate utilization data for ambulances procured under the project from other ambulances. Analysis of Data: In 17 of the 29 districts which supplied data, 100% of equipped ambulances in the district were supplied by the Project, and in all but four of the districts, the majority of equipped ambulances 38 were supplied by the Project. All districts report a very large ambulance vehicle fleet, but almost 93% of these vehicles are not equipped or are used for transport purposes only. Table 2.1 and Chart 2.1 present a summary of this data. In the three to five districts which supplied utilization data, ambulances supplied by the project were being used more intensively than other ambulances (higher kilometers per ambulance, higher mission numbers per ambulance). Ambulances supplied by the project are rarely if ever used for simple transport missions. They are rarely used for home visits (30-50 times less than for unequipped vehicles) or minor emergencies (only one in 10 missions). In summary, ambulances supplied by the project appear to be used intensively and appropriately for the most serious category of emergency call-outs. From the few districts able to differentiate clinical outcome data for different ambulance types, it is not possible to draw conclusions to be drawn about health outcomes of the Project. The data supplied also highlights the fact that ambulance response and patient movement times remain unacceptably long, with high numbers of patients found dead by the time the ambulance arrives. This provides support to the priority given to further development of an optimal ambulance network in the GOR's health strategy and in the planned second health project. Table 2.1 - Ambulances b Type, by District MEDICAL EMERGENCY EMERGENCEMER G ENCY PERCENTAGE OF TTL TRANSPORT ABLNE EQIPD EQUPEDGNC EQUIPPED DISRIC NUBTOTAHCLE WITHOULACE AMBULANCES: AMBULANCES: ABLNE DISTRICT VE~(WIHICLE WQITHOUT PROCURED ByOTE SURE PROJUECTY EQUIPMENT) EIPNT PROJECT OTHEOSORCE 1 AB 108 90 12 5 1 83 2 AG 74 36 32 6 0 100 3 AR 64 25 19 3 17 15 4 BC 121 116 2 3 0 100 5 BH 107 101 0 5 1 83 6 BN 79 70 3 5 1 83 7 BR 86 80 0 4 2 67 8 CV 50 43 6 1 0 100 9 DB 85 79 1 5 0 100 10 DJ 106 101 4 0 1 0 11 GJ 83 79 3 1 0 100 12 GL 86 81 4 1 0 100 13 GR 52 47 4 1 0 100 14 HD 113 106 4 1 2 33 15 HR 86 61 0 3 22 12 16 IL 52 50 0 2 0 100 17 MH 59 34 0 4 21 16 18 MM 100 94 1 5 0 100 19 NT 78 51 22 3 2 60 20 OT 78 74 3 1 0 100 21 PH 125 - - 4 1 80 22 SJ 65 55 9 1 0 100 23 SM 49 47 0 2 0 100 24 SV 80 79 0 1 0 100 25 TM 114 87 15 12 0 100 26 VL 88 85 1 1 1 50 27 VR 67 50 12 5 0 100 28 VS 74 71 0 3 0 100 29 B 198 175 0 23 0 100 TOTAL 227 2067 167 111 72 61 39 Chart 2.1 - Ambulances Supplied by Project as Share of Total, by District 0 20 40 60 80 100 120 140 160 180 200 AB AG 74 AR 64 BC 121 BN B N ae t .........= 1@ .sa.UB fE>ISmEs gE79 BR ~~~~ ~~~~~86 Cv 50 DB M85 |TOTAL NUMBER OF EMERGENCYAND DJ 106 TRANSPORT AMBULANCES 83 U EMERGENCY EQUIPPED GJ AMBULANCES: PROCURED BY PROJECT GL s#rr-Xrr@rty#t fi:HUlStg.198ti 186 O EMERGENCY EQUIPPED AMBULANCES: OTHER SOURCES GR 52 H D rFEr9#S99zFaDE#-r cW:___ht it5113 HR 86 IL 52 MH 59 MM~~~2 MM SimY99#9#t2@1po>@ks9#@pWmr999r9P-#a 100 NT ry9l5zvor922>9Sss>9E.78 NT OT 9 st##F9 !* 78 PH 1125 SM 6 TM 114 VL 88 VR 67 Vs 74 B 98 40 3. Summary of 1995 Beneficiary Survey of Decentralization and PHC Reform Pilots in Eight Districts In 1995, a formal evaluation was carried out of the experiment in decentralized management and financing reform for PHC by internal (Institute of Hygiene, Public Health and Health Services Management) and extemal evaluators (Institute of Health Sector Development, London, UK). (See report by Sue Jenkins et al. "Evaluation of the Health Reform in Eight Pilot Districts in Romania" October/November 1995, on project files). In the pilot, 84% of the population registered with family doctors and 1.6% had exercised the r ight to move from one doctor to another in the first year of the pilot; the attitude of doctors towards patients improved; referrals to hospitals and polyclinics decreased, but admission to hospitals increased by 3.2%; productivity seemed to increase (average earnings of family doctors was on average 15% higher than in non-pilot districts, while their workload increased by about 21% for consultations, 40% for home visits and 95% of family doctors provided 24 hour coverage; the income spread across doctors increased; most family doctors perceived that their professional and social status had not improved, but more than 80% reported an increase of income, and about 75% were willing to continue to participate in the reform. However differences in access between rural and urban areas persisted; the quality of many services included in the scheme was difficult to evaluate and equipment to provide some of these services was inadequate; there was a tendency for "inflation" in payments related to number of services provided which made it necessary to introduce some financial disincentives; there were no mechanisms in place to enable participation of family doctors in continuous education programs and to stimulate development of group practices; District Health Authorities were heavily involved in operational management, with most strategic decisions taken at Ministry of Health level; decentralization was limited to management of budget for new family doctors payment scheme and contracts. 41 Annex 9. Stakeholder Workshop Results Because of the complexity of the Project, it was not practical to cover all of the project objectives, activities and issues in a single workshop. Four workshops were conducted, focusing on components in areas in which the second health project was expected to carry out further investment and development, and for which it is therefore high priority to learn lessons from the experience of the Health Services Rehabilitation Project. Stakeholder Workshop 1: Health Information Systems February 15, 2000 List of Participants: 1. Irinel Popescu MOH - Secretary of State 2. Aurelia Dreve MOH- Project Coordination Unit 3. Costel Anca National Health Insurance House 4. Mariana Bazavan MOH - National Health Statistics Center 5. Sebastian Nicolau MOH - Medical Assistance General Directorate 6. Cristiana Dumitrescu MOH - 7. Dan Ursuleanu MOH - National Health Statistics Center 8. loana Persache MOH - National Health Statistics Center 9. Paul Vasilescu Institute of Health Services Management 10. Dan Farcas MOH - National Health Statistics Center 11. Theodor Stanescu Compaq Computers 12. Teodor Negru MOH - Budget General Directorate 13. Mircea Popa MOH - Public Health Directorate 14. Loraine Hawkins WB - Health Program Team Leader 15. Dr Roddy Neame WB consultant, health information systems specialist 16. Richard Florescu WB - Task Manager for HSRP 17. Silviu Radulescu WB - Public Health Specialist The workshop had two objectives: (i) to present the findings of the assessment of the HMIS component implementation and get stakeholders feed-back and (ii) explore the feasible options for the future developments. Objective (i) The following observations and conclusions were drawn from discussion: * Due to delays in the early stages of procurement, time pressure adversely affected program implementation. Romanian counterparts and the contractors were forced to focus mainly on the hardware side of the program (delivery of computers, installation and network tests) in order to expend funds before project closure date, leaving the "soft" side of implementation for later on. Hence many aspects remained to be implemented after the project closing date. * Reservations were expressed regarding the selection of the pilot district: instead of Neamt, a district 400 km far from Bucharest, the MOH should have selected a closer one or part of Bucharest, where more staff and expertise were available, and were MOH monitoring and control could more easily be exercised. * Lack of adequately skilled staff at local level and difficulties in hiring due to the financial constraints generally faced by the public institutions impaired implementation. * Continuous turnover of staff, both at the technical level (component management) and at the MOH senior management, led to many contlicting decisions, which often confused the main contractor and its sub-contractors. Many NHSC, DPHA, and hospital staff trained under the project left the system for better paid jobs outside the health sector. 42 * No one organization had full authority to implement the program across the boundaries of the organizations involved - authority was dispersed across MOH, NHSC, DPHA and hospitals. * Recommendations regarding the next steps to be taken in order to overcome remaining implementation problems identified during the field visits, were made by the Bank's team, and verbal agreement was reached upon them. These recommendations are described in detail in Dr Roddy Neame's report (WB consultant participating in ICR), available in project files. Objective (ii) In order to assess the desirability and feasibility of rolling out the pilot H-iMIS at a later stage (once the pilot was fully operational and evaluated), various options for further developments were presented by Dr Roddy Neame (a copy of his presentation is attached to his report) and discussed. The following conclusions and issues were drawn from the discussion: * It was recognized that an integrated system to cover the needs of all the health sector key stakeholders (MOH, NCHS, NHIH, other health insurance houses for the parallel health networks, professional associations and patients) would be the best approach for future developments, but consistent with decentralized investment. * At the same time, the MOH Secretary of State, advised that strong political commitment has to be reached for this purpose, through extensive consultations to be carried out among the stakeholders. * It was proposed that the second health project should include provision for TA needed to explore in detail various follow-up options, and take an appropriate decision. Stakeholder Workshop 2: Emergency Medical Services February, 2000 List of participants: 1. Dr. Irinel Popescu MOH, Secretary of State 2. Dr. Serban Bubenek MOH, General Director, Department of Health Policy 3. Dr. Carmen Angheluta MOH, PCU Director 4. Mr. lonut Bazac MOH, PCU Deputy Director 5. Dr. Radu Dop President of Emergency Medical Services (EMS) Commission of the MOH, surgeon Bucharest Emergency Hospital 6. Dr. Bogdan Martian MOH, Secretary EMS Commission, Department of Medical Assistance 7. Dr. Raed Arafat EMS Medicine Commission, ICU specialist, chief of Tirgu Mures EMS Pilot Project 8. Dr. Mircea Oprisan EMS Commission, Chief of Bucharest Ambulance Service 9. Dr. Beloiu representative of Bucharest College of Physicians 10. Prof. Gheorghe lonescu EMS Commission, chief of surgery Bucharest Emergency Hospital 11. Dr. Raducu Nemes EMS Commission, chief of surgery Dolj District Hospital 12. Dr. Dan Tulbure EMS Commission, chief of ICU Fundeni Emergency Hospital 13. Dr. Vlad Ciurea EMS Commission, chief of neurosurgery, Bagdasar Emergency Hospital 14. Col. Dr, Dan loan Manastireanu EMS Commission, ICU specialist Military Central Hospital) 15. Dr. lulian Coca EMS Commission, Ambulance Service in lasi 16. Dr. Gabriel Tatu Chitoiu EMS Commission, cardiologist Bucharest Emergency Hospital 17. Dr. Gabriela Pit EMS Commission, specialist in emergency medicine Cluj District Hospital) 18. Dr. Dana Farcasanu Institute of Health Services Management 19. Dr. Constanta Ciobotaru representative of Romanian Emergency Medical Services System project supported by Swiss Government 43 20. Mr. Dumitrean Parliamentary Health Committee 21. Dr. Silviu Radulescu WB, Health Specialist 22. Dr. Dana Burduja WB, team assistant and participant in MOH EMS working group The workshop had two objectives: (i) to assess implementation, impact and lessons learned in the investment in ambulance services under HSRP, and agree upon data to be requested in a beneficiary survey of ambulance services; and (ii) to assess implementation, impact and lessons learnt from the investment in rural health centers as part of a network of emergency services under HSRP. Objective (i) The following points and conclusions were reached in discussion: * From a technical point of view, new ambulances procured under the project are much prepared to offer services in major emergencies. * However organizational and staffing issues prevent the ambulance service from achieving higher efficiency. These issues are mainly the following: - Not all fully equipped ambulances are used 24 hours per day (some of them only 8 hours) because complete teams to operate them around-the clock are not available. - In most districts there are no specialist emergency physicians employed by the ambulance service and only 30% of other physicians have a certified competency in emergency care. - The number of nurses and ambulance drivers who are trained to participate in teams providing care in major emergencies is insufficient. - There are not enough operators for the dispatch unit; 24 hour coverage is usually provided only by the district ambulance station. - There is anecdotal evidence that equipped ambulances for major emergencies are not always used for the appropriate cases, but for inter-hospital transport or non-medical transport. * As a result, response times are in many districts still high, even for major emergencies (averages of 20-60 minutes) and the response times for rural areas are significantly higher. * Data completeness and quality is very variable. Criteria for classification of emergencies don't seem to be used appropriately, resulting in reporting by some district services of a proportion of major emergencies which is not plausible. e Future investment in the ambulance system should address organizational and staffing issues, and should be accompanied by improved data collection for monitoring and evaluation. Objective (ii) The rural health centers were intended to provide diagnostic and emergency treatment in support of primary health care in areas which are distant from a hospital. An evaluation by the MOH of the RHCs after the closure of the project revealed large differences among the various districts - from RHCs poorly functioning and facing lack of equipment as a result of re-distribution to hospitals of the equipment originally procured for the RHC, to Centers functioning as designed, which also managed to attract supplementary sources for financing recurrent costs and training of medical staff. In discussion among stakeholders, the following factors were viewed as the major considerations determining whether implementation was successful: * Attracting Medical Staff. The assumption that the provision of medical equipment would provide strong enough incentives to the physicians and nurses to settle or provide services in the remote areas where the RHCs were established, proved to be true only in part. Thus, some of these centers failed to attract the necessary trained medical staff, to use the medical equipment efficiently and effectively. Continuous medical education for the existing staff was not fulfilled properly in a significant number of places, due to the lack of a comprehensive training strategy, and lack of financial resources. 44 * Legal and financing framework. The introduction of the new Health Insurance Law, which does not provide clear (adequate) legal provisions regarding the reimbursement of the services provided by the RHCs, combined with poorly designed organizational regulations, has had a negative impact on the overall functioning of these centers. The current unclear legal status of the RHCs also prevents them from attracting alternative sources of financing. * Redistribution of the equipment. Besides the redistribution of equipment as a result of objective reasons such as: lack of medical staff, lack of adequate infrastructure facilities or lack of demand (where redistribution occurred with Bank endorsement), the assessment revealed also cases of equipment redistribution (ambulances included) on the basis of subjective criteria. The latter cases occurred mainly after the Loan closing date. * Local counterpart funds. Sites where there was good co-ordination between the District Public Health Authority and the Local Governments, tended to be able to meet properly the pre-requisites for successful program implementation. This cooperation enabled adequate local counterpart funds for the physical rehabilitation of the buildings to be provided, allowing the installation of the medical equipment in good conditions. On the contrary, where this co-operation was not effective, the lack of adequate funds prevented the completion of civil works in due time, and thus the installation of the equipment. Stakeholder Workshop 3: Rural Health Centers: discussion of RHCs in Arges District Health Services Planning Workshop February 11, 2000 Arges District Workshop The Bank ICR Mission took advantage of a health services planning workshop in Arges District, facilitated by EU/Phare, to conduct a discussion among stakeholders at district level of the effectiveness of establishment of rural health centers under the Project. Arges district is also designated as one of the pilot districts for participation in the proposed second health project List of participants: 1. Dr. Sorin Vasilescu Arges District Public Health Authority, Director 2. Dr. Gabriel Popa Pitesti Ambulatory Diagnostic Center, Director 3. Dr. Carmen Angheluta MOH, PCU Director 4. Dr. Marieta Bardut MOH, Department of Strategy and Management 5. Dr. Chris Aagard EU/Phare foreign consultant, Institutional Reform Project 6. Dr. Daniela Valceanu EU/Phare local consultant, Institutional Reform Project 7. Representatives of GPs and hospital directors in Arges district. 8. Loraine Hawkins WB, Health Program Team Leader 9. Silviu Radulescu WB, Health Specialist Summary of Stakeholder Views: As in a number of other districts, local representatives of hospitals and primary care in Arges were critical of aspects of implementation of rural health centers. Three centers in Arges were equipped under the Project. Criticism focused on the fact that some of the equipment in rural health centers is not being used and remains idle, while the nearest hospital lacks equipment of a similar type or standard. This has led to calls for the unused equipment to be transferred to the hospital, where it will be used, and where patients in practice continue to seek treatment in the absence of effective diagnostic services at primary care level, though at the time of the workshop this redistribution had not occurred. The expressed view of the MOH was that there is flexibility to reallocate such equipment to best meet local need. 45 EU/Phare advisers in the workshop highlighted that the health services plans for the district envisage a larger role for primary health care, and the need to reduce the flow of patients to hospital emergency departments. If rural health centers could be made functional, primary care providers would be better placed to access diagnostic support and treat patients locally, in keeping with the goals of the health services plan. An alternative local perspective was that the criticisms of the rural health center activities related particularly to one of the three centers which had recently lost its local doctor, and had experienced difficulty in recruiting a replacement doctor. Lessons learned, identified by discussants: * Staffing and training need to accompany equipping of rural health centers in order to make them effective. Measures are needed to overcome problems in attracting technically trained staff to operate x-ray and laboratory equipment in rural centers. * Leadership is needed from a local doctor, and community support is needed to attract and retain a doctor, to achieve continuity in local availability of primary care. * It is difficult to win support from key stakeholders in the health system for giving priority to primary health care development so long as essential hospital services are under-equipped and under-resourced, since hospitals are seen as the level of care which serves the sickest patients and most urgent cases. Stakeholder Workshop 4: Policy, Training and Project Management June, 2000 List of participants: I. Dr. Carmen Angheluta MOH, PCU Director 2. Dr. Aurelia Dreve MOH, PCU expert 3. Dr. Dana Farcasanu NIHSM Deputy Director 4. Dr. Cristian Vladescu Soros Foundation for an Open Society 5. Dr. Gabriela Scanteie NIHSM 6. Ec. Richard Florescu W.B. Task Team Leader 7. Dr. Simona Buse W.B. Team Assistant 8. Dr. Serban Bubenek MOH, General Director of Health Policy 9. Dr. Stelian Pop Senator, President of the Parliamentary Health Committee, Director of NIHSM 10. Prof. Dan Enachescu Medical University of Bucharest, Chief of Public Health and Management Department The objective of the workshop was to review and discussed the draft ICR report's assessment of policy and training activities, and project implementation management for these types of activities, and draw out lessons for implementation of these types of activities in the future. Some of the comments of the workshop regarding project management are of general application - not just in the policy and training sub-components. The participants agreed with the major conclusions of the draft report, but made the following additional points. * The Model User Guide (HEROM) for projection and analysis of health financing produced by Australian Health Insurance Commission was highly appreciated, though further effort should be made in order for the MOH and National Health Insurance House to use all its modules effectively. * Some recommendations made in this User Guide and also in the Romania Health Sector Reform Report (by InterHealth Institute, June 5, 1998) were used for the calculation of 46 District Health Insurance House budgets and of the budget allocated to primary health services in 1999. * Some of the young professionals with public health MA and MSc degrees trained under the Project are now spread throughout the health system and in other key organizations and they are continuing to collaborate and to teach in the NIHSM, and provide local consultancy services in a range of projects. * Initially, NIIHSM provided some general courses addressed mainly to stakeholders in key positions in the health system. These courses were progressively updated in concordance with the requests of from various participants and organizations in the health services system (e.g. new training modules were designed and delivered for GPs on Management of Primary Care Units). * Pilot approaches for testing health service delivery and finance conducted under the Project provided a very good base for further steps of reform. These pilot districts are perceived as now having better results than others working with the new financing arrangements in the system - perhaps reflecting longer experience as well as additional advice and training provided in the pilot districts under the Project. The workshop noted two critical gaps or shortcomings in the health reform framework so far developed: * A better definition of the roles of Romanian College of Physicians (and other health professional bodies) is needed in future. Current legislation gives the College a powerful role in resource allocation, which may conflict with the College's role as representative of the financial and other interests of doctors. - Because of the existing (mainly political) appointment processes in the National and District Health Insurance Houses, political instability in the MOH and GOR is transferred into the managenment of the health system. On this issue, workshop participants were critical of the consultants' and World Bank's recommendation to remove elections for the Boards of insurance houses from the Health Insurance Law. Thc workshcp participants highlighted the following lessons about how the implementation and development impact of these activities could have been strengthened: * More appropriate selection of technical assistance is needed, especially foreign consultants, so that experts are selected who have not only experience abroad but also solid knowledge about local conditions in Romania. * A legal advisor should have been involved from the beginning to the end of the project, to deal with legislative issues regarding reform implementation as well as legal issues arising in project implementation. * Management training modules need to be delivered in the districts as well as in Bucharest, because the demand for such training has increased, in part because of the refonns supported by the Project, and the Institute cannot meet this demand. Additional staff are needed. * There is a need to provide incentives to the highly skilled professionals trained under the programs developed in the Project to keep them in the health system. Health sector salaries are low relative to economy-wide averages. * Better coordination between WB procedures and GOR procedures or local laws would have avoided many implementation delays. * There is a need to maintain the same team for managing and implementing the whole project, and project management staff should work on a full time basis. 47 Annex 10. Borrower Contribution to the ICR MINISTRY OF HEALTH WORLD BANK PROGRAM COORDINATION UNIT HEALTH CARE SYSTEM REHABILITATION PROJECT WORLD BANK LOAN RO-3409 (Final Report) The Loan Agreement between the Government of Romania and the World Bank (WB) was signed in November 1991, whereby 150 million US$ were offered to support the Romanian health care system rehabilitation. The Romanian Government's projected participation amounted to 57.5 million US$, in Lei equivalent. The Loan Agreement was ratified through Law no. 79/1991. The project focussed on two main targets according to health care (HC) system specialists' priorities: - Primary medical assistance rehabilitation; - HC system financial and managerial restructuring. To meet these objectives, the following activities were contemplated: I. Primary medical assistance rehabilitation, through: - physical rehabilitation of, and medical equipment supply to rural dispensary units; - improving assistance to mother and child, and family planning services; - improving emergency medical assistance; - supplying basic drugs and consumables; - improving drug control according to international standards; - improving production and control of immunization and blood products, according to international standards; - medical and managerial staff specialized training; - Introducing health promotion programs. HI. HC system financial and managerial restructuring, through: - preparation and implementation of the first phase in HC system reform, HC system funding, new development of the system's organization and management legal framework; - experimental testing of the reform program in several counties; - development of the HC system reform; - introducing information technology (IT) to operate the HC system; - founding a HC management unit. What was particular to the HC rehabilitation system loan to Romania was the large amount of the loan as compared to other countries in the area, as well as the scope and complexity of its targets, circumscribing practically the whole HC system. This is the largest finalized loan that the World Bank granted to Romania in the last ten years. The initial loan was projected for a 5 years' utilization period, but the estimated loan withdrawal phasing proved to be too optimistic. The current WB estimation of the time required to implement WB loans as high as 100 million US$, is 7 years. Implementing the project would mean that all institutions involved are continuously focussing on the envisaged targets, trained managerial staff exists to conduct WB project operations, and introducing major changes in HC 48 policies and system structuring. Loan funds were but partially withdrawn by closing date (June 30, 1996). This was due, mainly, to the initial lack of institutional capacity for project implementation by the Ministry of Health (MH) and subordinated institutions, frequent HC policy changes, lack of specialized staff, as well as local legislation constraints (with distinct interpretation by the Court of Audit). All in all 63.3 million US$ were withdrawn. The utility of some of the loan-funded subprojects was re-examined during project operation period, resulting either in adjusting or changing subproject extent, or new project approval. Once completed, some of the subprojects whose costs had been only roughly estimated generated fund savings that were redirected to new projects. The Ministry of Health (MH) and Ministry of Finance (MF) advocated for a full utilization of the loan funding. As a result, the WB acknowledged the relevance of the proposed projects agreeing to 4 successive extensions of loan completion deadline. The Loan was finally completed on June 30, 1999. The 1998-1999 withdrawals totaled about 48 million US$. The Project Coordination Unit (PCU) was located in MH, charged with project management and accounting. The minister's Order appointed the project implementation coordinators. Several staff changes were operated in these units during project implementation. Hlere are some of the main project achievements, according to the initial objectives and new priorities identified during loan operation: 1. PRIMARY MEDICAL ASSISTANCE REHABILITATION 1. Primary and emergency medical assistance were thought of as major targets of the rehabilitation program, considering the existing health indicators (high general morbidity and death rate, lacking medical staff in certain geographical areas, low medical services quality due to outdated or lacking medical equipment). a) Rural Dispensary Units The project selected 419 rural dispensary units, about 10 in each county, on account of demographic criteria, local necessities and community support, for physical rehabilitation and medical equipment supply. The aim was to attract physicians to these dispensary units to provide better quality HC services. Government funds (up to 95% of the rehabilitation costs) were used. The dispensary units were provided with basic medical equipment, dentistry kits and equipment were acquired through loan-funded international bidding, and lab kits through PHARE funding. Currently, all equipment is installed and in operation in these dispensary units, providing better HC services quality. Because the initial costs of this project component were largely overestimated, remaining funds were redirected to other project components. The original dispensary unit program was completed by supplying refrigerators to dispensary units and Public Health Inspectorates, for storage and transport of immunization products. b) Emergency Medical Assistance Rehabilitation MH has developed the emergency medical assistance rehabilitation program, with technical assistance. The program included: - Development of Emergency Modules in 95 Rural Health Centers (RHC) - 2-3 RHC in each county, according to requests by County Health Care Directions -, provided with diagnosis and medical emergency equipment; 49 - Physical rehabilitation of and medical emergency equipment supply for 34 ERs in county hospitals; - Purchase of 250 medical ambulance vehicles for County Ambulance Services, county hospitals and university clinics; - Telecommunication equipment supplies. The program was aimed to improve effectiveness and attending speed in medical emergency calls, mainly in rural and barely accessible areas. RHiCs provide medical emergency assistance for patients belonging to about 10 neighboring rural dispensary units. Severe medical cases exceeding RHCs capability of coping with shall be sent to county hospital ERs. Some of the County Public Health Directions distributed part of the acquired medical equipment to big medical units (hospitals, policlinics) because the existing equipment was severely outdated, and also because hospitals had hardly been included in any of the rehabilitation schemes of the past few years. MH examined and approved several changes in the initial rehabilitation program. To succeed, the project depends largely on local beneficiaries' involvement (dispensary units, HC centers, and hospitals), County HC Directions and other units charged with providing local funding for physical rehabilitation of buildings, staffing or HC system operation. The frequent managerial staff changes operated in HC Directions resulted in reassigning the project medical units, affecting the coherent and timely achievement of project objectives. Project effectiveness can be examined by means of specific and non-specific indicators. The specific indicator analysis (number of tests, medical check ups, or emergency cases solved on site) was requested by MH to project beneficiaries, and is currently being drawn up. Project effectiveness examination through demographic indicator analysis (death rate, morbidity, etc.) takes into consideration the multitude of factors affecting the population health condition (living standard, environmental factors, education level, traditional and customary practices, etc.), and less the medical assistance quality. Ultimately, it should not be forgotten that medical equipment supplies were completed only in 1999, which makes their impact on medical service quality barely significant for the time being. 2. Improving Assistance To Mother And Child, And For Healthy Reproduction Extremely high infant and matemal mortality rate, excessive abortion rate, as well as an aggravated post-abortion medical condition have called for action to improve the situation. A project focussing on these issues was the outcome, with the following targets: - Improving pregnant woman's medical assistance, when risks are high; - Improving neo-natal assistance; - Improving early diagnosis of genital cancer; - Creation of a family planning and healthy reproduction network. To meet these targets the project developed and funded specific training programs for gynecologists, neo-natologists, general practitioners and medical assistants. The Mother and Child Protection Institute (MCPI) was appointed as Reference Training Center to improve assistance in high-risk pregnancy, and neo-natology. Following two international biddings medical equipment was delivered for MCPI and 50 maternity houses across the country. A comprehensive network (buildings, staffing and training courses, plus medical equipment) was created which included 230 family planning consulting rooms, and 11 family planning and healthy reproduction reference centers. The program also included contraception means to be distributed in the network, which were procured for MH through a FNUAP contract. 50 The mother and child assistance program was correlated with other healthy reproduction programs also, funded by FNUAP, WHO, USAID, NGOs, etc. Program implementation was earlier to start and become completed than were other programs. Abortion liberalization and a positive influence of family planning programs determined the last period's developments in maternal death rate, which dropped significantly, and continuously from 1.7%o in 1989, to 0.409%o in 1998. Nevertheless, as compared to other European countries (and involving a long-term change of behavioral pattern), this indicator is still too high. The overall number of abortions, after a significant increase in 1990, eventually dropped in a continuous manner due to the family planning program (from 691863 in 1990, to 271496 in 1998). Maternal death rate through obstetrical risk has constantly dropped from 263 maternal deaths in 1989 to 96 - in 1998, mostly owing to better assistance to pregnant women. Infant death rate is also descending in 1990-1998 (from 26.9%o living newborn babies to 20.5%o). Although the above indicator developments were positive in trend, their absolute numbers are still too high as compared to other countries' figures, reflecting poor living standards. 3. Supplying Basic Drugs And Medical Consumables In the HC System This activity refers to procuring drugs, medical consumables and immunization products through UNICEF, which was the MH procurement agent. Although drug requests and delivery were delayed with respect to the initial estimation, the health care system did benefit from low price drugs, medical consumables and immunization products, well below market price, at a time when the domestic offer still failed to meet the HC system requirements. The medical consumables and immunization products came free of charge. The great quantity of purchased consumables (syringes, syringe needles, catheters, surgical gloves, etc.) completely changed MOH's "syringe policy", from sterilizable to one-use-only ones, by far safer for use in infectious and catching diseases. The first volume of hepatitis B vaccine was bought through the WB program, and constituted the National Immunization Program basis. 4. Drug Quality Control Improvement The Program focussed on and supplied the Drug Control State Inspection (the present National Drug Agency) with specialized equipment allowing drug control according to international standards. 5. Improving the Production And Control of Immunization And Human-Use Biological Products The program funded the teams from "Cantacuzino" Institute and Human-Use Biological Products Control Institute to get technological equipment allowing significant improvement in their activities. Production capacity was updated and extended for the following vaccine types: influenza, BCG, measles, diphtheria, perthusis, as well as immuno-modulators. Modem vaccine production and control methods are currently used, allowing constant and controllable quality for most of biological products. Products meet GMP standards, and their prices can compete with those by prominent international firms. 6. Improving Endocrine Disease Control This project component focussed on the Endocrinology Institute specialists' training, supplying also the Institute with diagnosis and curative equipment for endocrine diseases. 51 7. Support For the National Anti-TB Program Due to the unremitting extension of TB cases over the last 10 years, TB control became a priority with MH policies (TB incidence was 101.2 cases to 100,000 inhabitants, in 1998, as compared to 58.3 cases to 100,000 inhabitants, in 1989). Although no TB control funding was initially projected, the WB agreed to include the National Anti-TB Program in the Loan to Romania. The Anti-TB program funding came from savings by bidding procedures for other program components. According to WHO methodology for TB control, the target was to enhance TB diagnosis capacity and monitor consumptive patients in an effective pneumo-phthisiology logistics and human resources network. To achieve this, an international bidding was held for the procurement of radiological and lab diagnosis equipment, TB monitoring and screening. The equipment was delivered to the Bucharest and county pneumo-phthisiology dispensary units, and the Pneumo-phthisiology Institute. Further external funds were spent on training programs for physicians in these units. 8. Day-Care Hospital Wards For Monitoring HIV-AIDS Sick-Children As HIV-AIDS monitoring is a top priority of national health policies, MH proposed to establish day-care wards in infectious disease hospitals over the country (Galati, Bacau, Vulcan, Giurgiu), and the WB agreed to engage in funding these wards. Fund savings by bidding procedures for other program components were directed to finance this component. The HIV- AIDS day-care wards were also provided with medical diagnosis equipment and consumables. A mobile unit for dental, dermatological and small-scale surgery interventions was procured, ready for medical assistance to HIV-AIDS sick-children in the territory. 9. Improving Diagnosis Capacity In Acute And Chronic Disease, And Children Hospitals After completing the programs jointly agreed upon with the WB in 1998, MH engaged the very last loan funds. The value of contracts resulting through bidding procedures was below the planned amount. In this program, emergency diagnosis equipment for children and adults, monitoring equipment for the chronic sick in some of the clinic and county hospitals over the country were procured (cardio-respiratory monitoring units, tridimensional echographs, biochemical and hematological analysis equipment, dialysis monitors, etc.). 10. Health Promotion The Program founded the National Health Promotion Center and a network of County Health Centers, which were provided with audio-video and office equipment. Health education materials were published. The national health promotion strategy was prepared. Population education programs on health promotion were published in cooperation with international organizations (WHO, UNICEF, FNUAP). II. HC SYSTEM FINANCIAL AND MANAGERIAL RESTRUCTURING 1. Reform Strategy MH and the Institute for Health Care Service Management (IHCSM) have operated this program. A detailed report analyzing the Romanian Health Care System was drawn up with foreign technical assistance (King's Fund College, British Council), containing proposals regarding the 52 medical service organization, funding and providing, and aimed to help develop the health care system reform strategy. A pilot program for decentralizing medical services in counties was developed. The experiment focussed on the following issues: - Population's efficient and universal access to primary medical assistance; - Improving medical service quality; - Patient's free option for one's physician; - Improving the family physician's social position; - Physician's payment system per capita and medical service provided. A team of foreign experts who took notice that the proposed targets had been met assessed the experiment system implementation, making a set of general and specific recommendations mainly on the financial structure, major objectives of the system, need for increased competence of system decision-makers. MH and IHCSM drew up a reform strategy draft project in 1996. After examining the draft strategy project the WB specialists commented on the lack of coherent financial support for the reform (mainly the health insurance component) and low capacity to implement the HC system reform by the institutions involved. A consultancy contract was concluded in 1997, to draw up a HC system financing study, in order to eliminate the HC system imperfections. The study focussed on the unambiguous designation of funding sources and possibilities, presenting a set of HC system financial policy options. Beside this study the consultants also developed a software program for modeling the national and county HC system incomes. Estimations of the current and future health insurance fund value were made by means of the developed model, as well the HC system income/expense ratio. The cost of medical services at country- and county level was also estimated considering the impact induced by the framework contract implementation. The Reform strategy was again improved in 1997, with PHARE consultancy support, and progress was made in topics regarding system funding, service organization and packaging, and human resources. The same year the Social Health Insurance Law was voted, and in 1998 the Public Health Law passed in Parliament, creating the legal framework for HC system reform. MH specialists initiated an international cooperation program in 1998, with consultancy by the InterHealth Institute specialists, and WB funding. Romanian and foreign specialists joined in teams (medical service packaging, medical service structure organization and operation, human and physical resource feasible distribution) to complete the final analysis and options on the HC system reform strategy. Recommendations were made on the following topics: - The minimum package of medical services in the health insurance system; - Incentives for patients and service providers; - Improving the legal framework; - An articulate definition of MH and HIH (Health Insurance House) role and structures; - Better training and territorial distribution of specialists; - Hospital operation restructuring; - Building rehabilitation and equipment updating. 53 Specialists from the WB PCU and the MH divisions developed the HC System Reform Strategy, following the report mentioned above. The Ministry of Health approved then the sectorial strategy paper. The strategic objectives in MH policies are to have healthier people living in Romania, lower morbidity and less premature deaths, fair access to HC services, and HC system effectiveness. The major issues to undergo reform that were identified in the present document were included in Romania's Sustainable Development Strategy. The HC system strategy highlights the current HC system deficiencies, main strategic intervention areas, as well as the main reform activities in: - HC system management and legal framework updating; - HC system funding; - Population health condition and medical services; - Human and physical resources in the Romanian HC system. 2. The Institute for Health Care Service Management (IHCSM) The project proposed the foundation of a medical service management school for training HC system managers and political analysts in the field. This school became subsequently the Institute for Health Care Service Management (IHCSM). The project funded technical assistance and equipment to develop the institute curricula, and provide advanced management know-how for specialists in IHCSM, MH and HC county divisions. A contract with reputed universities (London School of Tropical Medicine and Hygiene, Montreal University and New- York University) allowed the specialists and HC system decision-makers to attend two series of Master courses. In the three universities Romanian specialized to ensure teaching staff in the field, and HC service management competence series were held. Beside these courses, HC system managerial staff members (HC System County Directorates, hospital managers, etc.) attended short-term courses or grants. 3. IT Operation of HC System The Computer and Medical Statistics Center developed a new set of indicators for the Romanian Sanitary System, with foreign consultancy. The County Medical Statistics Laboratories and the Computer and Medical Statistics Center received software products to ensure the report data flow. During project implementation, the need to operate the sanitary system by information technology (IT) became unquestionable, and the framework project of the IT system was developed, as an information system for health care management. The WB approved the framework project, and also the task book and bid evaluation report. With WB clearance a contract was concluded with a system integrating company. The decision for project implementation was slow to carry out due to a complex set of factors that held it back. Thus, the project, which was drawn up in 1995-1996, had to be readjusted to the current needs and modifications operated since its development, considering also the changes in MH structure, the need to involve a great number of decision-makers and executives, complicated legal procedures, etc. The sanitary IT system designed for Romania is the most complex of its kind in East Europe. 54 Its objectives are to achieve: - Better knowledge of the population health condition, medical assistance needs and sanitary system output; - Better system planning and management; - Better resource allocation and effectiveness assessment in the sanitary system; - Operative information at all decision levels, on emergency situations; - Better decision-making through modern medical techniques; - Simple and prompt connections between the national health care system components, and with other domestic and foreign bodies. The project shall be carried out on a three-level basis, each with specific target units: 1. Strategic units: The Ministry of Health The Computer and Medical Statistics Center 2. Monitoring and coordinating units: The County Public Health Direction 3. Operative units: Hospital Health Care Center Policlinic Dispensary County blood transfusion center County emergency station Level 3 will be implemented in a pilot-county. IT system equipment and connections are completed. Communications are currently in progress. Communication lines between MH and the Computer Center are ready, and those between MH and the County Public Health Directions are being installed. Each system user has Internet Access, at all levels. The application programs required development cf new products. Work was not easy due to the complex set of factors to be considered, 1998 and 1999 were difficult transition years for the Romanian sanitary system, with major changes in its structure and operation (the new health insurance system, reorganization of the public health county authorities). Applications were adapted to the 'Year 2000' issue. The Computer and Medical Statistics Center will include a C'ompetence Center that operates the modifications appeared in an integrated procedure. The applications refer to three main fields: - Resource management; - Medical applications; - Reporting and decision-making. Staff training includes a complex program for training the future system users, and preparing new training courses for those engaged in teaching activities and the key operative staff. Four training centers were organized and fully equipped in IT in Bucharest, Piatra Neamt, Baia Mare and Timisoara. Despite the efforts to keep the trained personnel to operate the network, part of the staff left for better wages at domestic and foreign companies, in particular to the Health Insurance Houses 55 when the County HC Directions were reorganized. The training course teachers were not always satisfied with their students' initial knowledge level, pointing out risks for the project operation. Therefore, a number of IT specialists should be employed and well motivated, particularly in the Competence Center. The IT program becomes fully operational in December 1999, according to the contract between MH and the integrating company. The health insurance IT system shall be designed in another project, and it will include some of the components described above. Its functions shall differ, though, from the MH IT system, its equipment, programs and data will belong to other owners, as will be its operation and application maintenance. LOAN FUND ENGAGEMENT AND SPENDING Many of the programs initially designed for WB loan funding have suffered changes, according to the system developments in all these years. Some of the programs were supplemented (the emergency medical assistance program, IT system), others were only adjusted. Funds were saved as compared with the estimated necessary amounts, after the bidding and procurement procedures were completed. These savings were engaged in funding new programs (immunization products, medical consumables, anti-TB programs, HIV-AIDS programs, diagnosis and emergency equipment for acute disease and children hospitals). It is known now that WB procedures require that a deadline be established for loan utilization. Meanwhile, MH and MF were interested to spend the whole volume of the WB loan. The deadline for spending the WB loan was extended, in joint agreement with the WB, until March 31, 1999. As MH is currently in an advanced stage of operating new programs in March 1999, the WB agreed to extend the loan deadline to June 30, 1999, for all contracts concluded prior to March 31, 1999. In May 1998, MH had developed several programs (anti-TB, HIV-AIDS, an additional contract for the IT system, the ambulance program), and requested the WB agreement for them. Although the WB had initially intended to cancel the unspent amounts (about 12 million US$), it eventually agreed to continue the funding. Part of these funds (1.3 million US$) were returned to UNICEF in March, following completion of procurement procedures developed with this organization. Later, in January 1999, MH registered additional savings through the contracts resulting by WB agreed programs. These new funds were then used to provide hospital equipment. The speedy PCU activity succeeded to complete a new block of 78 contracts by March 31, 1999, with WB approved funding. All these activities are now finalized. All loan funds were engaged in estimative programs by January 1999, before the loan deadline (March 31, 1999). About 420.000 US$ were reported as savings after all shopping procedures had been completed in January-March, and the resulting contracts finalized, on March 31, 1999. The Ministry of Finance already annulled the saved loan amount, at MH request. On June 30, 1999, when payments were completed for all contracts resulting by shopping procedures, and by revaluation of older consultancy contracts that had been blocked, the balance showed 330,000 US$ as compared to the contracted sums. This balance results from differences in exchange value of the currency used for payment, and from differences arising between the requested and the accepted sums requested by various companies (when justifying documents were missing, payment of the requested sums was denied). 56 About 220,000 US$ resulted from the FNUAP contract for contraceptive products, and FNUAP shall return this sum to the loan account. The minister of health has already requested this fund transfer to be operated. The transfer eventually occurred in September 1999. All sums that were not used for the reasons above shall be cancelled from the loan amount when all financial operations between MH, MF and WB are completed. We also wish to point out that it is practically impossible to use a loan in totality, because there are always differences between the estimated sums and the sums contracted following bidding procedures. Other differences arise between the contracted and paid-off sums (exchange value fluctuations of the paying currency at various payment dates bring about these differences). PCU MANAGEMENT AND PROCUREMENT ACTIVITY A Project Coordination Unit (PCU) was established in MH to ensure project management, goods and services procurement, and project accounting. During the first project years PCU staffing structure underwent many modifications. Moreover, the staff was trained according to WB procedures in management and goods and service procurement. These issues, along with the frequent MH managerial team modifications, have negatively influenced program coherence and loan withdrawal rate. In the last two years, the stable PCU structure and well-trained staff allowed an accelerated loan withdrawal rate, which made it possible to operate and complete contracts amounting to about 50 million US$. In the first months of 1999, about 80 contracts resulted in 19 million US$ loan withdrawals. As compared to these figures, the first five years of PCU loan operations resulted in 50 million US$, and the remaining 50 million US$ were under UNICEF operation (UNICEF was the procurement agent for MH). During the project implementation period the following activities were organized: 19 international bidding procedures for goods, 42 international shopping procedures, 4 single source procedures, 7 consultancy acquisition procedures, and 2 types of direct international contracting procedures (LNICEF, FNUAP) for drugs, medical consumables, immunization products and contraception means. MAIN DIFFICULTIES ARISING DURING PROJECT IMPLEMENNTATION 1. Difficulties in Project Management The lack of training in the responsible institutions was the main problem at the beginning of the project operation period. The difficulties related to insufficient knowledge in project planning, decision-making, staffing, and WB procedures. The institutional and managerial incapacity was aggravated by the communication deficit between the ministry departments, the other institutions involved and the county health directorates. A net orientation of the sanitary policy remained uncertain for a long period. Decision planning and assuming was often put off or suffered frequent changes, even later, when the loan was already in operation, because MH decision-making teams had changed. The great number of persons involved in decision-making or as executives, which were also engaged in their daily activities, unrelated to the project, brought no benefit to project related activities. The counterpart funding (budget funds) management was inadequate. The starting period in project operation was not funded (counterpart funds) at all, and it brought about great delays in completing the physical rehabilitation of rural dispensary units; later on additional funds were necessary to continue this component because building materials prices went up sharply. Some 57 of the building works required local budget contributions, which were difficult to obtain. Moreover, the amounts in Lei had to be supplemented due to the VAT Law. The counterpart funds (57.5 million US$) were spent by December 1996. After that date, custom duties and VAT were paid from Health Budget funds. A major constraint was the frequent change of MH and PCU staff, and the staff's insufficient number and inadequate training. Scarce office equipment and bad communications, mostly in the first part of project implementation period, was also a cause for delay, requiring the PCU and MH considerable efforts to amend. 2. Legislative Contradictions The Loan Agreement, ratified through Law 79/1991, required the application of new regulations and procedures, some of which apparently contradicting domestic legislation. It was particularly hard to persuade the Court of Audit to accept the WB standard procurement documents. The Court of Audit - the preventive control division - rejected payment for some of the consultancy or procurement contracts with international bodies or institutions, which observed WB procurement procedures but did not concord with domestic procedures. Some of these payments were accepted after MH disputed the Court of Audit decisions, but other payments were solved only through Government Decisions, which required long-term to solve (2-5 years). The Loan agreement stipulation for a commitment commission to be applied to the total unspent funds, in combination with the far too optimistic estimation of withdrawal speed, or project implementation issues, brought about penal or civil responsibility for natural persons who were program coordinators, according to the Romanian legislation. In such cases when project implementation depends on a full variety of factors and great number of persons, finding a "guilty" person becomes an arbitrary and subjective decision. To avoid such situations a future loan agreement should specifically mention the withdrawal quotas, at the date when the borrower is ready to use these funds. This stipulation would also reduce the borrower's charge of reimbursable sums. TOTAL DRUGS, IMMUNIZATION PRODUCTS, 59,697,266.61USS CONSUMABLES TOTAL EQUIPMENT 86,659,897.52 US$ TOTAL CONSULTANCY, GRANTS 5,676,805.62 US$ GENERAL TOTAL 149,033,969.75 US$ 58 MINISTRY OF HEALTH WORLD BANK PROGRAM COORDINATION UNIT PROJECT ACTIVITY VOLUME BY YEARS YEAR NUMBER OF CONTRACTS CONTRACT VALUE (US$) 1992 15 18 158 295.56 1993 25 13 361 373.16 1994 34 9 036 957.09 1995 107 31 414 539.65 1996 45 19 591 432.86 1997 24 9 844 366.99 1998 10 29 316 190.90 1999 78 18 315 813.54 TOTAL 338 149 033 969.75 Schedule 1 Withdrawal of the Proceeds of the Loan Category Amount of the Lo" a Fial Allocation Allocated According to August 1, 1999 the Agreement Dated June 03, 199$ (USD)__ _ _ _ _ _ _ _ _ _ _ 1. Goods 61,400,000 56,697,266.61 a) medical supplies b) equipment, materials 79,600,000 86,659,897.52 spare parts 2. Consultant services and 7,000,000 5,676,805.62 training 3. Unallocated 2,000,000 420,000.00 cancelled 546,030.25 to be cancelled TOTAL 150,000,000 150,000,000 59 CONCLUSIONS ON PROJECT COMPLETION The experience gained in operating the project shows that the following requirements should be met to reduce risks and enhance success probability for WB funded programs: - Project objectives should concord with the country's overall policy, in particular with health policy, and the WB objectives; this will help maintain both the decision-makers' and the beneficiaries' commitment during project implementation; - Programs should be relevant on the medium- and long-term, by maintaining target on scope and priorities, population relevance, and good cost-benefit ratio; - Loan and counterpart (government) funding should be available and timely, to ensure adequate project implementation; the political sequence and beneficiaries' consensus are determining factors in the implementation stage. Several issues should be pointed out as determining the effective project management: - The reciprocal influence between the political, economic and social environment and the project components; - The timely involvement of all beneficiaries; - Establishing moderate objectives; - Establishing adequate implementation periods, considering the actual implementation capacity of the institutions involved; - Stipulating fund utilization by modules, with terms only when the borrower is ready to effective implementation, and avoiding additional costs for the borrower; - Avoiding contradictions between domestic legislation and WB procedures, by clear and straightforward stipulations in the Loan agreement and other official documents; - Establishing responsibilities and a simple operation mechanism for the components involved in project coordination and implementation. Note: The GOR's contribution included complete lists of all sites and institutions rehabilitated or equipped under the project, of all equipment procured under the project by different types of procurement method and beneficiary organization, and of other goods and services procured under the project. This data is available on project files. 60 ADDENDUM ACTIVITIES DEVELOPED WITHIN THE RO-3409 PROGRAM. FINANCED BY THE WORLD BANK DURING JANUARY 1S' - JUNE 30TH 1999 Consequently to the request on behalf of the Ministry of Health in March 1999 World Bank has approved the extension of all the activities in the project up to June 1999, keeping in mind the fact that the necessity of allocations and of spending all the amounts of the project mutually convened with the Ministry of Finance. Thus up June 30th contracts were concluded as a result of international shopping procedures (31 de international shopping activities resulting in 78 de contracts) and as a result of supplementing some contracts in development for the national emergency system (ambulance purchasing, biochemical analyzers, transportation ventilators). The system for the health information system was also supplemented. The afferent payments of the concluded contracts in 1999, 19 mil. USD, were completed up to August 1 5th, the date limit for the closure of the accounts of the project. 61 MAP SECTION 22' , 24 _ 26° ew 28IBR 28710 UKRAINE --- 7oS'ov tc R O M A N I A )T. \ / BO Kolomyyc . \2 / \ ELEVATIONS ABOVE 500 METERS O SELECTED CITES H U N G A R Y ~~~~~~~~~~~~~~~~~~~~~~~OSNI ŽD COLUNTY (JUDET) CENTERS YA Mae LT G BOo A NATIONAL CAPITAL Map -rsign 1tT osfY. ,EXPRESSWAY MALJOR ROADS 1. OTHER PORTS 1 A S ~ ~ ~ ~ 2~-RIVERS fe Tuolopect yc,,, - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~COUNTY [JUDET) BOUNDARIES --INTERNATIONAL BOUNDARIES a ndo nyo the rmf ormationshownon [ ' =- . - . / t 4 zELEQRM^yQ . -i \ ,7 t~~~~~~~~O so i c ~ suchbaDndorirr. 220 t 240 B U L G A R I A - 26 28° ToVarno~~~~~~~~~~~~~~~~~~~~~0 2 5 10 TooIpe ) A 9 d ~~ARAD MILES lk.~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~A' ('-16fi~ ~ ~ ~ ~ ~~~~~x- T. ~ ~ ~~~~~~~~~~~~1PAHV~\ -'' BAL ~ T EA FED. REP,OF MEHEDINT]RE f) ~ ~ YUGOSLAVIA cmi- CA~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~4' e8di oneat (Serbia/Montenegro) C1D al-koo 12 P~ LA C:N Thio -op -ne p,.dccd by the 4 4'- -4-Mop Deoign Unit of The W-odd8-k. sE/ Th. -ocd -e , xnon eonntio -nd ELEfe o mto eonoo ll~o coop do Ot fplyPI, o- the p.a ofT.V Th.e World SanS GDoop, -ny o,dpo-t a-y -ndrsm-t -racpt-nc of ZieT.iV.o -oh boud-ies, B UL GA R IA 22' 24' 26' 28' MAY2 195