ICRR 13522 Report Number : ICRR13522 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/16/2011 PROJ ID : P074053 Appraisal Actual Project Name : Health Transition US$M ): Project Costs (US$M): 88.2 52.5 Project Country : Turkey Loan /Credit (US$M): Loan/ US$M ): 71.1 50.2 Sector Board : HE US$M): Cofinancing (US$M ): Sector (s): Central government administration (60%) Health (30%) Compulsory health finance (10%) Theme (s): Health system performance (50% - P) Administrative and civil service reform (25% - S) Injuries and non-communicable diseases (25% - S) L/C Number : L4737 Board Approval Date : 05/20/2004 Partners involved : Closing Date : 12/31/2007 12/31/2009 Evaluator : Panel Reviewer : Group Manager : Group : Susan A. Stout Robert Mark Lacey IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: This project was the first of a two phased APL designed to support the Government of Turkey ’s Program for Health Transformation (PHT), the objectives of which are to “ improve the governance, efficiency, user and provider satisfaction, and long-term fiscal sustainability of the health care system in Turkey.� The PAD states that the Project Development Objective is : "to assist the Government to strengthen the institutional environment for the implementation of its Program for Transformation of Health that will improve system stewardship, streamline financing and service delivery, and build the institutional capacity to extend health insurance coverage to he whole population in a fiscally sustainable manner .� The same language is used in the Loan Agreement. Restructuring. During the first years of implementation, it became clear that legal and political difficulties with the passage of reform laws meant that two key outcomes (MOH no longer involved in service provision, and hospitals operating under a new autonomous model ) would take much longer to achieve than originally anticipated. The PDO was accordingly revised to " 1) to help expand the capacity of the MOH and the Social Security Institute (SSI) to formulate and effectively implement health policies, health sector regulatory mechanisms and health insurance functions; and 2) support the implementation of critical health service delivery reforms (family medicine and hospital autonomy) with the objective of improving access, efficiency, quality and fiscal sustainability of the health sector ". The rationale for the change was articulated in a formal project paper which was reviewed and approved by the Board. The legal agreements were also amended to reflect the change in the project development objective, and were consistent with the objectives as stated in the restructured project . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 06/21/2007 c. Components (or Key Conditions in the case of DPLs, as appropriate): The original project included six major component, as follows :* : A. Restructuring the MOH for Effective Stewardship (appraisal : Euro 9.6/ USD 12. 12.19 million; actual 16/USD 9.65 million ) aimed at supporting the transition of MOH from a provider of services to a Euro 7.16/ policy maker and regulator of service provision, while retaining key public health functions . The component included: (i) restructuring of the MOH, (ii) establishment of a National Pharmaceutical and Medical Devices Agency (NPMDA), (iii) quality assurance and accreditation of health facilities including establishment of a new quality unit in the MOH and accreditation of the National Hygiene Laboratory, and (iv) establishing a Monitoring and Evaluation (M&E) framework for the Heath Transition Program (HTP) and a M&E unit within the MOH. B. Building Capacity of the Health Insurance and Social Security Institutions (appraisal : Euro 3.0/USD 83/USD 2.32 million ) supported the establishment of a single health insurance 3.8 million; actual Euro 1.83/ fund through the consolidation of the four existing health insurance schemes, and expanding its reach to an additional estimated 22 million citizens who were not fully covered . C. Reorganizing the Delivery of Healthcare Services (appraisal : Euro 15. 67/USD 19. 15.67/ 19.90 million; actual 21/USD 9.16 million ) included: (i) introduction of family medicine as an organizational model for Euro 7.21/ the provision of outpatient or primary health care services, (ii) harmonization of MOH and SSK (Sosyal Sigortalar Kurumu, - Social Security Institute)hospitals towards greater autonomy, (iii) developing an effective patient referral system to reinforce system hierarchy, and (iv) strengthening population health programs, including disease surveillance, maternal and child health, prevention and control of communicable and non-communicable diseases (NCDs). The reorganization will be implemented in a phased manner, with pilots in large cities during Phase I, to be rolled out nationwide during the second phase. D. Strengthening Human Resources Capacity (appraisal : Euro 8.72/ 11.07 million; actual Euro 72/USD 11. 75/ USD 3.49 million ) to develop the skills and competence of the health work force, in line with the 2.75/ changing role of the MOH and its affiliated institutions . It had two sub-components: (i) health and social security human resources policy and planning, and (ii) strengthening the School of Public Health (SPH), to become a center of excellence in advocacy, training and research for the MOH 15.13/ E. Building Infrastructure for Health and Social Security (appraisal : Euro 15. 19.15 million; 13/USD 19. 13.06/ actual Euro 13. 16.58 million ) supported the development of national standards in line with the 06/USD 16. realignment of institutional roles and responsibilities in both MOH and MOLSS . This was to be done in a phased manner, first with the development of standards and the establishment of a records and information network in the social security system, and thereafter between the social security system and the hospitals. It had two sub-components: (i) building the health information system, and (ii) building the social security information system . 60/USD 3.30 million; actual Euro 2.55/ F. Project Management (appraisal : Euro 2.60/ 55/ USD 3.23 million ) Component supported project coordination between the two ministries as well as project implementation in each ministry, and an oversight mechanism for overall project guidance and policy support for both the HTP and the project itself. It funded project management, including financial management and procurement, as well as technical assistance needed to support the Project Management Support Unit (PMSU). G. Completion of Five Regional Training Centers ( Euro 2.5/US$3 US$ 3.17 million ) On July 17, 2005, a component was added to complete the construction of five Regional Training centers (reallocation of 2.5 million € from unallocated), which were initiated under the Second Health Project but not completed prior to the closing date. As this reallocation did not entail a change to the PDO or key outcome indicators, this change did not require Board approval. Restructured Components About 28% of the loan was disbursed at the time of restructuring. The restructured project realigned project components to reflect the change in the PDO, focusing the project effort on capacity building and making changes to clarify accountability for implementation, as follows : A,. A,. Restructuring the MOH for Effective Stewardship (appraisal : Euro 9.6/US$12 12.19 million; after US$ 12. 16.4/US$20 restructuring : Euro 16. 20.83 million ) . Consistent with the shift to emphasizing capacity building US$ 20. for the Ministry of Health, the component was adjusted to focus on strengthening the Strategic Planning and Policy Development Unit of the MOH and the development of a performance management framework for autonomous health facilities. This included the establishment of quality assurance and accreditation policies and procedures and establishing monitoring and evaluation capacity with a particular focus on supporting the health reforms set out in GoT's Health Transformation Strategy . Consultancies (4) were completed. B) Building Capacity of the Health Insurance and Social Security Institutions Euro 3.0/US$ 3.81 (Euro3 US$ 11. 11.30 million ) was modified to focus expanding the capacity of million; after restructuring : Euro 8.9/US$11 the SSI to formulate and effectively implement health insurance functions, including through training, IT software and hardware improvements to its information infrastructure and a public information campaign . C. Reorganizing the Delivery of Health Care Services (appraisal : Euro 15. US$ 19. 15.2/US$1919.3 million; after 11.6/US$14 restructuring : Euro 11. 14.7 million ) was modified to merge the small scale activities on improving US$ 14. the referral system (originally .29 million) with activities to expand the introduction of family medicine model and for the MOH to implement a systematic framework for stake holder consultation and public information campaigns in support of health sector reform . D. Strengthening Human Resources Capacity (appraisal : Euro 8.72/ US$ 11. 72/US$11 US$ 6.6 11.1 million; Euro 5.2/US$6 million after restructuring ) was revised to clarify accountability with the RSHI (National Hygiene Laboratory) which was originally supported through component A; and to strengthen the school of public health. Activities designed to support improvements in work -force policy and planning were moved to component A (for MOH) and component B (for SSI). Project activities supported public health functions and human resource training capacities. The project provided technical assistance, consultancies and training for doctors. 15.13/ E. Building Infrastructure for Health and Social Security (appraisal : Euro 15. 19.30 million ) 13/USD 19. The original component sought to support the development of an information system that would integrate MOH and SSI information systems was redesigned into two sub-coponents to be implemented by MOH and SSI separately. F. Project Management and G. Completion of Five Regional Training Centers . These components were unchanged. * The ICR presents all costs in Euro, this review has converted all costs to US dollars, using the USD/Euro conversion rate as of July 2010, the date of completion of the ICR (1 euro = US$1.26). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project costs Project costs were broadly as anticipated at appraisal, with some modifications through the implementation period. In June 2009 a loan amendment cancelled US$4.3 million because the GoT and Bank decided that activities which were not sufficiently advanced as this APL 1 reached its closing date, would be best pursued through the planned second APL, which is now under implementation . In October 2009 a further US$3.2 million was cancelled from the SSI portion of the loan because the envisioned activities could not be completed in the available time frame . At closure, 91.3% of the loan proceeds had been disbursed. Financing and Borrower Contribution A loan amendment in March 2006, reduced the Borrower's contribution for consultants and allocated US$57.6 million to goods so that the MOH could procure IT hardware and software required by the restructuring, and for support to the NPDMDA ((National Pharmaceutical and Medical Devices Agency). Allocations for hospital quality accreditation and hospital autonomy were substantially reduced . Funding for the monitoring and evaluation unit were substantially increased (by 260%). The initially small amount allocated for grants to NGOs was reduced from US$ 2.0 million to US$0.25 million because mechanisms to assess and accredit NGOs for participation were inadequate . Dates The project was formally restructured in June 2007, nine months after the mid term review.The project’s closing date was extended by two years to December 31, 2009. 3. Relevance of Objectives & Design: Relevance of original objectives is rated substantial . The original objectives were consistent with the high priority on health sector reforms sought by the Government elected in 2002. The PHT was incorporated into the Government's Urgent Action Plan which was closely monitored at the most senior levels of the new administration . The objectives of the project were also consistent with the CAS of 2003, which pointed to the need for systemic institutional change in service delivery, including the health sector, an objective which continues in the current CAS . Relevance of original Design is rated substantial . The choice of an APL was sound given the government's ambitious vision for health sector reform . The design, though complex, was geared to the improvement of the MOH's policy and technical leadership, while at the same time helping it to evolve away from its direct service provision role, in order to encourage greater stewardship over the sector as a whole. The design built on lessons from analytic work and two earlier investment projects that suggested that both the Bank and Government needed to work on systematic health reform instead of incremental improvements or health infrastructure, and that, therefore, capacity building and technical assistance would have more value than a focus on health infrastructure . Integration of information systems and insurance system coverage in and between MOH and SSI would help rationalize and extend coverage and improve service delivery, particularly through the extension and diffusion of the family medicine model which would in turn improve access, quality and efficiency of health services. Strengthening the use of information in both MOH and SSI would improve sectoral oversight and management and, together with effective project management of technical assistance and consultancy services to be provided through the project, would strengthen the institutional environment . The original results framework focused on tracking institutional and organizational changes . high. Relevance of the revised objectives is rated high During the mid-term review (MTR) in September 2006, the Bank’s evaluation indicated that the Government’s long term PHT objectives remained relevant, but that the path to reforms had taken a different turn, and certain important reforms were on an accelerated timetable while others were moving more slowly. In particular, it became clear that key legislative changes which would facilitate certain areas of the reform process, particularly passage of the Hospital Autonomy Act and the law restructuring the MOH by the National Assembly would require much more time than the original design anticipated . During the MTR, the Bank and the Borrower came to an agreement that it would be desirable to change the PDO to focus project efforts on improving MOH capacities to play a stronger stewardship role while the political and legislative process developed . The restructured approach sought to support the Borrower's view that capacity building within the MOH and SSI would encourage further progress toward the PHT’s goals and enable it to proceed with the large scale introduction and diffusion of the family medicine model, Relevance of the revised design is rated as substantial . The restructured design drew on extensive dialogue with the MOH to agree on revised objectives, continuing with the plan to phase Bank support to the PHT through a series of APLs, and included an implicit results chain wherein improved capacities within MOH and SSI would prepare these institutions to take on the larger stewardship roles anticipated in the overall vision for health sector reform . 4. Achievement of Objectives (Efficacy): Outcomes Before Restructuring : Objective a: Strengthen institutional environment for the implementation of its Program for Transformation of Health that will improve system stewardship The objective was not achieved because the MOH was not restructured and remains a service provider . With the transfer of SSI hospitals to the MOH, the Ministry is more engaged with direct service provision than it was at the start of the project. Outcome : Negligible Objective b . Strengthen the institutional environment [to] to] streamline financing and build capacity to extend health insurance of the whole population The integration of SSI hospitals (less efficient than MOH hospitals ) with the management of MOH hospitals at project start was achieved through government decision, although as noted in the ICR, this decision was not explicitly linked to the project . The project provided policy support for reforms that have dramatically increased health insurance coverage, which is now at 87% of the population. The project supported the PHT, to which the improvement can be attributed, by contributing to a stronger policy environment through consultancies and technical assistance . Outcome : Substantial Objective c . Streamline Service Delivery (family medicine and hospital autonomy ). This objective remained largely the same after restructuring as in the original design and is discussed under “After Restructuring� below (Objective c.). After Restructuring : Objective a . Expand the capacity of MOH to formulate and effectively implement health policies and health sector regulatory mechanisms . Outputs Strengthened Strategic Planning and Policy Development Unit at MOH, which prepared a draft Strategic Plan articulating the MOH’s vision for the health sector A draft law on pharmaceutical and medical supply regulation was prepared . Policies and procedures on supervision were prepared and disseminated . Training, and preparation meetings on Service Quality Standards, work to develop of indicators of clinical quality and methods for the evaluation of service standards at the hospital level were provided to 130 persons from 24 provinces, Program and Performance Based Budgeting Training were conducted for 140 participants from across MOH. A Performance Assessment Tool was designed and introduced through training of staff in 14 hospitals With the addition of performance payment, the monthly salary increased for health workers . For specialists the increase was 77%. Set up IT system and software, training program design and implementation, and built capacity to introduce prospective payment based contracts for all MOH hospitals . Central Hospital Appointment System was designed and piloted in Kayseri province . The capacity of the School of Public Health was strengthened. The School completed the preparation and distribution of Poisoning Diagnosis and Treatment Guidelines, and Guidelines for Primary Care in 2007. The Public Health School provides training in health management, health economics and finance, rational drug use, flagship training program, and epidemiological intelligence service . Research and development capacity of the Refik Saydam National Hygiene Center Presidency (RSHI) was built to provide training to strengthen oversight and accreditation of public health laboratories.The RHSI P3 laboratory was built and accredited. Outcome The new human resources (HR) strategy has been submitted for clearance to the Government, but not yet adopted. The MOH has the capacity to pilot-test and then introduced Performance Based Supplementary Payment (PBSP) system in 850 MOH hospitals. This is one of the first efforts to experiment with performance based payments at a significant scale in the public service . MOH prepared Performance Based budgets for 2009 and 2010. The MOH has the capacity to guide the introduction of performance payment and mechanisms, regulations governing the assessment of hospital performance, and development of a system for assessing and regulating hospital quality and performance . The ratio of full time to part time physicians increased from 42% to 75%, which is related to better compensation. Patient satisfaction in public hospitals increased from 38% in 2004 to 66.6% in 2009 (the target was 65%) Outcome : Substantial Objective b . Expand the capacity of the SSI to formulate and effectively implement health insurance functions . Outputs Developed training program plan for 25,000 SSI staff. Project management training for 21 senior administrative staff. Project financed 65% of the infrastructure and technology for a training center with e -learning capabilities which capacity to train 30,000 people per day. Set-up web-based accounting system and a system to enable the MOH’s legal department to track legal records. IT governance and security infrastructure were evaluated and new IT security certification regulations were developed. Outcome The SSI withdrew from direct service provision when authority over SSK hospitals was transferred from the MOLSS to MOH. Contracts based on prospective provider payments and global budgets were in place for all MOH hospitals. A modified prospective payment system* was established for all private and University hospitals. Insurance coverage among vulnerable groups increased from 24% in 2003 to 82% in 2008 (surpassing the target of 50%). The ICR notes (p 21) that enrollment increase is largely attributable to the Green Card Scheme which was financed directly by Government, and is therefore not directly attributable to the project. Outcome : Substantial *A prospective payment system is one in which the health care institution receives a set amount of money for each episode of care provided to a patient, regardless of the actual amount of care used. Objective c . Support the implementation of critical health service delivery reforms (family medicine and hospital autonomy ) with the objective of improving access, efficiency, quality and fiscal sustainability in the health sector . Outputs A Law enabling piloting and expansion of Family Medicine (FM) model was passed, creating the necessary legal framework for introducing family medicine with capitation payments . Overall, 6,557 physicians were trained in FM. In total, 120 PHC physicians have been certified in Family Medicine. 6 "Provincial Evaluation Meetings" were held where local health administrators , MOH staff and staff from provinces not yet implementing the family medicine model were able to share knowledge of how to introduce and improve the effectiveness of the FM model . A Family Medicine Advisory Council, a Family Medicine Scientific Consultation Board and a Public Health Scientific Consultant Board were established A law was submitted to Parliament to pilot-test hospital autonomy. Although full autonomy of hospitals was not achieved, hospitals achieved increased control over finance and administration at the facility level . Outcome Delivery Reforms (family medicine and hospital autonomy): Achievements in the diffusion and adoption of the family medicine model were greater than originally anticipated. Family medicine was introduced through a pilot in Duzce in 2005 with expectations that the lessons from the piloting effort would be extended in a follow -on APL. In fact, by the end of APL1 the family medicine model had been introduced in 33 provinces and had enrolled 23.1 million individuals, a major achievement . Available evidence (from the Turkey Household Based Survey and MOH service delivery data ) indicates that the expansion resulted in increased patient and provider satisfaction (in 2004, average patient satisfaction across all provinces was 69%, by 2009 patients in family medicine provinces had an 86% satisfaction rate, while non-family medicine provinces had a 75% satisfaction rate). Provider satisfaction in 9 provinces increased from 50% to 58% after the introduction of family medicine. Although full hospital autonomy, which requires legislation, was not achieved, MOH and SSI did make changes that gave hospitals and clinic directors more authority over procurement and investment decisions. Access: The number of examinations in public hospitals increased from 135 million (2004) to 210 million (2008), which may be related to more full-time equivalent (FTE) physicians working in hospitals and increased insurance coverage. Population enrollment in FM practices increased from 0% nationwide to 90-100% in the 33 provinces (surpassing the target of 90% in 15 provinces). Primary health care (PHC) visits increased from 2 visits per capita per year (2004) to 2.75 visits (2009). . Efficiency: A mandatory referral chain was not established so referral rate data are not consistent and reliable, making it impossible to assess reductions in the rate of referrals from PHCs to hospitals . The referral rate from secondary to higher-cost tertiary hospitals decreased from 30% in 2004 to 19% in 2009 (almost meeting the target of 20%). The ICR (p 25) reports a decrease in the average length of hospitals stay from 5.3 days in 2002 to 4.2 days in 2008. Quality: The ICR does not contain evidence on improved health service quality . However, data provided in an article in the British Medical Journal * indicates significant enhancements in health outcomes following the implementation of the project-supported PHT. For example, between 2000 and 2008, life expectancy rose from 70 to 73 years; infant mortality was halved from 38 to 19.4 per 1000 live births; and maternal mortality fell even more dramatically from 70 to 19.8 per 1000 live births. Under both latter indicators, Turkey moved over the same period from worst to best position compared to five other middle-income countries.** Fiscal sustainability : The ICR does not provide information on fiscal sustainability of the sector . However, the British Medical Journal article already cited points out that the strong political commitment to the PHT is reflected in the substantial increase in the share of the Government ’s budget allocated to health care (from 11.5% in 2000 to 16.5% in 2008), especially against the background of rapidly growing total public expenditures (5% per year). Good overall economic performance has helped to make the rising health expenditures sustainable – while total health expenditures rose by 40% in real terms between 2003 and 2008, their share of GPD rose only from 6.2% to 6.4% of GDP. Outcome : Substantial Enis Baris, Salih Mollahaliloglu, Sabahattin Aydin: “Health Care in Turkey: From Laggard to Leader,� British Medical Journal,12 March 2011, Volume 342, pages 579-582. ** Argentina, Brazil, Mexico, Russia and Thailand. Contribution to Achievement of Program Objectives Given the substantial achievement of the PDOs following restructuring, the project also made a substantial contribution towards fulfillment of the APL program goals of improving the governance, efficiency, user and provider satisfaction, and long-term fiscal sustainability of the health care system in Turkey. 5. Efficiency (not applicable to DPLs): The PAD (Annex 9) does not quantify the expected efficiency and fiscal impact of the health reform supported by the project. However, it estimates for each component the direction of future government and insurance expenditure if the component is implemented or not, and the financial incentive set to different stakeholders and how they might react to the reforms. The ICR does not include an efficiency analysis because of the difficulties to attribute efficiency changes to a policy reform project . However, both the PAD and the ICR identify the expected health sector efficiency gains through project-support. These include: The introduction of the PBSP (Performance Based Salary Payment), with hospital efficiency enhancements, led to reductions in the average length of stay for inpatients in all hospitals, which are costly. The family medicine model and complementary reforms led to substantial increases in the number of consultations per physician which are less costly than hospital admissions (from below 2,400 in 2004 to above 3,500 in 2007). Incentives for vaccination and other child health measures led to an increase in full vaccination coverage from 78% in 2000 to 96% in 2008, which are efficient measures to prevent costly diseases and epidemics among children. A shift towards spending on preventive and primary care . Project resources were managed with sufficient flexibility through the APL to enable the MOH to efficiently re-allocate funds to sectoral reform objectives . Resources for monitoring and evaluation were substantially increased (by 260%) to coordinate the efficient introduction and diffusion of the family medicine model. As part of the restructuring, loan funds were cancelled if their implementation was not deemed timely or necessary anymore. Efficiency is rated Substantial ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Efficacy of the two original objectives dropped at restructuring were, respectively, negligible and substantial. Relevance of objectives and design were substantial . The outcome of the project before restructuring is rated Moderately Satisfactory . Relevance of objectives of the restructured project is high, and both design relevance and efficacy are substantial. Efficiency is rated as substantial . Outcome is therefore Satisfactory . According to the guidelines for restructured projects, the overall outcome rating is based on a weighting according to the percentage of the loan (28% in this case) that was disbursed at the time of the restructuring. Outcome is thus rated Satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to the development outcomes achieved through the project is rated Moderate . The project primarily financed capacity building and institutional strengthening, which contributed to a larger health sector transformation program that continues to be championed and implemented by the Government . Project outcomes are likely to be sustained because they were developed directly with the concerned units of the line agency rather than by separate project entities . Capacity of the SSI was strengthened to a lesser extent, and frequent and ongoing changes in the SSI may weaken the progress on overall health sector reform. However, this risk is well identified by both Bank and Borrower, and is being addressed in the follow up APL, the Health Transformation and Social Security Reform Project (HTSSP), which seeks to build on the gains achieved through the PHT and this project . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at Entry . The project grew out of experience with two earlier investments in the health sector and an intensive program of analytic work on public expenditure and institutional governance issues in the health sector.* Lessons learned through these experiences were among the inputs to a strong push for health sector reform by the government elected in 2002. They also informed the Health Minister’s strong commitment to health sector reform. The Bank took this confluence of experience and government commitment as an opportunity to contribute to the accomplishment of the resulting Program of Health Transition. The decision to channel this support through an adaptable lending instrument was appropriate since responding to these opportunities would require flexibility . At the same time, the MOH saw the development of the project as a platform for launching the government wide program. However, the Bank team in the initial design did not adequately recognize potential obstacles to the rapid passage of legislation . The underlying assumption that reforms would flow from legislative changes to enable restructuring of the MOH and achievement of hospital autonomy proved optimistic. The Bank team chose the APL as a flexible instrument to mitigate this risk by using a phased approach to support reforms (PAD p. 7). Although the results framework for the project is distinct from that for the Government’s overall PHT, the initial formulation of project development objectives proved to be overly optimistic . *Turkey: Reforming the Health Sector for Improved Access and Efficiency (in Two Volumes) Volume 1: Main Report. Report No. 24358-TU March 2003, Human Development Sector Unit Europe and Central Asia Region, World Bank Supervision . The Bank team provided strong technical support during implementation and applied due diligence to fiduciary matters both prior to and following restructuring . The ICR points to difficulties in the quality of the Bank/Borrower relationship in the first two years of implementation (though it does not specify what they were), which, as reported by the current Task Team, were resolved through changes on both Government and Bank teams prior to the mid term review. The Bank took full advantage of the mid term review to assess opportunities and to identify ways to respond to changes in the political economy of health sector reform . The restructuring of the project, with accompanying changes in implementation arrangements, was opportune and apposite, and enabled the effective achievement of the restructured objectives. Missions were frequent and strongly staffed with sector expertise. The team was, however, unable to resolve M&E weaknesses (see Section 10 below). at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government : The Government provided a strong enabling environment, as evidenced by the passage of several important laws, including the family medicine pilot law, legislation to monitor quality as part of the PBSP system, establishment of SSI, and establishment of UHI among others. Its support was evident prior to restructuring and continues today, as MOH is currently working (in the context of the second APL) to expand the legislative framework for reforms. Implementing Agencies . Implementation arrangements in the initial design created unnecessary organizational boundaries between those responsible for ministry -wide reform efforts and those responsible for management of specific project resources . The Ministry of Health was highly committed to the overall health sector reform process, and to project objectives, and made progress on several elements of reform, including particularly the roll out of the family medicine model during the first years of the project. Implementation progress accelerated substantially when project management and budget authority were transferred directly to the line agencies . When some legislative goals could not be met (such as mandating the restructuring of the MOH), the leadership of the MOH focused on incremental reforms such as the establishment of strategic planning and improving quality regulation . One shortcoming was that M&E weaknesses were not rectified. The SSI's performance was less consistent than that of the MOH, reflecting uncertainty concerning the organizational design prior to restructuring and frequent changes in leadership in the later years of the project . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Design: Design The project’s monitoring and evaluation framework was not well articulated . Some of the indicators were not clearly defined and baselines for several were unclear . The selection of outcome indicators were of little value in reporting on progress in achieving institutional change and sectoral reforms which are typically more incremental in nature . Output indicators were also not well specified . The framework did not clearly set out roles and responsibilities for monitoring and evaluation . Implementation : Two shortcomings with M&E emerged before restructuring. First, the outcome indicators that were monitored were slightly different from those in the "Performance Monitoring Indicators" table of the PAD and Loan Agreement, and this discrepancy was not resolved. Second, at restructuring, the number and form of outcome indicators were adjusted and new indicators to better measure the progress of the family medicine model, hospital performance and capacity measure were introduced. However, the restructuring effort did little to clarify roles and responsibilities for M&E, which led to tensions between the Bank and Borrower on the appropriate use of M&E resources (which were increased significantly through the restructuring ). The Borrower decided to utilize these resources to pay for “field coordinators,� who, in addition to their M&E duties, were charged with providing �coaching support� on the introduction of key reform initiatives (including family medicine ). According to the Borrower’s ICR (ICR, page 47), “The field coordinators who were recruited as national consultants observed the countrywide implementation of the activities planned within the scope of the PHT, the problems of the regions on site, and evaluated them with the local managers and reported them rapidly and informed the senior management of MOH.� Nonetheless, “the field coordinators were not in the original Project design and did not provide a systematic M&E function � (ICR, page 10). Utilization : The ICR reports (page 12) that the M&E coordinating unit “did not systematically and objectively collect data,� and that “systematic M&E and its use in informing policy remained a critical issue recognized by both the Bank and the Government.� M&E is a “key priority� of the APL2 project. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): According to the ICR, the project did not trigger safeguard issues, there were no new civil works or any expected negative environmental impacts. The project is was rated category 'C' for purposes of OP 4.01. Fiduciary The ICR reports that procurement practices were generally satisfactory after the PMSU and PIU were fully staffed during the first year of implementation. In addition to prior reviews, annual procurement post-review missions were conducted during the project implementation . The ICR reports that no major deviations from the Bank’s procurement guidelines were identified during the ex -post reviews. According to the ICR, financial Management (FM) practices throughout implementation were generally satisfactory, and both implementing agencies, MOH and SSI, provided timely and satisfactory financial monitoring reports. Independent auditors’ reports were unqualified and received in a timely manner . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately The original design was excessively Satisfactory ambitious in scope and complexity . Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Several lessons can be drawn from the experience of this project : Strong Government ownership is key to project success, particularly when significant institutional reforms are at stake. Successfully supporting health sector reform requires long term engagement using a full range of instruments -- in this case a combination of analytic work focused on systemic and institutional constraints to service delivery and a flexible lending instrument . When the Bank seeks to support a process of sectoral reform, might it need to consider a programmatic set of results indicators which would be adjusted as experience develops, enabling a more adaptive approach to focusing dialogue and assistance . The ICR notes that client engagement and relationships are key to success, and suggests that the Bank should work to ensure consistency in client engagement skills across teams . In this case, implementation performance improved significantly following the MTR and a change of task teams . Flexibility and an appropriate response to evidence that the initial design was unrealistic were important to the achievement of the project’s goals. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR, provides in depth coverage and analysis of key health sector issues . However, it could have been more candid and specific on disagreements between Bank and Borrower, especially those prior to restructuring. Some indicators (health quality and fiscal sustainability ) are not reported on, and no economic or financial analysis of the project was attempted . The ICR will, nevertheless, be of interest to other teams working to promote health sector reform. a.Quality of ICR Rating : Satisfactory