Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) Report Number : ICRR0021154 1. Project Data Project ID Project Name P071160 IN: Karnataka Health System Dev Country Practice Area(Lead) Additional Financing India Health, Nutrition & Population P126846,P130395 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-42290,IDA-51610,TF-99435 31-Mar-2012 206,480,000.00 Bank Approval Date Closing Date (Actual) 22-Aug-2006 31-Mar-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 141,830,000.00 400,000.00 Revised Commitment 211,372,759.63 219,125.75 Actual 213,528,396.59 219,125.75 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives According to the Development Credit Agreement (page 22), the original project objective was as follows: •To assist Karnataka in improving the utilization of essential curative and public health services, particularly in the underserved areas and amongst vulnerable groups. The objective was similarly articulated in the Project Appraisal Document (PAD, page 8): Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) •To increase utilization of essential health services (curative, preventative, and public health), particularly in underserved areas and among vulnerable groups, to accelerate achievement of the health-related MDGs. "Underserved areas" are where essential services are generally not available; "vulnerable groups" are those who are more at risk of morbidity and mortality, such as mothers and small children (PAD, page 8) In December, 2012, Additional Financing (AF) was approved, and the project objective was revised as follows (August, 2012 Project Paper, page 6): • To improve health service delivery, public-private collaboration, and financing, particularly for the benefit of underserved and vulnerable groups in Karnataka. In March, 2016, the project underwent another Level 2 restructuring to modify several key indicators and outcome targets. Because of the revision of objectives, and later of outcome targets, a split rating is performed here. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 26-Dec-2012 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components 1. Strengthening Existing Government Health Programs Towards the Achievement of More Effective and Equitable Delivery of Services (Appraisal: US$ 115.07 million; Actual: US$ 131.4 million): This component aimed to support institutional development for public and private service providers and the delivery of essential health services. The first subcomponent (Organizational Development) was to support activities to strengthen the stewardship role of the public sector (specifically, the Department of Health and Family Welfare) including a move towards results-based management. Activities were to include: preparation of Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) Service Improvement Plans for health facilities at the district level; establishment of a Health Planning Unit; and strengthening of the accreditation process. The second subcomponent (Improving Primary and Secondary Care Services Effectiveness) was to support activities to increase delivery of essential health services by reimbursing the government for health expenditures upon achievement of milestones. 2. Innovations in Service Delivery and Health Financing (Appraisal: US$ 67.5 million; Actual: US$ 159.2 million): The first subcomponent (Innovations in Service Delivery Linked to Need and Performance) was to support investments in primary care and public health services focused on using an innovative approach. Activities included: establishment of a Service Improvement Challenge Fund for initiatives that scaled up the quantity and quality of services; establishment of a Public Health Competitive Fund for initiatives that scaled up public health, non-curative care featuring community involvement. The second subcomponent (Innovations in Health Financing) was to support reductions in financial barriers to accessing health services. Activities included a health insurance pilot program, including a feasibility study, and financial support for premium subsidies for poor families. 3. Project Management, Monitoring and Evaluation (Appraisal: US$ 15.16 million; Actual: US$ 23.6 million): This component was to support the establishment of a project management unit and State Health Information Center, as well as a dedicated health management information system. Under the multiple project restructurings, the following revisions were made to the project components: Component 1: Training and technical assistance for private sector providers were dropped in order to strengthen focus on developing capacity to manage public-private partnerships. The accreditation system for private sector providers was also dropped. Training of local-level managers of primary health centers, and a strategy to improve quality assurance at those centers, were scaled up. The AF further increased focus on quality at the primary health center level, with activities on capacity building for local staff (including for road safety and non-communicable diseases) as well as development of quality assurance tools and quality standards for community health centers and district hospitals. Component 2: Additional support was provided to upgrade health centers in “backwards blocks.” The citizen help desk and mobile clinic activities were dropped (due to procurement difficulties). The AF provided for two pilot programs: community sensitization and screening for selected non-communicable diseases; and a situation and needs analysis of two safety demonstration corridors (which included ambulance services and emergency health care facilities). Lastly, while the analysis of a national health insurance scheme was dropped, the AF still supported institutional development of the health insurance program for the poor (including financing of claims for hospital services). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost • The appraised project cost was US$ 206.5 million. The actual project cost was US$ 314.0 million. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) •Component allocations were adjusted throughout the project restructurings, with the most significant being an increase in Component 2 from the original allocation of US$ 60.7 million to US$ 115.9 million, due to activities under the AF. Financing • The original International Development Association Credit was appraised at US$ 141.8 million, which increased to US$ 211.8 million with the AF. US$211.8 million was actually disbursed. • The project design included the concept of results-based financing, through the use of milestone- based disbursements for the "Innovations in Health Financing" activity. Borrower Contribution • The appraised Borrower contribution was US$ 64.8 million. The actual Borrower contribution was US$ 100.3 million. • The ICR (page 26) reports that implementation delays (and therefore low disbursement levels) were interpreted by the State Treasury as low absorption capacity, which led to insufficient counterpart budget allocations at the start of fiscal years in the early project period. Eventually, budget allocations were increased by mid-year through supplementary budget. Dates • December 2010: The project underwent a Level 2 restructuring to modify the results framework (several key outcome indicators were shifted to intermediate outcome indicators, but the project's scope and outcome targets did not change). • February 2012: The project closing date was extended from March 2012 to September 2012 to allow time for appraisal of the Government's request for AF. • July 2012: The project closing date was extended from September 2012 to March 2013 to prepare the AF. • December 2012: AF of US$ 70.0 million was approved. At this time, the objectives were also revised, along with relevant modifications to the project components and the results framework. Lastly, the project closing date was extended from March 2013 to March 2016. • March 2016: The project underwent a Level 2 restructuring to modify the results framework (targets for key outcome indicators were revised). The project closing date was also extended from March 2016 to March 2017. 3. Relevance of Objectives Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) Rationale At the time of project appraisal in 2006, the Karnataka State's priority health outcomes (such as the infant mortality rate and stunting) had seen significant improvement and were among the best in the entire country. However, there remained significant gaps, including percentage of children fully immunized (60% in 1998), neonatal mortality (which comprised three-fourths of infant mortality, which stood at 55/1000 live births), and skilled birth attendance (only 51% of deliveries took place in institutions in 1998, even lower for the poor and socially disadvantaged groups). Disparities also persisted according to income level, including the under-five mortality rate and full childhood vaccination. Other system-wide weaknesses included weak management capacity (including the lack of a results focus), lack of focus on essential health services, inadequate health financing strategies, and lack of optimal use of private sector services. Both the original and revised objectives were highly consistent with the Bank's Country Partnership Strategy for FY13-17, which identified strengthened public and private health delivery systems as a key outcome, as well as the corporate goals of eliminating extreme poverty and promoting shared prosperity. (A new Country Partnership Framework (FY 18-22) is under development.) The country's 12th Five-Year Plan for 2013- 2017 also highlighted reductions in maternal and child mortality as its main health-related targets. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To assist Karnataka in improving the utilization of essential curative and public health services, particularly in the underserved areas and amongst vulnerable groups. Rationale The theory of change was sound, with support for improving health facilities and services, as well as training of health personnel, likely to lead to improved health service utilization, particularly among vulnerable groups as the project geographically targeted underserved areas and provided specific interventions for women (maternal health) and children (immunizations), as well as for the poor (health insurance). There was a strong focus on institutional development along with an emphasis on innovative approaches to health service delivery. Attribution is somewhat unclear, as there was a significant increase in public spending during the project period (including on health schemes specifically targeting pregnant women). The ICR (page 15) reported that health expenditure in the state more than doubled between 2004-05 (US$ 232.4 million) and 2010- 2011 (US$ 587.9 million), and per capita public expenditure on health grew from US$ 4.20 to US$ 9.67. In addition, project financing for Component 1 supported the government's broader health program. The ICR Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) (page 18) did suggest that "specific areas of support provided by the project played a catalytic role in shifting state government’s policy towards improving delivery of essential health services in the lagging districts; strengthening institutional mechanisms, efficiency and transparency of the health insurance agency; and piloting innovations in prevention and control of non-communicable diseases (NCDs) and promotion of road safety as well as promoting health financing with a clear focus on reducing the financial burden on the poor and promoting partnership with the private sector." There were 446.0 million direct beneficiaries (original and revised target: 306.0 million). Outputs • Construction/renovation and equipping of 392 health facilities (target: 375). Most health facilities constructed were in the seven underserved districts targeted by the project (no specific quantitative evidence provided). 47% of government Primary Health Centers now provide round-the-clock services. • Training of 91,415 health personnel (target: 25,500) on organizational development, quality assurance, and healthcare waste management. • Development of three Food Laboratories, including testing equipment and an operations manual. The project team reported that registration and licensing of about 60% of eligible food business operators was completed, and an online registration and complaints systems were functioning. Outcomes • Deliveries taking place in a health facility increased from 65% in 2005/6 to 94% in 2016, achieving the target of 90%. 5.8 million deliveries were attended by a skilled assistant (target: 5.2 million), of which 61.4% took place in a public facility. • Deliveries taking place in a health facility among the poor increased from 37% to 77%, surpassing the target of 55%. • The percent of the population receiving outpatient care in government facilities in seven less-developed districts increased from 46% in 2006 to 78% in 2017, achieving the target of 78%. • The percentage of children completing full immunizations increased from 55% in 2005/06 to 78% in 2009 (note: this key outcome indicator result was not reported in Annex 1, but was noted in the main text of the ICR (page 15)). It is not clear which project outputs contributed to this outcome. There were no data reported on the following, some of which were identified as "Key PDO Indicators" in the PAD (page 31) but later shifted to intermediate outcome indicators at the December 2010 project restructuring: •Percent of mothers and newborns visited within two weeks of delivery by a trained community health worker. Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) • Percent inpatient attendance. • Annual prevalence rate of malaria. • Percent of women receiving information on HIV/AIDS during pre/post natal or family planning visit. • Percent tuberculosis cure rate. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective To improve health service delivery, particularly for the benefit of underserved and vulnerable groups in Karnataka (ORIGINAL outcome targets). Revised Rationale Improved "health service delivery" is considered here to be substantively the same as "utilization" and therefore will be assessed under the same theory of change. "Service delivery" may be construed as broader than "utilization" to include also quality of services, but it is assumed that issues of quality are also reflect in levels of utilization, as beneficiaries will be more inclined to use services that are known to be of higher quality. Due to several developments under the country's National Rural Health Mission Program during the project period (i.e. several originally planned project activities were now funded by that Program), the project's focus shifted towards targeting key systems-level strengthening activities and reforms. The ICR (page 16) stated that the AF specifically focused on: (i) activities that could not be easily done through other mechanisms such as quality assurance and accreditation, (ii) activities that would benefit from technical engagement with the Bank by setting specific milestones for institutional development, verification, transparency, and grievance redress in the health insurance scheme, and (iii) pilot programs for NCDs and Road Safety, which if successful could be scaled up using Government funds. Outputs In addition to the outputs listed above: • Deployment of 42 specialist doctors in 7 underserved districts. • Development of training materials for NCD prevention and management. • Conducting of a pilot program for NCD screening and management, including for cervical cancer, diabetes, and hypertension. • Introduction of information technology initiatives, particularly to facilitate procurement. Outcomes Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) In addition to the outcomes listed above: • 27,156 women were screened for cervical cancer in pilot districts, falling short of the target of 60,000. • 60% of the eligible population, or 319,012 people, were screened for diabetes in the two pilot districts, and 91% of those diagnosed were put on treatment. There was no target set. • 77% of the eligible population, or 409,055 people, were screened for hypertension in the two pilot districts, and 95% of those diagnosed were put on treatment. There was no target set. Revised Rating Substantial PHINNERREVISEDTBL Objective 1 Revision 2 Revised Objective To improve health service delivery, particularly for the benefit of underserved and vulnerable groups in Karnataka (REVISED outcome targets). Revised Rationale Outcomes: • 27,156 women were screened for cervical cancer in pilot districts, surpassing the revised target of 1,000. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective [This Objective field is intentionally left blank.] Rationale [See below for the next objective, which was added at the time of Additional Financing.] Rating Not Rated/Not Applicable PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) To improve public-private collaboration and financing, particularly for the benefit of underserved and vulnerable groups in Karnataka. Revised Rationale This objective was added at the AF. Outputs • Development of guidelines and standards for trauma care services, including provision of ambulance services by private providers. • Establishment of 124 mobile health clinics (through the private providers) for unreached and underserved areas. • Outsourcing of diagnostic services for NCDs to private medical colleges. • Training of village health and sanitation committees, under the Public Health Competitive Fund. • Institutional capacity building for the state insurance agency, including implementation of a milestone- based disbursement approach and communication strategy to sensitize the poor. • Training for state and district officials on public-private partnerships. Outcomes • The number of road traffic accident victims transported by the state ambulance system, but run by a private operator, increased by 183%, surpassing the target of 10%. • 766 public and private facilities were empaneled to provide care to road accident victims under government programs, surpassing the target of 150. • 153,237 claims were paid by the health insurance pilot program benefitting Below Poverty Line households, surpassing the original target of 45,000 and the revised target of 150,000. • 123,462 claims were paid to private sector hospitals by the government health insurance program supported by the project, achieving the target of 120,000. • 256 private hospitals empaneled by the state insurance agency submitted at least one hundred claims, surpassing the target of 50. Achievement is rated High due to surpassing of several targets in establishing public-private collaborations and financing of insurance claims (which targeted the poor). Revised Rating High PHOVRLEFFRATTBL Rationale Under the original objective, the overall efficacy rating is Substantial. There were two revised objectives, one of which is rated Substantial and other rated High, and therefore the Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) overall efficacy rating under the revised objectives is also Substantial. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency The analysis in the PAD (Annex 9) was primarily an economic analysis of the health sector as a whole, not specifically on efficiency in the use of project resources. The discussion centered on the extent to which government expenditures are pro-poor (allocative efficiency and equity) and the extent to which the government is responding productively to market failure. There was no project-specific economic analysis such as internal rate of return (IRR) or net present value (NPV). The analysis in the ICR (Annex 4) also did not include an IRR or NPV estimate, but it focused on the cost- effectiveness of specific project interventions, given the project design’s strong focus on primary health, public health, and NCDs. The ICR suggested that preventative services were delivered at the primary care level at a lower cost than at specialized health facilities (no specific data or evidence provided), and also that the expansion of patient volume at project-support facilities and the increase in outpatient utilization indicate project efficiency. Specifically, the use of mobile clinics improved access to health care at likely lower cost: 107 mobile units served over 7.2 million patients, thus a per capita cost of 116 rupees. NCD screening was provided to over 27,000 women, with an estimated cost-effectiveness ratio of $10/life saved. Institutional deliveries also resulted in over 2,700 maternal deaths averted, thus a cost-effectiveness ratio of $699/life saved. These are considered highly cost-effective given the Karnataka gross domestic product per capita of approximately $3200. In addition, the project used the existing government system for civil works, with the engineering division of the Department of Health and Family Welfare executing all works, The introduction of information technology- based project management also likely contributed to project efficiency. Other elements of project design that likely contributed to efficiency were the targeting of underserved and vulnerable groups and focus on public health initiatives. Finally, the ICR suggested that project funding was significantly leveraged against government spending, as available government resources for health "increased substantially over the project cycle," and that the project did not represent more than 7% of total annual government health spending in the state (ICR, page 51). Although there were implementation challenges that likely diminished the efficiency in the use of project resources - high turnover of officials (leadership at state and project administrator level) and procurement delays, frequent number of project restructurings, and the use of integrated implementation arrangements in which Ministry staff had limited time to focus on project activities (hence a tradeoff between efficiency and sustainability) - project outputs were still delivered in full by project closing. Efficiency Rating Substantial Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the project objective is High. Achievement under the original objective to increase utilization of essential health services (curative, preventative, and public health), particularly in underserved areas and among vulnerable groups, and under the essentially identical revised objective to improve health service delivery, is rated Substantial due to evidence of achievements in skilled deliveries, outpatient utilization, and childhood immunizations, as well as NCD screening, road accident care, and health insurance support for the poor. Achievement under the second objective (added at the AF), to improve public-private collaboration and financing, particularly for the benefit of underserved and vulnerable groups, is rated High due to surpassing of several targets in establishing public-private collaborations and financing of insurance claims (which targeted the poor). Efficiency is rated Substantial due to the strong focus of the project design on cost-effective interventions and supporting data on cost-effectiveness for two specific interventions. Outcome The overall outcome rating for the project under both the original and revised objectives is Satisfactory, reflecting High relevance of design, Substantial efficacy, and Substantial efficiency. The overall outcome rating is therefore also Satisfactory, reflecting only minor shortcomings in the project's preparation and implementation. a. Outcome Rating Satisfactory Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) 7. Risk to Development Outcome The government has integrated a number of the project activities into its own sector programs, hence increasing the likely sustainability of the interventions. Development of institutional capacity was significant, as evidenced by the meeting of accreditation and quality standards by numerous health facilities, improved performance and financial sustainability of the Karnataka Drugs Logistics and Warehousing Society, information technology networking of blood banks and project management offices, and the absorption of special cells on public-private partnerships, NCDs, food safety, and road safety into the broader Ministry structure. The significant increase in government expenditure on health also signals the continuing government commitment to the project objectives. 8. Assessment of Bank Performance a. Quality-at-Entry The project design was overall sound, with interventions selectively focused on essential curative and public health interventions to improve maternal and child health and address communicable diseases. The project design also explicitly targeted underserved and poor areas. The ICR (page 7) noted that the project was designed "to better coordinate and maximize the impact of externally assisted projects and vertical national programs sponsored by the Government of India." The project was preceded by grant- funded technical assistance programs on improving the performance of the public sector, enhancing the involvement of the private sector in the delivery of essential services, and developing a health financing strategy. These three programs helped to inform the design of the project. A key lesson from the prior Karnataka Health System Development project (1996-2004) pointed to the need for incentives to improve performance and increase expenditures on health, and hence the project featured milestone- based approach to ensure implementation of key reforms. The risk assessment rated the overall risk as Moderate, identifying numerous potential risks in institutional capacity and ownership. These were addressed with appropriate mitigation measures. There was an underlying assumption that implementation would rely on the existing institutional Ministry structures; while this increased the likelihood of sustainable outcomes, the increasing focus on innovation through multiple restructurings represented a departure from the conventional implementation of the government program, and hence there was less ownership of these innovative elements and therefore delays. Some managers and staff considered the project-supported innvoations as an additional burden (ICR, pages 26-27). There were also shortcomings in the results framework, with multiple "performance indicator" lists and lack of baseline and target figures. Quality-at-Entry Rating Moderately Satisfactory Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) b. Quality of supervision The task team, which was field based, provided regular implementation support through the lengthy nine- year implementation period. Supervision ratings were candid, including the acknowledgement of slow project progress at certain intervals which was reflected in downgraded ratings. Project restructurings, although numerous, were well used to refine the project design as well as the results framework. The Bank team took into account the changing context - namely the high achievement in skilled birth attendance and child immunizations but persistent socio-economic and geographic disparities; changing morbidity profiles (increasing NCD morbidity and road traffic injuries); and increased government expenditures on health - and made adjustments to the project design accordingly at the time of AF. The results framework was clarified and better aligned with the objectives. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The original results framework was complex, with several shortcomings as noted in the ICR (page 27-28): numerous indicators, differing lists of "priority" indicators, and lack of baseline data and targets. For example, a table of "PDO Indicators" (PAD, page 31) listed nine indicators, but did not provide baseline or target figures; a separate table of "Priority Performance Indicators" (PAD, page 35) listed 16 indicators including baseline and targets, but there was minimal overlap with the list of "PDO Indicators." Data collection arrangements included drawing from existing information systems for vertical health programs on immunization, tuberculosis, and HIV/AIDS. These data were intended to be disaggregated by geography and vulnerability, though this was not specifically reflected in the project results framework. Household level data was also to be collected to verify results for the targeted beneficiaries. The project M&E design also included evaluative studies, including on the health insurance pilot, health care waste management, and quality of civil works. Lastly, the M&E design included the establishment of a State Health Informatics Center, and a new health management information system, to ensure reliable and timely data. b. M&E Implementation The various project restructurings clarified the results framework, notably by including specific indicators for Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) targeted vulnerable groups. The revisions were important for increased alignment with the objectives, including at the time of the AF. Information was collected as planned from multiple sources such as household surveys, administrative records, information systems, and facility surveys. Evaluative studies were carried out as planned. c. M&E Utilization The ICR reported that evaluations of pilot programs were conducted, with findings used to inform policy decisions. Also, M&E data was used to support the milestone-based approach to link disbursements with key institutional and health financing reforms. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as a Category B project due to potential adverse impact on the environment, triggering safeguard policies on Environmental Assessment (OP 4.01) and Pest Management (OP 4.09). An Environmental Assessment was conducted and determined that the project would only create minor, site- specific, and reversible environmental impacts. An environmental management plan was prepared to ensure mitigation of negative impacts. The PAD (page 109) discussed matters relating to the safeguard policy on Indigenous Peoples (OD 4.20), given the project's focus on disadvantaged populations, though it does not explicitly note that the policy was triggered. A Social Assessment was conducted, and the findings contributed to a Vulnerable Communities Action Plan to facilitate access to health services for these groups. The ICR reported that project supervision documents indicated overall satisfactory compliance with environmental and safeguard policies. The project supported capacity building to over 9,000 government health facilities on health care waste management; the ICR (page 24) also noted in particular that the project supported the State in "setting high standards for environment safeguards," for example, by achieving the status of "mercury-free" medical equipment in public health facilities, implementing a low cost bio-medical liquid disinfection initiative, and developing a state policy to give priority to environmental safeguards in handling biomedical waste and identifying specifications for medical equipment. Minor shortcomings in environmental safeguards compliance were reported as delays in implementation of the environment action plan and a break in the contracts of district consultants for health care waste management. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) b. Fiduciary Compliance Financial management: The ICR (page 31) reported that financial management (FM) was mostly rated in the satisfactory range throughout the project period, with the exception of the early years due to delayed finalization of the project's FM manual and internal audit arrangements. These issues were resolved, with FM reports and audit reports submitted in a timely manner from FY2010 onwards. The Mid-Term Review identified continued risks of weak FM capacity and low effectiveness of internal auditing. A consequent action plan including hiring an FM consultant. No major problems in FM were thereafter reported. Procurement: Procurement performance was also rated in the satisfactory range for almost the entire project period. Procurement for the traditionally financed activities was in accordance with Bank guidelines. Procurement for the programmatic activities was in accordance with the state's procurement reform action, consequent to the Bank's state procurement assessment report of 2001. A Detailed Implementation Review of the India Health Sector, conducted by the government, had addressed operational concerns relating to procurement of health sector goods and services, and required a progress review every six months. Minor shortcomings in procurement performance included occasional slow pace of decision making and procurement processes, delayed release of payments, and lax monitoring and control of contracts. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Quality at Entry is rated Moderately Satisfactory due to Bank Performance Satisfactory Moderately Satisfactory shortcomings in the results framework. Quality of M&E Substantial Substantial --- Quality of ICR Modest --- 12. Lessons Lessons drawn from the ICR, adapted by IEG: Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review IN: Karnataka Health System Dev (P071160) • Disbursement-linked indicators can be an effective intermediate step towards results-based financing, especially by catalyzing key policy reforms. In the case of this project, the disbursement-linked indicators approach opened the path to shift away from input-based financing. Technical assistance to build central and local government capacity to implement such an approach was critical to its success. • Well-designed pilot programs responding to emerging situations can be an effective tool for introducing the government to new initiatives and assessing practicalities of such initiatives. In the case of this project, the new initiatives on non-communicable diseases and road safety included evaluations to demonstrate what works and launch new policy dialogue. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided robust quantitative evidence of achievements; however, the analysis of the results chain was less satisfactory. The Efficacy section of the ICR was presented as a bullet point list of outcomes, lacking a comprehensive discussion of how project outputs led to the project's achievements. In addition, this section did not adhere to ICR guidelines for restructured projects, as it did not present a separate analysis for each project restructuring (those triggered by revisions in the objectives and of outcome targets). The Efficiency section would have been bolstered by an IRR or NPV analysis. The Safeguards section lacked an explicit discussion of which safeguard policies were triggered and whether the project was in compliance with each one. a. Quality of ICR Rating Modest Page 16 of 16