ICRR 13209 Report Number : ICRR13209 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/21/2009 PROJ ID : P050657 Appraisal Actual Project Name : Uttar Pradesh Health US$M ): Project Costs (US$M): 127.58 98.4 Systems Development Project Country : India Loan/ US$M): Loan /Credit (US$M): 110 84.83 Sector Board : HE Cofinancing (US$M): US$M ): Sector (s): Health (87%) Sub-national government administration (13%) Theme (s): Other communicable diseases (29% - P) Health system performance (29% - P) Population and reproductive health (28% - P) Gender (14% - S) L/C Number : C3338 Board Approval Date : 04/25/2000 Partners involved : Closing Date : 12/31/2005 12/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Alemayehu A. Ambel Ridley Nelson Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD) and the Loan Agreement (LA) state that the objective of the project was "to establish a well managed health system in Uttar Pradesh which delivers more effective services through, policy reform, institutional and human resources development, and investment in health services ." Due to the bifurcation of the state into Uttar Pradesh and Uttarakhand, separate LAs were signed with each state . However, the project development objectives (PDOs) remained unchanged. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project had the following two components, and each with three sub -components(PAD, pp. 28-30): Component -1. Policy Reforms, Management Development and Institutional Strengthening (Appraisal Estimate US$ 30.23 million, Actual n.a.). 1.A. Developing a Strategic Management Capacity (Appraisal Estimate US$0.7 million; Actual n.a. (see Comment under Section 2d)). To enhance competence for formulating and reviewing strategies within the Department of Medical, Health and Family Welfare (DOMHFW). This sub-component would finance equipment, furniture, supplies, studies, consultants, operations and maintenance, and salaries of incremental staff . 1.B. Strengthening Performance, Accountability and Efficiency (Appraisal Estimate US$13.2 million; Actual n.a. ). To build a management culture oriented towards operational effectiveness and efficiency, and holding managers accountable for results. This sub-component would finance equipment, supplies, training, studies, consultants, and operational expenses. 1.C.Building Implementation Capacity (Appraisal Estimate US$16.3 million; Actual n.a.). To establish and build the capacity of a dedicated Project Management Unit (PMU) so that it could implement and coordinate projects of this size and scope. Under this sub-component, the project would finance civil works, furniture, equipment, training, consultants, and operational expenses and salaries of incremental staff . Component -2. Improving Health Service Quality & Access (Appraisal Estimate US$ 97.35 million; Actual n.a.) 2.A. Improving Clinical Service Quality (Appraisal Estimate US$75.9 million; Actual n.a.) To improve the quality of clinical practices by improving the skills of health providers, introducing quality assurance (QA) systems, and utilizing appropriate facilities, equipment, drugs and medical supplies . 2.B. Improving Public Health Service Quality (Appraisal Estimate US$18.5 million; Actual n.a.) To re-establish a disease surveillance and control system, including for food safety, health sector waste management and health communications. 2.C. Improving Access to Health Services (Appraisal Estimate US$3.0 million; Actual n.a.) To engage with Non-Government Organizations (NGOs) to provide health care and facilitate health seeking behavior where there are few private sector alternatives particularly in remote areas . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Change in administrative structure and adjustment to the project : the ICR reports that due to the bifurcation of the state into Uttar Pradesh and Uttarakhand in November 2000, separate Project Implementation Plans (PIP) were developed within the original framework and without changing the components . Total costs were also divided in 86:14 ratio between the states of Uttar Pradesh and Uttarakhand (ICR, p. 3). Project Closing Extensions : There were two extensions: The original closing date of 12/31/2005 was first moved to December 2007, and later changed to 12/31/2008. The ICR indicates that extensions were forced by unsatisfa ctory performance on both implementation progress and achievement of development objectives . Actual Project Costs and Financing : The ICR does not report total actual project costs (loan amount and Borrowers contribution) by component. Actual estimates by component show only the loan amount (see ICR Annex 1, p. 24). In the meeting between IEG and the project task team, the project task team said that the Borrower's contribution by component was not available. 3. Relevance of Objectives & Design: Relevance of Objectives :(Rating: High). The objective of the project underscores effective services in low income states through policy reforms and institutional and human resources development as well as investment in health services. Citing documents available at the time of project preparation such as the Country Assistance Strategy (CAS January 1998) and also the Board's Discussion of the CAS (February 1999), the PAD reported that high priority was placed on provision of assistance to reforming states including Uttar Pradesh . The objectives of the project were also indicated as priorities in the 2004 and 2008 CAS documents. Relevance of Design : (Rating: Modest). The PDO focused on policy reforms and human resource development . Consequently, the key performance indicators were on utilization of essential health services and changes in public sector expenditures on health . These indicators were used to monitor implementation progress . The PAD envisaged outcomes for the poor in beneficiary states . However, the ICR reports that the project did not measure outcome by poverty status due to problems in proper identification of the Below Poverty Line population . Likewise, although the PAD sought an increase in referrals at Community Health Center (CHC) and Combined/District Hospitals from Primary Health Center (PHC), the ICR indicates that this was never reported on in either state . In addition, based on a 2002 review of the Bank's experience on health systems projects, the ICR identifies some design issues relevant to this project. These include, (i) development objectives did not explicitly identify health status improvements, (ii) facilities in India do not routinely record the patient's poverty status, (iii) there was no comprehensive vision to involve all health sector partners, and (iv) standardized designs became less relevant owing to "ever-widening" differences among states that prompted the project to move away from "facility-based" or "input oriented" to "population-based" or "output oriented" approach. Therefore, based on High relevance of objectives and Modest relevance of design, the overall relevance of objectives and design is rated Substantial . 4. Achievement of Objectives (Efficacy): The objective was "to establish a well managed health system in Uttar Pradesh which delivers more effective services through, policy reform, institutional and human resources development, and investment in health services ." (Rating : Substantial ) Output: Output The following output indicators address predominantly the first question in the objectives : The extent to which a well managed health system was established . Policy Reform, Management Development and Institutional Strengthening : 1. Project management capacity: relevant committees, boards and forums were established, organizational development studies were carried out, and HR management policies were reviewed . 2. Performance, accountability and efficiency : health management information systems were strengthened in both states, systems were developed for personnel information, hospital information, and beneficiary tracking in U ttar Pradesh, quality improvement teams were set up at all project facilities and strengthened in Uttarakhand . Health Service Quality and Access : 1. Clinical service : facilities garages, waste storage rooms, regional training centers were built, renovated or refurbished; equipment was repaired and maintained . 2. Public health service : In Uttarakhand, Biomedical Waste Management plan was developed for 34 health facilities; and in relation to this, Standard Operating Procedures (SOP) and awareness materials were developed and disseminated. On-site training was also given at each project facility . In addition, deep burial pits were constructed at project sites, and equipment and supplies were procured and supplied to all facilities . In Uttarakhand, Information Education and Communication (IEC) activities on safe drinking water were also implemented under the project. 3. Innovative schemes for disadvantaged population : In Uttar Pradesh, 169 NGOs provided basic health care services in 28 districts serving a total of about 1.25 million population. In Uttarakhand there were 3 NGOs providing the same services to a population of about 150,000. Other activities carried out in Uttarakhand include the procurement of 13 mobile vans and integration of a family of Indian systems of medicines ( AYUSH) with modern systems of medicine as well as undertaking preparations for maternal death audits and village health planning activities. Outcome : The following outcome indicators address predominantly the second question in the objectives : The extent to which the system delivered more effective services . However, although the ICR reports some adjusted figures to account for other factors that might have affected the indictors, it is not always possible to fully attribute the changes in the indicators to the project. 1. Outpatient department (OPD) attendance: On the performance of OPD the ICR notes an increase in the utilization of these health services in Uttar Pradesh . Similarly, the achievement in Uttarakhand was close to the target ( 92 percent). For women,in Uttar Pradesh, OPD attendance increased by 16.5 percent over the project period. In Uttarakhand, OPD attendance by women increased from 47.8 percent to 54.7 percent, which is a little over the project's target of 52 percent. 2. In-patients/ hospital admissions: The achievements in Uttar Pradesh did not meet the targets . However, there was a steadily increase in hospital admissions in Uttarakhand . The ICR reports that the shortfall in Uttar Pradesh could be attributed to lack of doctors in the facilities . In-patient services utilization by women increased only by about 1 percent from the baseline value of 57.2 percent to 58.4 percent. However, the target at 50 percent was lower than the baseline value . In Uttarakhand, hospital admissions for women increased to 68 percent while the target was 64 percent. 3. Institutional deliveries: In Uttar Pradesh, institutional deliveries increased at CHCs and PHCs over the project period. However it declined at district hospitals [District Hospital- Female DH (F)]. In Uttarakhand institutional deliveries surpassed the project's target . 4. Bed occupancy rate (BOR): BOR declined in all project facilities in Uttar Pradesh between the period 2000-2005. It increased in Uttarakhand at DH level but declined at CHCs level . 5. Health sector spending: (This is treated as an outcome because there was an objective to increase investment ) In Uttar Pradesh, there were fluctuations in the allocation between 2000 and 2005. More recently, however, the Government of Uttar Pradesh maintained its commitment to increasing the allocations in the health sector . By 2008-09, the health sector spending was 6.4 percent of total government expenditure which is close to the project's target of 6.5 percent. The budget allocated to non -wage expenditures was 28.7 percent of total health sector expenditure compared to the target of 30 percent. In Uttarakhand total expenditure on health increased from 4.4 percent of the total government expenditure in 2005-06 to 5.1 percent in 2007-2008. The estimate for 2008-09 was 4.9 percent. The ICR does not provide information on the performance of non -wage expenditure in the health sector in Uttarakhand . 5. Efficiency (not applicable to DPLs): There was no ERR estimated at appraisal and no aggregate ERR estimated in the ICR . On other measures of project efficiency, the ICR reported the results of the satisfaction survey that shows a significant increase from 1.97 to 2.56 on a 4-point scale in Uttar Pradesh while it remained static at 2.7 in Uttarakhand. However, implementation efficiency was affected by delays in project start -up, high staff turn-over and poor performance on key activities including civil works and procurement of goods and services . As a result, two extensions were made to the project resulting in a 60 percent delay in the overall project completion date . It is also reported that "operations and maintenance of equipment was not up to the mark in some places . In a number of locations some of the equipment were found to be nonfunctional or were operating without any appropriate maintenance system ". (ICR, P. 9). Procurement of goods accounted for more than 40 percent of the total project cost . Therefore, any inefficiency in equipment operation and maintenance would substantially undermine the overall efficiency of the project . In addition,in terms of growth rates of outpatient and inpatient services and institutional deliveries, the comparison between project and non-project facilities shows that performance was mixed, and in some cases, it was lower in project than non-project facilities. Finally, the inability of the Borrower or the Bank to show the allocation of actual costs by component suggests considerable uncertainty about allocative efficiency . Therefore, for these reasons, efficiency is rated Modest. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on Substantial relevance, Substantial efficacy, and Modest efficiency, the outcome of the project is rated Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR mentions some developments that would minimize risk to development outcome . For example, government allocations to health increased in recent years . In addition, the Government’s in the two states prepared a follow-on operation for Bank funding building on the investments made under the current project . However, the project involved a number of activities and institutions including NGOs . Therefore, maintaining the services would require more financial resources and coordination among various agencies . For these reasons, the risk to development outcome is rated Moderate. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: at - Entry (Rating: Moderately Satisfactory): Project design incorporated lessons from similar projects Quality -at- implemented in India as well as results of a social assessment and a patient satisfaction survey . Most of the targets were also based on baseline information and several technical studies were undertaken at the time of project preparation. However, some of the targets were unrealistically lower than the baseline values (ICR, p. 14). In addition, the project envisioned to measure poverty status of patients for service utilization without making sure if such indicators were routinely recorded by project area facilities or without introducing an M&E system to capture such information. Quality of Supervision (Rating: Moderately Satisfactory): Supervision missions were regular; aide memoirs documented detailed progress reports; and detailed benchmarks were agreed with the PMU to ensure that key actions were taken expeditiously . In addition, the Bank team facilitated biannual workshops, where various states shared their experiences in implementation, problem solving and innovation . However, the performance of the project could have been better if the Bank seized the opportunity of reprogramming the project and phasing implementation based on the mid-term review (MTR) recommendations. Therefore, based on Moderately Satisfactory performance of Quality-at-Entry and Moderately Satisfactory performance of Quality of Supervision, the overall Bank performance is rated as Moderately Satisfactory at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government performance (Rating: Moderately Satisfactory): The ICR mentions there was a long delay in project preparation activities due to national elections and inability to obtain clearance from the Government of India. However, during the course of implementation the Government was engaged in policy dialogue with the Bank. Implementing Agency performance (Rating: Moderately Satisfactory): Initial implementation delays were observed in both states. By the time of the midterm review the disbursement of the credit amount was only 23 percent and the project was two years behind schedule (ICR, p.5) due to several preparatory problems and frequent turnover of key officials (ICR, p.20). However, implementation performance greatly improved in both states after the MTR (ICR, p.20 &21). Therefore, based on Moderately Satisfactory Government performance and Moderately Satisfactory Implementing Agency Performance, the overall performance of the Borrower is rated as Moderately Satisfactory . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E design (Rating: Modest). There were some shortcomings in formulating key indicators . Some were poorly designed and difficult to measure . As discussed in Section 3 of this ICR review, the M&E system did not allow measuring changes in health status improvements . M&E implementation (Rating: Substantial ). In Uttar Pradesh, the ICR notes the successful implementation of the Health Management Information System (HMIS) component by mentioning the Government's commitment to owning the system and the Bank's testimonies that were reflected in aide memoirs . A comprehensive Personnel Information System (PIS) was developed and information on over 10,000 Health Directorate personnel was entered in the system. As well, health and hospital management system, and drug inventory system were expected to be uploaded. Other M&E tools developed in Uttar Pradesh include modules for national programs (RCH, TB, Immunization, Malaria, Blindness, Leprosy and Integrated Disease Control ). In Uttarakhand, data for allopathic doctors and paramedical staff were entered into the PIS and HMIS were rolled out at the district level and there were plans to extend the system to the block and sub -block levels. M&E utilization (Rating: Negligible to Low). The ICR does not provide evidence on the use of the results of M&E in guiding policy formulation or project preparation . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Procurement and Financial Management : There were procurement delays at the start of project implementation and hence assistance to the Project Management Unit (PMU) was provided by external procurement agencies . However, this was later handled by the PMU . On financial management, the ICR notes delays in external audits for some years and the inadequacy of internal audits . However, no information is supplied whether there were any audit qualifications. Quality Review : The ICR team commissioned assessment of the quality of construction and maintenance of health facilities as well as adequacy and maintenance of equipment (ICR, p. 9). The assessment shows satisfactory quality for civil works in both states, but in Uttarakhand for equipment only . In Uttar Pradesh, the team reported problems in quality of equipment maintenance and operations . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The ICR identifies 16 lessons on project design and implementation as well as on future operations that this ICR review concurs with. The following two are highlighted (with some adjustment from the ICR wording). 1. Project designs need to identify and include measurable health status indicators in addition to intermediate outcome indicators. 2. Developing systems that provide quality care to the poorest is a time -consuming process. It involves investment in strategic planning, human resources, and partnerships with all available stakeholders in the health sector. 14. Assessment Recommended? Yes No Why? The project is a demonstration of the shift in emphasis from building infrastructure and facility to systems development for effective health service delivery . Therefore, further lessons from the implementation of the project would inform the Bank's future engagement in promoting health systems development . In particular, efficiency warrants more exploration. 15. Comments on Quality of ICR: The quality of the ICR is rated, on balance, Satisfactory . The ICR provides sufficient information on project outputs, outcomes, risks, as well as Bank and Borrower performances . It attaches the summary of the Borrowers ICR . In addition, the ICR provides an assessment of quality of civil works and equipment operations and maintenance . Therefore, the available information was adequate to determine the ratings . The quality review section of the ICR is innovative and provides useful information on project implementation . On the other hand, the ICR presented the report by comparing and contrasting performance in the two states . This, however, was not the intent of the project and the format made the report somewhat difficult to follow and to link to outcomes . The ICR could have presented separate summaries for each state and then provided an overall composite rating with more weight, due to the scale, given to Uttar Pradesh. Other issue in the ICR is inconsistent reporting of actual project cost estimates (ICR, Annex 1, p. 24). Finally, a small point, the cost column heading of the "Staff Time and Cost" table on page 32 reads "US$ Thousands" while the actual figures seem to be reported . a.Quality of ICR Rating : Satisfactory