ICRR 13707 Report Number : ICRR13707 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/28/2013 Country : Romania Project ID : P100470 Appraisal Actual Project Name : Avian Influenza US$M ): Project Costs (US$M): 42.4 27.63 Control & Human Pandemic Preparedness & Response Project L/C Number : L4839 Loan/ US$M ): Loan /Credit (US$M): 37.7 24.2 Sector Board : Health, Nutrition and US$M): Cofinancing (US$M ): 4.67 3.43 Population Cofinanciers : Board Approval Date : 09/08/2006 Closing Date : 12/31/2009 12/31/2010 Sector (s): Health (50%); General public administration sector (35%); Agricultural extension and research (13%); Solid waste management (2%) Theme (s): Rural services and infrastructure (25% - P); Other communicable diseases (25% - P); Natural disaster management (24% - P); Pollution management and environmental health (13% - S); Health system performance (13% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Stephen Hutton John R. Heath Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The objective of the project at appraisal was to : "reduce the threat posed to humans and the poultry sector in Romania by HPAI [Highly Pathogenic Avian Influenza ] and other zoonoses, and to support the preparation for, control of, and response to influenza pandemics and other infectious disease emergencies in humans ." Loan Agreement Schedule 1, Technical Annex Page 13) After restructuring, the revised project objective was to "assist the Borrower in building its capacity to respond to potential future infections of humans by HPAI and other zoonoses, and preparing for, controlling and responding to influenza pandemics and other infectious disease emergencies in humans ." (Project Paper July 2009, page 4) b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 07/28/2009 c. Components: The original project had three components : A) Animal Health (Appraisal estimate EUR 14.3 million, Actual EUR 0.22 million), including: i) Strengthening the institutional framework through technical assistance for veterinary services, ii ) Strengthening disease surveillance, Diagnostic Capacity and Research through establishment of a integrated national animal disease system, purchase laboratory equipment and consumables, and training, iii ) Strengthening HPAI Control Programs through technical assistance and purchase of equipment for culling and disposal, and iv ) Improving biosecurity in poultry production through awareness and communications campaigns . This component was dropped after restructuring . B) Human Health (Appraisal estimate EUR 19.8 million, actual EUR 19.06 million), including: i) Enhancing Public Health Program Planning and Coordination through support for research and conferences, ii ) Strengthening of National Public Health Surveillance Systems by upgrading the national influenza laboratory to biosecurity level 3, and iii) Strengthening Health System Response Capacity by strengthening isolation and case management capacity in hospitals and upgrading influenza vaccine production . C) Public Awareness, Communications and Implementation Support (Appraisal estimate EUR 3.46 million, actual EUR 0.71 million), including: i) Public Awareness and Communications, and ii ) Implementation Support, Monitoring and Evaluation. Animal health-related communication and awareness activities were dropped after project restructuring . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The original project cost was Euro 37.5 million, of which IBRD was to finance EUR 29.6 million. By the restructuring date in July 2009, Euro 2.6 million had been disbursed, EUR 3.9 million was reallocated from Animal Health to Human Health, and EUR 7.3 million was cancelled. A total of EUR 18 million was disbursed by project closure, and the remaining EUR 4.3 was cancelled. Thus, 14 percent of the funds were disbursed before restructuring. 3. Relevance of Objectives & Design: a. Relevance of Objectives: The project was motivated by global concerns about the risk from avian influenza, rather than the existing country strategies. The project was broadly consistent with the 2009-13 Country Partnership Strategy which emphasized flexibility and responsiveness to evolving circumstances, but avian influenza, zoonoses, and pandemic preparedness were not specifically mentioned in any of the pillars of the strategy . Avian influenza posed a plausible threat to Romania . There had already been outbreaks of avian influenza among poultry in 2005 and 2006, leading to a million birds dying or culled at an economic cost of roughly Euro 90 million. While these outbreaks had already been contained when the project was designed, there was a plausible risk of further outbreaks; the Romanian Danube Delta is on major bird migratory routes, and biosecurity conditions among backyard poultry systems were poor . An additional outbreak occurred in March 2010, demonstrating that the threat remained relevant throughout the project . Avian influenza remained a relevant threat to humans and the poultry sector at closure, with a continued threat of outbreaks from infection by wild birds. The relevance of the original objective is rated Substantial. After restructuring, the project removed the emphasis on the poultry sector and an emphasis on threat reduction, and added a focus on building response capacity with an emphasis on infections in humans . The change was made because of poor implementation progress by the National Sanitary, Veterinary and Food Safety Authority . However, the shift away from addressing the poultry sector also reduced the relevance of the project for dealing with avian influenza concerns in humans . The revised objectives no longer addressed risk reduction measures that could reduce the risk of influenza outbreaks occurring in humans by reducing its occurrence in poultry , they only addressed building capacity to respond to outbreak in humans once they occur . As such, they removed a crucial component of an effective avian influenza strategy . The relevance of the revised objective is rated Modest. b. Relevance of Design: The original design drew on the standard Global Program on Avian Influenza design, including both animal health and human health investments and capacity building, combined with a public awareness campaign . Most elements of this design would have contributed to achieving the project objectives : support for biosecurity improvements on farms would reduce the risk of avian influenza outbreaks, surveillance and diagnostic improvements would help to identify any outbreaks, strengthening control programs would contain outbreaks and reduce the spread of disease . Improvements to human health laboratories would help to identify potential outbreaks or a pandemic, and improved vaccine production would help to mitigate a pandemic . However, while there was some room for improvement in veterinary service capacity, these gaps were not an emergency. There was already considerable capacity for emergency response and control of avian influenza, and activities supported under the project did not justify an emergency intervention . The relevance of the original design to the objectives is rate substantial . After restructuring, the project narrowed its scope considerably . The removal of animal health activities led to a disconnect between the project design and the objectives . It is not possible to address avian influenza and other zoonotic risks among humans appropriately without any significant effort to reduce the risk of outbreaks among animals. Consequently the relevance of the revised design to the objectives is rated modest. 4. Achievement of Objectives (Efficacy): Objective 1: Reduce the threat posed to humans and the poultry sector by HPAI and other zoonoses (original objective only) Cancellation of the animal health component and much of the communication component meant that the project did not make any contribution to this objective . Achievement of this objective is rated negligible. Objective 2: Preparing for, controlling and responding to influenza pandemics and other infectious disease emergencies in humans (original and revised objectives ) Outputs: The project provided equipment to intensive care units for two infectious disease hospitals in Bucharest, and established isolation and intensive care units in 7 regional centers, but at least two intensive care units were not operational at project closing . To a limited extent, contingency plans for avian influenza and human influenza pandemic were strengthened, and legislation was revised . Outcomes: It is difficult to assess achievement of outcomes of this objective . The intensive care unit upgrades and isolation wards were not operational by the time of the H 1N1 pandemic. Achievement of this objective is rated modest. Objective 3: Build capacity to respond to potential future infections of humans by HPAI and other zoonoses (revised objectives only) Outputs: Construction works for upgrading the National Influenza laboratory to biosecurity level 3 were conducted, though the laboratory was not operational at project closing, lacking equipment and certification . Staff at the Cantacacuzino Institute received training on vaccine production standards, but rehabilitation works for the influenza vaccine production unit were partially completed (purchase of new equipment for vaccine production ) while others were not completed (ampoule sealing and filling station ). Outcomes: Works at the vaccine production unit increased the capacity of the facility to produce influenza vaccine, with production increasing from 30 doses per week in 2008 to 79 doses per week in 2009. This capacity was helpful in responding to the 2009 influenza pandemic. Achievement of this objective is rated substantial . 5. Efficiency: The economic analysis in the ICR estimates economic rates of return of 28-86% in the base case scenario, but this is based on a methodology with a number of weaknesses . The analysis uses real historic data on the costs of 2005-6 avian influenza outbreaks among poultry, but then relies on arbitrary and arguably implausible assumptions about the probability of further outbreaks (severe outbreaks occurring every 5 years) and the degree to which the project investments would reduce the risk of outbreak among birds or would reduce the impacts of an influenza pandemic (halving the infection rate among humans ). The proposed impacts on infection rates among poultry are particularly difficult to imagine given the cancellation of the animal health component; no project investments were completed that could plausibly reduce the probability of poultry sector outbreaks. Impacts on other zoonoses were not quantified . Many other benefits are not quantified; improvements in the intensive care unit will be helpful for many medical needs that have nothing to do with influenza. After restructuring, the project did not follow a least -cost approach to reducing the risk from avian influenza, which would have involved containing the disease among animals . Implementation of the project was slow, many activities were canceled, and many other activities were not completed by the time of project closure . Given the inability to estimate a meaningful economic return and the evidence of inefficiency in implementation, efficiency is rated modest. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 86% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The original project objectives had substantial relevance, and design relevance was also substantial . One of the original objectives was modestly achieved; another objective was negligibly achieved . Efficiency was modest. This points to major shortcomings and an outcome rating of Unsatisfactory . After restructuring, the removal of the poultry sector from the objectives left the project with modest relevance of objectives, and the removal of the animal health component left it with modest relevance of design . One of the revised objectives was modestly achieved, the other substantially so . Efficiency was modest. In cases in which the project ’s objectives are formally changed through a restructuring with the Board, the harmonized evaluation criteria require the evaluator to assess the project under the original objectives and under the revised objectives, and to weight the resulting two outcome ratings according to the percent of the actual disbursements that were disbursed at the time of the restructuring . In this case, the rating for both the original and revised objectives is the same, so the weighting will result in an Unsatisfactory outcome rating. a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR notes that the Cantacuzino Institute also faces significant financial difficulties because of the weak economy (page 27) as it is funded mainly from revenue from selling vaccines and laboratory services with no dedicated government funding . Maintenance of equipment at infectious disease hospitals could be affected by economic downturns. An IEG mission in 2011 found that tight budgetary conditions mean that funding for the Ministry of Public Health is at risk; 2010 budget levels were below those requested . Hospitals are finding it difficult to utilize the improved intensive care units because of a lack of adequately trained staff and doctors, due in part to public sector hiring freezes and personnel cuts . Vaccine production at the Cantacuzino Institute shut down in 2012 and has not yet restarted, though upgrades to the vaccine production line were completed in 2012 under the Health Sector Reform (Adaptable Program Loan 2) Project. The Bank reported that the BSL 3 laboratory was function as of late 2011, but was seldom used. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The original project allowed for both the animal health and food safety and the human health ministry to work together in a single intervention . The project built on a GPAI intervention model, which would allow cooperation between donors and the World Bank . The Bank responded rapidly in preparing a project to meet client demand, but effectiveness was then significantly delayed. The Bank should have taken into account the ongoing changes occurring in Romania during project preparation. One of the main reasons the government supported the project was to demonstrate to the European Union that they were taking action to increase emergency response capacity . This motivation had vanished by project startup because EU accession had been achieved . The Bank should also have realized that what Romania needed was not emergency support for tackling immediate outbreaks of avian influenza, which had been contained by the time the project commenced, but rather support for longer-term capacity building. The project had civil works components that required permits and vaccine production that needed licensing; these activities were not well suited to a short -term project. Thus, an emergency loan may not have been the appropriate instrument . While "project design must be simple and take into account a realistic assessment of the existing capacity of the Borrower and other stakeholders " was listed in the appraisal document and ICR as a lesson learned from previous emergency projects, the project design here was very complicated, with a large number of subcomponents including many activities that were not of an emergency nature . While a certain level of complexity was inevitable in order to adequately meet the objectives, the design could have been simplified by leaving out some long-term activities that could not reasonably be completed within the three year emergency project timescale. The project's M&E framework was weak, with some indicators that had vague baselines and were difficult to monitor and assess. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Project implementation suffered significant delays, with only 14 percent of funds disbursed by the restructuring date, 5 months before the project was scheduled to close . However, the Bank made significant attempts to improve implementation throughout the project, including repeated attempts to notify the government and implementing agencies of problems with execution and partial suspension of disbursements in November 2008. When these efforts proved unsuccessful, the Bank played an active role in restructuring the project and dropping the non -performing animal health component. Although the TTL changed 3 times over the four-year span of the project, hand over was reportedly smooth in each case, and the core project team remained the same. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The government showed strong commitment to the project at the time of preparation . Commitment faded over time as the government became focused on EU accession (achieved in January 2007): government became more interested in working with the European Commission than with the Bank . Also, concern about avian influenza was overshadowed by the 2009 H1N1 influenza pandemic. There was limited coordination between the human and animal health project management units, and the steering committee did not provide much coordination or support to project activities (the committee did not meet until December 2007). Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The Ministry of Health was committed to the project, though frequent change in leadership and staff slowed implementation. Budget constraints resulting from the financial crisis made it difficult for the Ministry to provide counterpart funding on time . Delays in procurement activities meant that some of the Cantacuzino Institute activities were not completed by the time the project closed . Difficulties in the National Sanitary Veterinary and Food Safety Authority seriously constrained project implementation, and led to the cancellation of the animal health component . A lack of clarity in the chain of command and in the distribution of responsibilities slowed decision making and weakened accountability . Frequent changes in top management led to changes in priorities and lack of support for the project . Managerial weaknesses within the agency made it difficult for the project management unit to get contracts approved. A lack of familiarity with Bank procedures added to the problems . Disagreements between the legal and technical departments and the Bank made procurement difficult; almost every procurement contract was blocked by the legal department . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The initial outcome indicators were relevant but some were difficult to measure ("Appropriate contingency plans prepared") or to attribute to project activities ("Risk of human infections reduced and treatment improved "). There were too many intermediate output indicators (33) and they were difficult to monitor. Targets and baselines were very broad, making it difficult to assess progress . Intermediate outputs were linked to components, rather than to objectives . Intermediate indicators focused on completion of tasks rather than on results. b. M&E Implementation: M&E implementation was slow to start, with no progress made as of July 2007. Progress on M&E was rated unsatisfactory or moderately satisfactory throughout the project (except for December 2009). With project restructuring, two of the three outcome indicators were dropped, along with all intermediate indicators linked to animal health and public awareness /communications, but no other steps were taken to improve the relevance of the design framework. After restructuring, the task team observed the shortcomings of the indicators, but decided against further restructuring since the project was close to completion . c. M&E Utilization: The M&E system provided limited assistance to the project . Process-oriented indicators did identify which activities were not progressing and these activities were dropped in the restructuring, but the lack of progress would likely have been obvious even without these indicators . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: The project was an environment category B project, as there existed a risk of spreading avian influenza through improper waste management. The ICR reported that all works financed by the project were in compliance with environmental standards. b. Fiduciary Compliance: No major fiduciary problems were reported . Financial management within the Ministry of Health was satisfactory. A lack of financial management capacity within the National Sanitary Veterinary and Food Safety Authority was a factor in the delays for contract approval . c. Unintended Impacts (positive or negative): d. Other: 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory Cancellation of poultry sector objective Unsatisfactory and animal health activities undermined project relevance. Many project activities were not completed, and there is little evidence of project efficacy. The economic analysis was not meaningful and there were inefficiencies in implementation. Risk to Development Moderate Significant All main project gains are threatened Outcome : by limited funding and staffing. Bank Performance : Moderately Moderately Quality at entry suffered from Satisfactory Unsatisfactory significant weaknesses (see section 8). Supervision was proactive but ineffective in addressing implementation problems. Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: IEG finds that an emergency project may not be the right tool for long term capacity building . It can be difficult to implement civil works projects and purchase of complex specialist equipment under the time constraints of an emergency project. When the country already has significant emergency response capacity, then using an emergency instrument may not be necessary . 14. Assessment Recommended? Yes No Why? A PPAR was conducted to contribute to a cross -country review of the results and lessons from the Global Program on Avian Influenza. 15. Comments on Quality of ICR: The ICR was helpful in identifying some structural weaknesses in the project due to complexity, in identifying weaknesses in the M&E framework, and in outlining institutional weaknesses in the implementing agencies that delayed implementation and led to cancellation of the animal health component . However, little information was provided on the contribution of the project to outcomes (such as pandemic preparedness). The ICR failed to note significant weaknesses in project design and Bank performance . The economic analysis was weak, particularly as it included benefits for the poultry sector when poultry sector activities were canceled. a.Quality of ICR Rating : Satisfactory