ICRR 13630 Report Number : ICRR13630 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 07/25/2011 PROJ ID : P100451 Appraisal Actual Project Name : Avian Influenza And US$M ): Project Costs (US$M): 6.50 6.50 Human Pandemic Preparedness And Response Project Country : Tajikistan Loan/ US$M ): Loan /Credit (US$M): 5.00 5.00 Sector Board : Cofinancing (US$M ): US$M): 1.80 1.80 Sector (s): General public administration sector (37%) Health (29%) Agricultural extension and research (20%) Other social services (8%) Animal production (6%) Theme (s): Rural services and infrastructure (40% - P) Pollution management and environmental health (20% - S) Health system performance (20% - S) Participation and civic engagement (20% - S) L/C Number : CH244 Board Approval Date : 06/29/2006 Partners involved : EU, UNICEF, WHO Closing Date : 12/01/2009 08/31/2010 Evaluator : Panel Reviewer : Group Manager : Group : Ridley Nelson Robert Mark Lacey IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: The objective stated in the appraisal document was to minimize the threat of HPAI infection and other zoonoses posed to humans and poultry in Tajikistan, and to prepare for, control and respond to influenza pandemics and other infectious disease emergencies in humans . The legal document has the same objective . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Component 1 Public Awareness and Information (Appraisal US$1.00 million; Actual US$1.06 million). This component supported the development of a communication program aimed at minimizing the risk of an avian influenza outbreak and its spread, by ensuring that citizens understood the threat, and were aware of AI symptoms . It included (i) policy advocacy to heighten AI awareness among decision makers, and support to the government for crisis communication management; (ii) public education through mass media; and (iii) social mobilization through community/school-based targeted interventions . Component 2 Animal Health (Appraisal US$3.00 million; Actual US$2.15 million). This component provided support for prevention, control and total eradication of HPAI by: (i) enhancing animal health planning and coordination capability for HPAI prevention; (ii) strengthening field disease surveillance and laboratory diagnostic capacity; and (iii) strengthening HPAI outbreak containment plans, including the establishment of a compensation fund . Component 3 Human Health (Appraisal US$2.20 million; Actual US$2.90 million). This component provided support for reduction of the impact of a pandemic influenza virus by: (i) enhancing public health program planning and coordination; (ii) strengthening of national public health surveillance systems; and (iii) strengthening health care system’s response capacity. Component 4 Implementation Support and Monitoring and Evaluation (Appraisal US$0.30million; Actual US$0.39million). This component supported the coordination and management of the activities, including arrangements for financial management and procurement. It provided for continuous monitoring of project implementation, including operational audits of the Compensation Fund as well as annual project audits, and periodic surveys on project outcome indicators . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Towards the end of the project implementation period, the project resources were reallocated away from the compensation fund to be used for procurement of goods and works at the various laboratories and hospitals that were supported under the project . The closing date was extended for nine months to enable activities to be completed. 3. Relevance of Objectives & Design: Relevance of Objectives . Rated High . As noted by the ICR, (p. 9) the project did not emerge from a carefully crafted national strategy but was designed in response to a quickly emerging potential crisis, while at the same time building awareness and the capacity of crucial institutions that would help the country identify, prevent the spread of, and manage an emerging pandemic crisis. While none of the worst case scenarios became a reality because each potential pandemic was a largely localized phenomenon without a severe worldwide impact, the threat remains . Relevance of Design . Rated Substantial As noted by the ICR (p. 6) the design followed fairly closely the standard Bank model which appears to have been largely appropriate in this case . Essentially this consisted of support for veterinary and human health services with emergency response capacity building and awareness building across both professional workers and the potentially affected or involved population as a whole, particularly poultry farmers . The late reallocation of resources was appropriate at least in the short term to shift resources towards improved diagnostic and response capacity in health services. One issue in design was whether to develop national laboratory capacity or to send samples out for analysis . The ICR argues (p. 11) that developing capacity internally was unavoidable due to very poor trade relations with most of its neighbors, poor communication links by road or air, and the fact that external laboratory services still would have to be paid for. While overall, this argument is largely valid, if only given the need for rapid response time and the complementarities with related national laboratory services, the issue of payment remains a problem since resources would still be needed to cover purchased services, whether internal or external . 4. Achievement of Objectives (Efficacy): The extent to which the project minimized the threat of HPAI infection and other zoonoses posed to humans and poultry and prepared for, controlled, and responded to, influenza pandemics and other infectious disease emergencies in humans. Rated Substantial . Since no major threat emerged there is no disease control outcome to measure and therefore inevitably limited evidence to assess outcome or expected outcome. The ICR argues that the project developed intersectoral surveillance systems in both animal health and human health sectors and the transparent and effective establishment of a compensation funding mechanism to provide an incentive for farmers to alert veterinary authorities. A National Action Plan was prepared with the aim of ensuring adequate coordination across the various institutions as well as the provision of information to the public. Standard Operating Procedures were developed at the relevant institutions. The ICR argues that the procedures introduced under the project proved their worth in the promptness with which Tajikistan managed to properly diagnose the H1N1 virus in the Fall of 2009 and address the polio outbreak in the summer of 2010. However, these were threats of a much lower level of challenge than a major global Avian Flu outbreak. The ICR also finds that the improved coordination and communication established would serve Tajikistan in tackling other epidemiological events. The program to enhance awareness is considered by the ICR to have had largely satisfactory outcomes . However, there were some weaknesses . Some two thirds of the population in Tajikistan is aware of Avian Influenza and over half of the adults are cognizant of the risks related to contact with an infected animal according to a KAP survey undertaken by UNICEF towards the end of the project . The report from this survey was provided to IEG and was found to follow a sound methodology, however being a single survey it could not indicate change in awareness over time. The survey found that project participants who had had contact with the various media sources used had a higher level of risk perception about AI in birds and humans than those who had not . However, it is not clear how significant the differences in perception really were because responses were variable and on one question to children, adults and teachers about what to do with a dead bird at home, there were no differences in response between participants exposed to the awareness campaign and those who were not . The ICR reports that, according to veterinary specialists, awareness at regional and village level will help to improve the handling of animals and by-products related to other diseases such as the handling of birthing material from brucellosis infected animals . A free Hotline has improved surveillance with the population taking an active role in reporting emergencies in remote areas and it has reduced response times to emergencies . On regulatory and institutional measures, a National Action Plan with Standard Operating Procedures (SOPs), also applicable to a number of other disease outbreaks, was developed. An updated migratory bird survey was completed and increased controls were put in place with the establishment of a computerized veterinary database to better track infections. A compensation fund was set up laying the groundwork for a wider national epizootic compensation fund. This review gives significant weight to compensation since it is particularly important as an incentive for reporting . On animal health, there was a problem that lay outside the project's influence but has potential impact on its outcome. Although the ICR argues (p. 11) that the project dramatically improved the capacity for sampling, diagnosis and analysis at the regional and central levels, there were limited resources for operating the laboratories. Unanticipated at appraisal, the government implemented a reform that included the release of some 1,100 veterinarians. This proved disruptive at least in the short term even if, as noted in discussion with the project team, in the longer term it may be a more sustainable solution. While it appears that one aim was to shift resources to operating budgets, as so often happens with such public sector shifts in situations with very tight resource constraints, there was no increase in operating budgets towards the core services and the supply of reagents and adequate staffing of field laboratories remains a problem. The ICR finds that, while the ability to take and transport samples, undertake preliminary tests and determine the nature of infection has improved, the supply chain for maintaining this service remains precarious due to the lack of budget. The ICR notes that, "Given this situation, one could legitimately ask whether contracting of laboratory services with service providers abroad would not have been more sustainable." This is a valid question since although on the face of it insufficient resources might be expected to impact both internal or external laboratory work to the same extent, the reality is often that ministry operating budgets are formulaically tied to staffing. On human health, an assessment of project impact by UNWHO’s reported a significant improvement in the country’s capacity to identify and manage health emergencies. Five staff, consisting of two Sanitary-Epidemiological Service staff, two veterinarians and one from the NCVD, were trained in the proper procedures of shipment of high risk samples. The ICR reports the passing of important legislation in support of this regulation and operating procedures were adapted to improve planning and coordination across the various players, related to both animal and human health and emergency services. There was improved collection, transport, and testing as well as reporting through computerized interconnection. Hospitals in Sughd, GBAO and Khatlon and the main infectious disease hospital in Dushanbe benefited from rehabilitation of isolation wards. A Bio-security Level 2 diagnostic laboratory was installed in Dushanbe. Around 700 tests had been undertaken by the time of the ICR. Retesting by the London International Reference Laboratory was reported by UNWHO to show an 85% accuracy. There were problems with the refurbishment of old buildings unsuited to modern lab and ward requirements . The issue of sustainability is picked up again under the Risk to Development Section, however here it is noted that, in a health related issue where the capacity to rapidly respond in future years to an unpredictable threat is the essence of the challenge, questions about sustainability are an element in assessing outcome . 5. Efficiency (not applicable to DPLs): Efficiency is rated substantia l. The appraisal estimated the benefits from the reduction in losses from a potential pandemic based only on a projection of the reduction in poultry losses . This was conservative since it excluded any benefits to human health, either value of life benefits or avoided epidemic costs . Clearly these could potentially be very high both nationally and globally. The internal rate of return found on poultry alone was 21.3%, substantially above the opportunity cost of capital. (Notionally, this is recorded in the table below as representing a 50% coverage but coverage cannot be calculated.) Apart from any longer term human health valuations, there have been some other more immediate benefits . Looking more broadly across the sectors, the ICR finds (p. 12) that benefits in the health care sector, especially the provision of oxygenators and heart monitors provided by the project, are credited with having saved a number of lives of people suffering from various conditions . Also, the laboratory capacity has played a role in Tajikistan ’s ability to treat other human influenza and fulfill its international reporting responsibilities on that disease . Similarly, the refurbished veterinary facilities and laboratories provide benefits beyond the ability simply to diagnose and manage Avian Influenza. The country suffers from other zoonoses such as brucellosis and foot and mouth disease. The improved ability to collect samples and properly diagnose them is expected to play a significant role in preventing contagion in such diseases . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 21.3% 50% ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project had substantial relevance of objectives given the serious threat . Design, while arguably exhibiting some weakness in ensuring sustainability, is, on balance rated substantial . Efficacy is rated, on balance, substantial albeit with limited evidence. The efficacy rating draws partly from the positive WHO impact assessment . Efficiency is rated substantial given the substantial rating of efficacy suggesting readiness and potentially high avoided costs of a pandemic. Supporting substantial benefits, the appraisal ERR, using poultry benefits alone, was well above the opportunity cost of capital . However, as with a number of Avian Flu interventions there are some concerns about sustainability since so far the threat did not strike and almost inevitably attention is turned to other pressing priorities . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: There are particular concerns expressed in the ICR about the sustainability of the animal health component . Although it appears to have improved the capacity for sampling, diagnosis and analysis at the regional and central levels, it suffers from limited resources to maintain and operate the laboratories . As noted above, during the implementation period of the project, the government implemented a quite far reaching reform that included the release of some 1,100 veterinarians. A steady supply of sufficient chemical reagents and adequate staffing of the field laboratories remains problematic . WHO and FAO are currently working on plans (ICR p. 14) to give support for lab reagants and other consumables for the next few years . However, there remain questions about the level of government commitment and capacity to sustain relevant investments as the recent immediate threat recedes . Sustainability in such a situation is a difficult balancing act between maintaining readiness for a low probability longer term high cost event and immediate development needs . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Ensuring Quality at Entry . The project followed the standard Bank model . It was therefore somewhat complex in design and wide in scope without sufficiently accommodating Tajikistan's capacity and budget weaknesses . However, it would not have been possible to avoid both the livestock and the human health elements so there was an inherent complexity in the strategy. The appointment within the institutional design of a multi agency Technical Working Group was an important element. The contracting of UNWHO, UNFAO and UNICEF for surveys, training and other support was quite complicated, calling for coordination, but appropriate . They provided knowledge and funding and helped to bring the country towards international practice with improved disease information exchange . These organizations may be able to provide some support for longer term sustainability of project investments . Quality of Supervision . The ICR finds that the presence of the TTL in country aided quick Bank response and that there was regular supervision. It also argues that the reallocation of resources under the compensation fund carried out in 2010 was appropriate and responded to a real need . However, there is some question about whether this shift diverted attention too much from the longer -term Avian Flu readiness to more immediate but less targeted general health issues. There are arguments in both directions and insufficient evidence to draw a firm conclusion on this issue . at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Government Performance . The ICR notes that the Government, after some initial reluctance, became fully committed to the project, providing the regulatory framework, appointing a Joint Technical Working Group and establishing administration by a PMU with significant experience in implementing Bank -financed projects. Government commitment was also exhibited in establishing the compensation fund mechanism, the National AI Response Plan, the National Pandemic Plan, and the development of operating procedures and regulations . While the cutting of the veterinary service had some negative impact, in the longer term, which is of relevance to pandemic situations, such a move towards privatization of veterinary services was sound and has been widely adopted . Implementing Agency Performance . Three main agencies were involved, the PMU, SVIS, and MoH .. The PMU was initially overwhelmed by some of the technical challenges in the procurement of specialized equipment which led to initial delays. However, overall it performed satisfactorily in managing implementation. Financial management, procurement and M&E activities were effectively carried out. The Communication Specialist at the PMU with assistance from UNICEF was responsible for the activities carried out within the Public Awareness and Communication subcomponent. Some stakeholders reported that the communications activities under this project had a major impact in reducing the risks from a future pandemic . SVIS was responsible for the veterinary component and effectively implemented its project activities, including laboratory upgrading, staff training, and training for poultry producers. It implemented the serological monitoring for HPAI, upgraded the laboratories, and established the National Animal Disease Information System. It is now working with FAO on upgrading its capabilities in diagnosis and reporting so as to achieve international standards. The MoH developed a National Pandemic Influenza Preparedness Plan, secured adequate vaccine and drug supplies, strengthened laboratory capabilities, and trained staff. The construction of the hospital isolation wards, however, was difficult for MoH due to lack of technical expertise. MoH’s response to the H1N1 outbreak led to early detection and effective containment . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Design . The ICR reports that the preparation team developed a comprehensive M&E framework including benchmarks related to the drafting of legislation, regulations and operating procedures to ensure that all concerned institutions would be prepared for an emerging pandemic. M&E appropriately focussed largely on outputs since the actual containment of outbreaks could only be tested should an epidemic arise. As the ICR notes (p.8), the PAD contained a good results framework but its transfer to the ISR was not clear . Implementation . Although, the PAD contained a good results framework, its transfer to the ISR was not clear. The organization of the ISR indicators confuses outcomes and outputs . Notwithstanding this confusion, most of the indicators of achievement under the project were retained and data was collected. The monitoring of progress against deliverables was satisfactory and the information relevant. The ICR argues that the final M&E report provided good insight into what happened on the ground Utilization . The proper diagnosis of the H1N1 virus in fall/winter 2009 and the rapid containment of the polio outbreak in summer 2010 were both considered by the ICR to illustrate the system ’s capacity to react to an emerging health crisis event . a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards . The only safeguard triggered was OP 4.01 on Environmental Assessment. Due to the emergency nature of the project, a waiver was granted on having the Environmental Assessment done prior to approval . It was done later. The ICR reports several safeguard missions that suggested no major issues . There were some small issues with the height of incinerator chimneys for contaminated materials and some installation problems, but these were promptly addressed. Fiduciary . The ICR reports (p. 8) that most reviews rated fiduciary and procurement satisfactory . The details of those that did not are not discussed in the ICR . Audits were on time with moderate qualifications related to internal controls . Unintended Impacts . No unintended impacts were reported . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Significant Significant The ICR uses the term Substantial . Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The ICR offers three lessons which are summarized here with some adaptation : 1. Effective coordination of activities with other donors and projects requires time and can delay implementation therefore it should be planned for . Where cross-donor coordination is envisaged, it is prudent to attempt to project the sequencing and critical paths of the different elements and to incorporate flexibility into implementation schedules. 2. Early cooperation with internationally experienced UN agencies working in the same or related fields can offer synergy and provide a basis for sustaining programs in tight budgetary situations . 3. Refurbishing old buildings for laboratories or wards of hospitals is likely to be fraught with unpredictable design constraints that are costly to remedy. While it may appear desirable and pragmatic to take advantage of already available buildings and infrastructure, in cases of facilities such as laboratories and hospital wards the remedial costs may be prohibitive. This is particularly applicable to laboratories and other facilities for handling highly pathogenic and contagious materials that must abide by international standards . 14. Assessment Recommended? Yes No Why? To verify the ratings and document lessons learned in an area with limited evaluation evidence so far . 15. Comments on Quality of ICR: The ICR is generally satisfactory . It gives a clear description of the project . As with many of the Avian Flu interventions, data is limited. This is in the nature of projects that, so far, have not yet been called upon to prevent pandemics and are thus unable to demonstrate their preparedness . Nonetheless, more evidence, even qualitative, on the state of preparedness observed by the ICR mission would have been useful in the absence of hard data . Also, while most reviews rated fiduciary and procurement satisfactory it would have been useful if the ICR had provided some information on those that did not and how this was handled . a.Quality of ICR Rating : Satisfactory