ICRR 14132 Report Number : ICRR14132 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 04/02/2013 Country : Iraq Project ID : P107698 Appraisal Actual Project Name : Strengthening US$M ): Project Costs (US$M): 8.7 12.3 Emergency Health Response In Northern Iraq L/C Number : Loan/ US$M ): Loan /Credit (US$M): 8.7 12.3 Sector Board : Health, Nutrition and US$M): Cofinancing (US$M ): 0 0 Population Cofinanciers : Board Approval Date : 06/18/2008 Closing Date : 06/30/2010 06/30/2012 Sector (s): Health (93%); Central government administration (7%) Theme (s): Health system performance (67% - P); Other human development (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Judyth L. Twigg Ismail Arslan IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Emergency Project Paper (page 7) and the Grant Agreement (page 5), the objective of the project was: To build capacity in the establishment of rapid, coordinated and effective response services to health emergencies, including those resulting from acts of violence, accidents, or natural disasters . The project was prepared on an accelerated basis (according to OP/BP 8.5 on Emergency Recovery Assistance, subsequently replaced by OP /BP 8.00 on Rapid Response to Crisis and Emergencies ), given the post-conflict environment. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The costs per component reflect the following : Appraisal amount; Appraisal amount + Additional Financing amount; Actual amount). 1. Emergency Coordination Centers (Appraisal: US$4.4 million; US$7.6 million; Actual: US$7.2 million): This component aimed to strengthen emergency coordination and response in the three governorates of Kurdistan (Erbil, Sulaymaniya, and Dohuk). Activities included establishment of an emergency coordination center in each of the three governorates; installation of a telecommunications system connecting the centers, ambulances, and health facilities; and provision of ambulances . 2. Capacity Building and Training in Emergency Procedures (Appraisal: US$0.3 million; US$0.8 million; Actual: US$1.3 million): This component aimed to build capacity for patient treatment in medical and paramedical personnel providing pre-hospital emergency services . Activities included the development of new procedures for pre -hospital emergency services and training of health personnel and ambulance drivers . 3. Blood Banks (Appraisal: US$2.2 million; US$2.1 million; Actual: US$1.4 million): This component aimed to upgrade the capacity of existing blood transfusion services to provide a timely supply of safe blood and blood products. Activities included construction /rehabilitation and equipping of three blood banks . 4. Project Management (Appraisal: US$1.8 million; US$2.7 million; Actual: US$2.4 million): This component aimed to ensure effective administration and coordination of project activities . Activities included the establishment of a Project Coordination Unit and contracting out of project implementation to the United Nations Office of Project Services (UNOPS). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost : The appraised project cost was US$ 8.7 million, with the entire amount financed as a grant to the regional Ministry of Health. The actual amount disbursed was US$ 12.3 million. Costs for Component 1 increased due to increased costs of the upgraded telecommunications system . Financing : Additional Financing of US$4.5 million was approved, increasing the total grant amount to US$ 13.2 million. Following the recommendations of the Mid -Term Review (January 2010), the original technical design for the telecommunications systems was inadequate and needed to be upgraded, which would entail increased costs and an extension of the project period . There were no co-financiers for the project. Borrower contribution : There was no planned borrower contribution . Dates : June 2010: Additional Financing (US$4.5 million) was approved to improve the quality and expand the scope of the telecommunications system. The closing date was extended from June 2010 to December 2011. December 2011: The closing date was extended a second time from December 2011 to June 2012 to allow completion of activities. 3. Relevance of Objectives & Design: a. Relevance of Objectives: High. High . During the post-conflict period, violence became a leading cause of death among the population while the health system in the country significantly deteriorated . Although the improvement of emergency medical services was not explicitly discussed in the most recently available Interim Strategy Note (FY09-11), it is consistent with the strategic theme to support reconstruction and socio -economic recovery efforts, in order to restore and modernize physical and human capital. The country's National Development Strategy for 2010-14 similarly identifies ensuring of basic services, particularly in education and health, as a strategic objective . b. Relevance of Design: Substantial . The project design was intended to address bottlenecks in the provision of emergency medical services . The project would provide much needed equipment and facilities, alongside training of medical and paramedical personnel. These activities were likely to increase the timeliness and effectiveness of response services to health emergencies. 4. Achievement of Objectives (Efficacy): To build capacity in the establishment of rapid, coordinated and effective response services to health emergencies, including those resulting from acts of violence, accidents, or natural disasters . Substantial Outputs Establishment of 3 emergency coordination centers (ECCs), as planned. Each center is fully staffed with an average of 1 coordinator, 12 drivers, 14 paramedical personnel, and 20 dispatchers. The target for staffing was 1 coordinator, 1 driver, and 1 dispatcher per center. Establishment of a three-digit emergency phone number for citizen calls . A media campaign to educate citizens about emergency medical services was conducted . 18,419 calls were received in April 2012. However, there are some problems with incoming false calls and confusion between the emergency number and a separate information number. Provision of 29 fully equipped ambulances and equipping of 102 existing ambulances with telecommunications equipment. Training of 34 master trainers on the provision of emergency medical services, surpassing the target of 16 trainers. Training of 248 paramedical staff in emergency medical response, surpassing the target of 208. Training of 20 physicians, falling short of the target of 36 due to higher than estimated costs for training . Training of 34 health ministry staff on medical emergency management, surpassing the target of 4. Training of 21 emergency medical dispatchers, falling short of the target of 45 due to higher than estimated training costs. Training of 248 ambulance crew members in emergency medical response, falling short of the target of 600 due to higher than estimated training costs . Training of 199 ambulance drivers in safe driving and mechanical maintenance, falling short of the target of 400 due to higher than estimated training costs . Construction and/or renovation of 3 blood banks, as planned. However, A framework for emergency medical services outlining organizational structure, staffing, budget, and protocols was developed but not yet adapted . There were no provisions for training on safe handling of blood or the disposal of medical waste by the blood banks. At the time of project preparation, there were ongoing discussions with the World Health Organization to support this aspect of safe blood supply; however, it is not clear whether this support materialized . Outcomes The 3 emergency coordination centers were fully functioning with the following number of emergency incidents reported during the project period (i.e. emergency calls answered and dispatched, and patients transported to hospital): Dohuk: 287; Erbil: 27; Sulaimaniyah: 2710. The average response time (defined as the time between the call and arrival on scene ) was reported in the ICR as 7-10 minutes in urban areas. As emergency services did not exist in its present form prior to the project, a baseline figure was not available . Medical literature suggests that response time for a metropolitan area in the U.S. are in a similar range; and therefore, response services supported by this project are considered timely . 5. Efficiency: Modest , due to lack of sufficient information to assess whether project resources were used efficiently . Costs for managing the project comprised a relatively high proportion of total project costs (US$2.4 million out of US$12.3 million, or 19.5%), due to the fees paid to the contracted implementing agency, although these costs may have been appropriate given the post-conflict emergency environment . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of the project objectives is rated high, and relevance of the design is rated substantial . Achievement of the objective to build capacity to establish rapid, coordinated and effective response services to health emergencies is rated substantial due to evidence of improved timeliness and effectiveness of response services . Efficiency is rated modest due to lack of information . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: An overall institutional framework for emergency medical services has yet to be firmly established, including standard operating procedures and strategies for maintenance and M&E . Funding and capacity in the regional Ministry of Health remains limited, while coordination amongst the health departments in the three governorates is also weak. However, the Centers remain operational, and training activities are being integrated into the core curriculum of medical and paramedical schools . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project design drew upon lessons learned from other Bank emergency projects in the health sector . Of particular note, previous projects had experienced supervision difficulties due to security risks in the central part of the country, as well as the limited capacity of the central government . Therefore, the grant was provided directly to the regional Ministry of Health (MOH) in order to build local accountability and ownership and enable more direct supervision. The project risks identified in the PAD included inadequate project implementation and management capacity. This risk was addressed through contracting out day -to-day implementation of the project to the UN Office of Project Services (UNOPS). However, analysis and costing of several activities were inaccurate, including the choice of the telecommunications equipment and the technical requirements for implementation, and the scope of the training program (the estimated need was less than half of what was actually needed to achieve sufficient coverage of the population ). at-Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: There were coordination difficulties between the regional MOH and UNOPS, leading to implementation delays (see Section 9). The Bank team provided intensive and proactive supervision support, given these difficulties, in particular using the opportunities presented by the Mid -Term Review to improve project design and secure Additional Financing to complete implementation of activities . For example, technical training activities were expanded and the telecommunications system was upgraded, although the latter required increased coordination with other government agencies (the Ministries of the Interior and Communications ). Safeguards compliance with regards to medical waste management was unsatisfactory . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The regional Ministry of Health demonstrated strong commitment to the project objectives and allocated staff and office space for the Project Coordination Unit despite scarce resources . The Minister of Health appointed during the second year of the project period played an important catalyst role in increasing the engagement of the three governorates. However, the MOH did not provide sufficient supervision of UNOPS to ensure timely implementation of activities. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The Project Coordination Unit within the Ministry of Health was adequately staffed, including an expansion of the Engineering Technical Committee. Project management capacity improved overall during the project period . However, there were shortcomings in performance of the contracted implementation agency, UNOPS . For example, the contract arrangement with UNOPS was insufficiently supervised by the MOH to ensure timely and adequate delivery of outputs; extra time was needed for UNOPS to obtain clearances from its central headquarters; the UNOPS team was insufficient in number of staff and did not contain an optimal skills mix (i.e. information technology specialists, fiduciary specialists with knowledge of Bank procedures ). Safeguard compliance was unsatisfactory with regards to the disposal of medical waste . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Project monitoring indicators were to be tracked through a computerized emergency communication system . The M&E results framework had some shortcomings, such as unrealistic targets (i.e. the target for the number of staff trained was based on international personnel standards and did not take into account personnel shortages in the country). Indicators to measure improved "effectiveness" of response services could have been more clearly identified, as the Project Paper (page 10) had defined the "effectiveness" of the response as "the capacity to carry out a rapid assessment of the situation, to make the decision to mobilize appropriate resources in a coordinated way, to provide adequate pre-hospital care and reorganize patients /victims referrals." b. M&E Implementation: The computerized system installed in the Centers enabled some data collection . However, shortcomings in M&E were not addressed. c. M&E Utilization: According to the ICR (page 10), it was not clear whether the monitoring data generated by the Centers would be used to inform decision-making. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was classified as a Category "B" project, due to new construction . The requirement to carry out a limited Environmental Analysis as part of project preparation was waived, on the condition that the implementing agency, UNOPS, agree to several minimum standards in implementation . UNOPS applied the standards for (i) the inclusion of standard environmental codes in repair and reconstruction documents; and (ii) review and oversight of any major reconstruction works by specialists . However, according to the ICR (page 10), "it is unclear whether UNOPS applied the provision on safe waste disposal (particularly for the blood banks )" and therefore safeguard compliance is assessed in the ICR as "moderately unsatisfactory." b. Fiduciary Compliance: Financial management: The finalization of accounts and audit of UNOPS were delayed . According to the ICR (page 10), UNOPS had yet to submit the audit report covering the period July 1, 2010 to March 30, 2012. Although the ICR reports that financial management throughout the project period was rated as Moderately Satisfactory, there is little information provided in the ICR to confirm satisfactory financial management performance . Procurement: The contracting of UNOPS as implementing agent was in part a mitigation measure for procurement performance risk. However, the arrangements with UNOPS were initially inadequate, leading to delayed procurement. Issues included: centralization of UNOPS procurement activities at the headquarters in Copenhagen, which required legal clearance for large contracts; differences in UNOPS vs . Bank procurement procedures; and lack of technical expertise to provide proper input on procurement documents . However, the ICR (page 10) reports that procurement performance improved by the end of the project period and was eventually satisfactory . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn from the ICR (pages 18-19) and adapted by IEG: The sustainability of a project may be at risk when there are no clear plans for project transition in the design stage. In the case of this project, longer -term aspects of capacity building (such as establishment of an institutional framework, coordination arrangements within the government, and a strategy for long -term operations and maintenance) were not considered in the project design, and therefore the risk to development outcomes is considered significant . Lessons from IEG: Even in a post-conflict environment with the need for rapid response, technical aspects of the project design should be carefully assessed . In the case of this project, although the scope of project activities was narrowly focused on emergency medical services (as opposed to the entire health sector ), the technical requirements for the communications system needed to be re -assessed, which led to implementation delays and higher project costs. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is concise and overall consistent with guidelines . The quality of the evidence was adequate for assessing project performance. The ICR's lessons offer relevant considerations for implementing projects in post -conflict environments. a.Quality of ICR Rating : Satisfactory