Document of The World Bank Report No: ICR00004224 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-14785) ON A GRANT IN THE AMOUNT OF US$ 10 MILLION TO THE Economic Community of West African States - West Africa Health Organization (ECOWAS-WAHO) FOR A West Africa Regional Disease Surveillance Capacity Strengthening Project (WARDS) January 15, 2018 Health, Nutrition, and Population Global Practice Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective December 14, 2017) Currency Unit US$ 1 = XOF 555 XDR 1 = US$ 1.41 FISCAL YEAR July 1 – June 30 ABBREVIATIONS AND ACRONYMS ACGF Africa Catalytic Growth Fund AFENET African Field Epidemiology Network BMGF Bill and Melinda Gates Foundation CAD Canadian Dollar CBA Cost benefit Analysis CAMES Certificate of Specialized Studies CCISD Centre for International Cooperation in Health and Development CCT Country Coordinating Team CDC Center for Disease Control and Prevention CES Center for Epidemiological Surveillance CIDA Canadian International Development Agency CMU Country Management Unit DHIS2 District Health Information System 2 DL Disbursement Letter DSR Disease Surveillance and Response ECOWAS Economic Community of West African States EOC Emergency Operations Center EVD Ebola Virus Disease FELTP Field Epidemiology and Laboratory Training Program FETP Field Epidemiology Training Program FM Financial Management GDP Gross Domestic Product GHRF Global Health Risk Framework GHS Ghana Health Services ICR Implementation Completion Report IDA International Development Agency IDSR International Disease Surveillance and Response IFR Interim Financial Report IHR International Health Regulations IOM International Office of Migration IR Intermediate Results ISR Implementation Status Report M&E Monitoring and Evaluation MCMC Markov Chain Monte Carlo MoU Memorandum of Understanding MTR Mid Term Review NTD Neglected Tropical Disease ORAF Operational Risk Assessment Framework PASEi Projet d’Appui en Surveillance Epidémiologique Intégrée PCN Project Concept Note PDO Project Development Objective PMU Project Management Unit RCDCP Regional Center for Disease Control and Prevention (of ECOWAS) REDISSE Regional Disease Surveillance Systems Enhancement Project RESOLAB Veterinary Laboratory Network for Avian Influenza and other Transboundary Animal Diseases in West Africa RF Results Framework RFCDCP Regional Framework for Communicable Diseases Control and Preparedness RIAS Regional Integration Assistance Strategy SCADD Strategy for Accelerated Growth and Sustainable Development TEPHINET Training Programs in Epidemiology and Public Health Interventions Network UNDP United Nations Development Program USAID United States Agency for International Development WAFELTP West Africa Field Epidemiology Laboratory Training Program WAHO West Africa Health Organization WAHO-OOAS West Africa Health Organization- WHO World Health Organization WHO-AFRO World Health Organization – Africa Region Senior Global Practice Director: Timothy Evans Sector Manager: Trina Haque Project Team Leader: Enias Baganizi ICR Author : Oluwayemisi Busola Ajumobi Economic Community of West Africa States – West Africa Health Organization (ECOWAS – WAHO) West Africa Regional Disease Surveillance Capacity Strengthening (WARDS) Project CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes .............................................. 1 3. Assessment of Outcomes ............................................................................................ 2 4. Assessment of Risk to Development Outcome........................................................... 3 5. Assessment of Bank and Borrower Performance ....................................................... 3 6. Lessons Learned ......................................................................................................... 3 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ............ 3 Annex 1. Project Costs and Financing ............................................................................ 4 Annex 2. Outputs by Component ................................................................................... 5 Annex 3. Economic and Financial Analysis ................................................................... 6 Annex 4. Bank Lending and Implementation Support/Supervision Processes .............. 7 Annex 5. Beneficiary Survey Results ............................................................................. 8 Annex 6. Stakeholder Workshop Report and Results..................................................... 9 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 10 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 11 Annex 9. List of Supporting Documents ...................................................................... 12 MAP A. Basic Information West Africa Regional Country: Western Africa Project Name: Disease Surveillance Capacity Strengthening Project ID: P125018 L/C/TF Number(s): TF-14785 ICR Date: 05/18/2017 ICR Type: Core ICR Lending Instrument: IPF Grantee: ECOWAS Original Total USD 10.00M Disbursed Amount: USD 10.00M Commitment: Revised Amount: USD 10.00M Environmental Category: Category C Implementing Agencies: WAHO Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 04/29/2011 Effectiveness: 12/11/2013 Appraisal: 05/03/2013 Restructuring(s): Approval: 10/22/2013 Mid-term Review: 01/22/2016 04/25/2016 Closing: 06/30/2017 06/30/2017 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Grantee Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Quality at Entry Project at any time No None (QEA): (Yes/No): Problem Project at any Quality of Yes time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Major Sector/Sector Health Health 100 100 Major Theme/Theme/Sub Theme Human Development and Gender Health Systems and Policies 100 100 Health System Strengthening 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Makhtar Diop Makhtar Diop Country Director: Rachid Benmessaoud Colin Bruce Practice Trina S. Haque Trina S. Haque Manager/Manager: Project Team Leader: Enias Baganizi Enias Baganizi ICR Team Leader: Enias Baganizi ICR Primary Author: Oluwayemisi Busola Ajumobi F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The project development objective is to strengthen the regional disease surveillance and response system of ECOWAS member countries Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Percentage targeted districts CES which submit complete and on-time Indicator 1: reports for selected priority diseases using the standard reporting format Value 0 85% 95.2% quantitative or Qualitative) Date achieved 10/22/2013 06/30/2017 112% - target surpassed Although shown as surpassed, the calculation of this indicator was Comments observed during the ICR review to be problematic as the definition of the (incl. % achievement) indicator seeks to report on the percentage of targeted district CES that submit complete and timely reports for select priority diseases. However, the computation of the reported data reports on the percentage of reports received on time. Health personnel receiving training (number) of which percent female Indicator 2: Value 0 (percent of female 404 (25%) 543 (23%) quantitative or unknown) Qualitative) Date achieved 10/22/2013 6/30/2017 Comments (incl. % 134% - target surpassed. 92% by gender. achievement) Percentage of short term trainees in field epidemiology or laboratory skills Indicator 3: who are working at district 12 months after completing training, by gender and country of sponsorship Value quantitative or 0 80 82 Qualitative) Date achieved 10/22/2013 102 percent - target surpassed Comments To track and properly reflect the status of achievement of this indicator for (incl. % achievement) the ICR reporting purposes, the analysis makes use of most recent data provided by the implementing agency as of December 15, 2017 which only accounts for 20 short-term trainees in field epidemiology or laboratory skills (out of a total of 420, as reported in the intermediate outcome indicator 3) who have completed the 12-month period post- completion of the training program; Cote d’Ivoire (8), Senegal (8), and Togo (4). Of the 20 trained health workers, 70 percent report being in the same position post-completion of training program; 15 percent reported moving to a new role within the CES; while 15 percent were non- respondents. Of the 17 trainees who were reachable, 82percent reported working at the district 12 months post-training completion, representing a 102percent achievement. For the other health workers who benefited in the short-term training, the required 12-month period is still ongoing following project closure. Completion rate of FELTP trainees by gender and country of sponsorship Indicator 4: Value 0 85% 96% quantitative or Qualitative) Date achieved 10/22/2013 06/30/2017 113 percent - target surpassed. As reported below in the intermediated outcome indicator two, 50 residents were trained instead of the originally set target of 60 (which is the assumption used to derive the set target completion rate of 85percentpercent). The reason provided by the task team and implementing agency is that at the beginning of the training program, it was agreed with the US CDC that 30 residents will be trained under the francophone FELTP program as opposed to the originally set target of 40 Comments (while 20 residents were trained under the Anglophone program). (incl. However, the new target of 50 trained residents was not formally revised percentage in the results framework. Of the 50 residents trained, there was a 96 achievement) percent completion rate (48 of the 50 sponsored trainees completed the training program). Below is break down by gender and participating countries: ✓ Total residents (48) including 8 women, representing a 17 percent completion rate ✓ Completion rate by country of citizenship is reported as 100 percent for participants from 9 of 11 countries under the program (Benin, Burkina Faso, Cote d’Ivoire, Ghana, Guinea, Liberia, Mali, Niger, Senegal, Togo, Sierra Leone) Semi-annual regional disease surveillance bulletins published by WAHO Indicator 5: and disseminated to ECOWAS Member Value 0 7 quantitative or 6 Qualitative) Date achieved 10/22/2013 86 percent - target substantially achieved The actual number of semi-annual regional disease surveillance bulletins reported as an activity under the project is two. The activity was revised following request to WAHO from all ECOWAS member countries to publish weekly epidemiologic bulletins. At the request of the countries, WAHO agreed to the publication of weekly bulletins. 80 weekly epidemiological bulletins and 3 quarterly epidemiological bulletins were Comments produced and published on the WAHO website and shared with partners (incl. % and countries. The use of the weekly bulletins served to be more practical achievement) and useful to the ECOWAS member states and the sub-region (although no revisions were made to the results framework to take this into account). To appropriately reflect project achievements under this activity, the analysis of this indicator includes an additional 4 semi-annual bulletins computed from the 80 weekly and 3 quarterly bulletins that were reported as published and disseminated by WAHO to ECOWAS member countries (assuming that the weekly and quarterly bulletins hold similar content and serve the same purpose as the semi-annual bulletins) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Total Number of Centers of Epidemiologic Surveillance (CES) Indicator 1: established by the project Value 0 60 60 (quantitative or Qualitative) Date achieved 10/22/2013 06/30/2017 Comments 100percent - target achieved (incl. % achievement) Total Number of candidates who completed Long-term FELTP training, Indicator 2: by gender and country of sponsorship Value 0 54 48 (quantitative or Qualitative) Date achieved 10/22/2013 06/30/2017 Comments 96percent - Target substantially achieved. (incl. % achievement) For this indicator, the actual number of candidates sponsored under the project was used to compute percentage achievement instead of the originally set target of 54 (which was derived under the initial premise that 60 residents were to be sponsored under the program). In total, 50 residents were sponsored under the project of whom 48 completed the FELTP: Anglophone FELTP offered by the University of Ghana (20); Francophone FELTP offered by the CDC/University of Ouagadougou (28) - number was initially 30; death of one resident in the cohort during the program; and another francophone resident could not defend the required thesis for graduation due to ill-health. The distribution of the total number of graduated candidates by gender and sponsoring country is as follows: ✓ Total residents (48) including 8 women, representing 17percent ✓ Countries with no female residents: Burkina Faso (3), Côte d'Ivoire (4), Guinea (4), Mali (4), Senegal (4), Sierra Leone (2) ✓ Countries with 1 woman among the residents: Liberia (3), Niger (3), and Togo (4). ✓ Countries with two women among residents Benin (3), and Ghana (11). ✓ As of June 30, 2017, of the three remaining cohorts: two Francophone cohorts and one Anglophone cohort had completed the final examinations; The thesis defence and the graduation of the residents took place in July 2017. Total number of persons who completed short-term training in field Indicator 3: epidemiology and laboratory skills, by gender and country of sponsorship Value 420 (of which 99 0 350 (quantitative women) or Qualitative) Date achieved 10/22/2013 06/30/2017 120 percent - target surpassed. However, original set target had no entry by gender. Below is the breakdown by gender: CCISD: 172 health workers / trainers trained (including 34 females) ✓ 135 R/CES including 24 females Comments ✓ 35 trainers including 10 females (incl. % achievement) ✓ 2 health workers trained in M&E with internally generated funds Fondation Mérieux: 248 people trained including 65 females; ✓ 198 Laboratory technicians (including 51 females) ✓ 50 trainers including 14 females ✓ Cabo Verde (25 including 12 females); Côte d'Ivoire (72, including 15 females); The Gambia (20, including 5 females), Ghana (60 including 7 females), Guinea Bissau (22 including 8 females); Nigeria (81 including 23 females); Niger (18, including 4 females); Togo (6, with no female). Number of regional simulation exercises for outbreak response by WAHO Indicator 4: Value 0 7 0 (quantitative or Qualitative) Date achieved 10/22/2013 0 – target not achieved Comments (incl. % achievement) Activity not carried out due to insufficient funds. Direct beneficiaries (number), of which female Indicator 5: Value no entry 410 708 (168 females) (quantitative or Qualitative) Date achieved 10/22/2013 06/30/2017 Comments (incl. % Approximately 173percent – target surpassed. achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 12/23/2013 Satisfactory Satisfactory 0.00 2 06/03/2014 Satisfactory Satisfactory 0.30 3 11/25/2014 Satisfactory Satisfactory 2.31 4 05/15/2015 Satisfactory Moderately Satisfactory 2.49 5 11/23/2015 Satisfactory Satisfactory 4.72 6 06/27/2016 Satisfactory Satisfactory 6.90 Moderately Moderately 7 08/01/2016 6.90 Unsatisfactory Unsatisfactory 8 11/03/2016 Moderately Satisfactory Moderately Satisfactory 7.68 9 06/03/2017 Satisfactory Satisfactory 10 H. Restructuring (if any) Not Applicable I. Disbursement Profile – . 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Regional Context. The project was prepared at a time when infectious disease outbreaks were on the rise in the Economic Community of West Africa States (ECOWAS) member countries1, and taking a particularly heavy toll among the populations in the West African sub-region. In these countries, epidemiological and economic evidence showed the urgent need to improve regional and global disease control mechanisms. Epidemic-prone diseases frequently encountered in the sub-region include meningococcal meningitis, yellow fever, Lassa fever, cholera, poliomyelitis, measles, dengue fever and Influenza A(H1N1) virus. Fourteen of the Fifteen ECOWAS countries experienced at least one epidemic outbreak during the period 2008-2009; half of these countries experienced two outbreaks. Between 2005 and 2009, almost half a million cases of epidemic-prone diseases were recorded, including meningitis (178, 681), measles (150, 753), and cholera (13, 9580). About 18,000 of these cases were fatal, and more than 70percent of the victims were children under 15 years. From 2008 to 2009, the number of meningitis cases almost tripled. (ECOWAS, 2009) 2. The Integrated Disease Surveillance and Response (IDSR) strategy was instituted by the World Health Organization Regional Office for Africa (WHO/AFRO) in 1998 to promote rational use of resources for disease surveillance and response (DSR) by integrating and streamlining common disease surveillance activities of countries in the region. The IDSR strategy was endorsed and adopted by the health ministries of the respective African country governments using a district health approach, a model that has proven to be very successful for evaluating poliomyelitis (polio) eradication efforts, monitoring immunization programmes, detecting emerging and reemerging infectious disease outbreaks, and for evaluating disease control measures. However, a 2008 assessment carried out in the ECOWAS member countries showed a lack of qualified and dedicated human health personnel in surveillance at all levels of the health system, and the lack of trained personnel at the district level was reported to be a major obstacle affecting the successful implementation of the national IDSR strategy 3. Rationale for Bank involvement. In May 2010, the President of the ECOWAS Commission requested the assistance of the Africa Region of the World Bank (WB) to develop a regional project to support disease surveillance and epidemic response in the sub-region.2 The project aimed to support strengthening the capacity of the West Africa Health Organization (WAHO), and the WHO/AFRO to design and cost a regional disease surveillance and response system for ECOWAS member countries in order to meet their obligations under the IDSR strategy and the WHO International Health Regulations (IHR, 1 Benin, Burkina Faso, Cape Verde, Côte-d’Ivoire, Gambia, Ghana, Guinea, Guinea -Bissau, Liberia, Mali, Niger, Nigeria, Senegal, Sierra Leone, and Togo. 2 Letter from H.E. James Victor Gbeho to Africa Region Vice President, Obiageli Katryn Ezekwesili (May 6, 2010). 1 2005)3, and to enhance the capacity of countries in diagnosis, disease surveillance and response in the sub-region. 4. Although a larger scale (US$150 million) IDA-financed regional integration project, focusing on only 3 (Burkina Faso, Mali and Senegal) of the 15 ECOWAS member states was initially proposed, following the project concept note (PCN) review the WB decided to begin with a smaller scale regional investment which could inform the development of future larger-scale investments. The African Catalytic Growth Fund (ACGF) was used to finance the WARDS project and contributed to filling immediate funding gaps for priority disease surveillance activities in the sub-region. The project design was narrow in scope, with a limited budget of US$10 million. However, it aimed to facilitate the development of a larger scale operation in DSR; to lay the foundation for the development of a regional surveillance network of a global public good nature to enhance core IHR capacities; to further reinforce partnerships in DSR in the sub-region; and to demonstrate the priority and political commitment afforded to the issue by the ECOWAS member states through the mobilization of US$ 1.5million in counterpart funding. At the time of project preparation, the ACGF was under review for an extension and a replenishment of the trust fund, a process that had a significant impact on the length of time between project preparation and effectiveness (as further discussed in section 2 under project preparation). 5. Linkage to relevant strategies. The project was fully aligned with the three key objectives of the Bank’s 2009 Regional Framework for Communicable Diseases Control and Preparedness (RFCDCP), which were to: (a) develop regional integrated multi-disease surveillance and response capacity; (b) strengthen regional capacity for laboratory diagnosis of infectious diseases; and (c) strengthen regional institutions and networks for inter-country collaboration. Overall, the project aimed to promote collective action and cross border collaboration in the sub-region, and to increase investments in regional public goods in line with the World Bank Regional Integration Assistance Strategy for Sub- Saharan Africa (RIAS) Strategic Pillar III – Coordinated Interventions to Provide Regional Public Goods. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 6. Project Development Objectives. The project development objective (PDO) was to strengthen the regional disease surveillance and response system of ECOWAS member countries. 3 The WHO International Health Regulations (IHR 2005) requires country governments to develop, strengthen, and maintain the core capacities of national public health systems to detect, assess, notify, and respond promptly and effectively to health risks and public health emergencies of international concern; revisions to the IHR (2005) became effective in June 2007 2 7. The key outcome indicators4 were as follows: i. Percentage of targeted districts which submit complete and on-time reports for selected priority diseases using the standard reporting format ii. Health personnel receiving training (number) – of which percent female iii. Percentage of short term trainees in field epidemiology or laboratory skills who are working at district 12 months after completing training, by gender and country of sponsorship iv. Completion rate of FELTP trainees by gender and country of sponsorship v. Semi-annual regional disease surveillance bulletins published by WAHO and disseminated to ECOWAS Member States 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification N/A 1.4 Main Beneficiaries, 8. At the country level, the direct beneficiaries of the project included health technicians working in the statistical and epidemiological surveillance and laboratory services within the national and district health systems of the participating countries (specifically 708 people, of which 168 were women). A subset of the populations of the 15 countries of the ECOWAS were also identified as indirect project beneficiaries. 9. At the regional level, WAHO was a direct project beneficiary with project support to the enhancement of its institutional capacity for regional coordination and project management. 1.5 Original Components (as approved) The Project comprised 3 main components as follows: 10. Component 1 – Regional Capacity strengthening (indicative cost: US$4.96 million). The identified activities under Component 1 included: (a) Establishment of functional centers for epidemiological surveillance (CES) in targeted districts in all ECOWAS member countries; (b) Assessment of the core capacities to implement the international 4 although PDO indicators ii, iii, and iv could have been better defined to avoid sounding repetitive, the decision of the task team to use these three indicators under the project served to properly capture and measure project achievements with enhancing training capacity for DSR as it relates to training accessibility, completion, and job retention for utilization of newly acquired skillsets. 3 health regulations in two remaining ECOWAS countries where such an assessment had not been carried out at the time of appraisal (Guinea Bissau and Senegal); (c) Facilitation of the revision of national disease surveillance and response strategies and plans in line with the 2010 WHO’s Technical Guidance for IDSR in all ECOWAS countries; (d) Organization of regional simulation exercises on outbreak preparedness and response; (e) Documentation and sharing of lessons learned from inter-country/cross-border collaboration initiatives; (f) Regular publication and dissemination of an epidemiological bulletin for ECOWAS; (g) Harmonization of tools for data collection and reporting; (h) Creation of a regional platform for electronic information sharing; (i) Harmonization of training modules on disease surveillance and laboratory techniques for personnel of centers of epidemiological surveillance at district level; (j) Facilitation of exchanges among epidemiologists and other disease surveillance personnel in the ECOWAS sub-region; and (k) Enhancing the institutional capacity of WAHO for regional coordination and supervision. 11. Component 2 - Strengthening Human Resources (indicative cost: US$4.76 million). The activities supported under this component included: (a) Long-term specialized training at the masters level in disease surveillance and response building on the experience of the Field Epidemiology Training Program (FETP) and the FELTP programs; (b) Short-term and on the job training of frontline health workers at district level on disease surveillance using harmonized curricula based the Projet d’Appui en Surveillance Epidémiologique Intégrée (PASEi) approach to the development of CES; and (c) Short-term and on the job training of frontline health workers at district level on laboratory techniques using harmonized curricula based the RESAOLAB initiative of laboratory training of personnel at district level and the strengthening of national and regional laboratory. 12. Component 3 - Management Support (indicative cost: US$1.78 million). Under this component, the project aimed to strengthen the capacity of WAHO in project management through the following: (a) recruitment of a project manager, financial specialist, procurement specialist, and M&E specialist for a limited period of time until these functions could be taken over by WAHO regular staff; (b) provision of technical and financial support to enhance M&E strategies and implement project monitoring, including field supervision, and evaluation activities; and (c) participation of key staff in regional simulation exercises and other regional information sharing and learning opportunities. 13. Overall, the project aimed to: (i) support the development of a framework and operational strategy for a regional disease surveillance and response system, including specimen management; (ii) develop an integrated regional health information management system; (iii) develop a resource mobilization strategy to support the operationalization of the regional disease surveillance and response system; and (iv) increase the quantity and quality of human resources within the West African sub-region for field epidemiology and laboratory diagnostics. The project financed goods (e.g., vehicles, computers, training materials, and office equipment), consultants’ services, operating costs, and training programs. 4 1.6 Other significant changes N/A 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 14. Project Preparation. The project was prepared in approximately 2 years using an Investment Project Financing mechanism with funding received from the ACGF5. The unusually long preparation timeline was a consequence of internal Bank processes with the administration of the ACGF, which was initially scheduled to end within a year of project preparation, and thus made it necessary for the task team to hold off on moving forward with project preparation until the ACGF was extended and replenished. Upon replenishment of the project financing source in April 2013, the task team moved relatively quickly with the project preparation phase leading up to board approval, and project effectiveness. 15. Project preparation was carried out using a participatory approach, and included formal engagement of key stakeholders including the implementing agency (WAHO), and other development partners [the World Health Organization, Regional Office for Africa (WHO/AFRO) and Intercountry Support Team for West Africa (WHO/IST/WA); the United States Agency for Internal Development (USAID), the United States Centers for Disease Control (CDC); and the Canadian International Development Agency (CIDA)] in the consultation process to further understand the disease surveillance and response landscape, to clarify roles and responsibilities, and to minimize duplication of similar efforts in place by other partners. 16. Project Design. The design of the project components appropriately reflected what the project intended to achieve with regards to strengthening the regional disease surveillance and response systems of ECOWAS member states. The geographical scope of the project covered all fifteen ECOWAS member countries. Due to the lack of International Development Agency (IDA) funding for DSR at the time, the US$ 10 million ACGF provided an opportunity to support regional and country efforts with enhancing DSR capacities by providing a regional platform to test out innovations vis-à-vis the technical and operational approaches utilized under the project in alignment with ongoing country programs in support of the implementation of the IDSR strategy, and to inform the design of future Bank operations. The rationale for Bank engagement using a regional integration window of the ACGF was appropriate to: (a) support the efforts of ECOWAS and WAHO to promote collective action and cross-border collaboration among ECOWAS member states; (b) strengthen the capacity of WAHO to support the provision of a regional and 5 Project concept note review took place on April 29, 2011; Pre-appraisal was conducted from June 11-15, 2012; QER meeting was held on July 06, 2012; followed by an appraisal mission from July 30 - August 10, 2012; and Project was approved by the RVP on May 03, 2013. 5 global public good in a sustainable way; and (c) establish the mechanisms for deploying institutional arrangements to support policy sharing, and advocacy for DSR in a regional grouping. 17. Initially, two key risks identified were (i) the lack of ownership of regional projects by countries, with a failure to reach a consensus on similar priorities during the design phase, and (ii) the identification of institutions with the appropriate capacity to serve as centers of excellence. As a mitigation measure, the task team sought and secured endorsement of the project by the Ministers of Health of the ECOWAS member states to ensure ownership of the regional capacity building for IDSR. The project was also aligned with the 2010 Strategy for Accelerated Growth and Sustainable Development (SCADD) of the Government of Burkina Faso6, the host country of the WAHO. However, given the regional nature of the project, some development priorities were known to differ across the ECOWAS member states, which contributed to the lack of project ownership described in the preceding paragraph, and consequently, a low request for support across countries during the annual work planning process. In addition, the scope of the project, budget implications, broadness of the PDO, and choice and formulation of some indicators were not thoroughly considered during project preparation. This represents a shortcoming of the project design as is evident from the proposal to drop an intermediate outcome indicator (IOI 4: number of regional simulation exercises for outbreak response by WAHO) during the annual budget work planning process, and the late considerations to restructure the project during the implementation phase. These factors should have been properly assessed at the time of appraisal to provide more clarity on what the project specifically sought to accomplish and the targets set in the results framework. In this regard, the task team should have included an explicit definition of what constituted a strengthened disease surveillance and response system at both country and regional level. 18. Quality at Entry. The direct recipient of project funds was ECOWAS. A key outcome of appraisal was the agreement to have WAHO as the implementing agency, and for a separate Project Agreement to be entered between the Bank and WAHO, with funds transferred directly from the Bank to WAHO through a subsidiary agreement between WAHO and ECOWAS. In turn, WAHO would enter into sub-contractual agreements with six implementing partners (IPs) [Centre de Coopération Internationale en Santé et Développement (CCISD), CDC, Fondation Merieux, University of Ghana, University of Oslo, and WHO/AFRO] selected via a single-source procurement selection method for the implementation of project activities in all fifteen ECOWAS member countries; four of the six IPs (CCISD, CDC, Fondation Merieux, and WHO/AFRO) were identified during the project preparation phase for the provision of technical assistance to WAHO for training activities under the project. However, delays experienced with the process of obtaining signatures for the Memorandum of Understanding (MOU) between WAHO and ECOWAS member countries had a negative impact on the smooth commencement of implementation of project activities (as discussed further in the implementation section). 6 At the time of project preparation, Burkina Faso was rated as a top IDA performer with a 2009 CPIA score of 3.8 6 19. Given the unfamiliarity of WAHO with Bank procedures and the limited experience of the WHO/AFRO with implementing Bank projects, the fiduciary capacity of both institutions was a major cause for concern at the time of project appraisal, which was confirmed during the fiduciary assessment carried out at appraisal. In retrospect, all key risks including the limited capacity of the implementing agency and the risk of misuse of grant funds due to a limited knowledge of the World Bank FM and procurement guidelines at WAHO were carefully considered with appropriate mitigation measure put in place including establishing an adequately staffed project management unit (PMU) at WAHO to properly handle the project management functions, and fiduciary processes under the project as laid out in the Operational Risk Assessment Framework of the PAD. 2.2 Implementation 20. The implementation arrangement was rather complex with WAHO serving as the implementing agency (host of the PMU), and the overall coordinator for the six IPs responsible for the implementation of various activities under the project at country level. Despite the mitigation measures put in place to address the inexperience of WAHO with the management of Bank-financed projects, taking on this role was a challenging task especially apparent during the first half of project implementation. To address institutional capacity gaps identified during project appraisal, the first implementation activity of the project (as a condition of effectiveness) was the recruitment of a project coordinator, a procurement specialist, a financial management (FM) specialist, and a monitoring and evaluation (M&E) specialist to be staffed at the newly established PMU housed at WAHO. 21. All necessary steps taken by the project task team to adopt appropriate measures to strengthen the institutional capacity of WAHO as a PMU were carried out in an efficient manner. However, there were some delays with the commencement of implementation of project activities (post-project effectiveness) by the IPs that resulted in delays with the transfer of the first installment of the project funds to the project account. Specifically, this was due to hold ups with the receipt from WAHO of eligible signature of contracts for consulting services that were to be offered by the IPs, and to the turnover of the FM specialist assigned to the project in Ouagadougou to provide close support to WAHO on FM-related issues. 22. A commendable achievement leading up to the start of implementation was the transfer of the first instalment of counterpart funds (briefly discussed in the context section above) by WAHO to the designated project account before the transfer of Bank funds were made available; the first transfer of funds from the WB in the amount of US$ 300,000 into the designated account was received on May 22, 2014, over two months after WAHO had transferred its first tranche of counterpart contribution in the amount of US$ 250,000 on March 7, 2014. In total, WAHO contributed US$ 1.5 million in counterpart funding, doubling the amount initially agreed upon (US$ 0.75 million) during the design of the project. This showed a high level of dedication on the part of the implementing agency to the smooth implementation of project activities throughout the duration of the project. 7 23. As the implementing agency, WAHO displayed a high level of transparency from the commencement of implementation by putting in place an agreement with the ECOWAS member countries for the coordination of the implementation of country level activities under the project by the IPs. However, the perceived lack of project ownership displayed by some countries during the project preparation stage was confirmed at implementation, and during supervisory visits to the countries. Despite the role of countries in specific areas of the project design (such as in the choice of districts to be supported, and the selection of FELTP residents to be trained), government representatives of these countries felt that they had no control over the use of funds in their countries, and thus were not as committed to the project and the achievement of the PDO. A key explanation provided by the task team for the lack of country ownership is due to the regional nature of the project with the implementation of project activities coordinated by a regional body (WAHO) hence no arrangements were in place for countries to have direct access to project funds for the implementation of country-level activities. 24. The Bank procurement team worked closely with WAHO to streamline the hiring process for the selection of the six IPs under the project. By the first year of implementation (November, 2014), the project had recorded some achievements with the roll out of activities across each component of the project by five of the six IPs including the operationalization of the Centre de Coopération International en Santé et Développement (CCISD) coordination office in Bobo-dioulasso, Burkina Faso, and Dakar, Senegal; assessment of the core IHR capacities in the two countries without a prior assessment (The Gambia, and Senegal), and the commencement of masters level training of 10 FELTP residents in the University of Ghana. However, there were reported delays experienced by Fondation Merieux with the provision of training to district level laboratory technicians in disease surveillance under component 2 of the project. 25. The first project disbursement was carried out on May 22, 2014, and before the end of the first year, the project had recorded some progress in meeting the year one disbursement target of US$3.9 million; as of September, 2014, total disbursement figure amounted to US$2.3 million. In addition, the project also disbursed one-third (US$500,000) of the counterpart funding contribution during the same timeframe. By May, 2015 disbursement rates of the project, which was at 25percent at the time, stalled remarkably due to procurement delays resulting from complications with purchasing and the shipment of goods (i.e. vehicles) from one country to another despite existing customs agreement between ECOWAS member countries. 26. By the next supervision mission carried out in November 2015, implementation of project activities had progressed significantly with a reported disbursement rate of 47percent. During the period observed, it was reported that one of the five IR indicators (number of regional level simulation exercises conducted by WAHO) could not be achieved due to limited funds and thus, the activity was not included in the final project work plan. The task team resolved to consider dropping the indicator by midterm review (MTR). In hindsight, the acknowledgement of the unrealistic indicator included as part of the project design under a limited project financing from the ACGF presented a good opportunity for the task team to have carried out a restructuring of the project without 8 having to wait for MTR, and thus reflects a shortcoming of the task team specifically as it relates to project supervision. 27. The MTR was originally scheduled for December 2015 but was delayed to March and April 2016 due to political issues affecting Burkina Faso, host country to WAHO, at that time. At MTR, the project was faced with significant challenges; four of the five key project ratings (progress towards achievement of PDO, overall implementation progress, regional capacity development, and strengthening human resources) were downgraded from a rating of S to MU (during this mission, the decision to drop the IR indicator that measured the number of regional level simulation exercises conducted by WAHO was also confirmed). At MTR, the project had disbursed 59percentpercent of project funds, while the rest of the funds were committed towards activities to be implemented by the IPs, which recorded varying progress with the implementation of project activities due to significant delays experienced by some IPs with the implementation of project activities. 28. Delays with the receipt of the signatures for the MOU between WAHO and the ECOWAS member countries also had a negative impact on country ownership, especially regarding duty free privileges for the procurement of goods such as vehicles, and related customs regulations in each country. Without these MOUs country governments (especially the Ministries of Finance) did not acknowledge familiarity with the WARDS project in their respective countries, and were therefore reluctant to warrant duty free privileges for imported goods and equipment procured under the project although the implementing agency of the project, WAHO, is a specialized agency of ECOWAS, However, it is important to note that issues surrounding duty free privileges persisted after the MOU signatures were received. Another issue was the lack of an effective communication strategy for the project on how best to promote the project across ECOWAS member countries and to advocate for disease surveillance as a priority agenda for the various country governments, which contributed to the perceived indifference displayed by some countries to the project. To address this issue, an effective communication strategy was developed and put in place by the second half of project implementation to promote the visibility of the project in the sub-region, which led to an increase in the visibility of the project among the stakeholders, and DPs, and emphasized the importance of strengthening and sustaining disease surveillance capacities in the sub- region. 29. As mentioned above (in paragraph 25), political issues in Burkina Faso, (the host country of the implementing agency) justifies the decision of the task team to postpone the MTR from December, 2015, and to have a two-phased MTR mission from March 10-11, 2016 (in Dakar, Senegal), and from April 25-26, 2016 (in Annecy, France) due to the prevailing political climate in the country. The MTR mission reconfirmed that the PDO was justified to maintain its rating of high as it remained highly relevant to the priorities of the ECOWAS member countries, especially considering the 2014-2015 Ebola Virus Disease (EVD) outbreak, which exposed weaknesses in the disease surveillance systems in many countries in the sub-region. Despite the high PDO rating, a key recommendation from the MTR was to re-word the PDO to better reflect the activities of the project within the allocated financing of US$10 million. Consequently, there were two proposed revisions 9 to the PDO as follows: (i) “the project development objective is to increase the human and technical capacity for disease surveillance at country and regional level in ECOWAS region”; and (ii) “the project development objective is to increase selected technical capacity in the field of epidemiology and disease surveillance at country and regional level in ECOWAS space”. 30. Following the proposals to reword the PDO, a key decision of the MTR was to restructure the project to: (i) change the wording of the PDO to better reflect what exactly the project aims to achieve, and could support, and (ii) amend the Results Framework by revising the PDO and intermediate results (IR) indicators, and to ensure that the new indicators relate to competencies in both epidemiology and disease surveillance while also reflecting country and regional levels of interventions. 31. At post-MTR (November 2016), the project was back on track with achieving the PDO by the expected project end date (June 30, 2017). However, there remained concerns about the broadness of the PDO and the need to restructure the project, as confirmed during the MTR mission. To address these concerns, the task team initiated the process of obtaining approval, from the country management unit (CMU), of a restructuring package in November, 2016. However, the approval was not processed for two key reasons: (i) the final restructuring request was reviewed about four months before the expected close date of the project; (ii) the project had recorded an increase in performance post-MTR with an upgrade of four key project ratings to satisfactory, and significant achievements with meeting the PDO and IR indicators as laid out in the results framework. 32. Although a restructuring of the project early on during the implementation phase would have presented the task team with a good opportunity to properly address the issue around the broadness of the PDO and the formulation of some indicators, in hindsight, a restructuring of the project at an earlier time would have been challenging for the task team and the implementing agency given the diverted attention of key stakeholders towards the provision of emergency response to containing the spread of the 2014-2015 EVD outbreaks in the sub-region that occurred around the same time period as the start of full implementation of project activities (refer to Box 1 below on WARDS response to EVD outbreak in West Africa). This could also have contributed to the refusal of the CMU to process a restructuring package given a response of the WB to enhancing the capacity of ECOWAS member states for disease surveillance, preparedness and response was the preparation of the Regional Disease Surveillance Systems Enhancement (REDISSE) program 7 under the regional integration portfolio. 7 The REDISSE program is a series of interdependent projects involving the ECOWAS member countries, which promotes a one health approach that provides a platform for high-level policy and regulatory harmonization, cooperation, and coordination between the animal health and human health sectors within and across countries for the earlier detection of infectious diseases, and a more effective response to infectious disease outbreaks. 10 33. By the project end date, the project had recorded a 100percent disbursement rate, and all key project ratings were rated as S. Box1: WARDS Project Response to the 2014-2015 Outbreak in West Africa In the first year of the WARDS project implementation, the Ministers of Health of the fifteen Member States of ECOWAS met in Monrovia, Liberia from April 11-12, 2014 at the 15th Ordinary Assembly of Health Ministers to express their deep concern at the challenges posed by infectious disease outbreaks resulting in epidemics in the ECOWAS sub-region. During the meeting, a call was made to technical and financial partners to continue to intensify the support provided to countries in the sub-region, and the ECOWAS commission was urged to provide additional resources towards the establishment of a community fund for rapid response to public health emergencies in West Africa. This call was before the first transfer of fund from the Bank into the designated account and so the project could not contribute in kind to the EVD response. However, activities under the WARDS project helped improve the readiness of countries to respond to future outbreaks via the rapid deployment of well-trained Ministry of Health staff under the FELTP program to support outbreak surveillance and investigations. These outbreaks are communicated to WAHO in a timely manner via the regional DHIS2 platform developed under the project, which is also utilized for the publication and dissemination of routine weekly epidemiologic bulletins published by WAHO using data transmitted from all ECOWAS member countries. The WARDS project has also laid the foundation for the REDISSE program, which scales up some activities under the WARDS project in support of enhancing disease surveillance and response capacities across ECOWAS member countries and in the sub-region including the short-term and long term training in epidemiology and laboratory techniques, and a scale-up of the number of CES supported in the region from 60 to 207 with funding of Canadian$20 million from a trust fund supported by the Canadian International Development Agency (CIDA). 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 34. M&E Design and Implementation. An M&E plan was developed prior to the commencement of implementation. A regional platform for electronic information sharing called the WAHO – OOAS Regional Data Warehouse, and based on the District Health Information System 2 (DHIS2) software application was developed under the project for WAHO with technical assistance provided by the University of Oslo. 8 A visit to the CES in Richard-Toll, Senegal (in February 2017) by an external evaluation team from the Division of Laboratory at the central health level showed that improvements are needed to the DHIS2 platform, specifically with regards to making the system interoperable with other databases, and improving the simplicity of the data collection process to make usage of the platform less time-consuming. 35. The choice of key performance indicators utilized under the project appropriately captures, and is highly relevant to the PDO to strengthen the regional disease surveillance and response system of ECOWAS member countries using a catalytic funding source of 8 The platform is accessible online at: http://venus.dhis2.org/dhis/ 11 US$10 million. However, the absence of an M&E Specialist on the team throughout the project lifecycle impacted on the quality of the results framework. 36. M&E Utilization: The information sharing platform implemented by the University of Oslo has been in place and is operational. The country teams responsible for health data management have been trained to use the DHIS2. Fourteen out of the 15 countries of the ECOWAS regularly send data through the platform. Furthermore, the harmonization of automatic country data collection and transmission tools to the WAHO regional platform is in progress. This activity was implemented with additional funding from USAID, and the data is used to produce weekly and quarterly bulletins. ECOWAS countries requested from WAHO that the weekly and quarterly bulletins replace the semi- annual reports adopted in the WARDS project. By the end of the project, WAHO had produced a total of 80 weekly epidemiological bulletins, 3 quarterly newsletters, and 2 semi-annual bulletins for publication and dissemination to ECOWAS member states to provide up-to-date reporting on infectious disease outbreaks across member states and to better inform decision making for outbreaks detection, and rapid response. 2.4 Safeguard and Fiduciary Compliance 37. Safeguards. There were no safeguards policies triggered under the project. The project safeguards was categorized under Category C. 38. Procurement. The procurement rating for the project was rated as S over the course of the project (except for an MS rating), with an S rating recorded at the time of project closure. Despite initial procurement delays, procurement for the WARDS project was completed with an implementation rate of 100percent. The procurement plan approved for the project included 12 procurement items totaling US$ 10,270,000 of which almost all have been subjected to prior review. The post-review for the purchase of equipment was at US$251,540 or 2.4percent of the procurement. The procurement plans were financed with World Bank funds and WAHO counterpart funding, which explains the excess of US$ 10 million allocation quoted above for the procurement items. 39. During the first year of project implementation, two key procurement challenges were reported; the first was with regards to procured goods specifically related to the difficulties experienced with transferring procured vehicles, computers and other goods to the CCISD Regional Coordination Offices due to incumbent country customs procedures. The WAHO sought the guidance of the ECOWAS commission to address this issue within the countries. The second was the unavailability of some members of the Tender Committee for the opening sessions of the bidding process. Further procurement delays stalled the implementation of activities under components 1 and 2, which were also brought about by the low disbursement ceiling for the project set at US$ 300,000 in the disbursement letter (DL); the low disbursement ceiling was put in place as a risk mitigation measure due to the inexperience of WAHO with managing a WB-financed project prior to the WARDS project. Despite the delays with the procurement of goods at the beginning of the project, the six IPs implemented all assigned activities in their respective contracts by the close of the project. 12 40. WARDS project procurement activities were verified in two external audits, and findings from these audits were presented to the World Bank and declared acceptable. 41. Financial Management. The FM rating for the project was rated as Satisfactory over the course of the project (except for an MS rating), with a Satisfactory rating recorded at the time of project closure. 42. For budget planning purposes, a streamlined work planning process was adopted under the project with the proposed project budget developed for the entire four years of the project to provide a clear sense of activities to be carried out for the duration of the project. The first FM implementation support mission for the project was carried out in May, 2015. At the end of the mission, the overall performance of the project in financial management was rated satisfactory. Key observations were reported as follows: (i) the financial management staff remained adequate and the book-keeping was up to date, (ii) the quarterly interim financial reports were submitted on time and their quality rated as satisfactory, (iii) the disbursement rate of allocated funds was rated satisfactory. 43. Between 2014 to 2016, eleven (11) Interim Financial Reports (IFR) were completed and submitted to the Bank. Each of the IFR submitted was rated satisfactory. The IFR for 2014, 2015, and 2016 were completed by WAHO and presented for external audit. At the start of implementation, there were some recorded delays with the hiring of an external audit firm. However, the hiring process was eventually carried out and the first external final audit report was completed by June, 2015. Subsequently, all financial audits were completed and submitted to the world bank, published on WAHO website and certified as satisfactory by the world bank with no issue of ineligible expenses. 44. An internal audit was carried out by the internal audit department of the ECOWAS commission, which was complemented by ex-ante review conducted by the WAHO financial controller. By the end of the project, overall disbursement rate was 100percent and the project consumption rate was 100percent. 2.5 Post-completion Operation/Next Phase 45. The project satisfactorily achieved the objective of serving as a catalyst to mobilize additional funds for strengthening disease surveillance and response capacity in the sub- region of West Africa. As a catalytic project, the success of the WARDS project in leveraging additional financing of CAD 20 million from CIDA to scale up the establishment of new CES under the related REDISSE program highlights that the model developed under the project is a viable approach to mobilizing external funds for strengthening DSR capacities across ECOWAS member states. However, important lessons and its application to related operations can be drawn from the issues experienced under the project specifically as it relates to country ownership of activities carried out within country borders by non-government funded IPs. In addition, the WARDS project served as the first point of WB engagement with ECOWAS as a reliable partner for signing regional financing agreements, and in strengthening the institutional capacity of WAHO, as the PMU to manage larger operations including the regional level activities under the 13 REDISSE program, The REDISSE first series of project (REDISSE I)9 became effective prior to the closure of the WARDS project, which ensured continuity of funding for key activities such as support to existing CES, establishment of new CES, and support to WAHO for project management. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 46. The relevance of the project objectives is rated as high, design is substantial, and the implementation is substantial. Overall, the rating for project relevance is rated as substantial. 47. Relevance of Objectives: Since project effectiveness (December 2013), the West Africa region continues to be plagued with emerging and reemerging infectious disease outbreaks of epidemic potential including Avian Influenza (H5N1), EVD, Meningitis, Cholera, and Lassa Fever. The 2014-2015 EVD epidemic in the sub-region further emphasize the strong economic case for investing in regional disease surveillance and response systems. As mentioned above, the project is aligned with pillar III of the RIAS for the region (2008/rev 2011), building coordinated interventions to provide regional public goods. Under this pillar, the RIAS specifically references the establishment of regional and sub-regional programs to address the cross-border dimensions of disease prevention and control as a priority area. The project is also aligned with the priorities of development partners (DPs) to establish a regional health workforce to promptly respond to outbreaks and health emergencies, and the priorities of the ECOWAS member states to increase human resources for health capacity for surveillance, outbreaks investigations and rapid response by increasing the number of FETP and FELTP graduates in the sub-region based on the findings from a needs assessment conducted following the wake of the 2014- 2015 EVD outbreak.10 48. The relevance of project objectives is therefore rated as high. 49. Relevance of Design: The regional nature of the project was highly consistent with the PDO, and the coordination of implementation activities was carried out under the full responsibility of the WAHO. Each of the three project components were well designed to attain the PDO of strengthening disease surveillance and response system of ECOWAS member states. Although the stated PDO was broadly defined, the three project components were well designed with relevant activities under each component. For instance, activities that supported strengthening the management capacity of the WAHO under component 3 were very relevant to the attainment of the PDO by ensuring that 9 REDISSE I countries are: Guinea, Senegal, and Sierra Leone 10 The respective governments of the ECOWAS member states have set an optimal target of one trained field epidemiologist (or equivalent) per 200,000 population, who can systematically cooperate to meet relevant IHR and PVS core competencies. 14 WAHO is well equipped with the capacity to serve as a regional coordinating body for the efficient implementation of project activities at country level by the six IPs. In addition, given the project was mainly focused on the provision of technical assistance in core technical areas for strengthening disease surveillance and response systems across ECOWAS member states, the implementation arrangements adopted under the project using the respective expertise of the six identified IPs was relevant to the achievement of the PDO. 50. The relevance of project design is therefore substantial. 51. Relevance of Implementation: The catalytic project utilized the strengths of national and regional programs and institutions to implement the country-level and regional-level activities under each project component. WAHO was selected to serve as the primary implementing agency and the host of the PMU due to its technical capacity and earlier engagement at the country level on the implementation of the IDSR strategy including harmonization of legal frameworks, and strategy documents. In its role as the implementing agency, WAHO coordinated the implementation of the project activities by the six IPs across each project component in a relatively streamlined manner with the adoption of a harmonized reporting template that was used by each IP to provide quarterly implementation progress report to WAHO. However, there were some delays experienced with the implementation of project activities due to procurement issues (see section 2.2 above on project implementation). These issues were eventually resolved, and despite the initial impediment to implementation of activities, the project got back on track with the implementation of project activities as scheduled. 52. Consequently, the relevance of project implementation is rated substantial. 3.2 Achievement of Project Development Objectives 53. The achievement of the PDO (efficacy) is rated as substantial. 54. Considering both logic and evidence, the strength of a regional disease surveillance and response system is only as strong as the weakest link (the system of the representative member country under the regional body). Assessment of the achievement of the PDO will therefore account for project support to both regional and country level systems as follows: (i) strengthened regional disease surveillance and response systems; and (ii) strengthened country disease surveillance and response system of member countries under the ECOWAS commission. 15 Table 1: Percentage achievements of Results framework indicators Rating PDO indicators Intermediate Outcome Indicators Surpassed (100%+) 4 2 Achieved/Substantially (85%+) 1 2 Partially achieved (65%-84%) - - Not achieved (less than 64%) 0 1 Total 5 5 % surpassed and 100% 80% achieved/substantially 55. As shown in table 1 above, 100 percent of PDO level indicators were reported to have surpassed and achieved/substantially achieved their end of project targets; of the five PDO indicators, four indicators were reported as surpassed, and one as substantially achieved. Additionally, four of the five IR indicators were achieved or surpassed, except the indicator on the number of simulation exercises conducted in the region by WAHO. As noted at MTR, this indicator could not be achieved during the lifecycle of the project; with the formulation and inclusion of this indicator in the results framework, the task team had set an ambitious and unrealistic target considering the financing allocation of US$10 million, and the scope of activities supported under the project (to address this gap in achievement, two cross boarder activities that involved participants from four neighboring countries were conducted under the leadership of WHO to test the readiness of neighboring countries to respond to disease outbreaks). 56. Key achievements of the project with attaining the overall PDO to improve disease surveillance and response system across ECOWAS member states were reported under two outcome indicators: (i) indicator on percentage of target district CES which submit complete and on time reports for selected priority diseases; and (ii) indicator on the completion of long-term FELTP program. For the first indicator reported as surpassed, it was observed during the project Implementation Completion and Results (ICR) review that a different methodology was used in the computation of the indicator, which differed from the definition of the actual indicator; in this case, the number of reports submitted by the CES were reported for this indicator as opposed to the percentage of CESs which submit complete and timely reports. The second indicator target was revised at the beginning of implementation to a more modest target as agreed by every stakeholder involved but without an official change to the results framework11 . Hence, it was reported in the last ISR to have attained a substantial achievement (accounting for the 80percent completion rate reported in the ISR) when in fact, the achievement under the indicator was surpassed as presented in the ICR report (with a 96percent completion rate among the 50 residents trained). Project support to this activity has significantly contributed to filling some 11 At the beginning of the training, it was agreed with the US CDC that 30 residents will be trained under the francophone FELTP program as opposed to the originally set target of 40 (while 20 residents were trained under the Anglophone program). This was not formally revised in the results framework, and so was reported as substantially achieved in the last ISR. However, it is worth noting that of the 50 residents trained, there was a 96 % completion rate 16 existing gaps in the region with the availability of advanced level certified health personnel trained in disease surveillance and laboratory practices. 57. A similar indicator that measured the percentage of short-term trainees in field epidemiology and laboratory was reported in the last ISR of the project as not achieved at project closure due to a limited availability of information on the indicator because a core aspect of the indicator also sought to capture the continuous employment of the trainees by districts a year post-completion of the training program. Although this information was not available at the end of the project specifically due to some implementation delays with the commencement of the activity, this indicator is particularly relevant for highlighting the success of the project with achieving the PDO given that a key issue affecting human resources for health capacities in disease surveillance and response, especially at the district level, is the high turnover rate and lack of trained personnel sustained within the system to provide rapid frontline services for outbreak preparedness and response. Consequently, the analysis provided in the ICR has been carried out using the most recently available data (as of December, 2017) to track progress with the achievement of the targets set for this indicator by analyzing job retention rates among health workers trained in short-term field epidemiology or laboratory skills who had completed the mandatory 12-month period post- training completion. Of the 17 short-term trainees contacted, 82 percent reported working at the same district 12 months post-training completion, which represents a substantial achievement as compared to the originally set target of 80percent. On the assumption that a similar trend can be expected across all short-term trainees following completion of the mandatory 12-month post-training period, it can be concluded that the project has demonstrated its contribution to addressing retention issues and high turnover of trained health personnel. Similarly, under the long-term Anglophone FELTP program, the three FELTP residents from Liberia have taken up key roles in the newly created National Public Health Institute of Liberia. However, concerns were raised about the lack of a progress report to country ministries on the residents being trained to ensure future employment plans are thoroughly considered prior to the graduation of residents (this has been noted as an area to consider for future residents). 58. In the last project ISR, another indicator reported as not achieved was the indicator on the number of semi-annual bulletins published by WAHO with only two semi-annual publication reported compared to the set target of seven. The justification provided for this performance is that post-MTR, ECOWAS member countries made a request to WAHO that the weekly and quarterly bulletins replace the semi-annual reports as a more useful tool for keeping member countries updated on the status of disease surveillance and response in the sub-region. As a result, the semi-annual reporting arrangements in place for this indicator was replaced with weekly and quarterly reporting format. To appropriately reflect project achievements under this indicator, the analysis for the ICR extrapolates data from the total number of weekly and quarterly bulletins published and disseminated by WAHO (80 weekly, and 3 quarterly bulletins) to compute that an additional 4 semi-annual bulletins were published (based on the frequency of the actual activity carried out, and assuming the weekly bulletins serve the same purpose as the semi- annual bulletins). Using these figures, the indicator is reported to have been substantially achieved. 17 Was the disease surveillance and response system of the ECOWAS region strengthened? 59. The project has supported coordinated mechanisms currently being established to enhance disease surveillance and response systems in the region as a global public good. One of such direct efforts attributable to the project is related to supporting the scale up of the number of FELTP residents in the ECOWAS region to meet the set targets of at least one trained field epidemiologist in place per 200,000 population who can systematically cooperate to attain relevant IHR (2005) and PVS core competencies among member states (see Figure 2 on project results chain below). For instance, under the Anglophone FELTP program implemented by the University of Ghana, the 20 residents from the two cohorts funded under the project were deployed to various countries across the region to conduct a total of 41 disease outbreak investigations, which were mostly conducted during their second year in the program. However, residents of the francophone FELTP such as the Senegalese residents reported less opportunities for outbreak investigations in other countries, while other residents from the Anglophone cohort reported limited support over the course of the program (the issue of support provided to residents was not reported as a concern by residents under the francophone program).12 12 Other differences in the structure and curriculum of the two long-term FELTP training programs offered under the project highlights the need for a more streamlined FELTP training curriculum, and to improve overall program quality as discussed in the lessons learned section of the ICR 18 FIGURE 2: KEY RESULTS CHAIN OF THE WARDS PROJECT INPUTS: OUTPUTS: ACTUAL PROJECT OUTCOMES: UTILIZATION Strengthen regional and Development of the Use of DHIS2 platform for more effective country surveillance DHIS2 reporting system and efficient data management; surveillance and reporting platform data used for decision making and for 48 additional FELTP Increase Human Graduates available outbreak detection and rapid response resource capacity in across the ECOWAS Scientific information routinely shared with disease surveillance and member countries and peers, decision makers. the public and other response contribute to progress public health practitioners in the region and towards meeting target of globally. Strengthen the 1 trained FELTP per institutional capacity of Increase in the availability of trained health 100,000 population WAHO to contribute to personnel to support recent infectious the provision of a 543 trained health disease outbreaks (including H5NI, EVD, sustainable regional and personnel in and Meningitis) global public good epidemiology and Laboratory practices Increased investments in HNP projects Use of a catalytic supporting regional collaboration through ACGF financing Strengthened PMU with WAHO (Four WB investments; 25+ non- mechanism to inform 11 functioning staff WB investments managed by WAHO) the technical design of related large-scale 60 Centers for investments Epidemiologic QUALITY Surveillance with Support the provision of Surveillance and reporting systems improved technical epidemiologic services strengthened within a network of sentinel capacities from the district to the districts central level across 80 weekly and 3 quarterly Ministries of Health Staff demonstrate countries in West Africa bulletins published and mastery of the FETP core competencies disseminated 60. The support provided to the establishment of 60 CES (including the scale-up of the number of CES in the region from 60 to 207 using additional funds mobilized from CIDA under the REDISSE program), and to the training and graduation of 48 new FELTP residents in the region who serve as a resource available to all member states for disease surveillance, outbreak preparedness, and rapid response is evidence of the strengthened capacity in the region with a cadre of health professionals equipped with advanced skills in applied epidemiology and laboratory management. 61. Project financing also helped with improving the availability and reporting of regional level data on the state of disease surveillance in the sub-region via a regional platform for the analysis, harmonization, and transmission of electronic surveillance data (DHIS2), which are published and disseminated by WAHO to member states on a weekly basis and serves to promote the importance of improving evidence-based policy making for disease surveillance as a priority agenda for country governments and the ECOWAS commission. In contrast, the core regional level intermediate outcome indicator that sought 19 to directly provide further evidence of the impact of the project’s support to enhancing outbreak preparedness and rapid response at project completion was not achieved due to lack of funding, and concerns regarding the effectiveness of the intervention as a priority activity.13 Other evidence of the strengthened regional disease surveillance and response system brought about by the project include the enhanced institutional capacity of WAHO to serve as an effective regional institution capable of supporting efficient coordination and collaboration among ECOWAS member states using its unique convening power. Additionally, support to activities under the WARDS project has helped improve the readiness of ECOWAS member countries to respond to future outbreaks, and has resulted in the development of a regional platform for information and data exchange across countries, which are fully operational across all countries to provide rapid response to outbreaks in the sub-region. For example, trained staff under the FELTP program were deployed in Nigeria during the last Lassa Fever outbreak and in Niger during the Rift Valley Fever outbreak in 2016. These outbreaks were communicated to WAHO timely through the use of the routine weekly epidemiologic bulletin publication channels by WAHO using data transmitted from ECOWAS member States Was disease surveillance and response systems across ECOWAS member countries strengthened? 62. As mentioned above, project support to the establishment of 60 CES in target districts across member countries of the ECOWAS region has contributed to improving the mechanisms in place to report surveillance data by district health personnel to the central level. At the country level, adoption and use of the DHIS2 platform has led to more efficient and effective management of surveillance data. During a visit to Hohoe municipality in the Volta region of Ghana (a host district of the CES) in April 2017, district health managers, disease surveillance officers, and other staff interviewed displayed improved knowledge about the utilization of the DHIS2 platform and the Epi info software to generate country level surveillance data and for data analysis respectively, and the application of surveillance data for public health action including for laboratory confirmation of suspected cases, early notification to line ministries, and rapid response to disease outbreaks. Similar findings with the surveillance systems where reported at the site visit to the CES in Richard-Toll, Senegal during the same period. 63. The WARDS project has also contributed to strengthening the capacity and increasing the availability of trained health personnel in field epidemiology and laboratory practices under four different categories of training programs supported under the project: (i) 2-year Master’s level program; (ii) 3-month basic FETP course; (iii) short course on field epidemiology; and (iv) short courses in laboratory practices 64. In addition to serving as a resource for the ECOWAS region for outbreak preparedness and rapid response, the FELTP graduates trained under the project also 13 As mentioned in the implementation section above, a similar type of activity was carried out in lieu of the regional simulation exercise to test the outbreak preparedness capacity of the region. 20 support their respective country governments with facilitating the basic training program provided to frontline district health workers in disease surveillance and laboratory management by serving as resident advisors. The three-month long basic training program modeled after the FELTP has contributed to the enhancement of human resources capacity to support disease surveillance and response activities at the district level. Over the course of the project, residents and alumni of the FELTP have been mobilized by their country governments to participate in various disease outbreak investigations. For instance, several outbreak investigation teams comprised of the FELTP residents were deployed in response to outbreaks of Avian Influenza (H5NI) in Kpone Katamanso district, Greater Accra region, and Bawjiase Akpeteshi Camp, Central Region of Ghana between May 2015 to July 2016; EVD in the Maryland and Montserrado counties of Liberia between 2015 to 2016; and Meningitis in Nadowli-Kaleo, and Jirapa district, Upper West region of Ghana from January to March, 2017. The residents effectively displayed core competencies in field epidemiology including contact tracing, education of contacts on mode of transmission and prevention, sample collection and laboratory confirmation, and sensitization of health facilities in the affected districts for the implementation of effective control measures. 65. Many of the FELTP graduates have been retained in the position held prior to the start of the program, while other graduates have been offered higher positions within the governments and serve as FELTP mentors for incoming cohorts. At the time of the last supervision mission, WAHO had commenced discussions with the respective country governments of some residents to assign the residents jobs with higher level of responsibilities following graduation from the program. One resident had a confirmed assignment to work on the rapid response team of the sub-region; a second resident with the emergency operation center (EOC), and another resident was offered the position of a Technical Advisor to the frontline response team in Togo. 66. The quality of training supported under the project are exemplified in the acceptance of FELTP residents to present their work in several scientific conferences; 29 abstracts were accepted for presentation at the Africa Field Epidemiology Network (AFENET) conference, and an oral presentation was made by an FELTP graduate at the US CDC Epidemiologic Intelligence Scientific Conference, in August 2017. A total of 8 mini grants were also awarded by the Training Programs in Epidemiology and Public Health Interventions Network (TEPHINET) to the FELTP residents supported under the project highlighting the relevance of the program and its self-sustaining capacity to train public health leaders in field epidemiology as nominated by their respective country governments. 3.3 Efficiency 67. The economic costs of disease outbreaks are significant and preventing these outbreaks yields large economic benefits by reducing the threats of epidemics and pandemics. Besides the negative economic impact due to mortality and morbidity and health care costs, disease outbreaks affect economies at national and global levels by decreasing production, investment, exports, and personal income as well as reducing labor, capital, and productivity, which are all major components of growth (UNDP, 2014). Their 21 increasing toll on productivity, profitability and foreign investment owing to deaths and chronic debilitating illnesses has had a serious effect on the economic growth of some poor countries. For example, the estimated forgone output due to the latest Ebola epidemics in Guinea, Liberia and Sierra Leone was over 12percent of the countries’ combined output. The regional loss of output due to slower growth rate was estimated to be US$7.35 billion in 2014 (World Bank, 2014). 68. There are three primary rationales for a publicly-provided regional approach to disease surveillance and response network in West Africa. The first rationale is based on the sharing of resources to enhance efficiency. Examples of resurgent polio, meningitis, cholera and yellow fever in West African countries that were thought to have eliminated or controlled them demonstrate the need for a coordinated regional response. Costly high- level resources, such as laboratories, specialized research institutions, and advanced training facilities may efficiently serve the needs of more than one country. It would be wasteful and duplicative to establish these resources in every country, particularly when the critical mass of highly trained personnel and the volume of services are considered. The World Bank estimates that the total cost savings are 10 -15percent of the system’s total cost, depending on the prevalence of diseases (World Bank, 2012). Project activities such as establishing surveillance centers and trainings for strengthening human resources are regional public goods to increase efficiency. 69. The second rationale is simply the overwhelming economic burden that these diseases, individually and collectively, place on the region which constrain the economic development of individual nations and the region as a whole as was alluded to earlier. The third rationale rests on the status of a disease surveillance system as a global public good, which is both non-rival and non-exclusive. The benefit from preventing the spread of infectious disease is spread across individuals and countries, but there is no practical way to restrict the benefits to those who pay for maintaining it. 70. The project led to higher technical efficiency by improving skills and capacity of human resources through trainings. Because of skills acquired in trainings, these trained personnel and additional FELTP graduates can produce better health care services. Improving the technical capacities of the Centers for Epidemiologic Surveillance is another factor that increases technical efficiency by improving the productivity of health personnel. Development of the DHIS2 reporting system also contributes to technical efficiency by improving decision making processes, rapid outbreak detection, and response systems. In addition, sharing and efficient use of regional resources such as training institutions allows countries to shift resources to other priority areas and increase allocative efficiency. The project is operationally efficient - all activities have been completed within the implementation framework and the project was fully disbursed. 71. Investment in the WARDS project is justified principally for the public goods nature of the establishing surveillance centers, strengthening human resources, and for the positive externalities associated with improved population health outcomes. A quantitative cost-benefit analysis is a challenging task because it is difficult to reliably estimate the impact of the project’s investments in health professionals’ education on health outcomes. 22 Instead of doing a full cost-benefit analysis, the first approach taken here is to establish the economic case by presenting the mechanism through which potential regional and country level benefits will be achieved by the project activities first. 72. The regional level benefits of the project is to build capacity of WAHO to become an effective regional institution capable of supporting (i) coordination and collaboration among countries; (ii) harmonization of legal frameworks, strategy documents and technical approaches; (iii) sharing and efficient use of regional resources such as training institutions including University of Ghana and University of Ouagadougou; (iv) regional platforms for information and data exchange across countries; (iv) joint planning and evaluation; (v) introduction and assessment of innovations, such as Centers for Epidemiologic Surveillance (CES) and information technologies; and (vi) effective and transparent management of resources as well as resource mobilization. 73. Beside the regional benefits attained, the project has several country level benefits including (i) adoption and use of DHIS2 for more effective and efficient management of data; (ii) persons trained in field epidemiology and laboratory capacity; (iii) functional CES in 60 districts providing a network of sentinel districts for early detection of outbreaks). Even though the country level benefits are not evenly distributed across countries, improved capacity of any country for early detection and response to outbreaks/epidemics increases the health security of its neighbors by reducing the risk of an undetected outbreak spreading across borders. 74. The project also leveraged increased investment in regional collaboration for public health, nutrition and population through WAHO including several World Bank investments and more than 25 regional investments by other donors. World Bank Investments leveraged through WARDS project include the Sahel Malaria and NTD Project, the Sahel Women’s Empowerment and Demographic Dividends, the Harmonization of Pharma Regulations, the REDISSE Program Series of Projects, as well as co-financing from BMGF, CIDA, and China. It is important to note that the REDISSE Program and co-financing was influenced by WARDS not only in terms of institutional arrangements but also in terms of technical design. Both the successes and failures of WARDS were taken into consideration in the technical design of the REDISSE program. 75. The second approach is to identify potential benefits of the project’s components and activities, quantifies them into monetary units and compares with project costs through a cost-benefit analysis. However, measuring the impacts of disease outbreaks is a challenging task since they are sporadic events, with limited data points to estimate probabilities. This analysis adopts a comprehensive perspective by modeling the health and economic impacts of reducing the likelihood of disease outbreaks (and pandemics) in West Africa due to the implementation of the WARDS project. To overcome the uncertainties around the likelihood of a value of key parameters (probability of outbreaks and the associated economic damage), this analysis used the Markov Chain Monte Carlo (MCMC) simulation model. 23 76. The likely impacts of the intervention are treated as random variables with hypothesized distributions. More specifically, 1,000 simulations for the next 50 years (2018-2067) using an annual probability of an outbreak in West Africa within a range 0.01 to 0.03, which covers the possibility of a mild, moderate and severe outbreak (World Bank, 2012; Jonas, 2013). The analysis also assumes the total economic impact in a given year will impact the GDP within a range between -0.07 and -4.8 percent, which also covers the estimated impact of mild, moderate and severe pandemics (GHRF Commission, 2016; McKibben and Sidorenko, 2006). Project activities of establishing surveillance centers and trainings for strengthening human resources contributes to control future outbreaks, however, there are other important factors needed to be in place for a full control. Therefore, it is assumed that the projects contribution is limited to five percent. 77. The net present value of the expected annual benefits of the project is equal to US$70.9 million. This estimate includes the health-related benefits (avoided mortality and morbidity) and the economic benefits. The net present value of the project costs, assuming a constant rate of disbursement between 2013 and 2016, is estimated at US$10.8 million, which makes the benefit-cost ratio equals to US$6.6, i.e. for each US$1 invested through the project, there will be an expected return of US$6.6, which implies a high efficiency gain. 78. The project significantly contributed enhancing efficiency by allowing the countries to share costly resources and discouraging wasteful duplication in every country. 79. The efficiency rating is rated as substantial. 3.4 Justification of Overall Outcome Rating 80. Based on the substantial ratings recorded for project relevance, efficacy, and efficiency, the overall outcome rating is rated as satisfactory. Overall rating Relevance Efficacy Efficiency Satisfactory Substantial Substantial Substantial 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 81. The design of the project did not specifically consider the project’s direct impacts on poverty reduction hence, there are no data available measuring positive or negative impacts on poverty reduction over the course of the project. Under Component 2 (Strengthening Human Resources), a clear observation was the gender disparities apparent among the participant pool of selected FELTP residents. Of the 48 WAFELTP residents trained under the program, only 7 women were selected to participate in the program across countries in the sub-region. Eight countries (Burkina Faso, Côte d’Ivoire, The Gambia, Guinea, Mali, Senegal, and Sierra-Leone) with a total of 25 residents had no female representation in the program. This is also attributable to the overall gender imbalance 24 present in the profession. Notwithstanding, a substantial achievement was recorded with the outcome indicator measuring the overall training of health personnel (23percent female compared to the set target of 25percent). (b) Institutional Change/Strengthening 82. The project has contributed to strengthening the disease surveillance and response capacities across member countries of the ECOWAS region, and has also improved the M&E platforms utilized at both the regional and country level. In addition, the institutional capacity of the WAHO has been enhanced. Prior to the WARDS project, WAHO, a technical arm of the ECOWAS, was an institution with extremely weak fiduciary capacity as confirmed during the fiduciary assessment carried out at project appraisal. The support provided under Component 3 of the project has strengthened the project management capacity of WAHO to implement and manage other Bank-funded and several DP projects. Given its impressive performance as an implementing agency, especially with regards to DP and IP coordination, and the strengthened fiduciary management capacity under the project, WAHO is now implementing four additional WB-funded regional projects in various areas related to health system strengthening in the region. Lessons learned from the WARDS project in the areas of implementation, work planning and annual budget estimations, disbursement and expenditure, and the effective management of DPs were applied to these new projects to increase their likelihood of achieving their PDOs. (c) Other Unintended Outcomes and Impacts (positive or negative) 83. As previously mentioned, the strengthened project management capacity of WAHO has attracted various sources of financing from the Bank and other DPs to WAHO for the management of several health projects in the sub-region (four WB-funded projects and over twenty-five projects funded by multiple DPs). 84. A new mandate put in place by ECOWAS in 2016 following the assessment of the performance of WAHO as a PMU has resulted in the eligibility of WAHO to receive funding directly from the Bank without the need for a subsidiary agreement with ECOWAS, a consequence brought about by the support provided under component 3 of the project (management support). As the scope of operation of WAHO as a PMU expands, it will be important to maintain the momentum recorded with improving the capacity of WAHO to continue to carry out both its core mandate as a technical arm of the ECOWAS, and its project management functions as an implementing agency for the effective and efficient roll out of various health systems strengthening interventions in the sub-region. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops n/a 4. Assessment of Risk to Development Outcome Rating: Moderate 25 85. The key risk to development outcome is regarding the long-term sustainability of project interventions due to the different levels of prioritization of disease surveillance and response by the various country governments across member states. At the time of project appraisal, the risk related to sustainability as regards to inadequate resources available at the end of the project to support the implementation of follow-on activities was rated as low. This risk currently remains low due to the increased support by the Bank and DPs for disease surveillance and response, especially in the wake of the 2014-2015 EVD outbreak, which also resulted in the design and current implementation of the related REDISSE program14 in the ECOWAS region as a global public good, and a program which in the medium-term, supports the implementation of the related activities carried out on the WARDS project on a larger scale, and other activities to improve disease outbreak preparedness and response across member countries. 86. Although the strengthened institutional capacity of WAHO attributable to the project has a positive impact on the sustainability and long-term support for the development and implementation of similar projects including the mobilization of Canadian$20 million from CIDA for the continued support to the implementation of the CES, and for the establishment of new CES under the REDISSE program, there is a risk that the focus of country governments in the sub-region will shift to other competing and immediate health priorities across countries commonly faced with limited fiscal allocation for health. The momentum gained by the project with advocating for DSR as a priority agenda will therefore need to be further sustained in the long-term to ensure uninterrupted financing for disease surveillance, outbreak preparedness and response, and increased level of ownership of the agenda by country governments. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 87. The Bank team worked closely with the ECOWAS commission and WAHO during project preparation to ensure strong alignment between project objectives, activities, and key strategic documents including the RIAS and the RCDCP as well as harmonization with the SCADD of Burkina Faso, the operating environment of the WAHO. As a regional project, the Bank also participated in various consultations facilitated with the ECOWAS commission, WAHO and the WHO-AFRO from project preparation to effectiveness to ensure strong endorsement of the project by the ECOWAS Ministers of Health. The Bank also thoroughly accounted for the initial weak fiduciary capacity of WAHO confirmed 14 REDISSE 1 countries are: Guinea, Senegal, and Sierra Leone; REDISSE II Countries ae: Guinea Bissau, Liberia, Nigeria, and Togo; REDISSE III is currently under preparation and includes Benin, Burkina Faso, Mali, Mauritania and Niger. 26 during appraisal by putting in place all the required measures to enhance the capacity of WAHO to function as a PMU, and to provide efficient project management support. 88. Despite these successes, challenges that were not properly addressed during the appraisal stage posed a key challenge such as the broadness of the PDO, and issues with the results framework including the choice of indicators utilized under the project to capture project achievements, and some indicators that were not well defined. While the task team benefitted vastly from expertise in disease surveillance, two key areas lacking in the team composition were the expertise of a Health Economist to carry out a cost-benefit analysis, and the long-term engagement of a Monitoring and Evaluation Specialist to guide the team in the formulation and review of the PDO and the results framework. The Bank performance in ensuring quality at entry is therefore rated as Moderately Satisfactory. (b) Quality of Supervision Rating: Moderately Satisfactory 89. Supervision missions were led by the Bank and co-facilitated with WAHO who closely coordinated with the six IPs of the project to ensure reduced transaction costs and a streamlined reporting arrangements. In conformity with Bank processes, supervision missions were conducted every six months and comprised reasonably staffed teams. Supervision missions thoroughly reviewed the project activities and necessary steps were taken to resolve bottlenecks that arose with WAHO and the IPs over the course of project implementation. Similarly, when faced with difficult issues beyond the control of the team, such as the political climate during the MTR, the Bank team found other possible methods to carry out supervisory activities. 90. Ratings and reporting about issues were clearly articulated in the ISRs including findings from fiduciary capacity assessments, procurement delays affecting implementation of key activities, and issues with recording progress with the achievements of targets set in the results framework. However, some discrepancies were observed with the reporting of some key outcome indicators especially towards the end of the project, a consequence of the lack of expertise of an M&E specialist, and which negatively impacted on the quality of the supervision. Although there were significant improvements reported with the M&E arrangements, it remained an area for continuous monitoring by the Bank to ensure quality data collection and accurate reporting on the project achievements. The quality of supervision is therefore rated as Moderately Satisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 91. Based on the Bank performance in terms of both Quality at Entry and Quality of Supervision, the overall rating for Bank Performance is rated Moderately Satisfactory. 92. 27 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 93. WAHO initially carried out its function as a PMU with very weak fiduciary capacities and a lack of experience with managing Bank projects. This inexperience was reflected during the first half of project implementation with significant delays with implementation of project activities and procurement challenges reported. Following the appropriate staffing of the PMU with highly skilled financial management, procurement, M&E, and communications staff to support the management of the project, the performance of WAHO with providing management support to the project and coordinating the activities of the IPs significantly improved. Key areas of improvements included the establishment of streamlined and efficient coordination mechanisms to closely monitor the implementation of project activities, improved financial management ratings and disbursement performances, and the application of effective communication strategies to increase the visibility of the project achievements in the sub-region. Considering the overall performance over the course of project implementation, the government (WAHO) performance rating is therefore Moderately Satisfactory. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 94. Four of six implementing partners (CDC, CCSID, University of Ghana, University of Oslo) were reported as high performers over the course of project implementation. However, key issues with some implementing partners affected the timely implementation of core project activities including delays with the commencement of some activities such as the training of laboratory technicians provided by the Fondation Merieux, frequent delays and poor reporting by the WHO/AFRO, and discrepancies in reporting of project achievements by the CCSID which reported project achievements using a different formulation of indicators than what is outlined in the PAD The implementing agencies overall performance is therefore rated as Moderately satisfactory (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 95. Based on both the performance of WAHO and the six implementing partners, the overall borrower performance is rated as Moderately Satisfactory. 6. Lessons Learned 96. The following are general lessons learned from overall project management: • The need to streamline financial work planning process: The development of the proposed budget for the 4 years of the project (as opposed to the conventional annual work planning exercise) proved to be a very useful exercise, which 28 contributed to improving disbursement and consumption rates over the course of project implementation. Notwithstanding, the overall project design was ambitious (as discussed above), which became more apparent during the budget planning exercise. Consequently, key activities such as the regional simulation exercises were excluded from the project work plans, while shortfalls in funding for project management were covered under the related REDISSE first series of project (REDISSE I). • It is critical to formally assign key focal points from institutions that will be contracted to serve as project implementing partners: The establishment of a PMU within WAHO equipped with strong project management capacity has resulted in effective coordination and efficient monitoring of all WAHO projects financed by various DPs. However, the reporting difficulties experienced with some IPs including CCISD and WHO/AFRO was noted to be due to the lack of dedicated individuals assigned to support the project (as noted in the section on implementing agency performance, this was a major issue especially with WHO/AFRO). In future operations, these difficulties can be avoided by assigning specific focal points within the various institutions to support the implementation of project activities • Appropriate disbursement ceiling in disbursement letters should be established to limit procurement constraints: The lessons learned pertaining to the low disbursement ceiling of US$ 300,000 set under the WARDS project, which was reported to have contributed to delays with the procurement of goods have been applied to the disbursement ceiling allocation for other WB-supported projects managed by WAHO including the REDISSE, and the Sahel Malaria and Neglected Tropical Diseases (MNTDs) projects • The need to promote country ownership of regional projects through appropriate financing arrangements, and the development and implementation of an effective communication strategy under every project to ensure sustainability post-project closure: The lesson learned from the WARDS project with regards to lack of project ownership displayed by countries due to funds inaccessibility has been applied to the REDISSE program financing arrangements, which ensures part of project funds are directly accessible to countries as IDA grants and credits. Additionally, the importance of having an effective communication strategy is evident in the difference in performance recorded under the project prior to the development and roll-out of the strategy, and after the adoption of the strategy utilized under the project in the second half of project implementation. 97. Lessons learned related to the project design and the implementation of specific project activities under Component 1 and 2 are as follows: Component 1 • Establish CES with the capacity to support multiple countries and ensure appropriate linkage to laboratories: For the 147 new CES under the REDISSE program, it is recommended that the team considers the possibility of the establishment of a CES to support more than one country (specifically for cross- 29 border countries) for efficiency gains, and to encourage sharing best practices across neighboring countries. A needs assessment exercise prior to the set-up of the CES is also recommended to explore the feasibility of establishing a CES at other levels of the health system (beyond the district level). Additionally, attention should be given to improving the coordination between the CCSID and the MoH to ensure appropriate linkage of the CES with laboratory strengthening efforts by including in the ToR, as part of the key selection criteria for a CES, its close linkage to a district level laboratory to enhance disease surveillance capacities. Although this was the intention of the WARDS project, the linkage to laboratories was not enforced and is consequently reported as a weakness. Additionally, it was noted that for those CES that were successfully paired with a laboratory, the trainings provided were not conducted simultaneously. It is recommended that the trainings for the new CES should be coordinated at the same time as the trainings for the laboratories that will be linked to the CES, and challenges that persist with improving the functionality of the DHIS2 platform (and the linkage of generated data to laboratory facilities) are addressed. Component 2 • Streamline long-term FELTP design across ECOWAS member countries: It is recommended that WAHO works closely with the IPs to ensure harmonization of the current curricula of both Anglophone and Francophone FELTPs, and to ensure that the quality of the programs is comparable. For instance, the Anglophone FELTP implemented by the University of Ghana took approximately 2.5 months of preparations/negotiations before the commencement of the first cohort. Moving forward, it is recommended that a capacity assessment is carried out, and WAHO commences discussions with the IPs early on to prepare the new budget estimate that properly accounts for all the outlined criteria in the new ToR, and to establish a clear structure for the program. Additionally, due to the high cost of program implementation, there is a need to consider efficient ways of reducing the fixed costs in the next cohort without compromising the quality of the program • Advocate for program financing, and retention of FELTP graduates: It is recommended that WAHO closely support countries with advocating for the financing of the WAFELTP, and to begin these discussions with the countries during the early stages of project preparation to ensure a feasible sustainability plan is in place. Discussions should also address the retention of program graduates. For instance, discussions with the G-FELTP residents revealed that there are concerns among the residents with regards to the program’s visibility and impact, and how the program differs from the regular MPH program offered at the University of Ghana. Additionally, residents feel they are not utilized enough as a resource for the MoH and the Ghana Health Services (GHS) to recruit for matters related to Epidemiology and laboratory surveillance. There is thus a need to work closely with the MoH and the GHS on the development of a strategic plan that will be put forward as a recommendation to the Ghana human resources ministry in charge of outlining the career paths of government employees. It is recommended that WAHO holds regular discussions with country governments within the applicable ministries to advocate on behalf of the residents about the recruitment and retention 30 of FELTPs graduates in positions where there are opportunities to apply their newly acquired skillsets. • Improve the FELTP curriculum and overall program quality: As reported in the section on project achievement, specific modules of the WAFELTP curriculum such as the veterinary health, and the laboratory component remain weak. Addressing these gaps in skillsets should be a thorough consideration for the training program that will be offered to future residents. Another area for improvement is on the limited number of supervisors for the FELTP residents, which negatively impacted the quality of the program offered under the WARDS project. It is recommended that the FELTP graduates are offered opportunities to serve as supervisors to residents in future cohorts, and that adequate motivation are put in place for the FELTP resident supervisors such as the provision of support to attend conferences where future residents plan to present their work. Other practical approaches for improving the overall program quality should be thoroughly considered. In this regard, issues around language barriers were reported among past residents who were deployed to other countries for outbreak investigations. To promote better cross-border collaboration among countries, it is recommended that a practical language exchange module be incorporated into the revised curricula for future cohorts of the WAFELTP, and factored into the budget calculations. • Promote gender balance among program participants: To address the highly- skewed gender representation, a strategic plan for encouraging more female participation in the program should be incorporated into the selection criteria/ process of call-for participants. 7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors (a) Grantee/Implementing agencies (b) Cofinanciers/Donors (c) Other partners and stakeholders (e.g. NGOs/private sector/civil society) 31 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Components Appraisal Actual/Latest Percentage of Estimate Estimate Appraisal (USD Million) (USD Million) 1: Regional Capacity Strengthening 4.81 4.95 103% 2: Strengthening Human Resources 4.93 4.75 96% 3: Management Support 1.01 2.07 199% Total Financing Costs 10.75 11.77 109% Note: Appraisal estimates were taken from the PAD when counterpart funding was specified at US$ 750,000 (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Africa Catalytic Growth Fund 10.00 10.00 100.00 (ACGF) Counterpart funding 0.75 1.50 200.00 Financing from Trust fund linked to 0 0.27 REDISSE Project preparation Total Financing 10.75 11.77 109.5 Note: Counterpart funding contribution to the project from WAHO was doubled to US$ 1.5 million. Additional financing source reported was derived from a trust fund linked to REDISSE project preparation and was used under the WARDS project to enhance the PMU in preparation to manage other WB-funded projects and for work carried out by an IP in preparation for the REDISSE project. 32 Annex 2. Outputs by Component Below is a breakdown of project outputs by component and by implementing partner. Component 1: Regional Capacity Strengthening • Establishment of 60 Centers for Epidemiologic Surveillance (CES) • Evaluation of minimum capacity required for the implementation of IHR and development of report for Cape Verde, Mali and Senegal; • Draft and implementation of five-year IDSRS strategic plans. For Ghana, Liberia, Mali, Niger, Nigeria and Senegal • Capacity building of 44 national officials responsible for surveillance and laboratories from the 15 ECOWAS countries and Mauritania on IDSR • Revision and printing of 1,000 copies of the IDSRS (Integrated Disease Surveillance and Response System) for Burkina Faso. o . Component 2: Strengthening Human Resources • US CDC/ University of Ouagadougou: 29 residents trained under the Francophone Field Epidemiology and Laboratory Training program (FELTP); including 6 females • University of Ghana: 20 residents trained under the Anglophone FELTP including 3 females • CCISD: 172 trained health workers (including 34 females) o 135 R/CES including 24 women o 35 trainers including 10 women2 health workers trained in M&E with internally generated funds • Fondation Mérieux: 248 persons trained, including 65 females; o 198 Laboratory Technicians (including 51 female); o 50 trainers (including 14 females) • University of Oslo: 132 persons trained (including 30 women) • WHO: 74 (including 18 females) Component 3: Management Support • Development of the regional M&E Platform (DHIS2) • Production, publication and dissemination of 80 weekly epidemiological bulletin and 3 quarterly epidemiological bulletins • Staffing of PMU supported with the recruitment of Consultants (2 Project Coordinators, Financial Management Specialist, Procurement Specialist, M& E Specialist, and a Communications Specialist) • Procured goods: 33 Detailed list of project outputs under each component by Implementing Partner (includes all procured goods) No Components Output 1 Regional Capacity Strengthening 1.1 WHO - Revised the Gambia IDSR Technical Guide - Capacity building of 44 heads surveillance and laboratories in the 15 ECOWAS countries and Mauritania on the IDSR, on the adaptation of staff training modules and on the development of the IDSR strategic plan - Development and validation of national IDSR strategic plans in Ghana, Liberia, Mali, Niger, Nigeria and Senegal - Printing of 1000 copies of the Burkina Faso national IDSR guide and made available to health actors - Conducting the assessment of the capacities required for the implementation of the IHR conducted in Senegal, Mali and Cabo Verde with the action plans for reinforcing the essential capacities for the implementation of the IHR in the 3 countries - Organization of a cross-border meeting on epidemic-prone diseases including Malaria and Neglected Tropical Diseases and based on the "one-health" approach, that brought together 5 countries (Burkina Faso, Côte d’Ivoire, Liberia, Niger and Sierra Leone) - 1.2 - University of - Launch of the DHIS2 regional platform Oslo - Improvement in the use of data collected by member countries - Enhancement of the promotion for the use of the DHIS 2 platform by ECOWAS member countries. - 1.3 - CCISD - Conduct of feasibility missions in the 10 countries to host the Centers for Epidemiological Surveillance (CES) - Setting up and functionality of 4 Country Coordinating Teams (CCT) in Bobo-Dioulasso, Accra, Dakar and Abuja - Review of training materials for epidemiological surveillance in Portuguese - Finalization of the translation of the PASEi training modules into Portuguese and English - Coordination and operation of the 60 new CES in the countries concerned - Evaluation of PASEi centers in the 5 countries (Benin, Burkina Faso, Guinea, Mali and Niger) - Organization of R/CES training in French, English and Portuguese speaking countries 34 - Supervision of the new 60 CEs set up; - Evaluation and capacity building of 75 formers CESs (PASEi) in the 5 countries (Benin, Burkina Faso, Guinea, Mali and Niger) - - NB. The 15 CSEs of Guinea have been re-established; - - 2 - Strengthening Human Resource - - 2.1 - CDC/ - 29 Residents received Master’s degree in Field Epidemiology University of Ouagadougou - 2.2 - University of - 20 Residents received Master’s degree in Field Epidemiology Ghana - 2.3 - CCISD - Organization of training of trainers in French, English and Portuguese speaking countries - Conduct of training sessions of the 60 established R/CES - Organization of the training of trainers for the PASEi countries - Capacity building of the CES of countries from the PASEi zone - Organization of supervision sessions of 60 established CES - Organization of supervision sessions of 75 formers CES - 2.4 - Fondation - Translation of the training modules in laboratory techniques into Mérieux English and Portuguese - Organization of training of trainers in French, English and Portuguese speaking countries - Training of laboratory technicians from French, English and Portuguese speaking countries - Implementation of the e-learning - 3 - Management - PMU with 11 staff functioning Support - 10 vehicles deliver to 10 countries to aid supervision of CES - 4 Vehicles for CCT operations in Dakar, Accra, Abuja and Bobo-Dioulasso - 60 computers, printers and UPS bought for the functioning of the 60 CES - - Project’s - Availability of the project’s communication plan Visibility - Regular update of the WAHO website with project’s activities - Production and publication of Epidemiological bulletins and articles on the WAHO website; - Production and publication of weekly and quarterly bulletins on the WAHO website and sharing with countries and partners Annex 3. Economic and Financial Analysis 35 The economic costs of disease outbreaks are significant and preventing these outbreaks yields large economic benefits by reducing the threats of epidemics and pandemics. Besides the negative economic impact due to mortality and morbidity and health care costs, disease outbreaks affect economies at national and global level by decreasing production, investment, exports, and personal income as well as reducing labor, capital, and productivity, which are all major components of growth (UNDP, 2014). Their increasing toll on productivity, profitability and foreign investment owing to deaths and chronic debilitating illnesses has had a serious effect on the economic growth of some poor countries. For example, the estimated forgone output due to the latest Ebola epidemics in Guinea, Liberia and Sierra Leone was over 12 percent of the countries’ combined output. The regional loss of output due to slower growth rate was estimated to be US$7.35 billion in 2014 (World Bank, 2014). Globally, the economic impacts of severe pandemics have been estimated at 4.8percent of the global gross domestic product (GDP) or approximately US$3 trillion in the 21st century (Jonas, 2013). Compared to the estimated required investments to build a well-functioning global disease surveillance system and response, the expected annual returns on investment of avoiding such large losses are estimated as high as 123 percent (World Bank, 2012). Since the endemics, epidemics and emerging communicable diseases do not respect the national boundaries, a regional approach to control communicable diseases in West Africa makes sense both epidemiologically and economically. There are three primary rationales for a publicly-provided regional approach to disease surveillance and response network in West Africa. The first is simply the overwhelming economic burden that these diseases, individually and collectively, place on the region which constrain the economic development of individual nations and the region as a whole as was alluded to earlier. Communicable diseases decrease productivity, undermine the human resource base and deter foreign investment in Africa. For example, in 1999, Gallup and Sachs estimated that malaria alone inhibited economic growth by 1.3percent per annum in malaria endemic countries and cost the continent at least US$12 billion in lost GDP. The second rationale rests on the status of a disease surveillance system as a global public good, which is both non-rival and non-exclusive. The benefit from preventing the spread of infectious disease is spread across individuals and countries, but there is no practical way to restrict the benefits to those who pay for maintaining it. The third rationale is based on the sharing of resources to enhance efficiency. Examples of resurgent polio, meningitis, cholera and yellow fever in West African countries that were thought to have eliminated or controlled them demonstrate the need for a coordinated regional response. Costly high-level resources, such as laboratories, specialized research institutions, and advanced training facilities may efficiently serve the needs of more than one country. It would be wasteful and duplicative to establish these resources in every country, particularly when the critical mass of highly trained personnel and the volume of services are considered. The World Bank estimates that the total cost savings are 10 - 15percent of the system’s total cost, depending on the prevalence of diseases (World Bank, 2012). The WARDS project increase technical, allocative, and operational efficiency. It leads to higher technical efficiency by improving skills and capacity of human resources through 36 trainings. The trained personnel and additional FELTP graduates are able to produce better health care services due to the skills they acquired in trainings. Improving the technical capacities of the Centers for Epidemiologic Surveillance and development of the DHIS2 reporting system are other factors that increases technical efficiency by improving the productivity of health personnel. In addition, sharing and efficient use of regional resources such as training institutions allows countries to shift resources to other priority areas and increase allocative efficiency. The project is operationally efficient—all activities have been completed within the implementation framework for full disbursement. Cost-Benefit Analysis The economic analysis of the WARDS capacity strengthening project examines the economic rationale for investing in disease surveillance and response in West Africa. The analysis identifies potential benefits of the project’s components and activities, quantifies them into monetary units and compares with project costs through a cost-benefit analysis (CBA). (CBA) allows a direct comparison of costs and benefits of the project, the costs and benefits of alternatives use of the project resources (economic costs) and compares costs and benefits of interventions beyond the health sector. The analysis also includes an appraisal of the potential to leverage financial contributions from national governments and the international community to sustain the project’s interventions in the long run Assessments of disease surveillance and response systems have mostly focused on the economic returns of avoiding pandemics (World Bank, 2012; IOM, 2009). The modeling approach depends on the balance between data availability and model sophistication. This analysis adopts a comprehensive perspective by modeling the health and economic impacts of reducing the likelihood of disease outbreaks (and pandemics) in West Africa due to the implementation of the WARDS project. Measuring the impacts of disease outbreaks is a challenging task since they are sporadic events, with limited data points to estimate probabilities. To overcome the uncertainties around the likelihood of a value of key parameters (probability of outbreaks and the associated economic damage), this analysis used the Markov Chain Monte Carlo (MCMC) simulation model. The likely impacts of the proposed intervention are treated as random variables with hypothesized distributions. More specifically, 1,000 simulations for the next 50 years (2018-2067) using an annual probability of an outbreak in West Africa within a range 0.01 to 0.03, which covers the possibility of a mild, moderate and severe outbreak (World Bank, 2012; Jonas, 2013).15 The analysis also assumes the total economic impact in a given year will impact the GDP within a range between -0.07 and -4.8 percent, which 15 An alternative to be tested is to calculate impacts by using each of the seven categories of disease – the six categories from table 2 plus a seventh branch for a catastrophic pandemic originating in West Africa. 37 also covers the estimated impact of mild, moderate and severe pandemics (GHRF Commission, 2016; McKibben and Sidorenko, 2006).16 The key parameters and model assumptions are: • Annual probability of outbreaks in West Africa equal to 0.03 (World Bank, 2012; Jonas, 2013); • The probability of an outbreak event to occur in any year is independent of other events to occur in other years within 50 years’ time frame; • West Africa GDP equal to US$561.1 billion (2016 USD); • Assuming a 2.5 - 7.1 percent annual real GDP growth rate in the time frame of 50 years, and a loss of output associated with an outbreak event equal to 4.8 percent of the regional GDP; • The model does not consider global GDP losses, only GDP losses in West Africa. • The contribution of the project to overall effectiveness of surveillance and response system is five percent Results of the analysis The net present value of the expected annual benefit of controlling an outbreak in West Africa is, equal to US$283.6 million. This estimate includes the health-related benefits (avoided mortality and morbidity) and the economic benefits. Applying the distribution of costs proposed by the World Bank (World Bank, 2008), the estimated losses resulting from mortality, productivity losses due to morbidity and absenteeism, and the expected losses resulting from behavior changes to avoid infection are given in the table below. Table 1: Distribution of expected loss among main factors Reasons Distribution Cost $ mortality 12% $ 34,026,316 productivity losses due to 28% morbidity and absenteeism $ 79,394,738 losses resulting from 60% behavior changes to avoid infection $ 170,131,582 Total 100% $ 283,552,637 Distribution of Expected Number of Outbreaks in 50 years, West Africa 16 Previous work has estimated the economic loss that occurred as a result of each of the 20th-century pandemics as 0.7–4.8% of global GDP (GHRF Commission, 2016). 38 However, the results above are likely to underestimate the benefits of disease surveillance and response system since (i) it does not calculate the expected impact on endemic diseases; (ii) it considers only the impacts in West Africa; and (iii) it assumes the risk of pandemic events this century will be the same within the time frame considered (50 years). The net present value of the project costs, assuming a constant rate of disbursement between 2013 and 2016, is estimated at US$10.8 million, which makes the benefit-cost ratio equals to US$6.58, i.e. for each US$1 invested through the project, there will be an expected return of US$6.58. Sensitivity Analysis The sensitivity analysis examines alternatives to the main scenario given above. Given the uncertainties around key parameters of the model, a sensitivity analysis was conducted to estimate changes in the results given plausible variations on the value of key parameters. Additional simulations were tested based on the following parameters: • Annual probability of an outbreak: 1%, 2%, and 3% 39 • Economic costs of outbreak: 0.7%, 2.75%, and 4.8% reduction in output during a year with an outbreak (which covers the scenarios of a mild, moderate and severe outbreak); • The contribution of the project to overall effectiveness of surveillance and response system is three percent Table 2 contains the different scenarios resulting from a combination of different probabilities of pandemic and severity of an outbreak. Economic annual impacts vary from US$3.4 million (mild pandemic with a one percent probability) to a US$70.9 million (severe outbreak with a probability equal to three percent) with five percent contribution rate. However, the economic annual impact range decreases to US$2.1million (mild pandemic with a one percent probability) to a US$41.5 million (severe outbreak with a probability equal to three percent) with when the efficiency rate is three percent contribution rate. Table 2: Expected Economic Impacts - West Africa, 50 years, million US$ saved annually 5% efficiency Annual Probability of Outbreak Cost (Reduction in GDP) 1% 2% 3% -0.70% $3.40 $7.16 $10.75 -2.75% $13.03 $26.84 $40.82 -4.80% $25.66 $49.21 $70.89 3% efficiency Annual Probability of Outbreak Cost (Reduction in GDP) 1% 2% 3% 0.70% $2.13 $4.59 $6.61 2.75% $7.96 $16.40 $24.35 4.80% $14.58 $30.49 $41.52 The benefit-cost ratio was calculated for these scenarios as well. The benefit-cost ratio is between US$ 0.3 (mild pandemic with a 1percent probability) and US$1 (mild outbreak with a probability equal to 3percent) assuming that an outbreak causes a 4.8percent reduction in GDP. If the cost due to an outbreak is assumed 4.8percent of GDP, the benefit- cost ratio varies between US$2.4 and US$6.6 as shown in Table 3 below. Table 3: Annual Benefit-Cost Ratio, not cumulative - West Africa, 50 years, Annual US$ 5% efficiency Annual Probability of Outbreak Cost (Reduction in GDP) 1% 2% 3% -0.70% $0.32 $0.67 $1.00 -2.75% $1.21 $2.49 $3.79 -4.80% $2.38 $4.57 $6.58 40 3% efficiency Annual Probability of Outbreak Cost (Reduction in GDP) 1% 2% 3% -0.70% $0.21 $0.43 $0.62 -2.75% $0.74 $1.52 $2.26 -4.80% $1.35 $2.83 $3.86 41 References • Bogich, T.L., R. Chunara, D. Scales et al. (2012). “Preventing pandemics via international development: A systems approach”. PLOS Medicine 9(12): e1001354. • Chan, E.H., T.F. Brewer et al. (2010). “Global capacity for emerging infectious disease detection”. PNAS 2010; 107(50):21701-6. • GHRF Commission (2016). "Global Health Risk Framework for the Future. The neglected dimension of global security: A framework to counter infectious disease crises”. • IOM (2009). Sustaining global surveillance and response to emerging zoonotic diseases. Washington, DC: The National Academies Press. • Jonas, Olga B. 2013. “Pandemic Risk.” Background Paper, World Development Report. World Bank, Washington DC. • UNDP (2014). "Assessing the Socio-Economic Impacts of Ebola Virus Disease in Guinea, Liberia, and Sierra Leone: The Road to Recovery". • World Bank (2012). “The Economics of One Health”. Economic and Sector Work. People, Pathogens, and our Planet, volume 2. • World Bank (2014). "The Economic Impact of the 2014 Ebola Epidemic: Short- and Medium-Term Estimates for West Africa." 42 Annex 4. Grant Preparation and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending/Grant Preparation TTL ADM Enias Baganizi Senior Health Specialist GHN07 Responsible Procurement Mohamed El Hafedh Senior Procurement Specialist GGO07 ADM Hendah Responsible Financial Ngor Sene Financial Management Specialist GGO26 Management Aissatou Chipkaou Operations Analyst GHN13 Team Member Daniele A.G.P. Jaekel Operations Analyst AFTHW Suzanne Rayaisse Program Assistant AFMBF Team Member Social Safeguards Sveltana Khvostova Senior Environmental Specialist GEN01 Specialist John Paul Clark Senior Technical Specialist GHN07 Epidemiologist Lead Health Dominic Haazen Lead Health Policy Specialist AFTHW Policy Specialist Hocine Chalal Lead Environmental Specialist Samba Duale Consultant AFTHW Epidemiologist AFTM Financial Edith Atioumoutio ET Consultant W Management Aritseidis I. Panou Consultant LEGOP Lawyer Nicole Hamon Language Program Assistant AFTHW Operations Bintou Sogodogo Program Assistant AFMBF Operations Operations Daniele A.G.P Jaekel Operations Analyst AFTHW Analyst Edith Ruguru Mwenda Senior Counsel LEGAM Legal Aristeidis I. Panou Consultant LEGOP Lawyer Andre Ndikuyeze Consultant AFTHW Epidemiologist Mamata Tiendrebeogo Senior Procurement Specialist AFTPW Procurement Supervision/ICR TTL ADM Enias Baganizi Senior Health Specialist GHN07 Responsible John Paul Clark Senior Health Specialist GHN07 Peer Reviewer Volkan Cetinkaya Senior Economist GHN03 Economist Karine-Noelle Mouketo- Program Assistant GHN07 Team Member Mikolo Amba Denise Sangara Program Assistant GHN07 Team Member 43 Aissatou Chipkaou Operations Analyst GHN13 Team Member Suzanne Rayaisse Program Assistant AFMBF Team Member Begnadehi Claude Bationo Senior Operations Officer AFMBF Operations Claudia M. Pardinas Ocana Senior Counsel LEGOP Legal Haidara Ousmane Diadie Senior Health Specialist AFTHW Epidemiologist Luc Lapointe Consultant AFTPW Boubacar Diallo Consultant AFTPW 44 Annex 5. Beneficiary Survey Results N/A 45 Annex 6. Stakeholder Workshop Report and Results N/A 46 Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR Executive Summary The US$11.5 million WARDS (West Africa Regional Disease Surveillance Capacity Strengthening) Project, jointly funded by the World Bank and WAHO, was implemented by 6 partners (CCISD, CDC Foundation, Mérieux Foundation, WHO, University of Oslo and University of Ghana). The three main components: capacity building in disease surveillance and response at the regional level and at the ECOWAS member state level, increased resources and project management support, were achieved without major challenges. For the most part, the 5 predefined development objectives were largely met. In fact, all (100percent) the 60 surveillance centers were set up, based on the quota established by country, in the project document. In addition to the 60 CES set up in the 10 target countries, 15 new CES were funded in Guinea to address gaps noted during the evaluation. All the funded epidemiological surveillance centers regularly submitted comprehensive reports for the notification of priority diseases, with rates of completeness and timeliness being 100percent and 95.2percent respectively. Five hundred forty-three (543) health workers benefited from training in epidemiological surveillance and laboratory skills, representing 134percent of initial planned target of 404 health workers, per project document. All intermediate results indicators were also achieved, to the satisfaction of direct and indirect beneficiaries. Ministry officials, from countries covered by the project, agree that the project yielded very encouraging results and improved the performance of trained health workers. The expansion and sustainability of results remain a major challenge. For the countries who received REDISSE funding, it is recommended to capitalize on the experiences and consolidate the achievements of the WARDS project, for more effective use of resources provided by the World Bank. 1. Introduction WARDS (West African Regional Disease Surveillance Strengthening) Project was implemented by WAHO (West African Health Organization), a specialized health agency for ECOWAS (Economic Community of West African States). This project received US$10million funding from the World Bank along with US$1.5 million funding from WAHO. The implementation of the project lasted 4 years (2014 through the end of June 2017) and was carried out in partnership with six implementing agencies: CDC Foundation (Centers for Disease Control and Prevention), UiO (University of Oslo), UG (University of Ghana), WHO/IST (World Health Organization/Intercountry Support Team), MF 47 (Mérieux Foundation) and CCISD (Center for International Cooperation in Health and Development). These technical partners subcontracted the achievement of the three project components as follows: Component 1: Regional and ECOWAS Member State Capacity Building in disease surveillance and response was assigned to CCISD, University of Oslo and WHO. Component 2: Human Resources Strengthening was implemented by the Mérieux Foundation, University of Ghana, CDC, University of Ouagadougou and later on, CCISD. Component 3: Project Management Support was carried out by WAHO. Within this framework, this organization was responsible for coordination and daily project management, at the regional level, through the harmonization of policies and procedures; support to joint planning and the coordinated implementation of interventions in the countries; monitoring and evaluation of cross-country activities; documentation; and the dissemination of best practices in health within the ECOWAS setting. As a reminder, the WARDS project defined 5 PDOs (Project Development Objectives) and 5 IRIs (Intermediate Results’ Indicators), as outlined in Appendix 1. It should also be noted that the WARDS project targeted the following direct beneficiaries: health workers in the field, statistical officials, epidemiological disease surveillance officers as well as laboratory staff, within the national health systems. The indirect beneficiaries are the populations of the 15 ECOWAS countries. 2. Methodology The general objective of this evaluation is: “Assess the level of achievement of the project through Project Development Objectives indicators and Intermediate Results Indicators, in the 15 target countries.” To evaluate the level of achievement of the various defined development objectives, a methodology was developed and a survey designed to collect data within the framework of the mission. These were submitted beforehand to WAHO for review and amendment (Appendices 3 and 4). This methodology consisted of, among others, a review of project documents and utilization of the various available reports. These reports were carefully reviewed prior to data collection in the field. The facts reported in the process reports and quarterly reports, by the various partners, were compiled and validated during field visits, carried out in 3 preselected countries: Burkina Faso, Ghana and Senegal. For each country visited, a summary report and a 48 PowerPoint presentation were presented and discussed with Ministry officials during debriefing sessions, scheduled before the end of the missions in the referenced countries. Information compiled during field visits and the review of process reports, from the various implementing partners, enabled the assessment of achievements as outlined in the next chapter. 3. Achievements of WARDS project Findings in the field and analysis of reports enables the assessment of the achievement of development objectives and the level of achievement of intermediate results’ indicators of the project. 3.1 Level of achievement of WARDS’s five (5) indicators. The reports and findings from the field confirm that the first development objective was achieved. The 60 epidemiological surveillance centers were set up based on the quotas established by country, as outlined in the project document. In addition to the 60 CES set up in the 10 target countries, 15 new CES, involved in the FELTP Frontline, were funded in Guinea, in partnership with CDC Atlanta. All the supported epidemiological surveillance centers regularly submitted comprehensive priority disease notification reports – 95.2percent timeliness and 100 percent completeness. The second Project Development Objective, with respect to training of personnel, was also achieved and even surpassed. Five hundred and forty-three (543) health workers benefited from training in epidemiological surveillance and laboratory skills, being 134percent of initial target of 404 health workers, as per project document. However, it is important to note that the percentage of women who benefited from the various trainings is still low. For more detail on the number of people trained by gender and country, please refer to table 1 under PDO 2. Objective 3 focuses on the percentage of short-term interns in the field, in the epidemiology sector or having laboratory skills and who work in the district, 12 months after completing training (by gender and by sponsoring country). Even though it was reported that 100percent of health workers were trained by district, it will be premature to confirm their performance in their respective districts, especially since, many of them just completed their training (1 or 2 months before this evaluation). With respect to Project Development Objective 4, in terms of completion rates, 48 out 60 (80percent) FELTP interns, planned for in the project, successfully completed their training. For this objective, one must note the instructional challenges, CDC Foundation scaled down the number of interns to be enrolled in the French FELTP courses (20 instead 49 of 30 by cohort). If the completion rates are calculated based on this new data, the completion rate for the interns is 96percent (48 out of 50 interns registered for the training, 1 of the interns died during the training and 1 had an accident right before completing their thesis for end of training presentation. This will be rescheduled). For objective 5, related to the publication of quarterly newsletters on WAHO’s regional disease surveillance and its dissemination to ECOWAS member countries, only 2 quarterly newsletters out of 16 expected newsletters (29percent) were published. WAHO’s management changed the guidelines as it found that weekly newsletters were more appropriate for the exchange of information, to provide quick feedback and to organize potential real-time response. Consequently, 80 weekly newsletters and 2 quarterly newsletters were published to date. With respect to the achievements’ section, one must note that the various implementing agencies encountered long delays in the implementation of activities for various reasons. The major factors for this delay are summarized below: CCISD attributes delay in start of activities to challenges in procurement of computers and vehicles, especially challenges in delivery and customs exemption processes in the beneficiary countries. However, as soon as equipment was delivered, the project teams immediately started working and delivered results within the recommended deadlines. As previously mentioned, all the surveillance centers were set up and staff was trained and regularly supervised. However, per the WHO Burkina Faso representative, the World Health Organization encountered long delays, primarily due to the onset of the outbreak of the viral Ebola disease. This outbreak mobilized almost all the WARDS project staff during the first two years of the project. As soon as the outbreak was controlled, WHO achieved all (100percent) of all planned activities as listed below: • Evaluation of minimum capacity required for the implementation of IHR and available reports for the 3 target countries: Cape Verde, Mali and Senegal; • Drafting of five-year IDSRS strategic plans. These plans were drafted and implemented in 6 target countries (Ghana, Liberia, Mali, Niger, Nigeria and Senegal); • Capacity building of 44 national officials responsible for surveillance and laboratories from the 15 ECOWAS countries and Mauritania on IDSRS, adapting staff training modules and the development of an IDSRS strategic plan. • Printing 1,000 copies of the IDSRS (Integrated Disease Surveillance and Response System) handbook revised for Burkina Faso. With respect to the Mérieux Foundation, according to the Mérieux staff responsible for this component, delays were mainly due to World Bank’s required administrative 50 processes with respect to signature of contracts and to the sensitization of authorities in the countries facing certain operational challenges. It is important to also note that the preparation of modules for trainers and laboratory technicians and their translation into English and Portuguese took much longer than scheduled. There were difficult negotiations with the 8 target countries to convince them of the need to build capacity in laboratory skills which were often weak and neglected. Consequently, the design of modules into the various languages and trainings did not start until the second quarter of the third implementation year (2016) and ended right before the closeout of the project in June 2017. Despite this delay, Mérieux Foundation trained 50 trainers, double the planned number for the project; these trainers also trained 198 laboratory technicians in 8 countries (details on the number of people trained by country are found in tables 1A and 1B). Mérieux Foundation complained about the lack of financial resources and limited time to ensure training supervision of people who were trained. However, it hoped to monitor their professional development through the RESAOLAB project. The RESAOLAB project intends to continue activities carried out by the WARDS project. Within the framework of training on epidemiology in the field, the Universities of Ghana and Ouagadougou as well as CDC Foundation, did not note specific issues with respect to start and execution of the training on the FELTP program, besides the previously stated limited number of female candidates. 3.2 Level of achievement of IRI (intermediate results’ indicators) As is shown in table 1 B, most of the intermediate results indicators were achieved (100percent), to the satisfaction of direct and indirect beneficiaries. Only the simulation exercises, the planned 4th intermediate result, could not take place due to insufficient financial resources. It is important to note the incorrect costing of a practical yet untested methodology and especially that technical and multidisciplinary expertise and multisector coordination was not established. For these reasons, WAHO, during the midyear review of the project at Annecy, France, received approval from the World Bank and other participants to replace the simulation exercises with an international meeting. For this reason, an international meeting involving representatives from 8 ECOWAS countries, was organized in Monrovia, Liberia, in partnership with IOM (International Organization for Migration), WHO (World Health Organization) and UEMOA (West African Economic and Monetary Union). Despite encouragement and opportunities provided in terms of priority during selection process, gender equality was not adhered to, with the lack of female candidates in most of 51 the countries. Thus, women represent a low percentage in almost all the countries, except for Niger, where approximately 46percent of people trained in laboratory skills were women. The distribution of women by country is presented in table 2. a. According to regular reports received by WAHO and available in the department for Disease Control and Epidemiological Surveillance, the 60 epidemiological surveillance centers set up and funded by the project, are set apart by their performance in terms of completeness of 100percent and timeliness of 95.2percent. In the two countries visited, Ghana and Senegal, the personnel responsible for surveillance, trained and equipped with computers and modems through this project, serve as mentors and are called to act as facilitators during training of front line health workers (front line epidemiology training) or to build the capacity of health workers in districts that were not supported by the project. b. With regards to personnel trained in FELTP (field epidemiology) at the University of Ghana (20) and at the University of Ouagadougou (28), they are valued by their employers; three of them were already promoted to a higher grade, at the beginning of their field internship, even before receiving their certificates. They displayed competency and excellent performance during epidemiological research and the organization of response to the Rift Valley fever outbreak during their training in Niger, the Lassa fever outbreak in Benin and the measles outbreak in Senegal. Their expertise encouraged the countries to send and fund several candidates for FELTP training. c. Even though the reports state that there was a training of critical mass of laboratory technicians, it is difficult to see their contribution in the visited surveillance centers (At Richard Toll, Senegal, the sole CES laboratory technician stated that she has not yet received this training. This was also the case of the laboratory technicians that were met in the districts of Awutu Senya, Central Region, Ghana and Hohoe, Volta Region, Ghana). d. The assessment of the level of satisfaction of direct beneficiaries stems from the statements of health workers who confirm that they learnt a lot in theory and are grateful for the opportunity to put their skills to the test, during epidemiological field research. Most of their work was discussed in oral presentations or selected posters during large international fairs (AFENET, TEPHINET). During field visits, the recipients of the FELTP course, funded by WARDS, proved to and confirmed being able to master a couple software programs, such as EP-Info. They were also well- equipped for collection, analysis and interpretation of epidemiological data to carry out prevention and rapid response initiatives for any potential threats. The tables and graphs on epidemiological trends of diseases under surveillance are displayed on the walls of their little offices and are shown with pride by their authors, health workers trained and equipped by the WARDS project. 52 4. Funding and budget expenditures As stated in the introduction, this revolutionary project was funded by a US$10 million World Bank grant as well as a US$1.5 million grant from WAHO. All (100percent) funds were committed and used before the end of the project as shown in table 2, $11,485,550. Thus, 99.8percent of appropriated funds were spent on the implementation of various project components. As stated in the table, most partners spent more than 90percent, sometimes 100percent of the budget, allocated to them, except the WHO which spent 86 percent of initial budget allocation. From the table, it emerges that, in addition to the $1.5 million committed before the start of the project, WAHO allocated additional funds to cover management costs, not budgeted in the project’s initial document. Monitoring, evaluation and support systems, as recorded in the various process reports, were achieved and the expenditure ratios were consistent with the results and objectives of the project. One can even note that the cost-effectiveness ratio of the project is high if one considers that many activities were carried out and that a lot more health workers than planned were trained, as illustrated in the results by objective table. In addition to strengthening of epidemiological surveillance, it also included the training and mentoring of workers from other sectors, involved in the strengthening of health information systems, such as statisticians and logistics personnel, by the University of Oslo. Lessons learned Project startup experienced delays due to cumbersome and sometimes not well understood administrative procedures. This is also why equipment and some logistical resources were delivered late to the various countries, following issues with their exemption and customs clearance. Even though, this led to additional costs, these issues were appropriately resolved and delays were caught up by the various implementing agencies, as stated above. Furthermore, the actual implementation phase of the WARDS project coincided with the outbreak of the viral Ebola disease, which mobilized many human and material resources to quell it. This greatly affected the normal progress of the activities of various projects, including the WARDS project, which had just started in the sub region. It is important to point out that the training of the Francophone trainee cohort, in the WARDS-funded “FELTP”, field epidemiology program, occurred within a tense and difficult socio-political setting. The outbreak of the crisis in Burkina Faso, which led to political change, caused the course organizers to temporarily move the course to Senegal. The cost was entirely funded by the CDC Foundation, according to its technical advisor. This event disrupted the initial course schedule. However, it did not affect the quality of 53 instruction provided due to the proactiveness of the management team of the program and officials who facilitated the evacuation of all students, without constraints. Ministry officials, in the countries visited, admitted that the project produced very encouraging results and greatly improved the performance of trained health workers. Extension and sustainability of results remain a major challenge for most of the countries who received REDISSE funding. The extension of activities, through the creation of new epidemiological surveillance centers, has almost been achieved. However, the states need to commit to ensuring access to basic inputs such as office supplies (printer ink cartridges), laboratory reagents for the screening of diseases under surveillance as well as means of transport to facilitate epidemiological research and the secure transportation of laboratory samples. Positive points 1. The project played a coordinating role and facilitated interaction between Anglophone, Francophone and Lusophone countries. This broke the prevailing language barrier in the sub-region; 2. The FELTP course curriculum, already approved by CAMES, and ensures equivalence with the ongoing CES in public health; 3. The trainees greatly appreciated the selection process which was highly competitive rather than based on social networks; 4. Greatly improved visibility in field epidemiology training as well as results from the first cohorts of the FELTP programs created real excitement in training of a critical mass of people in FELTP to respond to requests made by the countries. It is important to note that several countries expressed the desire to increase the number of candidates using their own funds. 5. The skills of the trained people, especially their increased capacity in research and response to epidemic-prone diseases, were greatly valued by national officials. One can cite concrete examples of their involvement in research on the outbreak of the Rift Valley Fever in Niger and the Lassa Fever outbreak in Benin. 6. Supply of work tools, laptops and access to internet, for the submission of data and sharing of information in real time; 7. The establishment of an alumni association to maintain contact and strengthen the network of beneficiaries of the FELTP training in Francophone Africa. Negative Points 1. Cumbersome administrative processes (long delays in signature of contracts, delay in the procurement of materials and equipment) caused the delay in the start of the various project components; 54 2. The lack of mentoring of the trainees, at the country level. Countries do not have personnel adequately trained in epidemiology. The few trained health workers were very much sought after by the various ministry of health departments. 5. Insights or recommendations to ensure improved effectiveness during implementation of future projects WAHO: 1. Build on the achievement of the WARDS project, utilize lessons learned and best practices in planning and implementation of the ongoing REDISSE program, in more than 12 countries in the sub-region. 2. Reapply the major simulation exercises whilst considering the multidisciplinary features and multisector coordination and needed expertise and costs to effectively carry this out. To test the level of readiness and response by countries to address permanent epidemic and other public health threats to communities in the sub- region. 3. Evaluate procurement procedures of implementing agencies and, if possible, delegate to them responsibility for procurement and deliveries, whilst adhering to recommended rules and principles by audits and funding agencies. 4. Carry out advocacy with the States for the pursuit of harmonization in the equivalence of certificates. At the time where graduates of the FELTP program are automatically promoted and integrated with specialists holding CAMES’ Certificate for Specialized Studies, Senegal is waiting to certify FELTP program certificates and its integration in higher grades in civil service. This causes frustration in trained health workers. 5. Promote exchange and regular interaction between implementing agencies of new projects. If possible, encourage joint planning of interventions to be carried out in each target country of the project. 6. Create a working group on governance of medical biology policies (LD, referral labs, LNSP, …). ECOWAS countries: 1. Deploy additional efforts to convince and encourage women to register for FELTP training and to participate in the various capacity building trainings of technical skills; 2. Plan for mentoring and coaching of future promotions by FELTP alumni, including a little incentive; 3. Specific efforts to register women in future trainings is highly advised and even recommended. 55 Organizers of the FELTP course: 1. Improve the conditions of internships in the field through improved supervision of trained people by their trainers (follow the example of the University of Ghana which planned at least one visit for students during their time in their countries). 2. Plan and put in place systems of skills and duties transfer from the CDC to local personnel (2-3 years). Within this framework, starting this year, the University of Ouagadougou plans to appoint a Director for the Master’s program in field epidemiology. Conclusion This project provided the great opportunity to demonstrate its technical and management skills, unknown by certain partners. This also enabled it to excel both within and outside the ECOWAS zone. In fact, beyond the achievement of project development objectives and high achievement rates of intermediate results indicators, the project has successfully played its coordinating role in promoting the provision of significant funding by the World Bank (for instance, the REDISSE program) as well as other partners. 56 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders 1. Linkage to relevant strategies. • No mention is made on ECOWAS region existing health strategies : e.g. WAHO strategic plan as well as WHO:/IST health strategies for the West African countries; 2. Project design • On the line 17, the following assertion is not appropriate [The project was also aligned with the 2010 Strategy for Accelerated Growth and Sustainable Development (SCADD) of the Government of Burkina Faso, the host country of the WAHO]. The location of WAHO Headquarters in Burkina Faso has nothing to do with the “Strategy for Accelerated Growth and Sustainable Development (SCADD) of the Government of Burkina Faso”. WAHO scope of activities is the 15 Members states. The linkage made is not consistent, according to me. 3. Assessment of Bank and Borrower Performance • 5.2 Borrower Performance: why did you combine WAHO and IPs performance? This normally should be separated. WAHO should not be considered at the same level as IPs. WAHO played the role of overall coordination, though he carried out some activities, basically procurement activities. 57 Annex 9. List of Supporting Documents Fondation Merieux : Rapport Narratif Renforcement des capacités des ressources humaines dans le domaine du Laboratoire dans le cadre du projet WARDS. Période du 1er janvier au 31 mars 2017 List of WARDS FELTP Residents and Job Status Report The Ghana Field Epidemiology and Laboratory Training Programme (FELTP). Annual Report, 2015 Ghana FELTP and WAFELTP Report: Cross boarder Capacity Development to prevent avoidable epidemics, detect early threats and respond rapidly and effectively in infectious Disease outbreak The Ghana FELTP Report on Training of WAHO-Sponsored Residents, September 2014 - February 2015 The Ghana FELTP Progress Report - WAHO: August – December 2016 The Ghana FELTP. Progress Report - WAHO: January – April 2017 Rapport : Reunion Regionale CCISD/WARDS sur Les Centres de Surveillance Epidemiologique dans les Pays de la CEDEAO. Ouagadougou, Burkina Faso, du 24 au 25 mai 2017 Rapports Situation Evaluation Pays de la CEDEAO Sur « JEE » Lien pour les rapports JEE : http://www.who.int/ihr/procedures/implementation/en/ WARDS Project: Final Supervision Report on the. April 03 - 14 2017 Termes de Référence (TDR) pour les travaux de groupe Atelier sur le master FELTP Ouagadougou les 23 et 24 septembre 2009 University of Oslo: Implementing Data Warehouse Terms of Reference WARDS Implementation Support Mission : Aide-Mémoire. April 03 – 14 2017 WARDS Initial Implementation Status and Results (ISR) Report: Sequence No 1. December 23 2013 WARDS ISR Report: Sequence No 2. June 3 2014 WARDS ISR Report: Sequence No 3. November 25 2014 WARDS ISR Report: Sequence No 4. May 15 2015 WARDS ISR Report: Sequence No 5. November 23 2015 WARDS ISR Report: Sequence No 6. June 27 2016 WARDS ISR Report: Sequence No 7. August 1 2016 WARDS ISR Report: Sequence No 8. November 3 2016 WARDS Final (ISR) report – Sequence No 9. May 2017 WARDS Termes de Référence : Centre de Coopération Internationale en Santé et Développement (CCISD) 58 MAP INSERT MAP HERE AFTER APPROVAL BY SENIOR GLOBAL PRACTICE DIRECTOR AN ORIGINAL MAP OBTAINED FROM GSD MAP DESIGN UNIT SHOULD BE INSERTED MANUALLY IN HARD COPY BEFORE SENDING A FINAL ICR TO THE PRINT SHOP. NOTE: To obtain a map, please contact the GSD Map Design Unit (Ext. 31482) A minimum of a one week turnaround is required 59