Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) Report Number: ICRR0021737 1. Project Data Project ID Project Name P118858 GH-Statistics Development Program (FY11) Country Practice Area(Lead) Ghana Poverty and Equity L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-50100,TF-10425,TF-A6364 31-Aug-2016 41,421,762.66 Bank Approval Date Closing Date (Actual) 25-Aug-2011 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 30,000,000.00 14,500,000.00 Revised Commitment 44,500,000.00 14,500,000.00 Actual 41,421,762.66 14,500,000.00 Prepared by Reviewed by ICR Review Coordinator Group Paul Holden Judyth L. Twigg Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO) of the Ghana Statistics Development Project was to strengthen the National Statistical System (NSS) in the production and dissemination of timely and robust statistics relevant for evidence-based policy making and other uses (Financing Agreement, p. 5). The PDO can be parsed into 3 objectives: Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) • Strengthen the NSS in the production and dissemination of timely statistics; • Strengthen the NSS in the production and dissemination of robust statistics; • Strengthen the NSS in the production and dissemination of statistics relevant for evidence based policy making and other uses. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components There were four components. Component 1: Institutional reform and organizational change: Appraisal US$ 9.36 million; Actual US$ 3.18 million. The goal of this component was to strengthen the Ghana Statistical Service (GSS) through an institutional reorganization. Reforms were to consist of: (i) redesigning the GSS organizational structure; (ii) analyzing staff requirements under the new structure; (iii) undertaking capacity and competency assessments to identify suitably qualified staff within the new structure; (iv) redeploying eligible staff; (v) identifying outsourcing opportunities; (vi) undertaking any additional necessary recruitment; (vii) developing and implementing a staff buyout program; and (viii) undertaking a capacity development program. There were two sub-components: Sub-component 1a: Training and Re-deployment was to support a capacity building program directed firstly at staff who had been incorporated into the new institutional structure and secondly at those who had been assigned to statistical units at District, Municipal, and other levels. In addition, this component was to provide training to upgrade staff to a qualification standard equivalent to a Master’s degree, which had been designated as the minimum requirement for anyone entering employment in the GSS. Subcomponent 1b: Staff Buyout Program was to consist of a staff buyout program with the goal of assisting the GSS to buy out up to 300 original staff members. As part of this program, financial and other counseling was to be provided to retrenched GSS staff to assist with retraining or other appropriate forms of assistance. Component 2: Enhance statistical capacity and the underlying statistical framework to ensure production of quality statistics: Appraisal US$14.96 million; Actual US$ 14.43 million. Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) There were three sub-components: Sub-component 2a: Upgrading the Skills of the GSS Staff was to provide staff at both the national and regional levels with training on data collection, statistics production, and validation and analysis. Sub-component 2b: Improving the Statistical Framework was to support development of a master sampling framework derived from the 2010 population and housing census, including census post-enumeration mapping, cartographic mapping, and business register. This also included technical advisory services and the acquisition of goods. Sub-component 2c: Investing in Physical Infrastructure and Equipment was to support procurement for the various statistical offices of furniture, information technology, and other needed equipment to support the proper functioning of the statistical units. It also supported the establishment of the GSS data center, ten regional mobile registration centers to record births and deaths, and also transportation for data collection. Component 3: Data production and dissemination: Appraisal US$ 18.88 million; Actual US$ 22.11 million. The goal of this component was to improve the collection, accuracy, analysis, and dissemination of data. This was to be achieved through two subcomponents: Sub-component 3a: Data Development and Management was to implement the long-term census and survey program through enhanced data collection, strengthen management information systems, and develop a national statistics databank. Sub-component 3b: Communication was to enhance the dissemination of official statistics, communication, and outreach. Component 4: Project management and monitoring and evaluation (M&E): Appraisal: US$ 1.31 million; Actual US$ 1.47 million. This component was to establish a Project Implementation Unit within the GSS that would be responsible for all fiduciary components and provide a coordination function for the project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Comments on Project Cost, Financing, Borrower Contribution, and Dates: The original planned project cost was US$40 million funded through IDA (US$30 million) and the Statistics for Results Facility (SRF) (US$10 million). There were significant exchange rate losses resulting from a depreciation of the US dollar relative to the SDR. Additional SRF financing of US$4.5 million was fully disbursed. By closing, 100% of the project funds, amounting to US$41.2 million, was disbursed. There was a US$34.5 million planned borrower Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) contribution, with US$16.9 million actually disbursed. Overall, the original total project amount was US$ 74.5 million; actual total project costs were US$ 58.1 million. Dates: The project was approved by the Board on August 25, 2011, with a planned implementation period of five years. However, there were substantial delays arising from the time it took Ghana’s legislature to approve the project documents. This finally occurred in February 2014. The effective date of project commencement was April 28, 2014. There were two level 2 restructurings to revise the results framework, to extend the closing date, and to reallocate funds between the various components of the project, which were based on the recommendations of the Mid-term Review. The first restructuring occurred on May 6, 2016 and the second on June 21, 2017. Additional financing was approved on May 8, 2018. There were no other changes. 3. Relevance of Objectives Rationale At the outset of the project, the objective of enhancing Ghana’s statistics through strengthening the NSS was relevant to the National Strategy for the Development of Statistics (2009-2013). It remained relevant at closing to the National Strategy for the Development of Statistics II (2018-2022). It was also relevant to the World Bank Country Partnership Strategy (CPS) for FY13-FY18, which had three pillars, one of which was improving economic institutions, and which had identified statistical capacity building as being of critical importance. The CPS contained an appendix devoted to the assessment of Ghana’s statistics. At closing, a new Country Partnership Framework had not been completed. However, the 2018 Systematic Country Diagnostic identified the reliable collection of data and statistics as an important priority, stating that "effective budget planning will require reliable data collection and statistics” (Ghana, Systematic Country Diagnostic, p. 57). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen the NSS in the production and dissemination of timely statistics. Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) Rationale The causal chain underlying this objective was that better and more reliable production of statistics to support policymaking would lead to wider data dissemination that would also support evidence-based policy making. The inputs related to this objective related to enhancing the capacity of the GSS and Ministries, Departments and Agencies (MDAs) to collect, process and analyze statistics on a timely basis. Inputs also included upgrading the physical infrastructure and equipment associated with the statistical framework. The outputs related to the objective were enhanced information and communication technology (ICT) capacity, training for statistics professionals in the GSS and MDAs, and upgrading the knowledge base of GSS staff in the area of economic statistics. These outputs were to enhance the capacity of the NSS to produce and disseminate on a timely basis statistics that were in accordance with international best practices. The Indicator relating to this PDO was the number of months between data having been collected and the release of the main reports for the household surveys. These were disaggregated into: • The Annual School Census, which had a baseline of three months in mid-2011 and a target of one month by end December 2018. This was achieved. • The Ghana Living Standards Survey (GLSS), which had a baseline of 18 months at end December 2006, with an initial target of nine months for end-August 2016. This was revised down to six months by end December 2018, in the first restructuring. The actual achievement was eight months. • The Labor Force Survey (LFS), for which there was no baseline. The target for production was every three months by end December 2019. This was achieved by end December 2018. With two of the disaggregated indicators achieved and one largely achieved, the rating for this objective is Substantial. Rating Substantial OBJECTIVE 2 Objective Strengthen the NSS in the production and dissemination of robust statistics. Rationale The causal chain underlying this objective was that more comprehensive, accurate and reliable statistics obtained through the adoption of international standards would result in more reliable and useful data that would guide policy making and analysis and enable progress towards policy goals. Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) The inputs related to this objective were to enhance the capacity of the NSS framework, namely the GSS and MDAs, to collect, process, and analyze statistics. Activities included training for statistics professionals in the GSS and MDAs, and upgrading the knowledge base of GSS staff in the area of economic statistics. Inputs also included upgrading the physical infrastructure and equipment associated with the statistical framework. The outputs related to the PDO were enhanced ICT capacity. This would lead to the outcome, namely the increased production and dissemination on a timely basis of robust statistics that were in accordance with international best practices. The indicator relating to this PDO related to the production of better and more accurate economic statistics for two key series, rebased in accordance with internationally recommended practices, with a base year that was within the previous 5 years. The two series were: • The Consumer Price Index (CPI), which had not been rebased since 2002, with a target for producing the rebased CPI by end August 2016. This target was revised to end December 2019 in the restructuring. The ICR (p. 40) indicates that the formal target was not achieved. Although the statistics were rebased, they were still under review by the government statistician at the end of December 2019. • Gross Domestic Product statistics, which had been based on September 2011. They were successfully rebased by end December 2018. The rebasing of these two statistical series updated the underlying data and made them more robust. Rating Substantial OBJECTIVE 3 Objective Strengthen the NSS in the production and dissemination of statistics that were relevant for evidence-based policy making and other uses. Rationale The causal chain for this objective was that a strong NSS and NSS framework would produce statistics that were more accurate and timelier and would therefore be used more extensively for policy making that was based on evidentiary data, rendering them more relevant. This objective was to be achieved through a redesign of the organizational structure of the GSS, and an assessment of staff skills in order to best assign staff to positions where their competencies could be best utilized. Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) There were four outputs achieved under this objective. They were: • Prepare a draft statistics law for the NSS, that would establish the legal basis for the statistical framework. • Develop a new organizational chart for the GSS and re-post staff to positions in it. • Increase the number of staff who have training equivalent to a master’s degree. • Revise key Human Resources documents and Human Resources policies. The key outcome indicators were user satisfaction with statistical products and services of the NSS as measured by: i) User satisfaction with the statistical products and services of the NSS, measured by a User Satisfaction Survey (USS) with a reported baseline of 40% in 4011, an original target of 60%, a revised target of 90%, and achievement of 94.5% in 2018. ii) The Ghana Statistical Capacity Index score (overall score and disaggregated by the following components: methodology, source data, and periodicity and timeliness). The overall score improved from a baseline of 64 in 2011 to 71.1 in 2018, achieving the target of 70. iii) The proportion of censuses and surveys implemented by the NSS as per the long-term census and survey plan (GLSS, multiple indicator cluster survey, demographic and health survey, LFS, and integrated business enterprise survey). This indicator was dropped in the first restructuring. However, the PDO indicator, the “Statistical Capacity Index,” constructed by the World Bank and other international organizations, employs a composite statistical score based on a 100-point scale. The ratings on each of the components listed above are unclear as to what they actually mean and how they should be combined to achieve an overall rating and how they impact, positively or negatively, the achievement of the PDO. With regard to user satisfaction, there were issues with the survey data (ICR, p. 11), primarily from the two-year delay in project implementation. In particular, the USS had a 2011 baseline of 40% with an original target of 60% (ICR, p. 37). However, the first survey after the project became effective established a baseline of 88.5% even though project activities had not commenced. The target was formally revised in the first restructuring to 90%, with actual achievement by end December 2018 of 94.5%. The ICR highlights that the survey methodologies for the 2011 survey and that for the 2014 survey were different. There were no indicators related specifically to the increased use of data produced by the NSS for evidence- based policy making. The ICR (p. 23) refers to USS scores regarding the statistical products and services produced, which it points out show high levels of satisfaction on the part of users with respect to “the statistics, publications, databases and services offered by official statistics producers.” The ICR also points out (p. 23) that satisfaction increased between 2016 and 2018, when the second USS was undertaken. The ICR and discussions with the TTLs also offer additional evidence related to the circulation of statistics and their use as an indication of their importance for evidence-based policy making. “Among other policy decisions that benefitted from the availability of data is the Government of Ghana’s use of the LFS survey data through the Ministry of Employment to design the youth employment schemes and the use of the GLSS report by the Ministry of Gender, Children and Social Protection for targeting the poor and the vulnerable” (ICR p. 23) as well as for the targeting of community health outcomes. Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) The ICR (p. 23) points out that the USS indicates the increased use of statistical products in the country, which is in line with this objective. Although there were issues related to the statistical capacity index indicators, and the lack of an indicator directly to measure the use of statistics for evidence-based policy making, IEG rates the achievement of this objective substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale With all three of the objectives rated substantial overall efficacy is rated substantial. Overall Efficacy Rating Substantial 5. Efficiency No economic and financial analysis was undertaken. The ICR (p. 24) states that “the project does not lend itself to a financial analysis because the National Statistical Office has limited opportunities to recover costs” and that applying traditional cost benefit analysis would be difficult because of problems with quantifying the benefits in monetary terms. Furthermore, the ICR points out that the expected payback from the reforms would only accrue in the medium to long term with respect to fiscal savings and effectiveness. Efficiency gains accrued from the strengthening of the statistical framework, the physical and technological infrastructure of the GSS, and the statistical staff of the implementing MDAs. These gains have resulted in the wider circulation of statistics. Furthermore, the ICR (p. 25) points out that there implementation efficiencies achieved through smart reallocation of funds at the mid-term review, the payment by the government of MDA salaries, effective actions by the Bank of Ghana that ensured that project funds remitted by IDA were received and deposited in the designated account, and the effective work of the project secretariat. The delay in project effectiveness, which could have been anticipated during project design, was a minor shortcoming. Efficiency Rating Substantial Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s objectives were highly relevant to Ghana’s country conditions and to both the government’s and the World Bank Group's priorities at appraisal and closure. Overall efficacy is rated substantial, reflecting improvements in the production and dissemination of timely and robust statistics as well as compliance with international standards. Although there were issues with the definition and measurement of indicators for the relevance/use objective, these are considered to be minor shortcomings overall. Efficiency is rated substantial as a result of evidence of implementation efficiency. The overall outcome rating is therefore Satisfactory, indicative of only minor shortcomings in the project's preparation and implementation. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Risks at the closing of the project include (ICR, p 34) : • Potential lack of commitment on the part of MDAs to project implementation. • The possible failure of the government to fund capital expenditure and repair and maintenance of the system. • The Statistics Bill had not been passed at project closing, but there was confidence that it was only a matter of time before it would be passed. Furthermore, the government demonstrated its commitment to the project through its funding and the payment of salaries as described above. The institutional strengthening that had occurred under the project also reduces the risk to development outcome. Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) 8. Assessment of Bank Performance a. Quality-at-Entry Design of the project was based on in-depth analysis of the NSS that identified deficiencies in the existing system that hindered the provision of accurate and timely statistics to underpin policy decisions. Furthermore, there was extensive consultation between the Bank and GSS in order to ensure full agreement regarding reform priorities and the key elements of project design. The Bank team carefully aligned the project's design with the country's overall development plan. The Project Appraisal Document was prepared jointly between the Bank and the NSS in order to ensure that project resources were used most effectively and that risks to development outcomes were identified and mitigation measures put in place. Nevertheless, there were moderate shortcomings related to M&E design and the design of key outcome indicators. . Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Following the delays arising from the time it took to approve project documents, close supervision by the World Bank contributed to project achievements. The two level II restructurings allowed for the extension of project closing, which together with the Additional Financing contributed to the outcomes achieved. Regular supervision missions were conducted by the World Bank team and "facilitated effective project implementation and ensured value for money, transparency, and accountability" (ICR, p. 33). The ICR also points out that having the project team mainly based in Ghana allowed for ongoing support and follow-up with regard to implementation issues and also insured close communication with GSS. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) a. M&E Design The GSS coordination team was responsible for monitoring and evaluation, including the updating of the results framework for the project and compiling and reporting on performance indicators. There were shortcomings in the design of the M&E framework. Regarding the USS data, there were problems with the baseline survey data, arising primarily from the delays in commencing the project. Regarding the Statistical Capacity Index, it is unclear how its constituent elements were combined and how these translated into achieving the PDO. There were no indicators to measure the use of statistics for evidence-based policy making. b. M&E Implementation The ICR reports (p. 32) that data on performance indicators were collected every six months, with the timetable for collection coinciding with World Bank supervision missions. The ICR indicates that the Project Implementation Unit produced progress reports on a regular basis, including information on budgets, progress on capacity building, the production and dissemination of data, and information on procurement and financial management. This information was included in Implementation Status and Results Reports (ISRs). The Mid-term Review also tracked the indicators and recommended additional financing for the project. c. M&E Utilization The ICR (p. 32) reports that the indicators were included in the ISRs as an indication of whether the outcomes were being achieved, although it is difficult to ascertain how the Statistical Capacity Index was used, since it is only prepared sporadically. Nevertheless, once the delays had been overcome, the ISRs all rated progress with regard to achieving the PDOs as Satisfactory. Overall M&E Quality is rated substantial M&E Quality Rating Substantial 10. Other Issues a. Safeguards There were no environmental or social safeguard issues triggered during the project. It was rated Environmental Assessment Category C. Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) b. Fiduciary Compliance No fiduciary compliance issues were noted. The 2.5-year delay in project approval on the part of the legislature resulted in a US$3 million loss as a result of the weakening of the US dollar vis à vis the SDR. There were five audits over the life of the project, with the final audit indicating that there were no irregularities. Procurement was undertaken in accordance with World Bank procedures (ICR, p. 25). Of the 101 planned procurements, 100 were successfully completed, while one contract was terminated. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following lessons are taken from the ICR, with some adaptation of language by IEG: • Delays in project approval can increase project costs. The potential for delays can be reduced by addressing risks up frontduring project design. In the case of this project, the lengthy delay reduced resources available for the project. • Collaboration between stakeholders raises the chances of project success. In the case of this project, close stakeholder cooperation was enhanced through the establishment of a steering committee and could serve as a model for other projects of this type. • The effective prioritizing and sequencing of key activities, including capacity development to upgrade staff skills, can result in strong ownership on the part of country counterparts. In the case of this project, this prioritization maximized the use of project resources. Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review GH-Statistics Development Program (FY11) (P118858) IEG adds the following lessons: • Assessing project effectiveness can best be addressed through careful indicator design. In the case of this project, not incorporating an indicator to determine the extent to which the improved statistics were being used in policy and analysis limited the ability to assess project impact. • Using international indices as outcome indicators can be of limited usefulness. First, they often only become available with a lag and are therefore of little use in monitoring progress. Second, they are not customized to the specific circumstances of the country or specific project. Using the Statistical Capacity Index in the case of this project added little to the understanding of project achievements. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides a strong analysis of the project and description of the various aspects of performance and the causal chain. Shortcomings exist, however. The assessed achievement of outcomes, rather than of the PDOs. The discussion of the relationship between the outcome indicators and the PDOs is limited and insufficiently analytical. In particular, the description of the M&E framework lacks depth. a. Quality of ICR Rating Substantial Page 13 of 13