Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) Report Number: ICRR0022237 1. Project Data Project ID Project Name P133070 SL-Revitalizing Ed Development (FY14) Country Practice Area(Lead) Sierra Leone Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-61070,TF-16568,TF-17720,TF- 28-Feb-2017 31,392,314.77 A2012 Bank Approval Date Closing Date (Actual) 31-Jul-2014 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 21,314,800.00 21,314,800.00 Revised Commitment 21,070,094.71 21,070,094.71 Actual 21,070,094.71 21,070,094.71 Prepared by Reviewed by ICR Review Coordinator Group Shiva Chakravarti Judyth L. Twigg Joy Maria Behrens IEGHC (Unit 2) Sharma P163161_TBL Project ID Project Name P163161 Sierra Leone REDiSL AF ( P163161 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0 Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) Bank Approval Date Closing Date (Actual) 20-Jun-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the financing agreement (p. 4), the project's objectives were “to improve the learning environment in targeted schools and establish systems for monitoring of education interventions and outcomes.” b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1: Improving the learning environment and opportunities in targeted areas (Original cost: US$16 million. Actual cost: US$24.6 million): This component aimed at improving the learning environment through a package of improved financing of schools, training of teachers, provision of quality early childhood care and education (ECCE) services, and early literacy. Sub-component 1.1: Performance-based school grants (Original cost: US$5.1 million. Actual cost: US$13.9 million). This subcomponent aimed to improve access to and the learning environments in primary and junior secondary schools (JSSs) in targeted districts in Local Councils (LCs) that were identified as having the lowest education outcomes and highest rates of poverty. Financial incentives were to be provided to schools based on their delivery of outputs, with the disbursed amount being dependent on the results that schools achieved. Sub-component 1.2: Pilot approaches to increase school readiness (Original cost: US$1 million. Actual cost: US$2.5 million). This sub-component aimed to build on existing practices of ECCE, as traditionally exist in Sierra Leone, and to pilot a cost-effective approach to ECCE and develop quality assurance standards. The project was to construct ECCE classrooms that would be attached to existing government- Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) supported Community Learning Centers. The cost-effective approach included providing three weeks of pre-service training and ongoing in-service training, which led to certification. Sub-component 1.3: Strengthen reading outcomes at early grades (Original cost: US$6.9 million. Actual cost: US$5.8 million). This sub-component supported the Education Sector Plan’s (ESP's) aim of improving the pedagogical methodologies used by teachers to improve learning outcomes by supporting a comprehensive national reading program intended to improve literacy in early grades and create a culture of reading as a foundation for learning. Sub-component 1.4: Improve teacher management (Original cost: US$3 million. Actual cost: US$2.4 million). This sub-component aimed to support the establishment and operationalization of the Teacher Service Commission (TSC) to support the effective and efficient management of the teaching labor force in Sierra Leone. Revised Component 1: The project had accumulated savings primarily from the lower-cost development and delivery of supplementary readers for grades one to three supported under Component 1. During the project’s second level II restructuring, and given the UK Department for International Development's (DfID’s) decision to cancel its second tranche of funding (see Section 2e), the decision was made to reallocate US $924,000 from Component 1 to Component 2.  Performance-based school grants (sub-component 1.1). This sub-component was continued and scaled-up under AF 2 to include: (i) the provision of performance-based school grants to the 1,350 originally targeted schools for an additional five rounds; (ii) 500 new primary and JSS schools in four additional LCs for six rounds; and (iii) selected pre-primary schools/classrooms where the teachers that were trained under the original financing had adopted the play-based pedagogical approach.  Improvements to teacher management (sub-component 1.4). This sub-component was continued and scaled-up under AF 2 to provide technical assistance to the TSC to execute its functions in three (of the four) statutory mandate areas: teacher registration and licensing of teachers, teacher management, and teacher performance and development. Component 2: Strengthening education service delivery (Original cost: US$5 million. Actual cost: US$2.9 million). This component sought to provide support in three main areas: (i) implementing and monitoring specific elements of the education sector strategy (ESS); (ii) coordinating donor activities in education; and (iii) addressing capacity gaps at the Ministry of Education, Science and Technology (MEST) and local levels. Ultimately, this component sought to garner support from donors to work collaboratively with the project and MEST to strengthen education service delivery in Sierra Leone. Sub-component 2.1: Build the foundation for better measurement of learning outcomes (Original cost: US$0.5 million. Actual cost: US$0.5 million). This sub-component was designed to help the MEST determine what its students were learning and lacking, by building on outputs of the previous Education for All-Fast Track Initiative (EFA-FTI) operation and the work of donor partners. The project was to establish Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) and operationalize a semi-autonomous assessment unit, within the MEST, to coordinate the design and execution of assessments at lower and upper primary and JSS. Sub-component 2.2: Establish robust, consistent school data collection and analysis (Original cost: US$1 million. Actual cost: US$1.1 million). This sub-component aimed to support the MEST with the annual school census (ASC) that had previously carried out successfully, and overall, address data collection in the short term and establish a sustainable system for data collection and analysis within the Directorate of Planning and Policy. Sub-component 2.3: Establish a system for driving and monitoring the implementation of the Education Sector Strategy (Original cost: US$3.5 million. Actual cost: US$1.3 million). This sub-component aimed to support the MEST to bring about desired improvements in education and to monitor and evaluate the ESP by setting up systems and processes for delivering and monitoring key outcomes. Revised Component 2: Following DfID’s cancellation of the second tranche of its contribution to the project, the reallocation of funding from Component 1 was still insufficient to cover all of the activities initially planned under Component 2 in the original design. In response, the project team undertook a strategic prioritization of activities to retain those that were critical for achievement of the objectives. Although the scope of some of the project’s activities was reduced in response to the decrease in financial commitments from DfID, the changes made in response were commensurate with the lower commitment size.  Establish robust, consistent school data collection and analysis (sub-component 2.2). This sub- component was continued under AF 2 and financed the national roll-out of a digital ASC in 2018, using digital technology for data collection, as well as for data analysis, reporting, and dissemination. Component 3: Project management and supervision (Original cost: US$0.95 million. Actual cost: US$1.5 million). This component was to cover inputs required to support MEST for administrative management of the project. It was also to support capacity building, particularly at the decentralized levels, for fiduciary management, procurement, and auditing. Sub-component 3.1: Project management and supervision (Original cost: US$0.95 million. Actual cost: US$1.3 million). This sub-component was designed to provide support for the upgrade of the supervision mechanism, financial audits, qualitative audits, and impact evaluations (IEs), in order to complement technical assistance on improving data collection and management being provided to the MEST by donor partners. Revised component 3: Similar to the original financing, this sub-component also financed project management and supervision activities under AF 2. Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) Component 4 (Added): Support the implementation of the MEST Ebola Strategic Response Plan (ESRP) (Original cost: US$1.45 million. Actual cost: US$2.4 million).This component sought to address emergency needs identified by the MEST as a result of the Ebola crisis. When introduced during the project’s first level II restructuring, Component 4 had three subcomponents, and an additional sub- component was added under AF 1. Sub-component 4.1: Radio and television program (Original cost: US$566,325. Actual cost (AF 1): US$566,325). This subcomponent aimed to support the MEST’s efforts to maintain learning for pre-primary, primary, junior secondary, and senior secondary school students during the period of Ebola-related school closures by providing support for development of radio and television programing for remote learning. Sub-component 4.2: Establishing safe and secure learning environments (Original cost: US$763,200. Actual cost (AF 1): US$1,669,785). This sub-component aimed to support the MEST and the government in ensuring that schools were safe for students’ return, especially as some schools had been used as "holding centers" by the Ministry of Health and Sanitation, due to insufficient beds in hospitals and health centers, for suspect cases. The project provided support for disinfecting these schools and providing handwashing stations and hygiene kits. Sub-component 4.3: Monitoring of the Emergency Radio Education Program (EREP) (Original cost: US$0 [used existing M&E allocation]. Actual cost (AF 1): US$55,800). This sub-component aimed to support the monitoring of the EREP, which was to be coordinated by the Planning and Policy Unit of MEST that was already handling the M&E of the project and had a specific allocation dedicated to this activity. Sub-component 4.4: Social mobilization and public information. This sub-component was added under AF 1. It aimed to support the Social Mobilization Working Group (SMWG), which was created as a part of the Education Response Task Force consisting of MEST staff, key partners, and civil society. The SMWG worked on issues related to the reopening of schools in Sierra Leone to support: (i) the creation of an enabling environment for children, parents, and communities at large to ensure that schools were safe for children and had an overall positive impact on their health and well-being; and (ii) public information for communities and teachers, by relaying correct and consistent messages on safer school environments and infection prevention within schools through multimedia packages. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was approved on July 31, 2014 and became effective on September 29, 2014. It was initially financed by a US$17.9 million Global Partnership for Education (GPE) grant and a US$ 5.5 million Multi- Donor Trust Fund grant. The first restructuring of the project, completed on December 16, 2014, re- allocated US$1.45 million from its original funding to recovery efforts for the Ebola crisis (emergency phase) through the addition of a new fourth component. An Additional Financing (AF 1) in the amount of US$950,000 was approved on June 3, 2016 to expand recovery efforts in the context of the Ebola crisis, which included school reopening interventions (ICR, p. 8). This AF was financed by the Ebola Recovery & Reconstruction Trust Fund. In October 2016, DfID withdrew its remaining commitments of US$3.1 million of its original allocation of US$5.5 million from the Multi-Donor Trust Fund to instead invest funds directly into the President’s Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) Recovery Plan's priority areas for the crisis (ICR, p. 8). In response to DfID’s withdrawal, a second level II restructuring was approved on February 14, 2017. This involved three changes: (i) an extension of the project closing date - from February 28th, 2017 to December 31, 2017; (ii) re-allocation of funds from Component 1 to Component 2; and (iii) cancellation and narrowing of select activities under Component 2 (ICR, p. 8). On June 20, 2017, a second additional funding (in the amount of US$10 million, IDA credit) was processed to scale up and strengthen several activities (ICR, p. 8). The scale-up included expanded performance- based school grants and increased support for strengthening accountability and school-based management capacities (ICR, p. 8). The project closing date was also extended for an additional two years. Although the original financing closed on December 31, 2017, the additional financing became effective on December 7, 2017 and closed on December 31, 2019 (ICR, p. 8). 3. Relevance of Objectives Rationale Project objectives broadly aligned with the country context at the time of approval. In Sierra Leone, inequities exist for educational opportunities. The PAD (p. 6) described these inequities as existing along socio-economic, geographic, and gender lines. Additionally, “the education system was operating with little information management and almost no assessment information" (PAD, p. 6). The project attempted to address these inequities and information challenges. The PAD (p. 4) stated that one of the main challenges in the sector was “ensuring access and retention across all levels of schooling,” but other development partners were addressing access issues, and this project focused in a complementary manner on the learning environment and systems of monitoring (PAD, p. 7; ICR, p. 3). The project retained relevance during the Ebola crisis by adding a component to support the government’s ESRP. The project’s objectives were consistent with the Country Assistance Strategy (CAS, FY 2010-2013) at approval. The first objective of this CAS was "increased access to and completion of primary schooling especially for girls and out of school children," with an outcome indicator of "improved capacity to effectively and efficiently deliver education" (CAS, p. 18). The systematic country diagnostic for Sierra Leone also included a priority area to improve quality and access to education (SCD 2018, p. xvii). The objectives continued to be relevant to Bank strategy at project closing. At project closure, Sierra Leone’s draft Country Partnership Framework (CPF) for the period FY21-FY26 outlined the government’s progress towards achieving (i) access, equity, and completion; (ii) quality and learning outcomes; and (iii) systems strengthening. The project was also aligned with government policy. The PAD noted that the project was designed to provide the government with the first opportunity to implement key strategic areas of its ESS (PAD, p. 8). Rating Relevance TBL Rating Substantial Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve the learning environment in targeted schools in Sierra Leone Rationale The theory of change was convincing. The project aimed to address critical challenges of improving the learning environment: high education costs, limited capacity, a lack of ECCE classrooms and educators, and low levels of readings in early years (ICR, p. 15). To address these challenges, the project intended to provide performance-based grants to schools, construct pre-primary schools, certify ECCE educators, train teachers, provide reading material under a national reading campaign to encourage reading, and operationalize a TSC. In the changed context of the Ebola crisis, improving the learning environment also included continuity of student learning, and the project intended to support implementation of the ESRP through development of emergency radio and television programs for pupils and establishment of safe environments for reopening of schools. There were several embedded assumptions in the theory of change. For performance-based grants, there was a logical assumption that learning environments will be improved if schools are open on a greater number of days and teachers and students are present in the classroom. However, the presence of teachers and students in classrooms of open schools could still translate into poor or no teaching. Similarly, construction of ECCE classrooms may not translate into improved learning environments if they are not being used effectively. The project addressed some of these assumptions by using indicators like pupils’ reading efficiency and conducting an independent study to measure the effectiveness of teacher training. Overall, the planned activities were reasonably expected to improve the learning environment by providing students with better infrastructure, trained teachers, better quality and management of teachers through TSC, and reading material constructed to enhance skills. Outputs  Performance-based financing grants were paid to 1,350 originally targeted schools in four original LCs (Kenema, Pujehun, Kambia and Tonkolili) for five rounds and to 500 new primary and JSS schools in four additional LCs (Kono, Koinadugu, Falaba and Karene) for six rounds, and also to selected pre- primary schools/classrooms where the teachers that were trained under the project adopted a play- based pedagogical approach. A third-party verification and audit of the performance-based grants was successfully carried out in 2019 and recommendations shared.  Standardized operational guidelines, standard operating procedures, tools, and templates were developed in ten school-based management areas. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070)  An ECCE curriculum and minimum standards were developed and validated in 2016 and finalized by the MEST in August 2017. Due to a change in administration in May 2018, the aforementioned documents were reviewed by the new leadership and subsequently approved.  An early childhood development (ECD) policy was developed and validated in 2016. An inter-sectoral review meeting on the ECD policy with senior officials of the MEST and related ministries (health, social welfare, local government finance, etc.) was held on December 19, 2017, to finalize the policy. The Minister of Education has signed the policy, and Cabinet approval of the ECD policy is in process.  50 classrooms (and ventilated improved pit latrines) annexed to primary schools were constructed, furnished, and equipped in the four originally targeted LCs.  117 preschools (comprising 239 teachers, 61 caregivers, and 117 head teachers) from the four originally targeted LCs received three days of Level I and six days of Level II training in five ECCE core competency areas, in line with the new ECCE curriculum, in May and August/September 2017.  More than 2.2 million early grade readers for grades 1-3 (6 titles - 2 book titles/pupil per grade) were distributed to all primary schools (> 6,000 schools) throughout the country (public and non-public schools). Approximately 460,000 grade 4 readers (2 titles per pupil) and 13,000 teacher guides were procured and distributed to all schools (>6,000).  Approximately 4,000 in-service teachers of grades 1-4, 1,000 head teachers, and 170 inspectors and supervisors from 14 districts received a 10-day on training on effective early grade reading instruction in July-August 2017 (with two levels of training).  TSC district offices were constructed at 14 sites, and furniture, equipment, and motorcycles for the TSC district offices were procured and delivered.  Operational costs of the TSC Secretariat (with the exception of Commissioners’ sitting fees), including the internet, utilities, communications, security, housekeeping, travel and transport, stationery, etc., were funded by the project.  Policy guidelines were developed in three mandate areas - teacher management, teacher registration and licensing, and teacher-employer relations -- and have been validated at national and district levels with various constituents (teachers, head teachers, district officials, teachers' unions).  A TSC operational framework (including the TSC organogram) and TSC corporate policies on human resources were developed. TSC consultancies on corporate policies, a district operations manual, and harmonization and standardization of teacher policies on registration and licensing were completed.  During the Ebola crisis, 600 hours of radio and television programming were carried out targeting schools at all levels. Subjects taught included math, language arts, science, life skills, and psychosocial issues and information about Ebola. 5,970 schools were disinfected and received handwashing stations according to established guidelines and protocols. 36,000 handwashing stations were procured and distributed to schools. Outcomes  85 percent of schools in targeted LCs met the 60% performance target from a baseline of 0, meeting the formally revised target of 85 percent of schools by the project’s revised closing date. Indicators included in the performance targets included: 100% student attendance, 100% of teachers teaching, 50% of pupils with books, 100% of pupils being assessed, 100% of teachers being observed, and pupils’ reading proficiency. Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070)  84 percent of teachers were using improved reading methods, from a baseline of 0, exceeding the target of 70 percent. A project-sponsored randomized controlled trial (RCT) of 96 treatment schools and 96 control schools found that trained teachers, on a scale of 1-3, had an average score of 1.22 as opposed to a score of 1.03 average score for teachers who were not trained.  88 percent of payroll teachers were teaching in schools where they had been authorized to teach, from a baseline of 62 percent, exceeding the target of 75 percent.  Following the Ebola crisis, 8100 schools were reopened from a baseline of 0, meeting the target of 8100. Achievement of Objective 1 is rated Substantial, as the project essentially achieved the intended outcomes. Rating Substantial OBJECTIVE 2 Objective To establish systems for monitoring of education interventions and outcomes Rationale The theory of change for the second objective was convincing. The project aimed to address critical challenges related to monitoring of education interventions and outcomes: limited school data information, information management, learning data, and monitoring capacity. In order to address these challenges, the project intended to establish and operationalize an assessment unit, provide short-term support for an annual school census, and support capacity development of MEST. It logically follows that these activities would help establish M&E systems for education interventions and outcomes. Outputs  A national learning assessment policy framework was delivered, and a validation session on that framework was held.  The project provided support to the analysis of 2016 West African Examinations Council results, the ranking of schools in different performance categories (league tables), and determination of targets and regulations around exam performance. The ranking is public (on the MEST website).  A paper-based ASC was held annually from 2015-2019. The 2015 and 2017 ASCs deployed a full questionnaire, but the 2016 census used an abridged version. ASC 2015, 2016, and 2017 reports were prepared, used in government plans and reports, and made available on request. The 2018 ASC was finalized and published in February 2019. The 2019 ASC was also successfully completed, the first school census since the introduction of the Free Quality School Education program, a government subsidy initiative designed to eliminate all school fees for parents and guardians. Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070)  Data cleaning, analysis, and report generation of "Situation Room" data (a system to collect key data from schools every month during the Ebola crisis) were undertaken by an analyst financed by the project. Support for one M&E officer and one data analyst was continued until project closure.  Internet connectivity in the MEST was enhanced through internet protocol telephony and local area networking. The MEST website was developed and upgraded. Outcomes  Seven donors were working with the ministry by project closure to strengthen education service delivery, exceeding the target of five donors. This PDO-level indicator, according to the ICR, was intended to capture “the project’s efforts to work with development partners to build the capacity of the MEST to establish systems for monitoring of education interventions and outcomes" (ICR, p. 17), aimed at overcoming a tendency prior to the project for donors to work in parallel rather than collaboratively. It is unclear how this indicator reflected achievement of the objective.  A national learning assessment for grades 4-5 in math, English and science was implemented in approximately 1,000 schools in 19 LCs in June/July 2017.. Achievement of Objective 2 is rated Substantial, as the project essentially achieved the intended outcomes based on the conduct of the ASCs and national learning assessment. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Based on the information presented in the ICR, the objectives to improve the learning environment in targeted schools and to establish systems for monitoring of education interventions and outcomes were essentially achieved. Overall Efficacy Rating Substantial 5. Efficiency Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) The economic analysis conducted at appraisal and for the ICR showed positive results. The PAD conducted a cost-benefit analysis showing that both internal rate of return (IRR) and net present value (NPV) of costs and benefits deemed the project economically viable (PAD, p. 26). The PAD estimated the NPV of program benefits equivalent to US$ 22.04 million. The analysis estimated the impacts of improved learning environments (test scores) on learning and labor market outcomes (earnings) for project beneficiaries. These estimates were calculated using existing studies to identify average improvement in test scores and earnings. The associated IRR was 20 percent. The ICR used the same assumptions and approach and estimated project benefits equivalent to between US$18 million and US$ 29 million. These assumptions included constant education and cognitive skills across benefit time, a 15 percent discount rate for future returns, a 30-year work period for beneficiaries, a 10 percent maintenance cost for constructed classrooms, and no opportunity cost associated with teacher trainings. The associated IRR was 19 percent for a base case scenario. The ICR noted that there were delays at the beginning of project cycle due to the Ebola crisis. This crisis resulted in closing of schools, which affected initial project disbursement rates. These challenges were tackled by the first level II restructuring in December 2014. During and after the Ebola crisis, consistent operational, technical, and implementation support resolved issues and ensured the smooth flow of funds, according to the ICR (p. 88), throughout the project life. Additionally, the project benefitted from a 34-month extension of its closing date, which allowed it sufficient time for the achievement of planned activities (ICR, p. 19). The project successfully used resources in an efficient manner, despite the removal and reallocation of co-financier DfID funds due to the Ebola crisis (ICR, p. 20). There were savings from reading books delivered to grades 1-3 because the costs were lower than estimated budget. These savings were allocated to compensate for withdrawn funds by DfID and enabled the project to avoid scaling-down of planned activities. The project also efficiently targeted LCs for performance-based grants by using education and poverty criteria to ensure that highly marginalized LCs received funding (ICR, p. 25). The project ensured a continuity of task team by involving team members who had prior EFA-FTI experience to avoid steep learning curves. Even when there were changes, including change in TTLs, the project's operations experienced continuity and consistency (ICR, p. 27). The Bank team maintained continuous policy dialogue with the government and other donor partners by using the Education Development Partners Group, consisting of education ministry staff, donor partners, civil society organizations, and non-governmental organizations. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  20.00  Not Applicable 0 ICR Estimate  21.50  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) 6. Outcome Relevance of objectives is rated Substantial, as the objectives were aligned at project closing with the Bank's country strategy and the country’s education sector policy. Efficacy is rated Substantial, as the project achieved its development objectives. Efficiency is rated Substantial based on the economic analysis, tackling of project implementation challenges during the Ebola crisis, and overall implementation efficiency. Overall, there were only minor shortcomings in the project’s preparation and implementation. The project's outcome is therefore rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR provided two main reasons for low risk to development outcome: government commitment and capacity, and the Bank's continued engagement. The new CPF for Sierra Leone (FY 21-26) outlines the goal of improved learning outcomes in primary and secondary education, especially among girls, indicating continued government commitment to the sector. Moreover, the skills acquired by MEST staff through project technical trainings will continue to be beneficial after its close. In addition, the Bank's recently approved Free Education project (US $ 50 million, FY 2020), with objectives to improve management of the education system, teaching practices, and learning conditions, should help sustain achieved outcomes. The ICR (p. 20) agrees that “many of the project interventions are system-wide improvements requiring sustained efforts and continuous reforms.” The project team reported that the Free Education project will scale up performance-based grants to more schools. The project also supported system-wide improvements, including systems for data collection and analysis within MEST. The project also undertook capacity building trainings for teachers and ministry staff. It is a reasonable expectation that these skills will accrue benefits well beyond the project. Additionally, the project team reported that the government intends to expand some of the implementation mechanisms of the project to distribute school subsidies, e.g. use of digital tablets to collect timely data on performance. 8. Assessment of Bank Performance a. Quality-at-Entry The objectives were consistent with the Bank's country strategy and the government's ESP (2014-2018). Though one of the interventions (performance-based school grants) was new in the education sector, the PAD stated that it had been tested in the health sector with some success (PAD, p. 10). The ICR stated that the Bank's education task team worked closely with the government and MEST during the preparation process. According to the ICR (p. 25), the Bank’s involvement added value because of Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) prior experience in the sector and country. Additionally, the Bank had prior experience with GPE modalities in 36 countries, along with experience of effectively managing multi-donor trust funds (ICR, p. 4). The PAD assessed the risks for the project with effective mitigation measures. For instance, to mitigate the risk of performance-based grants, an untested measure in the education sector, the PAD included an independent entity for verification as a mitigation measure (PAD, p. 24). The PAD also iterated the lessons reflected in the project design (PAD, p. 20). These lessons included requiring a team in place prior to start of implementation that remains in place throughout the project cycle, and establishing monitoring and evaluation mechanisms early in the project cycle. Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR, adequate budget and staff resources were allocated, and the project was closely monitored. The task team carried out regular implementation support missions to review implementation progress and provide implementation support. Under AF 1, the team undertook seven technical and implementation support missions in one fiscal year (ICR, p. 32). The team also prepared regular aides- memoire to inform the government of project implementation issues that required special attention. Similarly, implementation status reports provided management with a transparent assessment of the project’s achievements and challenges (ICR, p. 33). World Bank support and flexibility was especially important during the Ebola crisis, given weak government capacity and the challenging context. The project team increased its efforts during the Ebola crisis by using teleconferencing and other mechanisms to engage with the MEST. It processed no-objections swiftly to mitigate delays in procurement due to the crisis. The ICR (p. 33) stated that technical support from the Bank was crucial for supporting activities that were delayed due to the crisis. The Bank team connected the government to service providers and experts and provided technical assistance to narrow down the curriculum for radio broadcasts (TTL interview). The team “worked closely with the ministry and UNICEF and provided limited financing and technical assistance to the establishment of the 'Situation Room,' which captured real-time data on enrollment, state of classrooms and schools, and distribution of hygiene kits. On the administrative side, the team seconded an advisory consultant to the MEST to support the presidentially-appointed School Reopening Coordinator. The consultant was based on site and coordinated the efforts of the various working groups" (TTL interview). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) a. M&E Design The project supported a M&E team under Directorate of Planning and Policy, maintained by the project secretariat. The project also contracted an independent agency to validate performance-based grants through site visits (PAD, p. 81). Its sub-component 2.2: Establish robust, consistent school data collection and analysis strengthened monitoring activities of the project (ICR, p. 28). The project also incorporated lessons from previous EFA-FTI grant operations, which “indicated that quality of learning and teaching improvements, as well as growth in enrollments, are difficult to measure over the limited period" (ICR, p. 27). The project defined indicators that were adequate to measure outputs and outcomes defined in the results framework. Each objective had direct and clear indicators associated with it. Quantitative indicators provided a good snapshot of the impacts of the project, including measurement of grant implementation, construction of infrastructure, and re-opening of schools with Ebola prevention measures. The project also conducted an RCT to assess the effectiveness of teacher training. A weakness was that some indicators were inadequate to establish a firm comparison with the counterfactual. For example, the project provided performance-based grants to schools (Subcomponent 1.1), and the PDO-level indicator associated with the grants was "percentage of schools that meet 60 percent of performance targets." However, the ICR did not report disaggregated data on the level of specific performance measures, which included school opening days, teachers in class, and students in class. Additionally, there were no quality indicators. For example, schools could be open, and students and teachers could be in class, but this may not necessarily translate into an improved learning environment if the teachers are not teaching effectively. In addition, as noted earlier (Section 4), one of the outcome indicators associated with the second objective, the number of donors working collaboratively with the MEST to strengthen education service delivery, was not a clear measure of the objective to establish systems for monitoring of education interventions and outcomes. b. M&E Implementation The ICR stated that “the project successfully carried out five annual school and related analyses between 2015 and 2019. Paper-based versions were used in 2015, 2016, 2017 (under GPE/MDTF), and in 2018 and 2019, there was a migration from paper-based to digital-based data collection using tablets (under AF 2).... [T]he project provided census reports and statistical bulletins based on the collected data … and supplied data for the development of two dashboards for Ministry of Basic and Secondary Senior Education and the Directorate of Science, Technology and Innovation under the Office of the President" (ICR, p. 28). The ICR also stated that the project successfully “established and paid salaries for a technical data collection and M&E team comprised of one M&E specialist, two M&E officers, and one data analyst. This team supported (i) the monitoring of MEST activities; and (ii) project monitoring in the Situation Room.” Moreover, as stated in the ICR, the project contracted an independent firm to collect data on performance-based financing indicators in 2016 and 2017. Monitoring ratings were moderately satisfactory through most of the project. The ICR stated that “the Bank endeavored to provide more support pertaining to the project’s M&E during the mid-term review, which took place in July/August 2016. As of December 2016, improvements in M&E were noted. Specifically, the M&E team followed up on project activities and delivered several monitoring outputs, including the report on the first round of disbursement of performance-based school grants, the Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) construction of early childhood education classrooms, and the results framework. As a result, by project close, M&E was upgraded from Moderately Satisfactory to Satisfactory" (ICR, p. 28). c. M&E Utilization The ICR noted that the project utilized project M&E in three ways (ICR, pp. 29-30):  The project used data from the school census as the baseline for the initial design of its components. The ICR also stated that the project used school census data over the following academic years, to provide updates.  The data from the census also informed the payment of tuition fees, the supply of teachers and learning materials, and the distribution of furniture by the government and was used by education partners in the design and implementation of interventions.  The collected Global Positioning System coordinates for all of the schools in Sierra Leone, as well as photos of their buildings, facilitated the approval process for schools’ receipt of performance-based school grants. M&E Quality Rating Substantial 10. Other Issues a. Safeguards At appraisal, the project was classified as Environmental Category B and triggered the Environmental Assessment OP/BP 4.01 safeguard policy because of performance-based school grants. These grants could have involved construction or rehabilitation, but environmental impacts were expected to be localized, small scale, and manageable via minimum mitigation measures. In response, the project developed an Environmental Safeguards Management Framework (ESMF). These safeguards classifications were maintained throughout the project. However, minor updates occurred with the addition of AF2. A budget was added to sub-component 1.1 to facilitate ESMF implementation, including providing head teachers and school management committees with training on environmental guidelines. This training also covered environmental safeguards monitoring and implementation concerns. Safeguards compliance was rated as Moderately Satisfactory throughout the majority of the project’s lifespan (ICR, p. 30). b. Fiduciary Compliance Financial management: The ICR (p. 31) stated that financial management (FM) was rated "Moderately Satisfactory" for most of the project's lifetime. However, it noted that several changes were made to improve financial management (ICR, p. 31). Specifically, “the Secretariat transitioned to using the FM software recommended by the Bank. Secondly, audit reports were undertaken and were of unqualified opinion as they revealed no significant irregularities. Thirdly, though Interim Financial Reports (IFRs) were Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) mostly submitted in a timely manner and in accordance with established Bank procedures, there were some issues noted in terms of incorrect classification of expenditures and advances and incorrect reporting on exchange gains. To correct this, the Bank worked closely with the Project FM Specialist to rectify and then revise the IFRs accordingly. As a result, by June 2019, the FM rating was upgraded from Moderately Satisfactory to Satisfactory, due to the project team’s submitting IFRs in a timely manner and overall, good project financial planning" (ICR, p. 31). Procurement: The ICR (p. 31) stated that procurement was rated as "Moderately Satisfactory" for most of the project’s lifetime. Moderate shortcomings related to challenges because of the Ebola crisis. Nonetheless, the ICR noted that “when needed, procurement capacity was augmented by the Bank’s recruitment and training of a short-term procurement officer. However, there continued to be issues in procurement processes and/or contract administration, including but not limited to the poor filing of procurement documentation and delays in the timely completion of contracts. Nevertheless, necessary action was taken to address issues as they came up and the procurement rating was upgraded from Moderately Satisfactory to Satisfactory in June 2019, on the basis that: (i) the procurement team was awarding and managing contracts in a timely manner; (ii) post-procurement audit (completed around that time) found no issue on procurement processes; and (iii) the project was compliant with the procurement policy" (ICR, p. 31). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) The ICR (p. 36) offered several lessons and recommendations, including the following lessons restated by IEG:  Flexibility in project design and implementation arrangements can enable response during crisis. The Bank’s ability and decision to add an Ebola-related component using the additional financing helped the government to manage the risks of the Ebola crisis in the education sector.  Digital technologies can be effective and important tools for continued education during a crisis. The project provided an emergency response to the Ebla crisis through television and radio programming. The experience from project implementation was helpful in enabling the government's response during the current COVID-19 crisis.  Successful operation in low capacity and fragile contexts requires participatory approaches and field support. The project was helmed by the ministry (MEST) but benefited from the presence of operational staff in the field by receiving timely implementation support. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was written well and described the project well. The ICR constructed a theory of change, which was missing in the PAD. Yet, the ICR did not always establish tight linkages between the project's components and intended outcomes and did not account for some external factors/assumptions. The ICR provided good evidence and data points in a comprehensive results framework. The document candidly discussed implementation issues and drew effectively from the PAD and other project documents. There were minor shortcomings. The ICR was unnecessarily repetitive in some places. The ICR also did not provide adequate explanations for some ratings during project implementation. For example, the ICR stated that M&E was rated Moderately Satisfactory throughout most of the project but did not provide adequate explanation for some of the delays (ICR, p. 29). Similarly, the ICR stated that necessary action was taken to upgrade the procurement rating from Moderately Satisfactory to Satisfactory, but it did not provide any description of the necessary actions (ICR, p. 31). a. Quality of ICR Rating Substantial Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Revitalizing Ed Development (FY14) (P133070) Page 18 of 18