Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) Report Number: ICRR0023087 1. Project Data Project ID Project Name P146797 Kenya GPE Primary Education Development Country Practice Area(Lead) Kenya Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-18863,TF-B0830 31-Dec-2021 97,141,427.63 Bank Approval Date Closing Date (Actual) 18-May-2015 31-Dec-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 97,875,000.00 97,875,000.00 Revised Commitment 97,801,967.29 97,801,967.29 Actual 97,141,427.63 97,141,427.63 Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Grant Agreement (page 7) and the Project Appraisal Document (PAD, page 22), the project objectives were as follows:  To improve early grade mathematics competency and to strengthen management systems at school and national levels. Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components 1. Improving early grade mathematics competencies (Appraisal: US$ 34.5 million; Additional Financing (AF): US$ 4.0 million; Actual: US$ 40.2 million): This component aimed to improve teacher competencies through the scaling up of interventions piloted under the Primary Math and Reading Initiative supported by United States Agency for International Development (USAID) and United Kingdom Department for International Development. Activities included: in-service teacher training for Grades 1 and 2 teachers on pedagogical techniques to improve pupil mastery of basic numeracy skills; training of teacher advisor tutors; instructional materials; classroom supervision visits; and sensitization for teacher colleges on the early grade mathematics (EGM) model. 2. Strengthening school management and accountability (Appraisal: US$ 38.8 million; AF: US$ 4.0 million; Actual: US$ 43.0 million): This component aimed to strengthen management and accountability of Boards of Management at the school level, including in mobilizing and utilizing school resources more effectively. This component targeted 4,000 primary public schools that were considered low-performing (those with a Kenya Certificate of Primary Education (KCPE) score below the average of 243 for public primary schools in 2012 and 2013). Activities included: provision of detailed KCPE performance reports to schools to identify teacher weaknesses; implementation of a teacher performance and development framework; support to schools for preparing school improvement plans (SIPs); and school audits to improve transparency and accountability in the use of grant funds. 3. Strengthening capacity for evidence-based policy development at national level (Appraisal: US$ 10.8 million; AF: US$0.5 million; Actual: US$ 10.1 million): This component aimed to strengthen Ministry of Education (MOE) capacity for evidence-based sector policy development. Activities included: capacity building of the Education Management Information System (EMIS) unit to collect data; conducting of Standard 3 national assessments; and studies on potential models to serve disadvantaged groups and on the effectiveness of the use of capitation grants. 4. Project coordination, monitoring and evaluation (Appraisal: US$ 4.3 million; AF: US$ 1.0 million; Actual: US$ 3.8 million): This component aimed to support management and monitoring of all project activities. At the time of additional financing, the following activities were added/revised:  Component 1: Enhanced supervision/pedagogical support to teachers on EGM, centered around a School Based Teacher Support (SBTS) initiative implemented in a parallel Secondary Education Quality Improvement Project (P160083) for English, science, and mathematics teachers. Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797)  Component 2: Integration of effective school management practices and accountability mechanisms (drawn from the school improvement plan activities) into the nationwide student capitation grant program.  Component 3: Migration of the paper-based EMIS system to the online National Education Management Information System (NEMIS), publication of a statistical booklet on secondary education, and capacity building of the National Assessment Centre (NAC) for effective monitoring of learning achievement. Lastly, three disbursement-linked indicators (DLIs) were added, as required by the Global Partnership for Education (GPE), that focused specifically on quality of learning, equity of learning, and efficiency in learning. These DLIs were as follows: development of a competency-based assessment for early years education; adaptation of the competency-based assessment for learners with special needs; and share of public schools complying with selected key elements of the new school capitation grant guidelines (including availability of costed annual work plan, maintenance of updated cash books, evidence of appropriate store ledgers, adherence to procurement procedures, and school level public disclosure of relevant information.) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost  The project cost at appraisal was US$ 88.4 million. With additional financing, the project cost was revised to US$ 97.9 million, of which US$ 97.2 million disbursed. Financing  The project was originally financed by a GPE grant of US$ 88.4 million, which disbursed in its entirety. Additional financing through another GPE grant of US$ 9.475 million was approved, of which US$ 9.402 million disbursed.  Of the additional financing approved, US$3.0 million was allocated to DLIs. Borrower contribution  There was no planned Borrower contribution. Dates  November 2018: The project closing date was extended from March 2019 to March 2020, in order to allow preparation of the AF proposal and completion of activities under the parent project.  December 2019: The project was restructured to add/revise activities (see above) and to add US$9.48 million in AF. The results framework was also modified to add intermediate indicators for the new activities and increase the number of primary beneficiaries due to scaling up of activities. The project closing date was extended from March 2020 to December 2021. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) 3. Relevance of Objectives Rationale Kenya is a lower middle-income country and the ninth largest economy in Africa, with recent decades of growth fueled by strong consumption, investment, and broad-based growth. Human development indicators have also been steadily improving over the last couple of decades, including life expectancy at birth, infant mortality, and fertility rate. Poverty has also been declining (46% in 2005 to 34-42% in the last decade), but inequalities remain high. In the education sector, specific interventions and increased public spending have resulted in near universal primary enrollment and reduced gender gaps in school enrollment. Free primary education was introduced in 2003 and led to expanded access to education (net enrollment rate in primary school was 95.9% in 2013). However, enrollment and completion rates are lower in arid/semi-arid land (ASAL) areas and in urban slum areas. In addition, the rapid expansion of the education system has affected Kenya's capacity to maintain and improve the quality of education nationwide. This has been reflected in low competency levels in early grade mathematics and reading and declining national test scores (UWEZO survey findings throughout 2009, 2011, and 2012 indicate that fewer than a third of children enrolled in grade 3 have basic grade 2 level numeracy skills). These low levels of learning have been attributed to low teacher productivity (absenteeism, time on task), low teacher pedagogical competency, insufficient budget and instructional resources, and ineffective management and accountability at the school and system levels. Kenya's economic development plan - Kenya Vision 2030 - sets the goal to become a newly industrializing middle-income country by 2030, including through developing a globally competitive quality education system. The National Education Sector Plan for 2013-18 is the sector implementation strategy linked to the Vision 2030 plan, and it prioritizes raising literacy and numeracy levels, reducing disparities, and improving the quality of education with a focus on teacher quality, school leadership, and monitoring results. The project objectives are also well aligned with the GPE goals to ensure mastery of basic literary and numeracy, build national systems, and target marginalized groups (note: USAID was implementing a parallel project in the domain of basic literacy). The Bank's Country Partnership strategy for FY2014-18 (later extended to FY20), in place at the time of project appraisal, identified human resource development as a primary domain of engagement, although there were no specific outcomes for education in the results framework. The new Country Partnership Framework (CPF) for FY22-27 was being prepared at the time of ICR Review preparation and will include reduced disparities in learning outcomes as part of the second CPF outcome of "inclusion and equality institutionalized" (as reported by the project team). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) EFFICACY_TBL OBJECTIVE 1 Objective To improve early grade mathematics competency Rationale The theory of change for this objective was clear. As articulated in the ICR (page 7), the results chain drew on national and international evidence that more effective teaching was required for improving learning outcomes. Project outputs therefore were to center on increased technical training and supervision/pedagogical support for teachers in EGM and provision of EGM textbooks/teaching materials. At the time of project restructuring, the supervision and support for teachers had been identified as an area of weakness, and therefore the AF was to strengthen teacher support on EGM methodologies, including formative assessment, to increase the monitoring of quality learning. The project was to focus on rural areas, poor urban areas, and ASAL counties. The long-term goal of the theory of change was to improve the learning outcomes of children in Kenya with higher numbers of learners completing school. Outputs Nearly 7.5 million students benefited from direct project interventions (target: 7.0 million). Of these, 53% were female students (target: 40%). Teacher support:  Training of 102,157 teachers (grades 1, 2 and 3) on EGM (target: 40,000) and provision of EGM teaching guides and lesson plans.  Training of 2,864 teacher advisory tutors (later known as Curriculum Support Officers (CSOs) and Quality Assurance and Standards Officers) (target: 1,000). 81.4% of the teachers rated the training useful to a large or very large extent. These tutors, in turn, supported 52,235 teachers (for grades 1, 2 and 3) through in-school peer support that involved peer lesson observation, lesson study, and teacher action research, as well as the formation of communities of practice where they shared experiences, good practices, and working strategies, sought solutions to challenges, and received immediate support and guidance.  Conducting of 218,547 classroom observations (target: 100,000). Classroom observations were greatly facilitated by the use of the tablets issued to CSOs under a parallel USAID project for online reporting. This teacher support mechanism has been institutionalized as part of the Teacher Performance Appraisal and Development (TPAD) system. Teaching materials:  Provision of 10.5 million EGM textbooks (target: 6.0 million), which were adopted for the national roll- out of the Competency-based Curriculum. This resulted in a pupil:textbook ratio of 1:1 in mathematics for grades 1 and 2, in accordance with the textbook policy. The MOE was able to develop content in- house rather than outsourcing it to a publisher, which reduced the cost of textbook production. 91.0% of teachers indicated that provision of textbooks led to ease of teaching Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) mathematics concepts, and 88.4% of the teachers reported greater participation of pupils in their classrooms.  Adaptation of EGM textbooks, teachers’ guides, and a competency-based assessment for learners with hearing impairment, low vision, total blindness, or physical impairment. Student assessment:  Conducting of baseline, midline, and endline student assessments on math competencies.  Development of competency-based assessment for early years (to support teachers in monitoring quality of learning). In addition to the above outputs, the following DLIs were achieved:  Development of the Kenya Early Years Assessment Framework for Monitoring Learner Progress (KEYA-MPL) framework, and related tools and training manual, which was then approved by the MOE. At least 4,000 teachers in grades 1, 2 and 3 received training on the KEYA-MPL.  Development of the KEYA-MPL framework for learners with special needs and disabilities. At least 2,450 teachers in schools with learners with special needs and disabilities received training on the adapted KEYA-MPL. Outcomes  The math competency level of grade 2 students, as measured by an Early Grade Mathematics Assessment (EGMA)* in the subtraction domain, decreased from 79.1% in 2017 to 76.6% in 2019, and then increased to 81.5% in 2021. This fell short of the target of 84.1%. On this assessment, girls' score increased from 80.0% to 82.4%, while boys' scores increased from 78.3% to 81.4%. As noted in the ICR (page 18), the mid-line EGMA (2019) pointed to the need to enhance teachers’ pedagogical skills and thus led to a mid-course correction with the provision of the AF (and an increase in competency level from that point on). The post-endline EGMA**, which was conducted in 2021 post- COVID-19 pandemic, showed increased achievement from the 2016 baseline in all five domains of addition, subtraction, multiplication, division, and number patterns, suggesting a positive impact of the project interventions. The standardized mean EGMA score increased from 500 at baseline to 506.1 at post-endline, an improvement of 0.26 standard deviations. * The indicator, which was specific to the domain of subtraction fluency, was selected because subtraction fluency is moderately correlated with other mathematical subtasks, implying that pupils who can solve subtraction sums at levels 1 and 2 could also perform the other subtasks at almost the same level. The performance index was based on the subtraction scores, with the benchmark score being 50 percent correct. The indicator was measured by a sample-based national assessment for grade 2 mathematics. ** The post-endline EGMA assessment was a national survey, conducted across 47 counties in Kenya, with a sample size of 300 randomly selected public and private schools previously used in the baseline, midline and endline studies of the project. The study was carried out among grade 3 learners at the beginning of the Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) second term of 2021 because it was expected that the learners would have completed the entire grade 2 content, and any learning loss mitigated as a result of the COVID-19 school closures. Achievement is rated Modest due to a significant shortfall in achieving the target for improved mathematics competency, although improvements following the introduction of focused interventions by the AF are noted. Rating Modest OBJECTIVE 2 Objective To strengthen management systems at the school level Rationale The theory of change for this objective was clear. As articulated in the ICR (page 7-8), the results chain was based on the premise that school improvement planning and better resourcing of schools utilized in an accountable manner would lead to schools having a more conducive environment for learning. The project outputs were therefore to center on provision of school grants to improve school resources and increased supervision of teachers to enhance accountability. The AF activity sought to promote efficiency through mainstreaming "best practice" processes into the governance of student capitation grants provided to all public schools. The activities were to be implemented through a combination of pilot interventions in 4,000 schools, which were identified by below-average marks on the KCPE (less than 243 out of 500). Outputs  Production of school-level KCPE reports for 31,200 schools (target: 4,000). This activity was eventually scaled up to all public and private primary schools in the country, due to interest from other education stakeholders receiving these reports, and recognition by the MOE of the benefit of providing the KCPE reports to all teachers. The KCPE reports were made available in 2015, 2017, 2018, and 2019. In addition, 188 staff were trained at the county level on utilization of the reports.  Appraisal of 32,685 teachers using the TPAD system, in particular in the area of teacher contact hours (target: 30,000 teachers). The MOE extended the TPAD program to all primary schools nationwide such that 225,263 additional teachers nationwide were appraised. The project supported the transition of the TPAD from a paper-based to an on-line system, which resulted in an increase in the percentage of teachers who completed the appraisal. The TPAD has been fully institutionalized within the MOE.  Conducting of annual school audits in the 4,000 participating schools (target: 4,000). According to the project team, the audits assessed school compliance with the established criteria of: formation of Board of Management, development of school improvement plan and budget, updated textbook Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) ledger, record-keeping for procurement, payment vouchers, etc). Schools that lagged in compliance were provided with capacity building support.  Capacity building in the preparation of school improvement plans, with all 4,000 participating schools submitting satisfactory plans and receiving annual school grants (target: 4,000).  Training of 46,366 members of school Boards of Management on utilizing student capitation grants in primary schools (target: 10,000). Of these, 31,993 members were male and 14,373 members were female. In addition, the following disbursement-linked indicator (DLI) was achieved:  32.4% of all public schools were in overall compliance with the requirements of the new school capitation grant guidelines and operational manual. This achieved the target of 31% and included all 4,000 of the project schools. Outcomes  4,000 schools completed the top two priorities in their SIPs. This achieved the target of 4,000. 88.3% of school heads indicated that the SIP process had greatly improved the management of their schools. "Best practices" identified included: directing of government funding to prioritized needs, collective training on the SIP process to foster ownership, enhanced capacity of all school stakeholders in implementing SIP activities, comprehensive annual school audits to improve accountability and transparency, use of a consultative process to increase ownership and effective prioritization, and monitoring of the grants to enhance accountability and improve efficiency of service delivery. Achievement is rated Substantial due to evidence of strengthened capacity and increased use of data for evidence-based decision-making at the school level. Rating Substantial OBJECTIVE 3 Objective To strengthen management systems at the national level Rationale The theory of change for this objective was clear. As articulated in the ICR (page 7-8), the results chain was based on the premise that increased availability of data would lead to sector management and schools Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) making more evidence-based decisions. The project outputs therefore centered on collection and dissemination of education statistics and student level performance data. Outputs  Development of the EMIS, including with EGM data. Learners were registered with unique personal identifiers, linked to their birth certificates. The percentage of primary schools submitting EMIS data increased from 60% in 2015 to 90% in 2021 (target: 98%).  Technical assistance to MOE for conducting national learning assessments, including training of 64 technical staff at the NAC.  Preparation of an Education Sector Analysis to review issues related to teachers, infrastructure, community, students, school leadership, school reforms, political dynamics, and language policy in schools. This analysis contributed to the development of the 2018-2022 National Education Sector Strategy Plan, also supported by the project. Outcomes  Standard 3 national assessments were conducted in 2016 and 2018, with findings disseminated. In addition to identifying learning gaps, the assessments informed policy development and curriculum modifications. According to the ICR (page 24), the MOE has institutionalized a culture of assessment data utilization through the robust dissemination of data down to the school level.  EMIS data was published in annual educational statistical booklets from 2017-2021. The MOE’s use of the system data includes student placement, disbursement of capitation for primary and secondary school grants, and provision of medical coverage to secondary school students. Achievement is rated Substantial due to evidence of strengthened capacity and increased use of data for evidence-based decision-making at the sector-wide level. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall efficacy is rated Substantial due to Modest achievement of the first objective to improve early grade mathematics competencies, Substantial achievement of the second objective to strengthen management Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) systems at the school level, and Substantial achievement of the third objective to strengthen management systems at the national level. Overall Efficacy Rating Substantial 5. Efficiency At appraisal (PAD, Annex 6), the economic analysis centered on Components 1 (improving EGM) and 2 (strengthening school management). The Component 1 analysis was a cost-effectiveness analysis comparing the EGMA intervention proposed by the project (specific instructional model with materials, teacher training, and in-classroom teacher support) with alternative interventions, in terms of cost per increase of one additional standard deviation test score. The estimated cost for the EGMA activities was $2.31 per pupil. Other interventions (in Kenya) were estimated at the following cost per pupil (note: these test scores increases were overall, not specifically for math): teacher incentives in year 2 ($15.90); girls' scholarships ($72.26); streaming by achievement ($2.87); and extra contract teacher + streaming ($50.74). For Component 2, a cost benefit analysis was presented, using benefits from wage premiums for increased years of education and costs derived from project costs, public costs of education, and associated private education costs. The net present value (NPV) was estimated at US$ 164.1 million, and the internal rate of return (IRR) was 32%. At completion (ICR, Annex 4), the cost-effectiveness analysis is revisited for both Components 1 and 2, which comprise 76% of project costs. Benefits are again derived from the additional years of education yielding higher income (the ICR noted that beyond wage premiums, other benefits from rising female education levels include not only higher skills, employment opportunities, and income, but also reduction in early marriage and teenage pregnancy; higher education levels for the general population also reduce crime). Component 1 is estimated to have benefited 7.5 million primary students, and therefore project benefits are estimated as the number of children with improved test scores, multiplied by the lifetime earnings gained from this improvement in test scores. For component 2, strengthening management and accountability in 4,000 schools impacted an estimated 1.7 million students, and therefore the project benefits are estimated as the number of children completing primary school who would not have done so without the project, multiplied by the average wage difference between workers with incomplete and complete primary schooling. The NPV of both components was estimated at US$ 342.0 million, with an IRR of 23%. The NPV for Component 2 only was estimated at US$ 36.1 million, with an IRR of 16.8%, which is lower than at appraisal. There were indications of efficient use of project resources. For textbook procurement, MOE moved from a school-based system to a centralized procurement and distribution process, contributing to an improved turnaround time from procurement to delivery of materials. Also, the content was developed in-house by the MOE rather than outsourced to a publisher, further reducing costs and contributing to the surpassing of the textbook indicator. Additionally, the MOE modified the teacher training venues from costly hotels (that were far from the teacher’s duty stations) to local school facilities close to teachers, which made it easier for teachers to participate and led to cost savings that allowed for an additional 62,157 teachers to benefit from training. COVID-19 slowed down implementation of the school-based teacher support activities, which was a school-based initiative, because of school closures; however, once schools reopened in January 2021, the MOE was able to roll out and complete the activity as planned. Efficiencies were also gained from the use of Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) technology (tablets provided to schools by a parallel project were used to conduct classroom observations, and the teacher appraisal system migrated from paper-based to on-line). All project funds disbursed for the original project essentially during the original project period, with AF disbursed during the project extension period. Despite the COVID-19 closures, all project activities were completed successfully, and targets were met. However, there were some moderate inefficiencies noted, including delays in the baseline study and procurement of the EGM textbooks, although these delays were resolved and activities were completed. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 44.00 Appraisal  32.00  Not Applicable 76.00 ICR Estimate  16.80  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance is rated High due to strong alignment with country condition and with country and Bank strategies. Overall efficacy is rated Substantial due to Modest achievement of the first objective to improve early grade mathematics competencies, Substantial achievement of the second objective to strengthen management systems at the school level, and Substantial achievement of the objective to strengthen management systems at the national level. Efficiency is rated Substantial due to evidence of implementation efficiencies, although the cost-effectiveness analysis of part of the project indicated less favorable NPV/IRR at completion compared to original estimates. Overall outcome is rated Moderately Satisfactory due to the shortcoming in achieving the project objective to improve early grade mathematics competencies, as evidenced by significant shortfalls in achieving the key project indicator target to increase math scores of Grade 2 students (as measured by an Early Grade Mathematics Assessment in the subtraction domain). a. Outcome Rating Moderately Satisfactory Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) 7. Risk to Development Outcome Project outcomes are likely to be sustained due to mainstreaming of numerous project activities and structures into the government's education management system. The teacher performance appraisal and development system (including classroom observations), guidelines for the use of school capitation grants, the school-based teacher support model, and the peer support clusters were all institutionalized in the MOE. Also, as the MOE had primary responsibility for project implementation and management, capacity was strengthened to continue activities. Key project activities will be continued and scaled up in the follow-up Primary Education Equity In Learning Project (P176867). Although there are some macroeconomic uncertainties related to the COVID-19 pandemic, overall medium-term economic prospects appear favorable. 8. Assessment of Bank Performance a. Quality-at-Entry The project objectives were closely aligned with country and Bank strategies and GPE goals, and drew upon well-established understanding of increasing effectiveness of teaching and learning. The EGMA activities built upon lessons learned from implementation of a USAID early reading project, while the school/sector management activities were designed as a pilot approach with integrated interventions. In a similar manner, lessons learned from the implementation of the parent project (and a parallel secondary education project) significantly influenced the design of the AF activities (including the use of results- based financing) to leverage improved management of the school capitation grants, scale-up of SIP activities nationwide, adoption of the SBTS initiative, and adapting of teaching practices for learners with special needs. The results chain and M&E arrangements were clearly identified. The overall project risk was rated Substantial due to capacity constraints at the school level (including Boards of Management) and capacity constraints in learning assessment and fiduciary management. Mitigation measures were provided for on-going training and technical assistance throughout implementation. Quality-at-Entry Rating Satisfactory b. Quality of supervision Supervision missions were regular and appropriately staffed with fiduciary and technical specialists. The Bank team used the opportunity provided by the AF to apply lessons from ongoing implementation experience as well as other relevant operations in the education sector, leading to the addition/revision of activities to further the gains in basic mathematics competency and broaden the project's impact on improving school management and accountability. The results framework was revised accordingly to account for the new activities, while maintaining focus on the project's objectives. There were some initial implementation delays—contracting a firm to conduct the baseline EGMA, NEMIS development, school- Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) based procurement of teaching and learning materials—but these were resolved sufficiently to ensure timely completion of activities. Institutional support was strongly focused on developing in-country capacity to sustain activities and outcomes. As reported in the ICR (page 31), the MOE, with Bank guidance, took actions to build the capacity of senior management regarding gender and sexual harassment issues. The Bank and government also began an important dialogue on gender and sexual harassment in the workplace. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The results chain and indicators to measure outcomes were specific and clearly articulated. The first objective was to be measured by improvement in mathematics competency level of grade 2 pupils (disaggregated by gender) in the subtraction domain. The second objective was to be measured by not only the preparation of SIPs but also capacity to execute each individual plans' top two priorities. Monitoring arrangements were developed as part of project activities (EMIS and TPAD), and evaluative activities were also included to assess project impact (student learning assessments, and evaluations of TPAD, SIP and EGMA). b. M&E Implementation The implementation of the EMIS, and subsequently the NEMIS, and the development of the TPAD were critical activities to monitor project progress and outcomes. Monitoring data was collected on registered learners with unique personal identifiers, placements, capitation grant information, school visits, and teacher observations. The school and county level implementing agencies provided reports on a regular basis to contribute to national level reporting on project progress. The project also supported student learning assessments and policy studies related to teacher deployment and utilization, cost effective models for enhancing participation in ASALs and informal settlements including for children with special needs, and equity-based financing for primary and secondary education in Kenya. c. M&E Utilization M&E data was used to monitor project progress, provide school level data, determine distribution of school capitation grants, secondary school student insurance, teacher training, and teacher promotions. The ICR (page 33) also reported that results of the learning assessments have been used as follows: (i) EGMA results have led to increased teacher training efforts, particularly school-level Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) support and peer-to-peer support activities; (ii) the SIP audit showed that SIP design was effective, such that SIPs have been institutionalized in the Government Policy Guidelines on Utilization of Learner Capitation Grants and Other School Funds and used nationwide; and (iii) the SIP audit also led to improvements in the training of school Board of Management members, and informed development of the next education sector plan. M&E Quality Rating High 10. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project with the safeguard policy on Environmental Assessment (OP/BP 4.01) triggered. An Environmental and Social Management Framework was prepared to provide guidance on and mitigation measures for potential adverse impacts from construction and/or rehabilitation of sanitation facilities, as well as disposal of e-waste (i.e. tablets). The safeguard policy on Indigenous Peoples (OP/BP 4.10) was also triggered due to the focus on vulnerable and marginalized groups. The government prepared a Vulnerable and Marginalized Groups Framework to ensure inclusion of and support from those communities. According to the ICR (page 33), safeguards were monitored consistently during project implementation, and Implementation Status Reports (ISRs) consistently rated safeguards compliance in the satisfactory range. The MOE ensured that there was an environmental and social management plan checklist included in the SIP manuals related to small SIP repairs, and Boards of Management were sensitized through training on indigenous peoples issues and inclusion. There were no safeguard problems reported. b. Fiduciary Compliance Financial management: The Bank’s financial management specialists provided capacity development workshops for project coordination unit (PCU) staff on record keeping and reporting. The financial management performance rating was in the satisfactory range throughout the project period. As reported in the ICR (page 34), the PCU submitted unaudited interim financial reports of acceptable quality, albeit with slight delays on occasion. The required audits were submitted to the Bank with unqualified opinions. Procurement: The Bank’s procurement specialists conducted regular procurement assessment missions and training to the PCU and counties to ensure compliance. The procurement performance rating was in the satisfactory range throughout the project period. As reported in the ICR (page 31), in 2017, there were three procurement complaints related to gender sensitive allegations. The procurement cases were addressed by the PCU/MOE in accordance with Bank procedures. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797) c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Overall outcome is rated Moderately Satisfactory due to the shortcoming in achieving the project objective to improve early grade mathematics Outcome Satisfactory Moderately Satisfactory competencies, as evidenced by significant shortfalls in achieving the key project indicator target to increase math scores of Grade 2 students. Bank Performance Satisfactory Satisfactory Quality of M&E Substantial High Quality of ICR --- Substantial 12. Lessons Lessons drawn from the ICR (pages 35-36), adapted by IEG:  Project designs that include pilots as demonstration models can be an immediate catalyst for system-level change, if lessons are widely shared in a timely manner. In the case of this project, the school management activities were initially implemented in 4,000 schools, but the timely feedback (due to data generated by the monitoring system, the learning assessments, and the evaluations) led to immediate scaling up of activities, by way of both the government budget and GPE Additional Financing.  Improving learning outcomes can be an achievable project objective if accompanied by a strong M&E framework and monitoring arrangements. In the case of this project, the project was able to establish a robust M&E system capable of systematically tracking progress, and project activities included relevant assessments and studies that influenced decision making and policy development. For example, the regular assessment data collection allowed for an evidence-based correction at the time of project restructuring, when assessments indicated that learning levels had decreased. Remedial action was taken, by introducing more targeted teacher pedagogical skills training and peer-to-peer learning to complement ongoing activities. Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya GPE Primary Education Development (P146797)  School-based teacher support systems are a critical ingredient for ensuring on-going quality improvements in schools. In the case of this project, ongoing support in content and pedagogical practices was provided by teacher advisory tutors/curriculum support officers. Teachers also received timely feedback on teaching approaches. Peer-to-peer collaboration also enabled teachers to work together, share experiences, and find solutions to challenges. Equally important was the feedback loop developed with the National Assessment Council to ensure teachers were familiar with learning gaps, particularly after the COVID-19 school closures. All of these were important for helping to motivate teachers and necessary for improving the quality of teaching. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is notable for its clear and strong articulation of the theory of change. Along with the strong quality of evidence, the clear results chain allowed for a coherent analysis of project achievements. The narrative of project implementation experience reflected good practice in using evidence (from monitoring data) to assess project progress and make mid-course corrections, in order to achieve outcomes. Lessons were relevant and clearly derived from project experience. a. Quality of ICR Rating Substantial Page 16 of 16