Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) Report Number: ICRR0023456 1. Project Data Project ID Project Name P154524 Third Punjab Education Sector Project Country Practice Area(Lead) Pakistan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-86200 31-Dec-2021 292,310,000.00 Bank Approval Date Closing Date (Actual) 03-Jun-2016 30-Jun-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 300,000,000.00 0.00 Revised Commitment 300,000,000.00 0.00 Actual 292,310,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Livia Maria Angelica Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Benavides Matarazzo Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO) was to support Punjab province to improve school participation, completion, and teaching-learning practices, with a particular focus on low-performing districts. The PDO was the same in both the Project Appraisal Document (PAD) and Loan Agreement. The ICR identifies three objectives as part of the PDO: a. To improve school participation with a particular focus on low performing districts. Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) b. To improve completion rates with a particular focus on low performing districts. c. To improve teaching and learning practices with a particular focus on low performing districts. Although some funds were reallocated and indicators adjusted at three project restructurings, the scope of the project was not changed, and therefore a split rating methodology is not required. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1: Improved access, quality, and education system management (appraisal: US$ 289.89 million; restructuring with reallocation: US$ 277.89 million; actual: US$270.31 million) This component was to be implemented using results-based financing in support of the Government of Punjab’s 2018 Education Goals and Growth Strategy. Disbursements would be on a reimbursement basis and linked to nine agreed-upon disbursement linked indicators (DLIs). Given the focus on low-performing districts (LPDs), this component targeted children and schools in those jurisdictions, when relevant. The component was divided into three strategic areas and nine subcomponents, as follows: Strategic Area 1: Improving access to schooling for the poorest. The goal was to provide students from the poorest households with financial resources to attend schools at the primary and secondary levels. Under this strategic area, the project would support: (a) the provision of education vouchers for children aged 5-16 to enroll in low-cost private schools. The targeting mechanism was to be improved to ensure that children in LPDs would be prioritized; (b) strengthening the public-private partnership (PPP) program run by the Punjab Education Foundation (PEF) to increase enrollment and quality of schools, using the same targeting mechanism for LPDs as in (a); and (c) the provision of stipends to secondary school girls to increase their transition from the primary level and their retention through to graduation. The project was to continue supporting the girls' stipend program, as in previous Bank-financed projects, in sixteen pre-defined districts, nine of which coincided with the LPDs. Strategic Area 2: Ensuring quality teaching and learning for all. The goal of this strategic area was to raise the quality of teaching and learning through: (a) the strengthening of early childhood education (ECE) in order to improve the cognitive, socioemotional, and physical development of children and facilitate their transition into primary school; (b) ensuring the merit-based recruitment of teachers to vacant posts; (c) improving teaching quality and learning in primary schools through teacher professional development, with a particular focus on multi-grade classrooms and classrooms with a large number of children, and through the development of a tool on a pilot basis (covering 100 schools) to provide structured learning activities for grades 1-3; and (d) improving the education assessment system to enhance quality of education through the development of an assessment policy framework, enhancing the quality of items and tests, and improving analysis and usage of assessment data. Strategic Area 3: Improving leadership, management, and accountability. Under this strategic area, the project would strengthen: (a) the system of allocation and accountability in the use of non-salary budgets Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) (NSBs), including a greater focus on rural schools, many of which are located in LPDs; and (b) the integration and usage of education data to improve evidence-based decision making and overall management of the education system. Component 2: Capacity building, project management, monitoring, and evaluation (appraisal: US$ 9.36 million; restructuring with reallocation: US$21.36 million; actual: US$19.03 million) This component was to support the implementation of Component 1 through the provision of critical and strategic technical assistance (TA) and of capacity building activities. This component would also support effective and efficient project management, including monitoring and evaluation activities. The component would be implemented under the traditional modality of fund advancements and submission of statements of expenditures. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was approved on June 3, 2016, became effective on October 5, 2016, had the mid-term review on April 23, 2019, and closed on June 30, 2022, six months after the original closing date of December 31, 2021. The World Bank loan of US$300 million was embedded within the overall provincial government education budget. As described above, about 97 percent of the loan was designed to be disbursed as results-based financing, with reimbursements upon compliance with agreed DLIs and against eligible expenditures. The overall yearly provincial education budget was about US$13 billion, so that even if the project were to be disbursed in one year, it would be no more than 2.3 percent of the total budget. In addition, the Government of the United Kingdom, through the Department for International Development (later renamed as Foreign, Commonwealth and Development Office, FCDO) provided parallel financing of approximately US$150 million (GBP100 million). The project was restructured three times in addition to a closing date extension of six months. The first restructuring after the mid-term review (June 2019, US$115.58 million disbursed) addressed a change in the professional development model in the province, including the replacement of the Directorate of Staff Development with the new Quaid-e-Azam Academy for Educational Development (QAED), as well as the creation of new roles and functions for staff in the organization. As a result of this institutional reorganization, DLI 6 (addressing continuous professional development for teachers) was not met for more than two years after the launch of the project. The DLI was rephrased to better focus on the new professional development model designed around promoting behavioral change to improve the teaching process. In addition, the results framework was updated, including three PDO indicators: (a) quality score of primary teaching-learning practices was modified in line with the changes made to DLI 6; (b) the school participation indicator did not change, but the methodology did; and (c) the school readiness indicator, which was missing at appraisal, was defined, and baseline and end-of-project targets were included. In addition, two intermediate outcome indicators were revised. The second restructuring (June 2020, US$186.41 million disbursed) focused on assisting the Province of Punjab to address challenges posed to the education system due to the COVID-19 pandemic, while also adjusting some DLIs (now renamed performance-based conditions, PBCs). Funds were transferred from Component 1 to Component 2 to finance activities aimed at compensating for school closures, such as dissemination of information on the disease, and providing distance education and mobilization campaigns Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) and support to schools once they reopened. This reallocation also was to allow for improved data management to ensure data integration. In addition, two PBCs were revised. PBC 3 on girls’ stipends was revised to improve the registration and disbursement of stipends. PBC 7 on student assessment was restructured to be aligned with the then-new assessment policy framework developed in the province. The intermediate results indicators were revised to consider the impact of the pandemic on school enrollment, particularly in private schools, some of which were expected to permanently close. The third restructuring (June 2021, US$224.59 million disbursed) modified the scope of four PBCs to respond to challenges posed by COVID-19 and contribute to effective implementation for the remaining life of the project. The restriction, due to fiscal constraints, on hiring of new teachers meant that the remaining US$6.44 million remaining under PBC 5 would not be disbursed. Out of these funds, US$3 million were reallocated to PBC 4 on ECE to provide a new remote platform to train monitoring officers, teachers, and caregivers on early learning. The remaining funds, US$3.44 million, were reallocated to PBC 8 on NSBs to purchase cleaning supplies and personal protection equipment, among other items. PBC 7 on student assessment was also revised (with no change in funding) to reflect delays due to school closures because of the pandemic. Finally, the project was extended in December 2021 by six months to June 30, 2022. The purpose of this extension was to allow for the completion of contracts under Component 2 of the project. 3. Relevance of Objectives Rationale The project objectives were highly relevant to the sectoral context in the Province of Punjab. Even though Punjab was the most populous and richest province in Pakistan, its capacity to deliver services faced a number of challenges. At the time of preparation, 21.9 percent of children were out of school, and in the case of LPDs, the number was 31 percent. Outcomes for school completion, dropouts, and learning outcomes were also poor, and enrollment had plateaued. Factors contributing to this weak performance included demand-side constraints, limited availability of quality ECE and of public classrooms and teachers, and a poor learning environment due to low teacher capacity. The project was well aligned with the FY2015-19 Country Partnership Strategy (CPS), the most recent for Pakistan, which recognized challenges in the education sector at both the national and provincial levels. The CPS aimed to support an increase of gross primary enrollment rates by at least 10 percent as well as the adoption of education quality assessments and improved learning outcomes, recognizing that a special effort would need to be made with respect to girls, given their vulnerability and the fact that they were lagging behind boys in education attainment. A Country Partnership Framework is currently under preparation, but it is important to note that the 2020 Systematic Country Diagnostic identifies education as one of the critical sectors, where girls particularly are underserved. The project was also aligned with the Punjab Education Sector Plan (2013-2017) and the subsequent “The New Deal” (the sector plan for 2018-2023, under the then-new government). Both these plans focused on improving access, retention, and quality of education in Punjab. Project indicators were also aligned with these plans, including the strengthening of PPPs as a strategy to improve access, improving teaching practices and student assessments to strengthen education quality, and establishing a management Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) information system to strengthen education management. Punjab’s education reform program was also supported by donors, notably the FCDO. The World Bank has had a long-standing engagement with the Province of Punjab in the education sector since 2003. Prior to this project, the World Bank supported two investment project financings (IPFs), the Punjab Education Sector Reform Program (FY2003-12), and the Punjab Education Sector Project II (FY2012-2017), as well as four development policy financings (DPFs) between 2004 and 2007 (see Implementation Completion and Results Report [Report No. ICR00003955] for a Second Punjab Education Sector Project, PESP II). The PESP II provided support in similar areas to this project, particularly focusing on improving the quality of teaching, and providing demand-side financing to increase access to education, including stipends for girls, student testing, supporting school councils, and strengthening budget management. PESP II was also a results-based operation using DLIs with a TA component. This project, PESP III, built on the lessons and experience gained with the prior two projects, continuing to support the Government of Punjab’s reform agenda and including new areas such as the expansion of ECE and PPPs. There was a short overlap between the two projects, ensuring continuity in the engagement with the School Education Department of the Government of Punjab. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve participation rates with a particular focus on low performing districts. Rationale This objective would increase school participation through the enrollment of children ages 6–15 years old in primary, middle, and secondary schools, as well as children ages 3-5 in ECE. Given the limited availability of adequate classrooms and teachers in the public sector for primary through secondary levels, the PEF would expand its PPP program and provide education vouchers to out-of-school children to allow them to enroll in low-cost private schools. In addition, girls ages 11-15 would receive a stipend to ensure that they would stay in school. At the time of project launching, only 30 percent of children attended pre-school programs, but these were mostly informal spaces focusing on child care rather than on readiness for primary education. Only 1,125 primary schools had an ECE classroom with a trained teacher and appropriate teaching-learning materials. Under the project, enrollment would be increased through the opening of new classrooms meeting agreed- upon quality teaching-learning standards. Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) Outputs and Intermediate results Overall, the number of additional children ages 6-16 who were provided with vouchers was exceeded as compared to the revised targets (as per the June 2020 restructuring which responded to the COVID-19 pandemic). A total of 211,139 children received vouchers, as compared to a revised target of 150,000 children. Of these, 114,015 were boys (target was 75,000) and 97,124 were girls (target was also 75,000). The number of new PPP initiatives supported by PEF under a quality assurance system was 1,380, above the target of 1,125. The number of additional children enrolled in PPP initiatives supported by PEF under a quality assurance system was 923,836 (target was 900,000 and was not revised due to the COVID- 19 pandemic), of whom 498,871 were boys (target 450,000) and 424,965 were girls (target was 450,000), thus overachieving the target for boys but partially underachieving the target for girls. An external evaluation showed that PEF schools tended to cater to poorer students and enrolled more girls than public schools (ICR, p. 13). There was no formal output indicator linked to the girls’ stipend, as this was a long-standing program and the project aimed to ensure continuity of this program in the sixteen selected districts in Punjab. With respect to ECE, the project aimed to enroll 210,000 children (50 percent of whom would be girls) in newly opened schools that met specific quality standards. At the endline of the project a total of 126,789 children had been enrolled in these schools, of whom 65,017 (51 percent) were girls. This underachievement of the target is attributed to the COVID-19 pandemic. School closures had a serious impact across Pakistan, where overall pre-school attendance fell from 33 percent in 2018 to 23.7 percent in 2020. On the other hand, the project aimed to support 7,000 ECE classrooms, and the endline number was 7,067 classrooms that met the quality standards. Teachers were trained remotely, using an EdTech platform, and were also provided with in-person support. Of the 7,067 classrooms, 2,532 were in LPDs. Outcomes The overall expected outcome for this objective was to increase the participation rate of children 6-15 years old from a baseline of 75.8 percent to a target of 79 percent, and in the LPDs from 68.5 percent to 73 percent. Both outcomes were achieved, as they reached 79.6 and 73.01 percent, respectively. When broken down by age and gender, the results for girls show that targets were exceeded, while for boys targets were not met:  Participation rate boys (6-10 years): baseline was 85.4, the target 87, and the latest value was 85.45 percent  Participation rate girls (6-10 years): baseline was 77.8, the target was 80, and the latest value was 81.97 percent  Participation rate boys (11-15 years): baseline was 75.4, the target was 79, and the latest value was 76.62 percent  Participation rate girls (11-15 years): baseline was 64.6, the target was 68, and the latest value was 72.33 percent The increase in participation rate for girls aged 11-15 is particularly noteworthy. This increase is partly attributed to the long-running stipends program, which is implemented in 16 districts, 9 of which coincide with the LPDs. During project implementation the stipend amount was tripled to increase the incentive for girls to enroll and stay in school. The payment system was also improved, using digital mobile phone payments. Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) School participation in stipend districts increased by 6.1 percentage points during project implementation, while it decreased by 0.02 percentage points in non-stipend districts. At the pre-school level, the outcome was measured through the school readiness indicator, which measures pre-literacy, pre-numeracy, fine motor skills, and the child’s level of language and socio-emotional development. At the start of the project, school readiness was two percent, while at the end of the project readiness was 8.35 percent, exceeding the expected target of 6 percent. This is attributed to the implementation of a new ECE policy that included a play-based, two-year curriculum and well-defined quality standards and a corresponding quality assurance mechanism. Rating Substantial OBJECTIVE 2 Objective To improve completion rates with a particular focus on low performing districts. Rationale The theory of change for this objective held that expanding girls' stipends at the secondary level would encourage girls to complete primary school, and that increases in the NSB linked to completion rates would provide incentives for school directors to focus on completion. The logic underpinning the demand side was further linked to assumptions that improvements in the quality of primary education would encourage parents to ensure that their children attended and completed primary levels, and that effective communications campaigns could inform parents and communities about the availability and importance of quality primary education. These inputs would be expected also to increase student motivation and retention. The objective focused on completion rates for primary schools, i.e. completion of grade 5. Primary completion in Punjab was low, with only 71 percent of children enrolling in grade 1 in government schools surviving to complete grade 5. While the project contributed both to government schools as well as to private and PPP schools, completion rates were only monitored in public government schools. Outputs and intermediate results The main outputs for this objective were intertwined with those from both objectives 1 and 3, with inputs related to these objectives contributing to completion rates. In particular, the girls’ stipend program (see objective 1) provided an incentive to ensure completion of primary so that they could progress to secondary, in order to earn said stipend. Also, the NSB (see a more detailed discussion under objective 3) provided a small discretionary budget at the school level allowing school directors to purchase inputs such as furniture, cleaning materials, and learning items. Notably, the allocation formula placed a premium on retention rates. In addition, an SMS (text messaging) campaign had a positive impact on enrollment and retention. Finally, quality inputs are likely to have had an impact on completion rates. These included hiring of better qualified teachers, the redeployment of teachers to classrooms with high teacher-to-student ratios (greater than 40 Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) students per teacher), and the implementation of a new teacher development program focused on improving the teaching-learning process. Outcomes At the provincial level, completion rates for grade 5 were exceeded as a whole, and also separately for boys and girls, as follows:  Completion rate, grade 5: baseline was 67, the target 70, and the latest value was 78 percent  Completion rate, grade 5 boys: baseline was 69, the target was 71, and the latest value was 76 percent  Completion rate, grade 5 girls: baseline was 66, the target was 69, and the latest value was 81 percent In the case of LPDs, targets were not determined at appraisal, but estimates were available through the Annual School Census. Results were as follows:  Completion rate, grade 5: baseline was 49 percent and the latest value was 69 percent  Completion rate, grade 5 boys: baseline was 52 percent and the latest value was 68 percent  Completion rate, grade 5 girls: baseline was 46 percent and the latest value was 71 percent The results for girls, both at the provincial level and in the LPDs, are exceptional, particularly in the context of COVID-19. This high result is attributed to several factors. Schools in Pakistan are commonly separated by gender, and girls’ schools appear to function better than boys’ schools. Also, girls responded better to SMS messages promoting learning at home during the COVID-19 lockdown and, later, to the call to return to school, once schools reopened. Finally, the stipend for girls at the secondary level provided incentives for primary school completion and subsequent progression to secondary school. As explained above, the project supported public schools, as well as PPP and private schools. The ICR and further conversations with the Task Team explained that there was a private school census in 2016, but a follow-up census could not be carried out. Data were therefore not available to determine completion rates for PEF-supported schools. Rating Substantial OBJECTIVE 3 Objective To improve teaching-learning practices with a particular focus on low performing districts. Rationale Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) This objective was to focus on improving quality of teaching and learning at the ECE and primary level across the province of Punjab. At the ECE level, the objective would be achieved through the implementation of an ECE policy that would adopt a play-based, two-year curriculum that would meet international standards and would have a quality assurance mechanism. At the primary level, the project would support a renewed teacher professional development program, hiring of new teachers, redeployment of teachers to reduce overcrowded classrooms, improvements in the student assessment process for decision-making, and allocation of NSB to schools in order to finance inputs to improve the quality of learning in the classroom. School councils were also to be supported in order to ensure community participation to improve children’s learning process and have an accountability mechanism in the use of NSB. Outputs and intermediate results The project supported the opening of 7,067 new ECE classrooms under the new policy’s quality standards that included teacher training, ECE kits, age-appropriate furniture, and wall posters. An independent assessment of ECE classrooms found that teachers were provided with support and inputs in line with the provincial policy on ECE. As of the first restructuring, the School Education Department (SED) reformed the teacher development program and implemented a new Continuous Professional Development program that included classroom observation and mentoring by trained specialists, a platform for digital learning, and face-to-face facilitated discussions with mentors. While this program was initially meant to be implemented in 18 districts, it was scaled up to all of the 36 districts in the province. The endline survey for the project showed significant improvement on indicators such as positive classroom behavior and student understanding of content. Fiscal constraints linked to the COVID-19 pandemic forced the government to restrict the hiring of new teachers, starting in year four of the project. Still, during the first three years over 96,000 new teachers were hired under a merit-based system. A survey showed that these teachers had higher qualifications and had better content knowledge than teachers hired under the traditional system. The project supported the development of an assessment framework that aimed at standardizing data, thus making it comparable across schools. The Punjab Examination Commission (PEC), responsible for student assessments, was strengthened and qualified staff was hired. By end of the project, the PEC had carried out two rounds of the new assessments (both school-based and large-scale) and a study on learning losses. The project also supported development of the Integrated Education Dashboard that monitored 27 indicators, allowing for informed decision making at the provincial and district levels. Non-teacher inputs were financed through the NSB that was transferred to each school. The project contributed to the simplification of the transfer process to ensure more expedient availability of these funds. Still, there were challenges in its implementation and, at end of the project, only 56 percent of the budget had been spent versus a target of 69 percent. At the community level, a school council mobilization strategy was developed to increase community participation at the school level. The purpose of the School Councils was to have oversight in the use of the NSB, to promote child health and nutrition, and to contribute to attendance and retention. A total of 505,092 Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) School Council members were reached through citizen engagement initiatives, far exceeding the target of 212,000. Outcomes At the outcome level, the ECE intervention was measured by level of school readiness, which assessed the child’s readiness for primary school, including pre-literacy, pre-numeracy, fine motor skills, language level, and socio-emotional development. The average value for school readiness improved beyond the target of 6 percent, from 2 percent up to 8.35 percent. Further, a survey carried out in the winter between 2021 and 2022 found that children who had attended pre-school answered 73 percent of age-appropriate questions, while children who had not participated in ECE were able to answer only 60 percent of the same questions. At the primary level, outcomes were measured through the quality score of primary teaching-learning practices. This score was based on classroom observation that included teaching-learning practices such as classroom organization, instructional support, and emotional support. The baseline for this indicator was 2.53 and the target was 2.65. The project achieved a value of 2.59, halfway achieving the targeted outcome. There are three underlying reasons that probably led to the partial achievement of this outcome. First, hiring of new qualified teachers was stopped at the third year of the project due to fiscal restrictions. Second, the Teacher Development Program launch was delayed and started only towards the third year of the project, thus reducing the available time to achieve an impact on teaching-learning practices. Third, schools were closed for at least ten months (in some cases more) because of the lockdown resulting from the COVID-19 pandemic, interrupting children's learning processes. Although there were notable improvements in teaching-learning practices at the ECE level, the shortcomings were significant at the primary level. Achievement of this objective is therefore rated Modest, with recognition of the substantial achievements at the ECE level. Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Objectives 1 and 2 are rated substantial due to achievement of most outcomes and targets. Objective 3 is rated modest due to only partial achievement of the teaching-learning practices outcome at the primary level, although the ECE readiness outcome was fully achieved. The overall efficacy of the project is therefore rated substantial. Overall Efficacy Rating Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) Substantial 5. Efficiency The PAD included a cost-benefit analysis of the project using the present discounted value method. The expected benefits from this project were: (a) additional primary and secondary students completing their education from public, private, and PPP schools who would earn higher wages than those who did not complete school; (b) improved relevance and quality of education that would lead to youth with better learning skills and therefore higher earning potential; and (c) lower repetition rates and dropouts from public schools, resulting in less wastage of both public and private resources as students would complete their schooling in a shorter period. The overall costs of the project included the NSB expenditures carried out during the project, expenditures under the new interventions of the PESP III project, and private costs, including out-of-pocket costs and opportunity costs from not having children contributing to the family income. The assumptions that were used to estimate the net present value (NPV) of project benefits, the benefit-cost ratio, and the internal rate of return (IRR) were: (a) an estimated one million additional children would be enrolled through the PPP and education voucher programs; and (b) inputs from the proposed project would increase grade promotion by 3 percent and reduce repetition by 3 percent with respect to the baseline. Without the project, the analysis estimated that promotion rates would decrease by 3 percent and repetition would increase by 3 percent with respect to the baseline. With respect to ECE, the analysis assumed that annual intake would increase by 2 percent per year during the life of the project. Project costs were forecasted based on data for the 2016-2017 fiscal year expenditures for NSB. Additional expenditures under the project were also costed and included in the estimate. As a whole, PESP III overall costs were estimated to be US$3.77 billion in nominal terms. On average, the private household expenditures for a child were US$29 per year for the primary grades and US$59 per year for children in secondary school. Based on existing data at the time, the analysis assumed that 7 percent of children of primary age and 22 percent of secondary school age would be involved in wage labor if they were not enrolled in school. The analysis estimated that US$399 million would be spent by households, and US$553 million would be the opportunity cost of foregone wages for the life of the project. Benefits due to increased lifetime labor earnings were estimated at US$728 per annum for children completing as compared to those who did not complete primary school, and for secondary the estimate was US$1,107 per annum for those who completed as compared to those who did not. Efficiency benefits due to reduced repetition were estimated at a reduction of 1.3 student years for primary and 1.2 student years for secondary students. The discount rate was 12 percent, and yearly inflation was calculated to be 5 percent per year. Based on the above reasonable assumptions, the estimated NPV of economic benefits for the project was US$4.25 billion with an IRR of 26 percent. The sensitivity analysis using varying quality premiums in employment rates resulted in a range of US$3.36-5.15 billion for the NPV and 23-29 percent for the IRR. The ICR focused its benefits estimates on enrollment data for public, private, and PPP schools. When compared to the economic analysis carried out at appraisal, this would be a lower bound given that benefits from quality inputs were not taken into account. On the cost side, a similar analysis was carried out as that of the PAD, taking into account public and private investment costs, as well as opportunity cost for additional enrollment of children at the primary and secondary level. The resulting NPV for the project was estimated at US$3.06 billion, Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) with an IRR of 27 percent. Given that the benefits were underestimated, results are similar to the ones at appraisal. In terms of implementation, the project suffered some initial delays due to political changes in government, the need to contract TA to support project implementation, and the processing of an initial restructuring to reflect an institutional reform linked to teacher professional development. An extension of six months was required to close the project. In the context of a pandemic, where schools were closed for more than ten months and government processes were disrupted, this extension was reasonable. Throughout project implementation and, as part of the three restructuring processes, DLIs/PBCs were revised in order to reflect changes in project design (for example the creation of a new agency responsible for teacher continuous professional development) or to respond to COVID-19 and to macro-fiscal constraints. Of the total of nine PBCs, seven were fully disbursed and two were partially disbursed. These were PBC 4 on girls’ stipends (US$4.35 million was not disbursed) and PBC 8 on NSB (US$3.34 million was not disbursed). Overall, project disbursement was US$289.98 million, i.e. 97 percent of the total loan. The Government of Punjab spent an estimated US$10.83 billion, and the FCDO program financed, under its full program, GBP430 million (98 percent of the original GBP427 million). Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  29.00  Not Applicable 100.00 ICR Estimate  27.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives is rated high, as the objectives were fully aligned with the Pakistan Country Partnership Strategy (FY 15-19), the Government of Punjab’s Education Sector Plan (2013-2017), and the subsequent “The New Deal.” Efficacy is rated substantial as most of the outcomes were obtained, with the exception of teaching-learning outcomes at the primary level. The cost-benefit analysis resulted in high returns, with some limited implementation shortcomings and a six-month project extension, producing an efficiency rating of substantial. Taken together, the overall Outcome rating is Satisfactory, reflecting only minor shortcomings in the project's preparation, implementation, and results. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) a. Outcome Rating Satisfactory 7. Risk to Development Outcome The risk to development outcome is deemed low for several reasons. First, the project financed only a small fraction of the overall program, and therefore it is likely that interventions will continue. Second, the long history of reforms, partly supported by four DPFs and three IPFs, demonstrates a strong commitment to improving coverage and quality of education in the Province of Punjab. Third, the project invested in strengthening several of the institutions within the SED. For example, the project supported the creation of the Teacher Development Program under the QAED, the establishment of a revamped ECE program, and the reform of student assessments under the PEC. Finally, the development of a strong information system allows for better decision-making and targeting of critical issues. However, Pakistan is facing a difficult macro fiscal environment that is forcing some restrictions in funding for the social sectors, including the education sector. At the same time, the number of school-age children continues to grow, demanding an increase in expenditures to provide these children with a spot in a classroom, be it in a public, private, or PPP school. 8. Assessment of Bank Performance a. Quality-at-Entry Project design built on the Bank's experience with the education sector in Punjab, including a series of DPFs and investment projects. The project took into account prior knowledge and supported activities that had been successful and needed continuity (for example, the provision of stipends for girls) and introduced new reforms (such as the classroom observation tool and the ECE program). In addition, the project benefited from having an ongoing project management unit tasked with managing all donor and multilateral funding. This meant that implementation capacity was in place at the time of project launching. Given that achieving improvements in quality of education is a long-term endeavor, this is an appropriate support strategy. The project was designed to focus on results. This was done by planning disbursements based on achievement of DLIs/PBCs. In addition, funds were directly disbursed to the SED, avoiding the need to open a designated account for the project. Project funds were allocated to provide technical assistance to the various institutions under the SED in order to support the achievement of the reforms, as well as supporting project implementation, monitoring, and evaluation. At appraisal, two key outcome indicators were not available: quality score of primary teaching-learning practices and level of school readiness for ECE. In the case of the first indicator, there was no global agreement on measuring teaching-learning practices. In the case of ECE, the intervention model had not been fully designed, and consequently the indicator could not be finalized. The team expected that these indicators would be available by December 2016, but they were only agreed upon in the third year of the project, as part of the restructuring of June 2019, and were set up only after an ECE monitoring system Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) was established and a survey to measure teaching-learning practices made available in March 2018 and July 2016, respectively. The definition of indicators could have been more accurate. For example, the project objectives stated that there would be a focus on LPDs, but this focus was not translated into all the PDOs. Data were disaggregated mainly for school participation, but not for other PDO indicators. Also, the project supported the education sector as a whole, including ECE, primary, and secondary. However, not all indicators covered all these levels. For example, completion rates were measured only for primary education. Furthermore, primary completion was measured only in government schools, while the project invested substantially in private and PPP schools. Quality-at-Entry Rating Satisfactory b. Quality of supervision The World Bank supervision team was composed of headquarters and field staff who provided fiduciary and TA support throughout project implementation. The team responded to changes in the institutional framework and to the lockdowns imposed by the COVID-19 pandemic through three project restructurings and one project extension. Institutional changes included the replacement of the Department of Staff Development (DSD) with the QAED, introducing a completely new teacher professional development model. Also, there were major changes in student testing strategies and the introduction of a new ECE program. The team supported these changes, but there were still substantial delays in the implementation of the teacher training program and the rolling out of ECE. There were also changes in leadership in the SED and the Project Management and Implementation Unit (PMIU), particularly during the first two years of project implementation. This had implications for the continuity of project implementation. In response, the World Bank team promoted the strengthening of the Project Steering Committee and the preparation of action plans to improve project implementation. Coordination with FCDO flowed well, with joint meetings in the quarterly roadmap stock-taking exercise that was led by the Chief Minister. Also, the team secured US$1 million Externally Financed Output from FCDO to assist with TA work linked to the implementation of education reforms in Punjab. The World Bank responded to the COVID-19 pandemic well and in a timely manner. The team worked closely with the client in the weeks prior to the declaration of a pandemic emergency and the consequent school closures, putting in place a distance learning program and a communications strategy to support parents. The team supported the client in the development of surveys to parents and a survey to schools once they reopened to assess dropout rates, among other issues. This is how they detected that 21 percent of adolescent boys and 8 percent of adolescent girls were lost from the education system. The World Bank also supported measuring learning losses once schools reopened. There was a six-month extension of the project, which seems reasonable, in the context of major disruptions resulting from the pandemic. Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) Quality of Supervision Rating Highly Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project's monitoring and evaluation design relied mainly on existing institutional monitoring systems that were under the purview of the various institutions of the SED. Several of these systems had been developed with support from the World Bank as part of prior operations. In total, twelve systems were relied upon for project M&E, three of which were specifically introduced in the PESP III project design. The project contributed to the School Information System, the 2016 Private School Census, and the Assistant Education Officer Classroom Observation Tool. The project's design also included third-party verification of expenditures for the girls’ stipend program, the PEF model (education vouchers for private schools and PPPs), NSB, and School Council mobilization. Project design included both baseline and endline surveys. The project objectives specified that LPDs would be a focus of the project; however, most PDO indicators were not designed to measure the impact specifically on LPDs as opposed to other districts in Punjab. Also, while the project encompassed ECE, primary, and secondary levels, most indicators focused on the primary level. In addition, the project made a substantial investment in strengthening the PEF model; still, data for private and PPP schools were not available to estimate completion rates, such that the indicators measured results stemming from only part of the project's intervention. In addition, while the project invested substantial amounts of funds on quality inputs, and the province had experience in student testing, no indicators were defined on improved learning. b. M&E Implementation Overall, the M&E systems were implemented as designed. A public school census was carried out on an annual basis, and Monitoring and Evaluation Assistants carried out monthly visits to schools to collect data for the information system. A private school census was carried out in 2016, but there was no follow-up census thereafter. Third-party assessments were carried out as planned for the four expenditure programs, thus allowing for disbursements for the corresponding DLIs/PBCs. Two impact assessments were carried out for the girls’ stipends program and for the SMS re-enrollment communications campaign in response to COVID-19 school closures. In addition, household surveys assessed the impact of COVID-19 on children’s enrollment in schools once they reopened. The endline survey was also carried out as planned. Data collection and management was carried out through government systems, thus ensuring their sustainability. Depending on the type of data, the responsible agency took on the task. The project supported the development and implementation of a Data Center within the PMIU to enhance data collection, integration, and storage. The ICR does not provide information on student assessments, beyond the fact that two full rounds of student testing under the new system were carried out. Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) c. M&E Utilization The monitoring systems developed under the project were implemented by the relevant units, providing them with information for the decision making process. The Integrated Education Dashboard, for example, is available at the district level and provides information on 27 indicators in real time. This dashboard pulls information from the Annual School Census, from the PEF (private and PPP schools), from the QAED teacher training program, classroom observation data, etc. The dashboard also has options to analyze data as needed. The PMIU provided training to district officials in all 36 districts on use of the dashboard. The Data Center, also supported by PESP III, is being linked to the National Database and Registration Authority, thus strengthening the institutionality of data management. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project did not trigger social and environmental safeguards. It did not operate in area with indigenous people. The project was Category C for environmental assessment, as it did not involve any civil work or school rehabilitation. b. Fiduciary Compliance Fiduciary compliance was satisfactory. Funds were disbursed directly to government accounts. Counterpart funds were always available as needed. As of the writing of the ICR, however, there was still US$2.3 million pending justification under Component 2 of the project, in addition to overdue interim financial reports and internal audit reports. To date, as reported by the Task Team, all expenditures have been fully documented, but final reimbursement of remaining funds is still pending, together with the final financial reports. c. Unintended impacts (Positive or Negative) None reported. d. Other --- Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Long-term engagement with a client can secure results in a sector such as education. Investments in education are long-term endeavors and, as such, require substantial commitment and availability of resources. This project came after a DPF series and two investment projects, allowing the client and the World Bank to build on prior interventions and learn from prior experience. Results-based lending allows teams to focus on outcomes instead of processes, but it requires robust information and fiduciary systems. This operations model relies on government systems to procure, contract, and deliver goods and services. Management and information systems must therefore be in place and operate under clear accountability mechanisms. Private schools provide a means to increase access in an environment of substantial population growth, but accountability is important. As observed in this project, low-cost private schools tend to cater to poor children, perform well, and serve vulnerable groups (such as girls). However, these schools need to be accountable to authorities by allowing regular monitoring and student performance assessments. This is not always the case. As reported for this project, private schools do not always allow government authorities to enter premises. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well written and provided extensive information on a large and complex project. The document appropriately divided the PDO into three objectives and carried out an in-depth analysis for each of them, enriching the analysis with additional information from studies carried out by the project to assess specific interventions. The ICR was candid, noting that there were gaps between the proposed focus on low-performing districts in the objectives and the lack of necessary monitoring data to assess them. Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Third Punjab Education Sector Project (P154524) However, the theory of change proposed in the ICR was not aligned with project objectives and outcomes, leaving out some very important project interventions, most notably ECE. As a result, the analysis of the objectives was sometimes not well organized and difficult to understand. In addition, the ICR did not assess much of the work supporting PEC in reforming the student assessment program and the implementation of several rounds of performance testing. Finally, the document could have benefited from some final editing, including a complete acronym list. Notwithstanding these shortcomings, the quality of ICR is rated as substantial. a. Quality of ICR Rating Substantial Page 18 of 18