Report No: ICR00177 IMPLEMENTATION COMPLETION AND RESULTS REPORT IBRD/87910 ON A LOAN IN THE AMOUNT OF US$ 120 MILLION TO THE PEOPLE'S REPUBLIC OF CHINA FOR THE China: Guangdong Compulsory Education Project 16 September, 2024 Education Global Practice East Asia And Pacific Region The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT CURRENCY EQUIVALENTS (Exchange Rate Effective {Sep 03, 2024}) Currency Unit = Chinese Yuan (CNY or RMB) 6.3515 = USD 1 FISCAL YEAR Regional Vice President: Manuela V. Ferro Country Director: Mara K. Warwick Regional Director: Alberto Rodriguez Practice Manager: Mario Cristian Aedo Inostroza Task Team Leader (s): Salman Asim ICR Main Contributor: Sajitha Bashir, Aya Kibesaki, Martin Moreno The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT ABBREVIATIONS AND ACRONYMS BBT Ban Ban Tong (classroom connection through ICT) CPF Country Partnership Framework CPS Country Partnership Strategy CPMO County Project Management Office DOF Department of Finance ECE Early Childhood Education EFA Economic and Financial Analysis EMP Environmental Management Plan GDP Gross Domestic Product IBRD International Bank of Reconstruction and Development ICR Implementation Completion and Results ICT Information and Communications Technology IE Impact Evaluation ISR Implementation Status and Results Report IRI Intermediate Results Indicator M&E Monitoring and Evaluation NPV Net Present Value PAD Project Appraisal Document PDO Project Development Objective PPMO Provincial Project Management Office RF Results Framework RMB Renminbi (Chinese Yuan) ROR Rate of Return RPF Resettlement Policy Framework The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT TABLE OF CONTENTS DATA SHEET ................................................................................................................................................. i I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................................................1 II. OUTCOME ...................................................................................................................................................7 III. KEY FACTORS AFFECTED IMPLEMENTATION AND OUTCOME................................................................... 16 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ............................ 17 V. LESSONS AND RECOMMENDATIONS ............................................................................................................. 22 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................................................ 23 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ....................................................... 30 ANNEX 3. PROJECT COST BY COMPONENT ......................................................................................................... 33 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................................................ 34 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ................................. 43 ANNEX 6. TEACH ............................................................................................................................................... 46 ANNEX 7. PROJECT COST SAVINGS AND ADJUSTMENTS ..................................................................................... 51 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT DATA SHEET @#&OPS~Doctype~OPS^dynamics@icrbasicdata#doctemplate BASIC DATA Product Information Operation ID Operation Name P154621 China: Guangdong Compulsory Education Project Product Operation Short Name Investment Project Financing (IPF) China: Guangdong Compulsory Education Operation Status Approval Fiscal Year Closed 2018 Original EA Category Current EA Category Partial Assessment (B) (Restructuring Data Sheet - 28 Jun Partial Assessment (B) (Approval package - 31 Oct 2017) 2023) CLIENTS Borrower/Recipient Implementing Agency PEOPLE'S REPUBLIC OF CHINA Guangdong Department of Education DEVELOPMENT OBJECTIVE Original Development Objective (Approved as part of Approval Package on 30-Oct-2017) The project development objective is to improve learning facilities and teaching quality in selected public primary and junior secondary schools in project counties. s s s s s s @#&OPS~Doctype~OPS^dynamics@icrfinancing#doctemplate FINANCING Financing Source Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 120,000,000.00 117,406,153.93 117,406,153.93 IBRD-87910 120,000,000.00 117,406,153.93 117,406,153.93 i The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Non-World Bank Financing 159,400,000.00 180,000,000.00 180,000,000.00 Borrower/Recipient 159,400,000.00 180,000,000.00 180,000,000.00 Total 279,400,000.00 297,406,153.93 297,406,153.93 RESTRUCTURING AND/OR ADDITIONAL FINANCING Amount Disbursed Date(s) Type Key Revisions (US$M) 23-Jun-2023 Manual 116.38  Loan Closing Date Extension 14-Jun-2024 Manual 117.11 @#&OPS~Doctype~OPS^dynamics@icrkeydates#doctemplate KEY DATES Key Events Planned Date Actual Date Concept Review 09-Jun-2015 19-Aug-2015 Authorize Negotiations 17-May-2017 17-May-2017 Approval 31-Oct-2017 31-Oct-2017 Signing 30-Nov-2017 16-Jan-2018 Effectiveness 26-Apr-2018 02-Apr-2018 ICR/NCO 16-Sep-2024 -- Restructuring Sequence.01 Not Applicable 14-Jun-2022 Restructuring Sequence.02 Not Applicable 28-Jun-2023 Mid-Term Review No. 01 04-Jan-2022 28-Mar-2022 Operation Closing/Cancellation 30-Nov-2023 30-Nov-2023 @#&OPS~Doctype~OPS^dynamics@icrratings#doctemplate RATINGS SUMMARY Outcome Bank Performance M&E Quality Highly Satisfactory Moderately Satisfactory Substantial ISR RATINGS ii The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Actual Disbursements No. Date ISR Archived DO Rating IP Rating (US$M) 01 08-Mar-2018 Satisfactory Satisfactory 0.00 02 19-Sep-2018 Satisfactory Satisfactory 5.00 03 03-Apr-2019 Satisfactory Satisfactory 5.00 04 23-Jul-2019 Satisfactory Moderately Satisfactory 5.00 05 26-Jan-2020 Satisfactory Moderately Satisfactory 17.15 06 06-Nov-2020 Satisfactory Moderately Satisfactory 35.88 07 03-Dec-2021 Satisfactory Moderately Satisfactory 66.39 08 28-Jun-2022 Satisfactory Satisfactory 85.92 09 16-Jan-2023 Satisfactory Satisfactory 97.13 10 08-Mar-2024 Satisfactory Satisfactory 118.45 @#&OPS~Doctype~OPS^dynamics@icrsectortheme#doctemplate SECTORS AND THEMES Sectors Adaptation Mitigation Major Sector Sector % Co-benefits Co-benefits (%) (%) Primary Education 33 0 0 Education Public Administration - Education 11 0 0 Secondary Education 34 0 0 Information and Communications ICT Services 22 0 0 Technologies Themes Major Theme Theme (Level 2) Theme (Level 3) % Human Development Access to Education 34 Education and Gender Education Facilities 56 iii The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Standards, Curriculum 56 and Textbooks Teachers 11 Social Development and Social Protection Disability 34 Protection iv The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT ADM STAFF Role At Approval At ICR Practice Manager Mario Cristian Aedo Inostroza Regional Director Alberto Rodriguez Global Director Luis Benveniste Practice Group Vice President Mamta Murthi Country Director Mara K. Warwick Regional Vice President Manuela V. Ferro ADM Responsible Team Leader Salman Asim Salman Asim Co-Team Leader(s) ICR Main Contributor Sajitha Bashir, Aya Kibesaki, Martin Moreno I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the time of appraisal, China was an upper middle-income country, with great geographical and social disparities in per capita income. Guangdong province was one of the most populous and the richest provinces, with an average per capita income in 2015 which was more than 2.5 times that of the poorest province.1 However, the ratio of rural and urban incomes within Guangdong was 1:3 in 2010, according to the Project Appraisal Document (PAD). Many workers migrated from rural to urban areas within the province, in search of better livelihoods, and about 1 million children of the workers were left in the rural areas. 2. Education was seen by national policy makers as essential to drive the next stage of economic development and create opportunities for social mobility for the population and Guangdong’s education policy focused on reducing disparities and improving quality. China’s National Plan for Medium- and Long-Term Education Reform and Development 2010-2020, in place at the time of appraisal. highlighted the necessity to make education more student-oriented by revamping teaching content and methods and to more resources to disadvantaged areas. Guangdong, being the richest province, had already achieved universal nine years of education and was close to universalizing secondary education in 2014. In line with national priorities, the province developed its education reform program “Guangdong Education Create Strong Schools and Modernization Program 2010-2020”, which was to be implemented in two phases. Phase 1 included the reduction in disparities in school facilities, teacher allocation and basic education resources, as well as connecting all classrooms through information and 1Gross Regional Product Per Capita in 2015 Gansu: 26,165 CNY, Guangdong: 67,503 CNY. (National Bureau of Statistics of China https://www.stats.gov.cn/sj/ndsj/2016/indexeh.htm) Page 1 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT communication technology (ICT), specifically the Ban Ban Tong (BBT) teaching platform.2 Phase 2 focused on improving education quality through a competency-based approach to teaching and learning and strengthening the training of teachers and principals and other initiatives. By 2020, all counties within the province should have met the Phase 1 standards and 85 percent should have met the Phase 2 standards. The project was aligned with education strategy at both the national and provincial levels. 3. China’s decentralized system of education financing, as well as the residency registration system (hukou), created disadvantages for poorer provinces and poorer counties, which were partly compensated by targeted support from the national and provincial governments. Each county is responsible for provision of compulsory education and relies largely on its own revenues to build schools, hire teachers and invest in other resources. According to the hukou system, individuals are registered in their county of birth and are entitled to public services such as education and health only in those locations, even if they migrate to urban areas. This created a group of “left-behind children”, which were recognized by the State Council of China as a particularly vulnerable group. 3 In Guangdong province, about 297,000 children were left behind by migrant workers from the poorest counties with grandparents or other family members, to continue school education. Another vulnerable group identified in the Guangdong Education Plan were children with special needs, estimated to number about 3,800 according to national criteria for identifying and classifying disabilities. 4. The 16 counties selected under the project suffered from great disadvantages at the time of appraisal . GDP per capita was just one-third that of the average for the province as a whole. These counties accounted for 34 percent of all students enrolled in basic education in Guangdong, but they were lagging behind in meeting the Phase 1 objectives of the Guangdong’s Education Plan. With a mere two percent of government revenues in the province, they had higher pupil-teacher ratios, lacked subject teachers, insufficient classrooms and poor quality of teacher training. Only 30 percent of primary schools and 48 percent of junior secondary schools in these counties had ICT equipment, compared to an average of 52 percent and 69 percent, respectively, for the province. 5. The project was aligned with the Country Partnership Strategy (CPS) of FY12-FY16, which notes the need for enhancing access to high-quality public services such as education, particularly in underdeveloped areas, and the Bank’s role in promoting innovation, knowledge sharing and cooperation. Specifically, the project was designed to address educational disparities in the most disadvantaged counties of the province (the pillar of “inclusive development” of the CPS), and support innovative pilot projects relating to teacher training/professional development and vulnerable children that could be provide lessons and be scaled up to support the education reform program of the province (which aligned with “innovation” as principle of engagement of the CPS). 6. The Guangdong Compulsory Education Project was considered an important demonstration at both the provincial and national levels. It was the first collaboration between the World Bank and the Guangdong Department of Education in compulsory education. Theory of Change (Results Chain) 2 BBT is a multimedia platform that provides a comprehensive classroom environment, including software (online teaching resources) and hardware (personal computer, television) components. (TEACH Report, 2023) 3 In 2016, the State Council defined left-behind children as minors under the age of 16 whose 1) both parents are migrant workers, or 2) one parent is a migrant worker and the other lacks custodial capacity. Page 2 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 7. The PAD presented a coherent theory of change and results chain, linking the four components of the project with the detailed activities, outputs, short-term outcomes which were to be achieved during the project’s lifetime, and long-term outcomes that were expected to be achieved after the project . It aimed to support the Guangdong Education Reform Program in reducing disparities between the 16 selected counties and the rest of the province in basic school, teacher and ICT facilities and the quality of teacher training, while providing lessons from pilot interventions to benefit the whole province. The underlying theory of change was that improved conditions for learning, improved teaching quality and policy guidance based on evaluations of innovative pilots to improve equity – short-term objectives directly realized by the project - would improve student learning outcomes after the project. (See Figure 1 below and description of PDO and project components later). 8. The project comprised activities financed by IBRD funds and by counterpart funds, which were largely distinct and funded separately by the two funding sources. IBRD financing focused on the more forward-looking initiatives of the province’s reform program (such as the use of ICT and improved teacher training programs) and piloting innovations to address equity and quality challenges in rural areas. Counterpart funding, on the other hand, financed important but more “conventional” activities, such as construction of classrooms, teacher dormitories and long duration pre-service teacher training.4 Counterpart funding was provided by the provincial government, and in this sense, World Bank funding helped to leverage additional resources for the province beyond its normal budget. 9. Nevertheless, three shortcomings of the theory of change and the link with the results framework can be identified. First, the project comprised activities that were separately financed by IBRD funds and counterpart funds, but this was not duly reflected in the design to consider them all as integral parts constituting the project. For example, while the project results chain included outcomes that were expected from activities funded by both sources, and the estimate of beneficiaries resulting from all project activities was included in the results framework, the latter excluded the outputs supported by the counterpart funds (specifically, construction activities and teacher training activities financed by the provincial government under components 1 and 2, respectively). Second, while the innovative, pilot activities under component 3, financed entirely by IBRD funds, were reflected in the results chain (see Figure 1) as making a significant contribution to the overall project outcomes and formed an important part of the rationale for the project as reflected in the PAD and CPS (discussed later), the PDO and PDO indicators did not reflect this intended objective. The results framework reflected these multiple innovative activities with just two intermediate result indicators (“total number of activity and resource teachers trained” and “a policy paper”) among some 20 other indicators listed for IBRD funded activities. As described in the PAD, the intended short-term project outcome was a policy paper and policy guidance to be issued by the Education Department of Guangdong Province to all counties, based on systematic evaluations of the cost-effectiveness and impacts of the pilots. The guidance would encompass how to scale up customized interventions for migrant and special needs children and innovative approaches for teacher professional development and science teaching. Finally, a third shortcoming was that critical assumptions and external factors that may have affected the various links in the results chain were not presented. These included assumptions regarding availability of local contractors (for civil works financed by counterpart funds), suppliers of ICT equipment and mobile laboratories and training providers, the staffing requirements at the provincial level, and the technical capacity of the provincial educational authorities to distil lessons from the pilots and develop policy guidance. 4The in-service (short duration) multisubject training and the BBT training, and also project management were funded by both funding sources. Page 3 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 10. Figure 1: Project results chain Project Development Objectives (PDOs) 11. The project development objective is to improve learning facilities and teaching quality in selected public primary and junior secondary schools in project counties. Key Expected Outcomes and Outcome Indicators 12. Outcome 1: Improve learning facilities Page 4 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Outcome indicator: Share of project classrooms where ICT is used (percentage) 13. Outcome 2: Improve teaching quality Outcome indicators: a. The proportion of trained teachers whose teaching practices meet a minimum threshold in targeted competencies b. The proportion of trained backbone teachers whose teaching practices meet a minimum threshold in targeted competencies c. The proportion of teachers trained in ICT whose teaching practices meet a minimum threshold in targeted competencies Components 14. Component 1: Improve School Equipment and Facilities (Total Planned: USD 168.0 million /IBRD USD 67.2 million. Estimated Actual: Total USD 160.7 million/IBRD USD 62.7 million). Subcomponent 1.1 – provision and installation of ICT equipment in classrooms; development of digital educational content for students made available on an existing online platform. Subcomponent 1.2 - construction of classrooms and teacher accommodation, particularly in rural locations. This is financed by counterpart funds. 15. Component 2: Strengthen the Training and Assessment System for Teachers and Principals (Total Planned: USD 57.6 million / IBRD USD 12.4 million. Estimated Actual: Total USD 54.6 million/IBRD 16.9 million). Subcomponent 2.1 - improvements in existing training content for in-service and pre-service training of teachers and principals as well as the development of new training content. Development of new content and training delivery models for multi-subject teaching (including both short and long-term courses). Counterpart funding will be used for the long-term training and a combination of counterpart and IBRD funding for the short-term training. Subcomponent 2.2 - improvements in the current training system by designing an enhanced quality assurance mechanism (QAME) for teacher continuous professional development (CPD). 16. Component 3: Pilot Reforms to Improve Equity and Quality of Education (Total Planned: USD 39.8 million / IBRD USD 39.8 million. Estimated Actual: Total USD 29.1 million/IBRD 29.1 million). The component supported four pilots: (a) Providing support, through equipment and personnel resources, to left-behind children in public primary and junior middle schools. (b) Enabling inclusive education in primary and junior middle schools with a particular emphasis on children with mild to moderate disabilities (equipment and personnel resources) (c) Providing mobile labs to schools in hard-to-serve areas. (d) Strengthening the existing system of school partnerships to improve teacher mentoring, teaching quality and school management practices. 17. Component 4: Project Management, Monitoring and Evaluation (Total Planned: USD 2.9 million / IBRD USD 0.3 million. Estimated Actual: Total 1 million/IBRD 0.47 million). Page 5 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT This component covered project management, technical assistance, supervision of the project, monitoring and evaluation, training of project implementation personnel, study tours, workshops and seminars. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) 18. The PDO, PDO indicators and targets and project components were not revised. However, after the mid-term review, the project was restructured in June 2022, when 71 percent of IBRD funds had been disbursed. Some intermediate results indicators were revised. The principal reason for the first restructuring was to utilize the savings arising from (i) lower prices of the teaching platform, BBT , than originally envisaged, through a drop in market price (especially of the hardware) as well as efficiency in procurement (the BBT included the hardware, software, and libraries of educational resources aligned with the curriculum) (ii) depreciation of the RMB vis-à-vis the dollar, (iii) cancellation of the development of digital educational content, and (iv) cancellation of external trainings. Further implementation of the pilot on mobile labs was cancelled after procurement of two labs for two counties, in consideration of the limited time left to carry out the open, competitive bidding and conclude the contract before the Project closing date. The government decided to exclude the development of digital educational content from the project, as it was to be covered through a separate large government program called "Guangdong Province Compulsory Education Stage National Curriculum Digital Textbook Large-scale Application", and external trainings were cancelled due to COVID-19. Around $38.5 million were savings; a large part of which came from the BBT procurement, since equipment was procured at less than half of the initially estimated cost. The bulk of the savings was allocated to procurement of additional BBT teaching platforms, which went up from the originally planned 15,200 to 41,101. This automatically increased the number of direct project beneficiaries, both students and teachers, which increased from 554,800 to 1,990,000. Some additional funding was also allocated to training of teachers (for BBT teachers, as well as backbone teachers) and increased the number of teachers to be trained from 23,600 to 50,800. 19. A second restructuring was done in June 2023, to extend the project by 5 months to allow for completion of some activities, namely: (i) to assess the impact of the teacher trainings and update the results framework, especially the PDO indicators (ii) to complete the remaining construction work (financed by counterpart funds) and acceptance in the two cities that had not yet completed it. 20. On the Guangdong provincial government side, the improvement of green building standards for infrastructure projects and the increase in labor, material and equipment costs during the construction period has led to more funds spent on classrooms and dormitory construction. 21. The savings and the original and final allocation by component are given in Annexes 3 and 7. 22. Other Changes 23. Rationale for Changes and Their Implication on the Original Theory of Change 24. The changes did not impact the theory of change, as they mainly entailed increasing the scale/beneficiaries. Page 6 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT II. OUTCOME A. RELEVANCE OF PDO Assessment of Relevance of PDOs and Rating 25. The relevance of PDOs is rated as High. The project’s objectives are aligned with the World Bank’s Country Partnership Framework at the time of project closing, which covers the period FY2020-2025. 5 The CPF highlights the need to reduce inequities in access to quality education to promote future broad-based prosperity. It also stresses the urgency of focusing on learning outcomes and to shift future financing flows to learning-enhancing investments with the aim of reducing disparities. The CPF specifically mentions the Guangdong Compulsory Education Project for piloting approaches to reduce disparities for the “left-behind” children of migrant facilities and for children with disabilities. The piloting of innovations and reforms, in general, is identified as one of the WBG’s greatest value additions. 26. Due to the selectivity criteria applied for IBRD lending to China, and a sharper focus on market and fiscal reforms, promoting green development and supporting a few services in lagging regions, there is no lending for education in the CPF. However, the Bank will support programmatic analytic work on promoting inclusive and quality education in China (FY22), which directly aligns with the objectives of this project. B. ACHIEVEMENT OF PDOs (EFFICACY) 27. The PDO is evaluated in two parts: (i) improvement of learning facilities and (ii) improvement of teaching quality. Both outcomes (objectives) were expected to be achieved in selected public primary and junior secondary schools in the 16 project counties. These objectives are evaluated separately, although in principle the two are inter-related. Specifically, improvements in learning facilities (especially those related to ICT) can lead to improvements of teaching quality. However, not all schools that received additional investments in equipment and infrastructure also received teacher training, and therefore the outcomes are evaluated separately. As stated earlier, the outcomes related to the four pilot interventions, although not part of the PDO, are also assessed given their importance in the overall results chain and the significance of the pilots for the country’s strategy and the World Bank’s CPF. 28. As the scope of the project did not change, and there was only a reallocation of the loan between components, with no revision of outcome targets, no split rating has been done and project outcomes are assessed against the original targets, which are also listed in the restructuring paper. 5 World Bank (2019). Country Partnership Framework for the People’s Republic of China. Report No. 117875-CN. Page 7 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 29. The overall PDO rating for efficacy is High. Assessment of Achievement of Each Objective/Outcome 30. PDO Outcome 1 (i) Improve learning facilities (in selected public primary and junior secondary schools in project counties) 31. Outcome Indicator: Share of project classrooms where ICT is used (Percentage) 32. The project achieved this outcome fully, as measured by the outcome indicator . The percentage of project classrooms using ICT reached 100%, exceeding the original target (70%) set at appraisal, and which was retained during project restructuring. Project classrooms included classrooms that lacked the BBT as well as new classrooms constructed with counterpart funding. As noted earlier, the number of schools and classrooms benefitting from these improved learning facilities far exceeded the original target (41,101, compared to 15,200 expected at appraisal), as a result of re-allocation of savings and the fall in the price of the equipment. The project succeeded in achieving utilization of the ICT equipment, as required by the PDO indicator, in this increased number of classrooms. Hence, the overall impact on schools in project counties was significantly greater than originally envisaged. The improvement in learning facilities also came about through the construction of an additional 961 classrooms, financed by government counterpart funding and which, as mentioned earlier, were not included in the project outcomes. 33. Information on the use of ICT in schools was collected not only through measurement of the outcome indicator but also through other ways. Schools and counties collected information as part of their regular supervision of teachers/schools and this data was aggregated at the provincial level. This was the main source of information for monitoring the indicator in the results framework, where the denominator was the number of classrooms were installed under the project and the numerator was defined as whether teachers are using the BBT platform during the classroom session, when schools do their routine checks. In addition, spot checks were done by the Provincial Project Management Office. World Bank supervision missions, which included an IT expert who provided regular advice on the desired functionalities of the platform, conducted spot checks and held discussions with teachers on the use of the BBT platform. In addition, Guangzhou University conducted an evaluation in late 2021, covering 160 schools and over 45,000 beneficiaries (students, teachers and parents). Responses were very positive about the BBT, specifically amongst teachers, who felt that it helped the teachers to prepare better for classes, share quality resources and teaching experiences with others, use master classes and access a larger variety of resources, and amongst students, who felt that classes were more interactive. 6 34. The BBT evaluation highlighted these findings: (i) more than 80% of the teachers were using the all-in-one machine and the projector stand frequently; (ii) more than 80 percent of student felt that BBT platform helped them to learn more and better; (iii) many schools used the platform to collect and record teaching /learning activities and share teacher developed resources; (iv) student evaluations and assessments have changed in many schools as teachers can record and analyze students’ learning progress and provide feedback and (iv) the BBT has functioned like continuous in-service training since teachers were able to watch expert lectures, observe good examples, or exchange and get feedback on the lesson plans, courseware and lessons. However, the extent to which teachers used all resources (or which resources were used most) on the BBT platform was not assessed. 6 Source: Guangzhou University, Project on BBT teaching platform; Performance Evaluation Report. Page 8 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 35. The universal use of ICT in the classrooms was facilitated by several factors . First, teachers were provided training specifically related to the use of the BBT platform and the resources that were provided on it. This was in addition to the much shorter vendor provided training on familiarizing teachers with the equipment. Second, the variety of resources on the platform and eventually, the ability to access national and provincial level resources through stronger internet connectivity at the school level, greatly empowered the teachers. These learning resources (power points, audio-video materials, games) were organized for specific lessons, enabling teachers to choose the ones that they needed. Third, the large size of the touchscreen (86 inches), the audio system and the related equipment made the platform accessible to all the students and kept their interest, while enabling teachers to engage more actively with students. Fourth, the equipment and software became progressively easier to use, requiring much less teacher effort to access resources. Finally, technical support to overcome system errors and problems was usually available either at the school level or the county level, minimizing the time teachers had to spend on this. Further, the vendor would also address equipment/software failures within a short time, as they were from nearby localities. 36. PDO Outcome 2 (ii) Improve teaching quality (in selected public primary and junior secondary schools in project counties) 37. Outcome indicators:  The propor on of trained teachers whose teaching prac ces meet a minimum threshold in targeted competencies (Non-cumula ve) (Percentage)  The propor on of trained backbone teachers whose teaching prac ces meet a minimum threshold in targeted competencies (Non-cumula ve) (Percentage)  The propor on of teachers trained in ICT whose teaching prac ces meet a minimum threshold in targeted competencies (Non-cumula ve) (Percentage) 38. As measured by these outcome indicators, the project exceeded its original end target of 70 percent of trained teachers meeting a minimum threshold in targeted competences, with 96.88 percent of total teachers, 100 percent of backbone teachers and 92.86 percent of teachers trained in ICT meeting this threshold. Further, due to the reallocation of savings to the BBT platform and additional teacher/principal training, the number of teachers trained exceeded the original targets, with 68,843 teachers trained on BBT, 859 backbone teachers and 532 principals. 39. Competences were systematically assessed through classroom observation, using an internationally validated tool called Teach, which scores teachers in time management, classroom culture, instruction and socio- emotional skills. The minimum threshold was set at 2.8 out of 5. Although the initial design envisaged comparing teachers in two rounds of classroom observations, both before and after the training under the project, in the final report of the study, only the second round of observations, conducted in March 2022, was used due to high level of sample attrition caused by teacher transfers, departures and leave. 7 The study therefore compared trained and untrained teachers at one point in time, rather than improvement in teaching competences over time for the same teachers. A total of 228 teachers in the 16 project counties were observed, of which 90 were backbone teachers and 138 were BBT teachers. 40. One factor complicating the assessment is that the target set by the project for achieving the minimum threshold of competences as a result of teacher training, was exceeded by untrained as well as trained teachers alike in the study. 87 percent of untrained teachers performed above the minimum threshold, as against a target 7According to the study report, “The department randomly selected new samples for the second data collection, ensuring equal participant numbers in the treatment and control groups as in the initial survey” Page 9 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT of 70 percent set for trained teachers. This raises the question whether the threshold (set by the provincial authorities in the absence of baseline information) was too low as well as the pedagogical significance of this threshold score. The use of a composite score to assess improvements in pedagogical practice, which is difficult to interpret, has limitations. Further, the instrument was not piloted in the local context, to understand the distribution of teachers’ pedagogical practice and set a meaningful threshold. Notwithstanding these shortcomings in the application of the tool, as the proportion meeting the minimum threshold was higher among trained teachers (95 percent), compared to 87 percent for untrained teachers, this enhancement was likely due to the project interventions on teacher training. In specific domains, such as socio-emotional skills, where only 32 percent of untrained teachers met the minimum threshold level, the project training contributed to a substantial 13 percentage point increase. 41. There were, however, differences between teachers who received training on BBT and backbone teachers who received training for their role. Amongst the former, a noteworthy 99 percent of the BBT trained group achieved the competency threshold, in contrast to 86 percent of BBT untrained groups that met the threshold. Amongst the backbone teachers, however, there was no statistically significant difference between the trained and untrained group (89 percent for both categories). Analysis of sub-competences showed that for training on BBT teachers, the training had the biggest impact on the domain of socio-emotional skills (improvement of 19 percentage points from 30 to 49 percent) and on instruction (from 91 to 99 percent). For backbone teachers, there was no statistically significant improvement in any domain. 42. The reasons for this difference are not explained in the study . It is likely that the targeted training on BBT and the possibility for teachers to immediately use the platform in the classroom and see improved engagement amongst students was responsible for its success. The training on BBT was highly focused on equipping teachers with knowledge of functions and use of the teaching platform, how to optimize teaching using BBT resources, and methods and strategies for curriculum and teaching innovations (including flipping classrooms, project based learning and deep learning/critical thinking). The training involved 1.5 days of face-to-face training and 20 hours of online training. Further, the teachers received ongoing support through teacher groups and informal networks within and between schools, as the use of the BBT spread amongst teachers. 43. One reason may be that the TEACH tool was not the appropriate tool to capture the impacts of the broader and deeper professional development program provided to backbone teachers under the project . The role of backbone teachers is to provide mentoring to other teachers, through continuous support and feedback in schools. For these teachers, the training involved deepening teachers’ knowledge of fundamental concepts of children’s learning and pedagogical theories, in line with their role as mentors to other teachers. The training program covered professional philosophy and teacher ethics, curriculum standards and teaching materials, development of teaching skills and capacity development in education research. The training was of longer duration, involving 55 days of intensive face-to-face workshops, 60 days of practice in respective schools, and 37 hours of online training. The objective of this training was not to improve the classroom practice of the backbone teachers. Additional information is not available to assess the effectiveness of the backbone teacher training program through other evaluation methods. 44. The third data collection using TEACH was carried out in 2023. Due to the high level of sample attrition and the lack of a control group (i.e., data on untrained backbone teachers in 2023), it is not possible to compare trained and untrained teachers in 2022 and 2023, as was done in the previous assessment employing a randomized controlled design. While the observed improvement cannot be causally attributed to the training provided by the project, the results provide suggestive evidence of its effectiveness. In 2023, 97 percent of trained teachers Page 10 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT met the competency threshold, including 100 percent of trained backbone teachers and 93 percent of trained BBT teachers. Notably, the total average score for trained backbone teachers rose to 3.83 (on a 5-point scale), up from 3.33 before they received training in 2022. Improvements were observed across all competency domains compared to their 2022 scores. 45. Other categories of teachers were also trained but no systematic evaluation of the impacts is available . Apart from backbone teachers, principals were also trained in areas similar to the backbone teachers as well as in administrative and educational leadership. 2,984 teachers received short-duration training in multi-subject teaching in order to address the issue of teacher shortage and small class sizes in rural primary schools. Finally, 2,394 teacher trainees have been undergoing a five-year pre-service training as multi-subject teachers, for which they were selected competitively. The first batch will graduate in 2024 and the second batch in 2025, and they will be posted in rural primary schools. This training was financed through counterpart funding and these outputs will contribute to long-term improvements in the quality of education in disadvantaged areas. 46. As noted in earlier sections, the project expended substantial efforts on pilot interventions, which were included as part of the intended project outcomes in the Theory of Change, but which were not included as part of the PDO or PDO indicators. From the point of view of overall project efficacy, therefore, the effectiveness of these pilots and their intended outcomes as sources of policy direction for Guangdong Province is important to assess. 47. Three of the pilot activities were implemented and one (on mobile labs) could not be continued, and while significant activities were carried out, in all cases, there was no systematic evaluation to assess if they could be scaled up or inform future policy. The pilot on “left-behind children” was an important policy priority for Guangdong given the large group of these vulnerable children and the goal of improving educational equity. Under this pilot, 100 schools had set up activity rooms, provided computers and phones for parents to communicate with children, organized cultural and sports activities, trained teachers on psychological counselling, and established care service working groups composed of principals and staff. Children received an average of 1 hour of support per day. Under the inclusive education pilot, 50 schools had set up resource rooms with furniture and equipment and working groups were established to provide special support to children with disabilities. The third pilot on school partnerships involved connecting schools, where 500 weaker schools were partnered with 500 stronger schools to improve teaching practices and administrative leadership. For example, 46 high-quality schools in Guangzhou helped 46 schools with weaker capacity in Zhanjiang City; sending 351 teachers to the recipient schools to support teaching, receiving 430 teachers from the recipient schools to do on- the-job learning, carried out mentor-apprentice pairing activities, and organized online and offline training etc. Positive impact on both teaching/learning and management was noted by both the recipient and receiving schools. Under the fourth pilot, 14 mobile science labs were planned to address the shortage of trained science teachers, improve science teaching in rural areas and by provide hands on learning opportunities to children. The pilot was discontinued due to limited time available to conduct the open, competitive bidding and conclude the contract before the Project’s closing date. The two mobile labs provided over 3,800 classes since February 2022. 48. Of the four pilots, the school partnership program seems to have elicited the most interest among teachers and valuable lessons could be learned for scaling up if a systematic evaluation is undertaken. Part of its success may have been due to the fact that such partnerships exist in other areas (such as Shanghai and other counties of Guangdong) and there was sufficient prior knowledge about how to implement it. The mobile lab program also appears to have had potential, but required addressing the implementation, procurement and maintenance problems. The Guangdong Education Action Plan 2023-2027, issued in 2023, mentions the importance of three Page 11 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT of these initiatives, but lacks details of the lessons drawn from the pilots, how the programs should be scaled and with what adjustments.8 Justification of Overall Efficacy Rating 49. The overall efficacy rating is High. The outcome related to “improve learning facilities” is rated as High, due to the project’s success in enabling the BBT teaching platform to be used in all project classrooms, and more than doubling the number of classrooms in which it was used. The outcome related to “improve teaching quality” is rated as High, as the BBT teacher training helped to improve the proportions of those who acquired minimum competencies. The backbone teacher training also helps improve the rate of teachers meeting a minimum threshold in targeted competences. C. EFFICIENCY Assessment of Efficiency and Rating 50. The ICR follows the basic methodology of the economic analysis used at appraisal, updating certain parameter values with information from project implementation and more recent studies, as well as correcting for some shortcomings in the original analysis. The cost-benefit analysis (CBA) involved identifying the annual investment and recurrent costs and benefits arising from expected improvements in learning outcomes in the long term, and future increase in earnings. Both IBRD and Government costs were included. At appraisal, components 1 (C1) and 2 (C2) were considered amenable for a CBA as the expected beneficiaries could be estimated, and the ICR adopts this approach, especially as no systematic information is available about the effects of the pilot interventions under component 3 (C3). The effect sizes of various interventions were derived from the literature, and these have been updated where relevant. The difference in assumptions for key parameters at appraisal and at project completion are provided in Table 1, and full details of the analysis are provided in Annex 4. 51. The most important changes in assumptions relate to the number of project beneficiaries (increased in the ICR, based on actual project data) and the effect size of the ICT intervention (reduced in the ICR, based on relevant studies). As can be seen from Table 1, the number of beneficiaries is significantly higher in this analysis (1,472,170) compared to that estimated at appraisal (663,600), because of (i) the supply of BBT platforms being almost three times more than the original target, due to the fall in prices and project savings and (ii) the addition of student beneficiaries due to the 961 classrooms built through government funds (46,128 beneficiaries).9 At appraisal, the students who were expected to benefit from teacher training programs were estimated and included in the economic analysis. However, they are excluded in the ICR, because most of the teachers trained were for using the BBT platform, and hence already included under (i), and because of the possibility of double counting as trained backbone teachers and principals also partly benefitted the same students. Furthermore, as previously indicated, the lack of control group comparisons makes it difficult to draw causal inferences about the impact of the teacher training interventions on backbone teachers. Therefore, it is reasonable to exclude the impact of this training on student learning from the analysis. With respect to assumptions of the effect size, the ICR uses a more relevant estimate of the impact of ICT use, based on a randomized control trial in Chinese rural schools. The evaluations 8Left-behind children, inclusive education, school partnerships 9The reason for the exclusion of beneficiaries of classroom construction is not explained in the PAD, even though the economic analysis includes the costs of classroom construction (financed from government funds). For the number of additional beneficiaries, original calculations at the project appraisal were using 35 students per classroom. This number has been updated to 48 students per classroom. Page 12 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT conducted under the project do not provide estimates of the impact of BBT training on student learning. The use of effect size of use of ICT from another rigorous study in rural China, as done during project appraisal, provides an appropriate methodology for estimating the potential benefits from the project’s intervention on using ICT . 52. Further, the PAD provided three scenarios based on different potential beneficiary numbers, while the ICR uses the actual number of beneficiaries and provides three scenarios based on sensitivity to different effect sizes (as there is uncertainty regarding this). The study of ICT in Chinese rural schools gives a range of values from 0.08-0.03 (the latter effect is not significant), based on how intensive the teacher training was. This range was used for the sensitivity analysis, as it is likely that the use of the BBT platform varied across the 41,101 classrooms, depending on the ability and interest of the teachers. 53. Annual investment costs were estimated at appraisal while the actual annual disbursements have been used for this analysis. Estimates of the annual recurrent costs and of the discount rate (5%) are the same as in the PAD. Other minor differences between the ICR and PAD analysis are mentioned in the Annex. 54. Table 1: Comparison of key assumptions, PAD and ICR PAD ICR Impact on student Used for scenario analysis, range Used for scenario analysis of effects for students benefitting learning (effect size) of effects for interventions similar from BBT use to project interventions Lowest for ICT Lowest: 0.03 (computer assisted learning of students, without extensive 0.117 (class size) teacher support) Highest: Highest for ICT: (buildings) 0.08 (computer assisted instruction by teachers with extensive training) Effects for students benefitting from new classrooms Classrooms 0.383 (buildings, as in PAD) Annual average gain in 0.38 (in standard deviation units 0.36 (in standard deviation units of tests scores) learning outcomes of tests scores) Estimated additional 0.31 years (for 0.117 effect size) 0.22 years (for 0.08 effect size) years of education 1.02 years (for 0.383 effect size) 0.08 years (for 0.03 effect size) (Calculated from previous two rows) 1.06 years (for 0.38 effect size) Number of student Three estimates of beneficiaries: Actual beneficiaries beneficiaries for C1 +C2: 663,600 C1 Component 1 (C1) and BBT Classrooms: 1,438,535 Component 2 (C2) C1 only: 532,000 New classrooms (counterpart funding): 46,128 C2 only: 131,600 C2 – not used due to possible double counting. C1: 15,200 BBT classrooms x 35 students per classroom C1: 41,100 BBT classrooms x 35 students per room C2: estimates beneficiaries based C1: 961 new classrooms x 48 students per room (Borrower’s ICR) on number of principals + Page 13 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT backbone teachers trained and average size of school Source: Author’s compilation 55. The Net Present Value (NPV) and the Internal Rate of Return (IRR) calculated under these assumptions show that the project had a positive economic impact, but the estimates vary significantly based on the effect size of the ICT intervention (Table 2.) The ICR conducts sensitivity analysis based on different assumed effect sizes of the BBT platform, which contributed the overwhelming share of student beneficiaries in the project. Overall, the net benefits from the project are positive and the IRR is also above the project discount rate of 5%, under all three scenarios. Comparing with the estimates of NPV and IRR in the PAD (for the high beneficiary scenario), the NPV and IRR are significantly higher when assuming a high effect size for the ICT intervention. This is due to the far higher number of beneficiaries from ICT included in the ICR analysis, as well as the inclusion of beneficiaries of classroom construction together with its high effect size. However, the sensitivity analysis shows that the NPV and IRR are highly sensitive to assumptions regarding the effect size of the ICT intervention. For instance, assuming a medium effect size, the IRR falls to 19% and a low effect size yields an IRR of just 9 % 56. In short, the sensitivity analysis shows the extent of benefit of the project would depend on the effectiveness of the use of the BBT platform in improving student learning . The information is not available from this project to reach a definitive conclusion. The highest effect size which is derived from another study assumes a high level of continuous support for teachers throughout the experiment. It is fair to assume that not all teachers who benefitted from the BBT training performed at the highest level of effectiveness in using the equipment and therefore the high effect size scenario is likely to overestimate the benefits. Information on this would have helped to arrive at a more accurate estimate of the economic benefits of the project. 57. Table 2: Project closing NPV and IRR – sensitivity analysis based on effect size of ICT intervention, common effect size for classrooms High effect size of ICT Medium effect size for Low effect size for ICT intervention intervention ICT intervention (0.03 SD) (0.08 SD) (0.05 SD) NPV (USD) 422,077,199 202,855,983 56,708,507 IRR (%) 32% 19% 9% Memo Item from PAD Economic analysis: NPV – 231, 680, 870 USD; IRR – 21% (high beneficiaries scenario) Note: Effect size for classrooms is assumed to be the same at 0.38 S.D across all three scenarios in the ICR. Source: Author’s calculations for ICR estimates. For PAD estimates see Guangdong Compulsory Education Project, Project Appraisal Document. World Bank (2017). 58. Implementation efficiency in reallocating project savings arising from the fall in prices of the BBT equipment (outside of the project control) as well as the efficiency in procurement was significant. This contributed to increasing the number of beneficiaries and project impact. Initial procurement of BBT packages were slower than planned, due to the lengthy discussions on the bidding document (e.g. qualification requirements) and in how many lots/contracts the bidding documents will be separated into for easier management. Following the delay, a BBT- specific implementation support mission was promptly organized to unblock the bottlenecks. 59. Based on the above considerations, efficiency is rated as Substantial. Page 14 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT D. JUSTIFICATION OF OVERALL OUTCOME RATING 60. The overall outcome rating is Highly Satisfactory, based on a High rating for relevance, a High rating for efficacy and Substantial rating for efficiency. E. OTHER OUTCOMES AND IMPACTS (IF ANY) 61. There are no other outcomes and impacts. Gender 62. This project was not gender-tagged because there were no significant differences in participation in school between boys and girls. While gender was not meant to be a focus, it could have been better mainstreamed in the program design and implementation. Support for left-behind students could have also considered gender dimensions. A supervision mission in September 2020 recommended addressing gender in teachers’ capacity building and conducting gender disaggregation. Institutional Strengthening 63. The project did not have a focus on institutional strengthening . However, by introducing new approaches to procuring teacher training services through competitive bidding, the project helped to enhance the provincial education department’s capacity to draw upon an enlarged pool of training providers.. The use of the TEACH tool helped decision-makers enabled them to use a rigorous approach to assess the effectiveness of training and contributed to institutional strengthening. An important avenue of institutional development would have been through the evaluation of the cost-effectiveness of the pilot interventions and using evidence for developing guidance. Mobilizing Private Sector Financing 64. The project was not designed to, nor did it, mobilize private sector financing. Poverty Reduction and Shared Prosperity 65. Higher quality education, targeted towards the disadvantaged, contributes to poverty reduction through improved learning, labor market participation and higher wages. It also promotes inter-generational improvements in welfare, as better educated parents invest in health and education of their children. The higher educational attainment, learning outcomes and enhanced earnings projected in the economic and financial analysis, as well as the project’s focus on the most disadvantaged counties and groups of children is expected to contribute to poverty reduction. Other Unintended Outcomes and Impacts 66. None Page 15 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT III. KEY FACTORS AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 67. The Guangdong Compulsory Education Project was the first Bank financed project focused on primary education in China, after a hiatus of about 20 years and had strong support from the provincial government as well as the World Bank. In seeking assistance from the World Bank, the Government of China at the central and provincial levels wished to learn from the Bank’s expertise in addressing the priorities that had been set out in its program. The provincial government expressed willingness to try out a number of innovative pilots. From the point of view of the Bank, the project was important to introduce evidence-based approaches in an upper middle-income country by leveraging its global experience. B. KEY FACTORS DURING IMPLEMENTATION Factors subject to the control of government/implementing entities 68. Shortages of staff and high turnover and government rules preventing the hiring of consultants to perform responsibilities of government officials affected project implementation in the first few years. This was addressed by the Guangdong Department of Education by late 2021 by appointing a new project director and new deputy directors, and staff working on procurement, safeguards, financial management, procurement and monitoring and evaluation, who stayed on until project closing. A consulting firm was hired in early 2022 to support the Provincial Project Monitoring Office (PPMO) and project counties, which had varying implementation capacity. Factors subject to World Bank control 69. Staff turnover at the World Bank affected continuity of supervision and implementation support in the initial phases. The project had four Task Team Leaders in five and a half years of implementation, and it is possible that, as a result, some project components were not as rigorously followed as others. However, the principal Bank consultant was on board from project preparation to completion, except for one year, and provided technical and hands on support to the project implementation. Factors outside the control of government /implementing agencies 70. The COVID-19 pandemic was an unforeseen and once in a century cataclysmic event which affected all aspects of Chinese society and the economy. Direct effects on school education resulted from the lockdown of schools, government offices and all establishments across China in the immediate aftermath of the outbreak of the pandemic in Wuhan province (January-February 2020). However, Chinese authorities were able to quickly make the shift to online learning, thanks to extensive prior investment in connectivity and use of ICT in education. In Guangdong province, face to face classes were resumed in a matter of weeks, by adopting social distancing and other measures, Page 16 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT with localized school closures based on the spread of the infection. Learning was therefore not as affected as in many other developing countries, as schools and education authorities quickly adapted to ensuring continuity of learning. Localized travel restrictions which prevented travel across counties did affect several of the project interventions, including the teacher training programs and school partnership pilot program, in which previously envisaged face to face visits or events had to be replaced by online interactions. This could have potentially affected the efficacy of these impacts. Overseas study visits were cancelled and converted into national trainings. Supply chain disruptions led to rise in prices of construction materials and labor, which affected the construction activities. The provincial government offset this by raising budget allocations for the project. The COVID-19 pandemic also affected the Bank’s supervision during 2020, when travel was not feasible. The Bank team adjusted to this by organizing virtual implementation support missions. 71. A positive factor contributing to the success of the project was the secular fall in prices of key technology items, in particular of the 80-inch flat screens that were used as part of the BBT teaching platform. The price fell by almost 60 percent between appraisal and during implementation.10 The rapid development of local Chinese firms with the capacity to supply and service the equipment, even in remote counties, facilitated the implementation of one of the project’s key components. In parallel, the development of software and educational resources in multi-media format at relatively low cost meant that the teaching platform could be easily used by teachers. 72. On the other hand, the implementation of the mobile laboratory pilot suffered from the fact that only one company in China produced such laboratories for primary and junior middle schools . With insufficient experience of this single firm in supplying mobile labs and after sales services, the lack of capacity at the county level to operate the labs and uncertainty about future demand given the pilot nature of the intervention, the original approach of purchasing 16 mobile labs was considered too risky. Instead, the Bank and the government agreed to procure non-consulting services from the company to deliver science teaching in selected schools in two counties on a “roaming” (pre-scheduled) basis. The lack of a competitive market, and lack of experience of other suppliers for mobile labs, therefore precluded the effective implementation of this pilot. 73. During project implementation, the Government of China introduced the far reaching “double reduction policy”, to eliminate private tuition and homework, lower the academic burden on students and foster non-academic pursuits . There is insufficient evidence about how this policy affected the behavior of schools and teachers. According to the PPMO, it had a beneficial effect on the project, in that there was a greater responsibility on schools to improve learning and support the all-round development of children, instead of relying on students having additional tutoring and studying long hours at home. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME 10The BBT unit was costed at 25,000 CNY, while in the end the average price of the 5 procurements came to 10,900 CNY per unit (approximately 3,900 USD vs 1,700 USD). Page 17 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 74. The M&E framework consisted of a set of quantitative indicators listed in the Results Framework for both outcome and intermediate indicators, as well as a set of evaluation studies related to the pilots. The PAD presented the system of collecting, verifying and analyzing data, including the implementation arrangements and designation of responsibilities. Much of the data consisted of administrative data which were collected from the schools and project counties and aggregated upwards. Project authorities were required to maintain the original data to allow for spot checks by supervisors as well as Bank missions. An innovative aspect of the M&E design was the use of classroom observation data collected using an internationally validated tool (TEACH), to assess the effectiveness of teacher training, which constituted a significant part of the project’s outcomes. By using data collected in a systematic manner from observations of how teachers teach in classrooms, using a randomly selected sample of trained and untrained teachers, the project went beyond traditional measures of teachers’ self-reports of training effectiveness (which were also collected), that had been the main method in the province earlier. 75. Compared to the efforts made to rigorous assess the effectiveness of teacher training, the evaluation of the four pilot activities was not systematically designed. The PAD outlined that the pilots were to be evaluated especially looking at the impact on equity and quality, efficiency and cost-effectiveness. Suitable indicators were not devised to track progress of each pilot, nor was the approach to their evaluation outlined.. M&E Implementation 76. The Bank provided technical support to strengthen the implementation of the M&E framework, with varying degrees of success. In the third implementation support mission the Bank team helped to define the calculation of each indicator, the source of information and the periodicity of collection. Administrative data were used to check progress in the construction of classrooms and dormitories. The Bank team also helped to adapt the original TEACH tool, which was originally designed only for primary teachers, for use in secondary classrooms and also helped the PPMO to analyze the data and prepare a report, which contributed to building institutional capacity. However, implementation was delayed, partly due to the frequent change in Bank TTLs and other delays in project implementation on the government side in the initial years. The original design of comparing the improvement in teaching quality from the baseline for trained and untrained teachers could not be adhered to, due to the attrition of the sample (through transfers and departures). This constituted a shortcoming in the implementation of the TEACH model and meant that the baseline survey results could not be used. The Bank team was able to suggest an alternative approach, using the comparison of trained and untrained teachers at one point of time towards the end of the project, which provided evidence about the effectiveness of the training program. The classroom observation instrument was not aligned with the content of the trainings developed for backbone teachers and teachers using BBT. This may also explain why the tool may not have captured the effectiveness of training of backbone teachers, as mentioned in the section on Efficacy of the PDO. 77. Training of teachers on ICT was also evaluated through additional methods . Interviews and questionnaires to teachers and students were administered after training institutions to feed into the next round of procurement and training. From late 2021 to early 2022, an evaluation of the BBT was carried out by Guangzhou University. 11 11 This included debriefings from project leaders and interviews of more than 120 principals and 248 primary and secondary subject teachers, Page 18 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 78. The PPMO’s draft proposal for evaluating two pilots (for “left-behind children” and inclusive education) were presented to the Bank mission in December 2018, which noted significant shortcomings .12 The Bank’s key concern was that it was not clear how the evaluation of the pilots would feed into the policy paper, which was a key intermediate results indicator, as the evaluations were expected to capture the impact, efficiency, cost- effectiveness of the pilots and produce recommendations for future policy making. Although the revised plans were to be submitted by mid-January, there is no evidence of further discussions or technical support being provided to the PPMO by the Bank to improve the quality of these evaluations and for the other pilots in subsequent implementation support missions. M&E Utilization 79. The PPMO and the County Project Management Offices (CPMO) used the key quantitative indicators on construction, delivery of the BBT platforms and teacher training to monitor project activities . The feedback from teacher training programs helped institutions delivering the programs to make improvements on the margin to their program. The TEACH tool was seen as a useful supplement to traditional methods of monitoring classroom teaching by principals and subject teachers, but delays in implementing the study prevented the midline results being used to adjust the content of training programs. Justification of Overall Rating of Quality of M&E 80. The project’s M&E system is rated as Substantial. It reflected most of the links in the results chain and was mostly implemented in a robust manner. One weakness related to the lack of attention to the pilot activities. Another weakness was related to the implementation of the TEACH survey, which was expected to generate evidence about the effectiveness of training programs. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 81. Environmental Safeguards: The project was classified as category B and triggered the OP4.01 (Environmental Assessment). The finalized Environmental Management Plan was included in the bidding documents and corresponding contracts. To enhance the project capacity to implement Environmental Management Plans, the PPMO set up a group of experts from universities, design units and environmental consulting institutions to provide technical assistance for preparation, implementation and inspection of environmental impact mitigation measures. Environmental management checklists were regularly filled by contractors and project schools. The environmental management system has been well established, environmental training activities have been organized regularly, the mitigation measures in the EMP have been generally well implemented by the project participants. For the mobile lab pilots, safety management before and after class, equipment safety inspection and electricity safety check of vehicle was done. The PPMO hired additional experts to advise on the revision of the equipment and chemicals, and also a professional company to ensure that solid waste produced by the labs shall be properly disposed, upon Bank team’s recommendation. classroom observations of 12 teachers (mainly between grade 5 and grade 9), and interviews of management teams and students. 8,567 teachers, 19,352 students, and 17,340 parents responded to the questionnaires. 12 Aide-Memoire, Implementation Support Mission, December 03-07, 2018 Page 19 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 82. Social Safeguards: All planned classrooms and teachers’ accommodation were to be built on existing premises of the project schools. The CPMOs followed the requirements of the prepared Resettlement Policy Framework. Although initially 31 households were to be resettled due to the construction, eventually only 16 households needed relocation (as a result of death, transfer etc.) based on the resettlement survey carried out by Sun Yat-sen University. The project did not trigger OP 4.10, as the number of ethnic minority students was just 0.5% of total enrollment, and they were fluent in Mandarin and the local dialect, well integrated with the majority Han community and of similar socio- economic development status. Social assessment was carried out and the public was consulted on education, including for students from vulnerable groups such as left-behind children, students with disabilities and low-income families. Establishing a fair and transparent allocation system of teachers’ dormitories was an area recommended through the assessment, but this was not clearly followed up in the implementation support missions. A Grievance Redressal Mechanism was set up, but no grievances were received. 83. Procurement and Financial Management: Towards the beginning of the project, there were some delays in the procurement and financial management, including delayed payments to service providers and delayed financial reports. These were mainly due to the insufficient number of project personnel, as well as personnel changes within the PPMO and other provincial departments.13 Implementation of Component 3 was delayed. The Bank provided significant technical support on the procurement of the BBT platform in every implementtion support mission as well as in between missions. On the government funding side, the construction implementation and disbursement was slow to pick up in some counties, but in the end the project was able to close with a five-month extension to allow for completion of construction related activities in a very few number of counties. Actions have been taken timely to address issues identified by auditors and through implementation support missions. C. BANK PERFORMANCE Quality at Entry 84. The Bank brought high quality technical resources and managerial support at the time of preparation. The project identification mission was led by the Education Sector Manager for the East Asia region, signaling the high importance that the Bank gave to the first basic education project in China in over two decades. The Bank helped to craft the project design basing itself on the policy priorities and suggested programs of the Guangdong Provincial Education Department. The Bank’s contribution in developing the logic of the project through the results chain, in separating out direct project interventions and pilot programs that could be scaled up based on systematic evaluations, and the design of a robust M&E system to assess progress towards intermediate and final results are apparent in the PAD. The Bank also decided strategically to focus on quality improvement activities, leaving construction of classrooms and dormitories to counterpart funding. 13The Aide Memoire from June 2020 mentions that the financial staff of PPMO had been changed three times since project effectiveness and the impact is felt for example in the fact that the required interim financial reporting was not submitted timely and the data was missing in the progress report. Page 20 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 85. However, the Bank did not pay adequate attention to developing an evaluation framework for the pilot activities. Given their significance as innovations, the Bank could have incorporated the design of these evaluations at the time of project preparation, or ensured that adequate technical expertise and support was provided through project funding during implementation. Quality of Supervision 86. During implementation, the Bank continued to provide technical support and feedback, but the quality varied across components. The technical support provided for the procurement of the BBT was exemplary, which partly contributed to efficacy of project outcomes related to “ICT use in project classrooms, through appropriate technical specifications of the product and considerable project savings. Each implementation support mission included an ICT specialist from the Bank’s country office, who was familiar with the local market, procurement of ICT equipment and use of the technology, who conducted spot checks and who provided detailed feedback as recorded in the aide-memoires. This was also the case for the environmental safeguards, where the Bank environmental specialist visited the constructed sites and made recommendations for example on air quality monitoring. Another exemplary area of support was in helping the provincial government team to use the TEACH tool and analyze its results, as well as to rectify the problems with the baseline survey. The Bank team also reviewed the content of the training programs and on the selection of the training institutions and teachers. The Bank team proactively restructured the project in 2022, to raise the targets for various indicators, as project savings and implementation effectiveness of certain components improved. 87. The Bank’s technical support for the four pilot initiatives was less than satisfactory during implementation . Given the importance of these initiatives for the overall outcome of the project, to reduce disparities and inequality, they were not systematically reviewed in the supervision missions. In particular, after the December 2018 mission which highlighted the Bank’s concern about the quality of the plans for the pilots and their evaluation, the Bank’s aide-memoires do not report regularly on the progress of the pilots, their content or the method of evaluation. The Bank implementation support teams do not seem to have included the appropriate technical experts for these activities. Following an updated market assessment, the Bank recommended an open, competitive bidding process for remaining 14 mobile labs instead of the direct contracting used for the first two mobile labs. This process could not be completed before project closing. This activity was thus cancelled at the MTR in 2022. Earlier discussions on such a topic in order to find a solution may have allowed for finding a solution for an innovative program to enhance science learning for children in hard-to-serve areas . Justification of Overall Rating of Bank Performance 88. Overall, Bank performance is rated as Moderately Satisfactory, with a satisfactory rating for quality of entry and moderately satisfactory rating for quality of supervision. D. RISK TO DEVELOPMENT OUTCOME 89. There are no foreseeable risks to the project’s development outcomes. The improvement in learning facilities through ICT equipment in classrooms is expected to continue being used as additional learning resources become available on the platform. Guangdong province plans to scale up use of technology and has benefitted from the project’s experience. The improvement in teaching quality as a result of teacher training provided under the project will Page 21 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT continue to benefit several cohorts of students. The training approach of selecting providers competitively and evaluating the success of the program will be sustained. V. LESSONS AND RECOMMENDATIONS 90. Effective implementation of ICT in education requires tools and technology that align closely with curriculum and what teachers are expected to teach, help them in their daily tasks and provide additional resources to engage students. Further, training of teachers needs to be closely linked to the use of the technology in their classroom environment and continuous technical and professional support is required to encourage and improve on the use of the technology. 91. In deploying educational technology, providing continuous and detailed procurement support, including advice on appropriate technical specifications and writing Terms of Reference that can help to lower costs, ensure quality and minimize risks of vendor lock-in. Technology that is available in the local market helps improve sustainability and provide local technical support, as is shown by the contrasting experience of the BBT and mobile labs. A detailed analysis of the local market and trends should be part of the appraisal. These aspects will become more important as more advanced technological applications, such as “personalized learning” and AI get deployed in education settings. 92. The effectiveness of different models of teacher training and support (in this case, training of BBT teachers, backbone teachers, principals and multi-subject teachers, as well as school partnerships) requires multiple approaches to evaluation, and not just a single impact evaluation focused on a few dimensions of teacher behavior in classrooms, in order to help influence policy on teacher training. 93. Independent evaluations of teacher training using internationally accepted instruments and robust techniques can complement existing systems of evaluation, if the Bank provides sufficient technical assistance in adapting the instrument to the local context and in building local capacity. In using instruments, such as TEACH, however, use of a composite score may affect interpretability and the value to policy makers (for instance, what a specific increase in the score means for teacher effectiveness). It may be better to focus on improvements in specific pedagogical practices. Piloting of the instrument in the local context is required to understand the baseline distribution of pedagogical practices and set meaningful targets. 94. A project with an important component on pilot interventions which are expected to be scaled up should include strong technical assistance for the design of the pilots, for developing an evaluation framework as an integral part of the design, and for building capacity to use these evaluations for policy making. Monitoring and evaluation of pilots should be provided adequate resources within the project. Page 22 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS @#&OPS~Doctype~OPS^dynamics@icrresultframework#doctemplate A. RESULTS FRAMEWORK PDO Indicators by Outcomes Improve learning facilities Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year Share of project classrooms where 0.00 Jul/2017 70.00 Nov/2023 100.00 Nov/2023 ICT is used (Percentage) Improve teaching quality Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year The proportion of trained teachers 0.00 Jul/2017 70.00 Nov/2023 96.88 Jun/2024 whose teaching practices meet a Comments on achieving targets Endling Teach Data submitted in June 2024 minimum threshold in targeted competencies (Non-cumulative) (Percentage) The proportion of trained 0.00 Jul/2017 70.00 Nov/2023 100.00 Jun/2024 backbone teachers whose Comments on achieving targets Endline TEACH data submitted in June 2024 teaching practices meet a minimum threshold in targeted competencies (Non- cumulative) (Percentage) The proportion of teachers 0.00 Jul/2017 70.00 Jun/2023 92.86 Jun/2024 trained in ICT whose teaching Comments on achieving targets Endline TEACH data submitted in June 2024 practices meet a minimum threshold in targeted Page 23 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT competencies (Non- cumulative) (Percentage) Intermediate Indicators by Components Component 1: Improve School Equipment and Facilities Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year 0.00 Jul/2017 1,990,000.00 Nov/2023 2,356,926.00 Nov/2023 Direct project beneficiaries Comments on achieving targets The original End Target was 554,800.00 but was increased to 1,990,000.00 at the Mid-Term (Number) Review. Total number of teachers 0.00 Jul/2017 60,000.00 Nov/2023 109,031.00 Nov/2023 (Number) Students benefiting from 0.00 Jul/2017 1,730,000.00 Nov/2023 2,247,895.00 Nov/2023 direct interventions to enhance learning (Number) 0.00 Jul/2017 39,350.00 Nov/2023 41,101.00 Nov/2023 Total number of classrooms with ICT installed (Number) Comments on achieving targets ICT equipment was installed in 2900 additional classrooms during the extension period with savings from mobile lab cancellations. Component 2: Strengthen the Training and Assessment System for Teachers and Principals Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year Total number of training courses 0.00 Jul/2017 14.00 Nov/2023 14.00 Nov/2023 developed (Number) Total number of training courses 0.00 Jul/2017 14.00 Nov/2023 14.00 Nov/2023 approved by the expert panel (Number) Teachers trained (Number) 0.00 Jul/2017 51,300.00 Jun/2023 70,307.00 Nov/2023 Page 24 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Comments on achieving targets 500 additional backbone teachers were trained in the extension period. End target for this indicator was increased to 51,300 trained teachers at the last ISM. Total number of targeted 0.00 Jul/2017 1,300.00 Nov/2023 1,407.00 Nov/2023 backbone teachers and principals trained (Number) Total number of targeted ICT 0.00 Jul/2017 60,000.00 Nov/2023 68,900.00 Nov/2023 teachers trained (Number) Participant satisfaction with quality of 0.00 Jul/2017 75.00 Nov/2023 85.80 Nov/2023 training content (Non-cumulative) (Percentage) Participant satisfaction with 0.00 Jul/2017 75.00 Nov/2023 85.50 Nov/2023 quality of backbone training content (Non-cumulative) (Percentage) Participant satisfaction with 0.00 Jul/2017 75.00 Nov/2023 86.30 Nov/2023 quality of ICT training content (Non-cumulative) (Percentage) Participant satisfaction with 0.00 Jul/2017 75.00 Nov/2023 85.70 Nov/2023 quality of principal training content (Non-cumulative) (Percentage) Proportion of trained teachers who 0.00 Jul/2017 75.00 Nov/2023 100.00 Nov/2023 pass the knowledge test after training (Non-cumulative) (Percentage) Proportion of trained 0.00 Jul/2017 75.00 Nov/2023 100.00 Nov/2023 backbone teachers who pass the knowledge test after training (Percentage) Proportion of trained ICT 0.00 Jul/2017 75.00 Nov/2023 100.00 Nov/2023 teachers who pass the knowledge test after training (Percentage) Component 3: Pilot Reforms to Improve Equity and Quality of Education Page 25 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year Total number of activity and resource 0.00 Jul/2017 200.00 Nov/2023 724.00 Nov/2023 room teachers trained (Number) Total number of teachers 0.00 Jul/2017 100.00 Nov/2023 534.00 Nov/2023 trained who will manage activity rooms (Number) Total number of teachers 0.00 Jul/2017 100.00 Nov/2023 190.00 Nov/2023 trained who will work in resource rooms (Number) Policy paper capturing 0.00 Jul/2017 1.00 Nov/2023 1.00 Jun/2024 recommendations based on pilots Comments on achieving targets DoE shared updated policy directions capturing recommendations based on pilots with the ICR team. and their evaluations (Number) Guidance based on policy paper 0.00 Jul/2017 1.00 Nov/2023 1.00 Nov/2023 issued by Department of Education Comments on achieving targets Policy Guidance was issued by the DoE in June 2023 on the scaling up of the pilot (Number) interventions under the Department's future reform program. Page 26 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT B. KEY OUTPUTS Objective/Outcome 1 Improve learning facilities Outcome Indicators 1. Share of project classrooms where ICT is used (percentage) 1. Direct Project Beneficiaries (number) 2. Total number of teachers (number) 3. Proportion of female teachers (percentage) Intermediate Results Indicators 4. Students benefiting from direct interventions to enhance learning (number) 5. Total number of classrooms with ICT installed (number) 1. 41,101 classrooms equipped with ICT equipment Key Outputs by Component 2. 961 classrooms constructed (linked to the achievement of the Objective/Outcome 1) 3. 5163 teachers’ dormitories constructed Objective/Outcome 2 Improve teaching quality 1. The proportion of trained teachers whose teaching practices meet a minimum threshold in targeted competencies 2. The proportion of trained backbone teachers whose teaching Outcome Indicators practices meet a minimum threshold in targeted competencies 3. The proportion of teachers trained in ICT whose teaching practices meet a minimum threshold in targeted competencies 1. Total number of training courses developed (number) 2. Total number of training courses approved by the expert panel (number) Intermediate Results Indicators 3. Total number of teachers trained (number) 4. Total number of backbone teachers and principals trained (number) 5. Total number of targeted ICT teachers trained (number) Page 27 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 6. Participant satisfaction with quality of training content (percentage) 7. Participant satisfaction of quality of backbone training content (percentage) 8. Participant satisfaction of quality of ICT training content (percentage) 9. Participant satisfaction of quality of principal training content (percentage) 10. Proportion of trained teachers who pass the knowledge test after training (number) 11. Proportion of trained backbone teachers who pass the knowledge test after training (number) 12. Proportion of trained ICT teachers who pass the knowledge test after training (number) 13. Total number of activity and resource room teachers trained (number) 14. Total number of teachers trained who will manage activity rooms (number) 15. Total number of teachers trained who will work in resource rooms (number) 16. Policy paper capturing recommendations based on pilots and their evaluations (number) 17. Guidance based on policy paper issued by Department of Education (number) 1. 14 training courses developed 2. 68,843 teachers trained on ICT/BBT Key Outputs by Component 3. 532 principals trained (linked to the achievement of the Objective/Outcome 2) 4. 859 backbone teachers trained 5. 2,984 teachers trained in-service on multisubject teaching Page 28 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 6. 2,394 teachers trained on multisubject teaching through pre- service training (not yet completed as first graduates planned for 2024) 7. 100 activity rooms for left-behind children managed 8. 50 resource rooms for children with disabilities managed 9. 1,000 schools participating in the school partnership activity 10. 1 policy paper based on pilots developed Page 29 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Salman Asim Team Leader Yi Dong Financial Management Specialist Jianjun Guo Procurement Specialist Yiren Feng Environmental Specialist Ross James Butler Social Specialist Meixiang Zhou Social Specialist Yan Zhang Procurement Team Tianxiu Kang Team Member Sajitha Bashir Team Member Aya Kibesaki Team Member Jose Martin Moreno Vigo Team Member Ying Li Team Member Kenglin Lai Team Member Dongqi Qian Team Member Tao Su Team Member Lingzhi Xu Team Member @#&OPS~Doctype~OPS^dynamics@icrannexstafftime#doctemplate B. STAFF TIME & COST Staff Time & Cost Stage of Project Cycle No. of Staff Weeks US$ (including travel and consultant costs) Preparation FY15 20.175 161,797.19 Page 30 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT FY16 39.039 239,898.69 FY17 27.093 265,177.82 FY18 11.213 120,645.30 FY19 0.000 384.59 Total 97.52 787,903.59 Supervision/ICR FY18 24.796 145,334.98 FY19 18.665 126,107.16 FY20 10.550 84,250.40 FY21 13.237 95,727.42 FY22 10.962 94,790.38 FY23 12.425 113,237.63 FY24 8.225 133,642.03 Total 98.86 793,090.00 Page 31 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Page 32 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT ANNEX 3. PROJECT COST BY COMPONENT Amount at Actual at Percentage of Amount at Approval Actual at Project Closing Percentage of Approval Approval Guangdong Project Closing Guangdong Approval IBRD Guangdong Components IBRD (US$M) (US$M) IBRD (US$M) (US$M) (%) (%) Component 1: Improve School 67.2 100.8 62.7 98.1 98.6 85 Equipment and Facilities Component 2: Strengthen the Training and 12.4 45.2 16.9 37.7 140.7 77.2 Assessment System for Teachers and Principals Component 3: Pilot Reforms to Improve 39.8 0.0 29.1 0.0 98.7 - Equity and Quality of Education Component 4: Project Management, 0.3 2.6 0.5 0.5 56.5 23.5 Monitoring and Evaluation Front-end fees 0.3 0.31 0.3 - 100 - Interest during - 10.8 - 17.1 - 164.8 constructions Total 120.00 159.4 109.414 153.3 103.1 86.8 “% of approval” is the spent amount vis-à-vis the revised allocation after the MTR/10 th supervision mission. If vis-à-vis the original amount at approval, the total percentage of IBRD would be 107.2% achieved (and not 103.1%). Note: Exchange Rate was 6.89 CNY for 1 USD in May 2017 according to the PAD. The rate used at the MTR was 6.35 CNY, and the rate used for the current table is 7.18 CNY.15 14 The Bank only disbursed $118.75 million as of late March 2024, and the differences in actual spent is coming from the better exchange rate at the time of utilization of the funds. 15 Rate from World Bank Client Connection as of January 31, 2024. Page 33 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT ANNEX 4. EFFICIENCY ANALYSIS 1. This annex documents how the economic analysis for the ICR of the Guangdong Compulsory Education Project was conducted. It follows the methodology adopted in the Project Appraisal Document but uses actual estimates of annual project costs and beneficiaries of key project interventions, as well as improved assumptions/more recent estimates of effects on learning to calculate project benefits. The changes in parameters are summarized in Table 1 of the main text and not repeated here. While the best effort has been made to update the analysis, it should be remembered that the methodology relies on a long chain of assumptions as stated in the PAD. For a proper sensitivity analysis, various assumptions in the chain would need to be changed. In the absence of data to make more robust assumptions, we have adopted the approach of varying the estimated effect of the most significant project intervention which had the greatest numbers of beneficiaries (the BBT teaching platform), to conduct the sensitivity analysis. 2. The annex is organized as follows. The first section reports the observed project costs. The ensuring sections lay out the expected learning impacts, the size of potential beneficiaries and estimate the impact on their future income. The final section integrates the costs and benefits estimated previously to calculate the cost-benefit analysis of the project, specifically, the present value and the internal return rate.16 Observed project costs 3. Table 1 shows the nominal and discounted investment and recurrent project costs. The total project cost is USD 279 million, of which USD 120 million correspond to the IBRD loan, and USD 159 million are funds provided by the Government of China (hereafter, the Government). Nominal costs are discounted at 5 percent, the customary discount rate used by the IMF and the World Bank. All cost projections are made until 2030, corresponding to the last year when the last cohort of students who will benefit directly from the project activities is expected to graduate from compulsory education. 4. Actual IBRD financed project investment costs are derived from the IBRD loan disbursement amounts reported in the World Bank ISRs and are reported in nominal US dollars of 2022. The annual amount of disbursement is calculated as the difference of disbursements between two different points in time. As the Government’s annual disbursement amounts are not available by the time this analysis was completed, it is assumed the disbursements follow the same patterns of IBRD. The overall investment cost is the sum of IBRD and Government annual disbursements. 5. Recurrent costs are derived from operation and maintenance (O&M) costs for infrastructure and equipment and incremental recurrent costs from ongoing teacher training. Given that no specific O&M costs are provided, this analysis uses the same costs reported as part of the PAD analysis and included in the Excel spreadsheet provided by the Bank’s project appraisal team. Specifically, this implies the following costs: 3 million RMB in years 3 and 4 of the project and the same values for the until 2030. Ongoing multi-subject teacher training are valued at 16,000 RMB for years per teacher trained between 2021 and 2024 for 5-year training, and 2021-2025 for 6-year training respectively. 6. Table 1. Nominal and Discounted costs (USD) of the Guangdong Compulsory Education Project 16 The details of the analysis are provided in an Excel spreadsheet Page 34 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Year Investment Investment Recurrent Recurrent Total costs costs costs costs costs (discounted) (nominal, (discounted at (nominal, (discounted at (USD) USD) 5%) (USD) USD) 5%) (USD) 2018 777,000 777,000 777,000 2019 11,519,000 10,970,476 10,970,476 2020 71,584,000 64,928,798 64,928,798 2021 71,850,000 62,066,731 62,066,731 2022 70,798,000 58,245,690 58,245,690 2023 52,472,000 41,113,185 41,113,185 2024 6,937,591 5,176,937 5,176,937 2025 5,544,267 3,940,207 3,940,207 2026 3,918,723 2,652,346 2,652,346 2027 2,293,179 1,478,203 1,478,203 2028 667,634 409,870 409,870 2029 435,414 254,577 254,577 2030 435,414 242,455 242,455 Total (USD) 279,000,000 238,101,881 20,232,221 14,154,595 252,256,475 Project benefits: estimated impact on student learning and years of schooling Estimated Impact on student learning outcomes 7. Table 2 summarizes the expected impact of different type of interventions to improve student learning. The table is separated in two panel. The first panel shows the impacts identified in the PAD, which corresponds to studies that were available before 2014, which guided the estimation of the analysis of benefits reported in the PAD. Panel B shows three more recent research papers and four estimates of interventions conducted in China and their impact on different learning outcomes. 8. The PAD used the minimum effect size of 0.117 (for class size) and maximum effect size of 0.383 (for buildings) to arrive at “a range of positive impact on student learning” for interventions that were considered similar to those in this project. This range was used to estimate the range of impacts on years of education in the following manner. For the students who benefitted from the entirely of the project during their nine years of education, the higher effect size/years of education was used, while for those who benefitted less than nine years, the lower effect size/years of education was used. By using the high effect size of buildings to apply to all students who benefitted from 9 years of education – the rationale for this is not clear – there is a likelihood that the benefits from the project were overstated. The analysis at appraisal did not try to estimate the benefits deriving from the specific project interventions that were planned. Page 35 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 9. The analysis used in the ICR follows a somewhat different approach which is more suited to the actual project interventions. Specifically, since the BBT intervention had the greatest number of beneficiaries, and received substantial reallocations after the project’s mid-term review, this analysis uses more up to date estimates of use of ICT specifically in China. It is important to remember that “ICT in education” can mean very different things, and therefore effect sizes reported in different studies may not relate to the same type of intervention. In Table 2, Bianchi, Lu and Song (2022) relates to an intervention (called “computer aided learning”) in rural China between 2004 and 2007, which installed satellite dishes, computer rooms and other multimedia equipment in rural schools with poorly qualified teachers, and provided recorded special lectures by expert teachers in Beijing, which were used in the target schools via the internet or CDs. The paper looked at effects on different cohorts of children using the China Family Panel Survey of 2010 and 2014. Both due to the fact that the data are from an early period, and that the intervention is substantially different from that in this project, this was not used. The paper by Clark et al.(2021) reports effects based on analysis of just 3 schools and for online education during COVID. 10. The analysis of computer assisted instruction/learning, using a RCT, reported in Bai et al (2016) approximates most closely to the BBT intervention in this project. Further, the study provides estimates for two types of interventions (with more intensive teacher training and engagement versus limited teacher support), and this provides a more realistic range for the impact of the BBT. The range of the estimated effect size is 0.03-0.08. In the study, in fact, the effect of the second intervention is not significant. Since the BBT involved a large number of teachers, it is reasonable to expect that there would be a range of effects of the intervention. 11. The other major intervention of the project was the classroom construction (financed by the Government) and in this analysis, the effect size of 0.383 is used as in in the PAD, as there are no updated estimates. 12. Further, the effect sizes for these two interventions are used separately on the estimated number of beneficiaries from the respective intervention, in order to provide a more accurate estimate of the benefits, as compared to the analysis in the PAD (see above). Page 36 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT TABLE 2. IMPACT OF EDUCATION INTERVENTIONS ON STUDENT TEST SCORES (IN LOW- AND MIDDLE-INCOME COUNTRIES AND CHINA) Category of Intervention (as Effect Size Authors (Year) Context defined by the authors) (SD) Panel A: Papers published by the time of PAD release Krishnaratne, White and Buildings 0.383** Low- and middle-income Carpenter (2013) countries Materials 0.160*** Teacher resources 0.284** Low- and middle-income countries McEwan (2014) Computers or technology 0.150*** Teacher training 0.123*** Class size or composition 0.117** Lai et al. (2015) Computer assisted learning 0.15*** China (Beijing) Lai et al. (2012) Computer assisted learning 0.20*** China (Qinghai) Mo et al. (2013) Computer assisted learning 0.16*** China (Shaanxi) Panel B: Papers published post PAD release Clark et al. (2021) Online Education during 0.22*** 9th graders from three COVID Chinese Middle Schools in a county in Baise City. Bianchi, Lu, and Song Computer aided learning – math: China (rural areas) (2022) use of recorded lectures from 0.18*** expert teachers and deployed reading: in rural schools via 0.23*** internet/CDs Bai et al. (2016) Computer assisted instruction CAI English: China (Haidong prefecture) (CAI), emphasis in provision of 0.08** equipment and more teacher engagement Computer assisted learning CAL English: (CAL), emphasizes in student 0.03 self-learning using software and reduced teacher engagement Page 37 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 13. Following the analysis in the PAD, the next step was to translate the learning improvements into increased earnings. This requires a few intermediate assumptions. First, the assumed effect sizes in standard deviation units are converted to additional years of schooling. Students in the education system are expected to show progress by improving their academic achievement measured using standardized tests. We use an estimate of the annual gain among students, 0.36 standard deviations which represents the expected amount of standard deviations in test scores that each year increases among students in the 50th percentile between grades 1 and 9 (Scammacca, Fall, and Roberts 2015). This is a more recent estimate than in the PAD, although it is similar to the estimate in the PAD. 14. Thus, the estimated impact on years of schooling is the additional gain that goes beyond the expected interannual gain. This is calculated as × . As noted in the PAD, the . estimate corresponds to students in the United States education system and they may be overestimating the annual growth which in turns would increase the effect size of the project expressed as years of education. 15. The results of the calculations for different effect sizes assumed for the two key project interventions under Component 1 and Component 2 (the BBT and classroom construction) are reported in Table 3. For the BBT, as indicated earlier, three possible effect sizes are assumed, ranging from 0.03 to 0.08. For classrooms, the effect size of 0.383 is used. TABLE 3. CONVERSION OF EFFECTS ON LEARNING OUTCOMES AND SCHOOLING Effect of ICT intervention Effect of (BBT) classroom constructi Scenarios on Treatment vs Control Note 1 2 3 4 Improvement on Effect of Computer 0.08 0.05 0.03 achievement in SD of assisted instruction 1 student test scores due to (CAI) interventions to enhance learning using ICT Improvement on Effect of classroom 0.38 achievement in SD of construction 2 student test scores due to interventions to increase classroom availability Improvement on Effect transformed 3% 2% 1% 15% achievement in percentage into percentage - 3 derived assuming normal distribution Page 38 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Effect size of project in Years of education 0.222 0.139 0.083 1.064 4 years of education Project benefits: impacts on future income 16. The estimate of incremental future earnings resulting from additional years of education is the same as in the PAD. The rate of return to education (due to an additional year of education) of 0.1025 is derived from the meta-analysis by Awaworyi and Mishra (2014; 2018) using China studies. The estimate controls by publication selection bias, moderating variables, and study characteristics. The implicit assumption is that there has been no change in the rate of return to education since project appraisal – for instance, the impact of COVID on earnings, if any, is assumed to have been short- lived. 17. Table 4 show that for gains of 0.222 to 1.064 additional years of schooling with a rate of return to education of 10.25% over an annual per capita income of USD 5098.47 translates into an annual incremental increase of earnings ranging between USD 116.13 (scenario 1 of computer assisted instruction) and USD 555.98.15 (classroom instruction). TABLE 4. PROJECT IMPACTS ON FUTURE EARNINGS Source Unit of Effect of ICT intervention (BBT) Effect of measure classroo Scenarios m construc tion 1 2 3 4 1 Rate of Return -ROR (%) Assumptio Percentag 10.25 n e 2 Per capita annual Datum Nominal 5,098.47 Earnings across 16 USD circa selected Guangdong 2015 counties 3 Additional years of Table 3, Years of 0.222 0.139 0.083 1.064 education Row 4 education 4 Additional Impact (%) on Derived: Percentag 2.28% 1.42% 0.85% 10.90% earnings per additional Row 1 * e year of schooling Row 3 5 Annual incremental Derived: USD 116.13 72.58 43.55 555.98 increase (USD) Row 2 * Row 4 Page 39 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Students benefitting from project interventions and graduating from compulsory education each year. 18. The methodology follows that in the PAD, except for the fact that actual beneficiary numbers for the two main project interventions are used and calculated separately. The beneficiaries are calculated as follows: 19. Students benefitting from “classrooms with ICT (BBT)” under Component 1, who graduate every year. This is derived as 41,101 classrooms X 35 students per classroom (1,438,535) divided by 9 = 159,837. 20. Students benefitting from classroom constructions under Component 1, who graduate every years. 961 classrooms were constructed benefitting a total of 46,128 students (48 student per classrooms). This translates into 5,125 annual beneficiary graduates from compulsory education. 21. Beneficiaries of other activities from Component 2, such as backbone and principal teacher trainings, were excluded to avoid a possible double counting of beneficiaries that are already benefiting from component 1, as it is not possible to identify which schools were exposed to each type of intervention. Further the impact evaluation of teacher training using TEACH showed no significant impact of the backbone teacher training. There was no systematic evaluation of the pilot programs under Component 4 and it is not possible to assess the beneficiaries. While the Government ICR estimates a high number of student beneficiaries as a result of the School Partnership pilot, it is not clear how this is arrived at, and there is no estimate of the possible effect size of this intervention. 22. Following the PAD methodology, the number of beneficiaries with different levels of completed education is estimated. The total number of beneficiaries is broken down using the share of the population in the 16 counties that attained each education level. This disaggregation allows to follow the cohort that benefit during the duration of the project, and allows to adjust the flow to consider that some beneficiaries might continue in the education system after concluding compulsory education, while others will leave the education system and go into the labor market. 23. Table 5 reports on these estimates. The number of beneficiaries at project completion differs significantly from the numbers estimated at appraisal reported in the PAD. In the PAD, under the high scenario, each year 73,733 students were to graduate each year from compulsory education and benefit from project components 1 and 2. In this analysis, the total number of graduate beneficiaries is more than double at 164,962. As shown above, this is largely driven by enormous increase in the number of classrooms with ICT equipment (41,101 classrooms compared to the original target of 15,200 classrooms. TABLE 5. EXPECTED NUMBER OF BENEFICIARIES WITH DIFFERENT LEVELS OF EDUCATION (STUDENTS GRADUATING FROM COMPULSORY EDUCATION EACH YEAR) ICT intervention Classrooms Total (BBT) Total (100%) 159,837 5,125 164,962 Compulsory graduates (35%) 55,943 1,794 57,737 Senior secondary graduates (37%) 59,140 1,896 61,036 Higher education graduates (28%) 44,754 1,435 46,189 Page 40 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Note: Table 5 is an updated version of PAD’s “Table A5.7. Expected number of beneficiaries in a graduating cohort each year”. Cost-Benefit Analysis MONETIZED BENEFITS ARE ESTIMATED SEPARATELY FOR EACH GROUP OF BENEFICIARIES BENEFITTING FROM THE TWO MAIN PROJECT INTERVENTIONS, USING THE ANNUAL INCREMENT IN EARNINGS REPORTED IN TABLE 4 (FOR THE THREE SCENARIOS OF THE EFFECT SIZE OF THE ICT INTERVENTION AND THE EFFECT OF CLASSROOMS). 24. Table 6 reports the discounted costs, discounted benefits and Net Present Value (NPV), for the first scenario of the ICT intervention, where the effect size is assumed to be high (0.08 SD), and adding the monetized benefits from classroom construction. This shows that the project achieves a net present value of USD 422 million, and Table 7 shows the estimated internal return rate (IRR) at 32%. This is significantly higher the IRR estimated at appraisal for the “high beneficiary” scenario (21%). As stated earlier, this is driven by the more than trebling of the classrooms with the BBT platform. 25. However, the estimates are highly sensitive to the effect sizes assumed for the ICT intervention (BBT). As reported in Table 8, when this effect size is assumed to be 0.05, the IRR drops to 19 %. When assumed to be 0.03, the IRR is just 9 %. Both NPVs and IRRs are still positive. TABLE 6. NET PRESENT VALUE (NPV) SUMMARY-ALL IN PRESENT VALUES Cost (5% discount Benefits (5% Benefits - Costs (5% Year rate) discount rate) discount rate) 2018 777,000 0 (777,000) 2019 10,970,476 7,137,254 (3,833,222) 2020 64,928,798 13,594,770 (51,334,028) 2021 62,066,731 19,421,100 (42,645,632) 2022 58,245,690 31,179,453 (27,066,237) 2023 41,113,185 41,773,924 660,739 2024 5,176,937 51,288,696 46,111,759 2025 3,940,207 59,802,574 55,862,367 2026 2,652,346 71,447,156 68,794,810 2027 1,478,203 81,847,124 80,368,920 2028 409,870 91,094,606 90,684,737 2029 254,577 99,275,782 99,021,205 2030 242,455 106,471,235 106,228,780 Totals (and NPV) 252,256,475 674,333,674 422,077,199 TABLE 7. IRR SUMMARY-CASH FLOWS Year Cost Benefits Benefits - Costs Page 41 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 2018 777,000 0 (777,000) 2019 11,519,000 7,494,117 (4,024,883) 2020 71,584,000 14,988,234 (56,595,766) 2021 71,850,000 22,482,351 (49,367,649) 2022 70,798,000 37,898,820 (32,899,180) 2023 52,472,000 53,315,289 843,289 2024 6,937,591 68,731,758 61,794,167 2025 5,544,267 84,148,227 78,603,960 2026 3,918,723 105,559,990 101,641,267 2027 2,293,179 126,971,752 124,678,574 2028 667,634 148,383,515 147,715,881 2029 435,414 169,795,278 169,359,864 2030 435,414 191,207,040 190,771,627 IRR 32% TABLE 8. SENSITIVITY ANALYSIS OF CBA – VARYING EFFECT SIZES OF ICT INTERVENTION ICT Scenario 1 + ICT Scenario 2 + ICT Scenario 3 + Measure Classroom Classroom Classroom construction construction construction Effect Size ICT intervention 0.08 SD 0.05 SD 0.03 SD (BBT) Classroom 0.38 SD 0.38 SD 0.38 SD Construction NPV 422,077,199 202,855,983 56,708,507 IRR 32% 19% 9% References Awaworyi, Sefa, and Vinod Mishra. 2014. “Returns to Educa on in China: A Meta-Analysis.” Discussion Paper 41–14. Discussion Paper. Monash University, Department of Economics. ———. 2018. “Returns to Educa on in China: A Meta-Analysis.” Applied Economics 50 (54): 5903–19. Bai, Yu, Di Mo, Linxiu Zhang, Ma hew Boswell, and Sco Rozelle. 2016. “The Impact of Integra ng ICT with Teaching: Evidence from a Randomized Controlled Trial in Rural Schools in China.” Computers & Educa on 96 (May): 1–14. h ps://doi.org/10.1016/j.compedu.2016.02.005. Bianchi, Nicola, Yi Lu, and Hong Song. 2022. “The Effect of Computer-Assisted Learning on Students’ Long-Term Development.” Journal of Development Economics 158: 102919. Page 42 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Chen, Jie, and Francesco Pastore. 2021. “‘Study Hard and Make Progress Every Day’: Updates on Returns to Educa on in China.” DP 14139. Discussion Paper. IZA. h ps://www.iza.org/publica ons/dp/14139/study-hard-and-make-progress- every-day-updates-on-returns-to-educa on-in-china. Clark, Andrew E., Huifu Nong, Hongjia Zhu, and Rong Zhu. 2021. “Compensa ng for Academic Loss: Online Learning and Student Performance during the COVID-19 Pandemic.” China Economic Review 68 (August): 101629. h ps://doi.org/10.1016/j.chieco.2021.101629. Scammacca, Nancy K., Anna-Mária Fall, and Greg Roberts. 2015. “Benchmarks for Expected Annual Academic Growth for Students in the Bo om Quar le of the Norma ve Distribu on.” Journal of Research on Educa onal Effec veness 8 (3): 366–79. h ps://doi.org/10.1080/19345747.2014.952464. World Bank. 2023. “China: Guangdong Compulsory Educa on Project (P154621). Implementa on Status & Results Report ISR 9.” Appendices APPENDIX 1. ANNUAL GAINS IN EFFECT SIZE FOR NATIONALLY NORMED TESTS FOR BASIC EDUCATION STUDENTS (GRADES 1 TO 9) AT THE 50TH PERCENTILE Reading Math Science Grade (Spring) 50th 50th 50th Average 1–2 1.03 0.99 0.54 0.85 2–3 0.56 0.66 0.56 0.59 3–4 0.38 0.51 0.33 0.41 4–5 0.38 0.46 0.38 0.41 5–6 0.33 0.31 0.3 0.31 6–7 0.21 0.24 0.27 0.24 7–8 0.27 0.29 0.27 0.28 8–9 0.21 0.14 0.26 0.20 Median 0.36 Source: Scammacca, Fall, and Roberts (2015) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS 26. The Guangdong Compulsory Education Project is an important project implemented by Guangdong Province, which aims to broaden the financing channels for compulsory education, to learn from domestic and international Page 43 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT experiences in promoting the balanced development of compulsory education, and to allow the province to promote the balanced, high-quality and standardized development of compulsory education and promote educational equity. It is also the only compulsory education project implemented by the World Bank (Bank) in China. 27. Under the guidance and support of the Bank team, the Guangdong Provincial Development and Reform Commission, and the Department of Finance of Guangdong Province, and through the joint efforts of the Provincial Project Management Office (PPMO), the County Project Management Offices (CPMOs) and the project schools, the project construction has been a success. And it has been welcomed by the Education Bureaus of project counties, and widely welcomed by teachers and students of project schools. 28. The project design is accurate, which effectively addressed the weaknesses in the development of compulsory education. a. Careful selection of project counties. The main principles for selecting project counties are: i) Populous counties in the province with a relatively weak foundation for compulsory education; ii) Counties with low levels of per capita financial security. Based on the above principles, Chaoyang, Wengyuan, Wuhua, Haifeng, Lufeng, Suixi, Lianjiang, Leizhou, Wuchuan, Dianbai, Huazhou, Chao'an, Huilai, Puning, Jiexi and Luoding were selected as the project counties. These counties are the key and difficult areas for promoting balanced, high-quality and standardized development of compulsory education in the province, and are the weaknesses in the development of compulsory education in the province. b. Careful arrangement of project construction contents. The project funding was concentrated on 4 major sub- projects (school expansion and reconstruction, support for special groups, development and share of high- quality educational resources, and teacher development and welfare), based on the national strategic plan (promoting high-quality and balanced development of compulsory education), the project design principles (balanced development, highlighting key areas, reform and innovation, and introducing talents), and the outstanding shortcomings and key links existed in the project counties in promoting the balanced and high- quality standardized development of compulsory education. It focused on addressing the outstanding shortcomings in the counties, such as the insufficient provisions of town school places and education security for special groups, and establishing a mechanism for sharing high-quality educational resources and improving the mechanism for teacher management and guarantee, which are the 2 key links. 29. The project management is good, and all construction tasks have been successfully completed on time. a. Strengthening Project Management.  Improving the project management mechanism. A project leadership group has been established in Department of Education of Guangdong Province (DoE), and DoE set up the PPMO for the project. The counties also established their own project leadership group and set up the CPMO for the project. The project management has been strengthened and the project implementation has been carefully organized.  Strengthening the workforce of the PPMO. More staff has been assigned to work in the PPMO. The staff has been trained. An expert team has been established. And a consulting firm has been hired to provide assistance.  Providing more guidance to CPMOs. The PPMO has paid more visit to the project counties to facilitate the project implementation. b. All construction tasks have been successfully completed on time. The Project launched in January 2018, and was closed successfully on November 30, 2023. Besides the pre-service multi-subject teacher training program is still implementing as planned (the last group of students will graduate in June 2025), all other tasks have Page 44 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT been successfully completed. The expected outcomes were achieved. All tasks financed by the Bank loan have completed settlement. The total amount of disbursement of the Bank loan has reached 117,406,200 US dollars, accounting for 97.84% of the total loan amount. The counterpart funding was allocated on time. The project funding is cost-effective. 30. The project was fruitful and met its expected objectives. a. All the indicators in the Results Framework have been met. The Bank team conducted the 13th Implementing Support Mission in November 2023. As reviewed by the Bank team, all the 3 PDOs and 20 intermediate result indicators have been met. The direct project beneficiaries have reached 2,379,000, exceeding the original target of 554,800. The additional direct project beneficiaries have reached 1,824,200, exceeding 328.80% of the original target. b. The project efficacy is high. The level and quality of compulsory education development in project counties have been improved.  The school places of the compulsory education have been effectively increased. Additional 45,000 school places were provided by the project. Thanks to the achievements, together with the classrooms newly built through other channels financed by the government, the oversized classes have basically been eliminated, which ensures students’ physical and mental health and improves educational and teaching quality.  The level of education informatization has been significantly improved. 41,101 BBT teaching platforms have been equipped in project schools and 68,843 teachers using BBT have been trained to develop their ICT skills, which improved the level of education/teaching informatization and digital application for project schools, and established an effective mechanism for sharing quality education resources.  The teacher quality, competence and ability have been effectively increased. Under the project, two groups of pre-service multi-subject teachers have been trained for the counties, and all of them are qualified rural teachers who are willing to retain in the counties. 73,666 in-service teachers in the counties have been trained in various trainings. All the trainings have effectively improved the teacher’s comprehensive quality, professional competence, and innovative ability.  The teacher’s dormitories helped the teachers to devote themselves wholeheartedly to teaching. The teacher’s dormitories constructed could substantially improve the accommodation of teachers in rural locations, and attract excellent teachers to teach there with peace of mind, happiness and for a long time.  Support special groups to promote education equity. The left-behind children pilot was implemented to explore the working systems and modes in providing support to the left-behind children, which enables the left-behind children to have a good childhood. The inclusive education pilot was implemented to improve the guarantee system and the teaching/education quality of enabling inclusive education in schools with a particular emphasis on children with disabilities.  The school partnership promoted the education balanced development. The strong schools focus on providing support to the weak schools on educational philosophy, school management, education and teaching, and teachers’ professional quality, which updated the philosophy of the weak schools, improved the teachers’ teaching capacity and promoted the balanced development of compulsory education among regions. c. The development of compulsory education across the province has been promoted to be more balanced, high- quality and standardized. The implementation of the project has increased the investment in compulsory education in project counties, which effectively addressed the shortcomings and weaknesses of compulsory education in project counties, enhanced the development level and quality of compulsory education in project counties, and played an important role in promoting the development of compulsory education across the province to be more balanced, high-quality and standardized. Page 45 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 31. The project is highly sustainable. a. Policy sustainability. The state and the province are committed to promote the high-quality development of compulsory education. The project's construction contents and design concepts are in line with national and provincial requirements for promoting the high-quality development of compulsory education, which provides a policy guarantee for promotion of the project's results, and for playing a sustainable demonstration and guidance role. b. Project implementation mode sustainability. The project implementation mode is highly practical and can be scaled up. It could be used and developed not only in project counties and project schools, but also could be applied and promoted in other counties in Guangdong Province. c. Personnel and management sustainability. The management capacities of the project units have been improved since the project implementation through lots of trainings and on-site guidance by experts. Related professional knowledge and service capacities have been significantly improved, thus providing a service guarantee for the sustainable operations of the project in the future. d. Funding guarantee sustainability. In 2013, the province's free compulsory education public funding subsidy standard was uniformly adjusted to 750 CNY per primary school student per school year and 1,150 CNY per junior middle school student per school year. In 2023, the standard was uniformly adjusted to 1,150 CNY per primary school student per school year and 1,950 CNY per junior middle school student per school year. In the future, Guangdong will also raise the free compulsory education public funding subsidy standard in urban and rural areas, which could provide financial guarantee for the continued operations of the project. ANNEX 6. TEACH Page 46 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 36. Excerpt from: Evaluating Teaching Practices in the Guangdong Compulsory Education Project: Deep Dive into using a Teach Classroom Observation Tool (June 2023, Provincial education department of Guangdong Province, China and World Bank) 37. Teach is a classroom observation tool that provides a window into one of the less explored and more important aspects of a child’s education: what goes on in the classroom. Teach captures both the quantity and quality of teaching practices, which are the time spent on learning as well as the quality of teaching practices. With respect to the quality of education, it measures three areas: (1) classroom culture: how a teacher creates a culture that is conducive to learning; (2) instruction: whether a teacher instructs in a way that deepens student understanding and encourages critical thinking; and (3) socioemotional skills: how a teacher fosters socioemotional skills. 38. In the project-supported counties of Guangdong, China, Teach helped the policy dialogue on teaching practices in the classroom, ultimately resulting in enhancing quality assurance mechanisms for teacher continuous professional development. Teach was applied in 16 project counties of Guangdong, China, in 2020-2022 to monitor the status of ongoing teaching practices and to evaluate the success of teachers’ training under the Guangdong Compulsory Education Project. Data was collected from 228 teachers across the 16 counties in 2020 and 2022. The results from this Teach application provided evidence of the strengths and weaknesses of teaching practices and enabled the provincial government to evaluate the outcome of the two teacher training interventions supported under the Guangdong Compulsory Education Project. 39. Supported by the World Bank, the Guangdong Compulsory Education Project aims to bridge the gap by providing equal educational opportunities for all students in Guangdong, with a particular focus on schools in rural areas. At its core, the project has focused on enhancing teacher capacity through improving training content and targeted teacher training. The aim is to strengthen teachers' teaching skills and pedagogical practices. As of May 2022, the project had successfully developed 14 sets of training materials for in-service training programs. 40. In addition, two types of pilot interventions were implemented in the project counties. The first was a training program for backbone teachers 17 aimed at enhancing teaching practices through activities such as analyzing exemplary lessons, refining lesson design and implementation, and improving classroom management. The second was the Ban Ban Tong intervention, which focused on improving teaching quality by exploring effective teaching strategies, innovative curricula, and teaching methods in utilizing the BBT teaching platform, which is a multimedia platform that provides a diverse range of educational content for students. The BBT platform was provided to schools as part of the Guangdong Compulsory Education Project. As of March 2023, a substantial number of teachers and principals participated in the training under the project, with a total of 38,852 educators, including 253 principals, 454 backbone teachers, and 33,145 teachers using the BBT platform. 41. With the objective of identifying current teaching practices, the Teach assessment tool was utilized to conduct classroom observations among a total of 228 teachers in the 16 project counties. In addition, local observers from the 17Backbone teachers are lead teachers in China who play leadership and modeling roles in school (Bu & Han, 2019, Promoting the development of backbone teachers through university-school collaborative research: The case of new basic education (NBE) reform in China. Teachers and Teaching) Page 47 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT education system, mostly teachers, were trained and certified in using the tool. Two to three observers were assigned to rate the first and last 15 minutes of each class. The provincial education department conducted two rounds of classroom observations, both before and after the training of teachers under the project. The initial data collection took place in January 2020, before the training, while the second round of data was collected in March 2022 after the completion of the backbone training and installation of BBT equipment with accompanying training. This report utilized class-level observations based on the data collected during the second round to gain insights into ongoing teaching practices and evaluate the outcomes of the teacher training interventions 18. 42. Table 1. Overview of Teach Framework Components Areas Elements Score Time on Task Time on Task Time On Learning 1-5 points Supportive Learning Environment Classroom culture Positive Behavioral Expectations Lesson Facilitation Checks For Understanding Quality of Teaching Instruction Feedback Practices Critical Thinking Autonomy Socioemotional skills Perseverance Social & Collaborative Skills Note: To ensure consistency across all elements, scores on Time on Task have been converted to a 5-point scale. The author modified based on Molina & Pushparatnam (2022)19 Proportion of Trained Teachers Meeting Competency Threshold 43. The project aims to ensure trained teachers reach targeted competence levels by monitoring and evaluating their teaching practices. The department has set a minimum threshold of 2.8 out of 5 in Teach assessment for teaching practice quality 20 . Table 2 presents the results of the proportion of trained and untrained teachers meeting the competency threshold in 2022, separated by teacher groups. 44. Overall, the trained groups consistently exhibited a greater proportion of teachers meeting the competency threshold compared to the untrained groups. In the absence of interventions, approximately 87% of teachers demonstrated proficiency in meeting the criteria for quality teaching practices. Notably, among the trained group, a vast majority of 95% of teachers attained a score of 2.8 points or higher (108 out of 114), exceeding the initial goal of 70%. In the area 18 Sample continuity between the initial and second surveys posed challenges due to participant transfers and leaves (Please see Appendix F). The department randomly selected new samples for the second data collection, ensuring equal participant numbers in the treatment and control groups as in the initial survey. 19 Molina, E., & Pushparatnam, A. (2022). Teach Primary: Observer Manual. 20 The project development goal of the Guangdong Compulsory Education Project is to achieve a target of 70% of trained teachers whose teaching practices meet the minimum threshold. Page 48 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT of socioemotional skills, only 32% of untrained teachers met the targeted competencies. In comparison, the trained teachers demonstrated a higher proportion (45%), with a 10-percentage point advantage. 45. When considering the specific training interventions, a noteworthy 99% of the BBT trained group achieved the required score (68 out of 69), while 86% of BBT untrained groups met the desired threshold (59 out of 69). Consistently, the trained BBT groups show a higher proportion in all Teach areas. On the other hand, the same proportion of backbone teachers achieved the targeted competency (40 out of 45). 46. Overall, the TEACH results in 2022 demonstrated the potential advantages of teacher training interventions, with trained teachers consistently surpassing their untrained counterparts in all assessed competencies. These findings underscored the positive influence of training in increasing the proportion of teachers who met the competency threshold across multiple areas, thereby highlighting the effectiveness of training interventions in improving teaching practices. 47. Table 2. Proportion of Trained Teachers Meeting Competency Threshold Total Backbone BBT Traine Untrained Trained Untrained Trained Untrained d N (%) N (%) N (%) N (%) N (%) N (%) Average score on the quality of teaching practices No pass 15 6 5 5 10 1 (Below 2.8) (13) (5) (11) (11) (14) (1) Pass 99 108 40 40 59 68 (2.8 or higher) (87) (95) (89) (89) (86) (99) Classroom culture No pass 2 1 2 1 0 0 (Below 2.8) (2) (1) (4) (2) (0) (0) Pass 112 113 43 44 69 69 (2.8 or higher) (98) (99) (96) (98) (100) (100) Instruction No pass 11 5 5 4 6 1 (Below 2.8) (10) (4) (11) (9) (9) (1) Pass 103 109 40 41 63 68 (2.8 or higher) (90) (96) (89) (91) (91) (99) Socioemotional skills No pass 77 63 29 28 48 35 (Below 2.8) (68) (55) (64) (62) (70) (51) Pass 37 51 16 17 21 34 (2.8 or higher) (32) (45) (36) (38) (30) (49) Page 49 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT Total 114 114 45 45 69 69 Note: The table facilitated the conversion of class-level data into a teacher-level by employing the mean calculation of TEACH scores. Guangdong class observations for TEACH in 2022. 48. To gain a better understanding of the results for backbone teachers, another round of data collection was conducted in 2023 to observe the TEACH scores of these groups after they received training. As mentioned earlier, the absence of a control group (i.e., data on untrained backbone teachers in 2023) prevents a direct comparison between trained and untrained teachers in 2022 and 2023, as was done in the previous 2022 assessment using a randomized controlled design. While the results suggest the effectiveness of the backbone teacher training, the observed impact cannot be conclusively attributed to the training. As shown in Table 3, after receiving the training, backbone teachers demonstrated higher competencies in all domains compared to their scores before the training in 2022. Overall, 100% of backbone teachers met the competency threshold in 2023, increasing from 88% prior to their training in 2022. This improvement is particularly evident in the socioemotional skills domain, where 84% of backbone teachers met the threshold after training, compared to only 42% before their training in 2022. 49. Table 3. Proportion of Backbone Teachers Meeting Competency Threshold before Training in 2022 and after the Training in 2023 Backbone (Before Backbone (Trained) training) 2023 2022 N (%) N (%) Average score on the quality of teaching practices No Pass (Below 2.8) 5 0 (12) (0) Pass (2.8 higher) 38 43 (88) (100) Classroom culture No Pass (Below 2.8) 3 0 (7) (0) Pass (2.8 higher) 40 43 (93) (100) Instruction No Pass (Below 2.8) 6 0 (14) (0) Pass (2.8 higher) 37 43 (86) (100) Socioemotional skills Page 50 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT No Pass (Below 2.8) 25 7 (58) (16) Pass (2.8 higher) 18 36 (42) (84) Total 43 43 ANNEX 7. PROJECT COST SAVINGS AND ADJUSTMENTS Page 51 The World Bank China: Guangdong Compulsory Education Project (P154621) ICR DOCUMENT 50. The project has one disbursement category so the budget reallocations were not included in the official restructurings. Project main cost savings – IBRD funded activities Yuan USD21 Savings from BBT (For the procurement of the initially planned 15,200 sets, the project paid 196.8 million CNY, instead of the estimated 380 million CNY) 183,177,200 25,512,145 Digital education resource development (Cancelled as another project covered the activity) 30,000,000 4,178,273 Mobile labs (Remaining labs cancelled) 33,600,000 4,679,666 External training (Cancelled due to COVID-19) 5,400,000 752,089 252,177,200 35,122,173 Project main reallocations – IBRD funded activities  BBT sets - increase by 25,901 sets  BBT teacher training - increase by 38,500 teachers  Backbone teachers’ training - increase by 200 teachers  Left-behind children and Inclusive Education: Funds per school - increase by 50,000 CNY per school (for 150 schools)  Psychosocial counseling training - increase by 300 teachers  Hire consultants for project management 21 Using 1USD =7.18CNY. Page 52