FOR OFFICIAL USE ONLY Report No: ICR00005802 INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT AND/OR INTERNATIONAL DEVELOPMENT ASSOCIATION IMPLEMENTATION COMPLETION AND RESULTS REPORT Credit Number 5878-VN ON A CREDIT IN THE AMOUNT OF SDR 67.8 MILLION (US$95 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR THE ENHANCING TEACHER EDUCATION PROGRAM PROGRAM-FOR-RESULTS December 15, 2022 Education Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective October 17, 2022) Currency Unit = Vietnamese Dong US$ 1= VND 24,335 SDR 1 = US$ 1.276580 FISCAL YEAR July 1 - June 30 Regional Vice President: Manuela V. Ferro Country Director: Carolyn Turk Regional Director: Daniel Dulitzky Practice Manager: Mario Cristian Aedo Inostroza Task Team Leader(s): Dung Kieu Vo ICR Main Contributor: Sandra Beemer ABBREVIATIONS AND ACRONYMS ASEAN Association of Southeast Asian Nations AUN-QA ASEAN University Network Quality Assurance BOET Bureau of Education and Training CBE Competency Based Education COP Communities of Practice COVID-19 Coronavirus disease CPD Continuous Professional Development DLI Disbursement Linked Indicator DLR Disbursement Linked Results DOET Department of Education and Training ECOP Environmental Codes of Practice EHS Environment, Health and Safety ESSA Environmental and Social Systems Assessment ETEP Enhancing Teacher Education Program FCER Fundamental and Comprehensive Education Reform FM Financial Management FSQL Fundamental School Quality Level GEC New Competency Based Curriculum GNI Gross National Income GOV Government of Vietnam GRM Grievance Redress Mechanism HCMUE Ho Chi Minh University of Education HNUE Hanoi National University of Education HUE Hue University of Education—University of Hue ICT Information, Communication and Technology IDA International Development Association IP Implementation Progress IPDA International Professional Development Association IRI Intermediate Results Indicator ISR Implementation Status Report IVA Independent Verification Agency LMS Learning Management System LTTUs Lead Teacher Training Universities M&E Monitoring and Evaluation MOET Ministry of Education and Training MTR Mid-term Review NAEM National Academy of Education Management NTEP National Teacher Education Program ODA Overseas Development Assistance OECD Organization for Economic Cooperation and Development PA Performance Agreements PAD Project Appraisal Document PDCA Plan, Do, Check, Act PDO Project Development Objectives PforR Program-for-Results PISA Program for International Student Assessment PMU Program Management Unit POM Project Operations Manual RF Results Framework RGEP Renovation of General Education Project SDR Special Drawing Rights SEDS Socio-Economic Development Strategy SEDP Social Economic Development Plan STEP Systematic Tracking of Exchanges in Procurement TEIDI Teacher Educational Development Index TEMIS Teacher Education Management Information System TNUE Thai Nguyen University of Education TPACK Technological Pedagogical Assessment Content Knowledge TTU Teacher Training University UDNE Da Nang University of Education—University of Da Nang UNESCO United Nations Education Science and Culture Organization Viettel Viettel Telecoms Corporation WCC Worker’s Codes of Conducts TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROGRAM CONTEXT AND DEVELOPMENT OBJECTIVES.................................................... 5 A. CONTEXT AT APPRAISAL AND THEORY OF CHANGE .................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .....................................10 II. OUTCOME .................................................................................................................... 14 A. RELEVANCE ..........................................................................................................................14 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................15 C. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................24 D. OTHER OUTCOMES AND IMPACTS (IF ANY) ...........................................................................25 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 26 A. KEY FACTORS DURING PREPARATION ...................................................................................26 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................28 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 29 A. QUALITY OF MONITORING AND EVALUATION .......................................................................30 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................31 C. BANK PERFORMANCE ...........................................................................................................32 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................33 V. LESSONS AND RECOMMENDATIONS ............................................................................. 34 ANNEX 1. RESULTS FRAMEWORK, DISBURSEMENT LINKED INDICATORS, AND PROGRAM ACTION PLAN ...................................................................................................................... 36 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 53 ANNEX 3. PROGRAM EXPENDITURE SUMMARY ................................................................... 56 ANNEX 4. BORROWER’S COMMENTS ................................................................................... 57 ANNEX 5. SUPPORTING DOCUMENTS .................................................................................. 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) DATA SHEET BASIC INFORMATION Product Information Program ID Program Name Financing Instrument Vietnam Enhancing Teacher Education P150060 Program-for-Results Financing Program Country IPF Component Vietnam No Organizations Borrower Implementing Agency Socialist Republic of Vietnam ETEP PMU, Ministry of Education and Training Program Development Objective (PDO) Original PDO The project development objective is to strengthen teacher training institutions to enhance teacher and principal effectiveness through improved continuous professional development. Page 1 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Administered Financing 95,000,000 47,647,015 48,040,291 IDA-58780 Total 95,000,000 47,647,015 48,040,291 Non-World Bank Administered Financing Borrower/Recipient 5,000,000 0 0 Total 5,000,000 0 0 Total Program Cost 100,000,000 47,647,015 48,040,291 KEY DATES Program Approval Effectiveness MTR Review Original Closing Actual Closing P150060 29-Jun-2016 05-Jun-2017 06-Nov-2019 30-Jun-2022 30-Jun-2022 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 27-Apr-2021 38.84 Change in Program Scope Change in Results Framework Cancellation of Financing Reallocation between and/or Change in DLI 13-Apr-2022 48.04 Cancellation of Financing Reallocation between and/or Change in DLI KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial Page 2 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) RATINGS OF PROGRAM PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 23-Dec-2016 Satisfactory Satisfactory 0 02 27-Jun-2017 Satisfactory Moderately Satisfactory 0 03 13-Feb-2018 Moderately Satisfactory Moderately Unsatisfactory 0 Moderately 04 16-Aug-2018 Moderately Unsatisfactory 3.95 Unsatisfactory 05 20-Mar-2019 Unsatisfactory Unsatisfactory 3.95 06 20-Aug-2019 Unsatisfactory Unsatisfactory 6.42 Moderately 07 27-Dec-2019 Moderately Unsatisfactory 10.21 Unsatisfactory Moderately 08 30-Jul-2020 Moderately Unsatisfactory 20.17 Unsatisfactory Moderately 09 05-Mar-2021 Moderately Unsatisfactory 38.84 Unsatisfactory 10 17-Aug-2021 Moderately Satisfactory Moderately Satisfactory 45.84 11 22-Feb-2022 Moderately Satisfactory Moderately Satisfactory 48.04 12 23-Jun-2022 Satisfactory Satisfactory 48.04 SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Other Education 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Page 3 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Human Development and Gender 102 Education 102 Access to Education 38 Education Financing 25 Science and Technology 13 Teachers 13 Standards, Curriculum and Textbooks 13 ADM STAFF Role At Approval At ICR Regional Vice President: Victoria Kwakwa Manuela V. Ferro Country Director: Achim Fock Carolyn Turk Director: Claudia Maria Costin Daniel Dulitzky Practice Manager: Harry Anthony Patrinos Mario Cristian Aedo Inostroza Task Team Leader(s): Michel Jack Welmond Dung Kieu Vo ICR Contributing Author: Sandra F. Beemer Page 4 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) I. PROGRAM CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL AND THEORY OF CHANGE Context 1. In the 25 years prior to appraisal (March 2016), Vietnam had experienced rapid and inclusive economic growth. The economic renovation (‘Doi Moi’) reforms launched in 1986 had transformed Vietnam from one of the world’s poorest countries to a lower-middle-income one—with per capita gross national income (GNI) of US$1,980 in 2015. Growth had also been inclusive: incomes have risen across the income distribution, with only modest increases in inequality. The percentage of people living in extreme poverty dropped from 50 percent in 1993 to 1.8 percent 1 at the time of appraisal. Key social indicators and access to basic infrastructure had also improved substantially. 2. In education, the country had made significant progress, with universal primary education essentially achieved, and moving towards universal preschool education for five-year-old children and universal lower secondary education. Between 2001 and 2012, net enrollment rates at lower and upper secondary levels increased from 70 percent to 85 percent and 33 percent to 60 percent, respectively. 2 The Ministry of Education and Training (MOET) developed the “Fundamental School Quality Level (FSQL)” in 2006, which established minimum standards for physical facilities, school organization and management, teaching materials and teacher support, and school-parent linkages for primary students. Full-day schooling for primary and lower secondary education had gradually expanded to increase learning hours. Vietnam also attained higher levels of student learning achievement. Its average performance on the 2012 Programme for International Student Assessment (PISA) 3 surpassed the Organization for Economic Cooperation and Development (OECD) country average. 4 3. While achievements were impressive, Vietnamese authorities were concerned that the education system was not well positioned for the age of globalization and international competition because the system was not providing the population with the skills and competencies needed for producing greater value for tomorrow’s economy. 5 The 2015-2020 Social Economic Development Plan (SEDP) confirmed that the quick development of high-quality human resources was essential for the country’s industrialization, modernization, and the development of a knowledge-based economy. 4. In 2013, the 11th Party Congress, at its eighth meeting session, adopted the Fundamental and Comprehensive Education Reform (FCER), 6 which aimed for education to meet “the requirements of industrialization and 1 In 2016 and 2020, the people living in extreme poverty is 1.8 and 1.0 percent, respectively. World Bank, 2022. From the last mile to the next mile, 2022 Vietnam poverty and equity assessment. 2 United Nations Education Science and Culture Organization (UNESCO). 2015. Vietnam: Education for All 2015 National Review. 3 PISA (Programme for International Student Assessment) is a worldwide study by OECD in member and non-member nations of 15-year-old school pupil scholastic performance in mathematics, science and reading. Out of 65 countries and economies participating in 2012, Vietnam placed 17th in math, 19th on reading, and 8th place on science, with average scores significantly above the OECD mean in all cases. 4 OECD. 2012. PISA 2012, Result in focus retrieved from www.oecd.org/pisa/.../pisa-2012-results-overview.pdf. 5 Skilling Up Vietnam: Preparing the Workforce for the Modern Market Economy, World Bank. November 2013 6 Resolution No 29/NQ-TW issued on 4th November 2013 of the 8th Conference of the 11th Session of the Central Executive Committee on Fundamental and Comprehensive Education Reform. Page 5 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) modernization in the socialist oriented market economy and international integration.” It constituted a political and legislative umbrella for a broad sector-wide reform. The FCER’s priority was general education (grades K-12), where learning approaches were to become less content-based and more competency- and quality-based. It intended for all Vietnamese students to obtain higher order cognitive and behavioral skills, such as the ability to think critically and creatively, apply knowledge from many subject areas to solve practical problems, work in teams, and convincingly communicate verbally and in writing. 7 To that end, general education introduced, inter alia, new curricula and methods of instruction, developed a comprehensive learning assessment system, and reformed teacher education. 5. Teachers were at the center of the reform process. The success of these reforms depended almost exclusively on the preparedness of the teacher to master the new pedagogical paradigm. One of the underlying principles of the FCER was that teachers should be equipped to respond to different and constantly changing contexts. The FCER required teachers to become increasingly professional, i.e., to adapt methodologies and knowledge to real-time and concrete problems as they arise in the classroom and school. Training and support for such teachers required greater interaction between professionals, reciprocity, hands-on mentorship, coaching, and on-time advice to solve identified problems. 7 Ibid. Page 6 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Theory of Change (Original Project Appraisal Document (PAD) Results Chain)—Figure 1 Rationale for PforR Support, and Program Scope and Boundaries 6. In response to the FCER, the MOET prepared the National Teacher Education Program (NTEP) 8, to enable the teacher education system to adapt more effectively to the changing needs of teachers and principals. The NTEP focused on one of the FCER Programs -- the Reform of Preservice and In-service Training -- to produce a comprehensive strategy for realigning teacher education to the new requirements of general education. The NTEP was a comprehensive strategy that aimed to realign teacher education to the new requirements of general education. It proposed several paths of interventions: pre-service training for new teachers, retraining for some teachers to obtain additional qualifications, in-service training programs that bring teachers to training centers for specific objectives, and continuous professional development (CPD) that provides support and training to teachers inside schools and classrooms. The MOET NTEP provided the framework to achieve this goal with six solution areas: (i) strengthening teacher education management planning; (ii) renovating pre-service teacher education; (iii) establishing and delivering continuous 8 The NTEP, was approved by Government on April 29, 2016. Page 7 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) professional development programs for teachers and school managers; (iv) building capacity of key lecturers and managerial staff of teacher training institutions; (v) enhancing the physical environment and facilities/equipment of teacher training institutions; and (vi) promoting socialization and international cooperation in teacher education. (See Figure 2) Figure 2: National Teacher Education Program in Context 7. Of the different elements of the NTEP, the Government and the World Bank agreed that the Enhancing Teacher Education Program (ETEP) would focus primarily on Solution Area 3—establishment and delivery of CPD through a Program for Results (PforR) approach. The enhanced CPD system was to provide more training and support that was of high quality, relevant, and timely -- directly to teachers and principals at the school level. ETEP was to use the revised curriculum to develop the CPD programs for teachers and principals. 8. The ETEP was designed to enhance teacher education and help teachers more effectively adopt new methods and master new competencies with a long-term goal of improved student learning outcomes and increased adult labor force participation to support Vietnam’s national development goals. ETEP aimed to reinforce the provision of CPD through two main vehicles: (i) core teachers and principal advisors, who would be assigned to schools and school clusters to provide face-to-face peer support and (ii) an online platform that could provide needs-based, interactive training and support directly to teachers and principals. To ensure the dynamism and quality of the enhanced CPD system, eight LTTUs in respective catchment areas (corresponding approximately to regions) were to train and provide support to core teachers and principal advisors; develop the online services and applications; and help develop the needs assessment systems. The LTTUs were to provide technical leadership and guidance for the school based CPD system. In addition, the design sought to mobilize world-class expertise to help develop and operationalize the strategies for improving school based CPD for teachers and principals. This technical assistance (TA) would allow MOET and LTTUs to: (i) elaborate and operationalize policies and directives, (ii) inform and improve the technical quality of the expected results, and (iii) reinforce the implementation capacity, including monitoring and evaluation, of the Program. (See Figure 1 for Original PAD Program theory of change diagram.) Page 8 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) 9. It is also important to note that the World Bank supported the Renovation of General Education Project (RGEP) (P150058), that was implemented in parallel with ETEP. The RGEP supported the FCER focus on the development and implementation of a new competency-based curriculum (GEC 2018) 9 for all school subjects from grades 1 – 12 (See Figure 2). The expectation was that ETEP CPD would prepare teachers and principals to meet the professional standards newly developed aligning with the reformed curriculum. The RGEP closed December 31, 2020, and by project closing, the MOET had endorsed and launched 29 new competency-based school curricula consisting of a general curriculum and 20 subject-specific curricula. The new curricula were used in the development of the ETEP CPD. Program Development Objectives (PDOs) 10. The PDO was to strengthen teacher education institutions to enhance teacher and principal effectiveness through improved continuous professional development. The PDO statement in the Project Appraisal Document (PAD) and Financing Agreement are the same. Key Expected Outcomes and Outcome Indicators 11. Progress towards the achievement of the PDO would be measured through two PDO indicators, as follows: • PDO 1: improved performance of the eight 10 lead teacher training universities (LTTUs) to provide CPD as measured by the Teacher Educational Development Index (TEIDI) 11; and • PDO 2: number of teachers and principals satisfied with school based continuous professional development. Program Results Areas and DLIs 12. ETEP had four results areas that were part of the NTEP and constituted the principal strategies for establishing an effective and well-functioning CPD system that could provide school-based support and training to principals. The Program results areas and associated DLIs, as described below: 9 GEC 2018 is the new general education curriculum introduced by MOET by Circular 32 in 2018 which aimed to shift general education from knowledge based to competency based. The ten (10) core competencies to be developed by students are: (i) autonomy and self-learning, (ii) communication and collaboration, (iii) problem solving and creativity, (iv) language, (v) calculation, (vi) science, (vii) technology, (viii) computing, (ix) aesthetics, and (x) physique. 10 Eight LTTUs include: (i) Hanoi National University of Education (HNUE), (ii) Thai Nguyen University of Education (TNUE), (iii) Hanoi 2 Pedagogical University, (iv) University of Education-Vinh University, (v) Hue University of Education - University of Hue (HUE), (vi) Da Nang University of Education - University of Da Nang (UDUE), (vii) Ho Chi Minh University of Education (HCMUE), and the (viii) National Academy of Education Management (NAEM). 11 The TEIDI index is composed of two basic elements: (i) assessment of the specific areas for TTUs (mainly on teaching, research and services); and (ii) assessment of the development of TTUs (serving regular training process). It includes seven dimensions: (i) strategic vision, governance, and quality assurance; (ii) training programs; (iii) research, development and innovation; (iv) external collaboration and relationship; (v) educational environment; (vi) teaching support; (vii) learning support. TEIDI applies a comprehensive approach to the planning, management and governance of teacher training universities (TTUs) as well as to identifying the strengths and weaknesses in the capacity of TTU’s training and retraining. The index helps identify the needs for teachers, organizational requirements, readiness and efficiency of the system providing pedagogical services, including sensitivity to the needs of disadvantaged groups and ethnic minorities, disadvantaged areas, and places with limited access to internet or other supports to provide regular training programs and materials for teachers and management staff of schools fostering teachers and educational managers in order to meet the requirements of general education reform and international integration. Page 9 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) • Results Area 1: Improved teacher capacity of LTTUs and central teacher management units to enhance teachers' and principals' effectiveness. (DLI 1) • Results Area 2: Development of teacher and principal training needs assessments to inform CPD program development. (DLI 2) • Results Area 3: School based and continuous professional development for teachers and principals. (DLI 3 and DLI 4) • Results Area 4: Teachers and principals have access to CPD programs and resources through an information and communications technology-based system. (DLI 5) 13. The DLIs each had one or more disbursement-linked results (DLRs) that would trigger periodic disbursements. The original DLI disbursement periods were defined as: (i) period 1—the period from date of signing the Financing Agreement (FA) to December 31, 2017; (ii) period 2—January 1 to December 31, 2018; (iii) period 3—January 1 to December 31, 2019; (iv) period 4—January 1 to December 31, 2020, and (v) period 5—January 1 to December 31, 2021. As discussed below, the DLI matrix was revised and is presented in Annex 1. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs Outcome Targets, Result Areas, and DLIs 14. The PDO and the Results Areas were not revised during the project period. However, there were revisions to the PDO and intermediate indicators and to the DLIs at the time of the April 28, 2021 restructuring. They were as follows: (i) two PDO indicators were revised (see Table 1 for details), (ii) all DLIs were revised (Table 2 for details); and (iii) all intermediate indicators (See Table 3 for details) were revised. Table 1: April 2021 Restructuring Revisions to PDO Indicator Original Indicator Revised Indicator Original Target Revised Target PDO 1: Improved PDO 1: Improved 15 percent increase over the TEIDI score of each LTTU performance of the eight performance of the six 12 baseline in TEIDI score for increases to at least level 5 LTTUs to provide CPD as LTTUs to provide CPD as each LTTU (>4.43) 13 measured by the TEIDI measured by the TEIDI Description of changes. The number of LTTUs was reduced from eight (8) to six (6) for the ‘capacity development investment’ areas of the LTTU performance agreements (PAs) that had been signed with MOET. The removal of the two LTTUs was largely due to the low achievement of PA targets. Besides, implementation delays related to the development of the CPD program materials made it difficult for the MOET to continue to provide technical and supervision support for the PA capacity development for all the LTTUs. While the two LTTUs did not receive the resources for ‘capacity development’, they did continue to receive capacity building from the remaining six LTTUs (judged to be high performing according to TEIDI) that was required and necessary for the rollout of their CPD activities as well as the infrastructure resources as agreed in their PAs before the restructuring. All 8 LTTUs continued to provide training for core teachers and principal advisors and support the core teachers and principal advisors in the rollout of the CPD to teachers and principals in their provincial catchment areas, thereby ensuring that there was no reduction in the scope of the Program. The unit of measurement was revised from “percentage increase” to “score increase” because the six LTTUs were 12Of the original 8 LTTUs Hanoi 2 Pedagogical University and NAEM had not received funds for development activities to gain the TEIDI target scores because of limited ability to implement TEIDI and were removed from the Program. 13 TEIDI scores are based on the 7 TEIDI dimensions. The TEIDI different levels scores are: (i) Level 1: 1 < score <=1.86; (ii) Level 2: 1,86< score<=2.71; (iii) Level 3: 2.71< score<=3.57; (iv) Level 4: 3.574.43) was fully achieved (see Table 4 for actual TEIDI scores for all six LTTUs). The IRI indicator associated with this PDO—number of LTTUs that sign PAs with MOET increased from the 2017 baseline of 0 to 8 in 2019, meeting the original target and continued with the PA TEIDI assessment requirements for six (6) LTTUs meeting the 2021 revised target of 6. In addition, the DLI target for periods 1, 2 and 3 for all eight (8) original LTTUs were disbursed as designed. These disbursements were contingent upon eight LTTUs conducting baseline TEIDI assessments and signing PAs with MOET. The PAs also included infrastructure improvement. All 8 LTTUs conducted their baseline TEIDI 16The scoring of TEIDI indicators is evaluated on the seven-dimensions, which, as indicated, include qualitative and quantitative factors and specific for each indicator. Qualitative factors are determined according to the level of satisfaction in the 4-step PDCA cycle: Planning (Plan) – Executing the plan (Do) – Checking (Check) – Action (Act). The PDCA cycle is a four-step problem- solving iterative technique used to improve business processes. The cycle draws its inspiration from the continuous evaluation of management practices and management’s willingness to adopt and disregard unsupported ideas. Page 15 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) assessments and by 2018 all 8 LTTUs had signed their PAs with the MOET. As indicated, once the number of LTTUs was revised in 2021, TEIDI assessments for six LTTUs were completed and all DLR balances for periods 4 and 5 were verified and disbursed. Table 4: Six LTTU TEIDI Score Achievement LTTUs 2017 2022 2022 Baseline End Target Actual verified Achievement by IVA verified by IVA Hanoi National University of Education (HNUE) 3.47 level 5 (>4.43) level 5 (5.00) Thai Nguyen University of Education (TNUE) 3.30 level 5 (>4.43) level 5 (4.72) University of Education—Vinh University 4.22 level 5 (>4.43) level 5 (4.86) Hue University of Education-University of Hue (HUE) 3.55 level 5 (>4.43) level 5 (4.95) Da Nang University of Education - University of Da Nang (UDUE) 3.34 level 5 (>4.43) level 5 (4.88) Ho Chi Minh University of Education (HCMUE) 3.10 level 5 (>4.43) level 5 (4.91) 25. Results Area 1: Improved capacity of LTTUs and central teaching management units to enhance teacher and principal education effectiveness had been fully achieved. As a result of the ETEP, the six LTTUs that continued the annual TEIDI assessments have seen overall improvements in the following areas: I. University governance and leadership. LTTUs now have enhanced strategic planning through the development of their PAs as well as through the implementation of their plans. Firstly, the TEIDI instrument which was the main instrument to measure LTTU improved capacity was modified for use by Vietnamese LTTUs and the framework guidelines were developed and approved in 2019 and updated in 2020 and 2021. The guidelines provided the LTTUs with concrete guidance on TEIDI use. As indicated, all eight of the original LTTUs participated in the baseline TEIDI assessment that the LTTUs used to develop their original PAs which included the mission, vision and development strategies with long-term, medium term and short-term goals. The PAs had annually developed implementation plans with specific key performance indicators and budget requirements. The plans focused on all seven TEIDI dimensions. The plans were publicly posted on the LTTUs websites. The LTTUs used the PDCA (Plan, Do, Check, Act) cycle, the key performance indicators, development of a stakeholder feedback system and digital technology for continuous improvement in governance as well as teaching and learning. It should be noted that the inclusion of PAs within the ETEP was the first time that the GoV used performance agreements for strategic planning and budget allocation—in practice ETEP was a pilot for the use of PAs. The government has recognized the benefits of PAs process and is now moving toward establishing this methodology for other higher education institutions. LTTUs also established Teacher-Learner Communities (between LTTU faculty and core teachers) and communities of practice (COPs) 17 in competency-based teacher education (among faculty members of all LTTUs). The COPs helped faculty have better insights into classroom and school contexts and thus enhanced their practices in pre-service teacher education. The LTTUs strengthened partnerships between the LTTUs and provincial governments in continuous assessment of CPD needs and school-based professional development support for teachers and principals. 17Communities of practice are collectives of people with shared interests in achieving the same goal. The COPs interactions include LTTU faculties, LTTU faculties and core teachers/ principal advisors, core teachers and principal advisors, core teachers/ principal advisors and teachers/ principals. Page 16 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) II. Capacity development in research, development, innovation and for teacher training lecturers. • Research, development, and innovation. LTTUs have enhanced the capacity of faculties for conducting research with a stronger focus on educational science based on a better understanding of general education and classroom practice. To support them in these efforts, the six LTTUs established six research groups with members from the different LTTUs and Birmingham City University on six educational science research topics that were presented at the UK-based International Professional Development Association (IPDA) conference. The LTTUs used their research activities to create more demand-driven teacher education curriculum, on-line courses and blended programs and deployment of relevant pedagogical and assessment methodologies to enhance the LTTU graduate employability skills, and delivery of a large-scale CPD program in coordination with multiple stakeholders. Moreover, LTTUs issued policies, developed, and implemented long-term plans on scientific research, development and innovation in line with their vision, mission, and strategies, the budget investment and equipment for the renovation. The LTTUs promulgated new research regulations with incentives for researchers. Additionally, over five years, the six LTTUs increased the number of international research topics and publications, including in journals on the list of International Scientific Indexing (ISI)/Scopus. (See Table 5 for publication details.) Table 5: LTTU Publications Total publications Total publication Percentage 2017 2022 increase in international publications Total Total Total Total Publications International publications International publications publications and % of Total and % of Total Hanoi National University of 847 190 (22.4%) 1,342 262 (19.5%) 38.0 Education Thai Nguyen University of 272 3 (1.1%) 140 35 (25.0%) 1066.0 Education University of Education—Vinh 112 54 (48.2%) 333 152 (45.6%) 181.0 University Hue University of Education- 320 57 (20.9%) 340 138 (40.0%) 242.1 University of Hue Da Nang University of 301 33 (10.9%) 397 108 (27.2%) 227.0 Education - University of Da Nang Ho Chi Minh University of 129 4 (3.1%) 263 26 (9.9%) 550.0 Education Hanoi National University of 113 2 (1.7%) 539 40 (7.42%) 1900.0 Education 2 (dropped from TEIDI) National Academy of Education 56 16 (28.5%) 181 47 (25.9%) 193.0 Management NAEM (dropped from TEIDI) Source: MOET ETEP Completion Report, 2022 • Capacity development for teacher training lecturers. Within the framework of ETEP training activities, lecturers from all 8 original LTTUs participated in workshops and training and learned and applied modern Page 17 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) approaches that contained both theory and hands-on practical experience. Capacity building activities for teams selected from training faculties (including core teacher training lecturers and core education management lecturers). All 8 LTTUs also participated in workshops and training activities that were supported by education experts from the United Kingdom, Hong Kong, SAR, China, Taiwan, China and Australia, as well as local experts recruited by ETEP. Notably, the new CPD mechanism established continuous hands-on support by LTTUs to teachers via a web-based system. The COPs set a new channel for the LTTUs to keep updated with classroom practice and teacher training needs that never existed within the system prior to ETEP. The new mechanism has helped the LTTU faculty apply both theory and hands-on practical experience. In addition, all 8 LTTUs increased their percentage of lecturers with doctoral degrees (except for UEVU which had the highest baseline percentage which slightly decreased during the Program implementation), leading to improved quality in research development and innovations (Table 6). Table 6: Percentage of Full-time Lecturers with Doctoral (PhD) Degrees LTTU/year Number of Full- Full-time lecturers Percentage of time lecturers with PhDs Lectures with PhDs Hanoi National University of Education 2018 749 419 55.9 2022 626 18 397 63.4 Thai Nguyen University of Education 2018 338 164 48.5 2022 272 204 75.0 University of Education—Vinh University 2018 317 206 64.6 2022 317 194 62.2 Hue University of Education-University of Hue 2018 251 140 55.8 2022 221 143 64.7 Da Nang University of Education - University of Da Nang 2018 240 109 38.3 2022 238 138 52.5 Ho Chi Minh University of Education 2018 509 167 32.8 2022 513 231 45.0 Hanoi National University of Education 2 (dropped from TEIDI analysis) 2018 325 107 36.0 2022 289 154 53.3 NAEM (dropped from TEIDI analysis) 2018 105 43 41.0 2022 96 46 47.9 Source: MOET ETEP Completion Report, 2022 III. Capacity development in quality assurance, curriculum development, and pedagogical renovation. 18 Hanoi National University of Education experienced a decrease in the number of enrolled students since 2020 as the university ceased the distant learning/ e-learning for master/ PhD programs and informal education (while keeping the blended learning mode for bachelor programs). The number of lecturers has significantly decreased accordingly. Page 18 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) • Quality assurance—Embedded in the TEIDI was a set of standards for quality assurance and accreditation of Vietnam’s higher education institutions. The standards reference the Association of Southeast Asian Nations (ASEAN) University Network-Quality Assurance (AUN-QA) 19 standards. All quality assurance activities at the 6 LTTUs that continued with the annual TEIDI assessments were evaluated based on these standards. • Curriculum development and pedagogical renovation—During ETEP, 51 CPD modules (24 for general education teachers and 27 for general education managers) were developed for primary, lower secondary and upper secondary levels teachers and principals by the LTTUs. Of the 51 modules, there were nine (9) compulsory CPD training modules 20 that were developed by seven21 LTTUs for the national scale rollout of the CPD to all 63 provinces. The process was highly participatory and included, but was not limited to, stakeholders within the MOET, DOETs, BOETs, CPMU, curriculum development and teacher training national experts including LTTU faculty, principals and teachers and international consultants. All the CPD modules were aligned with the GOV Circulars related to the establishment of the CPD program as well as the Circulars and Decisions related to the rollout of the GEC 2018. In addition, the CPD program was designed in coordination with curriculum experts to ensure strong alignment between pedagogical approaches, curriculum and learning assessment. The CPD modules went through a rigorous quality assurance process 22 that drew on state-of-the-art teaching and assessment knowledge as well as modern school governance practices and were verified as satisfactory by the IVA. The modules provided practical guidance for teachers as well as new pedagogical approaches. The materials were developed in formats to meet diverse learning needs such as video, audio, and printed materials as well as in digital formats to be able to meet the needs of the CPD program. Importantly, the CPD was being rolled out in conjunction with the new K-12 competency- based education (CBE) program supported by the FCER implementation. Moreover, the LTTUs developed CPD e-learning courses consisting of 18 training modules for two target groups—core and general education teachers and managers, at three learning levels 23—all meeting the quality assurance process described above. The e-learning courses became increasingly important during COVID-19 because they allowed for the continuation of Program activities and mitigated some of the negative impacts on the delivery of the CPD. 19 The AUN-QA Network was created to harmonize educational standards as well as to seek continuous improvement of academic quality of universities in the ASEAN region. 20 The nine compulsory modules are: (i) Module 1: Teacher: Guide to implementing the 2018 Education Program; Management staff: Managing teaching and educational activities; (ii) Module 2: Teacher: Teaching methods and techniques; Management staff: Human resource management; (iii) Module 3: Teacher: Student assessment; Management staff: Financial management; (iv) Module 4: Teacher: Development of CBE lesson plan; Management Staff: Management of infrastructure, equipment and technology; (v) Module 5: Teacher: Student counselling and support; Management Staff: Education quality assurance at schools; and (vi) Module 9: Teacher: ICT in teaching and assessment; Management Staff: ICT in school management. 21 The seven LTTUs were: (i) Hanoi National University of Education (HNUE), (ii) Thai Nguyen University of Education (TNUE); (iii) University of Education-Vinh University, (iv) Hue University of Education - University of Hue (HUE), (v) Da Nang University of Education - University of Da Nang (UDUE), (vi) Ho Chi Minh University of Education (HCMUE), and (viii) National Academy of Education Management (NAEM). 22 The quality assurance process was a 16-step participatory and robust process, agreed between the Bank and MOET, to ensure training materials and e-courses met international standards. Specifically: The CPMU established an independent expert group to directly provide insights on the development of outlines and materials. Independent experts included managers, lecturers, and researchers, with expertise and experience related to the module contents. The independent expert group advised the PMU and the material development team on technical issues related to ensuring the materials met the requirements of Decision No. 4660/QD-BGDĐT and the GEC 2018 while also being relevant with the work of general education teachers and managers. The CPD materials were piloted and then modified based on consulted with relevant stakeholders (i.e., international consultants, experts of LTTUs, relevant MOET departments, secondary school principals and teachers). Modules and materials were appraised at university level and cleared at ministerial level. Materials/ e-courses were revised based on comments of the appraisal council and the final verification council and updated after the delivery of the CPD module to general education teachers and managers and based on feedback from learners. 23 The three learning levels are primary level (grades 1-5), secondary level (grade 6-9) and high school (grades 10-12). Page 19 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Overall, ETEP created an effective CPD model for general education school teachers that moved from a centralized professional development model to a continuing, on-site/blended model (using in-person and on- line training through the LMS support by ETEP) that is continuously supported by teacher training universities (TTUs). In addition, ETEP provided, for the first time, a knowledge repository of 170 sets of training materials for 51 modules, including e-courses for teachers and principals at primary, lower and upper secondary levels. The materials were developed following a quality assurance process that included peer review by national and international experts. This was never done in any MOET CPD programs before—an important and considerable achievement. In addition, the LTTUs learned how to develop pre-service teacher training programs with support from international consultants based on the Technological Pedagogical Assessment Content Knowledge (TPACK) concept. 24 IV. Infrastructure—upgrading of facilities and equipment. As indicated, all eight original LTTUs signed PAs that included capacity development and infrastructure components. All eight LTTUs completed the upgrading of facilities as well as their IT projects. The new IT infrastructure included but was not limited to: upgraded core software for training, upgraded network transmission systems, Wi-Fi access, new servers, internet connection and library software, learning materials production equipment, installation of learning materials production studios, installation and upgrading of functional modules in the LMS system, providing SSO accounts for staff, etc. The completion of these activities improved the procurement capacity and infrastructure management capacity. Perhaps more importantly, the new IT and digital application within the LTTUs contribute to unlocking their development potential to reach regional and international levels and maximizing efficiency in using invested facilities in all activities of the school (i.e., teaching, research, learning support, digital resource development, CPD, organizing activities to support learners, academic and pedagogical environment, and landscape, and connecting international networks). The equipment has also enhanced LTTU collaboration, supported data sharing among the affiliated units of LTTUs (i.e., data center, network infrastructure, server systems), and ensured consistency in operations management, document storage, centralized data management and overall management. The LTTUs also established an online training research institute. Again, during the COVID-19 pandemic, LTTUs used their new systems to mitigate the negative impacts of learning by providing teaching and learning activities virtually—a critical activity for ensuring the successful completion of ETEP. 26. Achievement of Objective 2: Enhanced teacher and principal effectiveness is judged to be Substantial. This objective was measured by several interlinked monitoring tools. The PDO indicator linked to this objective was—number of teachers and principals satisfied with the new school based continuous professional development. The three IRI indicators linked to this objective were: (i) number of TEMIS-generated annual reports on CPD needs by DOET; (ii) number of deployed core teachers and/or principal advisors having completed different priority training module sets for the year; and (iii) number of teachers and school principals having completed different mandatory online modules of the CPD program set for the year through the LMS. The Results Areas and associated DLIs/DLRs related to this objective were: (i) Results Area 2: Development of teacher and principal training needs assessments to inform CPD program development (DLI 2); (ii) Results Area 3: School based and continuous professional development for teachers and principals. (DLI 3 and DLI 4); and (iii) Results Area 4: Teacher and principals have access to CPD programs and resources through an information and communications technology-based system. (DLI 5). 27. The PDO indicator—number of teachers and principals satisfied with the new school based continuous professional development increased from the baseline of 0 in 2016 to 637,279 in 2022 exceeding the 2021 revised 25The occupational standards were regulated in Circular No. 14/2018/TT-BGDDT dated July 20, 2018 and Circular No. 20/2018/TT-BGDDT dated August 22, 2018. Page 20 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) target of 320,000 by 317,279—surpassing the target by 99.1 percent. It should also be noted that the original target of 480,000 was surpassed by 157,279 or 32.5 percent. Importantly, teachers and principals’ satisfaction was measured by a composite index that consisted of: (i) interactive support provided; (ii) quality of ICT based interactive CPD services with regards to availability, relevance to meeting new teacher and principal standards, usefulness in the classroom; and contribution to improved student outcomes. Teachers and principals satisfied with the interactive CPD were those whose response had an average score of 3 and above (on a scale of 1 to 4 with 4 as totally satisfied) and had no scores below 2.5. These numbers were all achieved due to the introduction of the TEMIS and LMS systems that had been delayed but ultimately allowed teachers and principals to access the satisfaction survey as well as the CPD training materials and the teacher self-assessments. The restructuring made the correct judgement to reduce the number because of implementation delays. However, the Bank and GOV teams, working together, made substantial efforts to meet the expected goals of this indicator. 28. In addition to the indicators, the effectiveness of teachers and principals was assessed by teacher and principal supervisors based on the MOET professional standards 25 through their reviews of teachers and principals and was recorded in the new TEMIS. TEMIS data showed that (i) the percentage of principals assessed as distinguished’’ (out of the ranking of unmet, threshold, competent and distinguished) increased for all 18 criteria set out in the professional standards when benchmarking the 2019-2020 and 2020-2021 performance assessment results; (ii) the percentage of teachers assessed as “distinguished” (out of the same ranking as the one for principals) increased for 14 out of the 15 criteria set out in the professional standards when benchmarking assessment results between the two same period. (See Table 7) Importantly, the teachers assessed as “distinguished” all increased for criteria 2, 4, 5, 6, 7 26 in the professional standards. This is important because one aspect required within criteria 2, 4, 5, 6, and 7, is that teachers’ students demonstrate substantial learning progress or exceed the set targets towards the end of the academic year. 27 The increased percentage of teachers rated as ‘distinguished’ for these criteria therefore implies substantial progress by their students in academic results. Table 7: Results of Assessment by Supervisors of Teachers and Principals (%) Based on Professional Standards Unmet Threshold Competent Distinguished 2019-2020 2020-2021 2019-2020 2020-2021 2019-2020 2020-2021 2019-2020 2020-2021 Teachers 0.30 0.86 3.54 2.88 51.45 41.42 44.71 54.52 Principals 0.05 0.13 2.38 1.76 46.43 35.12 51.57 62.58 25 The occupational standards were regulated in Circular No. 14/2018/TT-BGDDT dated July 20, 2018 and Circular No. 20/2018/TT-BGDDT dated August 22, 2018. 26 Criterion 2. Teachers’ working style. Criterion 4. Development of teaching and instructional plans for student development in personal qualities and competencies. Criterion 5. Use of teaching and educational methods to develop students’ personal qualities and competencies. Criterion 6. Evaluation and assessment for students’ personal qualities and competencies development. Criteria 7. Student counselling and support. 27 The criterion on teachers’ students demonstrating substantial learning progress is measured against targets set for each student, by their teachers, at the beginning of the academic year and actual achievement of those targets at the end of the academic year. Students’ improvement is determined by end-of-year standardized test scores. The test scores are graded on a 10-point scale where passed=5-6, competent=7-8, and excellent=9-10. Page 21 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) 29. Results Area 2: Development of teacher and principal training needs assessments to inform CPD program development. The IRI indicator that was directly linked to Results Area 2—number of TEMIS 28-generated annual reports 29 on CPD needs by DOET increased from the 2016 baseline of 0 to 45 in 2022, meeting the target of 45. It is important to note that ETEP developed and adopted new teachers and principal professional standards to align with the new curriculum and that the assessment in TEMIS uses the new set of standards. TEMIS was deployed in all 63 provincial Departments of Education and Training (DOET). All of them can extract reports on teacher performance and CPD for teachers and principals, with 45 of the 63 meeting the established quality standards 30. The associated “DLI 2: systems are in place to (i) measure whether teachers and principals meet new professional standards; (ii) track and record evaluations of CPD programs and (iii) record assessment of CPD needs of teachers and principals” was also achieved with the final target of 45 TEMIS reports generated. To meet the DLI target, MOET outsourced the provision of the TEMIS system to Viettel Telecoms Corporation (Viettel) in 2020 and Viettel signed an agreement with MOET to provide the TEMIS services for free until June 2026 31. In addition, the LMS system, which is also operated by Viettel, was connected to the TEMIS so that the two systems interfaces. Notably, the TEMIS now houses teacher and principal self-assessment data and reports with supporting evidence, peer reviews, and management appraisal and is the first repository of its kind in Vietnam. The annual TEMIS reporting process moved the yearly performance assessment from paper-based to online, and therefore, teacher education information is now available at national, provincial, and district levels which has increased the accountability of teachers, school principals and local authorities in conducting honest assessments. The MOET has also institutionalized user roles and processes through official guidelines and user training provided for all 63 provinces. As a result, reliable information on CPD needs and CPD programs is available for evidence-based planning, budgeting, and implementation. 30. Results Area 3: School based and continuous professional development for teachers and principals. The IRI indicator that was directly linked to Results Area 3—number of deployed core teachers and/or principal advisors having completed different priority training module sets for the year increased from the 2016 baseline of 0 to 30,134 (26,319 core teachers and 3,815 principal advisors completing 6 modules cumulatively) in 2022 exceeding the target of 28,600 (6 modules cumulatively) by 1,534 or 5.4 percentage increase over the target. Thirty-six (36) modules were delivered to the 31,134 core teachers and principal advisors at primary, lower secondary and upper secondary levels in 63 provinces through in-person training and the LMS (blended model). The CPD blended model was essential in expanding equal access to CPD opportunities for teachers and principals all over the country. The on-line learning with on-site support has radically shifted the CPD paradigm for teachers and principals. As indicated, 36 modules were delivered to more than 30,000 core teachers and principal advisors at primary, lower secondary, and upper secondary levels and 30 (among the 36) compulsory modules were delivered by 911 core faculties from the LTTUs to over 600,000 teachers and principals 28 TEMIS is a Management Information System for General Education Teachers and Principals. MOET, through the ETEP program, conduct annual assessment of teachers and principals according to the new professional standards, as regulated in Circular No. 20/2018/TT-BGDDT dated August 22, 2018 and Circular 14/2018/TT-BGDDT dated July 20, 2018, respectively. 29 The TEMIS reports include: (i) professional development trends among general education teachers and principals based on an annual performance assessment; (ii) evaluation of general education teachers and principals on CPD programs; and (iii) training needs of general education teachers and principals. 30 The quality assurance process includes 14 steps with expected outputs, sources of evidence, lead agencies and collaborating agencies. 31 Viettel is the government owned Telecoms Corporation. MOET and Viettel signed a memorandum of understanding (MOU) to provide TEMIS services. The MOU stipulates that Viettel will: (i) grant user right (view/manipulate account information, organization information, view/synchronize the latest data for statistical reports, view assessment results of teachers/administrators), and data explore at all levels according to management requirements of MOET; (ii) provide support for administration and operation of the system; maintain user service quality, assuring safe and smooth operation of the system 24/7 for generation of TEMIS reports as regulated in the operations manual (OM) at: https://temis.csdl.edu.vn for the period of 5 years (June 2, 2021 to June 2, 2026) and(iii) provide full support for data requirements as proposed by MOET. Page 22 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) supported by the trained core teachers and principal advisors. The on-line learning proved its advantage during the COVID-19 pandemic, where in-person delivery was impossible in most provinces over six months in 2021. It also helped strengthen the instructional design of core faculty and the ICT skills of teachers and principals and ensured knowledge was not diluted through the cascading training model. The blended delivery model also allowed learners to learn at their pace and at a time that would not disrupt classes. Another important achievement of ETEP was that the core teachers and principal advisors’ selection processes, as well as their roles/responsibilities, were institutionalized by the MOET to provide continuous school-based support for teachers across the country. This policy decision has contributed to the sustainability of ETEP. The final targets for the associated DLI 3: Number of core teachers and principal advisors selected and trained to provide systematic face-to-face school-based continuous professional development to teachers and principals and DLI 4: Number of core teachers and principals advisors providing school-based continuous development to teachers and principals mirrored the IRI target of 28,600 and was met as indicated. 31. Results Area 4: Teacher and principals have access to CPD programs and resources through an information and communications technology-based system. The IRI indicator that was directly linked to Results Area 4— number of teachers and school principals having completed different mandatory online modules of the CPD program set for the year through the LMS increased from the 2016 baseline of 0 to 527,532 (5 modules cumulatively) in 2022 exceeding the target of 480,000 (5 modules). The associated “DLI 5: number of teachers and principals who are satisfied with ICT-based interactive CPD services provided under the Program reached 320,000 in period 5. As indicated, teachers and principals now have access to CPD programs through the LMS in all 63 provinces. An important aspect of the CPD programs was the blended on-line learning with face-to-face support, which increased CPD access for teachers while maintaining high quality training offered by accredited teacher training universities, unlike the cascade model used before ETEP. In addition to the web-based learning management system, social media was used for rapid support by the LTTU faculty and core teachers to ensure quality and consistency in CPD at provincial levels. MOET also issued guidelines on: (i) CPD planning and monitoring at provincial and LTTU levels; (ii) online pre-learning activities; (iii) peer support; and (iv) monitoring and reporting. MOET and the DOETs have recognized the timely development of the CPD mechanism, which prepared general education teachers and principals for the national GEC implementation in grades 1, 2, 3, 6, 7 and 10 32. In 2021-2022, the the survey to determine satisfaction levels of teachers and principals with the CPD online programs in five areas, as recorded on TEMIS, indicated that ETEP CPD programs were relevant, useful, and practical. Most teachers and principals agreed or strongly agreed that the online CPD program was relevant to the needs of continuing professional development and useful to apply to the learning curriculum and help improve students’ learning performance. (See Table 8) Table 8: Teacher and Principal CPD Satisfaction Assessments Online Availability of Relevance with Usefulness for CPD contribution to interactive online CPD CPD needs in line application in improvement of academic support programs with occupational classrooms performance of secondary standard students Teachers 95.93 96.00 97.18 96.84 96.51 Principals 96.07 96.17 97.21 97.09 96.82 Source: TEMIS, 2022 32. Examples of teacher’s responses indicated that: • Availability of online CPD programs allowed for self-paced learning. Teachers and principals were highly appreciative of the fact that CPD learning materials on LMS were provided timely with adequate presentations that included good audio and images, data sheets, video files, electronic lectures, teaching software, and online 32 The curriculum roll-out for grades 4,5,8,9,11,12 will take place in academic years 22/23 and 23/24. Page 23 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) simulation experiments and were comprehensive and understandable. They indicated that access to the LMS was effortless and convenient anytime, anywhere, with assessment tests, practice exercises, and online practice instructions. Learners could take initiative to study the CPD program online anytime, anywhere. • Online interactive support was important. Most teachers believed that the online professional support provided by the core teacher training lecturers for self-learning was rational with adequate online video lectures, easy observation and provided a space for discussion with colleagues. IT support for access and use of the LMS was timely and effective. Learners could promptly: (i) contact core teacher training lecturers to discuss professional issues, and (ii) could participate in regular discussions with colleagues through several channels on social networks such as Facebook, Zalo, email, messengers, etc. Rating of Overall Efficacy 33. The overall efficacy rating of substantial. As indicated, both PDO indicators were exceeded, and all IRIs were met or exceeded. ETEP enhanced MOET’s capacity in strategic management and, by extension, the strengthening and institutionalization of school-based CPD for teachers and principals which includes: (i) issuing and implementing relevant teacher professional development policies and regulations with extensive stakeholder consultation, (ii) establishing and monitoring new professional standards for teachers and principals, (iii) identifying teacher and school-level demand for school-based CPD, (iv) assessing the effectiveness of CPD interventions and ensuring their quality via enforcement of rigorous quality process in CPD materials and delivery at qualified teacher training institutions, and (v) using assessment and evaluation to shape CPD programs and initiatives. The MOET capacity in performance-based budgeting and allocation has also been further strengthened based on the implementation of LTTU PAs which was the first time PAs had been used in Bank-support education programs. The ETEP LTTUs have significantly improved their capacity in: (i) strategic planning and implementation of plans using the PDCA cycle and key performance indicators; (ii) research in educational science and creation of more demand-driven teacher education curriculum through on-lines courses and blended programs; and (iii) ICT management along with new ICT infrastructure that allowed for the continuation of activities during the COVID-19 pandemic and enabled the LTTUs to develop integrated management information systems and blended learning platforms. LTTUs have also established the Teacher-Learner Community (between LTTU faculty and core teachers) and communities of practice in competency-based teacher education (among faculty members of all LTTUs) and have strengthened partnerships between the LTTUs and provincial governments in continuous assessment of CPD needs and school-based professional development support for teachers and principals. Moreover, teachers and principals who benefited from the ETEP training program have improved teaching and management practices. The DLIs were achieved and put in place the needed mechanisms, monitoring tools, etc., that were necessary to operate the Program, and these mechanisms and tools have been used at scale to train hundreds of thousands of teachers a year. All are leading to systemic change and a paradigm shift for teacher training and continuous professional development. C. JUSTIFICATION OF OVERALL OUTCOME RATING 34. The overall outcome is rated Satisfactory. ETEP’s objectives were Highly relevant at appraisal and continue to remain so today. The Project directly responded to the key issues and challenges facing Vietnam’s systemic challenges within the teacher training system, particularly related to continuous professional development as well as the World Bank’s CPF 2018-2022. In terms of its efficacy, the project is rated Substantial having met or exceeded all PDO indicators and IRIs. As indicated, the DLIs established the foundational mechanisms for a paradigm shift and systemic change in the delivery of teacher education and continuous professional development for teachers and principals. Page 24 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) D. OTHER OUTCOMES AND IMPACTS (IF ANY) Not applicable. Gender and Minorities 35. ETEP was a gender-informed project that also had a focus on ethnic minorities in remote areas. ETEP sought to ensure that female teachers and principals had equal access to regular and continuous professional development training opportunities and that the training opportunities were relevant to the cultural needs of ethnic minorities. During ETEP implementation, consultation workshops were organized with women and ethnic minorities on the development and implementation of the CPD program. ETEP issued consultation guidelines to facilitate meaningful consultation and to inform participation of beneficiary principals and teachers about ensuring inclusion of ethnic minorities and vulnerable groups, through a process of free, prior and informed consultation. In addition, the CPMU, in conjunction with the eight LTTUs, organized a communications strategy and plan 33 to target women and ethnic minorities to build awareness of the CPD program and to ensure their participation in all aspects of the CPD program. The results areas also had specific areas requirements that focused on females and minorities: (i) result area 2: the new teacher professional standards included a requirement for mastery of ethnic minority languages as required in the new teacher professional standards and TEMIS provides gender and minority disaggregated data so that DOETs could focus on the CPD needs of women and ethnic minorities; and (ii) result area 3: strongly encouraged DOETs to select ethnic minority teachers and principals to become core teachers and principal advisors, as guided in MOET’s Official Note 3587 34. Selecting core teachers and principal advisors with ‘’gender balance’’ was also mentioned in 3587. Moreover, the CPD program was developed based on a non-bias framework provided by the Bank. 36. As a result, women made up 60 percent of the core teachers and principal advisors throughout ETEP implementation and ethnic minorities comprised approximately 10 percent of the core teachers and principal advisors. The 2021 TEMIS reports also provided gender disaggregated data on the female participation rate in the professional standards assessment. (See Table 9 for details.) Moreover, 2021 TEMIS data showed that 96.7 percent of all female teachers were satisfied with the new school-based CPD and that 97.13 percent of the female ethnic minority teachers were satisfied. These are important gender and ethnic minority participation-related accomplishments for the sector. Table 9: Female and Minority Teachers and Principals Participating in Professional Standards Assessment Number of teachers & Number of teachers & % of teachers & principals principals principals completing completing Professional School Year (SY) Professional Standard Standard Assessment 2020-2021 Assessment SY 2020-2021 SY 2020-2021 Total 877,499 836,453 95.3 Female 613,773 593,008 96.6 33 Communication activities are dynamic and diversified, inclusive of ethnic minorities and teachers from difficult areas and Including official mass media and online social networks; 169 news/articles were produced in the last 6 months; 3 communication conferences for local teachers and managers and 2 press conferences were organized. Many virtual learning communities have been created and are being active. 34Official Note 3587 required ethnic core teachers to have been working full-time 3 years and longer while normal core teachers were required to have at least 5 years’ experience in direct teaching at the time of selection. Page 25 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Ethnic minority 90,529 89,044 98.4 Female ethinic minority 58,185 57,280 98.4 Source: MOET Project Completion Report, 2022 Poverty Reduction and Shared Prosperity 37. While ETEP did not include any specific impact measurement of poverty reduction and shared prosperity, its objectives were meant to contribute to human capital formation by supporting the improvement of teaching and learning throughout Vietnam by supporting CPD for teachers and principals. In addition, ETEP wants to ensure equal access to regular and continuous professional development training opportunities in disadvantaged areas so that these areas could benefit from improved teaching. Throughout ETEP’s life, about 29 percent of the core teachers and principals from disadvantaged areas were consistently kept as key participants of the core team. In addition, there was a high participation rate of teachers and principals from disadvantaged areas in the CPD ETEP training modules because teachers/principals could access the training courses and support online. (See Table 10 for details.). These participation rates are in line with the representation of teachers/ principals from disadvantaged areas in the total population of teachers/ principals 35. This was particularly impressive because the disadvantaged areas are usually remote and often face challenges due to limitations in internet transmission. Moreover, a 2021 survey of teachers and principals from disadvantaged areas showed that: (i) 98.2 percent were participating in the professional standards assessment (compared to 95.3 percent nationally), (ii) 96.9 percent of teachers were satisfied with the new school-based CPD (96.5 percent nationally), and (iii) 96.6 percent of principals were satisfied with the new school-based CPD (96.8 nationally). Table 10. CPD Participation in the ETEP Program modules 1, 2, 3, 4, 5 and 9 Module Total Number of CPD Number of Participants in Percentage of Disadvantaged participants Disadvantaged Areas Area Participants Module 1 662,627 168,434 25.24 Module 2 631,571 163,595 25.90 Module 3 623,485 160,934 25.81 Module 4 572,065 144,185 25.20 Module 5 457,746 106,068 23.17 Module 9 172,894 40,242 23.28 Source: MOET Program Completion Report, 2022 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 38. Program Financing. As indicated, the Program was financed by an International Development Association (IDA) Credit of US$95 million. The financing instrument was a Program-for-Results (PforR) and was to be implemented by the MOET. The PforR was highly appropriate given that the Bank, through ETEP, was providing direct support for the government’s FCER and the MOET’s NTEP with a specific focus on the implementation of the new CPD teacher training programs. Additionally, the PforR provided an opportunity to: (i) bolster the capacity of the MOET to implement and 35 2020/2021 TEMIS data indicates a percentage of 23 percent. Page 26 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) manage the entire education reform through its own systems and procedures; (ii) use country systems to simplify disbursement, financial management, and procurement; and (iii) support the Ministry of Finance’s efforts to ensure that the technical agencies utilized overseas development assistance based on unified and coherent programmatic frameworks. 39. Program Design. The theory of change behind the Program was sound. The PDO was appropriate for the Program design given that the Program was specifically focused on the CPD aspects of the NTEP. The results framework (RF) was developed to measure achievements related to the implementation of the new CPD program and the results areas and their related DLIs were directly linked to the achievement of the PDO. The inclusion of specific resources for TA was an important design element that was effectively used for (i) elaborating and operationalizing policies and directives, (ii) reinforcing the implementation capacity, including monitoring and evaluation, of the Program; and (iii) supporting the development of the CPD program. As indicated, the two restructurings (2021 and 2022) did not affect the original theory of change but reflected the adjustments needed to accommodate implementation delays. These design revisions were highly appropriate. 40. The Program's technical design drew from globally recognized approaches to improving teacher education through training and ongoing coaching and mentorship at the school level. The component activities, as inputs to achieve the PDO, were appropriate and selected in collaboration with the MOET. The ETEP was aligned with Government strategic priorities as laid out in the GoV’s FCER and the MOET’s NTEP. It focused on four results areas that would lead to establishing an effective and well-functioning CPD system that would provide school-based support and training for teachers and principals. This was done through (i) improving capacity within the LTTUs to deliver training and technical support for teachers, principals and DOETs and the provincial level; (ii) developing a teacher and principal needs assessment system that would inform the CPD program; (iii) providing school-based support for teacher and principals, and (iv) ensuring teachers and principals had continuous access to CPD programs and resources. The inclusion of PAs for LTTUs was an important and highly successful design element that led to improved capacity and LTTU ownership of the reform process. The nationwide targeting of Program beneficiaries was highly appropriate given the introduction of a new CPD program aligned with the curriculum reform and the goal of overall systemic change within ETEP and the NTEP and, again, an integral part of the Project design. 41. Implementation Arrangements. The implementation arrangements for the Program were embedded within the formal sector structure. Embedding the implementation arrangements within the formal structures was an important design feature because it ensured that capacity would be developed at all levels to fully implement the new CPD program and the PforR. It was recognized that initial capacity constraints could slow early Program implementation however, the long-term benefits of system capacity would lead to long-term sustainability. The MOET was responsible for the strategic management of ETEP and, by extension, the strengthening and institutionalization of the CPD program. MOET was also responsible for: (i) operating the overall monitoring and evaluation (M&E) system for ETEP, (ii) overseeing the implementation of LTTU PAs, (iii) creating a Project Management Unit (PMU) that would be responsible for tracking the reporting on the DLIs and transmitting disbursement requests to the Bank; and (iv) hiring an IVA that would confirm the results reported by the LTTUs and MOETs related to the DLIs as well as provide LTTUs with quality assurance capacity development. The provincial DOETs were responsible for undertaking overall quality control of the CPD process at the provincial level through: (i) administrative oversight of the core teachers and principals; (ii) ensuring that CPD was implemented at the local level; (iii) tracking the needs assessments through TEMIS and associated training based on the assessments; and (iv) overseeing the evaluation of teacher’s standards. The LTTUs were responsible for implementing CPD improvements envisaged by the NTEP through training, researcher, online services and support to the core teachers and principals. As indicated, the LTTUs also signed PAs with the MOET which specified their needed institutional capacity Page 27 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) development based on the TEIDI assessments. The overall arrangements were instrumental for ensuring system-wide implementation and systemic improvements. 42. Risks Assessment. The overall risk for the original Program was rated substantial. Key risks and appropriate mitigation measures were identified during preparation. Risks were related to the technical design, capacity constraints, collaboration between stakeholders, and fiduciary because of the limited experience with the PforR instrument. The Program’s activities were designed to directly address risks through (i) inclusion of technical assistance, (ii) inclusion of capacity development activities throughout the system, (iii) inclusion of LTTU PAs that set criteria for promoting collaboration with other LTTUs and DOETs, (iv) inclusion of PAs between MOET and LTTUs to ensure timely provision of resources to LTTUs along with capacity development related to LTTU internal audits to mitigate fiduciary risks, (v) utilization of the TEIDI tool to measure the effectiveness of providing quality teacher training and professional development that was needs-based, and (vi) inclusion of the IVA to ensure the technical design and DLIs were achieved as designed. B. KEY FACTORS DURING IMPLEMENTATION 43. There were numerous factors that played a role in the challenges and successes of the Program during implementation. 44. Factors subject to Government and/or implementing entities’ control. Overall, the Program's success and achievements were due to the work of the MOET, LTTUs and DOETs. However, there were three years where implementation progress (IP) was rated unsatisfactory and moderately unsatisfactory due to extended delays in implementation. The challenges the government faced were as follows: 45. Delays in program effectiveness. Program implementation was delayed initially for one year due to two conditions of effectiveness not being met. The World Bank Board of Directors approved ETEP in June 2016 and it took until June 2017 for the government to sign the financing agreement. In addition, completion of the program operations manual (POM) (a condition of effectiveness) was only completed in June 2017. The Bank provided considerable support through the Korean-World Bank Partnerships Facility to build capacity within the MOET to prepare the POM. 46. Implementation delays. In 2017, there were delays in the allocation of funds for ETEP due to the delay in budget registration in the Medium-Term Investment Plan (MTIP) and the delay in the program financing regime, which led to delays in signing LTTU PAs. The delays in funds allocation were related to the GoVs reforms in public financial management (i.e., the implementation of the Law on Public Investment, the Law on State Budget and decrees and circulars related to the management of public financial resources and overseas development assistance (ODA) and concessional foreign loans) that disrupted the process of preparing, allocating and approving the annual budgets of the PMU and LTTUs. In particular, Vietnam's state budget management regulation issued in 2018 only allows to use concessional ODA loans for civil works and goods, but not for capacity building activities. This regulation presented a major obstacle to achieving Objective 1. MOET and LTTUs worked to mobilize other funding sources to carry out capacity building activities. Funds were also mobilized from provinces to finance e-learning and professional development activities for teachers and principals. 47. By 2018, some of the constraints had been addressed, and progress was being made, but there were still challenges. The PMU did not have the technically capable staff to support the inclusion of international practice in the development and appraisal of teacher education programs nor the development of the CPD modules. The PMU also had Page 28 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) challenges in hiring the IVA needed to ensure monitoring and validation of Program activities. There were also weak and fragmented institutional implementation arrangements with roles and responsibilities not well defined for the LTTUs and DOETs. There were delays within the MOET related to the development and implementation of the LMS and TEMIS. Initially, MOET was going to develop a new LMS and TEMIS system. However, the development and management of these systems proved to be technically challenging for MOET. These challenges led the Bank to draft an issues paper outlining the actions needed to address the challenges. 48. By the November 2019 mid-term review (MTR), the MOET had made significant progress in: (i) giving LTTUs autonomy in planning and budgeting of ETEP activities 36 that were directly linked to DLI outcomes, (ii) clearly articulating ETEP institutional arrangements for implementation 37, and (iii) finalizing agreements on the implementation and management of the LMS and TEMIS38 through existing systems. All these actions led the Bank to upgrade IP to MU. There were still challenges related to DLI achievements because of the delays outlined above. As a result, and as previously indicated, the GoV requested a restructuring to reduce the capacity development activities in two LTTUs, removed the development of a new LMS, used existing technology for the LMS, and cancelled some of the IDA credit related to DLIs that had been delayed. Once all of these arrangements were in place, disbursements doubled, and there was a strong turnaround in implementation with ETEP. The improvements were such that by 2021, IP was upgraded to moderately satisfactory; by closing, IP was upgraded to satisfactory, and the development objectives were achieved. 49. Factors subject to World Bank control. ETEP benefitted from the fact that there was a consistent World Bank team that supported implementation from Hanoi. This allowed the team to interact regularly with the MOET, PMU, LTTUs, DOETs and other government units, which facilitate the implementation of the Program. There were regular implementation support missions and a formal MTR as well as informal bi-monthly and sometimes daily Bank staff communications with the PMU and LTTUs regarding delays in implementation. Prior to the MTR the Bank conducted a formal assessment of implementation delays and provided the GoV and Bank management with an issues paper outlining recommendations for improving implementation performance. This proved to be instrumental in guiding the 2020 restructuring and led to rapid improvement in implementation and the successful completion of the Project. The PforR instrument was effective in ensuring that the Government focused on prioritizing results that led to a paradigm shift in the delivery of teacher and principal professional development. The PforR also led a continuous dialogue with the Government regarding progress towards the completion of the activities incentivized by the DLIs/DLRs. All-in-all there was strong and continuous technical support and capacity building from the Bank in-country team during implementation. 50. Exogenous Factors. In 2017 there were delays in updating the curriculum at teacher training institutions under NTEP due to delays in the issuance of the general education curriculum of basic education by MOET. The COVID-19 outbreak initially contributed to delays in implementation because the face-to-face teacher and principal training could not take place. However, COVID-19 also played a role in accelerating the implementation of on-line self-paced CPD training which, as indicated, beneficiaries have judged to be a real benefit of ETEP. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME 36 Ministry’s Decision 4318/ QD-BGDDT dated November 13, 2019. 37 Decision 5140/BGDDT-GDTrH dated November 11, 2019. 38 Decision 4315/QD-BGDDT dated November 12, 2019 and Decision 1180/QD-BGDDT dated November 5, 2019. Page 29 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) A. QUALITY OF MONITORING AND EVALUATION M&E Design 51. The PforR’s links between the results areas, outputs, outcomes and the PDO were sound (as shown in Figure 1). The PDO was concise, with two sub-objectives. It was specified at appraisal and remained the same throughout the Program's life. PDO-level indicators, intermediate outcome indicators and DLIs/DLRs were clearly defined at appraisal. The stated results areas and indicators sought to establish and implement the needed governance structure and frameworks to ensure systematic change within the teacher and principal continuous professional development. The M&E design was developed to monitor and report on progress toward meeting the targets of the PDO and intermediate indicators in the Results Framework through routine monitoring and reporting by the PMU. The M&E also included LTTU PAs, a new and innovative design feature for donor support activities with the education sector and utilization of the TEIDI instrument for greater transparency and ease of tracking results. The online platform (TEMIS) was included so that DOETs (nationwide in 63 provinces) could track the teacher and principal training needs and activities as well as performance assessments. In addition, the design included the development of an LMS for the delivery of online training. Finally, the design included an IVA to confirm the achievement of DLIs/DLRs which were also linked to indicator targets. The IVA hired was the higher education quality unit of the Vietnam National University, Hanoi, which provided capacity development for LTTUs in quality assurance during the verification process. M&E Implementation 52. Reporting data for the RF indicators was done throughout the project's life. The selected performance indicators were tracked regularly, and the RF was updated on a timely basis for supervision missions, including the MTR. The MOET signed a contract with the IVA to conduct their independent assessments of DLI/DLR achievements which were then used for the disbursement of ETEP IDA resources and provided all LTTUs with training in quality assurance to build their capacity. Moreover, (i) all 8 original LTTUs developed PAs based on the TEIDI instrument; (ii) the online LMS system was developed; and (iii) the TEMIS was developed. The PMU developed and promulgated a quality assurance process for CPD material development, training of core teachers and principal advisors and teachers/ principals which was the legal basis for LTTUs to strengthen supervision and quality assurance of CPD material development. This involved developing M&E regulations and guidelines and reporting forms for M&E between the CPMU and the LTTUs. Moreover, LTTUs established M&E departments within the universities. These departments were responsible for monitoring: (i) implementation of their individual PAs; (ii) agreements related to the achievement of DLIs and outcome indicators based on the results framework; and (iii) submission of LTTU evaluation reports based on PMU guidance. The LTTU M&E departments also monitored the quality of CPD material development and other program activities following QA processes introduced by ETEP PMU with support from participants from the quality assurance department and other key technical departments. M&E Utilization 53. Data was made available regularly for the results framework and was used to: (i) analyze project progress toward reaching the project development objectives and suggest remedial actions when risk to their achievement occurs, (ii) determine modifications for the two project restructurings, and (iii) inform the analysis of the ICR. The IVA reports were used to validate DLI/DLR reports and disburse IDA resources. The TEIDI assessments were used to develop LTTU PAs and the PAs for use by the MOET to allocate resources to the LTTUs for agreed activities and actions. The LTTU M&E departments ensured the quality of the CPD materials that were ultimately rolled out to all 63 provinces in Vietnam Page 30 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) through the LMS. Provinces generated TEMIS reports on teacher assessments against the professional standards and used this information to determine teacher professional development training needs for both online training and face- to-face training and support. The multi-pronged CPD material quality assurance established for CPD by ETEP has now been institutionalized throughout the system—a substantial achievement. Justification of Overall Rating of M&E Quality 54. Based on the analysis above, the overall rating for M&E is Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental and Social 55. Safeguards. Overall, the Environmental and Social Systems Assessment (ESSA) determined that ETEP was not expected to have negative environmental impacts. Although some of the LTTU upgrading activities might cause negative environmental impacts associated with civil works, such as generation of noise, dust, wastes including solid waste and wastewater, and health and safety risks, they would take place exclusively within university premises. Thus, potential negative environmental impact was assessed to be minor, temporary, localized, and could be mitigated. The negative environmental impacts during construction were to be adequately mitigated through the application of Environmental Codes of Practice (ECOP) and Worker’s Codes of Conduct (WCC) as well as communication activities in construction contracts to manage Environment, Health, and Safety (EHS) issues at construction sites. The ESSA was disclosed in the World Bank InfoShop on June 8, 2016 and in-country on April 12, 2016. Safeguards performance was monitored and supervised regularly during project implementation by the PMU, relevant consultants, and responsible LTTU authorities and rated satisfactory (S) in Implementation Status and Results Reports (ISRs) with the exception of one six-month period in January 2021 when safeguards were rated moderately satisfactory (MS). The downgrade was related to the fact that: (i) not all bidding and contractual documents issued by the LTTUs had covered environmental provisions as required by the program operations manual nor the Bank’s guidance on COVID-19 pandemic control and prevention measures and (ii) the LTTUs had not provided a sound management plan to avoid dumping of expired ICT equipment and infrastructure. By August 2021, these issues had been resolved, and safeguards were upgraded to satisfactory. A Grievance Redress Mechanism (GRM) was established for ETEP. The GRM was a multi-pronged mechanism that consisted of (i) the ETEP website which contained a link for the public to provide complaints and comments; (ii) a feedback survey mechanism for core teachers and principals that participated in LTTU training; (iii) MOET provided software that was developed to house complaints within the system which also included ETEP complaints; (iv) LTTU hotlines; and (v) People’s Inspectorates that were already established within each LTTU. This information was made available through the program operations manual and ETEP’s Official Note 85 on information disclosure and handling of complaints and denunciations. 56. Social Safeguards. During ETEP implementation, the social safeguards were continuously rated as satisfactory. ETEP supported the development of detailed guidelines for the facilitation of meaningful consultation with vulnerable groups and a detailed communications plan that was fully implemented by the PMU and LTTUs. The LTTUs: (i) integrated the “Bias Avoidance Framework” in the development of the CPD training materials; (ii) ensured that the training materials included regional differences as well as ethnic minority and gender issues; (iii) disaggregated teacher feedback by ethnicity, gender and disadvantaged areas in their reports; and (iii) extended the e-learning time for core teachers and principals in the ethnic minority and remote areas because of connection difficulties. LTTUs also conducted consultations and surveys to gain feedback from female teachers, ethnic minority teachers and teachers from disadvantaged areas to improve their activities and to report to the ETEP PMU on ways to implement ETEP activities for these groups better. Page 31 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Fiduciary 57. Financial management. FM performance rating was satisfactory or MS throughout project implementation except between December 2018 and December 2019 when it was rated moderately unsatisfactory (MU). This was largely due to inefficient ETEP budgeting and planning, specifically for (i) 2018 financial plans for LTTUs, which were approved late and (ii) prolonged delays in the confirmation and allocation of the Medium-Term Investment Plan (MTIP) budget for the ETEP, which led to delays in the allocation of capital expenditures for the LTTUs. By December 2019, the FM performance rating was MS and remained MS until ETEP closing. The MOF maintained a separate account for ETEP funding to ensure the monitoring of ETEP funded activities. In addition, an IVA was contracted in accordance with the financing agreement. Qualified FM specialists were maintained throughout implementation and when needed, capacity development workshops were conducted by the Bank FM specialists for the PMU and LTTU FM staff to help implement the FM requirements. The required audits were submitted to the World Bank in accordance with the Financing Agreement, and over the Program period, the auditors’ opinions were unqualified (clean). The final audit report is due December 31, 2022. Disbursements were made based on the IVA validation of all the DLI/DLR achievements as required by the financing agreement. The original project financing was US$95,000,000 equivalent. As indicated, there were two cancellations of Program funding for a total of US$46,959,709 equivalent. The total Program disbursements were approximately US$48,040,000 equivalent or 100 percent of the revised Program costs. 58. Procurement. The procurement performance rating was satisfactory and moderately satisfactory throughout project implementation except between December 2018 and December 2019 when procurement was rated MU. This was largely due to slow progress in procurement of the important packages related to contracting the IVA and the development of the LMS-TEMIS as well as delay in the MTIP as indicated above. Again, as indicated previously, the delays in the LMS-TEMIS resulted from MOET’s indecision regarding how the system would be developed and maintained. By December 2019, these issues had been resolved, and procurement was rated MS and S for the remainder of ETEP implementation. Procurement activities were being implemented in compliance with the Procurement Action Plans (PAPs), and ultimately, all procurement activities were completed as outlined in the PAPs. During implementation, World Bank procurement staff provided training to the PMU and LTTUs to ensure compliance with the procurement guidelines. Procurement plans were updated regularly and submitted to the Bank and always of good quality. C. BANK PERFORMANCE Quality at Entry 59. The program preparation team ensured that the design was closely aligned with Bank’s 2012-2016 CPS that consisted of three pillars (i) strengthening Vietnam’s competitiveness in the regional and global economy; (ii) increasing sustainability of its development and (iii) broadening access to economic and social opportunity. The FCER shared the first and third objectives of the CPS and aimed to prepare Vietnam for greater integration in the global economy by providing students with new skills and competencies. The NTEP was seen as a key part of the FCER and the ETEP support the catalytic component of the NTEP—teacher continuous professional development. The PDO was precise, and key indicators were appropriate for measuring progress toward achieving the PDO. The results areas along with the DLIs/DLRs were directly linked to the PDO. As indicated, the DLIs, and associated DLRs, were well articulated, each with a clear definitions and protocol to evaluate achievement and were reasonable in terms of expected achievement during implementation. The PforR financing instrument was the correct instrument to effect the needed changes to teacher professional development system. The preparation team carefully considered the implementation Page 32 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) and M&E arrangements as well as the appropriate risks and incorporated design features to mitigate them. Based on the above, the quality at entry for ETEP was Satisfactory. Quality of Supervision 60. Key to the project’s achievements was the consistent supervision by the World Bank team in-country that comprised the needed technical expertise to support the nationwide implementation of ETEP. Overall, there were 12 World Bank review and implementation support missions, which included an MTR. In addition to the traditional Bank implementation support, the in-country Bank team was able to reach out as often as necessary to the Program teams to solve implementation challenges. The project team was actively engaged in supporting the government in its efforts to implement the project. Whenever implementation challenges arose, the World Bank team worked with the government to find appropriate solutions that would not compromise the integrity of the design. The supervision teams consistently reported on safeguards, FM, and procurement progress during supervision missions and worked with the Program management units to build their capacity in these areas. They also systematically documented project progress in aide memoirs, back-to-office reports, and ISRs, keeping the World Bank management informed of progress and providing the foundation for the ICR analysis. The Bank had an ongoing relationship with the MOET representatives to ensure that the project’s goals were achieved. The Bank also included technical and area-specific experts when needed on supervision missions. The World Bank’s willingness to respond to the government’s request for the two Program restructurings was an important aspect of the Program that led to the achievements of ETEP and the systemic changes within teacher education and CPD. The Bank’s good collaboration on implementation and coordinated work with MOET contributed to the many ETEP successes. Based on this, the quality of supervision is rated Satisfactory. Justification of Overall Rating of Bank Performance 61. Based on the above, the overall Bank performance rating is Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 62. The risk of sustaining the development outcome is low. Firstly, ETEP developed a complete CPD mechanism that includes an LMS that has demonstrated that it can deliver online and face-to-face training and has contributed to systemic change and a paradigm shift in how CPD is conducted for teachers and principals. Moreover, the core teachers and principals now provide teachers with coaching and mentorship at the school level to help them improve their teaching. This system is now delivering training at a national scale with high levels of satisfaction of participants, all creating an expectation that it will continue to be the core teacher training system. Secondly, the LTTUs have shown enhanced capacity and a demonstrated commitment of their leadership and mid-level managers through the achievement of a level 5 TEIDI score—they did not just maintain their scores but improved them through ETEP. Thirdly, the ETEP deliverables have been taken over by the Department of Teacher and Education Manager (DTEM) within MOET. DTEM has issued guidelines for 2022 TEMIS reporting and teacher training with the CPD model and committed to annual guidance in subsequent years. This includes utilizing the CPD material development quality assurance process, coordinating with the LTTUs in developing the materials for following training modules and amending Circular 19 39 on teacher CPD. Fourthly, the DTEM and ICT department have agreed to ensure the inter-operationalization between TEMIS and the education sector database and to continue the ETEP ‘s CPD blended delivery model. Finally, it has been agreed 39MOET Circular 19/2019/TT-BGDDT dated November 12, 2019 is the regulation on CPD of teachers and principals of preschools and general education schools, and continuing education centers. Page 33 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) that TEMIS will be maintained until 2026 by the current provider and that MOET is committed to developing an in-house TEMIS as specified in the Ministry’s information technology master plan. V. LESSONS AND RECOMMENDATIONS 63. Lesson 1. The PforR and ETEP gave MOET the opportunity to think about comprehensive reform and alignment of the system. ETEP, in parallel to RGEP, aligned the new general education competency-based curriculum with the teacher and principal CPD and training. This required: (i) moving from a centralized professional development model to a continuing, on-site/blended model (using in-person and online training through LMS); (ii) building the capacity of LTTUs to participate effectively in the reform, (iii) preparation of training modules, recording of participation to link to incentives to participate, (iv) creation of core teachers and principals to support teachers/principals and giving them the time to do their work, and (v) effectively linking the financing to program results. ETEP has shown that results-based financing can lead to enhanced accountability, systemic change and a paradigm shift in the delivery of services. ETEP was Vietnam’s first experience with results-based financing in education and now serves as a model for results-based financing for GoV given that the GoV is making a gradual transition to a performance-based state budget allocation. 64. Lesson 2. For quick and effective crisis response, leveraging existing (online/remote) systems is key. As indicated, the LTTUs had developed CPD e-learning courses with ETEP support as well as the infrastructure to deliver these courses. These e-learning courses became increasingly important during COVID-19 because they allowed for the continuation of Program activities and mitigated some of the negative impacts on the delivery of the CPD. ETEP created an effective on-line CPD model that could easily be leveraged at the time of COVID-19 for general education school teachers that was decentralized that allowed for the successful completion of ETEP. 65. Lesson 3. The creation of COPs that linked LTTUs with schools and other stakeholders and the provincial and district level created an important feedback loop throughout the system. Teacher-learner communities and COPs provided LTTUs with important links to teachers and principals at the school level and critical insights into classroom which enhanced LTTUs practices in both the in-service and pre-service teacher education. These connections were strengthened during ETEP implementation and have been institutionalized. This feedback loop for both in-service and pre-service providers to understand the challenges that teachers face in the classroom is essential for improving the quality of the system—a substantial achievement. 66. Lesson 4. Performance agreement contracts are an important mechanism for providing autonomy to institutions as well as ensuring transparency and effective monitoring of institutional reform. The use of PAs proved to be an important mechanism for LTTUs to negotiate annual budgets based on targets and results. The management culture within LTTUs is now shifting from an inputs-based orientation to one focusing on results based on their TEIDI assessments. The use of PAs as a financing mechanism pushed the Government to think beyond business as usual for financing LTTUs while also focusing on the system's quality. The broader impact has been that PAs have been recognized and are being considered for expansion within the education system. 67. Lesson 5. Inclusion of specific gender and ethnic minority criteria for participation is critical for ensuring equal access to Program benefits. ETEP focused specifically on the participation of women and ethnic minorities in disadvantaged areas. There were consultation workshops organized with women and ethnic minorities on the development and implementation of the CPD program. A communications strategy targeted women and ethnic minorities to build awareness of the CPD program which led to active participation of women and minorities as core teachers and principals. and to ensure their participation in all aspects of the CPD program. As a result, women made Page 34 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) up 60 percent of the core teachers and principal advisors throughout ETEP implementation, and ethnic minorities comprised approximately 10 percent of the core teachers and principal advisors. The TEMIS reports also provide gender and minority disaggregated data so that DOETs can continue to focus on the CDP needs of women and ethnic minorities. 68. Lesson 6. Appropriate institutional implementation arrangements are critical for ensuring the implementation of a broad reform agenda. Given that this operation was supporting a part of a broader reform, having the MOET fully manage the activities was an appropriate design element and essential element for implementation. As indicated, initially, there were weak and fragmented institutional implementation arrangements with roles and responsibilities not well defined for the LTTUs and DOETs. Once MOET clearly defined the roles of the implementation units, there was considerable improvement in implementation activities. In addition, the MOET, PMU, and LTTUs began to use technical assistance in strategic areas that brought in the needed expertise to support the inclusion of international best practices in the development and appraisal of the teacher education programs and the development of the CPD modules. Even though there was a PMU, the substantive/educational work was done by units in the MOET, which enabled the relevant connections to the broader reform as well as strong systemwide capacity development. . Page 35 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) ANNEX 1. RESULTS FRAMEWORK, DISBURSEMENT LINKED INDICATORS, AND PROGRAM ACTION PLAN Annex 1A. RESULTS FRAMEWORK (i) PDO Indicators Objective/Outcome: Strengthened teacher education institutions to enhance teacher and principal effectiveness Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improved performance of the Text Baseline established 15% increase over the TEIDI score of each Hanoi LTTU 5.0; Thai six Lead Teacher Training for each LTTU baseline ịn TEIDI for LTTU increases to at Nguyen LTTU 4.72; Universities to provide CPD each LTTUscore least level 5 (>4.43) Vinh LTTU 4.86; Hue as measured by the Teacher LTTU 4.95; Danang Education Institutional LTTU 4.88; HCMC Development Index (TEIDI) LTTU 4.91 31-Dec-2017 30-Jun-2022 30-Jun-2022 30-Jun-2022 Comments (achievements against targets): Achieved (100 percent). Page 36 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of teachers and Number 0.00 480,000.00 320,000.00 637,279.00 principals satisfied with school-based continuous 30-Jun-2016 30-Jun-2022 30-Jun-2022 30-Jun-2022 professional development Comments (achievements against targets): Exceeded (199 percent). (ii) Intermediate Results Indicators Results Area: Improved capacity of Lead Teacher Training Universities to enhance teacher and principal effectiveness Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of LTTUs that sign Number 0.00 8.00 6.00 6.00 Performance Agreements with MOET. 30-Jun-2016 30-Jun-2022 30-Jun-2022 30-Jun-2022 Comments (achievements against targets): Page 37 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Achieved (100 percent). Results Area: Development of teacher and principal training needs assessment system to informed CPD program development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of TEMIS-generated Number 0.00 45.00 45.00 45.00 annual reports on CPD needs by DOET. 30-Jun-2016 30-Jun-2022 30-Jun-2022 30-Jun-2022 Comments (achievements against targets): Achieved (100 percent). Results Area: School-based and continuous professional development for teachers and principals Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 38 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Number of deployed core Text No core teachers/ 28,600 28,600 30,134 core teachers teachers and/ or principal principal advisors and principal advisors advisors having completed completed any completed six priority different priority training training modules training modules modules set for the year 30-Jun-2016 30-Jun-2022 30-Jun-2022 30-Jun-2022 Comments (achievements against targets): Achieved (105 percent). Results Area: Teachers and principals have access to CPD programs and resources through a ICT based system Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of teachers and Text No teachers/ 640,000 480,000 527,532 teachers/ school principals having principals having school principals have completed different completed any online completed 5 mandatory online modules of module of the CPD mandatory online the CPD program set for the program through the modules of the CPD year through the Learning LMS(s) program through the Management System(s) LMS(s) (LMS(s)) 30-Jun-2016 30-Jun-2022 30-Jun-2022 30-Jun-2022 Comments (achievements against targets): Page 39 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Achieved (110 percent). ANNEX 1B. DISBURSEMENT LINKED INDICATORS DLI 1: Institutional capacity of LTTUs to support the new Continuous Professional Development system for teachers and principals enhanced (Text) From the effective date From January From January From January From April Baseline of the FA to to December to December 2020 to 2021 to Total December 2018 2019 March 2021 March 2022 2017 A total of A total of Up to eight (8) Up to six (6) Up to six (6) eight (8) eight (8) LTTUs conduct LTTUs conduct LTTUs conduct LTTUs have LTTUs have LTTUs have an annual an annual an annual inadequate conducted conducted TEIDI self TEIDI self TEIDI self capacity to TEIDI baseline TEIDI baseline assessment assessment assessment Original values support the assessment assessment and MOET and MOET and MOET continuous and signed and signed publishes the publishes the publishes the professional Performance Performance TEIDI scores TEIDI scores TEIDI scores development Agreements Agreements on its website. on its website. on its website. with MOET. with MOET. Four LTTUs Four LTTUs Eight LTTUs Six LTTUs Six LTTUs Actual values conducted conducted conducted an conducted an conducted an baseline baseline annual TEIDI annual TEIDI annual TEIDI Page 40 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) assessment assessment assessment assessment assessment and signed and signed and MOET and MOET and MOET Performance Performance published the published the published the Agreements Agreements TEIDI scores TEIDI scores TEIDI scores with MOET with MOET on its website. on its website. on its website. Allocated amount ($) 3,210,000.00 3,210,000.00 4,280,000.00 4,210,000.00 0.00 14,910,000.00 Disbursed amount ($) 0.00 6,420,000.00 4,280,000.00 4,210,000.00 0.00 14,910,000.00 Comments (achievements against targets): The DLI was fully achieved and all DLRs were verified. DLI 2: Systems are in place to:(i)measure whether teachers/principals meet new professional standards;(ii) track & record evaluations of CPD programs;(iii) record assessments CPD needs of teachers/principals (Text) From the effective date From January From January From January From April Baseline of the FA to to December to December 2020 to 2021 to Total December 2018 2019 March 2021 March 2022 2017 Existing systems MOET issues a Up to thirty Up to forty- to assess circular (30) DOETs five (45) teacher and describing publish a DOETs publish Original values principal CPD guidelines and TEMIS- a TEMIS needs are not procedures to generated generated aligned with assess CPD Annual CPD Annual requirements of teachers and Report for Report for FCER nor principals their their Page 41 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) adequately against new respective respective inform the professional province. province. development of standards. CPD programs MOET issued 0 Thirty DOETs Forty-five Circulars published in DOETs describing 2020 a TEMIS- published in guidelines and generated 2021 a TEMIS- procedures to CPD annual generated assess report for its CPD annual teachers and respective report for its principals province. The respective against new annual reports province. The professional present trends annual reports standards. on present trends Actual values 0 professional on standards, professional beneficiary standards, assessments beneficiary of CPD assessments programs and of CPD assessments programs and of CPD needs assessments of teachers of CPD needs and principals. of teachers and principals. Allocated amount ($) 0.00 1,426,500.00 0.00 2,859,000.00 714,500.00 5,000,000.00 Disbursed amount ($) 0.00 1,426,500.00 0.00 2,859,000.00 714,500.00 5,000,000.00 Page 42 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Comments (achievements against targets): The DLI was fully achieved and all DLRs were verified. DLI 3: Number of core teachers and principal advisors selected and trained to provide systematic face-to-face school based CPD to teachers and principals (Text) From the effective date From January From January From January From April Baseline of the FA to to December to December 2020 to 2021 to Total December 2018 2019 March 2021 March 2022 2017 A corps of LTTUs provide LTTUs provide LTTUs provide LTTUs provide trained school- basic training basic training basic training basic training based to 25,000 core to 25,000 core to 25,000 core to 25,000 core specialists that teachers and teachers and teachers and teachers and provide 3,600 3,600 3,600 3,600 Original values systematic face- principal principal principal principal to-face school- advisors advisors advisors advisors based CPD to teachers and principals does not exist. Actual values 0 0 LTTUs LTTUs LTTUs Page 43 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) provided basic provided basic provided basic training to training to: (i) training to 19,496 core 804 core 25,000 core teachers and teachers and teachers and 3,555 15 principal 3,600 principal advisors on principal advisors on Module 1 advisors on module 1 (batch 2) (ii) Modules 5, 9. (batch 1). 20,300 core teachers and 3,570 principal advisors on Module 2 (iii) 24,975 core teachers and 3,600 principal advisors on Modules 3, 4. Allocated amount ($) 0.00 0.00 0.00 8,705,870.00 526,202.00 9,232,072.00 Disbursed amount ($) 0.00 0.00 0.00 8,705,870.00 526,202.00 9,232,072.00 Comments (achievements against targets): The DLI was partially achieved (94 percent) and all DLRs were verified. DLI 4: Number of core teachers and principal advisors providing school-based continuous professional development to teachers and principals (Text) Page 44 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) From the effective date From January From January From January From April Baseline of the FA to to December to December 2020 to 2021 to Total December 2018 2019 March 2021 March 2022 2017 Up to 18,000 Up to 25,000 core teachers core teachers and 2,700 and 3,600 A corps of principal principal trained school- advisors advisors based provide provide specialists that school-based school-based provide continuous continuous Original values systematic face- professional professional to-face school- development development based CPD to to teachers to teachers teachers and and principals and principals principals does during Period during Period not exist. 4, as per their 5, as per their individual individual annual plan. annual plan. 0 0 13,260 core 25,000 core teachers and teachers and 1,422 3,288 principal principal Actual values 0 advisors advisors provided provided school-based school-based continuous continuous professional professional Page 45 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) development development to teachers to teachers and principals, and principals, as per their as per their individual individual annual plan. annual plan. Allocated amount ($) 0.00 0.00 0.00 2,099,526.00 0.00 2,099,526.00 Disbursed amount ($) 0.00 0.00 0.00 2,099,526.00 0.00 2,099,526.00 Comments (achievements against targets): The DLI was partially achieved (71 percent in period 4 and 99 percent in period 5) and all DLRs were verified. DLI 5: Number of teachers and principals who are satisfied with ICT-based interactive CPD services provided under the program (Text) From the effective date From January From January From January From April Baseline of the FA to to December to December 2020 to 2021 to Total December 2018 2019 March 2021 March 2022 2017 0 0 Up to 160,000 Up to 320,000 teachers and teachers and principals are principals are Original values 0.00 0 satisfied with satisfied with ICT-based ICT-based interactive interactive CPD services. CPD services. Page 46 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) 0 0 160,000 320,000 teachers and teachers and principals principals were satisfied were satisfied Actual values 0 with ICT- with ICT- based based interactive interactive CPD services CPD services Allocated amount ($) 0.00 0.00 0.00 1,440,000.00 1,400,000.00 2,840,000.00 Disbursed amount ($) 0.00 0.00 0.00 1,440,000.00 1,400,000.00 2,840,000.00 Comments (achievements against targets): The DLI was fully achieved and all DLRs were verified. ANNEX 1C. PROGRAM ACTION PLAN PAP TBL Achieved Action Timing Completion Measurement (Yes/No) The Recipient shall develop and execute a Due Date 05-Jun-2018 Yes A Capacity Building Plan developed, costed and Capacity Building Plan for the LTTUs to executed (in cooperation with LTTUs). ensure sustainable provision of teacher school based continuous professional development support, all in form and substance satisfactory to the Association. Page 47 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Comments: The Plan was finalized in December 2019 and was fully executed. The Recipient shall carry out a study – Due Date 30-Jun-2020 Yes Study validated by MOET under the terms of reference acceptable to the Association - to identify options and recommendations for financing teacher- continuous professional development. Comments: The study was finalized and validated by MOET. The Recipient shall disclose information on Recurrent Continuous Yes Evidence of information disclosure the Program’s activities to the public, including financial information throughout the implementation of the Program. Comments: Information on the program's activities, including financial information, is disclosed via the PMU/ LTTUs' websites and fan page. PMU issued a guideline in April 2019 on information disclosure, among others, to be followed by PMU/ LTTUs. The guideline was fully implemented. The Recipient shall maintain a system of Recurrent Yearly Yes Log of complaints and reports. receiving, recording and following-up on complaints, including anonymous ones. Page 48 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Comments: MOET maintains the system. During the proram, MOET recorded only one complaint In January 2022, related to Vinh LTTU’s infrastructure investment activities. Vinh LTTU submitted a full report to the Bank on January 28, 2022 and the case was fully resolved March 25, 2022. The Recipient shall maintain a database on Recurrent Yearly Yes Database verified complaints and denunciations from all the participating agencies and follows up on the resolution of those complaints and denunciations. Comments: MOET maintains the database. During the proram, MOET recorded only one complaint In January 2022, related to Vinh LTTU’s infrastructure investment activities. Vinh LTTU submitted a full report to the Bank on January 28, 2022 and the case was fully resolved March 25, 2022. The Recipient shall (a) review LTTUs Due Date 05-Jun-2019 Yes Internal audit capacity assessed and LTTUs capacity internal audit capacity; and (b) prepare building plans developed, costed, approved and and execute – or cause to be executed - executed (in cooperation with LTTUs). plans to build the capacity of internal audit function in LTTUs. Comments: This action is completed. The Recipient shall issue guidance on the Recurrent Continuous Yes Guidance on the performance based budgeting/ performance based budgeting/allocation allocation was issued by MOET. Page 49 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Comments: The action was completed. All proposals for detailed designs, Recurrent Continuous Yes n/a construction supervision and bids for civil works, whether below or above cost estimates, shall be evaluated; proposals and bids shall not be rejected only on the basis of minor, non-substantive deviations. Comments: Complied, as noted in the implementation support missions. The Recipient shall design and adopt a Due Date 05-Jun-2018 Yes Issuance of a circular on teacher continuous mechanism – satisfactory to the professional development which includes the Association - to coordinate teacher regulated coordination mechanism. continuous professional development initiatives and activities among teacher education providers and concerned teacher management department Comments: Circular 19/2019/TT-BGDDT was issued on November 12, 2019. At least 60 percent of contracts for Recurrent Continuous Yes n/a consultancies and 85 percent of contracts for works shall be competitively bid annually. Comments: Page 50 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Complied, as noted in the implementation support missions. No contract having cost estimate above Recurrent Continuous Yes n/a the thresholds for using DC/SSS method set forth in the national public procurement law and regulation will be awarded on DC/SSS basis. Comments: Complied, as noted in the implementation support missions. Firms on the local, national, or Bank Recurrent Continuous Yes n/a debarment list shall not be allowed to participate. Comments: Complied, as noted in the implementation support missions. The Recipient shall develop a guideline to Due Date 05-Jun-2018 Yes Formal issuance of the guideline facilitate meaningful consultation and inform participation of beneficiary principals and teachers, including those from ethnic minorities and vulnerable groups. Comments: The guideline was formally issued in November 2018. Page 51 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) The above referenced guideline, which will Due Date 05-Jun-2018 Yes n/a be implemented by LTTUs should fully operationalize the existing Vietnamese legislation with respect to ethnic minorities through a process of free, prior, and informed consultations. Comments: No significant issue was noted in the implementation support missions. The Recipient will prepare an Operational Due Date 05-Jun-2017 Yes OM agreed by the Bank Manual. Comments: This action was completed. The Recipient will execute the Operation Recurrent Continuous Yes Implementation Support Missions will review according to the Manual. implementation quality Comments: Complied, as noted in implementation support missions. Page 52 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Michel J. Welmond Task Team Leader(s) Thang Toan Le Procurement Specialist(s) Cung Van Pham Financial Management Specialist F. Halsey Rogers Peer Reviewer Michael F. Crawford Team Member Tobias Linden Peer Reviewer Michael Trucano Team Member Suhas D. Parandekar Team Member Keiko Miwa Peer Reviewer Franco Russo Team Member Chau-Ching Shen Team Member Binh Thanh Vu Team Member Juan Manuel Moreno Olmedilla Team Member Nguyet Minh Nguyen Team Member Simon Thacker Peer Reviewer Andrew B. Ragatz Peer Reviewer Dung Kieu Vo Team Member Khang Van Pham Environmental Specialist Shabnam Sinha Team Member Page 53 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) Huong Thi Lan Tran Team Member An Thi My Tran Team Member Nga Thi Anh Hoang Team Member Giang Tam Nguyen Social Specialist Keiko Inoue Peer Reviewer Fei Deng Team Member Evarist F. Baimu Counsel Anna Coronado Team Member Anh Lan Vu Team Member Supervision/ICR Dung Kieu Vo Task Team Leader(s) Thang Toan Le Procurement Specialist(s) Cung Van Pham Financial Management Specialist Mary A. Dowling Team Member Aristeidis Panou Counsel Khang Van Pham Environmental Specialist Thu Ha Le Counsel Huyen Thi Thanh Le Team Member David Jorge Baringo Ezquerra Social Specialist Quynh Thuy Hoang Social Specialist Sandra Beemer ICR Main Author Page 54 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY14 1.344 8,565.28 FY15 26.337 195,607.42 FY16 63.388 315,709.38 FY17 0 1,973.82 Total 91.07 521,855.90 Supervision/ICR FY15 0 2,211.33 FY16 0 0.00 FY17 41.781 344,364.77 FY18 24.330 297,022.65 FY19 18.988 152,105.12 FY20 16.053 120,140.20 FY21 15.054 114,646.81 FY22 18.600 128,459.05 FY23 4.475 26,338.91 Total 139.28 1,185,288.84 Page 55 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) ANNEX 3. PROGRAM EXPENDITURE SUMMARY Actual Expenditures (Disbursement) Source of Program Type of Co- Estimates at Financing (US$ million) Financing Appraisal Percentage of Percentage of Actual1 Appraisal Actual World Bank IDA US$95.00 US$48.042 50.56 100.0 Counterpart funding – Borrower US$5.00 US$2.543 50.08 101.0 central Government Counterpart Borrower funding – US$0.00 US$3.424 N/A 100.0 LTTUs Other Partners N/A N/A N/A N/A N/A Total US$100.00 US$54.00 54.00 100.0 1 The actual expenditures are based on final audit reports for 2017-2021 and the final draft audit report for 2022. 2 There were two cancellations of funds totaling US$46,959,709 equivalent. 3 There was one cancellation of funds totaling approximately US$2.5 million. 4 The counterpart funding by the LTTUs was not budgeted during appraisal process but was mobilized during the program implementation and accordingly recorded in the audit reports. Page 56 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) ANNEX 4. BORROWER’S COMMENTS In a letter dated November 1, 2022, the Government confirmed its agreement with the findings of the ICR and had no additional comments. The Government provided their final completion report to the Bank, which has been filed in WBDocs for information and reference. Page 57 of 58 The World Bank Vietnam Enhancing Teacher Education Program (P150060) ANNEX 5. SUPPORTING DOCUMENTS Project Appraisal Document, World Bank, 2016 Project Restructuring, World Bank, 2021 Project Restructuring, World Bank, 2022 Aides-memoire and Implementation Status Reports 2016–2022 Project Completion Report, Ministry of Education and Training, 2022 Page 58 of 58