Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005947 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF0A4920 ON A SMALL GRANT IN THE AMOUNT OF USD 3.00 MILLION TO THE The Socialist Republic of Vietnam FOR VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) January 19, 2023 Education Global Practice East Asia And Pacific Region Regional Vice President: Manuela V. Ferro Country Director: Carolyn Turk Regional Director: Daniel Dulitzky Practice Manager: Mario Cristian Aedo Inostroza Task Team Leader(s): Anh Thuy Nguyen, Douglas Sumerfield ICR Main Contributor: Lauri Veli Johannes Pynnonen ABBREVIATIONS AND ACRONYMS COVID-19 Coronavirus Disease Pandemic that started in 2019 CPF Country Partnership Framework DFAT Department of Foreign Affairs and Trade DOET Department of Education and Training EM Ethnic Minority FY Fiscal Year GPRBA Global Partnership for Results-Based Approaches GoV Government of Vietnam HCMUE Ho Chi Minh City National University of Education HNUE Hanoi National University of Education IDEO Intergenerational Deaf Education Outreach IRI Intermediate Results Indicator IVA Independent Verification Agent OBA Output-Based aid M&E Monitoring and Evaluation MCV Manually Coded Vietnamese MOET Ministry of Education and Training PDO Project Development Objective PMB Project Management Board PMU Project Management Unit QIPEDC Quality Improvement of Primary Education for Deaf Children RF Results Framework VNIES Vietnam National Institute of Education Sciences VSL Vietnamese Sign Language WB World Bank TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 Theory of Change (Results Chain) ................................................................................................5 II. OUTCOME .................................................................................................................... 12 Table 3. Deaf student learning outcomes in project schoolyears ................................................ 14 *The students passed the semester when they scored a five or higher from the end of semester tests and attended at least 75% of classes. .............................................................................. 14 Justification of Overall Efficacy Rating ....................................................................................... 15 Assessment of Efficiency and Rating.......................................................................................... 16 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 16 A. KEY FACTORS DURING PREPARATION ................................................................................... 16 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 17 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 17 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 17 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 18 C. BANK PERFORMANCE ........................................................................................................... 19 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 21 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 23 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 28 ANNEX 3. SUPPORTING DOCUMENTS .................................................................................. 29 HOW VIETNAM OPENED NEW DOORS FOR DEAF CHILDREN ................................................. 29 STORY HIGHLIGHTS................................................................................................................... 29 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name VIETNAM QUALITY IMPROVEMENT OF PRIMARY P160543 EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) Country Financing Instrument Vietnam Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency The Socialist Republic of Vietnam Ministry of Education and Training Project Development Objective (PDO) Original PDO The project development objective is to increase access for deaf children to Vietnamese sign language-based primary education in selected schools within the participating provinces and improve their learning outcomes. Page 1 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A9420 3,000,000 2,226,333 2,226,333 Total 3,000,000 2,226,333 2,226,333 Other Financing Borrower/Recipient 76,000 126,000 70,476 Total 76,000 126,000 70,476 Total Project Cost 3,076,000 2,352,333 2,296,809 KEY DATES Approval Effectiveness Original Closing Actual Closing 27-Dec-2018 31-Jul-2019 30-Jun-2021 30-Aug-2022 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 15-Jun-2021 1.59 Change in Loan Closing Date(s) Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 02-Apr-2019 Moderately Satisfactory Moderately Satisfactory 0.00 Page 2 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) 02 26-Nov-2019 Moderately Satisfactory Moderately Satisfactory 0.60 03 02-Dec-2020 Moderately Satisfactory Moderately Satisfactory 0.88 04 03-Dec-2021 Satisfactory Moderately Satisfactory 1.82 05 22-Aug-2022 Satisfactory Satisfactory 2.17 ADM STAFF Role At Approval At ICR Regional Vice President: Victoria Kwakwa Manuela V. Ferro Country Director: Ousmane Dione Carolyn Turk Director: Daniel Dulitzky Practice Manager: Tobias Linden Mario Cristian Aedo Inostroza Anh Thuy Nguyen, Douglas Task Team Leader(s): An Thi My Tran, Inga Afanasieva Sumerfield ICR Contributing Author: Lauri Veli Johannes Pynnonen Page 3 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. At the time of appraisal of the Quality Improvement of Primary Education for Deaf Children Project (the project, QIPEDC), Vietnam had one the highest rates of hearing loss among children in the world 1. In 2015 this translated to about 116,000 Vietnamese children having a hearing impairment, of which approximately 40,000 had access to education through 50 specialized or hundreds of all-inclusive schools around the country2. 2. Vietnam is internationally recognized for its achievements in access to and quality of its basic education 1,2. However, despite this and Vietnam’s long history of education for deaf children, Vietnam still lacked favorable conditions in providing quality education for these children. Majority of the deaf children never attended school and those who were able attend often dropped out before completing the fifth grade1. 3. A major impediment to the education quality of deaf children was the lack of an established practice of teaching in the Vietnamese sign language (VSL), the first or “native� language of the Deaf community3. Even in many specialized schools, deaf students tended to study in aural language rather than sign language due to lack of qualified sign language teachers and teaching assistants. The issue was even further compounded by the shortage of standardized VLS signs. Because instruction was oral and Vietnamese being tonal and therefore particularly difficult to lipread, it normally took Deaf people 8 or 9 years to finish 5 grades of school4. 4. Between 2012 and 2016, the World Bank’s (WB) Intergenerational Deaf Educati on Outreach (IDEO) project successfully prepared 250 pre-school deaf children for formal schooling through VSL. The project used a joint family- institution model, most notably with young deaf adults as mentors, which supported children’s cognitive development through communication in a language that they could understand and use. Following the project’s achieved results, the Government of Vietnam (GOV) created positions for teaching assistants and qualified deaf mentors that could assist the assistants in teaching deaf students in sign language. 5. The IDEO project proved the importance and efficiency of a joint institution-family outreach model in deaf education in Vietnam. However, the project was only implemented at the pre-school level, and the beneficiaries expressed a strong desire for the bank to extend the service delivery model to the primary school level. To fulfill initially the gap in general education, the GoV identified an ecosystem of teachers, teaching assistants, caregivers and deaf adults that would need to be trained in order to provide on-site and outreach support for deaf children through using VSL. The ecosystem was meant to ensure that most of the developmental gains of deaf children entering primary 1 Nguyen et al. 2019. Situation of hearing loss among children aged 2 to 5 at kindergartens in Hai Duong Province, Vietnam. DOI:10.5530/srp.2019.1.33 2 Gian & Inho. 2015. Comprehension of Figurative Language by Hearing Impaired Children in Special Primary Schools. DOI:/10.1016/j.sbspro.2015.04.448 3 Klaudia, K. (2014). THE BENEFITS OF SIGN LANGUAGE FOR DEAF CHILDREN WITH AND WITHOUT COCHLEAR IMPLANT(S). European Scientific Journal, ESJ, 9(10). https://doi.org/10.19044/esj.2013.v9n10p%p 4 Woodward, Nguyen. 2011. Teaching sign language and deaf studies at university level to hearing people in Vietnam. DOI: 10.7877/jasl.20.5 Page 4 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) education would continue to grow and be consolidated. The first iteration of this ecosystem was to be implemented through the QIPEDC project. Theory of Change (Results Chain) 6. The investments made under Component 1 would improve deaf children’s access to Vietnamese sign language - based primary education by developing animated video lessons for Mathematics and Vietnamese language curricula and by creating visual recordings of VSL signs frequently used in primary education. The funding of activities under Component 1 would be input-based. 7. Investments under Component 2 would complement those from Component 1 by delivering capacity building for teachers, teaching assistants, parents and caregivers of deaf students, as well as deaf mentors to work with the deaf students. The funding for development of the training materials would be input-based, whereas the actual delivery of training would be funded through output-based aid. 8. Component 3 would combine the benefits from Components 1 and 2 by incentivizing schools via output-based subsidies to enroll deaf children and teach them using VSL. The subsidies would cover additional learning materials and learning aids, support for teachers and teaching assistants working with deaf children, and outreach support by teachers, teaching assistants and deaf mentors for deaf children. These would be complemented by covering the costs of communication activities such as holding of teacher-parent conferences to share experience and good practice in supporting deaf children’s education, project reporting requirements, and other related costs. Page 5 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) Figure 1 Theory of Change Project Development Objectives (PDOs) 9. The project development objective was to increase access for deaf children to Vietnamese sign language-based primary education in selected schools within the participating provinces and improve their learning outcomes. Key Expected Outcomes and Outcome Indicators 10. The project’s key expected outcomes were increased access to VSL-based primary school education and, consequently, increased learning outcomes of deaf children. 11. This was measured through two PDO Indicators: The number of primary education deaf students enrolled in the project and taught in Vietnamese Sign Language, and the share of the deaf students taught in VSL that pass the semester tests or equivalent assessments. Page 6 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) The project was implemented in 20 provinces that were spatially representative of the three regions of Vietnam: North, Central and South. The participating schools were located in urban and rural areas, as well as mountainous and coastal areas. The project focused on four groups: (i) primary school deaf students, (ii) teachers and teaching assistants who work with deaf students, (iii) parents/caregivers of deaf students participating in the project, and (iv) deaf mentors. 12. The project also assisted schools in their engagement with parents and communities at large to facilitate the integration of deaf children into mainstream education. The project aimed to benefit 2,040 primary school deaf students, including 1,800 in specialized schools and 240 in inclusive schools; 500 teachers and teaching assistants; 2,040 parents and caregivers, and 400 adults from the Deaf community as mentors. 13. The project’s results framework included intermediate results indicators designed to track progress towards outputs and outcomes that were perceived as necessary to achieve increased access and improved learning outcomes. These were: (i) Number of parents and caregivers of deaf children that participated in the program and were trained in deaf education and communication (ii) Number of VSL-based animated videos covering curriculum of Math and Vietnamese language lessons from Grade 1 to 5 developed (iii) Number of new VSL signs developed (iv) Number of deaf adults trained as mentors to support and facilitate learning for deaf children (v) Number of teachers and teaching assistants trained in deaf education and VSL. 14. Taking into consideration the exponential growth of disruptive technologies, especially in communication and education, the project was deliberately planned to include advanced educational technologies such as e-learning and mobile learning in designing teacher training programs, animated visual learning materials and VSL dictionary for deaf children of primary education age. These were to be transferred to and stored on virtual platforms, to be accessible from various digital devices, such as tablets, laptops and smart phones. 15. Gender sensitivity was placed at the center of the social safeguards of the project. The project aimed to ensure that data collection and selection of beneficiaries was gender sensitive and in alignment with the guidelines and requirements of Vietnam and the World Bank’s Gender Action Plan. The developed learning materials and sign language dictionary were to be carefully examined so that deaf students were equipped with sufficient and effective language and communication skills to deal with any sexual harassment situation that might occur. Components 16. The project had four main components: 17. Component 1: VSL-based material development for primary education (US$ 450,000, Actual US$ 441,279). Activities under this component supported the development of: (vi) 150 video lessons for primary school Mathematics and Vietnamese language curricula, (vii) 4,000 VSL signs frequently used in primary education. Payments for costs of Component 1 were input based. Page 7 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) 18. Two sets of VSL-based animated video lessons covering the curriculum for Math and Vietnamese language were developed for grades one to five of primary education. The content was developed in line with the national curriculum for the primary level. Each video lesson was designed to end with an assessment of students’ understanding of the material. A learning guide and a teaching guide accompanied each set of the video lessons. Since all Vietnamese students were studying the content of the existing textbooks - independent of their hearing ability, the video lessons were to help deaf students assimilate the same content through VSL. The videos included sound, animated pictures, and subtitles to make the content equally accessible to teachers, parents/caregivers and hearing students. The signed and subtitled videos allowed deaf students to learn in both VSL and written Vietnamese. 19. The existing list of 2,000 VSL signs developed under the IDEO project was expanded to include additional 4,000 signs fundamental for learning at the primary education level. MOET selected Hanoi National University of Education (HNEU) and Ho Chi Minh City National University of Education (HCMUE) to coordinate the design and implementation of the new lexicons. A key requirement for HNEU and HCMUE was that deaf designers were to be a key part of the development process. The materials were to be made available through a project-specific portal hosted under the MOET domain5. As officially approved by MOET, all video lessons, teaching and learning guides, as well as the 4,000 signs will be moved to an existing eLearning portal which is managed by MOET. The e-portal will also be the platform for further exchanges and a forum for sharing experiences and resources amongst teachers, teaching assistants, schools and parents and caregivers. 20. Component 2: Training of teachers, parents/caregivers and deaf mentors (US$ 1,150,000, Actual US$ 790,114). Activities under this component supported the development of training materials and the delivery of capacity building for teachers, teaching assistants, parents, caregivers, as well as deaf mentors that would work with deaf students. Component 2 consisted of two subcomponents: 21. Subcomponent 2.1: Development of Training Materials (US$ 200,000, Actual US$ 139,878). This subcomponent financed the development of training materials for the training programs for the participating groups: teachers, training assistants, parents/caregivers and deaf mentors. The training programs and training materials assisted them in mastering and using VSL in working with deaf students effectively, assisting them in teaching, communicating and supporting them at school and at home. 22. MOET outsourced the development of the teacher, TA, mentor, and caregiver training materials to the national universities of education, HNEU and HCMUE, which had considerable expertise in deaf education and development of sign language. As an example, the teacher trainings addressed: (i) cognitive development of deaf children (ii) pedagogical skills for teaching deaf children (iii) a basic VSL grammar system and a minimum set of signs necessary for effective teaching (iv) methodologies to deliver Math and Vietnamese language in a VSL-medium (v) deaf children’s behavior and communications with gender sensitive emphasis (vi) supporting deaf children to integrate into the community (vii) VSL interpretation (viii) on-site support and coaching for teachers before and after certification. 5 https://qipedc.moet.gov.vn/dictionary Page 8 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) 23. The training program was provided face-to-face in 2020-21 and via a combination of face-to-face and online methods due to the COVID-19 pandemic in late 2021 and early 2022. The length of the training was determined by its content; however, to minimize disruptions the training did not exceed 150 hours of face-to-face sessions per topic per school year. 24. Subcomponent 2.2: Output-Based Aid (OBA) subsidy for Delivery of Training (US$ 950,000, Actual US$ 650,235). This subcomponent financed the delivery of training for 400 teachers and 100 teaching assistants through certified training programs, training and support to 2,040 parents/caregivers of deaf students, and training and capacity building of 400 deaf mentors with the needed skills to teach, guide, tutor and mentor deaf children throughout the school year. 25. While payments for activities under Subcomponent 2.1 were input-based, payments for activities under Subcomponent 2.2 were made on a subsidy basis in which the training organizers were reimbursed based on the number of teachers, teaching assistants, parents/caregivers and deaf mentors that they enrolled and that completed the trainings. The training programs for all these target groups took place in the first year of project implementation with some refresher courses in the following years that consolidated and supported newer participants. MOET identified two training institutions that had faculty specialized in deaf education to deliver the training programs for teachers and teaching assistants of deaf children: Hanoi National University of Education (in the north), and Ho Chi Minh City University of Education (in the south). 26. The project aimed to emphasize the importance of creating and maintaining a conducive learning environment for deaf children at home and in the community. This was attempted by providing the caregivers of participating deaf children with opportunities to attend formal training and meet with teachers, other parents, and practitioners of community of practice. These methods offered opportunities for caregivers to learn and exchange experiences in guiding and supporting their children, as well as to teach them VSL. They learned about behaviors, psychology and communication patterns of deaf children, including gender specific needs and other disabilities. The communication, methods, and training materials for beneficiaries from ethnic minority groups were conducted in accessible languages. Due attention was given to adequate attitude, behavior, language, and teaching methods to ensure more effective work with deaf ethnic minority children in a culturally appropriate manner. Further consultations with ethnic minority (EM) beneficiaries were carried out during implementation. 27. To support children’s learning and integration with the community that they lived in, the project provided 399 deaf mentors in 20 provinces with necessary knowledge, skills and sharing through learning opportunities. The role of deaf adult mentors was to guide and ensure a safe and conducive learning and communication environment for deaf children. At the beginning of the project, each province identified a group of deaf mentors, around 20 in each province, who had relevant experience, interest and time availability to mentor deaf children in their community. The training and work with parents and mentors were delivered by provincial teacher training institutions. 28. Component 3 – Output-Based Aid (OBA) subsidy for VSL-based primary education of deaf children (US$ 1,100,000, Actual US$715,219). The purpose of this subsidy was to incentivize schools to enroll deaf children and to teach them using VSL, in line with government policy of increasing access to education for children with disabilities. In total, 1,929 deaf children were enrolled for specialized and inclusive schools. Both private and public schools in participating provinces were eligible for the subsidy, as long as they provided VSL-based education services to deaf children. Page 9 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) 29. Specialized schools received US$300 and inclusive schools received US$500 per annum per beneficiary student if the student met the inclusion criteria. Subsidy payments were made when: (i) The student was enrolled at the beginning of the school year (ii) The student attended at least 75% of each semester and obtained a score of 5 points or above (on a 10- point scale) in semester tests or equivalent assessments. 30. The project took advantage of the mandatory summative tests at the end of each semester to determine whether the subsidy requirements were met and to track the progress students made against the prescribed standard skills and knowledge for primary education. The results of the tests were verified by the independent verification agents and served as proof for output-based reimbursements. Figure 2 An example of implementation arrangements and the flow of funds 31. The difference in per student subsidy amount between inclusive and specialized schools was meant to provide inclusive schools that enrolled much smaller amount deaf students per school (normally 1 or 2 and maximum 4 students per inclusive school) with sufficient resources to assist them. The subsidy was meant to cover additional learning materials and learning aids, allowances for teachers and teaching assistants working with deaf children, outreach support by teachers, teaching assistants and deaf mentors for deaf children, communication activities such Page 10 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) as holding of teacher-parent conferences to share experience and good practice in supporting deaf children’s education, project reporting requirements, and other related costs. All participating schools plan to continue using the results of the project including VLS, teaching and learning materials, methodologies that teachers have been trained to teach deaf students in their schools. MOET has formally approved the training materials and plans to publish and encourage the use of learning materials and VLS signs nationwide. 32. Component 4 – Project management and independent verification (GPRBA US$ 300,000, Actual US$279,704. GoV US$ 126,000, Actual US$ 70,748). Three input-based activities were supported under this component: (i) Project management, (ii) Project audit, and (iii) Independent Verification Agent. 33. A Project Management Unit (PMU) consisting of staff members from the Project Management Board (PMB), Departments of Primary Education, Ethnic Minority Education, Teachers and Education Managers; and Planning and Finance was housed in the PMB of MOET. The PMU was responsible for the overall project planning, coordination, reporting, monitoring and evaluation. Under the component MOET hired procurement, administrative, financial management and technical expertise. Other eligible activities were as follows: (i) Preparation and dissemination of promotional materials and communication campaigns to publicize the project activities. This lasted throughout the project lifecycle but concentrated mostly in the first two years of implementation. (ii) Supporting MOET technical panels that reviewed and appraised the mathematics and Vietnamese language video lessons and the teacher training course. (iii) Supporting MOET and PMU in undertaking due diligence, baseline data collection, and other relevant analysis, evaluation and selection of participating provinces and schools. (iv) Supporting visits to schools to implement the project, including social campaigns, assistance with monitoring and reporting requirements. 34. MOET signed the annual contract with the Vietnam National Institute of Education Sciences (VNIES) as the Independent Verification Agent (IVA) to verify the expected outputs of the project. VNIES had unique experience in deaf education and insight into the general education operation system; plus, they were independent from the teacher training universities that delivered the training for teachers and teaching assistants. VNIES also had experience as the IVA for a prior World Bank results-based-financed project (i.e., School Readiness Promotion Project). Table 1 Resource Allocation by Component (in US$) Grant Counterpart Project Categories Estimated Actual Estimated Actual 1 VSL-based material development for primary education 450,000 441,297 0 0 Training of teachers, teaching assistants, deaf mentors, 2 1,150,000 790,114 0 0 and parents/caregivers Output-based Aid (OBA) subsidy for VSL-based primary 3 1,100,000 715,219 0 0 education of deaf children 4 Project management and independent verification 300,000 279,703 126,000 70,748 Total 3,000,000 2,226,3343 126,000 70,748 Page 11 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) Project Timeline 35. The project was restructured in 2021 to extend the project closing date by 14 months from June 30, 2021 to August 30, 2022. The extension required no additional project financing; however, the counterpart funding to cover extended project management costs was increased by USD 50,000 to a total of USD 126,000. Rationale for Changes and Their Implication on the Original Theory of Change 36. The project was originally designed to be implemented over a period of three years with at least two years dedicated for school level interventions. However, the two rounds of review of the grant by the donor and the multiagency reviews by the government’s side resulted in delayed approval of the grant and delayed countersigning of the grant agreement. As a result, the de facto implementation period of the project was reduced to less than two years. The project utilized the first year of implementation for the development of the VSL learning materials, training materials for teachers, parents, and deaf mentors; and for training before rolling out the VSL program at the school level. 37. The Department of Foreign Affairs and Trade (DFAT), the trust fund donor supporting the project, extended its Administrative Agreement with the Global Partnership for Results-Based Approaches (GPRBA) until December 2022. This in turn provided the opportunity to extend the QIPEDC project by another year, which allowed it to complete all its planned activities. The extension enabled the project to show another year of learning gains for students, providing further evidence of the impact of the project and building the case for expanding the activities to other students through regular MOET programming. 38. The restructuring did not affect the design of project components nor the Theory of Change. II. OUTCOME Assessment of Achievement of Each Objective/Outcome Assessment of Relevance of PDOs and Rating 39. QIPEDC was the second World Bank managed sign language project in Vietnam and it followed closely the successful model set by its predecessor, IDEO. Like IDEO, the project was fully aligned with the World Bank’s twin goals of “eliminating extreme poverty by 2030� and “boosting shared prosperity�. While Vietnam’s service provision and access have greatly improved over the last 10 years6, equitable access for marginalized groups remains a challenge7. The project targeted vulnerable groups that have not significantly benefited from recent economic growth and were difficult to reach through national general education programs. 40. The project continues to remain highly relevant and fully consistent with the second focus area of the CPF FY18-22, which highlighted investing in people and knowledge, with a specific aim at ensuring a non-differentiated service quality and access to education while moving toward a sustainable financing model. The Vietnam Development Report 2019 highlighted inclusiveness and identified the need to provide universal access to quality education. In the Vietnam 2035 report the government aimed to balance economic prosperity with environmental sustainability, and to promote equity and social inclusion8. For its part, QIPEDC focused on inclusive education while promoting a results- based approach. 6 2022 Vietnam Poverty and Equity Assessment – From the Last Mile to the Next Mile. World Bank, 2022 7 VNIES 2022.Summary Report Vietnam Education Sector Analysis 2011 – 2020. 8 Vietnam 2035 is a joint Government of Vietnam (Ministry of Planning and Investment) –WBG initiative, finalized in 2016. Page 12 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) 41. The problem the project aimed to solve was clearly articulated and the approach was appropriate. The involvement of the Deaf community was crucial, as the representatives from the community were able to provide insights into the current and desired state of deaf education in Vietnam. They were able to highlight the special characteristics of VSL, which for a non-standardized language were hard for non-VSL speakers to comprehend. This led to appropriate levels of ambition, where the project focused on building the vocabulary, pedagogy and student support, but refrained from overachieving goals, such as standardizing VSL grammar. 42. Finally, the previous sector experience achieved through the IDEO project guided the project’s implementation, ambitions, and structure. The foundations laid by IDEO and built upon by QIPEDC provided considerable evidence of the impact of investing in VSL-based deaf education and built the case for expanding the activities to other subjects besides Vietnamese and Mathematics and levels beyond primary through regular MOET programming. Therefore, the relevance of the PDO is rated High. Achievement of PDOs (Efficacy) 43. The achievement of the PDO is assessed according to progress toward its two sub-objectives: (PDO 1) increasing access for deaf children to Vietnamese sign language-based primary education in selected schools within the participating provinces and (PDO 2) improving their learning outcomes PDO 1: Increasing access for deaf children to VSL based language-based primary education 44. Based on PDO indicator 1, the project achieved its first objective of increasing access to VSL-based primary education. The student enrollment reached 95% of the project target of 2,040 students, as by end of the project, 1,929 primary education deaf students had been enrolled in the project and taught in Vietnamese Sign Language. 1,733 deaf students (out of targeted 1,800) attended specialized education and 196 (out of targeted 240) deaf students attended inclusive education. There were three major causes for students being prevented from being counted as deaf students participating in the project: (v) Some students had multiple disabilities. (vi) Some students were not certified as deaf by the authorized entity. (vii) Participating schools were fixed at the beginning of the project therefore deaf students of the non-project schools could not be eligible. Table 2 Student enrollment in detail Students Target Actual Percentage Specialized education 1800 1733 96.3 Inclusive education 240 196 81.7 Total 2040 1929 94.6 PDO 2: Improving the learning outcomes of deaf children to VSL-based primary education 45. Based on PDO indicator 2, the project achieved impressive results in improving the learning outcomes of deaf children and laid the foundations for continued quality improvements in the future. The deaf students’ learning results Page 13 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) far surpassed the annual targets with 96% of students passing their end-of-semester tests (vs 50% targeted). Table 3 provides detailed data of the learning outcomes, expressed through the exam results of the two semesters of the 2020- 21 and 2021-22 school year. The table shows the continuous and constant high pass and attendance rates in project schools. Table 3. Deaf student learning outcomes in project schoolyears Semester/School STUDENTS IN SPECIALIZED SCHOOLS STUDENTS IN INCLUSIVE SCHOOLS year Mathematics Vietnamese Attendance* Mathematics Vietnamese Attendance* Points <5 >=5 <5 >=5 <75% >=75% <5 >=5 <5 >=5 <75% >=75% Semester 1/20-21 11 1489 10 1490 1 1499 20 172 24 168 2 192 Semester 2/20-21 13 1487 9 1481 8 1492 3 191 3 191 2 192 Semester 1/21-22 34 1,520 38 1,516 26 1,528 3 165 2 166 1 167 Semester 2/21-22 58 1,496 58 1,496 54 1,500 4 164 4 164 3 165 *The students passed the semester when they scored a five or higher from the end of semester tests and attended at least 75% of classes. 46. Comparing pre-project data for 2019-2020 in project provinces, learning outcomes of deaf students improved significantly, as seen in table 4. For example, in the northern mountainous provinces of Yen Bai and Thai Nguyen the proportion of deaf students passing their final exams increased from 82% and 92.7% to 100% and 99.96%, respectively. In Lam Dong, representing central highlands, the pass rate rose from 93.2% to 98.6%. In Quang Ngai, a province in the southern central region, the increase was from 92.5% to 100%. Table 4. Examples of learning outcomes from selected project provinces - pre-project data (schoolyear 2019-20) vs project data (schoolyear 2020-21) % of students with passing scores for grade 5 Attendance Province for Math and Vietnamese 2019-2020 2020-2021 2019-2020 2020-2021 Yen Bai 82% 100% 93.5% 100% Thai Nguyen 92.70% 99.96% 87.02% 100% Lam Dong 93.2% 98.6% 97.3% 100% Quang Ngai 92.50% 100% 93% 100% 47. The improvement to learning outcomes was attempted through a comprehensive approach developing the teaching methods, learning materials, and classroom support. Learning materials were complemented by creating supportive videos for 150 mathematics and Vietnamese language classes (IRI2, 150/150). The students’ caregivers received training in deaf education and communication in VSL (IRI1, 1762/2040). Four thousand VSL lexicons were Page 14 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) recorded and made available through a searchable video database on the MOET website (IRI 3, 4000/4000). Three hundred ninety-nine deaf adults were trained as deaf mentors to support and facilitate learning for deaf children (IRI 4, 399/400). Five hundred twenty-nine teachers and teaching assistants participated in training sessions covering deaf education and VSL (IRI 5, 529/500). Table 5. Intermediate Result Indicators – Targeted and Achieved Intermediate Result Indicator Target Final Number of parents and caregivers of deaf children that participated in 1 2,040 1,762 the program and were trained in deaf education and communication Number of VSL-based animated videos covering curriculum of Math and 2 150 150 Vietnamese language lessons from Grade 1 to 5 developed 3 Number of new VSL signs developed 4,000 4,000 Number of deaf adults trained as mentors to support and facilitate 4 400 399 learning for deaf children Number of teachers and teaching assistants trained in deaf education 5 500 529 and VSL. Justification of Overall Efficacy Rating 48. Overall efficacy is rated High, based on the achievement of both components of the PDO and its impact on institutional strengthening that will allow its benefits to be institutionalized and expanded to other subjects and other provinces in the future. 49. The PMU managed to stay lean because it outsourced some of the technical and non-routine tasks, thereby maximizing the funding allocated to beneficiaries. Important external support groups such as the deaf community and independent evaluators provided ample support by monitoring the performance of project schools and contracted teacher training HNUE in Hanoi and HCMCUE in Ho Chi Minh City and were able to provide timely technical feedback. Due to these external support mechanisms, the PMU was able to react to changes quickly and before materials were finalized and released to full use. 50. The results-based disbursement approach focused on measurable outputs that could be evaluated through a quality assurance procedure. The verification of the results was done by the third party (i.e., VNIES) and carried out using reliable methods. At the same time, the verification process identified feedback issues and recommendations, which in turn helped to improve the quality and implementation of activities. 51. The verification process was refined throughout the project. However, the initial verification model caused significant delays during the first year of the project. Schools provided evidence of student enrollment and learning Page 15 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) outcomes without having received sufficient guidance. This made the verification unstructured and caused unnecessary delays in reimbursements, which in turn placed a needless financial burden on schools already lacking financial stability. Assessment of Efficiency and Rating 52. The project was implemented efficiently within its extended schedule, without the need for additional financing. The project fully achieved its results but spent only 74% of its grant funds. Similarly, after adjusting the schedule to take into account the initial delays in the verification process the project maintained its timeline and completed all activities according to plan. 53. There are two key reasons for underspending. Firstly, COVID-19 pandemic enforced social distancing measures and converted many of the in-person trainings into lower cost online learning sessions, and it either prevented or delayed the implementation of activities including the recruitment of the deaf adults, procurement and other support to deaf students. Secondly, the output/result-based financing approach was completely new to the participating schools. Therefore, there was aversion for procurement since schools had very little available resource for advance payments. 54. The project’s efficiency is rated Satisfactory. The project’s results suggest that it was cost-effective, and its returns represented by learning outcomes were higher than those estimated at appraisal. By improving the access and quality of VSL-based primary education services in project provinces, the project’s outcomes either met or exceeded original targets, and updated assumptions related to deaf education and to VSL-as-first language for deaf children. Overall Outcome Rating 55. The project met or exceeded its expected outcomes as defined and measured by PDOs and IRI targets and, equally important, strengthened institutional capacity at the provincial levels and among key educational institutions, which will continue replicating project outcomes in the future. The project’s overall outcome is rated Satisfactory, based on its high relevance, high efficacy, substantial efficiency, and significant impact that is expected to be expanded by the government beyond the initial scope. Other Outcomes and Impacts 56. Feedback from DOETs confirmed that the project schools and even non-project schools would continue the project activities by using the learning materials (150 video clips and 4,000 VSL lexicons) and sign language for teaching deaf students. MOET also has a plan to establish a center for inclusive education in each province to ensure implementation of the “no one left behind� strategy. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 57. During preparation of the project the team acknowledged that the planned actions were not a part of the GoV’s plan, nor were they budgeted by MOET. This could have caused challenges for an output-based financing project, as the government could not give non-budgeted advance payments to schools that the World Bank could then reimburse after providing a no objection based on the verified results. At the same time the schools had no budget of their own to pay for the initial investments. This was addressed in the project plan by splitting the financing into three parts. Thirty percent of the budget was disbursed to schools when the schools submitted a list of participating students. Page 16 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) This was an action that incurred no costs for the schools, and therefore gave them some initial financial capital. Other thirty percent was released when the schools submitted verified test results for the first school year, and the final forty percent when they submitted verified results of the second school year. 58. The project design took into consideration the exponential growth of disruptive technologies, especially in communication and education, and deliberately included educational advanced technologies in its design. These included actions such as e-learning and mobile learning in designing teacher training programs, animated visual learning materials and VSL video-dictionary for deaf children of primary education. These materials were transferred to and stored on virtual platforms, which permitted access from various digital devices, such as tablets, laptops and smart phones. These resources turned out to be invaluable during COVID-19 related restrictions and lockdowns as they enabled and facilitated remote learning for beneficiaries. B. KEY FACTORS DURING IMPLEMENTATION 59. Despite the difficulties posed by the COVID-19 pandemic between 2020 to 2022, the QIPEDC project provided training for 1,762 parents, 399 deaf mentors as well as 529 teachers and teaching assistants. Social distancing measures prevented several face-to-face training as trainers could not travel to the provinces. Many parents of deaf students did not have good quality computers/smart phones or stable internet connections. To solve such problems, a hybrid approach was applied, where lecturers delivered online training, while the trainees were gathered at the provincial DOET facilities with teaching assistants for added support. Deaf coordinators were also invited to participate and support such training. Given the impact of the training courses and needs of the teachers in specialized and inclusive schools, many project provinces shared training materials with other schools that did not participate in the project, permitting more teachers to have access to the project resources and enable them to practice their daily work. 60. The procurement of project materials, such as displays and computers, was delayed at the early stage of project implementation. The delays came from schools’ low procurement capacity and the lack of specific guidance in the initial Project Operations Manual, which did not clearly specify which assets would be reimbursable; these delays were compounded by the slow verification process during the first half of the project. Given the short implementation period, these delays in procurement led to schools being unable to use their full budget, which in turn led to lower intended disbursements. 61. During project implementation, some bottlenecks in the documentation and funds flow between the PMU and participating schools of Component 3 were observed, which caused delayed disbursements. The PMU took measures to address the issues identified including, among others, revision of operational manual and financial management guidelines and organization of trainings, provision of hands-on support to project schools/provinces. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) 62. The performance of the M&E function of the project was Substantial. All project results were achieved and fully verified by the IVA by project closing. 63. The project’s M&E was designed to: (a) track the progress, provide timely feedback and verify the achieved results to project management, partnering institutions, and the World Bank on implementation status and learning Page 17 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) outcomes achieved in the participating schools; (b) estimate the effectiveness of the project; and (c) summarize lessons learned from the design and implementation that could be applied to future similar interventions. 64. Routine monitoring was to be tracked based on the project’s results framework (RF), using independent verification agents. For the most part the project’s RF was concise, consisting of two PDO and six IR indicators that would be tracked regularly by the PMUs, consolidated by the PMO, and forwarded to the World Bank. The indicators and associated targets were generally appropriate, with the exception of PDO Indicator 2. The achievement of almost two times higher than the end target (97% vs 50%) of PDO Indicator 2 could be explained by using different measurements of deaf student’s learning outcomes. The original approach is to use the end-semester exams for hearing students to measure the learning results of deaf students. During the implementation, the tests for deaf students were designed according to their personal education plans to fit with the curricula used for teaching deaf students in schools and reflect better their competence. Such practice regulated by the MOET will be continued to use for measuring learning outcomes for disabled students in general and deaf students in particular. 65. The verification of project results, including the training of the four groups and deaf students’ p articipation, was completed. All told, eight independent verification (IV) reports of good quality were produced. While the first IV reports were submitted with substantial delay in 2021, the subsequent reports were completed in a timelier manner. Due to the COVID-19 pandemic, the last three IV reports for participation and learning outcomes of deaf students were prepared using a hybrid approach – desk review of supporting documents and online interviews. Class observations were also constrained due to the social distancing measures. Provincial on-site coordinators were hired to support the IV teams to collect data and verify the results. Justification of Overall Rating of Quality of M&E 66. The project’s PMB, PMU and IVAs routinely collected and monitored project data, and the PMU team acted upon the results to address emerging issues before they became problems. The adequate design of the results framework, good implementation of the data monitoring and verification and the utilization of the M&E system all helped inform the managements’ decisions. The monitoring data from the project schools collected and analyzed by the independent evaluators provided the basis to evaluate the project’s achievement of its PDO, albeit without the benefit of a formal impact evaluation. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 67. Gender sensitivity was placed at the center of the social safeguards of the project. The project ensured the data collection and steps of beneficiaries’ selection were gender sensitive, and in alignment with the guidelines and requirements of Vietnam and the World Bank’s Gender Action Plan. The developed learning materials and sign language dictionary were carefully examined so that deaf students would be equipped with sufficient and effective language and communication skills to deal with the sexual harassment situations if such were to occur. 68. The project took several significant steps in removing barriers that prevented deaf children, their parents, deaf mentors and deaf teaching assistants from identifying or communicating sexual harassment. The project disseminated a video on how to identify risks and report cases of sexual harassment for deaf children through different channels, including schools, clubs of deaf adults, the project website, and relevant events. Notably, the project advised MOET to issue Dispatch No. 67/QLCDA-QIPEDC on disseminating the learning resources on how to identify risks and report cases of sexual harassment for deaf children. Page 18 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) C. BANK PERFORMANCE 69. The World Bank’s Performance is rated Satisfactory. At entry, the chosen design was reasonable and aimed to support important aspects of inclusive learning. The project was implemented in close collaboration with both the beneficiaries and local authorities, simultaneously supporting 20 districts across the country. The selection of the project locations proved to be very appropriate and representative of the whole country. The project design was closely based on the demands and relevant to the local conditions and traditions. 70. The project design was comprehensive and included necessary aspects to improve Vietnamese sign language- based curricula and teacher training. The project's interventions were suitable for the available resources, local curriculum, inclusive culture and community connections. The project supported and disseminated actions that facilitated VSL-based learning, and due to the participatory nature of the project, this made it easier for the hearing- impaired children and the Deaf community to adapt and absorb them. The participation of schools and communities was mobilized in each of the project steps (from designing, implementing, and monitoring to adjusting and evaluating). This helped develop ownership, improve effectiveness, and ensure the sustainability of the project’s results. 71. There were some inconsistencies in the initial design of the project target setting . One key inconsistency for IRI1 was that the number of participating caregivers was based one-to-one on the number of participating children. Here the target for caregiver beneficiaries should have been stated as a percentage of participating children. The flaw in the indicator made the absolute number of the caregivers to be trained become unreachable in a case where fewer than the target of 2,040 children participated, or when caregivers had more than one child participating in the project. 72. An additional issue with target setting at the design phase was that the initial 50% target pass rate for deaf students for PDO#2 was set based on different exams than were used for project implementation, leading to a target that was too low. The target rate was based on qualitative data acquired from participating schools, where the pass rates of students were identified as “extremely low�. Based on consultations with the original task team, this led to a quantified target of 50%. In retrospect this target was unrealistically low, as the student-specific personalized learning goals supported by the project led to an initial result of almost 90% in Year 1 of the project. 73. The achievement of almost double the end target (97% vs 50%) for PDO Indicator 2 merits further discussion. It can be explained by the use of different measurements of deaf student’s learning outcomes. The original approach was to use the same end-semester exams for hearing students to measure the learning results of deaf students. However, during project implementation, tests for deaf students were redesigned according to their personal education plans to fit with the curricula used for teaching deaf students in schools and reflect better their competencies. This project-supported practice is included in MOET regulation (Circular 27/2020/TT-BGDDT, Article 8 dated September 4, 2020) and will continue to be used for measuring learning outcomes for disabled students in general and deaf students in particular. Target issues aside, this regulation and practice can be considered a significant project success. 74. During supervision, the task team proactively and frequently provided technical and other necessary support to the project management unit. The project extension in 2021 was a very timely adjustment to compensate for the initial delays in the beginning of the project activities and permitted full achievement of the PDO. The World Bank team made frequent visits to the project sites, and these visits permitted reallocation of saved funds to more resources for Page 19 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) project schools to implement a variety of activities including teachers' professional exchanges, awards for good performing teachers and students, and procurement of equipment required for better teaching and learning. In addition, the participation of deaf adults increased through the deaf mentor program, which contributed to the overall project results. D. RISK TO DEVELOPMENT OUTCOME ANK PERFORMANCE 75. Risk to development outcome is considered Low. At the time of appraisal, the overall risk to the project development outcome was Substantial. This rating was primarily based on the lack of risk-sharing arrangements in the project as GPRBA funded both the “upstream investment� to develop the VSL materials and the OBA subsidies to schools. Normally, risks associated with the investment side of the project would be borne by other entities or the government and GPRBA would only provide the OBA subsidy to bridge the service delivery gap. However, the education budget in Vietnam was decentralized and managed by local and provincial authorities; MOET was responsible for policy development and monitoring with no discretionary budget to support the pilot. GPRBA weighed the financial and performance risk of its being the sole funder against the merits of testing the OBA approach to provide a wider inclusion of deaf children and decided to proceed with the project with the understanding that, if successful, the approach may be scaled up through MOET directives to local and provincial governments. 76. The project achieved expected outcomes and can be considered a success. As expected by GPRBA, MOET has embraced the projects achievements and plans to sustain its outcomes by using the developed materials, methods and models with future cohorts of children. This will be supported through continuous development of training materials created under the project, adaptation of the professional development materials for regular in-service teacher training and by expanding the project activities to cover additional schools and provinces. The digital materials, namely the video lessons and VSL lexicons, will be moved to a domain under MOETs own website where they will be more likely to be discovered by the beneficiaries. 77. Another substantial risk was associated with institutional capacity for implementation and sustainability. This was on the account of 178 schools participating in the project, possible resistance from parents to VSL education, and the novelty of the approach. The development of a host of animated learning materials required considerable coordination and stakeholder willingness to reach consensus. To mitigate these risks, the project made provisions for capacity building for MOET and its regional education departments, ensured the selection of participating schools with strong commitment to the program, and carried out awareness-raising activities for parents and the communities in which the project activities were conducted. 78. The mitigation of the institutional capacity risks through capacity building managed to alleviate the potential challenges related to the high number of participating schools. According to the IVA reports, the awareness-raising activities were well received by parents and their communities. The mitigative actions contributed towards increased sustainability and at closing MOET, participating DOETs and project schools all show strong willingness to adhere to sustaining the projects outcomes in the future. 79. The lack of maintainable funding mechanism for deaf mentors poses a small but distinct risk to the development outcome. The project schools have budget to maintain a functional force of teachers and teaching assistants, but no financial allocations exist for the continued employment of the deaf mentors. At the time of closing the schools planned to offer deaf mentor positions to graduated deaf students on a voluntary basis, but without financial compensation it remains to be seen if the voluntary model becomes sustainable. Page 20 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) V. LESSONS LEARNED AND RECOMMENDATIONS 80. As one of the very few World Bank financed projects that focused predominantly on deaf children, the project offered several lessons to further expand Vietnam’s inclusive education approach. • Well-designed projects that count upon alignment with a government program, detailed analytical work that identifies issues, commitment to and readiness for implementation, and solid institutional arrangements that address strengths and weakness have the necessary ingredients for success. The project ticked many of these boxes, but somewhat missed others. As the World Bank’s financing instrument was not fully aligned with the government’s financial and budgetary policies, the full benefit of the flexibility of output-based financing model remained out of reach for some of the project schools. In the end many schools had to fulfill the high accountability requirements of the government’s financial policies in order to unlock the funds according to the World Bank’s results-based financing requirements. For output-based projects, the government should have the necessary regulations in place to allow for results-based budgeting/disbursement, and as such regulations should allow certain flexibility of the use of sub-grants at the school level, and any changes to the regulations, if needed, should be made in a timely manner. • Capacity building to meet the quality requirements of the verification process is crucial for an output-based project. For most of the participating institutions the financing model and especially the verification model appeared to be new and did not receive sufficient attention and training at the start. This presented several barriers to the verification process and caused delays during the first and second years of the project. • Clearly defined reporting and invoicing templates help schools identify reimbursable expenditures. The participating schools lacked financial personnel skilled in the type of reporting required by the project. Giving them clear report templates accompanied by step-by-step guidance would have facilitated the procurement process and permitted them to fully benefit from the grants allocated to them. • External shocks can be used as opportunities to improve system resilience. The universities and schools were forced to innovate to account for the challenges caused by the COVID-19 related social distancing regulations. This most likely contributed to the higher visibility of the project’s digital resources and led beneficiary schools to provide access to digital classes to other schools not participating in the project. • Setting a minimum school specific subsidy permits inclusive schools to meaningfully participate. The project’s subsidy structure was based entirely on the student headcount, which worked well for special education schools with large cohorts of qualifying students. However, inclusive schools with only a few students suffered from administrative overhead and therefore benefitted less from the project. Having a minimum grant per school that would be complemented by student numbers past a certain threshold could have helped inclusive schools with fewer deaf students to participate more meaningfully and benefit more fully. • Including representatives of the community the project aims to serve throughout project lifecycle makes the project more relevant. There’s a saying in the disability rights movement: “Nothing about us without us�. In essence we cannot build systems for people with specific needs without involving them in the process. We might know all the right success criteria to meet, have confidence in methods and designs, yet we are not our representative beneficiaries. Getting feedback from the Deaf community throughout the Page 21 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) project’s lifecycle built trust between the implementing parties and the Deaf community, ensured relevance and began to close the gaps between people implementing and benefitting from the project. . Page 22 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To increase access for deaf children to Vietnamese sign language-based primary education in selected schools within the participating provinces Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion PDO 1: Number of primary Number 0.00 0.00 2040.00 1694.00 education deaf students enrolled in the project and 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 taught in Vietnamese Sign Language (boys/girls) Number of female deaf Number 0.00 0.00 1020.00 785.00 students enrolled in the project and taught in VSL 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 Comments (achievements against targets): Objective/Outcome: To improve deaf children's learning outcomes. Page 23 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion PDO 2: The share of the deaf Percentage 0.00 0.00 50.00 97.40 students taught in VSL pass the semester tests or equivalent 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 assessments Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: VSL-based material development for primary education Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IR 2: Number of VSL-based Number 0.00 0.00 150.00 150.00 animated videos covering curriculum of Math and 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 Vietnamese language lessons from Grade 1 to 5 developed Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Page 24 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) IR 3: Number of new VSL signs Number 2000.00 0.00 6000.00 6000.00 developed 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 Comments (achievements against targets): Component: Component 2: Training of teachers, parents/caregivers and deaf mentors Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IR 1: Number of parents and Number 0.00 0.00 2040.00 1518.00 caregivers of deaf children participated in the program 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 trained in deaf education and communication Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IR 5: Number of deaf adults Number 0.00 0.00 400.00 400.00 trained as mentors to support and facilitate learning for deaf 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 children Page 25 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IR 6: Number of teachers and Number 0.00 0.00 500.00 484.00 teaching assistants trained in deaf education and VSL 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 Comments (achievements against targets): Component: Component 3 – Output-Based Aid (OBA) subsidy for VSL-based primary education of deaf children Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion IR 4: The number of deaf Percentage 0.00 0.00 1326.00 1650.00 students that pass the semester tests or equivalent 01-Jan-2019 01-Jan-2019 30-Aug-2022 30-Aug-2022 assessments Comments (achievements against targets): Page 26 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 1. Number of primary education deaf students enrolled in the project and taught in Outcome Indicators Vietnamese Sign Language (boys/girls) 2. Number of female deaf students enrolled in the project and taught in VSL 1. Number of VSL-based animated videos covering curriculum of Math and Vietnamese language lessons from Grade 1 to 5 developed 2. Number of VSL-based animated videos covering curriculum of Math and Vietnamese Intermediate Results Indicators language lessons from Grade 1 to 5 developed 3. Number of parents and caregivers of deaf children participated in the program trained in deaf education and communication 1. 150 VSL-based animated videos covering curriculum of Math and Vietnamese language lessons from Grade 1 to 5 developed Key Outputs by Component 2. 4000 new VSL signs developed (linked to the achievement of the Objective/Outcome 1) 3. 1518 parents and caregivers of deaf children trained in deaf education and communication Objective/Outcome 2 1. The share of the deaf students taught in VSL pass the semester tests or equivalent Outcome Indicators assessments 1. Number of deaf adults trained as mentors to support and facilitate learning for deaf Intermediate Results Indicators children 2. Number of teachers and teaching assistants trained in deaf education and VSL 1. 1650 deaf students passed the semester tests or equivalent assessments Key Outputs by Component 2. 400 deaf adults trained as mentors to support and facilitate learning for deaf children (linked to the achievement of the Objective/Outcome 2) 3. 484 teachers and teaching assistants trained in deaf education and VSL Page 27 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1: Vietnamese Sign Language (VSL) -based 0 .45 0 material development for primary education Component 2: Training of teachers, parents/caregivers 0 1.15 0 and deaf mentors Component 3: Output-Based Aid (OBA) subsidy for VSL- 0 1.10 0 based primary education of deaf children Component 4: Project management and 0 .37 0 Independent Verification Total 0.00 3.07 0.00 Page 28 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) ANNEX 3. SUPPORTING DOCUMENTS How Vietnam Opened New Doors for Deaf Children Ly-Et was enrolled in a school accommodating deaf children. The headmaster remembers her arrival. STORY HIGHLIGHTS • The Quality Improvement of Primary Education for Deaf Children Project developed new sign language gestures and trained deaf teachers, mentors, and caregivers. • Expanded sign language facilitated the integration of deaf children in Vietnam into the mainstream and special education. • The success of the project makes it suitable for expansion around the country and to older students. Page 29 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) Lo Mu Du Ly-Et was born in 2010 to deaf parents belonging to the Cil (K’ho) ethnic group in the central highland province of Lam Dong in Vietnam. Ly-Et’s parents’ deafness was considered a burden to their families, but they overcame that stigma, raised a family together, and wanted their child to have opportunities that had been unavailable to them. “Ly-Et came to us without any language skills, not in writing, reading or sign,� said Nguyen Thi Ngoc Minh. “We were lucky to be able to access a good set of sign language materials suitable for primary school students so that she and her friends could learn quickly.� The sign language that enabled Ly-Et to thrive was an outcome of the just-concluded Quality Improvement of Primary Education for Deaf Children Project (QIPEDC), administered by the World Bank with funding from the Global Partnership for Results-Based Approaches. This project has enriched the lives of Ly-Et and almost 2,000 other deaf children from 20 provinces across the country well more than the 1,200 originally envisioned. In addition to a set of 4,000 gestures for communication among deaf children and the deaf community, the project also developed a series of 150 video lessons to cover math and other subjects for students in grades one through five. Though sign language is not new to the Vietnamese deaf community, it has not been introduced widely as an official language for teaching and learning for children. Since the World Bank- managed Intergenerational Deaf Education Project (IDEO) introduced a set of 2,000 gestures in 2015, however, sign language has gradually become part of preschool and kindergarten curricula. Even so, most of Vietnam’s estimated 116,000 deaf children rely solely on methods such as reading lips or wearing hearing aids to communicate. A lesson using QIPEDC provided materials in Xa Dan school for deaf children in Ha Noi capital city. Page 30 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) When QIPEDC wound down in September 2022, Vietnam’s Ministry of Education and Training agreed to authorize the use of such materials nationwide to ensure these valuable training and learning resources would continue to benefit teachers and students beyond the life of the project. Increasing the ability of hearing-impaired children to communicate delivers benefits not only to them individually, but to the deaf community and the broader society. “Thanks to the project, deaf students are better connected and communicate with one another much more,� said Nguyen Hoang Lam, one of the project’s deaf coordinators. Students’ ability to communicate through sign language also opens the door for learning opportunities, and teachers are provided with additional resources for better teaching and communicating with deaf students. Nguyen Kim Ngan, a deaf student in Hanoi’s Xa Dan primary school is one student for whom expanded interactions through signing have been life-changing. “I love social science and natural science subjects at school,� she said. “I love to come to school to play with my friends, too.� The project piloted an approach to integrate deaf children into the mainstream and special education through the engagement of parents, caregivers, teachers, and deaf mentors, as well as the community at large. Almost 400 deaf adults like Lam received training on the new signs to enable them to provide sign language education for deaf children. Tran Duc Loi, Deputy Director of Lam Dong Province Department of Education and Training said a particular value of the project was that it connected deaf children, deaf adults, parents, and teachers and created a support community for deaf children. “Now, they have the confidence to communicate with others,� Loi said. The mother of a deaf child, Le Thi Huyen, had the same epiphany. After working as a facilitator and sign language interpreter for the family support team in the IDEO project, she pursued a bachelor’s degree in special education to become a deaf teacher. She is one of the almost 1,800 parents that have received training under the QIPEDC project, despite the difficulties posed by the COVID-19 pandemic. In that respect, she was similar to Nguyen Thi Ngoc Anh, one of the 429 teachers who received training under the project, and who believed that deaf children could learn similarly to their hearing peers if they were taught sign language as early as possible. As the positive impact of the training courses became apparent, many project provinces have shared training materials with other schools outside the scope of the project to meet the needs of the teachers in specialized and inclusive schools. Page 31 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) A lesson using QIPEDC provided materials in Thuan An school for disabled children, southern Binh Duong province. With this additional training and the dissemination of training materials, the test scores of deaf students in the schools supported by the project improved significantly, achieving a 97 percent examination pass rate, well above expectations. With better learning results, Tong Thi Nga, the mother of a deaf child who has successfully passed the primary education level and now progressed to 6th grade, has an expanded view of her son’s future. Perhaps he may someday qualify for a scholarship for continued studies, maybe even abroad, she said. “I hope that he can achieve his dream,� Nga said. The project’s delivery of both educational and humanitarian benefits means the approach is worth expanding to reach older students, said Nguyen Thi Minh, Vice Minister of the Ministry of Education and Training. “Implementation has been difficult, but the results are rewarding,� she said. “The 4,000 new signs are a good start, and we will need to work together on the next phases to help with other education levels.� Page 32 of 33 The World Bank VIETNAM QUALITY IMPROVEMENT OF PRIMARY EDUCATION FOR DEAF CHILDREN PROJECT (QIPEDC) (P160543) Teacher and students in Ba Ria school for deaf students during a lesson on the Vietnamese language with video material supported by QIPEDC. Page 33 of 33