Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00006187 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF B2692 ON A SMALL GRANT IN THE AMOUNT OF EUR 2.6 MILLION (US$3.0 MILLION EQUIVALENT) AND AN ADDITIONAL SMALL GRANT IN THE AMOUNT OF EUR 1.4 MILLION (US$1.6 MILLION EQUIVALENT) TO THE REPUBLIC OF ARMENIA FOR THE EU4INNOVATION STEM PROJECT June 24, 2023 Education Global Practice Europe And Central Asia Region Regional Vice President: Antonella Bassani Country Director: Sebastian-A Molineus Regional Director: Fadia M. Saadah Practice Manager: Rita Kullberg Almeida Task Team Leader: Renata Freitas Lemos ICR Main Contributor: Juan Prawda Witenberg ABBREVIATIONS AND ACRONYMS ADS Armenia Development Strategy BETF Bank-Executed Trust Fund CFEP Center for Education Projects CPF Country Partnership Framework EC European Commission ECA Europe and Central Asia EIP Education Improvement Project E&S Environmental and Social EU European Union GACE Gordon Academic College of Education GRM Grievance Redress Mechanism ICR Implementation and Completion Results Report ICT Information and Communication Technology ISR Implementation Supervision and Results Report MM Monitoring Matrix MoESCS Ministry of Education, Science, Culture and Sport NCEDI National Center for Education Development and Innovation NIE National Institute of Education PDO Project Development Objective PID Project Information Document PIU Project Implementing Unit POM Project Operational Manual PP Procurement Plan RETF Recipient-Executed Trust Fund RF Results Framework STEM Science, Technology, Engineering, and Mathematics STEMTC STEM Technical Committee TA Technical Assistance WB World Bank TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME ...................................................................................................................... 7 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 11 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 13 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 17 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 18 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 23 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 24 ANNEX 4. SUPPORTING DOCUMENTS .................................................................................. 25 ANNEX 5. BETF ACHIEVEMENTS ........................................................................................... 27 The World Bank EU4Innovation STEM Project (P170001) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P170001 EU4Innovation STEM Project Country Financing Instrument Armenia Investment Project Financing Original EA Category Revised EA Category Organizations Borrower Implementing Agency Republic of Armenia Ministry of Education, Science, Culture and Sport Project Development Objective (PDO) Original PDO To increase the capacity of the National Center for Education Development and Innovation (NCEDI) to formulate and manage key policies affecting general education quality and improve the learning environment for STEM education in selected regions. FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-B2692 4,336,963 4,336,963 4,364,832 Total 4,336,963 4,336,963 4,364,832 Total Project Cost 4,336,963 4,336,963 4,364,832 Page 1 of 29 The World Bank EU4Innovation STEM Project (P170001) KEY DATES Approval Effectiveness Original Closing Actual Closing 30-Apr-2020 30-Oct-2020 31-Dec-2022 31-Dec-2022 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-Mar-2022 0.68 Additional Financing Change in Components and Cost KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Moderately Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 18-Sep-2020 Satisfactory Satisfactory 0.00 02 01-Nov-2021 Satisfactory Satisfactory 0.33 03 16-Nov-2022 Satisfactory Satisfactory 4.12 ADM STAFF Role At Approval At ICR Regional Vice President: Cyril E Muller Antonella Bassani Country Director: Mercy Miyang Tembon Sebastian-A Molineus Director: Fadia M. Saadah Fadia M. Saadah Practice Manager: Harry Anthony Patrinos Rita Kullberg Almeida Katia Marina Herrera Sosa, Hanna Task Team Leader(s): Renata Freitas Lemos Katriina Alasuutari Page 2 of 29 The World Bank EU4Innovation STEM Project (P170001) ICR Contributing Author: Juan Prawda Witenberg Page 3 of 29 The World Bank EU4Innovation STEM Project (P170001) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. Country poverty and economic context.1 The share of the Armenian population living below the national poverty line has dramatically decreased over the past two decades, falling from 53.5 percent in 2004 to 29.4 percent in 2016, and the economy has been on an overall upward trend since 2017. Despite this, Armenia had one of the highest poverty rates in Europe and Central Asia (ECA) at the time of appraisal with nearly 30 percent of the rural population living in poverty. Armenia’s development challenges included low human development indicators, regional and gender inequities, and rankings below that of the ECA average on the United Nation’s Human Development Index. 2. Employers in Armenia struggled to recruit and retain workers with the required occupational and transversal skills. The lack of graduates with the appropriate workforce skills was cited as a major obstacle to the activities of employers and as a significant constraint to the country’s growth. The Armenia Development Strategy (ADS) 2014–2025 identified the need to develop its knowledge-based economy by strengthening science and technology education to address some of these issues. 3. Education sector. Access to general education was universal, with a reported 91.6 percent net enrollment in general education in 2015. The national curriculum at the time of appraisal, however, was fragmented and vague, and its goals were not closely aligned to modern labor market needs. Curriculum regulations and standards were neither coordinated nor consistent and in numerous instances, links between the subject learning objectives, requirements for learners, and subject core content were not clear. Subject curricula were often overloaded and included outdated knowledge that was not linked to explicit learning goals, and the curricula of science, technology, engineering, and math (STEM) subjects were theoretical and disconnected from real-world applications. Teaching practices, instructional materials and textbooks were also outdated throughout the education system. Traditional lecture-based teaching that included tangential theoretical content, for example, was prevalent while student-centered, active learning was relatively rare. 4. Issues of equity and quality of education were persistent problems, with poor and rural students at a particular disadvantage. Teachers in rural areas, for example, were less qualified than their urban peers and lacked access to high-quality professional development. In terms of the educational attainments of the population, over 20 percent of the working age non-poor had completed a university education, compared to only 10 percent of the poor. Rural residents had the lowest educational attainment with only 9 percent having completed tertiary education compared to 25 percent in Armenia’s secondary cities and 31 percent in the capital city of Yerevan. Learning outcomes in rural areas also lagged behind those of urban areas. In mathematics, for example, the average exam score of 12th graders from large cities was 14.3 (out of 20), compared to 12.1 and 11.9 for students from areas at the border with neighboring countries and remote villages, respectively. For 9th graders, math scores were 13.3 (large cities), 12.3 (border areas), and 12.5 (remote villages). 5. Institutional capacity to guide change in the education sector was limited. The role of the now defunct 1The information source for this section of the ICR is the World Bank Armenia EU4 Innovation Stem Project - Project Information Document (PID), Report PIDC178511, dated March 20, 2019. Page 4 of 29 The World Bank EU4Innovation STEM Project (P170001) National Institute of Education (NIE) in Armenia (currently the National Center for Education Development and Innovation [NCEDI]) was vague, and it functioned like a service delivery institution by carrying out various activities defined by annual contracts with the Ministry of Education, Science, Culture and Sport (MoESCS). Several government institutions have been created since early 2000 in the field of examination, accreditation, and textbooks, but their institutional capacity was by and large limited or overlapped with the mandate and activities carried out by the NIE/NCEDI. 6. The World Bank (WB) has been supporting general education in Armenia through the recently extended Education Improvement Project (EIP - P130182) since 2014.2 This project aimed to: (i) support the review and improvement of the curricula and standards for all grades, (ii) extend preschool coverage, (iii) improve the quality of general education, and (iv) promote greater links between higher education institutions and the labor market. It also supported the National Center of Education Technology with the aim to (i) strengthen its capacity to monitor Armenian schools and provide them with adequate education-related information and communication technology (ICT) coverage, and (ii) become a national center for educational statistics. The outcomes and lessons learned from the EIP contributed to the design of this project and related Bank-executed activities. 7. Relationship with the European Union (EU) Policy Framework. This project is part of the European Commission-World Bank Partnership Program Part III for ECA Programmatic Single-Donor Trust Fund - EU4Innovation STEM Pilot Activities. It is in line with goals of the new European Consensus on Development. At the request of the Armenian Government, the European Commision (EC) entrusted the World Bank (WB) to support the Armenian MoESCS redevelop STEM curricula and pedagogies (including teacher development and management) and pilot activities in the Tavush region before implementing them nationwide. 8. The project was financed by the EC through a Trust Fund (TF073260) amounting to EUR10,150,000. This Trust Fund consisted of a Bank-executed Trust Fund (BETF), a TF management fee, and a Recipient- executed Trust Fund (RETF). The RETF, in the amount of EUR2,662,985.40, was executed by the recipient (MoECS). The BETF (P167562) grant agreement was signed on December 2018 while the signing of the RETF grant agreement (TF0B2692), also referred in this document as the project, was completed in September 2020 and declared effective on October 30, 2020. 9. The BETF was designed to expand the curriculum revision achievements of the EIP with a particular focus on: (i) finalizing the revision and modernization of the STEM curricula, (ii) developing teaching and learning materials packages across grades and subjects, and (iii) training school principals and teachers for the piloting the new curriculum in Tavush. The RETF, which was based on the finalized curriculum revision, was used to (i) strengthen the NCEDI’s role in improving teaching quality and STEM education, particularly through teacher training on modern, student-centered pedagogies, and (ii) provide laboratory equipment and furniture to improve the learning environment for STEM education 2 This operation was approved on March 13, 2014. The original closing date of April 30, 2022 was extended to October 31, 2022. In April 2022, additional financing of EUR22.6 million was approved by the WB with a new closing date of December 2025. This additional financing will, as explained in the ICR, support the scaling up of STEM-related activities. Page 5 of 29 The World Bank EU4Innovation STEM Project (P170001) in Tavush.3 The Center for Education Projects (CFEP), established in 1996 under the MoESCS, was delegated the task of managing the EIP and was also designated as the Project Implementing Unit (PIU) for the RETF. 10. Relevance of Project Development Objective (PDOs). The relevance of PDOs is high for the following reasons. First, the PDO is well aligned with the WB’s Country Partnership Framework (CPF) for the Republic of Armenia FY2018–2022 in that it directly responded to Focus Area 2 -- Enhancing Human Capital and Equity -- and one of its objectives -- Enhancing access to good quality educational services for skills development and employability -- by improving the quality and relevance of education and by improving access to economic opportunities for the poor and vulnerable. Second, the PDO to upskill the labor force by modernizing the education system was aligned to the EU Policy Framework of economic development through enhancing human capital. Third, the RETF was also in line with the 2016 “Joint Staff Working Document: Eastern Partnership – Focusing on Key Priorities and Deliverables�, which recognized the need to enhance the quality and relevance of education in Eastern Partnership countries.4 Finally, the PDO was also aligned to the priorities of the ADS 2014–2025 with respect to the development of education and the sustainable development of science and advanced technologies in that it supported the improvement of scientific infrastructure and the stable growth of highly qualified workforce in scientific and technical fields. 11. Additional financing. On May 19, 2022, the WB Management proposed a restructuring and additional financing to the Government of Armenia that would extend the project to a total of EUR4,106,000. The additional financing (EUR1,443,014.60) was to be transferred from the BETF to the RETF for the acquisition of science equipment (for chemistry, biology, physics, and geography) and establishing ICT laboratories for the 80 pilot schools in Tavush (23 general education, 52 secondary education, 4 high schools and 1 college).5 12. The RETF closed on December 31, 2022 and the BETF will close on June 30, 2023. This Implementation and Completion Results (ICR) Report for the project focuses only on the RETF portion of the Trust Fund. Given the interdependency of the project with the BETF, a description of the BETF and its key accomplishments feeding into the RETF is included as Annex 5 in this document. Project Development Objectives (PDOs) 13. The PDO of the RETF was “to increase the capacity of the NCEDI to formulate and manage key policies affecting general education quality and improve the learning environment for STEM education in selected regions�. Key Expected Outcomes and Outcome Indicators 3 The EU and the MoESCS agreed on Tavush being the pilot region mainly based on two factors: (i) building on the existing infrastructure of the "Dilijan Education Cluster" established through the international UWC College of Dilijan and AYB School, both located in Tavush, and (ii) Tavush being a borderline province having a high poverty rate. 4 The Eastern Partnership Countries include Armenia, Azerbaijan, Belarus, Georgia, Moldova, and Ukraine. 5 General Education covers grades 1 through 9, Secondary Education covers grades 1 through 12, and High School covers grades 10 through 12. Page 6 of 29 The World Bank EU4Innovation STEM Project (P170001) 14. Progress towards meeting the PDOs of the RETF were to be measured by: a. Draft teacher accreditation standards developed by a fully staffed NCEDI; and b. 80 percent of pilot regions’ middle and high schools equipped with STEM materials. Components 15. As per the Administrative Agreement between the EU and the WB, the BETF and RETF comprised three components. The BETF included Component A (Improving STEM education and piloting redeveloped curricula and pedagogy) and Component B (Program management, administration, and implementation support). 16. The RETF included Component C - Increasing capacity of the NCEDI to regulate policies affecting teaching quality and support to the STEM pilot rollout. This was to be achieved through the following two subcomponents: 17. Subcomponent C.1 - Increasing capacity of the NCEDI for Professional Development of Teachers. This subcomponent, in coordination with component A, aimed to improve NCEDI’s capacity to develop and manage continuous professional development for teachers by providing advisory services and technical assistance (TA). This subcomponent was intended to finance: (i) drafting the teacher professional development framework in correspondence with requirements set forth by the new curriculum, international best practices, and stakeholder consultations, and (ii) outsourcing the provision of teacher training to eligible training providers. 18. Subcomponent C.2 - Acquisition of hardware and equipment for the rollout of the STEM Pilot in Tavush. This subcomponent aimed to provide science equipment (for chemistry, biology, physics, and geography) and ICT laboratories for the 80 pilot schools in Tavush. The subcomponent was to also install the new equipment and ensure safety compliances were met. II. OUTCOME Assessment of the Achievement of Each Objective/Outcome 18. The ICR assesses the outcome of this project in terms of the PDOs as achieving a satisfactory rating. A set of two PDO-level results indicators and four intermediate results indicators were designed to monitor progress towards the achievement of the PDOs. Both PDOs indicators have been achieved or exceeded, while two of the four intermediate indicators were achieved or exceeded. The remaining two intermediate indicators were partially achieved as explained below. Outcome 1: Increase the capacity of the NCEDI to formulate and manage key policies affecting general education quality. Page 7 of 29 The World Bank EU4Innovation STEM Project (P170001) Indicator: Draft teacher accreditation standards developed by NCEDI staff with an end-of-project target of a completed draft of teacher accreditation standards. 19. Target fully achieved. The NCEDI has developed (i) standards for the accreditation of in-service teacher training providers, (ii) draft questionnaire to be used for the quality assurance system to evaluate and monitor delivery of training, and (iii) draft teacher attestation system. NCEDI has leveraged technical support from the British Council to develop the attestation system. 20. Currently, 50 public and private educational institutions with a legal entity and state registration have been selected by the NCEDI to be of in-service teacher training providers to 14,000 teachers. NCEDI is simultaneously carrying out an evaluation of these training providers. Additionally, 42 schools have been designated by the NCEDI as “mentoring schools� with the aim to support the nationwide scale- up of teacher training and mentoring-related interventions. The NCEDI is working on building capacity for the proposed scale-up by addressing the need for sufficiently qualified teacher trainers and has therefore designed a 12-hour hybrid training program (6 hours online and 6 hours offline) on the structure and the focus of the new standards. To roll out this training program, NCEDI will hire all teachers from the 42 mentoring schools on a contract basis with government resources to train the remaining teachers in Armenia’s schools. 21. The attestation system is aligned to the new subject learning standards (State Standard of General Education) developed by the project under the BETF, adopted by the Government of Armenia in April 2021, and piloted in September 2021. The teacher attestation system outlines the knowledge, practice, and professional engagement required from teachers. The development of these standards has been guided by: (i) international best practices, (ii) frequent stakeholder consultations, and (iii) alignment with the requirements set forth by the new curriculum (developed under the BETF). Outcome 2: Improve the learning environment for STEM education in selected regions. Indicator: Pilot regions’ middle and high schools equipped with STEM material with an end- of-project target of 80 percent. 22. Target exceeded. One hundred percent of the 80 Tavush middle and high schools were equipped with appropriate STEM teaching and learning materials, which is 20 percentage points above the end-of- project target. The above procurement included (i) laboratory equipment and supplies for physics, chemistry-biology, geography, and ICT, and (ii) computer equipment for STEM classrooms (all-in-one computers, Wi-Fi routers, lab teacher notebooks, uninterruptible power supply, 3D printers, LCD projectors, and multifunction devices). Ninety-six percent of the 80 Tavush middle and high schools (77 schools) were equipped with laboratory furniture and infrastructure. The procurement and delivery of ICT materials (cables, electronic accessories, power tools, storage) was canceled due to lack of funds (due to the devaluation of the euro against the dram) and will be instead financed by the government. 23. Minor civil works the provision of adequate supply of water and drainage systems in chemistry and biology laboratories were carried out in 76 schools in strict compliance with the WB environmental guidelines. Page 8 of 29 The World Bank EU4Innovation STEM Project (P170001) 24. Furthermore, a quasi-experimental design using a difference-in-differences technique was implemented in May 2022 by the WB team under the BETF. Its purpose was to provide a causal estimate of the effects on student learning of the combined activities of the BETF and the RETF during the pilot implementation in Tavush.6 Learning assessments were carried out in pilot and non-pilot grades of approximately 210 schools in three regions across the country (Tavush, Lori, and Shirak). The first difference to be estimated was between the treated grades (2, 5, 7) and the immediately following untreated grades (3, 6, 8). The second difference was between the treated region (Tavush) and untreated regions (Lori and Shirak, which showed parallel trends in terms of student outcomes and school inputs and thus served as an appropriate comparison group). Preliminary results have shown an average positive impact of 0.13 of a standard deviation on student performance in mathematics and selected science subjects in the first year. 25. The above outcome and intermediate results accomplishments have been largely underpinned by the achievement of the interconnected BETF portion of this project as described in Annex 5. 26. RETF intermediate results. As shown in table 1, among the four intermediate results indicators, two were fully achieved and two were partially achieved for the reasons explained below. Table 1: Assessment of Intermediate Results Indicators Subcomponent Intermediate End-of- Assessment results indicator project target Target partially achieved. In March 2022, the Gordon C1: Increasing Percentage of 25% Academic College of Education (GACE) in Haifa, Israel was capacity of the relevant NCEDI competitively selected to provide the first stage of technical NCEDI for staff trained in assistance (TA), which was intended to strengthen the Professional the teacher NCEDI’s governance structure, charter, and functions. GACE Development accreditation provided training to NCEDI staff on: (i) innovation in of Teachers standards pedagogy and education, (ii) age appropriate pedagogy for primary and secondary school education, (iii) continued professional development of in-service teachers, (iv) innovative learning environments, (v) integrating film and media into teaching and learning, (vi) digital pedagogy, (vii) integrating informal educational approaches into formal education, (viii) assessing the quality of training programs, and (ix) monitoring and evaluation in education. This first stage was satisfactorily completed, that is, 14 out of a total of 34 relevant NCEDI staff (staff in the departments of management, training, disciplines, assessment) received the first stage training. This is equivalent to 41.1% of relevant staff. 6The WB designed the evaluation and the instruments and is doing the data analysis. The data collection was undertaken by the Ayb Education Foundation, an Armenian private organization founded in 2006 supporting the professional development of teachers. Page 9 of 29 The World Bank EU4Innovation STEM Project (P170001) The second stage of TA focused on further strengthening the quality assurance mechanism at the NCEDI to regulate policies affecting the professional development of teachers. This was to be accomplished by developing software for accreditation standards of training providers but had to be canceled under the project due to lack of funds. The end-of- project target would have been surpassed if the training was completed in full. There are currently no plans for the completion of this activity. In addition, 50 laptops were provided to the NCEDI, which allowed them to better organize training and monitoring activities of the pilot. Target partially achieved. During a survey that took place in C2: Acquisition Percentage of May 2022, 95.2 percent of teachers responded yes to the of hardware beneficiaries in 80% following question: are teaching and learning materials in and pilot region line with the curriculum? equipment for satisfied with the rollout of quality of STEM A follow-up teacher satisfaction survey was carried out the STEM Pilot learning materials between April and May 2023 to provide data for the ICR as in Tavush proposed under the November 2022 Implementation Status & Results Report to directly measure satisfaction levels. The following question was asked: on a scale of 1–5, rate how satisfied or dissatisfied you are with the quality of teaching and learning materials in [STEM subject]. Twenty percent responded that they were very satisfied and 50.8 percent were satisfied, meaning a total of 70.8 percent of teachers were satisfied with the quality of STEM teaching and learning materials. Only 26.7 percent responded that they were neither satisfied or dissatisfied, 0.8 percent were unsatisfied and 1.7 percent were very unsatisfied. Percentage of Target fully achieved. All Tavush middle schools were middle school equipped with appropriate STEM teaching and learning 100% students in pilot materials meaning that all students in those schools region benefiting benefitted from these materials. from STEM materials Percentage of Target fully achieved. All Tavush middle schools were middle school equipped with appropriate STEM teaching and learning 100% female students materials meaning that all female students in those schools in pilot region benefitted from these materials. benefiting from STEM materials Page 10 of 29 The World Bank EU4Innovation STEM Project (P170001) 27. The above achievements were largely supported by the achievements of the BETF intermediate indicators as explained in Annex 5. Overall Outcome Rating: Satisfactory 28. The ICR rates the overall outcome of the project as satisfactory because there were minor shortcomings in the achievement of its objectives (intended outcomes). Other Outcomes and Impacts 29. The results and lessons learned from this pilot intervention will inform the national rollout of the curriculum reform in the next two to three years. The EU and MoESCS are currently discussing the funding for the scale- up in three more regions (Lori, Shirak, and Syunik). The additional financing approved for the EIP on April 2022 for EUR22.6 million (US$25 million equivalent) will support the acquisition of approximately US$8.8 million (35.2 percent of the total additional financing) worth of STEM laboratory equipment and furniture for roughly 200 general education schools. These funds will also be used towards: (i) teacher training on the use of the equipment, and (ii) regularly monitoring and recording the use of the equipment through classroom observations. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME Factors that Positively Affected Implementation7 32. Strong counterpart ownership and continuity. Intermediate and final deliverables developed by the group of national and international experts under the RETF were frequently reviewed, discussed, and vetted in several well-structured workshops for education experts, teachers and leaders, university professors and the broader education community in Armenia. During these workshops, the education curriculum/STEM reform, teacher training, and teacher accreditation standards were explored and discussed. As a result of this collaborative process, there were no serious issues at the approval and endorsement stages of final deliverables. Fixing the technical specifications for the acquisition of STEM learning materials and equipment did, however, require multiple and at times, lengthy reviews. 33. Robust selection of local and international TA. This factor underpinned the entire project implementation process including: (i) the selection of GACE to support the initial stage in strengthening of the institutional capacity of the NCEDI, and (ii) the TA supporting the CFEP to develop technical specifications for the acquisition of STEM teaching and learning materials under the project. 34. Appropriate sequencing of implementation stages. By and large, the different activities under the project complied with the appropriate implementation process sequence. The first stage included the collection and analysis of data to identify and understand relevant issues and to develop appropriate ways to address them in the Armenian context. Following this, qualified national and international 7This section of the ICR is based on the analysis of the content of the project and the outcomes of the online meetings held between the heads of the CFEP and NCEDI and the WB fiduciary and technical teams. Page 11 of 29 The World Bank EU4Innovation STEM Project (P170001) experts developed the corresponding envisaged deliverables (e.g., the teacher attestation system, the procurement STEM and ICT equipment, and the recruitment of qualified TA to support NCEDI). Next, key stakeholders discussed the intermediate deliverables, usually in the form of well-structured and functional workshops. Before the final deliverables were submitted to the MoESCS for endorsement and approval, there was a vetting and reviewing process. Finally, once the deliverables were being used, evaluations were made to assess their impact. 35. Following the above sequential process, the pilot of the redeveloped curriculum was implemented in grades 2, 5, 7, and 10 in all schools in Tavush. Teacher training on monitoring and mentoring accompanied the pilot. An evaluation of the pilot is currently being carried out under the BETF and will be completed in June 2023. Preliminary results of this evaluation point to the fact that the revised curriculum is still overloaded and will require a second revision to streamline it under the BETF. The NCEDI is participating in the current second revision of the curriculum. The final evaluation findings will provide informed recommendations for the rollout of the revised STEM curriculum and contribute to a global roadmap of nationwide curriculum reforms. 36. Robust articulation of project activities. For example, half of the experts involved in the development of the STEM teaching and learning materials were directly involved in the provision of mentoring STEM teachers in the 2021–2022 period covering the main pilot year, in coordination with NCEDI. This mentoring specifically focused on identifying the problems that teachers came across in implementing the new curricula and new teaching. The four types of mentoring meetings included: (i) explaining new topics in the curriculum and methods used for their presentation in the classroom, (ii) weekly updates, (iii) reviewing specific project work that might require further guidance, (iv) use of formative assessments, and (v) classroom observations. 37. Visibility. All project activities are conducted in accordance with EU Visibility recommendations and guidelines including EU branding on all documentation and presentation materials. 38. Disbursements. As shown in the table below, the project that closed on December 31, 2022 was fully disbursed. Grant Allocation (EUR) Disbursement (EUR) Disbursement ratio (1) (2) (2)/(1) TF0B2692 4,106,000 4,106,000 100% Factors that Negatively Affected Implementation 39. CFEP’s initiation of procurement activities based on financial ceiling expectations without evidence that the corresponding financial resources were available. The initial grant agreement (TF0B2692) was signed in September 2020 for EUR2,662,985.40 and the first transfer of funds from the BETF to the project took place in May 2021. In the same month, the EU agreed in-principle to an additional financing to the RETF grant agreement as a transfer from the BETF for a total of EUR4,106,000 (US$4,084,238 equivalent). The WB gave no objection to a Procurement Plan (PP) for EUR4,106,000 in June 2021, however, without having yet processed the grant agreement amendment. The Page 12 of 29 The World Bank EU4Innovation STEM Project (P170001) amendment to the grant agreement to reflect this additional financing was only signed in May 19, 2022, and became effective on October 30, 2022. The CFEP signed four vendor contracts for laboratory furniture and ICT equipment and for the training of NCEDI staff prior to the grant agreement becoming effective. For this reason, the CFEP was unable to pay these vendors on time, incurring in late payment fees and, in one case, arbitration procedures. 40. Four-month delay in transfer of funds due to interpretation disagreements between the EU and the WB over one article of the Grant Administration Agreement. There was a four-month delay in processing the project’s increase of funds caused by the EU's suspension of the final payment request made by the WB on June 13, 2022. This was due to a disagreement over the interpretation of whether increases to existing grant agreements were subject to Article 12 of the Administration Agreement.8 This disagreement was resolved on October 11, 2022, and thus, the Amended Grant Agreement between the Government of Armenia and the WB framing this additional financing to the Project became effective at the end of October 2022 (11 weeks before the Project’s closing date). This delay generated significant reputational and financial risks to the CFEP and the WB described above. 41. Devaluation of the euro against the Armenian dram. The devaluation of the euro against the Armenian dram (-30.6% between March and October 2022) resulted in a financial gap difficult to buffer, and led to the cancellation of: (i) the second stage of TA for strengthening the capacities of NCEDI, (ii) the recruitment of TA to develop software for accreditation standards of in-service teacher training providers, and (iii) the purchase of part of the necessary ICT equipment.9 42. The invoicing requirement to reach a disbursement threshold of 70 percent of the BETF prevented a more timely invoicing to the EU concerning the additional financing to the project. The request to make the final transfer of funds from the BETF to the project was subject to an agreement concerning the above-mentioned 70 percent disbursement threshold. This threshold was reached by June 2022, at which time the request/invoicing for the final transfer was made by the WB to the EU. 43. COVID-19 protocols. The COVID-19 pandemic disrupted activities in 2020 and a significant part of 2021. Many project-related activities needed to be conducted online, thus slowing down the speed of the implementation process. No field supervision missions were held during this period. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME Bank performance Rating: Moderately Satisfactory 44. Quality at entry rated satisfactory. First, the RETF’s PDOs were highly relevant at the time of preparation and they continue to be consistent with the current WB Armenia CPF at the closing of the RETF. Second, the intermediate outputs of the BETF’s implementation guided the drafting of: (i) ToRs for the selection of a qualified TA to support the institutional capacity development of the NCEDI, and (ii) technical specifications for the acquisition of appropriate STEM-related teaching and 8 Article 12 states that “Individual procurement and grant contracts under this Administration Agreement shall be signed by the World Bank Group no later than 36 (thirty-six months)� from the start of the implementation period (December 23, 2018). 9 EU/WB allocated funds in EUR but purchases of goods and services were mostly in Armenian drams. Page 13 of 29 The World Bank EU4Innovation STEM Project (P170001) learning materials and laboratory and ICT equipment. Third, the teacher training modules utilized in the pilot phase were created largely in response to the results of the TEACH classroom observation tool, which captured the quality of teaching practices including classroom culture, instruction, and socioemotional skills. This instrument was also useful in identifying gaps in teachers’ skill levels, professional development needs, and use of instructional materials and classroom practices, which was critical to the design of the teaching accreditation standards by the NCEDI. Fourth, the WB team supported the development of a Project Operational Manual (POM), which was approved and adopted in May 2020. 45. Quality at supervision rated moderately satisfactory. The WB reviewed the project and BETF every six months and reported their findings and recommendation in substantive Aide-Memoires, which reflect the achievements (including frequent updates to the results framework and monitoring matrix), challenges, and ways forward in the short and medium term.10 Second, the supervision teams had the right mix of skill sets required to ensure: (i) the implementation of the project consistently adhered to best international practices, (ii) the provision of institutional support to the NCEDI, and (iii) the acquisition of appropriate STEM learning and teaching materials and equipment. 46. An issue that arose during supervision, however, was the WB’s no objection to the RETF PP without ensuring that the grant agreement increase was signed. The CFEP moved forward with procurement activities without additional financing in place. The WB team rectified this issue and was able to and ensure the procurement of STEM teaching and learning equipment and materials before the closing of the project. However, this oversight still created reputational risks to the CFEP and the WB, and caused the cancelation of key activities. 47. Justification of the overall rating of Bank performance. The ICR rates the overall Bank performance as moderately satisfactory in view of the satisfactory rating given to the quality at entry and the moderately satisfactory to the supervision. M&E Quality Rating: Substantial 48. Project implementation progress and achievement of outcomes and intermediate outputs were tracked through a monitoring and evaluation (M&E) design that included two sets of data sources: (i) the indicators in the results framework (RF) and monitoring matrix (MM), and (ii) the outcomes generated by several tailored-made studies and surveys. The following aspects of the M&E design were appropriate and relevant: (i) the inclusion of a set of two PDO-level results indicators and four intermediate results indicators in the RF, (ii) the evaluation of the pilot, and (iii) the use of the TEACH classroom observation instrument to assess teaching practices in Tavush. 49. The indicators in the RF and MM were updated every six months. The TEACH instrument was used before the start of the pilot to inform the development of teacher training modules and teacher accreditation standards. The first measurement of the pilot was completed in May 2022 and a second 10 Due to COVID-19 protocols, on-site visits were suspended in the latter part of 2020 and during most of 2021, and meetings were instead conducted online. Site visits resumed in 2022. Page 14 of 29 The World Bank EU4Innovation STEM Project (P170001) survey was made in April 2023. The project’s M&E was therefore timely and systematically implemented. 50. The M&E findings have been utilized to identify successes and inform the scaling up of the pilot to other regions. They have been also used to correct course. The evaluation of the pilot, for example, demonstrated that the revised curriculum was still overloaded. A second revision of the curriculum was therefore made with the support of the NCEDI to further simplify it. 51. Justification of the overall rating of M&E quality. The ICR rates the overall quality of M&E as substantial because: (i) there were no shortcomings in its design and implementation, (ii) its design was sufficient to assess progress and flag issues to be addressed, and (iii) it informs the future direction of the project (the scaling up of the pilot to other three regions). Compliance issues 52. Financial management performance under the project was in full compliance with the financial covenants of the legal agreements. The CFEP maintained adequate financial management of staffing and organization, project planning and budgeting, accounting, funds flow, reporting, internal controls and external audit. The three WB financial management supervision missions undertaken during project implementation reported that all biannual, unaudited financial statements were submitted and found to be acceptable. Only one audited financial statement covering the entire implementation period was submitted within six months after the project closing date. 53. Procurement performance under the project was in full compliance with the procurement covenants of the legal agreements. The CFEP followed WB procedures in managing the fiduciary function of procurement with regard to: (i) planning (with one exception explained below), (ii) budgeting, and (iii) procurement/selection activities. Additionally, CFEP was staffed by experienced procurement specialists that regularly attended trainings organized by the WB. 54. The procurement process for seven lots under the project was launched by the CFEP in mid-2021 and completed at the end of 2022. All contracted equipment was satisfactory delivered with the exception of lot 4 (chemistry equipment), where 12 line-items (out of a total of 131 in this contract) had significant deviations from the technical specifications. Because the corresponding firm refused to replace these products, the recipient returned the items and deducted the corresponding amount (EUR6,356, roughly 1.7% of the total value of the contract) from its last payment. Procurement of ICT materials was also satisfactorily completed with the exception of lot 6, which was cancelled in October 2022. The contract with the winning supplier was not awarded due to the lack of funds caused by: (i) the significant appreciation of the Armenian dram against the euro, and (ii) the delay in completing the additional financing transfer from the BETF to the project. 55. One key shortcoming of procurement performance was the WB no objection to the PP in June 2021 at which time the transfer of the grant agreement increase had not yet been processed and made effective. Based on this financial ceiling expectation, the CFEP launched the corresponding procurement tenders and signed the contracts with the selected vendors, but could not carry out the resulting payments in a timely manner. Page 15 of 29 The World Bank EU4Innovation STEM Project (P170001) 56. Environmental and social (E&S) requirements were met. An Environmental and Social Commitment Plan and an Environmental and Social Management Plan were developed and approved for the project. No environmental and social issues were flagged during the project implementation process. The four relevant E&S standards applicable to the project included: (i) ESS1: Assessment and Management of Environmental and Social Risks and Impacts, (ii) ESS10: Stakeholder Engagement and Information Disclosure, (iii) ESS2: Labor and Working Conditions, and (iv) ESS3: Resource Efficiency and Pollution Prevention and Management. All were rated satisfactory by WB supervision and support missions with the exception of ESS1, which was rated moderately satisfactory in 2021. The was due to the fact that the project had not yet recruited an E&S specialist, meaning the E&S reporting requirements were not followed. This issue was satisfactorily solved in January 2022 with the recruitment of an E&S specialist, which allowed the corresponding E&S required activities to be implemented. This, in turn, led to upgrading the rating to satisfactory. 57. Grievance redress mechanism (GRM) performed well. The project used the GRM detailed in the EIP POM. This GRM was accessible to project-affected people, which ensured that beneficiaries and the wider community had a means to provide project staff with practical feedback. Such feedback improves accountability, transparency, responsiveness to beneficiaries, and increases stakeholder involvement in the project. CFEP staff received periodic guidance from the WB upon request. No project related grievances or questions were logged except for the vendor that started arbitration procedures due to late payments. Risk to Development Outcome Rating: Moderate 58. The results from the BETF and the RETF will inform the scale-up of the curriculum reform in the next two to three years throughout the country. At the time this ICR was drafted, the EU and the MoESCS were discussing funding for the scale-up of the STEM project in three more regions (Lori, Shirak, and Syunik). The approved EIP Additional Financing was also being used to: (i) support the acquisition of STEM teaching and learning material and equipment for a further 200 general education schools, and (ii) train teachers on how to both use the equipment and how to monitor its use through classroom observations. 59. Despite the overall positive trajectory of development outcomes, there were some setbacks in terms of strengthening the institutional capacity of the NCEDI. The second stage of TA that was meant to support professional development for teachers and the creation of a quality assurance mechanism of in-service teacher training was canceled in June 2022. Up to the drafting of this ICR, there is no sufficient clarity on how this required TA is going to be financed to further support capacity building of NCEDI, thus posing a moderate risk to the development outcome in the medium-term. Despite this setback, the NCEDI is working on building capacity for the proposed scale-up by addressing the need for sufficiently qualified teacher trainers through the setting up of mentoring schools and the hiring of their teachers to provide cascade training to remaining teachers in Armenia. 60. Justification of the overall rating of M&E quality. The ICR rates the above risks to sustaining the development outcomes as substantial because once the Government of Armenia and other partners Page 16 of 29 The World Bank EU4Innovation STEM Project (P170001) have agreed upon the financial support, the plan provide further technical assistance to the NCEDI has already been made, and the revised streamlined curriculum could be implemented. V. LESSONS LEARNED AND RECOMMENDATIONS 61. The following lessons and recommendations can be drawn: a. The effective and timely sequenced articulation of the BETF and RETF resulted in a robust project design and implementation, and that by and large achieved its intended objectives. The shared management of available financial resources by the WB and by the recipient is a lesson worth emulating if it is appropriately articulated and effectively monitored. b. Continuous on-site monitoring is required to assess what is working and what is not so that actions can be modified in a timely manner. Examples of positive lessons learned during the implementation of the project that would be worth emulating in the national scale-up include: (i) the mentoring undertaken during the training of STEM teachers, and (ii) the class observations and use of formative assessment that monitored the teaching of STEM subjects and how well students were learning. c. The clarification of definitions in the agreements among the signing parties can be improved to ensure a smooth and timely implementation process. The four-month delay in processing the transfer of additional financing due to disagreements in the interpretation of Article 12 of the Grant Administration Agreement is an example of a lesson learned that should be avoided or mitigated in future operations. . Page 17 of 29 The World Bank EU4Innovation STEM Project (P170001) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increase the capacity of the National Center for Education Development and Innovation (NCEDI) to formulate and manage key policies affecting general education quality. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Draft teacher accreditation Text No teacher Draft teacher Draft teacher Draft teacher standards developed by NCEDI accreditation accreditation accreditation accreditation staff standards. standards developed standards developed standards developed 01-Apr-2020 01-Apr-2020 30-Nov-2022 30-Dec-2022 Comments (achievements against targets): Target fully achieved. Accreditation standards approved by the MoESCS and in use. Objective/Outcome: Improve the learning environment for STEM education in selected regions. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Pilot regions’ middle and high Percentage 0.00 80.00 80.00 100.00 schools equipped with STEM material 01-Apr-2020 01-Apr-2020 01-Jun-2022 30-Dec-2022 Page 18 of 29 The World Bank EU4Innovation STEM Project (P170001) Comments (achievements against targets): Target exceeded as 100% of pilot region’s middle and high schools were equipped with appropriate STEM teaching and learning materials and equipment. A.2 Intermediate Results Indicators Component: Increasing capacity of the NCEDI for Professional Development of Teachers. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Relevant NCEDI staff trained in Percentage 0.00 25.00 25.00 0.00 the teacher accreditation standards 01-Apr-2020 01-Apr-2020 30-Dec-2022 30-Dec-2022 Comments (achievements against targets): Target partially achieved as 14 NCEDI staff (out of 34 relevant NCEDI staff in the departments of management, training, disciplines, assessment), equivalent to 41.1% of relevant staff, received initial, albeit not complete training. This activity was cancelled in June 2022 due to lack of funds (due to the devaluation of the euro against the Armenian dram) to finance the second stage of institutional capacity strengthening. The end-of-project target would have been surpassed if the training was completed in full. Component: Acquisition of hardware and equipment for the rollout of the STEM Pilot in Tavush Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Beneficiaries in pilot region Percentage 0.00 80.00 80.00 70.80 Page 19 of 29 The World Bank EU4Innovation STEM Project (P170001) satisfied with quality of STEM 01-Apr-2020 01-Apr-2020 30-Dec-2022 31-May-2023 learning materials Comments (achievements against targets): Target partially achieved. A survey carried out in May 2023 asked the following question: on a scale of 1–5, rate how satisfied or dissatisfied you are with the quality of teaching and learning materials in [STEM subject]. 20 percent responded that they were very satisfied and 50.8 percent were satisfied, meaning a total of 70.8 percent of teachers were satisfied with the quality of STEM teaching and learning materials. Only 26.7 percent responded that they were neither satisfied or dissatisfied, 0.8 percent were unsatisfied and 1.7 percent were very unsatisfied. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Middle school students in pilot Percentage 0.00 100.00 100.00 100.00 region benefiting from STEM materials 01-Apr-2020 01-Apr-2020 30-Jun-2022 30-Dec-2022 Middle school students in Percentage 0.00 100.00 100.00 100.00 pilot region benefiting from STEM materials--Female 01-Apr-2020 01-Apr-2020 30-Dec-2022 30-Dec-2022 Comments (achievements against targets): Target fully achieved as per above survey. Page 20 of 29 The World Bank EU4Innovation STEM Project (P170001) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1: Increase the capacity of the National Center for Education Development and Innovation (NCEDI) to formulate and manage key policies affecting general education quality. 1. Draft teacher accreditation standards developed. Target fully Outcome Indicators achieved. Accreditation standards approved by the MoESCS and in use. 1. Percentage of relevant NCEDI staff trained in the teacher accreditation standards going from a baseline of 0 to an end-of-project target of 25%. Target partially achieved. 14 NCEDI staff (out of 34 relevant NCEDI staff in the departments of management, training, Intermediate Results Indicators disciplines, assessment), equivalent to 41.1% of relevant staff, received initial, albeit not complete training. This activity was cancelled in June 2022 due to lack of funds (due to the devaluation of the euro against the Armenian dram) to finance the second stage of institutional capacity strengthening. 1. Subcomponent C1: Increasing capacity of the NCEDI for Professional Development of Teachers. First stage of technical assistance Key Outputs by Component provided by GACE completed. Second stage cancelled due to lack of (linked to the achievement of the Objective/Outcome 1) funds. Objective/Outcome 2: Improve the learning environment for STEM education in selected regions. 1. 80 percent of pilot regions’ middle and high schools equipped with STEM material. Target exceeded as 100% of pilot region’s middle and Outcome Indicators high schools were equipped with appropriate STEM teaching and learning materials and equipment. Page 21 of 29 The World Bank EU4Innovation STEM Project (P170001) 1. 80 percent beneficiaries in pilot region satisfied with quality of STEM learning materials. Target partially achieved as per survey carried out in May 2022. Intermediate Results Indicators 2. 100 percent of middle school students in pilot region benefiting from STEM materials. Target fully achieved. 3. 100 percent of female middle school students in pilot region benefiting from STEM materials. Target fully achieved. 1. Subcomponent C2: Acquisition of hardware and equipment for the rollout of the STEM Pilot in Tavush. Target exceeded as 100 percent of middle and high schools (80) have been equipped with appropriate STEM teaching and learning materials including: Physics Laboratory Equipment, Chemistry-Biology Laboratory Equipment, Chemistry- Biology Laboratory Electrical Equipment, Chemistry-Biology Laboratory Supplies, Geography Lab Equipment, ICT materials and Chemistry- Biology Laboratory Materials. Laboratory furniture was provided to all Key Outputs by Component schools in Tavush region including water supply and drainage in Key outputs by (linked to the achievement of the Objective/Outcome) chemistry-biology laboratories. In addition, computer equipment for STEM classrooms was also acquired under the Project including: all-in- one computers for students, UPS, Wi-Fi routers, lab teacher notebooks, 3D printers, LCD projectors, and multifunction devices. The Center for Education Projects PIU (CFEP), established in 1996 in the MoESCS, has also completed procurement of notebooks for NCEDI needs and provided 50 notebook computers to NCEDI, which allowed them to better organize the monitoring activities of the Pilot. Page 22 of 29 The World Bank EU4Innovation STEM Project (P170001) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Increasing the Capacity of the NCEDI to Regulate Policies Affecting General Education Quality and Improve the 3,001,450.84 4,627,872.59 154 Learning Environment for STEM Education in Selected Regions Total 3,001,450.84 4,627,872.59 154 Page 23 of 29 The World Bank EU4Innovation STEM Project (P170001) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS No comments provided. June 24, 2023 Page 24 of 29 11 The World Bank EU4Innovation STEM Project (P170001) ANNEX 4. SUPPORTING DOCUMENTS At the preparation stage World Bank. Armenia: EU4Innovations STEM Project. Project Information Document (PID) Identification/Concept Stage. Report PIDC178511, March 20, 2019. Administration Agreement between the European Commission and the International Bank for Reconstruction and Development and the International Development Association concerning the European Commission - World Bank Partnership Program Part III for Europe and Central Asia Programmatic Single-Donor Trust Fund (TF073260) EU4Innuvadon STEM Pilot Activities(Donor Reference No. 20181404-659), December 28, 2018. Country Partnership Framework (CPF) forthe Republic of Armenia FY2018-2022 At the restructuring stage World Bank. Armenia: EU4Innovations STEM Project. Project Paper on a Proposed Restructuring and Additional Financing. April 20, 2020. World Bank. Armenia: Education Improvement Project (EIP) Additional Financing. Project Paper. Report No PAD 4443. April 29, 2022 At the supervision stage Amendment No 1 to the Administration Agreement between the European Commission and the International Bank for Reconstruction and development and the International Development Association concerning the European Commission – World Bank Partnership Program Part III for Europe and Central Asia Programmatic Single-Donor Trust Fund (TF073260) – EU4Innovations STEM Pilot Activities (Donor Reference No. 2018/404-659), September 1, 2020. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status Report 1 June 2019. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status Report 2 December 2019. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status Report 3 December 2020. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status Report 4 June 2021. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status Report 5 December 2021. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status June 24, 2023 Page 25 of 29 11 The World Bank EU4Innovation STEM Project (P170001) Report 6 June 2022. European Commission – World Bank. Armenia: EU4Innovations STEM Pilot Activities. Implementation Status Report 7 December 2022. World Bank. Armenia: EU4Innovations STEM Project (P170001). Implementation Status & Results Report (ISR) No. 1, September 18, 2020 World Bank. Armenia: EU4Innovations STEM Project (P170001). Implementation Status & Results Report (ISR) No. 2, November 1, 2021 World Bank. Armenia: EU4Innovations STEM Project (P170001). Implementation Status & Results Report (ISR) No. 3, November 16, 2022 World Bank. Countersigned Amendment No 1 to the Grant Agreement. May 19, 2022. June 24, 2023 Page 26 of 29 11 The World Bank EU4Innovation STEM Project (P170001) ANNEX 5. BETF ACHIEVEMENTS 1. The PDO of the BETF was “to improve the quality of STEM education in general education in the pilot region of Tavush�. 2. The BETF included the first component, Component A - Improving STEM education and piloting redeveloped curricula and pedagogy. This component aimed to: (i) modernize STEM curricula and teacher training (grades 1 to 12) for student-centred pedagogies that use best international practices, and (ii) develop teaching and learning materials for secondary education (grades 5 to 12). This component included the following two subcomponents: 3. Subcomponent A.1 - Redevelopment of STEM curricula and pedagogies. This subcomponent aimed to (i) set up a STEM Technical Committee (TC), (ii) support the review of the content, sequence, levels of difficulty, and coherence of the current STEM curricula in grades 1 to 12, (iii) develop teaching and learning material, including lesson plans, draft textbooks, and assessment instruments compatible with the redeveloped curricula, and (iv) develop teacher training modules for STEM pilot for grades 5 to 12. 4. Subcomponent A.2 - Pilot of redeveloped STEM curricula and pedagogies. This subcomponent aimed to evaluate the effectiveness of new draft STEM curricula in increasing aggregate learning among students in the pilot region by (i) testing the revised curriculum and teaching and learning materials at the classroom level, and (ii) implementing the redeveloped curriculum in classrooms and laboratories of grades 5 to 12 for one year in the pilot region. This subcomponent financed the following key activities: 5. Providing training to STEM teacher and senior school administration to emphasize modern, student- centered pedagogies and innovative methodologies (e.g., blended learning) for effective delivery of STEM education; a. Developing and rolling out of the pilot in the Tavush province to evaluate the new draft curricula (subject syllabi) and teaching and learning materials (lesson plans, draft textbooks, and inputs to assessment instruments); and b. Formulating recommendations to the government based on the results of the STEM pilot, and preparing final draft curricula, draft textbooks, and associated package of materials for grades 5– 12. 6. The BETF also included another Component B - Program management, administration, and implementation support. This component aimed to support program governance arrangements and Trust Fund related activities like meetings, planning, communication, outreach, dissemination, reporting, monitoring, and evaluation. 7. Progress towards meeting the PDO of the BETF was to be measured by: a. Draft STEM curricula (for grades 1 to 12) and related materials for grades 5 to 12 (like syllabi, lesson plans, draft textbooks, and inputs to assessment instruments) developed and submitted to the MoESCS; The draft STEM curricula for grades 1 to 12 and related teaching and learning materials including subject syllabi, lesson plans and inputs for assessment instruments have been developed and approved by the MoESCS. These materials are being currently used in the pilot June 24, 2023 Page 27 of 29 11 The World Bank EU4Innovation STEM Project (P170001) grades (2, 5, 7 and 10) in Tavush. In addition to the pilot grades, resources, teaching and learning materials have been also developed for all STEM subjects for grades 1, 3, 4, 6, 8, 9, 11 and 12 and are being currently piloted in Tavush. All these materials have been a valuable input for the NCEDI’s realization of its envisaged tasks, like for example, the setting of teacher accreditation standards. b. Increased average score of graduation exam in math of 9th grade students in the pilot region as compared to a baseline of 10.82 out of 20. This outcome still needs to be fully assessed, although there are strong indications from an impact evaluation carried out under the BETF that student performance has improved. The end-of-project measurement would be assessed and recorded two years after the end of the pilot for students who are currently in grade 7 on or about the school year 2024-2025. 8. The above achievements were largely supported by the target attainments of the BETF intermediate indicators as shown in Table 2 below: Table 2: Assessment of BETF Intermediate Results Indicators Under the BETF Subcomponent Intermediate results End-of-project target Assessment indicator Number of redeveloped Draft syllabi for grades 1- Achieved. New draft STEM subject syllabi and 12 subject standards for lesson plans grades 1-12 have been produced and adopted by the MoESCS A1: Redevelopment of Number of revised STEM At least 10 revised STEM Achieved. Local working STEM curricula and textbooks in line with the draft textbooks for grades groups have completed pedagogies new curricula and 5-12 the draft of 60 sets of international standards teaching and learning materials and teacher guides for all STEM subjects and grades, but targeting more grades 7 to 12, that will serve for the development of STEM textbooks. A2: Pilot of Selection of teacher A firm or firms will be hired Completed redeveloped STEM training and mentoring to undertake the curricula and teacher/school leader pedagogies training under ToRs satisfactory to the WB and the MoESCS Development of teacher The modules of the new Completed training modules STEM standards and the teacher training fully developed June 24, 2023 Page 28 of 29 11 The World Bank EU4Innovation STEM Project (P170001) Implementation of the A training firm will handle Completed teacher training and all the logistical/substantive mentoring preparation for the teacher Teacher training and training to take place with mentoring has taken place. guidance and advice from the WB team. The teacher Over 600 teachers have training will include been trained in online and mentoring along the school in-person meetings. year of teachers so that they can improve their At the time of drafting this teacher practices. ICR a total of 28 Teacher Group Meetings, 31 Mentorship Meetings and 130 lessons observations have taken place. Ongoing. Evaluation of the pilot An evaluation report Learning assessments in designed to capture lessons selected subjects have and findings from pilot been carried out in 210 implementation as well as schools, including pilot its results finalized. and non-pilot grades, in three regions across the country. Surveys have also been administered to principals, students and teachers. Second survey to be administered in April 2023 and preliminary results to be incorporated in the ICR, if available. June 24, 2023 Page 29 of 29 11