ADMINISTRATION AGREEMENT BETWEEN THE EUROPEAN COMMISSION ON BEHALF OF THE EUROPEAN UNION AND THE INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT CONCERNING PART II EUROPE 2020 PROGRAMMATIC SINGLE-DONOR TRUST FUND Trust Fund No. TF073685 EC Contract No. REFORM/IM2021/022 SLOVAKIA Digital transformation and national curriculum reform of primary and lower secondary schools in Slovakia (P176583) Output 5 Proposed guide for scaling up the results and lessons from the rapid results interventions November 2022 This project is carried out with funding by the European Union via the Technical Support Instrument and with the support and the partnership of the European Commission's Directorate General for Structural Reform Support (DG REFORM) Ministry of Education, Science, Research and Sport of the Slovak Republic DISCLAIMER This document was produced with the financial assistance and the partnership of the European Union. The views expressed herein can in no way be taken to reflect the official opinion of the European Union. This report is a product of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretation and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of the World Bank, the European Commission or the Slovak Government. The World Bank does not guarantee the accuracy of the data included in this work. COPYRIGHT STATEMENT The material in this publication is copyrighted. Copying and/or transmitting portions of this work without permission may be a violation of applicable laws. Rights and Permissions The material in this work is subject to copyright. Because the World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522- 2625; e-mail: pubrights@worldbank.org. This report has been delivered in November 2022, under the EC Contract No. REFORM/IM2021/022 under TF TF073685, signed between the European Commission and the International Bank for Reconstruction and Development on August 16, 2021. It corresponds to Output 5 – Proposed draft guide for scaling up the results and lessons from the rapid results interventions under the above- mentioned agreement. ACKNOWLEDGEMENTS This document has been delivered under the provisions of the Administration Agreement between the European Commission on behalf of the European Union and the International Bank for Reconstruction and Development concerning the Part II Europe 2020 Programmatic Single-Donor Trust Fund signed on August 16, 2021. The document has been prepared under the guidance and supervision of Alberto Leyton (Lead Public Sector Specialist, Governance Global Practice, Europe and Central Asia) by Andrea Sitarova (Senior Public Sector Management Expert, Governance Global Practice, Europe and Central Asia), with inputs from Robert Hawkins (Senior Education and Technology Specialist, Global EduTech team), Cristobal Cobo (Senior Education and Technology Specialist, Global EduTech team) and Daniel Manitsky (Senior Rapid Results Catalyst, Governance Global Practice, Europe and Central Asia). The World Bank team would like to express its gratitude for the excellent cooperation, commitment and drive to the team of the Ministry of Education, Science, Research and Sport of the Slovak Republic, led by Zuzana BaranoviÄ?ová (Director, Institute of Education Policy) to test the Rapid Results Interventions in the context of the Slovak Recovery and Resilience Plan, which created the insights captured in this report. The World Bank team is also grateful to the Ministry team for the comments provided to this report and to Agota Kovacs (Country Coordinator, European Commission, Directorate General for Structural Reform Service) for continued guidance for this work. This document also benefited from the overall guidance and supervision of Fabian Sender (Practice Manager, Governance Global Practice, Europe and Central Asia), Harry Anthony Patrinos (Practice Manager, Education Global Practice, Europe and Central Asia) and Lasse Melgaard (Country Manager, Bulgaria, Czech Republic and Slovakia). ABBREVIATIONS AND ACRONYMS EC European Commission EU European Union ICT Information Communication Technologies MoE Ministry of Education NIVAM National Institute for Education and Youth OECD Organization for Economic Cooperation and Development PISA Program for International Student Assessment RC Regional Center for curriculum management support RRI Rapid Results Interventions RRP Recovery and Resilience Plan SELFIE Self-reflection on Effective Learning by Fostering the use of Innovative Educational technologies TET-SAT Technology-Enhanced Teaching Self-Assessment Tool 4 Contents EXECUTIVE SUMMARY....................................................................................................................7 I. Objective and context of the report ................................................................................... 11 II. Slovakia’s approach to improving 21st century skills of pupils ............................................. 12 2.1. The challenges......................................................................................................................... 12 2.2. Curricular and digital improvements envisaged in the Recovery and Resilience Plan ........... 13 III. Rapid Results Interventions in the context of Slovak education sector ................................ 14 3.1. Definition of RRIs .................................................................................................................... 14 3.2. How and why the RRIs were used in Slovakia’s education and RRP contexts ............................. 16 IV. Transforming the Slovak primary schools to use ICT as catalysts to teaching differently ...... 18 4.1. MoE’s assignment for schools................................................................................................. 18 4.2. How the MoE implemented the RRIs...................................................................................... 19 V. Results and lessons from the RRIs ...................................................................................... 22 5.1. Results achieved by schools .................................................................................................... 22 5.2. Proof of concept of the RRI approach..................................................................................... 25 5.3. Learnings from the RRIs .......................................................................................................... 26 VI. Implications of the results of RRIs for the education system transformation ....................... 30 LIST OF ANNEXES: ANNEX 1: Summary of the evidence of changes in the processes at schools, practices in line with objectives of the curricular reform and impact of pupils ....................................................................... 37 ANNEX 2: Case studies with innovative solutions generated by RRI schools ......................................... 41 LIST OF FIGURES: Figure 1: Slovakia’s RRP component Skills for 21st century structure......................................................... 14 Figure 2: Typical process steps and timeline of the RRIs ............................................................................ 16 Figure 3: RRIs in the context of the RRP ..................................................................................................... 17 Figure 4: The 8-Affordances Framework for exploring pedagogical possibilities of digital educational resources ..................................................................................................................................................... 18 Figure 5: The objectives of the curricular reform ....................................................................................... 19 Figure 6: Framework for analyzing the school data to identify opportunities for improvements ............. 20 Figure 7: Roles and responsibilities in the RRIs run by the ministry ........................................................... 22 Figure 8: Dedicated YouTube channel for Slovak education RRIs .............................................................. 23 Figure 9: Testimonials from the school teams about the added value of the RRIs .................................... 24 Figure 10: Learning from applying the key elements of the RRI approach ................................................ 25 Figure 11: Examples of school data to identify their improvement priorities and actions ........................ 26 Figure 12: MoE support to schools in how to optimize the use of existing digital equipment .................. 28 Figure 13: Changing practices of school management, teachers, pupils in digital transformation ........... 29 5 Figure 14: Examples of insights and intermediary outcomes generated in RRIs........................................ 30 Figure 15: Current placement of the RRIs in the institutional support system for curricular and digital changes ....................................................................................................................................................... 34 Figure 16: Placement of RRIs recommended by the MoE .......................................................................... 34 LIST OF TABLES: Table 1: Results achieved by schools by the end of RRIs against their targets .......................................... 22 Table 2: Possible improvements for the RCs from using the RRIs .............................................................. 35 LIST OF BOXES: Box 1: Difference between an RRI and a pilot ............................................................................................ 15 Box 2: Examples of applications of RRIs across the sectors and governance levels ................................... 16 Box 3: Examples of targets set by the schools ............................................................................................ 21 Box 4: MoE recommendations from RRIs for the school level ................................................................... 31 Box 5: Testimonials of the MoE team in how RRIs inspired new ways of working for the ministry .......... 36 6 EXECUTIVE SUMMARY 1. The objective of this report is to summarize the results and lessons from Rapid Results Interventions (RRIs) conducted in Slovakia to generate insights on how to implement the reform programs from the Recovery and Resilience Plan (RRP) more effectively. The RRIs were implemented with support of the World Bank by the Slovak Ministry of Education (MoE) with the objective to simulate at the level of selected primary schools how to better coordinate, integrate and operationalize for better results different reform streams from the RRP component Skills for the 21st century. 2. The added value of this approach is that while the Recovery and Resilience Facility disbursements are conditioned by meeting a set of RRP investment targets and reform milestones, the RRIs place themselves in the space between the investments are conducted, reforms approved and that of the intended outcomes, i.e., aim to improve the value for money with a focus on results. While the RRIs were illustrated on the RRP education component, the approach is more general and applicable to other sectors or parts of the government’s RRP portfolio. 3. According to the MoE’s assessment, the Slovak education system does not prepare pupils sufficiently for the changing demands of the digital era. The learning outcomes are consistently below the OECD average (PISA, 2018), and only about half of the 16-19-aged students have above basic digital skills (Slovakia education monitor, 2021). The pupil results are significantly influenced by their socio-economic background, and approximately half of the poor families is without access to internet or learning devises (MoE, 2019). 4. The public schools curriculum continues to be mainly focused on memorization of large volumes of facts rather than capacity to apply the learning in real life. It is centralized and prescriptive in terms of what and when the teachers have to teach, without much connection between the subjects and leaving little time to adjust to the pupils’ needs, capacities or to create more space for their individual activities. Prior reforms have failed on the lack of support from the central level in translating the changes to the teaching practice in classrooms as well as on lack of teacher motivation. This presents a potential implementation risk also for the upcoming curricular reform under the RRP. 5. The RRP with its cross-cutting focus on digital transformation brings impetus for the Slovak education system reform, supported by a large investment package into school and digital infrastructure. The MoE plans to improve the teaching processes through the curricular reform and upskilling of the teachers for the new content and form of the curriculum. The intended outcome is to improve the literacy to the OECD average and 21st century skills of the pupils. In the course of past year, the MoE conducted significant preparatory activities both for the investments and well as the content of the reforms, the curricular reform is nearing its finalization and will be piloted from next school year. The MoE’s preparation included also testing of a results-focused change management approach – RRIs – to see how to operationalize the reform programs from the RRP’s component for better results. 7 6. RRIs can be used in the planning phase to generate insights on how to operationalize reforms or subsequently to accelerate their implementation, which were used for the first time in the education setting. In the context of the RRP component, they were to reduce the implementation risks by simulating at school level: A. how to operationalize the RRP programs and put them into school practice B. what other activities were needed to better achieve the intended outcomes or what enabling conditions the schools needed from the central authorities C. how to better integrate the curricular and digital transformation reform packages D. how to better coordinate the reforms with investments, the latter both providing enabling conditions as well as incentives for the schools to effectively put the reforms into practice E. how to measure the intermediary outcomes as lead indicators for the intended outcomes. 7. As such, the RRIs presented a radically different way of working for the MoE. The ministries typically perceive their role to manage and control through the laws and budgets. In this case, the MoE created a strategic assignment, and asked the schools to use data to identify their gaps in being able to start improving their pupils’ 21st century skills. The ministry subsequently mentored the teachers through rapidly testing their own ideas how to achieve the set-out goals in 50 or 100 days. Teachers were in role of innovators. The MoE’s leadership was thus in steering the experimentation process and building local capacities to learn through experimentation. The enabling role of the school principal was key too, in showing school level commitment and creating space and time for teachers to test new ways of teaching. The approach was built on a) urgency, b) data, c) targets, d) permission to experiment, e) sponsorship, f) diverse teams, g) coaching and h) positive competition between schools. 8. The MoE set the experimentation assignment for schools as How to teach differently with ICT as an accelerator of the pedagogical change. The new ways of teaching were to lead to the objectives of the curricular reform, i.e., student-centered approach, building digital skills across subjects, connecting better the subjects and being able to apply the knowledge to solving real life problems. The MoE based this assignment on theory that while there is limited evidence of ICT improving the learning outcomes, ICT does “enable active learning and student engagement, increases student motivation to learn, and augments student satisfactionâ€? (Mochizuki et al. 2019). 9. At the end of the RRIs, six school teams have all achieved or overachieved their targets and as a by-product generated 11 innovative solutions for how to teach differently with ICT. All showed evidence of practices in line with the objectives of the curricular reform. As such, they kickstarted the digital and curricular transformation of their schools. All schools became more goal-oriented and increased the use of data, which can help them in building their school digital strategies. The RRI approach brought the proof of concept that when school teams are given the mandate to test their ideas, set a clear target, prioritize ways to achieve it in a concrete time, they 8 can get activated to start improving the teaching processes that lead to better quality of pupil learning. 10. The MoE drew several lessons from the RRIs. The first one was that in order to activate the changes, giving schools “freedom to experiment within boundariesâ€? seemed more effective than prescribing them blanket solutions to implement. The MoE recognized based on school data that each was at different points of their digital transformation journey and with different school contexts. The ask was to experiment in the set-out assignment areas and use data to identify opportunities for improvements. For the solutions that came out of the RRIs, the MoE supported collegial knowledge sharing among the participating schools around what worked in which school context, what the process was how the school arrived there and promoted positive competition among schools. 11. The second lesson was that schools that are not fully equipped do not need to wait and can also find ways how ICT can help them activate improvements in pupil learning in a short time. Most of the schools, upon starting the RRIs, cited the equipment-related barriers to greater use of ICT in lessons. But in fact, the barriers related to being able to apply knowledge from trainings in lessons or the attitudes towards teaching with ICT were also important. The significant shift that the MoE achieved through the RRIs was for the schools to both start using more effectively the equipment they already had (and was underused) as well as to move the discussion from needed trainings to what kind of improvements the teachers wanted to achieve among the pupils and work backwards from there. 12. The third lesson: ICT as trojan horse to changes in teaching practices. While the interaction with schools started around the digital transformation, the digital has proven to be the entry point to the changes in teaching practices and ultimately the behavioral change in the classroom. As such, many teachers transitioned in RRIs from the perceived burden of learning how to use ICT in lessons to lesson planning becoming easier and lessons becoming more engaging for the pupils who were better able to remember what they experienced. Teachers were able to use the multimodal character of ICT for better pedagogical illustration of the learning matter, adjust the content easier to different pupil needs, automatize some of the tasks (e.g., quizzes for quick checking if pupils have understood) or take the learning outside of the classroom. 13. In the context of the RRP, the RRIs started to generate evidence of what the change in the classroom looks like when all the investment and reforms streams meet there, what does it take, what else is needed and how to measure it. The emerging evidence from six schools unpacks in more detail what changes in teaching and management processes need to happen at school in order to start seeing the evidence of teaching differently with ICT (in line with the objectives of the curricular reform) and how the intermediary outcomes could be measured. All schools showed evidence of improvements in their pupils’ capacities to collaborate, present their work more confidently, create digital content, use ICT to research, systematize and evaluate the information, as well as show greater responsibility and motivation for learning. 9 14. Having completed the pilot testing of RRIs, the MoE issued a policy paper recommending the decentralization of their use to the Regional centers for curricular management that will be to support schools in bringing the curricular and digital changes to practice. In case of current operating model of the RCs, this is likely to i) increase their impact with the goal-oriented approach to working with schools and focusing on school as a unit of change (as opposed to individual teachers, as it is the case now), ii) turn the RCs into more of think tanks for the region through collecting the “what worksâ€? through RRIs, while at the same time to become important feedback partners to the central level on the potential adjustments needed (not just an extended hand for top-down implementation), iii) allow RCs to be more strategic with the use of school data to select the sample of schools to work with in order to improve the average results and improve the equity, and iv) improve the sustainability of the RC interventions by activating the inner potential of schools to come up with solutions, own them and hence be more likely to implement and sustain them (as opposed to waiting to get trainings). 15. The lessons from the RRIs point to additional options the MoE could explore for integration of the component reform streams for better results. These include: a) readiness to teach better with ICT to be included among the criteria to prioritize the schools for ICT investments (ICT as a motivating factor for schools that upgrade their teaching practices e.g. through active participation in RRIs), b) using data to evidence the added value of RRIs (e.g. by collecting SELFIE data from schools at the regional level, benchmarking them and measuring against a control), c) using RRIs in close coordination with curricular pilots to activate the curricular changes better with digital as an entry point to change of practices (i.e. either with RRIs allowing to pre-test and experiment and then for the schools to revise their school curricula, or to use RRIs to activate the revised school curriculum into practice), and d) rethinking RRIs in the context of the implementation support system for curricular and digital changes at schools (including the roles and responsibilities of the different actors involved in the delivery chain of the reform) in order to continue the learning cycles from the RRIs. 10 Proposed draft guide for scaling up the results and lessons from the rapid results interventions I. Objective and context of the report 1. The objective of this report is to summarize the results and lessons from Rapid Results Interventions (RRIs)1 conducted in Slovakia to generate insights how to implement the reform programs from the Recovery and Resilience Plan (RRP) more effectively. The RRIs were implemented with support of the World Bank by the Slovak Ministry of Education (MoE) with the objective to simulate at the level of selected primary schools how to better coordinate, integrate and operationalize for better results different reform streams from the RRP component Skills for the 21st century. The added value of this approach is that while the Recovery and Resilience Facility disbursements are conditioned by meeting a set of RRP investment targets and reform milestones, the RRIs place themselves in the space between the investments conducted, reforms approved and that of the intended outcomes, i.e., aim to improve the value for money with focus on results. While the RRIs were illustrated on the RRP education component, the approach is universal and well applicable to other sectors or parts of the government’s RRP portfolio. 2. This report is developed in the context of the EU-funded and World Bank-implemented TSI “Digital transformation and national curriculum reform of primary and lower secondary schools in Slovakiaâ€? with the Slovak MoE as its main beneficiary. Its development objective is to build the capacities of the MoE staff to use RRIs to inform how to implement the RRP programs more effectively. The desired impact to be achieved by the MoE is also to inspire i) new ways of working and ii) new policy solutions. The project has the following four main components: Component 1: Rapid results interventions and guidance for scaling up their results Component 2: Adaptation of the rapid results methodology and recommendations for its integration with relevant performance management processes Component 3: Analysis of the options for implementing the curriculum reform, its management and quality assessment and recommendations Component 4: Support the design of a methodology for the management of the new curriculum for primary and lower secondary schools This report corresponds to Output 5 under component 1. It describes the results of a year-long effort in testing the RRIs in Slovak education setting. The key interface with the schools was the MoE team in order to maximize the capacity building opportunities in line with the development objective. The World Bank team provided the necessary guidance, international examples, technical inputs related to the RRI approach and reform curricular and digital content as well as peer review of the processes and support with documentation and knowledge management. The report is organized in six chapters. This chapter provides the context. Chapter 2 outlines the 1 Chapter 3 defines the RRI approach and how it works in detail. 11 challenges of the Slovak education system and the MoE’s approach to addressing them through the RRP. Chapter 3 defines what the RRIs are and how they were to add value in the RRP context. Chapter 4 gives the theoretical basis for the strategic assignment that the MoE set for the school experimentation and how the RRIs were implemented with the selected schools. Chapter 5 describes the achieved results in terms of the innovations developed by the schools, proof of concept of the approach and the main lessons. Chapter 6 concludes with recommendations for scaling up the RRIs and specifically discusses how their MoE-proposed decentralization to the Regional centers for curricular management could make the latter more impactful in supporting schools in bringing the curricular and digital changes to practice with focus on results. II. Slovakia’s approach to improving 21st century skills of pupils 2.1. The challenges 3. According to the MoE’s assessment, the Slovak education system does not prepare the pupils sufficiently for the changing demands of the digital economies. The learning outcomes of the Slovak pupils are consistently below the OECD average as well as the average of the Central European peers (OECD, 2018). There is growing demand for the 21st century skills on the labor market, but only about half of the 16-19-aged students have above basic digital skills (Slovakia Education and training monitor, 2021)2 and their capacity to apply them in practice has worsened in 2021 mainly at the schools with worse results (IT Fitness Test3, 2021). As such, these shortcomings are accentuated among the pupils from disadvantaged backgrounds. About 50% of those from the poorest quarter of the population show poor test results (Spending review, Ministry of Finance, 2017) and approximately half of such families are without access to the internet (MoE, 2019). 4. One of the reasons behind these shortcomings is the setup of the state curriculum focused on memorization of the large volumes of facts rather than capacity to apply the learning in the real life. The transmitted information is typically not anchored in the broader life context, does not promote critical thinking, sufficient digital skills, nor balance of the intellectual and physical development of the pupil and overall wellbeing. This is because the curricula are not sufficiently connected between the different subjects and do not follow the joint objectives. Rather the state curriculum dictates in detail what should be taught in each subject and when, which leaves little space for the teachers to adjust to real individual needs or capacities of the pupils and provide support to pupils’ individual activities. 5. The prior curriculum reforms have not translated to visible improvements in the quality of the education system and remained as bureaucratic acts. The state curriculum has been strongly centralized. Its 2008 reform allowed the creation of the school level curricula, but the schools received little support in translating them into practice. This reform did not contain any changes 2 EC (European Commission). 2021. Education and Training Monitor: Slovakia. Luxembourg: Publications Office of the European Union. https://op.europa.eu/webpub/eac/education-and-training-monitor-2021/en/slovakia.html. 3 IT Fitness test is used in Czech Republic, Slovakia, Poland and Hungary to assess pupil digital skills and their capacities to apply them in practice to solve problems. 12 to the objectives and content of the curriculum at the central or school level. In 2015, the MoE attempted another reform aiming to innovate the curriculum, but schools received little time and support to transpose it, so the changes were just formal at the bureaucratic level. This evidence points to implementation risks for the currently developed new curriculum under the RRP4. 6. The pandemic kickstarted rapid digital upskilling of the teachers and pupils, but also uncovered the gaps in digital infrastructure, teaching skills and attitudes to use ICT in teaching processes. From the technology-related ones, Slovak schools seem well equipped with two pupils per computer vs the EU average of 18, yet a share of this technology is outdated5 and only 8% of schools at primary and 17% at lower secondary level have access to high-speed internet.6 This presents the first bottleneck. At the same time, the teachers cite also the pedagogy- and attitude- related obstacles. In the TALIS survey (2018), 40% of Slovak teachers stated there is no suitable offer for in-service training for them. The work under this project also revealed that while an increasing number of teachers were benefiting from digital trainings, they have difficulties applying them in lessons or do not see benefits in using ICT in lessons. Only about 17% of Slovak schools have a clear vision how they want to use ICT compared to an EU average of 30% (EC, 2019)7. 2.2. Curricular and digital improvements envisaged in the Recovery and Resilience Plan 7. There is clear need to improve the teaching processes and the RRP with its cross-cutting focus on digital brings an impetus to Slovakia’s education system reform supported by a large investment package for school and digital infrastructure. The objective of the RRP’s component on Skills for the 21st century (component 7) is to improve the quality of the teaching processes in order to increase the pupils’ literacy above the OECD average8 and improve their 21st century skills (communication, collaboration, critical thinking, digital skills etc.). 8. The MoE intends to achieve improvements in literacy and 21st century skills through a reform of curriculum and upskilling of teachers for delivering it. As Figure 1 shows, the center of the component is the reform of the objectives, content and form of the curriculum. By creating three learning cycles and new learning areas of integrated subjects, the schools will have more flexibility to adjust their curricula to the specific needs of their school. Optimizing the school curriculum by 4 The education authorities plan to finalize the new curriculum in December 2022 and approve at the central level in January 2023. It is part of reform 1 of RRP component 7. 5 Only 67% of pupils are at schools where 90% of this equipment is operational. 6 EC (European Commission). 2019. 2nd Survey of Schools: ICT in Education. Brussels: European Commission. https://ec.europa.eu/digital-single-market/en/news/2nd-survey-schools-ict-education. 7 idem 8 Specifically, the MoE aims to improve the reading, mathematical, scientific and financial literacy of the pupils, reduce the share of 15-year-old pupils at risk levels in reading literacy by 10% (in Slovak and international scores), reduce the impact of the socio-economic background on learning outcomes to the level of OECD average, and increase the motivation to learn sciences, math and reading to the level of OECD average (Slovakia’s RRP, 2020). 13 removing overlaps of content across subjects will give teachers more time to focus on pupil needs and their own activity. As such, the updated curriculum should ensure better connections between the subjects, applying knowledge in the real-life context and using digital skills across the subjects. Teachers play a key role in delivering the updated curriculum and will need to be upskilled both in pedagogical use of the ICT, as well as innovative student-centered teaching methods. Figure 1: Slovakia’s RRP component Skills for 21st century structure Source: Adapted from the Slovak RRP (2020). 9. The MoE started implementing this RRP component in 2020 and has done significant preparatory efforts, including testing ways to strengthen the implementation capacities with a focus on results. The ministry completed a gap analysis for the digital equipment needs of the schools (Q2/2022) with the objective to equip fully 80% of schools with ICT by the end of 2023. It launched a network of 16 Regional centers for curriculum management (RCs) in Q3/2022, which are filling the gap between the central level and schools and their role will be to support schools in transposing the central level curricular and digital changes to their school practice. The digital and inclusive teaching competencies were updated in the teaching standards (Q1/2022) and teachers can tap into purpose-bound financial support for lifelong learning (Q3/2022). The state- level curricular changes are nearing completion and will be piloted from the 2023/2024 academic year. The MoE also completed a year-long testing of the approach called RRIs to see how to better coordinate and more effectively operationalize the multiple reform strands with focus on results. III. Rapid Results Interventions in the context of Slovak education sector 3.1. Definition of RRIs 10. RRIs are an innovative results-focused change management approach. It can be used in the context of complex reforms either in the planning stage to generate ideas for how to operationalize the strategic objectives or in the implementation stage to accelerate the 14 implementation of reform plans. The RRIs typically draw upon user-centered design9 and user-led innovation10, adaptive and network leadership,11 evidence-based practice (plan-do-study-act cycle)12, change management theories and facilitation, coaching and behavioral change techniques, behavioral insights and process optimizations. As such, the approach is aimed to help better bridge the gap between the policy and the user experience, in view of achieving better results on the ground. Box 1 shows the differences between an RRI and a pilot. While both aim to decrease the risks of rolling out large scale reforms by testing them on a small scale locally first, the RRIs typically rally a frontline team of service providers and users around an ambitious and time-bound target and give them the mandate to rapidly test their own ideas how to achieve them. This aims to tap into their creativity and ownership of the change, rather than to impose it on them. Box 1: Difference between an RRI and a pilot RRI Pilot • sets in advance an ambitious target for • typically evaluates the pilot ex post improvement to achieve • works under an ambitious time constraint • has reasonable time to implement the pilot • is composed of the frontline service providers • is composed of a designated team that comes across different functions, typically including to run the pilot in the local setting also service beneficiaries • gets the mandate to test their own ideas for • implements a top-down assignment and how to achieve the set-out targets design of the pilot 11. The RRIs typically follow a set of process steps. The preparatory stage is dedicated to gaining clarity over the high-level assignment for the RRIs, selection of and engagement with the frontline service providers, their team creation, data collection and agreement over the working arrangements for the RRI. The actual RRIs are then punctuated by three key meetings, to launch 9 User-centered design (UCD) is a creative approach to problem-solving which places the user as its center. Thus, a UCD researcher first tries to build empathy for the users that (s)he is designing for, works with them to build understanding of the issues they face, and generate together with them ideas for possible solutions, test them through rapid prototyping with end users and eventually roll out the innovative solutions. IDEO. (2015). Design Kit: Human-centered design toolkits. IDEAO. https://www.ideo.com/post/design-kit. 10 User-led innovation is innovation led by intermediate users and end users, rather than producers. Bogers , M. Afuah, A., and Bastian, B. (2010). "Users as innovators: A review, critique, and future research directions", Journal of Management, 36 (4): 857–875. http://www.marcelbogers.com/Pubs/Bogers-Afuah-Bastian_2010_JOM_Users- as-innovators.pdf. 11 Adaptive leadership is a leadership approach which focuses mobilizing a group of individuals to handle tough challenges and emerge triumphant in the end. Network leadership emphasizes the collective, bottom-up, distributed approach to leadership. Heifetz, Ronald A., Alexander Grashow, and Marty Linsky. (2009). The Practice of Adaptive Leadership: Tools and Tactics for Changing Your Organization and the World . Boston: Harvard Business Review Press. 12 Plan-do-study-act practices, where the goal is set out (plan), the plan is implemented and data gathered (do), the data is subsequently analyzed and learnings examined (study), and decisions are taken on how to adjust or scale up the developed solutions. Langley GL et al. (2009). The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd edition). San Francisco: Jossey-Bass Publishers. 15 them, take stock of progress at the midpoint (which allows to recalibrate or accelerate) and the final meeting to take stock of the results and agree how to sustain them (Figure 2). Figure 2: Typical process steps and timeline of the RRIs 12. The three key roles that need to be distinguished in the RRIs are those of a sponsor, mentor and an RRI team. The sponsor is the one that sees the need to run the RRI to fix a performance problem or generate specific insights for how to operationalize the reform programs. He or she sets the high-level assignment. The mentor sets the framework process for the RRI and guides the RRI team through it, monitors progress, provides or mobilizes support as needed and helps the RRI team distill the learnings from the process (what has worked and what has not). The RRI team sets its own target within the high-level assignment and a plan to achieve it. Throughout the RRI, this team then rapidly tests their ideas for how to achieve the targets, measures the progress and at the end takes stock of the results and makes a plan for how to sustain them in their own organization. 3.2. How and why the RRIs were used in Slovakia’s education and RRP contexts 13. The RRIs have been used around the world in different sectors, strategic planning cycle stages or levels of governance. In Slovakia they were used for the first time in the education sector. Box 2 shows the application of RRIs across selected countries and sectors. Perhaps the use of RRIs in the Slovak education sector to achieve rapid improvements may seem counterintuitive. The evidence has shown that the purpose of an ambitious performance target in this context was mainly to accelerate the throughput of innovations in how to operationalize the reform programs. Box 2: Examples of applications of RRIs across the sectors and governance levels United Kingdom National Health Service: in strategy operationalization stage to see how to achieve demand reduction on the healthcare system through co-creating with hospital, local community and patient teams ways how to reduce avoidable admissions. Emirati Government Accelerators: in strategy implementation phase, to see how to increase the breast cancer preventative screening or how to reduce the number of accidents on selected busiest roads. Selected cities across the USA: at the local government level, how to reduce homelessness. 16 14. In the context of Slovakia’s RRP, the RRIs were to reduce the implementation risks. The motivation came from the project’s main counterpart, the MoE’s Analytical Unit with the legacy in promoting results-based practices.13 Specifically, these implementation risks were the “white space riskâ€? and “integration riskâ€?. The first one means that when the managers write the RRP component which is to involve multiple organizations and stakeholders working over four years, it may be hard to predict all activities that will be needed to achieve the set-out results. The second risk means that even if all activities were to be anticipated, it may still be hard to pull them together at the end to an intended result (Matta, 2003). As such, the RRIs were creating microprojects in the space between the reform programs and the intended outcomes, to simulate at the level of the service providers ( 15. Figure 3): A. how to operationalize the RRP programs and put them into school practice B. what other activities were needed to better achieve the intended outcomes or what enabling conditions the schools needed from the central authorities C. how to better integrate the curricular and digital transformation reform packages D. how to better coordinate the reforms with investments, with investments both providing enabling conditions as well as incentives for the schools to effectively put the reforms into practice E. how to measure the intermediary outcomes as lead indicators for the intended outcomes. Figure 3: RRIs in the context of the RRP 16. As such, RRIs presented a radically different way of working for the ministry. The ministries typically perceive their role to manage and control through the laws and budgets. In this case, the MoE was to test how to i) manage complexity of the RRP component by RRI microprojects, ii) capitalize on building the relationships with frontline and other actors in the system involved in delivering the component, and iii) experiment and learn from it. The usual top-down approach 13 The Unit’s team comes from the Prime Minister’s Implementation Unit where its role was to track and accelerate the implementation of the recommendations from the Finance Ministry’s Spending Reviews. 17 was hence replaced by the MoE creating conditions for the schools to innovate by giving them space to test their ideas. IV. Transforming the Slovak primary schools to use ICT as catalysts to teach differently 4.1. MoE’s assignment for schools 17. The MoE set the experimentation assignment for schools as How to teach differently with ICT as an accelerator of pedagogical change. With the expected outcomes of the RRP in mind (improved literacy and 21st century skills), the MoE based its assignment on the theory which showed that while there is limited evidence of ICT improving the learning outcomes, ICT does “enable active learning and student engagement, increases student motivation to learn, and augments student satisfactionâ€? (Mochizuki et al. 2019; Vorvoreanu 2014). Prior evidence from Slovakia14 and abroad15 showed the need to connect the use of ICT with a purpose in order to have this activating effect. The World Bank supported the MoE in the use of the 8-Affordance model (Figure 4) to animate the initial discussion with schools to explore how digital tools and resources can foster pedagogies that create more effective ways of teaching and learning. While in the first round of the RRIs the experimentation assignment was fully open to schools in the context of How to teach differently with ICT as accelerator of pedagogical change, in the second round the MoE went on to define what “teaching differentlyâ€? means in line with the stated objectives of the curricular reform which was under preparation (Figure 5). Figure 4: The 8-Affordances Framework for exploring pedagogical possibilities of digital educational resources Source: Mochizuki et al. 2019. 14 The ICT procured for schools often stayed in storage or e.g., the interactive boards were used just to display content. (from interviews with school upon the launch of the RRIs). 15 For example, the One Laptop Per Child programs points to the limited success when ICT is introduced in classrooms without a vision of purpose of how it will be integrated into the teaching and learning goals (Smart and Jagannathan 2018; World Bank 2017; RTI International 2013). 18 Figure 5: The objectives of the curricular reform 1) Student centered approach From providing information through lectures to building the capacities of the pupils to find solutions themselves. 2) Building digital competencies across subjects Not only in computer science lessons but building digital skills of pupils across subjects. 3) Teaching with better connections between Connecting isolated subjects through joint the subjects assignments. 4) Applied learning From memorizing information to being able to apply it (e.g., for local problems). Source: Adapted from MoE conceptual model for the curricular reform, 2022. 4.2. How the MoE implemented the RRIs 18. The MoE’s Analytical unit took the lead in the RRIs as a sponsor and mentor. The Director of the Unit set the assignment and the Unit’s technical team provided mentoring support. This was a new task for this Unit otherwise focused on providing analytical and forecasting inputs to education strategy and policy development. At the same time, one of its dedicated teams covers also the monitoring and reporting on the RRP, so the task under this project to help other MoE’s technical units with insights on how to better implement the RRP programs fit naturally.16 19. The MoE selected a sample of six primary schools17 to experiment in RRIs in two rounds of 50- and 100-days. The MoE’s sample creation strategy was to start with motivated schools that at the same time have a clear vision of the digital transformation they want to achieve. The final sample was balanced it terms of school i) size, ii) type, iii) level of ICT equipment, iv) geographical placement, v) share of pupils from disadvantaged backgrounds, and vi) presence of digital coordinator at school. 20. The MoE’s engagement with schools started through the orientation for the principals and school nominations of their teams. The role of the principal was to outline his or her vision for the school’s digital transformation, but at the same time create space for and support the teacher team in bringing their own ideas.18 16 At the same time, the unit plans on creating a model based on this testing and pass it on to other MoE’s department and its regional structure. 17 Upon the announcement of the call to participate in the RRIs, 90 primary schools applied. The MoE has been documenting the RRIs on the social media, so also the schools that were not selected were able to benefit from the generated insights. 18 For this, it helped when the school team assigned a team leader different than the principal. Contrarily, when the principal became a team leader, he became “a spokespersonâ€? for the school and the school team was more hesitant about testing their own ideas. 19 21. While the MoE did not prescribe for the school teams what improvements to focus on, it asked them to use the data to arrive at a consensus on prioritized areas for improvement. The MoE guided the schools to collect data with SELFIE19, TET-SAT20 and created an additional questionnaire for more detailed qualitative information on the hardware, software and digital resources available and used at schools, teaching practices with the use of ICT and individual teacher visions for what kind of improvements they would like to achieve in the classrooms. The collected information was subsequently organized into a framework21 which tracked what inputs the schools had, how the teachers were able to use them, how they were able to teach with them with a specific purpose and how this was translating to pupil capacities to use ICT for specific purpose (Figure 6). Figure 6: Framework for analyzing the school data to identify opportunities for improvements 19 EU’s SELFIE (Self-reflection on Effective Learning by Fostering the Use of Innovative Educational Technologies) is a tool that schools can use to understand the level of digitization of their school and the direction they want to take about including ICT in the teaching and learning processes. The data is collected from the school management, teachers and pupils across a range of areas related to how well the school is equipped with digital technologies and internet connection, professional development practices, capacities to teach with ICT with different objectives. The questions for the pupils are related to how they use ICT for learning. 20 A tool for self-reflection of teachers on their capacities to use ICT in teaching. The teachers can find themselves in 5 different proficiency levels and subsequently plan and monitoring their progress over time. The tested areas include a) pedagogical use of ICT, b) use and creation of digital content, c) digital communication and collaboration, and d) digital citizenship. 21 This framework is flexible and the RRI mentor team is improving it further. 20 22. During the launch, the MoE team led the schools to set ambitious improvement targets to be achieved under a time constraint and a plan to achieve them. The collected data provided the basis for the discussion on the possible improvement areas as well the strengths that the school can build upon. The MoE team led the schools to gradually narrow down to the specifics of what improvement they would want to work on in the identified area, analyzed why they had not done it yet (i.e. what barriers they will need to address in the plan), further to the specifics of which subjects or grades would be covered in the RRI and finally the school team set a target which was i) focused on result rather than an activity, ii) measurable, iii) ambitious yet realistic within the given time constraint and iv) developed and owned by the team (not by the mentor, ministry or the World Bank). Box 3: Examples of targets set by the schools School 1: How to improve in 100 days the motivation of pupils from disadvantaged backgrounds to learn by ensuring that at least 40% of them master digital technologies at the required level to process their term papers with help of digital technologies (baseline: 0) School 2: How to ensure in 100 days that at least 200 pupils learn actively in lessons with the digital resources from the teacher-created school platform (baseline: 0) School 3: How to achieve that teachers use digital technologies in 40% of class hours in 50 days (baseline: 22%) 23. Throughout the RRI, the MoE held regular meetings with the school teams mentoring them through the implementation progress, not only to progress quicker with implementation but also to reflect upon the learning what worked in implementation. This interaction of the MoE with the schools was unique, as it was acting like a learning organization, making just-in-time adjustments through work with schools (as opposed to long implementation cycles). Figure 7 outlines how the roles and responsibilities were split between the MoE, schools and the World Bank support team in the different process steps of the RRIs. 21 Figure 7: Roles and responsibilities in the RRIs run by the ministry V. Results and lessons from the RRIs 5.1. Results achieved by schools 24. From the perspective of the school assignments, school teams have all achieved or overachieved their targets. As a by-product they generated 11 innovative solutions for how to teach differently with ICT. Table 1 lists the targets that the schools achieved. Error! Reference source not found. includes the evidence of how the schools were adjusting their teaching processes and ANNEX 2 a selection of case studies for the solutions developed by schools on 1) how to improve the motivation of students from disadvantaged backgrounds to learn with the help of ICT, 2) how to use ICT for thematic teaching (connecting two or more subjects) and improve the pupil motivation to learn with it, 3) how to use a platform of digital resources to activate teacher and pupil collaboration (peer learning). The rest of the solutions are documented on a dedicated youtube channel and the process of how the schools arrived there additionally at the dedicated Facebook page. Table 1: Results achieved by schools by the end of RRIs against their targets Wave School Indicator Baseline Target Actual (end of RRI) Primary Share of class hours with the use of ICT 30% 50% 50% School, in learning Humenne (source: school questionnaire) 22 Wave 1 (50- Primary Share of teachers with e-career 0% 50% 50% days) School portfolios on the school website22 Rabcice (source: school edupage) Share of pupils achieving higher than 0% 50% 50% average digital literacy (source: NIQES scores before/after) Primary Share of class hours with the use of 22.7% 40% 42.42% School digital technologies in learning Uhrovec (source: school questionnaire) Share of teachers with at least basic 30% 80% 80% digital skills (from total at school) (source: school questionnaire) Wave 2 Primary Share of pupils that have been taught at 0 554 554 (100 School Sered least one lesson covering more than two (100% (100% days) (J. A. subjects and have shared the learning from from Komenského) with other group of pupils with the help total) total) of ICT (source: teacher records) Primary Share of pupils that master digital 0 40 46 School technologies at the required level to (58 % (67 % Uzovske produce their term paper with it from from Peklany (source: teacher records) total) total) Primary Share of pupils that actively learn in 0 200 259 School Zilina lessons with the digital resources from (37% (48% (Gastanova) the teacher-created school platform from from (source: teacher records) total) total) Figure 8: Dedicated YouTube channel for Slovak education RRIs 22 The purpose of this improvement was to raise awareness of the digital teaching practices of various teachers and promote knowledge sharing and collaboration in lesson design, delivery and inter-subject connections. 23 25. All participating schools became more goal-oriented and increased the use of data, which will be useful for developing or updating their school digital strategies. Significant part of the RRIs was spent on supporting the school teams in creating a logical results framework (including a problem, goal, activities to achieve the goal and measurement indicators). Following the RRIs, the schools showed increased capacities to develop their plans to sustain the improvements or plans for their own subsequent RRIs. In these, the schools skillfully thought not only of different trainings, but also a range of additional activities needed to bring about the change, including the leadership support, incentives, hands-on support, attention to equity in how also the beginner ICT users can be brought in or activation of certain school dynamics. Schools started to use more frequently and flexibly various data collections and diagnostic tools in their work. In their testimonials, schools emphasized collaboration, acceleration and delivery (see Error! Not a valid bookmark self-reference.) and how the RRIs kickstarted the transformation of how they teach and use ICT. Some schools claimed increased interest of the parents and the public following their activities in the RRIs. Figure 9: Testimonials from the school teams about the added value of the RRIs 26. In the context of the MoE’s assignment, all schools showed evidence of practices in line with the objectives of the curricular reform and improvements on the 21st century skills of their pupils even in the short time. ANNEX 1 shows specific examples of how the changes in the teaching processes were leading to a shift from the teacher-centered to a pupil centered approach, greater use of applied learning mainly through project-based learning assignments, which typically were involving two or more subjects, and in the process of this, building pupil digital skills (also outside of the computer science lessons). All schools showed evidence of improvements in their pupils’ capacities to collaborate, present their work more confidently, create digital content, use ICT to research, systematize and evaluate the information, as well as show greater responsibility and motivation for learning. 24 5.2. Proof of concept of the RRI approach 27. The method brought the proof of concept that when school teams are given the mandate to test their ideas, set a clear target, prioritize ways to achieve it in a concrete time, measure the results and learn from it, they can get activated to start improving the teaching processes that improve the quality of pupil learning and their motivation to do so. Figure 10 takes stock of what worked in successfully operationalizing the key elements of the RRI approach, i.e., the use of a) urgency, b) data, c) targets, d) permission, e) sponsorship, f) diverse teams, g) coaching and h) positive competition. Figure 10: Learning from applying the key elements of the RRI approach 25 28. Both the MoE as well as the principals played a crucial role in enabling the schools to innovate though the RRIs. The MoE was explicit at all midpoint and sustainability review meetings with the schools that they are giving schools space to bring their own ideas. “We are in the listening mode today. We are listening to your experiences. We appreciate and recognize t he efforts you are putting in place to reach your targets.â€? – MoE staff Thus, the MoE’s leadership was in process steering, providing encouragement and supporting capacity building at the school level to design and deliver improvements themselves. The second player to enable the school innovations was the school principal. First by showing the commitment of the school to change, a direction, but also by stepping away and creating conditions for the teaching staff to try their own ideas for doing things differently. Teachers were in role of innovators. The principals started to discover the talents of their staff and were giving them a role in driving the digital transformation of their schools, or were motivating them through bonuses linked with sustaining the innovative practices. 5.3. Learnings from the RRIs 29. Rather than to prescribe schools blanket solutions to implement, the MoE found it more effective to set a strategic direction, experimentation process and tell schools to use data to identify the improvements to work on with focus on end beneficiary (pupil) in order to help MoE achieve its strategic objectives. The MoE observed through data and interactions with schools in RRIs that each was at different points in their digital transformation journey, with different school context and different gaps that needed to be addressed in order to start achieving improvements in 21st century skills of their pupils. The MoE gave schools the freedom what to work on and how to achieve it, but the improvements needed to be in line with the MoE’s strategic assignment and be based on data. The MoE needed to provide capacity building support to schools to be able to use data and think in a results-oriented way. “Reforms need to be clear and general (some basic rules) but giving schools freedom in order to define their own way to reach the goal.â€? – MoE staff Figure 11 shows the differing situations of the selected schools and how data helped them prioritize their improvement plans. Figure 11: Examples of school data to identify their improvement priorities and actions School A was well equipped with computers (each child had a tablet), but the school was in the beginning of its digital transformation journey. Most of the pupils were from disadvantaged backgrounds and were often lacking motivation to learn or were dropping out early. The principal’s leadership and a well working collegial body were the school’s strengths. Yet, they were not clear what changes they wanted to achieve, what it would take, and also both teachers and pupils needed to upskill significantly in being able to use different applications, but also to use them in teaching and learning. 26 So for the RRI, they prioritized experimentation with how to use ICT to improve pupil motivation to learn and they would evidence it on being able to do their term paper (pupil passion project) better with the help of ICT. While there were many areas that the school could be improving, this gave them clarity over what to prioritize and what all building blocks need to be rapidly put in place to achieve this improvement among the pupils. School B was less well equipped, the internet was slower, but already had some experience in innovative teaching and using ICT for different purposes and this mirrored also in pupil digital skills. However, teacher collaboration and sharing what and how they teach has suffered during the pandemic. This was the only red area on their chart. For the RRI, they decided to prioritize it to explore how creating an internal shared platform of digital resources could help them also improve collaboration among the teachers. They not only created the digital solution with the learning resources, but also activated the internal dynamics of peer support in using ICT, teachers reusing other teachers’ digital materials in lessons, upgrading them or providing feedback. They took this new dynamic also among the pupils and created with these digital resources a project-based learning assignment across several subjects for the whole school (i.e., for the pupils of the whole school to write a book) and activated the peer learning between grades. This school was already testing how the ICT can help them accelerate implementation of the envisaged curricular reform changes. 30. In this context, the MoE seemed to have valued both the solutions that were generated in the RRIs as well as building the capacities to innovate and learn. The learning was promoted at three levels. First was the experimentation with the actual RRI approach. Having learnt from the first wave, the MoE made adjustments in the design of the second wave (e.g., specifying further the assignment, emphasizing the need to finalize data collection prior to setting the targets or testing sustainability elements in RRI design). The second learning loop was taking place at the level of the schools. The MoE steered regular monitoring sessions, which allowed the schools to reflect on what they were learning. Through the milestone meetings, the MoE created space for collegial exchange of good practices among the schools, both in terms of what worked in specific contexts as well as what the process was to arrive there. The third learning loop was from the schools to MoE, which gathered insights how some of its RRP programs could be operationalized for results. 31. One of the most important insights generated in the RRIs was that also schools that are not fully equipped do not need to wait and can find ways how ICT can help them achieve improvements for pupil learning in a short time. This shows to the MoE that they can incentivize schools to 27 reform no matter their pre-existing status. In the data collected within the project and in the initial discussions with schools, most considered the state of digital technologies and internet connection at their school as the key barrier to using ICT more in lessons. One of the first questions the schools posed was whether they are going to receive ICT through the RRIs. Most of the initial improvement plans of the schools were focused on digital trainings for the teachers on various ICT applications. The significant shift that the MoE achieved through the RRIs was both for the schools to start using more effectively what they had and also to shift the discussion on what kind of improvements the teachers want to achieve among the pupils and with what purpose. More effective use of school technology related to a) in some cases starting to use the equipment that the school had either in storage, not connected or starting to think of new ways how to use it (e.g., when digital devises were used solely for screening information), b) if the computers for pupils were located mainly in specialized classrooms, adjusting the scheduling to optimize their use to be able to increase the share of digital content in lessons (see Figure 12), or c) finding creative solutions when the equipment was not sufficient for learning, i.e., through using pupil cell phones in targeted way in lessons for assessment or relying on home computers. Even in a short time, the schools were able to shift thinking to a coherent framework connecting the different parts, i.e., the leadership, skills, motivations, and purpose around the expected improvements among the pupils (Figure 13). Seeing the results in a short time was powerful for the schools. As one teacher stated: “we realized we can achieve the teaching improvements and do not need to wait for anyone to tell us how.â€?23 Figure 12: MoE support to schools in how to optimize the use of existing digital equipment Note: the picture shows the MoE staff working with the school team to adjust the school schedule to optimize the use of the computer labs (Source: MoE, 2021). 23 Teacher feedback during the RRI Sustainability review meeting, 2021. 28 Figure 13: Changing practices of school management, teachers, pupils in digital transformation Source: Adapted from World Bank Education Dashboard. 32. While the interaction with schools started around the digital transformation, the digital has proven to be the entry point to the changes in teaching practices and ultimately the behavioral change in the classroom with focus on pupils. While originally learning to include more ICT in lessons may have been perceived as a burden by the teachers, starting from having to do lesson plans and intentionally thinking with what purpose the ICT would be used in the different steps, at the end of the RRIs most teachers claimed the ICT made their lesson planning easier, lessons became more engaging for the pupils and they were able to better remember what they experienced or the knowledge they produced themselves. Teacher were able to use the multimodal character of ICT for better pedagogical illustration of the learning matter, adjust the content easier to different pupil needs, automatize some of the tasks (e.g., quizzes for quick checking if pupils have understood) or take the learning outside of the classroom. This is an important insight that the MoE can use in the phasing the reform implementation or linking incentives for schools to put the curricular changes into practice with the digital. 33. In the context of the RRP, the RRIs have generated specific insights how to operationalize the reform programs for results, how to better integrate the interventions inside the component and what else is needed to start turning them into the intended results. Figure 14 shows the emerging evidence from six schools that unpack in more detail what changes in teaching and management processes need to happen at school in order to start seeing the evidence of teaching differently with ICT (in line with the objectives of the curricular reform) and how the intermediary outcomes could be measured. Already in the short-term, the schools started observing evidence of improved 21st century skills of their pupils (mainly collaboration, presentation skills, capacity to research and critically review information), increased responsibility for their learning, better digital skills and confidence in use of ICT for different learning purposes, improved lesson engagement, motivation to learn and classroom climate. 29 Figure 14: Examples of insights and intermediary outcomes generated in RRIs VI. Implications of the results of RRIs for the education system transformation 34. Through the RRIs, the MoE piloted the process for the schools to experiment. The data and generated innovative solutions started flowing from selected schools to the MoE. The following chapter proposes recommendations for what to do with these insights next and how to continue the learning cycles to improve the effectiveness of achieving the intended RRP outcomes. These are based on i) the MoE policy paper on the RRI experience and recommendations for the policy and school level, ii) additional measures for better integration the RRP component emerging from the lessons from the RRIs, iii) analysis of how the MoE- proposed approach of decentralizing RRIs to the RCs could increase their impact with focus on results and finally, iv) how the emerging elements of the learning system in the education sector could be sustained and developed further, subject to further research and experimentation. 6.1. MoE recommendations for the scale up of insights from the RRIs 35. The MoE issued a policy paper on the pilot experience with the RRIs and recommends to use the approach further at the decentralized level. It makes recommendations both for policy as well as school practice. Specifically, the MoE recommends NIVAM (National Institute for Education and Youth), its directly managed agency, to become the trainers for the RRI approach and for the RCs24 to run the RRIs with schools. The additional insights generated from RRIs link 24 There are currently 16 RC operational and MoE intends to establish a full network of 40 RCs by Q3/2024, under reform 1 of RRP’s component 7. 30 directly with the RRP milestones, such as to i) promote sharing of school digital resources, sample lesson plans and digital transformation strategies at the dedicated platforms25, ii) find sustainable arrangements for financing digital coordinators at schools,26 iii) use the RRI method in order to accelerate and measure the implementation of the curricular reform in the area of digital literacy, iv) support the supply of trainings for the development of digital capacities of the teachers. Box 4: MoE recommendations from RRIs for the school level • For the schools to use the available tools, e.g., SELFIE or TET-SAT to identify their opportunities for improvements in digital transformation. • For the principals to actively support the use of RRI approach at schools to drive the improvements on their digital strategy. • Promote sharing examples of lesson plans, digital resources and other good practices among teachers, e.g., through digital platforms (accessible inside the school or also by other schools). • To develop the strategy of digital transformation of the school as part of the school curriculum and share it with other schools through dedicated platforms. • Support the lifelong learning of the teachers through “updating programsâ€? 27 focused in improving the digital competencies of the teachers (e.g., in peer mentoring). • Tie the remuneration criteria for the teachers (bonuses) with their achieved and practiced digital competencies (e.g., linked with sustaining the share of class hours with the use of ICT). 6.2. Additional options for better integration of the RRP component based on RRI lessons 36. The lessons from RRIs point to additional options for better integration of the component reform streams for better results. 1. Readiness to teach better The MoE started the RRIs with a sample of schools that with ICT as a criterion to were motivated to make improvements in how they prioritize the schools for ICT teach with ICT. Yet, the situation may not be the same investments across schools. Some school managers or teachers may need greater incentives for improving their teaching practices, taking the extra effort of including ICT in their lessons and learning to teach with them. Some genuinely want to try, but the often outdated or insufficient number of ICT devices for teaching at schools is a deterrent. The RRP ICT investments for schools are currently guided by the school needs, based on the gap analysis. Yet, the RRI evidence has shown that having ICT on its own does not translate to improved teaching practices. 25 This platform is to be established by Q3/2022 under reform 1 of component 7 of the RRP. 26 The MoE finances the digital coordinator program at 170 primary schools for a duration of one year from the EU Structural Funds. 27 According to §57 of Law 138/2019. 31 Therefore, the MoE could consider including “active graduationâ€? from RRIs among the criteria for prioritizing schools in receiving the ICT equipment. 2. Using data to evidence the The MoE currently collects the data on schools through added value of RRIs the management information system and ad hoc surveys. The SELFIE data is used primarily for school purposes and for their reflection on improvement opportunities and progress. The RCs could consider testing at the regional level to collect SELFIE data from schools, benchmark schools and try diversified approach based on the school digital maturity. This could help also with the sample creation and prioritizing schools for RRIs (e.g., the schools with lowest scores to enter the digital accelerators by default). As the RRIs have shown that to observe changes on SELFIE scores, it is better to allow more time than that of the quick RRI, the RCs could consider annual collection of the SELFIE data, measuring the effects of RRIs at participating schools against a control group within their benchmark. 3. Using RRIs in close Prior curricular reforms have not translated to visible coordination with curricular changes in the school practice. The evidence from the pilots to improve the school that revised its school curriculum (as part of the implementation of the curricular pilot of NIVAM) and then participated in RRIs curricular changes in showed that the teachers were not able to bring to practice with digital as an teaching practice the changes they made in the entry point to change of curriculum. The RRI helped them activate the curricular practices changes, including with the necessary teacher collaboration and peer support dynamics with help of ICT. NIVAM and MoE through its RCs could consider to use RRIs prior to supporting the school in revising its school curriculum to first create space to deviate from the curriculum and experiment. Then, based on that, revisions to the curriculum could be made. Under this scenario, teachers would better own the changes as the ideas how to operationalize them would be coming from them. Alternatively, once the school revises its curriculum, the RRI could help to bring the changes on the paper into practice. For this, the curricular pilots which will be launched from Q3/2023 (under reform 1 of RRP component 7) should be closely coordinated with the RRIs, including the measurement framework to evaluate the effects of the curricular changes in the classroom practice. The MoE should be adjusting the RRI strategic 32 assignments based on the testing needs to bring the curricular and digital changes into practice. 4. Rethinking RRIs in the The MoE intends to help train the NIVAM trainers in RRIs context of the and support the RCs in being able to run them with the implementation support schools. With this decentralized approach, the ministry system for curricular and intends to increase the volume of schools that will go digital changes at schools through the RRIs. This also presents an opportunity to rethink the delivery system to promote learning in it. While the actual delivery of the RRIs will come to the regional or local level, this should not become a one-way process. Rather the central authorities can rethink their role in the RRIs, so that the learning from the frontline can flow back in cycles to the central level for possible policy adjustments, new strategic assignments in line with the implementation schedule of the RRP component. This can involve also reviewing or clarifying the roles of the different actors involved in the delivery of the curricular and digital changes (i.e., MoE and NIVAM at the central level, NIVAM county office and RCs at the regional and local level, and then the school, digital ecosystem and teacher learning communities), based on their strengths, weaknesses and capacities. RRIs could then be rethought in the content of the broader system, where best the data collection and analysis could be placed, technical and methodological inputs to the reform content by who in what capacity, who to select the assignments, feed learning from RRIs back to the central level etc. 6.3. RRIs for greater impact of RCs in supporting school in curricular and digital changes 37. RRIs could make the RCs more impactful in their support to schools with focus on results. RCs are newly established structures of the MoE’s Department for Curriculum and Innovations in Teaching. They present a middle layer that has been missing in previous efforts to reform the curriculum between the central authorities and schools in order to provide them with hands-on support in translating the curricular reforms to practice. Presently, the RC operating model includes mentors who work with individual teachers to help them with a) trainings, b) mentoring and c) networking. As such, they present the extended hand of the central authorities to help them deliver the changes. The advantage of the RC staff is that typically they have good knowledge of the school context in their region as they mostly come from among the school principals or teachers and are building a network of training service providers and various professional networks in their region. In parallel, NIVAM has been running a limited number of 33 curricular pilots (testing mainly adjustment of the school curriculum to cycles) and the MoE’s Analytical Unit has been running the RRIs ( 38. Figure 15). Figure 15: Current placement of the RRIs in the institutional support system for curricular and digital changes 39. Therefore, the MoE’s decision to place the RRIs at the RC level can make them both more impactful as well as better connect the otherwise siloed system through learning cycles. Figure 16 shows the recommended placement of the RRIs by the MoE for going forward and 40. Table 2 summarizes the four main benefits it can bring for the RC current operating model. Figure 16: Placement of RRIs recommended by the MoE 34 Table 2: Possible improvements for the RCs from using the RRIs 1. Greater impact • By working with a school as a unit of change, rather than individual teachers (i.e., capitalizing on benefits of peer support, collaboration in teaching and principal support) • Goal-oriented interaction with schools in terms of expected improvements for pupils (rather than focus on conducting trainings) 2. RC as a think-tank and better • Documenting and learning through RRIs builds the feedback partner for the knowledge base of the RCs that can be shared with central level other schools in similar contexts as inspirational model examples and networking contacts for specifics of operationalizing similar improvements etc. The case studies, templates and insights generated in MoE-run RRIs can be transferred to the RCs as the foundation of their knowledge base and teachers from schools that graduated from the RRIs can provide further support and inspiration to new cohorts of RRI schools. • With this knowledge base, the RC gets better evidence and strengthens its position as feedback partner to the central in view of possible policy adjustments that can enable the schools to implement the reforms better and quicker. 35 3. Strategic choices and learning • Collecting the data from schools at the regional level based on data would allow to i) benchmark them, ii) adjust the approach by level of digital maturity of schools and iii) make strategic choices over the creation of the RRI school sample in view of achieving the average improvements on high level intended outcomes more effectively, and with focus on reducing the performance gap in correlation with the socio- economic background of the pupils. • Possibility to better evidence the added value of the RRIs by measuring the improvements at RRIs schools against a control. 4. More sustainable impact • As the RRIs rely in activating the inner potential of the frontline team to find solutions themselves, school teams better own the generated changes, are more likely to implement and sustain them. With this, the limited resources of the RCs can be more effectively allocated for better results. 6.4. The emerging MoE-run learning system as an opportunity for further research and experimentation 41. With the RRIs, the MoE tested new ways of working through building better relationships with the frontline service providers, experimenting and jointly learning from it. While there is emerging evidence of the MoE as a learning organization, sustaining these practices further creates space for further research and experimentation. In the feedback survey, the MoE saw the primary benefit of RRIs in improving the capacity to learn through experimentation and secondarily in generating solutions to be scaled up. The use of analytical framework (such as the 8-Affordances framework) helped the MoE team break down the approach to RRP component implementation into process steps and test through RRI microprojects. The team was very open to learn from the frontline practice and saw the biggest value of the diversity of views and experiences of the central and frontline teams in generating the solutions and learning from them. The whole experience helped improve trust between the ministry and participating schools. For the ministry’s practice, the main learning from the RRIs echoed by the MoE team was that it was more effective to set the direction, basic conditions and rules, but then as it is the schools that ultimately need to implement the changes, to motivate them through giving them space to pick their own challenges and let them define their own way how to reach their goals. Box 5: Testimonials of the MoE team in how RRIs inspired new ways of working for the ministry 36 “The approach of leaving it up to schools to select their own challenges they want to tackle seems very motivating. It is the teachers in cooperation with school management who choose what really bothers them and then they come up with solutions to improve the situation.â€? “Reforms, laws and standards will never be able to set the rules for the schools by themselves since every school is in a different situation with its own needs. The experimentation has to be about collaboration between the stakeholders, while the achieved results have to be implemented by the school itself. On the other hand, it is not the best idea to give full freedom for the schools in the decision making, so a set of basic rules is also important.â€? “Reforms need to be clear and general (some basic rules) but giving schools freedom in order to define their own way to reach the goal.â€? “The Ministry [learnt] that the best way to implement a reform is to give freedom through experimentation with general rules and topics on it.â€? “Most importantly, the other schools in our country, which will be willing to experiment [can take the approach forward]. The basic methodology should be kept but the areas of experimentation should be different in each cycle. On the other hand, since the area of implementation of accelerators is so vast, maybe other segments of the public sector can start experimenting too.â€? 37 ANNEX 1: Summary of the evidence of changes in the processes at school, practices in line with objectives of the curricular reform and impact of pupils EVIDENCE OF SCHOOL PROCESS IMPROVEMENTS EVIDENCE OF PRACTICES IN LINE WITH EVIDENCE OF IMPACT ON CURRICULAR REFORM OBJECTIVES WITH PUPIL SKILLS AND HELP OF ICT COMPETENCIES 1. Management support: 1. Student-centered approach • Improved collaboration (need • Improved strategic planning skills (in case of school • Most schools increased the use of to come up in a group how digital strategy), basing the decision on data diagnostic facilitated learning in detriment of they will solve the task, of teaching practices with ICT, better goal-orientation teacher lecturing, e.g., pupils came up respect for each other’s and capacity to prioritize actions which lead to through brainstorming with research opinions, collaboration achieving the goals (this includes not only provision of tasks to work on, they created (or were activated by peer learning), trainings, but strategic thinking on what all is needed supported in creating) groups, planned leading to better classroom to make the change happen) their work, used ICT for carrying out the climate. • Better support to giving space to teachers to innovate, tasks) discovery of teacher talents and their use to drive • Most schools activated peer learning, • Improved creativity (while digital transformation of school with pupils presenting or explaining their having a voice how they • Motivation to teachers through linking bonuses with work to their peers and getting feedback approach the task, pupils used sustaining teaching with ICT practices. • Teachers placed responsibility on pupils in lessons ICT to create digital for taking care of the digital equipment content e.g. cartoons, 2. Teacher trainings and peer support: they received animations, quizzes, poems • Trainings on how to use specific applications used for • Teachers adapted the teaching and and shared with peers). specific pedagogical purposes learning content to the pupils’ differing • Sharing of learnings from the trainings with colleagues needs and interests (e.g., with the • Improved digital skills and • Peer support in applying the learnings from trainings multimodal content of ICT). greater confidence in use of in lessons (special support for elderly or beginner ICT ICT for delivering the tasks (to users), e.g., though classroom observation, tandem 2. Building digital skills across subjects search information online and teaching, lesson feedback, sharing of examples • Pupils used ICT to deliver a range of systematize, evaluate and • Regular online and offline meetings between teachers research and learning product critically review, create digital sharing good practices, motivating each other development tasks with help of ICT content, e.g., videos, • Increased interest to learn from other school good • Pupils used ICT for better communication presentations; and share practices (e.g., knowledge exchange with other and collaboration with peers and with responsibly with others). schools though RRI milestone events, feedback on teachers Facebook documentation of the process, knowledge • Pupils created their term papers in a • Better communication skills exchange with the teacher awarded as the best subject of choice with help of ICT, (e.g., in delivering teacher in Slovakia etc.). recorded in shared space, presented to presentations and results of their peers and received feedback. their learning, increased 38 3. Teacher collaboration in planning and delivering lessons communication between across subjects: 3. Teaching with better connections across pupils and pupils and • Use of shared documents and application to quicker subjects teachers. plan project-based learning assignments or lessons • Several schools designed learning exploring the same topic across subjects, as the basis experiences across subjects (e.g., • Improved responsibility for for teacher discussions about and collaboration on through regular project Fridays or one- learning (pupils became more lesson delivery. off larger thematic days that included up of coordinators of their work, to all classes of the school). These were were more actively involved 4. Creation of platforms for sharing resources and within thematic areas or including in learning e.g., by researching collaboration: different thematic areas of subjects and information and working with • Creation of shared space where teachers post the in line with what was planned in the it further, not only receiving lesson plan models with the use of ICT, other teachers curriculum. knowledge but also creating re-use and customize these lesson plans further, it, taking responsibility to provide peer feedback or rating of the digital 4. Applied learning review the quality of their resources • The thematic days or project Friday to work, planning the • Creation of e-carrier portfolios of teachers on a shared the extent possible involved applying the development of the learning space, which allow the teachers to get to know each knowledge to solving real problems at outputs, e.g. a presentation other better, contact each other based on common the schools or within the community and taking effort to finalize it interests, possible collaborations, queries for support (e.g., the how to save energy at our in quality, taking responsibility with the tools they master etc. school day, where the pupils encoded for the received digital what they learnt to a QR code and took equipment, e.g. tablets). 5. Use of ICT for communication: to their homes to co-create new energy • Greater use of internal platforms (e.g., on MS Teams efficiency practices with their families). • Improved research skill and or EduPage) for communication among teachers, The other examples included active skills to evaluate information: between teachers and pupils, and with parents knowledge making applied in creating a (e.g., pupils formulated simple • Management using ICT tools to collect information book by selected grades of the school. hypotheses, recorded the from teachers, pupils or parents. The teachers adjusted the complexity of information or observations, the assignment and related use of ICT for evaluated the information and 6. Optimized use of ICT equipment: the different grades, at the same time reached conclusions, • Mapping of the digital equipment and e.g., if creating opportunities for pupils of defended their arguments). computers for pupils are located in specialized different grades to collaborate. Pupils classrooms, optimization of their use through explored the process of book production, adjustments in scheduling, allowing to increase the which different professions are involved, share of digital content used in lessons. they planned their work in teams to 7. Teachers use data to diagnose teaching practices and develop their books in different subject prioritize improvements: areas, used various applications to collaborate and produce the book, from writing, proofreading, illustration, to 39 • Teachers increase the use of data in lessons (e.g., to printing and creating a library of books to diagnose the practices of pupils, collect lesson share them with others, read each feedback etc.) other’s works etc.). • Teachers use various self-reflection tools such as SELFIE, TET-SAT, ucitel21 etc. to identify their individual or collective areas for improvement to prioritize the actions to address them. 8. Teachers adjust their teaching practices with help of ICT, they: • participate actively in the development of digital learning resources • improve the quality of pedagogical illustration with the multimodal use of ICT • move from lecturing to facilitating the research and active knowledge creation of pupils with the help of the ICT, turn learning into experiences • use ICT to automatize some of the pupil feedback collection or assessment, to rapidly and continuously check understanding of progress • collaborate with other teachers to design learning experiences exploring the same topics across subjects • adjust the learning content to the individual needs of the pupils • use ICT to promote collaboration among the pupils (e.g., jam board, word walls, zoom rooms) • use ICT to increase active engagement of pupils in lessons and positive competition • create practices to increase the responsibility of pupils for their learning and for the digital equipment. At the end of these activities, • teachers claimed they found lesson planning and delivery with help ICT resources and tools easier • they increased the pedagogical use of digital for specific purposes in lessons 40 • two schools made adjustments to their school curricula to reflect how they used ICT in lessons • schools noted increased interested of parents and public in the teaching practices at their schools. 41 ANNEX 2: Case studies with innovative solutions generated by RRI schools These case studies describe the solutions for How to teach differently with ICT as an accelerator of pedagogical change in the 100-day RRIs which took place 2021/2022 school year. Each case study describes the a) starting point of the school, b) the problem definition, c) solution, d) implementation steps, e) results, f) measurement and g) proposal for how to sustain the new practices. The list of case studies: 1. How to increase the motivation of students from disadvantaged backgrounds to learn using digital technologies 2. How can digital technologies help with thematic teaching to increase the motivation of pupils to learn 3. How a shared repository of digital resources can increase collaboration at school 42 CASE STUDY 1 How to increase the motivation of students from disadvantaged backgrounds to learn using digital technologies Video link: https://www.youtube.com/watch?v=CzzbC6AQeuA Participating school: Elementary school with kindergarten, Uzovské Pekľany, Eastern Slovakia Number of pupils: 69 Number of teaching staff: 11 Average age of teaching staff: 37 years Note: elementary school only has 1st grade (1st-4th year) Starting point at school Almost 90% of the children that attend school in Uzovské Pekľany are from socially disadvantaged backgrounds or have disabilities. The teachers know some of the pupils have lower motivation to learn or finish school, also through often times low literacy of the parents or their low involvement in following their children’s studies. When pupils enter the first grade, teachers focus mainly on the adoption of hygiene standards, and gradually try to improve the children's motivation to learn. The school operates in new premises, it is technically very well equipped, every student has a tablet, the classrooms have interactive whiteboards. Despite this, neither pupils nor teachers were able to sufficiently use digital technologies in learning or teaching. However, the school had great support from the management, and had a cooperative and creative team. Problem definition Despite the sufficient ICT equipment in classrooms, pupils and teachers had low digital skills and were not able to use ICT in learning or teaching. 43 Solution Teachers agreed on a goal to increase the motivation of pupils to learn by preparing their term paper with the help of digital technologies. The subject areas were assigned by the teacher, the specific topic of the work was determined by the students themselves. Digital technologies provided students with opportunities to independently study a selected topic, process it creatively and share its final work with classmates via the MS Teams platform. A prerequisite for completing the assignment was rapid training and mentoring in the use of selected applications and the MS Teams platform of both teachers and students by the school’s digital coordinator. Implementation process 1. The teachers jointly define the thematic areas of the pupils' term paper. 2. In cooperation with the digital coordinator, teachers choose suitable applications for the pupils to use for preparation of term papers. 3. The digital coordinator creates a training plan for teachers so that they know how to use the applications themselves and at the same time explain their use to the pupils. He gradually trains colleagues on selected applications, mentors and motivates them. 4. The teachers help the pupils to choose a specific topic from the given thematic areas and prepare the course outline. He or she guides the pupil to research the topic using digital technologies. 5. The teachers help the pupils to create an account in MS Teams, where they gradually save materials related to the term paper. The pupils work on the term paper in MS Teams both during classes as well as from home. 6. The teachers make sure that the pupils uses digital technologies as much as possible when completing the tasks and diagnoses where and what the pupils need help with. 7. The pupils upload the final version of the course work through OneNote in the MS Teams environment. 8. At the end, each pupil presents his or her work to other classmates. The results • Teachers strengthened their ability to lead pupils and develop their creativity, empathy, encourage cooperation between older and younger pupils, e.g. when presenting their term papers. They managed to start motivating the students to take responsibility for the learning devises entrusted to them, but especially for their own learning, thereby better preparing them for further education. • The teachers' approach through independent class work increased the pupils’ self- confidence (especially when presenting the achieved results), which can significantly contribute to the development of their internal motivation to learn. The use of digital technologies (mainly tablets) and available applications (e.g. when creating comics, animations, poems, puzzles) increased their creativity and ability to cooperate with classmates. 44 • Having seen the pupils’ term papers, also the parents got motivated to continue learning. Parents committed to take courses at the school to strengthen their digital skills. How can the use of this innovation be measured? • The share of pupils, or the number of students from the total number at the school who can produce the term paper using digital technologies and upload it to the MS Teams platform. • The share of pupils, or the number of students from the total number at the school who use school technology for home learning.28 A proposal for long-term sustainability • The introduction of the so-called user licenses for using tablets. The conditions for using and taking devices home are determined by the school itself. This license can be obtained if the student demonstrates sufficient skills regarding the care of the tablet and can subsequently take the tablet home, or work with it before class and after class. • Use of term papers with help of ICT as part of the teaching process in each school year. 28 This is helpful to extend the learning hours or ensure continuity when the child is ill 45 CASE STUDY 2 How can they digital technologies to help with thematic teaching and increase the motivation of pupils to learn Video link: https://www.youtube.com/watch?v=OlT9osFnEvY Participating school - data in the 2021/2022 school year Elementary school of Jan Amos Komenský, Komenského 8, SereÄ?, Galanta district Number of pupils: 547 Number of teaching staff: 36 Average age of teaching staff: 46 years Note: fully organized elementary school (1st-9th grade) Starting point at school The school was relatively well equipped with digital technologies, teachers improved their digital skills through training even during the pandemic. They tried to apply the acquired knowledge in classes. The school participated in pre-testing of the new curricular pilots. They revised the school's educational program for the 2020/2021 school year, removed duplications of curriculum in individual subjects, which allowed the teachers to gain more time for practicing the learning matter and pupil creative work. Problem definition Despite having revised their school curriculum, the school was not able to bring into practice teaching with inter-subject connections. Also, the actual use of digital technologies by teachers was low. Solution 46 Teachers agreed on a goal to promote thematic teaching to allow them to better connect the contents of individual subjects. They used digital technologies and applications to better develop cooperation between teachers and also students and motivate them to learn. Implementation process 1. Teachers choose a communication platform (e.g. MS Teams ) to be used for recording, planning and sharing of created digital resources. 2. Teachers agree on the topic they are to collaborate on in thematic teaching (based on what is coming up in their curricula). 29. 3. The assigned teacher creates a table that contains the list of teachers, the goals that the teachers want to achieve jointly and from the perspective of each subject and the list of applications and aids that are needed to achieve the goal 30. Source: ZÅ  SereÄ? 4. A prerequisite for successful implementation is that teachers are able to work with digital technologies. Therefore, it is important that they receive training on those applications that they want to use in class (e.g., Vimboard , PlantNet ). 5. The trainer can make a recording of each training, which will be uploaded to the designated location. Gradually, a database of materials for teachers can be created from which they can draw. The role of the designated teacher is to coordinate individual activities and trainings and to help colleagues in the use of digital technologies. 29As part of the thematic teaching, the school used the teaching of younger pupils by older pupils on the occasion of Earth Day, when the older pupils encrypted information into QR codes, which the younger pupils had to decipher, sort and transfer the information into a readable form in science classes. 30To connect the content of the subjects, the school decided to first make test rooms in several classes. In the first degree, you e.g. the teacher chose the topic Our City and devoted herself to it in several subjects. . In the fifth grade, the school chose to combine biology and mathematics. The school tried to implement thematic teaching in the past, at least once a month. 47 6. After the implementation of the thematic teaching, reflection and feedback is appropriate, which may result in updating the table for the thematic teaching plan, including observations based on this practical experience and recommendations for further use of the assignment plan. The results • Increased digital skills of teachers and students (resulting from greater use of digital technologies in lessons). • Increased attractiveness of teaching by using innovative methods and digital technologies in classes. • Pupils' learning in context, developing their creativity and critical thinking, increased motivation to learn. • Increased attractiveness of the teaching process. • Increased the number of teachers working with multiple applications. This was achieved by sharing their knowledge/experience after completing the training. • Improved cooperation among teachers and among pupils. How can the use of this innovation be measured? • The share of teaching hours or the number of lessons learned within thematic teaching with an emphasis on connecting the content of several subjects (from the total of hours). • The share of pupils or the number of pupils who actively participated in thematic learning with an emphasis on connecting the content of several subjects (from the total number of pupils). A proposal for long-term sustainability • Implement thematic teaching as part of the teaching process in each year throughout the school year. 48 CASE STUDY 3 How a shared repository of digital resources can increase collaboration at school Video link: https://www.youtube.com/watch?v=mMBGjooGY3A Participating school - data in the 2021/2022 school year Elementary school with kindergarten, GaÅ¡tanová 56, Žilina Number of pupils: 537 Number of teaching staff: 39 Average age of teaching staff: 54 years Note: fully organized primary school (1st - 9th grade), the project focused on first grade (1st - 4th grade) Starting point at school The school is relatively well equipped with digital technologies, some equipment is older and not all classes had the necessary technical equipment (e.g., interactive whiteboards). Before the start of the RRI, the teachers completed various training courses. However, they did not share much information from them, their cooperation was limited especially during the pandemic. Most teachers searched for digital materials independently. Teachers mainly used materials from zborovna.sk but found that many of the learning resources were outdated and finding the useful ones took time. If teachers were sharing the learning resources, it was mainly in the printed format. The school management supported professional development of teachers in innovative teaching approaches, experimentation at school and was interested to involve pupils in development of learning resources, so that they are useful for them. 49 Problem definition Insufficient communication between teachers and lower digital skills of part of the teachers prevented them from creating and sharing information and learning resources in digital format. The educational content was therefore not very attractive. Solution Teachers agreed to create a shared internal repository for digital resources (e.g. on uschovna.sk ) which would allow them to reuse each other's lesson plans or materials, customize or upgrade and provide each other feedback. It means that the solution was technological but accompanied by teacher dynamics which allowed to innovate the learning materials and approaches further. Greater use of digital content was to make the lessons more attractive for pupils and improve their motivation to learn. Implementation process 1. The teachers individually diagnose their digital skills31 and identify which applications they could train their peers in and which ones they would need help with. 2. Together, they agree on the structure, content and functionality of the shared internal repository for digital resources. 3. The digital coordinator (or other designated person) creates a shared table to record the required and completed teacher trainings. Source: ZÅ  GaÅ¡tanová 56, Žilina 4. The digital coordinator (or another designated person) creates the repository on https://uschovna.zoznam.sk. He or she creates for each grade folders for individual subjects. 5. Teachers gradually upload materials in digital format (text documents, presentations, tests, links to used applications) to the folders. 6. Teachers record in the table how many pupils actively learnt with the given digital resources from the shared storage. During the active use of the material in class, the teachers record feedback from the pupils, which they later record in the table, too32. 7. Once a quarter, teachers evaluate the quality of shared materials in order to improve the quality of shared content for their colleagues. This improves the cooperation between teachers in teaching who help each other improve the quality of teaching materials this way. 31Based on already used tools such as SELFIE or TET-SAT, or through self-reflection in individual areas of using digital technologies. 32For example by interaction: students who liked the lesson raise their hands, and those who did not like it do not raise their hands. Based on the number of likes and dislikes, the teacher gives the feedback in the table. 50 The results Teachers: • More efficient lesson planning and saved time with the preparation of materials for the lesson. • Simplified access to digital materials that are up-to-date, adapted to school conditions and can be readily used. • Continuous development of digital skills by working in a digital environment (e.g. when creating teaching materials, using the functionality of the internal storage, but also when teaching and using digital technology more in lessons). • Improved cooperation among teachers by creating an internal repository and using internal help and experience of other colleagues with the use of the given digital content. Pupils: • More interesting lessons, supporting students' creativity, imagination and motivation to learn. • A more fluid course of the lesson in classrooms where digital technologies are used and content is drawn from the internal online storage. • Increased students' independence and degree of cooperation during lessons. How can its successful implementation be measured? • The number of digital resources in the shared repository • Share of teaching hours using materials from the from the shared repository (out of the total number of teaching hours). • Share of pupils from the total number who worked with materials from the shared repository during lessons. • Satisfaction rating from the reuse of the listed digital resources (feedback from colleagues who used the material from the shared repository). A proposal for long-term sustainability • Complex setting of the structure of the internal storage - labeling of years, years and precise labeling of folders and files. • Regular checking and cleaning of unnecessary folders and files that are out of date, unnecessary, etc. • Annual measurement of digital competences and soft skills of teachers through e.g. SELFIE , TET-SAT or Ucitel21 . 51 52