Lessons on Implementation Stakeholders’ Experiences with Digital Personalized Learning Platforms Sophia D'Angelo, Juan D. Barón, Haani Mazari, Daniel Morales, Santiago Ospina Tabares, Paola Polanco, Tom Kaye  CONTENTS In this report Executive Summary 4 1. Introduction 7 2. Literature review: Digital personalized learning 13 3. Methodology 16 4. Findings 19 5. Discussion and Recommendations 30 TABLES Table 1. Summary of differences in implementation 10 across Prográmate cohorts Table 2. Sample demographics 17 Table 3. Concise version of coding framework 18 ACRONYMS ALEKS Assessment and Learning in Knowledge Spaces DPL Digital Personalized Learning EdTech Educational Technology EGRA Early Grade Reading Assessment ICT Information and Communication Technology LMIC Low- and Middle-Income Country MINERD Ministry of Education in the Dominican Republic UNESCO United Nations Educational, Scientific, and Cultural Organization UNICEF United Nations Children’s Fund Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms   2 Acknowledgements The successful implementation of the Prográmate project in the Dominican Republic would not have been possible without the invaluable contributions of several key individuals. We extend our deepest gratitude to Additionally, we acknowledge the Miguel Ángel Ortiz, MINERD Technician, unwavering support from our colleagues who played a pivotal role in supervising at the World Bank, including José Mola, and implementing the project in Carmen Maura Tavares, Omar Jiménez, schools, contributing significantly to the Janina Cuevas, and Astrid Pineda, achievement of its objectives. Manuel who contributed significantly to the Paredes, also a MINERD Technician, project’s implementation, monitoring, demonstrated exceptional leadership in and evaluation, as well as Diego Angel- overseeing the project’s implementation. Urdinola and Cristóbal Cobo, who, We extend our gratitude to Ancell as peer reviewers, provided valuable Scheker, Vice Minister of Technical and comments for this report. Lastly, we Pedagogical Affairs, for placing trust in express our gratitude to Alexandria the World Bank and providing unwavering Valerio, World Bank representative, for support throughout the project. Yissell her continuous support of the project, Crisóstomo, of the Technical Pedagogical and to Molly Jamieson Eberhardt, Vice-Ministry, played a crucial role in EdTech Hub’s Director of Engagement, internal coordination within the MINERD. for her support of our research Special thanks to Susana Michel, partnership. Director of Secondary Education, for facilitating project implementation and overseeing the entire management team of the participating schools. Their collective efforts have truly made a lasting impact on education in the country. ATTRIBUTION D'Angelo, Sophia, Juan D. Barón, Haani Mazari, Daniel Morales, Santiago Ospina Tabares, Paola Polanco, and Tom Kaye. 2024. Lessons on Implementation: Stakeholders' Experiences with Digital Personalized Learning Platforms. World Bank and EdTech Hub. Washington, DC: World Bank. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms   3 Executive Summary This study explored the perceptions and experiences of students, parents, teachers, and other education stakeholders involved in Prográmate, a project that uses a digital personalized learning (DPL) platform to develop secondary school students’ math skills and knowledge in the Dominican Republic. DPL refers to the use of technology to tailor → Engagement varied across schools, students, educational experiences to individual student and teachers. Much of this had to do with needs, preferences, and learning styles. In how each cohort was designed and the initial countries facing challenges to teaching quality, DPL presentation of the project to teachers and can help address disparities in access to quality other education personnel. education, improve student engagement, and → In some schools, structured time was enhance learning outcomes through the provision allocated for students to use the platform of customized learning experiences at scale. during the regular class schedule or after- Administrative data from the DPL software school hours. Some teachers also provided suggests that students who used the DPL students with homework assignments to use technology more frequently, benefitted more the DPL at home. than others, increasing their performance in the → Some students were more motivated to use DPL exercises, and outperforming their peers in the platform than their peers. Some students national standardized exams. However, the time were extrinsically motivated, encouraged students spent using the platform was different by teachers, parents, friendly competition across and within schools. This qualitative with their peers, or other incentives. Other study builds on the quantitative data to try and students opted to use the platform on their understand the implementation process and the own, especially to study for a test or exam. enabling and constraining factors that shaped student engagement with the platform. It draws → Engaging with the DPL platform at home or on interviews and focus group discussions with school depended upon students’ access to 231 students, parents, teachers, school leaders, devices and reliable internet , which varied and other education stakeholders. The key greatly. Some schools had access to devices findings are summarized below in relation to the that were donated as part of the government’s three research questions. Digital Republic project. However, in many cases, these devices were outdated or needed In what ways do key stakeholders, maintenance. At home, parents acted as such as students, parents, teachers, gatekeepers to devices, determining where and other education personnel, and when children could use them. engage with the DPL platform? Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Executive Summary  4 How do stakeholders describe → Many students, however, preferred to learn directly from their teacher, because they felt the benefits and drawbacks of the teachers could more easily adapt the feedback DPL platform? and support that they provide compared → Students and teachers generally found the to the platform. They valued teachers’ oral platform engaging and fun to use. explanations, rather than having to read feedback provided by the platform. → Teachers learned new mathematical strategies to use with their students, and → Both students and teachers also felt valued the data analytics, which allowed them frustrated by some of the technological to monitor student progress and mastery of challenges when trying to access a device certain topics. However, some teachers would or the Internet to use the platform. They did have liked the different topics in the platform not always have the digital skills or resources to be more aligned with their daily lesson available to do this. plans. They often found the time required to support students as burdensome in addition What are the enabling and to their normal workload. Teachers generally constraining factors that shape the agreed they needed more pedagogical and implementation of the project in technical support. schools and at home? → Likewise, students enjoyed being able to self- → Access to reliable internet, electricity, monitor their learning and found excitement and devices (especially large-screen devices, in watching their progress and/or competing as these worked better when accessing the with their peers. Seeing their advancements platform). Access tended to be even lower in motivated them; seeing their regression schools or homes based in rural and remote discouraged them. communities. → Students valued the platform’s repetition → Teachers’ and students’ digital skills, and and explanation of incorrect answers. They ability to navigate the platform. appreciated how the app provided access to a calculator and an option for them to look up → Teachers’ ability to monitor student the meaning of an unfamiliar word. Students engagement and progress by accessing the also enjoyed the use of real-life examples and data analytics and using that data to motivate content that was relevant to their interests and encourage students. and lives. → Students’ own interests, whether intrinsically → Some students mentioned how they or extrinsically motivated. learned new skills, including critical thinking → School leaders and pedagogical coordinators’ and problem-solving, digital literacy, or involvement, especially efforts to create reading skills. structured time and opportunities for students to use the DPL platform (either during regular school hours, during their computer or math class, or after school). → Parents’ involvement and encouragement , which especially influenced how and when students accessed the DPL platform outside of school. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Executive Summary  5 Based on these findings, the report offers six key recommendations for designing and implementing effective DPL programs in public schools of the Dominican Republic and other lower- and/or middle- income countries: 1. Co-design school-based DPL 4. Ensure an enabling technological programs with teachers and environment. A key challenge was access other education personnel. This to devices and reliable Internet connectivity in school and at home. Governments should will help ensure that project goals, content, turn to innovative solutions to address this, and structure are relevant and aligned to including partnering with private companies teachers’ workloads and school conditions. or tech specialists that can support resource It will also help enhance uptake and the allocation, distribution, maintenance, or use. integration of DPL programs as a whole-school approach, including through the engagement 5. Diversify learning content, of parents, families and other community not just by level, but also members. by modality. While most students 2. Ensure adequate timing and valued the personalized learning platform, sufficient duration of project others struggled to read the feedback and explanations it provided. DPL platforms implementation. Consideration for should consider students’ low literacy levels academic calendars and extracurricular and provide support in not just written text activities is critical, and ensuring teachers and but also through audio and images that can students have sufficient time to familiarize enhance understanding. themselves with the DPL platform will help maximize impact on learning. 6. Fund, commission, and conduct 3. Provide ongoing support to more DPL implementation teachers, so they can better research, including in formal support students. Teachers are critical school settings. There exists a large actors that provide technological, pedagogical, evidence base on how to design effective and motivational support to students. They DPL products, but there is limited research must be equipped not only with an initial investigating the best ways to use those training or workshop but also with ongoing products, especially in public schools in low support, so that they are better able to and/or middle-income countries. This study facilitate implementation and student learning. contributes to this evidence gap; more studies like this are needed. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Executive Summary  6 1. Introduction The world is facing a learning crisis. Across the globe, students are progressing through education systems without developing key skills, knowledge, and competencies. In low- and middle-income countries in particular, even before the COVID-19 pandemic, more than half of students were in learning poverty, defined by the number of children who cannot read a simple text designed for their age group (World Bank 2022a). Marginalized students, including girls, enhance students’ learning in mathematics socioeconomically disadvantaged students, and using a DPL platform. This study focuses on the students from rural areas, typically face larger experiences of students, parents, teachers, and learning loss than other students. This learning other education personnel who participated in loss, especially among this vulnerable population, the project to better understand how DPL can is cumulative and can lead to school dropouts be effectively implemented to improve learning (Azevedo et al., 2021; UNESCO 2021; World Bank outcomes. The study addresses three research 2022a). While educational technology (EdTech) questions: (a) In what ways do key stakeholders, can open up pathways to support learning such as students, parents, teachers, and other through multimodal approaches, it also has the education personnel, engage with the DPL potential to help improve learning quality in platform? (b) How do these stakeholders describe LMICs and for vulnerable populations of learners. the benefits and drawbacks of the DPL platform? A key example of an EdTech approach is the and (c) What are the enabling and constraining use of digital personalized learning (DPL), which factors that shape the implementation of the shows great promise in supporting teachers to project in schools and at home? improve learning outcomes for learners who The findings of this study fill a gap in robust are falling behind in their education (Major and quantitative and qualitative research in Francis 2020). implementation of DPL interventions (Major and This study examines the Prográmate project, Francis 2020). It also paves the way forward which was designed and implemented by the for future DPL interventions for learners in Ministry of Education in the Dominican Republic classrooms and at home in the Dominican (MINERD) in partnership with the World Bank. Republic and other LMICs. The Prográmate project was developed to Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Introduction 7 many parents and caregivers to perceive schools 1.1. as more safe and inclusive (MINERD 2021, 101). In Education in the 2016, the government launched a competency- based curriculum, which includes standards for Dominican Republic students to develop a range of skills including The Dominican Republic has one of the highest digital literacy, and critical, creative, and logical economic growth rates in the Caribbean thinking skills (MINERD 2016). In addition, the region. This growth has catalyzed investments government increased teacher salaries by 96 in education, where the percentage of gross percent from 2012 to 2019, attracting more domestic product (GDP) allocated to the qualified candidates to the profession. education sector doubled 10 years ago. These However, there remain challenges to achieving additional allocations have allowed the country to quality and equity in education. Even before realize various achievements. They have enabled the COVID-19 pandemic, 80 percent of 10-year- improvements in the access to and quality of old children in the Dominican Republic were in education by expanding school construction, learning poverty (World Bank 2019). According to hiring more teachers, and universalizing the results of international standardized tests educational policies, such as the extended school such as the 2018 Programme in International days and provision of school meals. Before the Student Assessment (PISA), only 9 percent COVID-19 pandemic, the Dominican Republic also of 15-year-old students (in the third year of attempted to bridge the digital gap in education high school) reached minimum proficiency in through an initiative known as República Digital, mathematics (OECD 2019). The other 91 percent which provided students and teachers with remain in the lowest performance level. Similar laptops and tablets and equipped schools with results are found in students in lower grades. internet connectivity. According to UNESCO’s ERCE1 study, 80 percent Between 2012 and 2020, the Dominican of third graders and 77 percent of sixth graders government built 23,528 new educational spaces, in the Dominican Republic only reach the lowest including 15,696 new classrooms across more performance level in mathematics, meaning they than 1,600 new schools, 2,944 refurbished fail to reach minimum proficiency (Tejada 2021). classrooms, 380 science labs, 564 computer These low levels of learning have been labs, and nearly 1,000 libraries (MINERD 2021, exacerbated by the COVID-19 pandemic, which 104). The most tangible result of the increased resulted in school closures. School closures investment in education has been the increase negatively impacted student learning and in student access and participation due to the widened learning gaps in the country (Abreu Malla availability of classrooms, especially at the et al., 2022). While students from private schools primary level, where 95 percent of learners are with a higher socioeconomic status were able currently enrolled (MINERD 2021). According to to maintain some continuity to education, the data from the 2019–2020 school year, 1.3 million lack of access to the internet and technology for students across nearly 5,000 public schools (71 students from low socioeconomic levels caused percent of enrolled students) were benefitting hundreds of thousands of students in the public from the extended school day (MINERD 2021). sector to be excluded from education (Diario These students also received other benefits, such Libre 2021). Studies indicate that just 56 percent as the government’s school feeding program, free of all households in the Dominican Republic have uniforms, and backpacks. These improvements access to fixed internet connectivity, and the to school and classroom environments have led high cost of internet creates barriers to access, 1 ERCE (Estudio Regional Comparativo y Explicativo) also known as the Comparative and Explanatory Regional Study is an international comparative assessment conducted by UNESCO in Latin America and the Caribbean region. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Introduction 8 resulting in only 53 percent of households with learning experience, progress and mastery of the internet access and 26 percent of households competencies, and flexible learning environments with digital devices (Lara Ibarra, Comini, and (Groff 2017). Although personalized learning Gelvanovska-Garcia 2022). does not depend on technology, the latter helps the implementation among large populations During the pandemic, MINERD trained teachers of students with the help of algorithms and and created an online platform for remote or large datasets. Research suggests that this distance education that served as a repository of type of EdTech can increase learners’ access to online teaching and assessment resources. This materials and learning activities, improve learning contributed to the use and adoption of diverse outcomes, and decrease educational gaps for digital technologies for teaching and learning. marginalized students (Major and Francis 2020). MINERD launched remote education programs through TV lessons and printed brochures with From an institutional point of view, the content and activities for self-learning, and possibilities of developing educational policies instructions for teachers to monitor student associated with DPL have due support in the learning. However, there was no administrative Dominican Republic. In an attempt to discover system in place that kept track of which students innovative mechanisms that can help improve used the available resources or how often these learning, MINERD, in collaboration with the resources were accessed. World Bank, designed and implemented the Prográmate project to enhance students’ Still, the pandemic brought with it new learning in mathematics using a DPL platform. educational innovations for the Dominican While DPL has the potential to offer significant Republic. All aspects related to digital promise to improve student learning outcomes, educational resources are promoted by MINERD there is limited research that explores the in accordance with the secondary-level academic implementation of DPL technology in formal curriculum and pedagogical standards. Prior to classroom or school settings. This study seeks the pandemic, MINERD had created the Dirección to fill that gap by documenting the experiences General de Medios Educativos (General Office and perceptions of students, parents, teachers, of Educational Media) to support educational school leaders and other education personnel innovation in the country, but it did not have who participated in Programáte across three the opportunity to implement any major years of implementation (2019, 2020, and 2022). changes until the COVID-19 pandemic resulted In doing so, it highlights promising practices in school closures. Although some schools in and challenges of DPL implementation, and the Dominican Republic already had access to brings together key insights that can contribute digital media before the pandemic, activities to improving future DPL projects and broader undertaken during the pandemic helped achieve EdTech innovations in the country and other unprecedented progress in using EdTech for LMICs. teaching at home and in schools (MINERD 2020; MINERD 2021). For example, the current official high school textbooks are in electronic format, 1.2. which indicates the government’s commitment Prográmate to digitizing learning resources for students (Government of the Dominican Republic 2022). The Prográmate project uses McGraw Hill’s DPL software called the Assessment and One type of EdTech is DPL. DPL technologies Learning in Knowledge Spaces (ALEKS). This provide adapted content and/or instruction DPL platform uses adaptive questioning to based on the learner’s needs (UNICEF 2022a). determine students’ knowledge of different math A personalized learning experience focuses on topics quickly and accurately. It then presents learners’ strengths, interests, ownership of the each student with targeted activities based on Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Introduction 9 their individual performance. After logging into used in each of Prográmate’s four cohorts, the platform for the first time, students are reaching over 11,000 students in the Dominican presented with an Initial Knowledge Check or Republic between 2018 and 2022. diagnostic assessment. This assessment is then All of Prográmate’s cohorts were implemented used to identify topics that the student has in formal government school settings and during already mastered and to, accordingly, determine the normal school day; however, they varied individualized learning pathways for each student. slightly in their design and approach. The first As students work through the DPL platform, cohort was conducted as a pilot in five primary it periodically assesses their performance to schools in 2018. In 2019, 2020, and 2022, the appropriately adapt their learning experience. If project was implemented in 51, 33 and 45 students answer three consecutive questions schools respectively, to students in secondary incorrectly, they regress to previous topics; if school (high school). This study explores the students answer three consecutive questions cohorts of secondary school students (see correctly, they advance to a new topic. Table 1 below). Each Prográmate cohort began The software draws on thousands of test with an initial workshop for teachers and other questions from nearly 550 math topics to education personnel, including school principals provide a tailored curriculum to each individual and instructional leaders. The initial workshop student. In this way, it caters to students with for the 2019 and 2020 cohorts were in-person, heterogeneous levels of learning within the same while the initial workshop for the 2022 cohort grade, and through a personalized curriculum. For was facilitated virtually. The duration of the the Prográmate project, a team at McGraw Hill project implementation also varied. The 2019 adapted its DPL software to the Dominican grade cohort lasted three months. While the 2020 9 school curriculum, excluding the content of the cohort was planned to last longer, it was abruptly software’s math component that was not part of interrupted by the COVID-19 pandemic just after the MINERD national curriculum. Each student, two months. The most recent post-pandemic teacher and school principal had a unique cohort started late and only lasted one month. account within the DPL platform that allowed None of the cohorts were designed to formally for real-time, detailed tracking of individual involve parents or families. Instead, schools and utilization and progress. The DPL platform was teachers determined household engagement; this is further addressed in the findings. Table 1. Summary of differences in implementation across Prográmate cohorts Prográmate cohort 2019 2020 2022 Number of schools 51 33 45 Date of initial workshop Early March Late January Early May Modality of initial workshop In-person In-person Virtual Timing of implementation March–June January–March May–June Duration of implementation 3 months 2 months* 1 month Use Diagnostic Diagnostic Diagnostic assessment and 90 assessment and 90 assessment only, minutes per week in minutes per week in school and home school, homework school, homework use optional optional optional Source: Authors Note: *Interrupted by the pandemic. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Introduction 10 The way in which the project was presented months of using a DPL software, for an average to participants also varied from one cohort to of one hour a week, led to an increase of 0.29 the next. In both the 2019 and 2020 cohorts, standard deviations in MINERD math standardized participating students were provided with the task tests. Learning gains in math were concentrated of first completing the diagnostic assessment and among students in the bottom half of the then using the DPL software regularly throughout performance distribution, calling attention to the the week. Students were asked to use a school important potential impact that a DPL intervention computer for at least 90 minutes per week as can have on some of the most vulnerable part of the extended school day (Barón, Taveras, students.2 Similar results were found in the 2020 and Zúñiga 2018). The math teacher also had the data, mainly prior to school closures due to the option to assign online homework and encourage COVID-19 pandemic, but the correlation between students to use the platform outside of school. DPL software use and student performance The most recent 2022 Prográmate cohort received weakened after schools closed. different instructions. They were asked to focus Given these findings, it is critical to explore how, specifically on the diagnostic assessment to and under what conditions, students engage measure learning loss caused by the pandemic, with DPL platforms for individual learning. This rather than to use the platform to develop a qualitative study seeks to explore stakeholders’ pathway of personalized learning. In other words, perceptions and experiences with the DPL students were not asked to use the platform in a platform to try and understand the heterogeneity structured way (e.g. for a minimum of 90-minutes across schools. Documenting and reflecting per week) as in the prior cohorts. As this study upon these insights will help identify any finds, however, for the 2022 cohort, teachers took enabling and constraining factors that shape the it upon themselves to encourage students to use project’s uptake and implementation. This can the DPL platform in school or at home. This study ultimately contribute to ensuring that future DPL thus explores how students, parents, teachers, interventions in Dominican public schools are and other education personnel engage with the developed by drawing on evidence and lessons DPL intervention, and what happened when key learned. Prográmate has reached more than 11,700 stakeholders created their own opportunities and students in the Dominican Republic. Although conditions to continue using the DPL software all students who participated in Prográmate outside the parameters of the intervention itself. attend Dominican public schools, they constitute Despite differences across cohorts, quantitative a heterogeneous group of learners. They come data suggest that the Prográmate project had a from both rural and urban communities; they positive impact on participating students across have distinct learning needs; they have different all cohorts. Administrative data from the DPL levels of access to electricity, technology, internet platform from the 2020 to 2022 cohorts suggest connectivity, and pedagogical support at school or that students were able to increase mastery of at home. This profile of students suggest that the between 9 percent to 11 percent of the topics in findings of this report may have the potential to just two months of use. Use of the DPL platform shape future adaptive learning interventions not positively correlates to learning measures, both only for the Dominican Republic, but also for other internal and external to the platform. Data from LMICs. 2019 was also positive; findings show that three 2 Although the positive impacts of the project are less clear for other subjects, the study did observed some spillover effects on standardized test scores in social studies, natural sciences, and language, ranging from 0.25 to 0.29 standard deviations. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Introduction 11 To help build the global knowledge base on DPL, this report highlights how different stakeholders — students, parents, teachers and other education personnel — perceive the benefits of adaptive learning technologies, and it uncovers factors that drive stakeholders’ use and adoption in under-resourced contexts. The study is organized as follows: a literature review of DPL research (section 2), the study’s methodology (section 3), key findings (section 4), and a conclusion with discussion and recommendations for future DPL implementation in the Dominican Republic and other LMICs (section 5). Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Introduction 12 2. Literature review: Digital personalized learning This section examines the literature on digital personalized learning (DPL). It summarizes the available evidence on how to design effective DPL products, (2.1), as well as teachers’ roles in DPL interventions (2.2). suitable learning objectives or aligning teaching 2.1 and learning strategies (Shemshack and Spector Designing effective digital 2020). Learning can be personalized at an individual level, where teachers identify struggling personalized learning products students and provide feedback or extra support Digital personalized learning (DPL) refers to for them (Holmes et al., 2018), or personalized the use of technology to tailor educational learning can target groups of students with experiences to individual student needs, similar learning levels (Robinson and Sebba 2010). preferences, and learning styles. DPL leverages While effective, delivering personalized learning artificial intelligence (AI) and machine learning, can be overwhelming and time consuming for combining data and analytics to provide teachers, even in high-resource schools in high- appropriate content to support teaching at income countries (Grant and Basye 2014). The the right level (World Bank 2022b). Student affordances of EdTech can help address some of achievement is largely attributed to the quality these issues by empowering teachers (D’Angelo et of teaching (World Bank 2022a). Approaches that al., 2022) to better cater to the needs of learners personalize learning, through which instruction at scale (Major, Francis, and Tsapali 2021). is contextualized to the learner’s experience, are Studies indicate wide support for EdTech, which widely understood to effectively improve student offers the potential to improve learning outcomes learning (World Bank 2022b). In traditional because it can be personalized to learners’ classrooms, teachers may use targeted pedagogy education needs, based on age, attainment level, to adapt their instruction to the capabilities, prior knowledge, and personal relevance (Major, interests, and dispositions of their students Francis, and Tsapali 2021, 3). Multiple studies (Natriello 2013; Kucirkova, Gerard, and Linn 2021). support the effectiveness of DPL products in They can cater to learners’ needs by defining Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Literature Review: DPL  13 improving the learning outcomes of school-age languages of students (Hennessey et al., 2021). children from LMICs (Major, Francis, and Tsapali Curriculum content is also a key aspect of DPL 2021), especially those who are furthest behind in personalization: DPL interventions are more learning (Muralidharan, Singh, and Ganimian 2019; effective when content is closely aligned to the Nedungadi, Jayakumar, and Raman 2014, cited in local curriculum (Banerjee et al., 2007, cited in Upadhyay, Shoobridge, and Coflan 2021). These Upadhyay, Shoobridge, and Coflan 2021). tools can be expensive to develop and currently DPL products offer personalized feedback, which only cater to certain curricular topics, especially is widely noted as having the potential to improve mathematics and reading (Major, Francis, and student performance (Narciss et al., 2014) and Tsapali 2021). Nevertheless, DPL interventions motivation (Koenka and Anderman 2019), while that adapt or adjust to the learner’s level have empowering students to self-regulate their been found to lead to significantly greater impact learning (Ouyang and Jiao 2021). There is limited than those only linking to learners’ interests or evidence on how K–12 students in particular are providing personalized feedback, support, and/or able to self-regulate their learning (Maier and assessment (Major, Francis, and Tsapali 2021). Klotz 2022); however, evidence at the university DPL products enable “adaptive learning”3 that level suggests that students may be unwilling “uses machine learning to adapt to students’ to respond to feedback if they perceive the behaviors and competency” (Bulger 2016, 8). feedback that does not accurately reflect their They provide data-driven learning; DPL products effort (Lim et al., 2020), or if students feel they can be used to gather data through a single do not receive the same level of feedback as assessment or multiple assessments performed their peers (Henderson et al., 2019). On the other at set intervals to personalize the learning hand, feedback is effective when students trust experience of students (UNICEF 2022a). Artificial its quality and their teacher’s care toward their intelligence (AI) produces learning analytics which learning (Lim et al., 2020). Further, feedback include whether students are completing learning over media including audio, video, or screencast content and exercises, the amount of time they recordings can help improve how learners’ make spend on their learning, and their results on sense of personalized feedback (Henderson et al., exercises or assessments; some products also 2019). include analyses of failed tasks, overall progress, and learning feedback (Van Shoors et al., 2022). DPLs can also be used to personalize content, 2.2 identify learning paths, and provide feedback to Teachers’ roles students (Major and Francis 2020). in DPL interventions DPL products are most effective when the Teachers play a critical role in DPL interventions. content is relevant and aligned to local culture, Despite the common misconception that adaptive language, and curricula. UNICEF found that all DPL tools could eventually replace the role of 40 DPL products that were available in LMICs teachers (Cavanaugh 2020), research shows provide some level of personalization in terms that learning gains from DPL are higher when a of what to learn, when or for how long to teacher is available to offer students additional learn, the pace of learning, and the choice of feedback and support (Hennessey et al., 2022). language (UNICEF 2022a). Language is particularly Teachers have a continuous responsibility to important when designing EdTech in LMICs, due to low-literacy levels and the diversity of local 3 While the terms “personalized learning” and “adaptive learning” are distinct, they are frequently used interchangeably across literature (Shemshack and Spector 2020). In this study, personalized learning is used while sometimes drawing on evidence from research on adaptive learning. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Literature Review: DPL  14 make well-informed decisions based on the also play a critical role in creating an institutional learning data acquired through DPL tools (Basham culture that is conducive to teacher collaboration et al., 2016; Van Shoors et al., 2022, 3). and EdTech innovation (Hennessey et al., 2022). Teachers can use DPL to teach the formal While there is a dearth of evidence exploring the curriculum, or draw on DPL data to open implementation of DPL interventions in schools avenues for remedial instruction, so they can of LMICs, insights from the broader EdTech target teaching to the “right” level for students literature point to the importance of ongoing at scale (Major, Francis, and Tsapli 2021). This school-based support that provide teachers is particularly useful in contexts with high opportunities to reflect, discuss, and apply what learner-to-teacher ratios or teacher shortages they learn (Hennessey et al., 2022). As in any (Ito, Kasai, and Nakamuro 2019). DPLs can be EdTech intervention, the continuous support for used to free up teachers’ time by performing teachers can help overcome major obstacles in diverse functions, ranging from routine tasks, effective implementation of DPLs to improve such as skill classification, to intensive tasks in student learning. In cases where teachers are data analytics, including analyzing patterns in required to use new technologies, availability of students’ learning and supporting personalized on-demand technical support has been found to feedback (Perera and Aboal 2019). Teachers enhance their willingness to adopt information also play an affective and motivational role. and communication technologies (ICT) (Aldheleai Without a teacher’s timely communication or et al., 2019; Ferede et al., 2021). encouragement, students may grow disengaged To support teachers in DPL interventions, it is or resistant to DPL (Cavanaugh 2020). also important to create a classroom or school To maximize the effectiveness of DPL environment that is conducive to DPL. Poor interventions, therefore, teachers must be infrastructure, unreliable or intermittent internet trained and supported. Professional development connectivity, high data costs, or a lack of battery opportunities can equip teachers with the skills power for devices are common challenges and knowledge to integrate personalized learning (Upadhyay, Shoobridge, and Coflan 2021; with other teaching activities (Major and Francis Aadil, Nazir, and Akhtar 2021) that can lead to 2020) in order to help students become active frustration and discouragement among students collaborators in their own education (Cavanaugh and school personnel (Ferede et al., 2021). UNICEF 2020, 5). To effectively deliver personalized (2022a) found that most DPL products in LMICs feedback, teachers can be supported in their were primarily designed for implementation in efforts to learn how to adequately use the urban settings and were not designed to target technology, interpret the data, and scaffold marginalized learners living in remote or rural student learning with pedagogical support (Van areas, where access to technology and the Shoors et al., 2022; Maseleno et al., 2018). Several internet is even more scarce. studies have found links between teachers’ While there have been numerous studies of limited digital skills and their resistance to using DPL products and their benefits, this review technology in the classroom (Zubairi et al., 2022; of the literature on DPL shows that there Vanderlinde and Van Braak 2010; Aydin, Gürol, and remains a general lack of evidence on the use Vanderlinde 2016). Yet even in interventions where of DPL in public schools, at scale, in LMICs. teachers have access to and familiarity with This study seeks to fill this evidence gap with the devices used for delivery, implementation its examination of a DPL intervention in the challenges can emerge from attitudinal barriers Dominican Republic. It provides insights that can (Aadil, Nazir, and Akhtar 2021; Aldheleai et al., inform future DPL implementation in the country 2019). Teacher training can help address these and other LMICs. attitudinal barriers and support teachers’ use of and experimentation with EdTech. School leaders Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Literature Review: DPL  15 3. Methodology The objective of this study is to explore the experiences of students, parents, teachers, and other education personnel who participated in the Prográmate project in one of its three cohorts (2019, 2020, and 2022) in public schools across the Dominican Republic. In particular, the study explores the opportunities and challenges associated with using a DPL platform to teach secondary students the Dominican mathematics curriculum. In doing so, this study contributes to the broader literature on personalized learning and DPL implementation in formal public school settings. Within the sampled schools, a diverse selection 3.1. of students were chosen from each Prográmate Sample selection cohort. Students were selected from two cohorts: one cohort that participated before For this study, focus groups and interviews were the pandemic in either 2019 or 2020 and were conducted across 10 participating public schools now in the fifth year of secondary school (high in the Dominican Republic. The participating school); and one cohort that participated in 2022 schools were chosen across four school districts, and were in their third year of secondary school three in the country’s largest urban cities of (high school). Teachers, school leaders, and other Santo Domingo and Santiago, and one in Azua, a education personnel were selected to participate rural province on the country’s southern coast. based on their role in the Prográmate project. In Schools within each of these provinces were each school the math and computer teachers and diverse; some schools were situated in more school principal, who were invited to participate, developed and socioeconomically advantaged received usernames for the DPL platform, which communities than others. A purposeful sampling could be accessed as an app or through a web approach was used. School performance, based browser. In some schools, instructional leaders, on student learning through DPL, was used teacher assistants, other teachers or supervisors, to select a diverse set of schools from each as well as caregivers, parents, and other family province. In the cities of Santo Domingo and members, were also invited to participate in the Santiago, schools with high, medium, and low study. A total of 231 stakeholders participated, performance levels were selected. In the rural including 167 students, 11 parents or caregivers, province of Azua, one high-performing school and 53 teachers or other education personnel was selected to understand what works well in (Table 2). remote areas. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Literature Review: DPL  16 Table 2. Sample demographics City (District Number) SD (15) SD (10) STI (08) Azua (03) Total Students 2019 or 2020 31 6 16 16 69 Students 2022 32 28 26 12 98 Teachers and education 17 15 10 5 47 personnel School principal 3 2 0 1 6 Parents and caregivers 0 6 2 3 11 Total 83 57 54 37 231 Source: Authors’ calculations. Note: SD = Santo Domingo; STI = Santiago. The average student participant across the three particular, stakeholder engagement with the DPL cohorts had a similar profile. Here we present platform from 2019, 2020 and 2022. All interviews some data on the 2019 cohort as an example. and focus groups were facilitated in Spanish In 2019, 57.5% of the student participants were and audio-recorded. A total of 18 hours of audio female, and 42.5% were male. The average age recordings were analyzed. was 14.8 years. Students came from diverse socio-economic levels. Most students were either from the fourth quintile (45%) or third 3.3 quintile (33%). Some students also belonged to Data analysis the first (4%) or second quintile (14%), while other Data were analyzed using a thematic coding were in the fifth or last quintile (4%). Student framework (Table 3). Different themes were used participants in the 2019 cohort on average spent to code interviews with students, and other 10.5 hours on the DPL platform over the course of research participants, such as teachers and other the project. education personnel, who commented on the broader implementation process (the X in Table 3.2. 3 identifies which themes were used for each Methods type of research participant). Key themes and subthemes explored stakeholders’ perceptions Interviews and focus group discussions were of Prográmate as a school-based intervention, semi-structured and served a twofold purpose. and their engagement with, and perceptions The first purpose was to examine stakeholders’ of the DPL platform. The themes explore both experiences with and perceptions of the DPL benefits and drawbacks or challenges of the platform. The second was to understand intervention, its design, and the DPL technology. their experiences with school closures related This coding framework was designed and adapted to the COVID-19 pandemic and returning to iteratively throughout the data analysis process. school. Prográmate’s first cohort (2019) took Data were analyzed in Spanish by three Spanish- place before the pandemic; its second cohort speaking researchers, who met numerous times (2020) was interrupted by school closures to discuss the coding process and held a final caused by the pandemic; and the third cohort online workshop to reflect on the findings with a (2022) participated after the post-pandemic wider research team. Responses from students reopening of schools. This report only draws on and teachers were extracted from the data and data relevant to the Prográmate project and, in translated to English for inclusion in this study. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Literature Review: DPL  17 an hour. No data on students with disabilities 3.4. or special education needs were available, and Limitations therefore more research is needed to determine how these young people engage with and benefit Despite the limited number of schools that from DPL interventions in public schools. participated in the project, the diversity of Importantly, we found no major differences contexts adds to the generalizability of the across subgroups of students, teachers or other study. Nevertheless, only a limited number of education personnel. Boys and girls, women and parents and caregivers participated, due to their men, seemed to benefit equally from the DPL busy schedules. Also, many of the 2019–2020 platform. They also faced similar challenges. cohort of students did not recall much of Heterogeneity was found in relation to the their engagement with the DPL platform in the different cohorts (2018-2022) and rural or urban Prográmate project. This led to some interviews communities. For that reason we include this running very short. Interviews and focus group information in the findings below. discussions ran between 10 minutes to over Table 3. Concise version of coding framework CODES RESEARCH PARTICIPANT Themes Subthemes Student Other Perceptions of Of initial workshop X Prográmate Of program timing X Of Prográmaté’s relevance or importance X Description of user Time and place of use X X engagement with the Technology and infrastructure X X DPL platform Role of teacher X X Role of other education personnel X X Role of family (caregivers, siblings, etc.) X X Motivation to use the DPL platform X Perceptions of using Benefits of the DPL platform X X and learning the DPL Drawbacks of the DPL platform X X platform Source: Authors Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Literature Review: DPL  18 4. Findings The findings of the study include answers to three main research questions: (i) In what ways do key stakeholders, such as students, parents, teachers, and other education personnel, engage with the DPL platform? (ii) How do these stakeholders describe the benefits and drawbacks of the DPL platform?, and (iii) What are the enabling and constraining factors that shape the implementation of the project in schools and at home? onboarding workshop. The idea was to respond to 4.1 their concern, questions, and adapt as much as Implementation of Prográmate possible to the needs of each school. Each Prográmate cohort had a specially designed This section details the project’s implementation. initial workshop. The differences in these It considers how the implementation of the workshops were noted by research participants. Prográmate project was affected by the initial Some education personnel reported the lengths workshop and the launching of the DPL platform of the workshops as lasting two days, one day, or in schools. It also examines the effectiveness of just a couple of hours. Also, there were variations DPL implementation including the timing of the in the groups that attended the workshops. In intervention, teacher and student involvement, the 2019 and 2020 cohorts, teachers and other and technical difficulties. education personnel from the same district attended the workshop. In 2022, the workshop INITIAL WORKSHOP included educational personnel from districts The Prográmate project began with an invitation from all over the country. While in the pre- from district authorities to implement the DPL pandemic implementation there were in-person intervention in a school. The invitation informed initial workshops (the 2019 and 2020 cohorts), the directors, coordinators, and professors the workshops became fully online after the that their school was selected for the program, pandemic (the 2022 cohort). The 2022 cohort’s described the utility of the adaptive platform initial workshop was therefore more flexible in for student learning, and invited them to an its location and time, so it was more feasible to have education personnel from different districts attend simultaneously. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 19 However, there were common challenges with I feel that if we are going to the quality of the initial workshops across all continue working on this, we need to cohorts. Teachers and other education personnel be trained. Because when you come raised concerns about the content and methods and face the problems, then students of both the in-person and online workshops. believe that you know how to solve Teachers and education personnel also described them. These students ask you and you the workshops as being very theoretical and not have to demonstrate that you know. practical, or not going in-depth into the resources – Mathematics teacher and functionalities of the DPL software. Education personnel reported having difficulties following the online workshop due to connectivity SETTING UP AND LAUNCHING issues interfering with the audio and image. This THE DPL PLATFORM affected their ability to attend and learn from Following the initial workshop, teachers and other the initial workshop. Overall, the opinions about education personnel were involved in helping these workshops were mixed. students create user profiles and passwords to We didn’t learn much from the launch the program at their respective schools. initial workshop. That is one of Research participants reported difficulties during the weaknesses that I see with the this logistical process. Many students did not program, it happened in 2019 and have email accounts, or they did not know the it happened now [2022] too. It was details of their email accounts since they did very theoretical and from there they not use them often. Students also reported [workshop facilitators] left us to that their assigned usernames and passwords defend ourselves however we could. were sometimes incorrect, so they had to be re-programmed. At one school, for example, a – Mathematics teacher group of teachers and students had to recreate On the other hand, a number of teachers and all their usernames due to technical problems. other education personnel reported that the Implementation also required all participating training on how to use the platform they received schools to have student email accounts before during the workshops was helpful. making the platform available, which meant that even when some schools finished collecting all Despite whether teachers and other education email addresses and assigning usernames, they personnel had a positive or negative perception had to wait for other schools to finish before they of the workshop, they recognized its importance. could start using the DPL platform. This delay They thought it was crucial for the success ultimately added to the short duration of program of the project, since they needed to have the implementation. According to teachers and other skills to facilitate students’ participation on the education personnel, this process resulted in platform. Teachers needed to be trained to use students missing out on valuable learning time, the platform in order to provide guidance and which they would have otherwise had to engage technological support for students; the initial with the DPL platform. workshop helped teachers to better assist their students. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 20 TEACHERS INTRODUCING THE DPL PLATFORM and depended on students’ motivation and TO STUDENTS willingness to engage with the platform. To encourage their students, teachers emphasized The motivation and incentives that education that using the DPL platform was not a graded personnel communicated to students also played activity, so the students would not feel pressured a key role in students’ use of the DPL platform. or anxious to participate. Instead, teachers The following dataset show two main distinctions affirmed the advantages of practicing math when teachers introduced the DPL platform to topics and encouraged students to use the DPL their students: DPL as a graded activity and DPL platform as much as possible. as a nongraded activity. The teachers were very influential. 1. DPL as a graded activity. Schools They said that we should connect [to communicated that students’ use the platform] as much as possible of the DPL platform would affect [and] as long as possible, so that the their grades. In this strategy, explanations they give after that schools might include positive will be better, and we will have a incentives, such as earning clearer idea of the exercises. additional points, or negative incentives, such as losing points, based on the time students – Third-year secondary student, rural spent using the DPL platform. Schools applied school (2022 cohort) these strategies when they did not have enough students entering the platform. These incentives TIMING OF IMPLEMENTATION then served as a form of extrinsic motivation for students, leading them to use the platform One recurrent aspect mentioned by education more frequently. However, student participation personnel in the interviews was the short was thus catalyzed by their desire to comply period of time given to implement and use of with the requirements, rather than to engage in the platform—implementation was limited by meaningful learning. the availability of funding. The short notice to start and the limited time to use the platform There was a great deficit of made education personnel move faster than they students who did not enter to do anticipated, leaving them feeling pressured to the registration and the diagnosis, achieve the goals in a compressed time frame. and that is where the threat of This was especially concerning for teachers, who losing 20 points in the grade came described how working with a new platform from; we all did it, but I think that required substantial time for them to familiarize only 10 percent of the students did themselves with the technical aspects and to it conscientiously (I am in that establish enough expertise in the DPL platform 10 percent). before providing guidance and support to their – Third-year secondary student, urban students. school (2022 cohort) My advice is to have more time 2. DPL as a nongraded activity. and work more directly with the Schools implemented the teacher so that he can work better DPL platform separately from with the students and the core lesson plans and student information can flow. evaluations. Student use of – Mathematics teacher the platform was optional Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 21 Another factor that influenced stakeholders’ use of the platform was the timing of the 4.2. project’s implementation. Education personnel Student and teacher reported that the project started too late in the academic year. The scheduled implementation engagement with DPL was in February or March, and the MINERD academic calendar ends in June. This caused TIME AND PLACE OF USE conflicts with the schools’ planned end-of- The total time students invested, or were year activities and final exams. The timing expected to invest, on the platform varied also interfered with teachers planned lessons greatly across and within each Prográmate for teaching math content over the course of cohort (see Table 1). Some students made their the academic year. Teachers raised concerns own decisions on how they engaged with DPL, about navigating multiple and competing roles, while other students were assigned to use the including implementing the DPL software while platform by their teachers or another adult. In simultaneously learning how to use it, supporting other words, students spent time using the DPL their students, and delivering their regularly software if they were either self-motivated or planned lessons. encouraged by teachers, peers, or parents. How Due to the short notice and the fact that the much time students spent on the internet or on program was implemented at the end of the devices engaging with the DPL software was also year, education personnel suggested that future likely shaped by socioeconomic factors, given implementations start at the beginning of the the high costs of devices and internet access in year, which would allow schools to plan their the Dominican Republic (Larra Ibarra, Comini, academic years accordingly. Teachers would have and Gelvanovska-Garcia 2022). Indeed, various more time to prepare themselves to address stakeholders identified the limited access to any challenges that arise for their students, and devices or the Internet at home and in schools, students could depend on teacher support to as discussed further below. help them take full advantage of the learning Some schools and teachers scheduled time opportunities available on the DPL platform. for students to use the DPL software in the In contrast, students in focus group discussions computer labs during their regular math or pointed to potential benefits of the timing; computer lessons. These schools often had they cited advantages to having a DPL software access to smartphones and laptops donated at their disposal at the end of the year. Many during the government’s República Digital students who were approaching Pruebas program, and which students could use. Some Nacionales (the national standardized tests) teachers also assigned students homework to asked if the software would still be available to use the DPL platform. They then monitored in- them after the project ended, so that they could home practice by reviewing student progress in use the DPL platform to study and prepare for the platform. Students who were encouraged the subjects in which they felt less confident. to use the DPL platform in school and at home This points to important implications regarding often reported spending more time using the how students may monitor their learning and platform when compared to their peers. Using engage with DPL for self-study or independent the platform in spaces beyond the classroom learning outside of the classroom. opens up the possibility of students spending more time engaging with DPL. While many schools may have explicitly encouraged students to use the platform at home, this option was Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 22 only available for those students who had access Technological challenges in schools.The to devices, such as smartphones or personal implementation of the Prográmate project computers, at home. was affected by the schools’ technological infrastructure and access to devices. Schools Teachers’ ability to monitor and provide located in smaller cities or towns in rural and incentives to their students influenced the remote areas reported more problems with overall time that students used the DPL platform. technology when compared to those located in Teachers and other education personnel large urban areas. Issues pointed out by students described how they drew on data analytics and education personnel include the following: to track which of their students entered the platform, and how much time they spent using Unstable supply of electricity. the platform. They would then follow up and Many schools and/or student provide encouragement or support (see more homes did not have stable, reliable below in relation to teachers’ role). When electricity throughout the entire teachers asked them to do so, students invested school day, affecting their ability to more time on the platform. engage with the DPL platform. We used it here or at home using Inadequate devices. Some schools the phone, I did it at home because had computer laboratories without the teacher said so and also functioning hardware. Many of because I had interest. the government-issued laptops in schools were outdated and could – Fifth-year secondary student, rural not be used reliably to access school (2020 cohort) the DPL platform. Large-screen devices like Other students, however, described feeling computers, rather than smartphones, were more motivated to use the platform of their own reliable but in short supply. volition, either because they could visually see Inadequate internet connections. how they were progressing and enjoyed it, or Access to and the speed of the because they could compare their progress with internet hindered the use of their peers, and thus compete in a friendly race the DPL platform. Students and to advance through the topics (see section 4.3 for teachers often had to wait for more on scores). the DPL content to load, at times taking over an hour, which interrupted learning time and caused TECHNOLOGICAL CHALLENGES frustration. During the focus group discussions, students, Teachers, education personnel and students teachers and other education personnel tried to employ different strategies to overcome reported that one of the main challenges of these technological challenges. They used their the implementation was the lack of proper own resources and took advantage of what was technological infrastructure. This affected the available and within their reach. These strategies time students invested in the platform and included: the overall satisfaction of the students and education personnel. The data suggest that a. Spending extended time in school. Since these challenges were found in both schools and there were limited devices and internet homes across regions. connectivity, extended school hours helped the students have more time with the DPL platform. This allowed the schools to use the computer laboratory at its maximum capacity. To do this, different groups of students Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 23 took turns spending extra time in school. computers or asked classmates to share their This strategy required clear communication smartphones or laptops with their peers. Two between schools and parents. Teachers and or more students often shared computers and school leaders met with parents to seek worked together in the computer lab in small permission for their children to stay after groups or as partners. These arrangements regular school hours, and parents agreed that raise questions about the accuracy of DPL’s the extra time spent on the DPL platform personalization for individual learners. would be beneficial for their children. b. Using smartphones. When schools could not Technological challenges at home. The use of the provide enough computers to students, they DPL platform at home was highly encouraged by found that turning to the use of smartphones some schools to provide extra time for students was a reasonable solution. Teachers and to practice. It was also necessary due to the other education personnel asked students challenging technological conditions in accessing to bring their devices and allowed their use the platform in schools. However, the data suggest during class time, something that some that there were also technological challenges at schools usually prohibited during regular home. Challenges were especially common for school time. However, both education students from low socioeconomic backgrounds, personnel and students reported different and are similar to those in schools: lack of devices challenges arising from this approach. and access to stable internet connectivity. Among Education personnel pointed out that some the factors that helped the students in these students engage in nonacademic activities situations are: on their phones, while students mentioned Availability of government-issued that the DPL software was not optimized to laptops. MINERD provided some be used on smartphones. They reported that students with personal computers all the information was not visible on their through the República Digital smartphones, and they sometimes struggled program. Students could use these to introduce an answer with the small screen laptops in schools, and some and keyboard. Also, not all students had schools allowed students to take them home. their own smartphones, and their parents’ Students reported using the DPL software on smartphones were not available to bring these laptops at home. However, some schools to school. In other words, parents acted mentioned that the laptops needed maintenance as gatekeepers to smartphones for certain and repair to work properly. students. Since they gave us the password, we c. Sharing data packages. With connectivity could use it [DPL platform] at home problems in classrooms, students were and that’s when they gave us the asked to use their own internet to carry out República Digital laptops. So I put the activities on the DPL platform. However, in my password and did exercises by not all students could afford it. In these myself. cases, classmates and teachers shared their cellular data with students who did not have – Fifth-year secondary student, rural their own internet connection so they could school (2020 cohort) have the opportunity to work with the DPL Family devices. Parents or relatives software. provided devices for use at home, d. Sharing devices. In cases where schools did allowing students to use them, not have enough devices or students did not have smartphones, teachers shared their Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 24 when available. However, this was not ideal Pedagogical coordinators had the executive because students had to adapt to their families’ role of supervising teachers, overseeing the schedules to use the devices. implementation process, monitoring progress, and maintaining clear communication with all In my case, I had to wait for stakeholders, including parents and families. my mom to finish working on her laptop to use it. Teachers were responsible for helping students use DPL in schools, or providing additional – Third-year secondary student, urban assignments and homework to use the DPL school (2022 cohort) platform at home. Math teachers and, in some Family cellular data. Households cases, computer teachers, were in charge of often lacked a reliable internet directly implementing the DPL program with connection. While some parents or students, and they oversaw activities such as the other family members could buy following: cellular data packages for students Monitoring progress. Teachers to practice from home, not all parents and logged into the DPL platform to families could. monitor each student’s progress. I used it [the DPL platform] every The platform provided them with day, but not always, because there information on the time that was no internet at home. So it was students spent using the DPL platform each internet from data packages, and you day, as well as their progress in mastering the always had to top it up, so my mom different math topics. This helped teachers to helped me with that. realize how to implement different strategies for students to engage with and learn from the – Fifth-year secondary student, rural platform, especially for students whom the school (2020 cohort) project could identify as falling behind. Teachers Parents, teachers, and other education personnel sometimes shared these results publicly in supported students in overcoming tech-related class to congratulate students when they made challenges. This points to the important role progress or motivate students when they were these key stakeholders play. behind. Motivating students. Teachers ROLE OF TEACHERS AND OTHER maintained direct communication EDUCATION PERSONNEL with students throughout the Implementing DPL successfully depended on the implementation of the project. This coordination of all education personnel, including helped them motivate students to school principals, pedagogical coordinators, and log into and engage with the DPL especially teachers. platform. They usually highlighted the benefits of doing so, mentioning that DPL would help School principals played a leadership role in them to succeed in current or future topics and providing instructions and aligning resources and courses. human capital. Among their concerns were the availability of devices and connectivity to reach Providing additional explanation all the students in the program, and to increase and support. Teachers were able to their school’s performance. guide and support student learning during the implementation. They helped explain the platform to students, showing them how to use it, solving technical issues that students may have, and Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 25 providing additional explanation of the learning Despite the challenges they faced, teachers also content itself. According to the focus groups, reported benefits of the project. They used the teachers felt inadequately trained and skilled DPL software to learn math in a new way and to to provide the sort of pedagogical and technical get innovative ideas on how to plan their regular support students needed. classroom lessons. They mentioned how the variety of content and exercises allowed them to Teachers faced challenges, especially when incorporate a broader range of activities in class trying to juggle their regular lesson planning with and make their work easier. DPL-related support. As one teacher described, the curriculum content that students were However, teachers also expressed anxiety and learning may have been aligned to the national frustration at the implementation process. curriculum, but it was not always aligned with Teachers reported that many students had the “curriculum of the day,” or the content of the questions that required pedagogical and technical daily lesson that the teacher had planned. This support they did not feel adequately trained for, meant that the teacher’s lesson may have been and they felt pressure from school principals or interrupted with requests to learn the content coordinators who expected results and progress from the DPL platform that the teacher was not with the project in addition to their required prepared to teach and that the class had not yet regular lesson schedule and other academic started to learn. tasks. Teachers described having to fulfill their duties within the project without proper devices, Students highlighted the important role of internet connectivity, or adequate technical or teachers in the implementation process. They pedagogical support. said teachers were crucial to complement explanations from the DPL platform. Students reported that having teachers in-person allowed ROLE OF PARENTS AND FAMILIES them to have more immediate support. Teachers Although parents were not often mentioned could answer their specific questions in a during the interviews, education personnel personalized way, something that they could not and students provided some clues about their find in the DPL software. According to students, role during the implementation. Their level teachers could adapt feedback and explanations of involvement depended on the approach of to each student. Consequently, many students each school to project implementation. Some preferred oral explanations provided by their institutions did not expect much parental teachers to the written explanations provided by involvement; in these schools, parents did not the DPL software. play a key role. However, in one case, when a The teacher is more available. If you school met with the parents to introduce the are curious about a topic you have program and explain the purpose of working with the teacher to ask whatever you want. DPL, the families became involved and influenced With [the DPL platform] you don’t. student participation, not only by encouraging their children to use the platform but also by – Third-year secondary student, rural providing technical support, devices and cellular school (2022 cohort) data so that the children could use DPL at home. It’s easier to learn orally with the My aunt, my cousin and my mother teacher because they explain better helped me. and you have to read a lot in [the DPL platform]. – Fifth-year secondary student, rural school (2020 cohort) – Third-year secondary student, urban school (2022 cohort) Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 26 they learned in previous years but either had 4.3 forgotten or did not remember fully. They also Perceptions of found it helpful when studying for the regular math course they were enrolled in. Students the DPL platform were able to strengthen their knowledge of In their focus group responses, students and confidence in the math topics through the described the benefits and drawbacks of various platform. Students described how repetition and features of the DPL software, including the continuous review contributed to their ability diagnostic test, the explanations, the content, to retain information and master different math the scores, and the personalization of the topics. experience. They also compared it with other I really liked it because it helped me technological tools. remember topics, for example, numbers in fractions. I have never been able The diagnostic test. Students to master that subject well. So I found it insightful to know their went into the explanations and the strengths and weaknesses. They first time I didn’t understand it, the thought it was useful to know second time I didn’t understand it, which topics they had forgotten and needed to then I saw it later and I was able to review. Many commented on the visual display master it more, and since a month ago of their scores, which were shown with graphics the platform has been helping me a lot. and in color, making it clearer and easier for – Third-year secondary student, rural them (and their teachers) to track their progress. school (2022 cohort) Some students reported being surprised, disappointed or ashamed by their results. Others With the platform adapting content to the compared their scores with their peers engaged student’s ability level, some of the students in friendly competition. The diagnostic test were exposed to more difficult topics, which enabled students to measure their progress and allowed them to study at more advanced levels encouraged further practice and learning on the in school. This provoked divided opinions. Some DPL platform. students found it useful and considered it as an advantage since they thought it would better The diagnostic test is good because prepare them for future courses. However, some that way the platform can know the students expressed frustration because more student’s knowledge in order to advanced content required more explanation reinforce it. If a bad result came out and more individual effort. In addition, teachers in the test, the application could described how this required their additional time provide feedback. to support students in subjects that they had not – Fifth-year secondary student, rural yet learned in class. school (2020 cohort) Explanation and feedback. Some students thought the feedback from the DPL platform was Content. The platform provides clear, noting the platform included an “I don’t personalized content based on know” option, which could be selected to provide the level and previous knowledge a variety of examples on a specific topic to of the students. Students support their learning. Other students described found it useful to review topics how the platform drew on real-life examples Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 27 which they could identify with. It also provided also reported that these scores were useful in techniques to resolve exercises more quickly knowing their areas of improvement and the and easily without having to do too much extra topics that needed more effort to master. Other work. Students highlighted how the platform students, however, described how seeing their adapted to their level of understanding to provide scores drop discouraged them. more or less explanation depending on their Other subjects and skills. The individual progress, making the learning process data show that DPL can help more efficient than classroom experiences when acquire skills and competencies teachers had to provide general explanations in math and other subject areas. for all students to fully grasp the day’s lesson. The equations they learned on the Students also appreciated how the platform DPL platform could benefit other repeated exercises when they made mistakes. subject areas, such as chemistry and physics. They reported that the platform explained their Students also found that the DPL platform mistakes, indicating why an answer was incorrect, provided texts in the context of math exercises and then provided additional explanations and that could also help them improve their Spanish, examples from which they could learn how to get reading skills, or soft skills, such as critical the correct answer. thinking and problem-solving. Using the platform However, many students also needed their also helped some students and education teachers to expand on some explanations from personnel acquire better digital skills; however, the DPL software or to provide additional support some teachers, who did not feel they had for their individual questions. They felt that the adequate training on computers or the platform, platform did not always adapt the explanation and some students without basic digital skills or to their specific characteristics as a learner, computer knowledge, mentioned that working and that teachers could tailor feedback and with the DPL platform was quite challenging. explanations more effectively to meet their There are students who don’t even know needs. how to turn on the computer. Almost Scores. The ability to see each all of my students who entered were student’s score in the platform because I entered with them because was a recurrent topic that they don’t know how to use a computer. emerged across the focus groups. – Mathematics teacher Students reported being able to see their scores, and how they either increased DPL and other digital tools in or decreased depending on their progress. the classroom. When students Overall, students described this functionality compared their experience using as useful to measure their results and motivate DPL software to other digital tools, them to practice more. This is highly related to such as Kahoot and Khan Academy, the competitive spirit among some classmates. they emphasized that the DPL Pairs or groups of students in the same class software was more personalized and automated. usually compared their scores, and the ones They noted how it had extra integrated tools, that were behind felt motivated to do more such as a calculator and an option for them exercises, get more points, and surpass their to look up the meaning of an unfamiliar word. classmates. Some students described having a Some students described using the YouTube competition with themselves. Even if there was channel that was associated with the DPL no direct competition with other peers, students software to access additional learning content with low scores sometimes felt frustrated and and video explanations. However, this feature made an effort to get better results. Students was not known to all students or teachers, Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 28 calling attention to the importance of the initial workshop and training provided to teachers and other education personnel (see section 4.1). Overall, students who used the DPL software for both the diagnostic test and regular practice recognized the advantages of having a tool that is adapted to their level of knowledge, and they appreciated the explanations and feedback the platform provided. However, in some schools, students were less optimistic about learning with the DPL platform since they mainly used it for the diagnostic test and not for learning practice to help them review prior and new content. Consequently, they did not receive explanations or detailed feedback on exercises, and they were not encouraged to further their learning practice of math topics on the DPL platform. Even in these latter scenarios, students recommended the DPL platform for evaluating math competencies. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Findings 29 5. Discussion and Recommendations This study has explored the perceptions and experiences of students, parents, teachers, and other education stakeholders involved in Prográmate, a project using a digital personalized learning (DPL) platform to develop secondary school students’ math skills and knowledge in the Dominican Republic. Research participants identified various benefits devices, especially large-screen devices, such as of the DPL software, namely the importance computers. The platform did not operate as well of continuous assessment and the platform’s on small-screen devices, such as smartphones. ability to adjust the difficulty level of the learning Training on the platform was also cited as a content to meet student needs. Students were challenge; some students and teachers did not motivated to use the DPL software, especially always know how to properly navigate the DPL because they were able to see their learning platform. This calls attention to the importance progress, use their scores to compete with their of ensuring DPL technology is aligned with user peers, and engage with EdTech to improve their skill levels and is accessible on personal devices. learning. Students described the platform as a It also points to the diverse socioeconomic valuable learning tool, often using it to study conditions of schools and households, or review a specific topic before taking a test. where students are expected to access the Teachers also valued the platform and the data DPL platform but may not have adequate it provided on their students. It allowed them infrastructure to reasonably benefit from EdTech to monitor how their students were doing, and projects. These challenges can lead to frustration they could offer motivation and constructive among students and teachers and are well encouragement, as well as instructional support, documented in the literature on EdTech-related especially on topics their students needed to projects (Upadhyay, Shoobridge, and Coflan 2021; review and practice. Ferede et al., 2021). Research participants also identified several Some students also struggled to understand the challenges with the DPL platform. They reported written explanations and feedback provided by technological challenges such as limited access the DPL software, and they emphasized their to reliable electricity, internet, and functioning preference for learning through oral explanations Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Discussion and Recommendations  30 provided by their teacher. This points to another the platform at home, or carving out structured critical issue regarding students’ literacy levels: time during the week to go to the computer lab though the DPL platform personalized the math and use the platform in class. The more time content it provided for individual students, it students spent using the DPL platform, the more did not adjust the literacy levels of the learners likely they were to increase their math skills and associated in the delivery of the written text knowledge.4 Therefore, it is critical that teachers, or explanations of the math content. Prior students, and other stakeholders are equipped research has suggested that DPL tools should be with the skills on how to use the platform designed in consideration of low literacy levels effectively; teachers should also have the know- of its learners (Gove and Cvelich 2011). Because how and confidence to monitor student progress of the challenges that students had with the and encourage their participation. explanations and feedback, teachers played a vital Effective implementation of DPL requires human role in providing instructional scaffolding for their engagement and understanding, monitoring, students as they engaged with the DPL platform. and the intervention of adequately trained This study has also highlighted key insights teachers. To maximize the effectiveness of regarding limitations of the implementation future Prográmate cohorts, teachers can be process. According to research participants, a better equipped with the training to integrate major challenge that limited the impact of the DPL with their other teaching activities (Major Prográmate project was the timing and duration et. al., 2020). Some teachers were able to enter of implementation. The project was introduced the platform and access learning analytics data to schools toward the end of the academic year, to monitor the amount of time students spent which meant that teachers felt overwhelmed on learning, their completed learning content, with other academic and extracurricular results of exercises or assessments, and their responsibilities and students only had access to overall progress in mastering the math topics, the DPL software for less than three months (see all affordances that have been highlighted in the Table 3). Another limitation in the implementation literature as improving the implementation of DPL was the technical challenges many schools faced projects (Van Schoors et al., 2022). Stakeholders when setting up usernames and passwords for reported that student learning on the DPL students to access the platform, which resulted platform was maximized when the teacher was in delaying the implementation process. Teachers there to provide additional support (Hennessy et also reported technical issues that affected the al. 2022). This study has shown that the success implementation of the DPL platform. Teachers of the intervention was hampered by the limited reported leaving the initial workshop feeling support teachers received during the initial anxious or apprehensive about their ability to workshop and throughout the duration of the effectively use the DPL platform or to support project implementation. their students in their use of it. The lack of When implemented effectively, DPL interventions confidence in their training by some teachers have the potential to provide tailored support to and education personnel led them to only ask diverse groups of students. The DPL software their students to complete the diagnostic test. for the Prográmate project was adapted to the In other schools, students were encouraged Dominican national curriculum in Spanish, and the not only to complete the diagnostic test but to math content was tailored to students’ diverse use the platform to practice learning the math learning levels. However, the written explanations topics both in school and at home. They sought presented a challenge for some students, likely to maximize their students’ engagement with due to their lower levels of literacy skills or the DPL platform, by assigning homework to use reading comprehension. This calls into question 4 According to administrative data from the DPL platform, for all cohorts (2019, 2020, 2022) Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Discussion and Recommendations  31 the need to personalize all aspects of learning 2. Ensure adequate timing content, or adapting the written explanations to and sufficient duration of meet the literacy levels of the learners using the platform. Literacy is particularly linked to equity, project implementation. where literacy levels can impact the students’ Given the heavy workloads of teachers, it is ability to engage with or benefit from DPL crucial that a DPL intervention does not impose interventions. Evidence suggests that effective an additional burden. Important academic and DPL models integrate learning content that is extracurricular dates and schedules should be adequately aligned to national curricula, student factored into the initial planning of a project, learning levels, and local language (UNICEF so the implementation of the project does 2022a). However, more research needs to be not interfere with teachers’ other duties. done; there is dearth of literature on how textual Attention should also be paid to the duration or written language in DPL tools may impact of implementation to ensure maximum results. student learning through DPL. The more time students spend on the DPL platform, the more they become familiar with This study has sought to provide valuable it and learn from it. insights into the use of DPL interventions in the context of public schools in the Dominican 3. Provide ongoing support to Republic. Given the findings of this study, the following six recommendations are offered to teachers, so they can better help improve student learning for future DPL support students. interventions in the Dominican Republic and Technology should empower and strengthen other LMICs. These recommendations are teachers’ roles in the classroom (D’Angelo et particularly important for designing and al., 2022). One-off workshops are ineffective in implementing DPL interventions at scale in equipping teachers with the training necessary formal school settings and for diverse learners. to successfully integrate new digital technologies into formal schooling. Ongoing and continuous 1. Co-design school-based DPL support should be provided to teachers and programs with teachers and other education personnel, including, for other education personnel. example, in the form of a remote helpdesk that provides just-in-time technical support, follow- Involving school actors in the design of DPL up in-person supervision and workshops, or an interventions can increase stakeholder buy-in and in-person or remote community of practice. ultimately yield a more fruitful implementation Participating schools should have opportunities process (Aadil, Nazir, and Akhtar 2021). To to share best practices and brainstorm solutions maximize the uptake and effectiveness of DPL to the challenges they confront, and model interventions in formal school settings, it is teachers or schools can be identified to share critical that these interventions are aligned with exemplary efforts. For example, this study the realities of, and work environments in which, has illustrated the critical role of teachers teachers and other education personnel operate. in fostering student engagement, monitoring Teachers and other stakeholders, including progress, and designing classroom or homework pedagogical coordinators or supervisors, school activities to enhance DPL use. Ensuring teachers leaders, and even parents and families should are provided ongoing support to complete be engaged and consulted in the process of their multiple roles effectively is of paramount designing and/or adapting DPL efforts. A co- importance. creative process will also help ensure that project goals are relevant, and that structures or systems are in place to adequately support all actors throughout the implementation process. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Discussion and Recommendations  32 4. Ensure an enabling 6. Fund, commission, and conduct technological environment. more DPL implementation Despite the efforts made by teachers, students, research, including in formal families, and other key stakeholders, major school settings. recurring challenges for schools were the While there is substantial evidence on the design limited technological infrastructure, internet of DPL products and interventions, there is less connectivity, and access to devices. DPL so on the process of implementing DPL models, interventions must consider how to foster especially in formal school settings. This study enabling technological conditions. For example, has revealed the potential positive impact schools could be equipped with well-functioning that DPL interventions can have on students computers or laptops, or their computer and teachers in public schools. Students may laboratories should have working devices. feel more motivated to use digital technology Schools or families could receive subsidies for to practice and monitor their learning, while internet data costs, and education personnel or teachers can draw on student data and DPL external ICT experts could have integrated digital learning analytics to provide pedagogical and skills training and support for students and affective support to their students. Nevertheless, teachers, which would help ensure that they are without creating enabling classroom and able to effectively navigate the DPL platform and school conditions, or considering the individual digital tools. Governments should also consider language and literacy needs of students, the partnering with private Internet providers or use of DPL to improve student learning in math tech companies that can support with resource may be jeopardized. More research is therefore allocation, distribution, or maintenance. needed to better understand the enabling and 5. Diversify learning content, constraining factors shaping the implementation of DPL interventions in public schools, especially not just by level, but also in LMICs. Generating and disseminating more by modality. evidence on how teachers are effectively using The DPL platform was well received by research DPLs and how DPLs can help teachers teach participants. All stakeholders found the more effectively, will help inform education personalization of the math learning content policy and practice in the Dominican Republic and to be particularly useful to identify student other LMICs (Major, Francis, and Tsapali 2021). strengths and weaknesses and to sharpen their knowledge or skills in a specific topic. However, the explanations that the platform provided were always in the form of written text, which required students to have a certain level of literacy and reading comprehension skills. To overcome this challenge, DPL tools could provide explanations and feedback through multiple modalities, including oral explanations through videos or audio recordings, or images and other visuals to scaffold student understanding. Prior research has suggested that feedback provided through media including audio, video, or screencast recordings can help improve learners’ understanding when using DPL technologies (Henderson et al., 2019). Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms Discussion and Recommendations  33 References Aadil, Fareeha, Rabia Nzir, and Misbah Akhtar. Ferede, Bekalu, Jan Elen, Wim Van Petegem, 2021. “Investigating the Impact on Learning Adula B. Hunde, and Katie Goeman. 2021. Outcomes Through the Use of EdTech During “Determinants of Instructors’ Educational ICT Use Covid-19: Evidence from an RCT in the Punjab in Ethiopian Higher Education.” Education and Province of Pakistan.” Working Paper 4, EdTech Information Technologies 27: 917–936. https:// Hub. https://doi.org/10.53832/edtechhub.0067. doi.org/10.1007/s10639-021-10606-z. Abreu Malla, Laura S., J. Alfonso Aísa, Daniel Gove, Amber, and Peter Cvelich. 2011. Early Morales, Catherine Rodríguez, and Monica Reading: Igniting Education for All. A Report Y. Pagans. 2022. “¿Qué hemos aprendido by the Early Grade Learning Practice. de la educación a distancia en República Revised Edition. Research Triangle Park, NC: Dominicana durante la pandemia?” World Bank Research Triangle Institute. https://eric. Blog, April 12, 2022. https://blogs.worldbank. ed.gov/?id=ED520290. org/es/latinamerica/que-hemos-aprendido- de-la-educacion-distancia-en-republica- Government of the Dominican Republic. 2022. dominicana-durante-la. “Consejo Nacional de Educación aprueba uso de libros digitales en los centros educativos a Aldheleai, Yahya M., Roselan Baki, Zaidatun Tasir, nivel nacional.” Government of the Dominican and Waleed Alrahmi. 2019. “What Hinders the Republic (website), February 16, 2022. https:// Use of ICT Among Academic Staff at Yemen’s presidencia.gob.do/noticias/consejo-nacional- Public Universities?” International Journal of de-educacion-aprueba-uso-de-libros- Humanities and Innovation 2 (1): 7–12. https://doi. digitales-en-los-centros-educativos. org/10.33750/ijhi.v2i1.30. Grant, Peggy, and Dale Basye. 2014. Personalized Azevedo, João P., Marcela Gutierrez, Rafael de Learning: A Guide for Engaging Students with Hoyos, and Jaime Saavedra. 2021. “The Unequal Technology. Eugene, OR and Washington, Impacts of COVID-19 on Student Learning.” DC: International Society for Technology in Primary and Secondary Education During Education. COVID-19: 421–459. https://doi.org/10.1007/978- 3-030-81500-4_16. Groff, Jennifer S. 2017. Personalized Learning: The State of the Field and Future Aydin, Mehmet K., Mehmet Gürol, and Ruben Directions. Center for Curriculum Redesign.  Vanderlinde. 2016. “Evaluating ICT Integration in https://www.media.mit.edu/publications/ Turkish K–12 Schools through Teachers’ Views.” personalized-learning/. Eurasia Journal of Mathematics, Science and Technology Education 12 (4): 747–766. https://doi. Henderson, Michael, Michael Phillips, Tracii Ryan, org/10.12973/eurasia.2016.1227a. David Boud, Phillip Dawson, Elizabeth Molloy, and Paige Mahoney. 2019. “Conditions that Banerjee, Adhijit V., Shawn Cole, Esther Duflo, Enable Effective Feedback.” Higher Education and Leigh Linden. 2007. “Remedying Education: Research and Development 38 (7): 1401–1416. Evidence from Two Randomized Experiments https://doi.org/10.1080/07294360.2019.16578 in India.” Quarterly Journal of Economics 07. 122 (3): 1235–1264. https://doi.org/10.1162/ qjec.122.3.1235. Hennessy, Sara, Sophia D’Angelo, Nora McIntyre, Saalim Koomar, Adam Kreimeia, Lydia Cao, Barón, Juan D., Carmen M. Taveras, and Janinan Meaghan Brugha, and Asma Zubairi. 2022. C. Zúñiga. 2018. “Adaptive Technology to “Technology Use for Teacher Professional Help Improve Math Learning in the Dominican Development in Low- and Middle-Income Republic.” World Bank Blogs, December 19, Countries: A Systematic Review.” Computers 2018. https://blogs.worldbank.org/education/ and Education Open 3 (100080). https://doi. adaptive-technology-help-improve-math- org/10.1016/j.caeo.2022.100080. learning-dominican-republic. Holmes, Wayne, Stamatina Anastopoulou, Heike Basham, James D., Tracey E. Hall, Richard A. Schaumburg, and Manolis Mavrikis. 2018. Carter Jr., and William M. Stahl. 2016. Journal Technology-Enhanced Personalised Learning. of Special Education Technology 31 (3): 126–136. Stuttgart: Robert Bosch Stiftung. https:// https:/doi.org/10.1177/0162643416660835. s3-eu-west-1.amazonaws.com/www.studie- personalisiertes-lernen.de/en/TEPL_en.pdf. Bulger, Monica. 2016. “Personalized Learning: The Conversations We’re Not Having.” Working Ito, Hirotake, Keiko Kasai, and Makiko Nakamuro. Paper, 07.22.2016. Data and Society Research 2019. “Does Computer-Aided Instruction Improve Institute. https://datasociety.net/pubs/ecl/ Children’s Cognitive and Non-Cognitive Skills?” PersonalizedLearning_primer_2016.pdf. Evidence from Cambodia.” Discussion papers 19040, Research Institute of Economy, Trade D’Angelo, Sophia, Sara Hennessy, Adam Kreimeia, and Industry, Tokyo, Japan. https://papers.ssrn. Saalim Koomar, Lydia Cao, Nora McIntyre, com/sol3/papers.cfm?abstract_id=3834470 Meaghan Brugha, and Asma Zubairi. 2022. “Technology Use For Teacher Professional Koenka, Alison C., and Eric M. Anderman. 2019. Development in Low- and Middle-Income “Personalized Feedback as a Strategy for Countries: Recommendations for Policy from Improving Motivation and Performance among a Systematic Review.” Policy Brief, EdTech Hub. Middle School Students.” Middle School Journal https://doi.org/10.53832/edtechhub.0080. 50 (5): 15–22. https://doi.org/10.1080/00940771. 2019.1674768. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms References 34 References Kucirkova, Natalia, Liddy Gerard, and Marcia C. Muralidharan, Karthik, Adhijeet Singh, and Linn. 2021. “Designing Personalised Instruction: A Alejandro J. Ganimian. 2019. “Disrupting Research and Design Framework.” British Journal Education? Experimental Evidence on of Educational Technologies 52 (5): 1839–1861. Technology-Aided Instruction in India.” American https://doi.org/10.1111/bjet.13119. Economic Review 109 (4): 1426–1460. https://doi. org/10.1257/aer.20171112. Lara Ibarra, Gabriel, Niccolò Comini, and Natalija Gelvanovska-Garcia. 2022. “Universal, Narciss, Susanne, Sergey Sosnovsky, Lenka Affordable, and Reliable Internet Connectivity Schnaubert, Eric Andrès, Anja Eichelmann, is a Key Ingredient for Inclusive Recovery.” George Goguadze, and Erica Melis. 2014. World Bank Blogs, December 21, 2022. https:// “Exploring Feedback and Student Characteristics blogs.worldbank.org/latinamerica/universal- Relevant for Personalizing Feedback Strategies.” affordable-and-reliable-internet-connectivity- Computers and Education 71: 56–76. https://doi. key-ingredient-inclusive. org/10.1016/j.compedu.2013.09.011. Lim, Lisa-Angelique, Shane Dawson, Dragan Natriello, Gary, ed. 2013. Adaptive Educational Gašević, Srećko Joksimović, Anthea Fudge, Technologies: Tools for Learning and for Learning Abelardo Pardo, and Sheridan Gentili. 2020. about Learning. Washington, DC: National “Students’ Sense-Making of Personalised Academy of Education. Feedback Based on Learning Analytics.” OECD (Organisation of Economic Co-operation Australasian Journal of Educational Technology and Development). 2019. Programme for 36 (6): 15–33. https://doi.org/10.14742/ajet.6370. International Student Assessment (PISA) Results Maier, Uwe, and Christian Klotz. 2022. Personalized from PISA 2018. Paris: OECD. Feedback in Digital Learning Environments: Ouyang, Fan, and Pengcheng Jiao. 2021. “Artificial Classifications Framework and Literature Intelligence in Education: The Three Paradigms.” Review.” Computers and Education: Artificial Computers and Education: Artificial Intelligence Intelligence 3 (2022): 100080. https://doi. 2 (2021): 100020. https://doi.org/10.1016/j. org/10.1016/j.caeai.2022.100080. caeai.2021.100020. Major, Louis, and Gill A. Francis. 2020. Technology- Perera, Marcelo, and Diego Aboal. 2019. “The Supported Personalised Learning: A Rapid Impact of a Mathematics Computer-Assisted Evidence Review. EdTech Hub Rapid Evidence Learning Platform on Students’ Mathematics Review. https://docs.edtechhub.org/lib/ Test Scores.” Working Paper Series, Maastricht A2II5ZV7. Economic and Social Research Institute Major, Louis, Gill A. Francis, and Maria Tsapali. on Innovation and Technology, UNU-MERIT, 2021. “The Effectiveness of Technology- Maastricht, The Netherlands. https://ideas. Supported Personalised Learning in Low- and repec.org/p/unm/unumer/2019007.html. Middle-Income Countries: A Meta-Analysis.” Robinson, Carol, and Judy Sebba. 2010. British Journal of Educational Technology 52 (5): “Personalising Learning Through the Use 1–30. https://doi.org/10.1111/bjet.13116. of Technology.” Computers and Education Maseleno, Andino, Noraisikin Sabani, Miftachul 54 (3): 767–775. https://doi.org/10.1016/j. Huda, Roselee Ahmad, Kamarul A. Jasmi, and compedu.2009.09.021. Bushrah Basiron. 2018. “Demystifying Learning Shemshack, Atikah, and Jonathan M. Spector. Analytics in Personalised Learning.” International 2020. “A Systematic Literature Review of Journal of Engineering and Technology 7 Personalized Learning Terms.” Smart Learning (3): 1124–1129. https://doi.org/10.14419/ijet. Environments 7 (33). https://doi.org/10.1186/ v7i3.9789. s40561-020-00140-9. MINERD (Ministerio de Educatión de la República Tejada, Franklin. 2021. “La UNESCO destaca avaces Dominicana). 2016. Diseño Curricular Nivel en los logros de aprendizaje de República Secundario. Santo Domingo: Dominican Republic. Dominicana.” Dominican Republic United https://www.ministeriodeeducacion.gob. Nations (website), November 30, 2021. https:// do/docs/direccion-general-de-curriculo/ dominicanrepublic.un.org/es/160898-la- RtcE-diseno-curricular-del-nivel-secundario- unesco-destaca-avances-en-los-logros-de- primer-ciclopdf.pdf. aprendizaje-de-república-dominicana. MINERD. 2020. “MINERD coninuará entrega de UNICEF (United Nations Children’s Fund). 2022a. computadoras a estudiantes y maestros.” Trends in Digital Personalized Learning in Low- Ministry of Education (website), May 31, 2020. and Middle-Income Countries. New York: UNICEF. https://www.ministeriodeeducacion.gob.do/ https://www.unicef.org/globalinsight/reports/ omunicaciones/noticias/minerd-continuara- trends-digital-personalized-learning. entrega-de-computadoras-a-estudiantes-y- maestros. UNICEF. 2022b. Where are We On Education Recovery? New York: UNICEF. https://www. MINERD. 2021. Plan Estratégico. Santo Domingo: unicef.org/reports/where-are-we-education- Dominican Republic. https://www. recovery. ministeriodeeducacion.gob.do/transparencia/ media/plan-estrategico-institucional-pei/ planificacion-estrategica-institucional/toJ- rrf-plan-estrategico-2021-2024-peipdfpdf.pdf. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms References 35 References UNESCO (United Nations Educational, Scientific and Cultural Organization). 2021. The State of the Global Education Crisis: A Path to Recovery. Paris: UNESCO. https://www.worldbank.org/en/ topic/education/publication/the-state-of-the- global-education-crisis-a-path-to-recovery. Upadhyay, Arjun, James Shoobridge, and Caitlin M. Coflan. 2021. Effective Use of EdTech for Remedial Learning Programmes: Considerations for Mongolia and other LMICS. EdTech Hub. https://doi.org/10.5281/zenodo.3958080. Van Schoors, Rani, Jan Elen, Annelies Raes, Stefanie Vanbecelaere, and Fien Depaepe. 2022. “The Charm or Chasm of Digital Personalized Learning in Education: Teachers’ Reported Use, Perceptions and Expectations. TechTrends 67: 315–330. https://doi.org/10.1007/ s11528-022-00802-0. Vanderlinde, Ruben, and Johan Van Braak. 2010. “Implementing an ICT Curriculum in a Decentrailsed Policy Context: Description of ICT Practices in Three Flemish Primary Schools.” British Journal of Education Technology 41 (6): E139–E141. https://doi.org/10.1111/j.1467- 8535.2010.01111.x. World Bank. 2019. Dominican Republic: Learning Poverty Brief. Washington, DC: World Bank. https://thedocs.worldbank.org/en/ doc/212111571223351532-0090022019/original/ LACLCC3CDOMLPBRIEF.pdf. World Bank. 2022a. Guide for Learning and Recovery and Acceleration: Using the RAPID Framework to Address COVID-19 Learning Losses and Build Forward Better. Washington, DC: World Bank . https://documents.worldbank. org/en/publication/documents-reports/ documentdetail/099063023145523057/ p17857701e1a0a0e008dc00c2c22f619135. World Bank. 2022b. Knowledge Pack: Technologies for Personalized and Adaptive Learning. Washington, DC: World Bank. https://documents.worldbank.org/ en/publication/documents-reports/ documentdetail/099120004132256958/ p1742520d80f840b5092f20b334bf33e41b. Lessons on Implementation: Stakeholders’ Experiences with Digital Personalized Learning Platforms References 36 © 2024 International Bank for Reconstruction and Development / The World Bank