Using Education Technology to Improve K-12 Student Learning in East Asia Pacific: Promises and Limitations © 2023 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpreta- tions, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy, completeness, or currency of the data included in this work and does not assume responsibility for any er- rors, omissions, or discrepancies in the information, or liability with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the en- dorsement or acceptance of such boundaries. Nothing herein shall constitute or be construed or considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions—The material in this work is subject to copyright. Because The World Bank encour- ages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. This research was financed in part by the China-World Bank Partnership Facility. Attribution—Noah Yarrow, Cody Abbey, Sharon Shen, Kevin Alyono (co-authors). With thanks and appreciation for Pinyi Chen, Neil Butcher, Mike Michalec, Chuyu Song, Lauri Pynnonen (contributors) and Andrew Ma- son, Mike Trucano and Yue Ma (reviewers). This Operations Note does not constitute World Bank Policy or requirements and is not mandatory. It is a deliberative internal document that is updated peri- odically and not intended for distribution outside of the World Bank. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@ worldbank.org. Cover photo credits, Alamy Stock Photo and World Bank Flckr. Designed by Veronica Gadea. Contents Abstract1 Definition of Terms 2 Section 1 Limits and challenges: EdTech as a means for post- COVID-19 learning recovery 3 1.1 Intervention scale matters for drawing policy-relevant lessons 5 1.2 “The Implementor Effect”: education interventions by NGOs versus governments 8 1.3 Taking implementation heterogeneity into account: the importance of uptake and dosage 10 Section 2 What happened? Pre-pandemic learning inequalities and learning losses during the COVID-19 pandemic 13 2.1 Overview of the learning crisis before and during the COVID-19 pandemic13 2.2 There is a “perception gap” between decision makers and the experience of students and families with EdTech 18 Section 3 Moving forward: How EdTech can support learning for all in EAP 21 3.1 Recommendations 21 Section 4 Conclusion 31 References 33 Annex 1 41 Abstract We use global and regional data to show that it is pos- not all families and schools are able to pay for, access, sible to use EdTech to improve student learning in EAP. and use it effectively. In this companion paper to the We present evidence that the broadcast/dual teacher EAP regional flagship “Fixing the Foundation: Teachers model often supports student learning gains, while other and Basic Education in East Asia and Pacific” (Afkar et approaches, including assistive EdTech, show promise. al, 2023), we present the results of a regional survey Others, such as e-readers, remote teacher-training and of middle-income countries showing that, contrary to AI interventions have yet to demonstrate positive im- available evidence, most education decision makers pacts on student learning at scale in the EAP context. believe that EdTech was effective in supporting student Based on evidence from the EAP region and globally, we learning during COVID-19 school closures. We recom- show that as the scale of EdTech interventions increases, mend several evidence-based EdTech interventions in the effect on learning generally decreases. The largest EAP including the “broadcast” or dual–teacher model, impacts tend to come from smaller-scale interventions and call for improved approaches for future research conducted by non-governmental institutions rather than that consider scale, dosage and heterogeneity of impact large-scale interventions by governments. We find that to evaluate EdTech interventions. as the use of EdTech expands in the EAP region, it tends to increase existing learning inequalities, since 1 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Definition of Terms In this report, our primary focus is the use of Educational Technology (EdTech) among K-12 student populations in the East Asia Pacific (EAP) region.1 We define EdTech as the use of hardware, software, digital content, data, and information systems in education, following “Reimagining Human Connections: Technology and Innovation in Education” (World Bank, 2020). Students and teachers use different devices to communicate and share information in voice, text, and video formats, and they use platforms established by governments, non-profits, and for-profit firms. Within this diversity of use, we consider radio and TV-based instruction to be distance education but not specifically EdTech, while we consider teaching and learning using messages on a one-to-one or group chat function to be EdTech use. As discussed in the paper, these distinctions become less and less clear as tech becomes more and more a part of everyday life, including education processes. Another concept is the uptake and dosage of education interventions. While these two terms are generally associated with medical contexts, we define “uptake” as whether the intervention is used by the teacher and/or students (Wilichowski & Cobo, 2021; World Bank, 2020), and “dosage” refers to the amount of instruction provided, which typically includes the number of intervention sessions and the length of each session (Mason & Smith, 2020). 1 This is with the exception of one study conducted in Tonga (Macdonald et al., 2017), which discusses the effect of an educational intervention on pre-K-12 students. 2 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Section 1 Limits and challenges: EdTech as a means for post- COVID-19 learning recovery Available evidence indicates that EdTech has promise in specific settings, but whether it can improve student learning at scale is harder to say. During the COVID-19 pandemic, high-income countries experienced an average learning loss equivalent to one-third to one-half of a typical aca- demic year (Patrinos et al., 2022). Countries with shorter school closures and faster implementation of distance-learning programs generally faced lower learning loss levels (Patrinos et al., 2022; Schady et al., 2023). A key lesson from prolonged school closures during the pandemic is the need for more adaptive and equitable remote learning systems in EAP countries, which can be utilized during future crises from pandemics, climate change and other risks. In this section, we present evidence to help answer the question of how effective will EdTech be in supporting recovery from the COVID-19 pandemic and addressing the human capital gap in the EAP region? We have attempted to compile all published rigorous impact evaluations of education interventions in EAP that include measurements of student learning outcomes. Altogether, we found 52 education studies with a total of 109 effect sizes on student learning in Middle Income Countries in the EAP region before, during, and slightly after the COVID-19 pandemic.2 Among these, 24 are EdTech inter- ventions, with 19 focusing on a specific mode of EdTech: Computer-Assisted Learning interventions.3 Examples of other EdTech studies in the dataset include an intervention providing digital textbooks to primary school students for use in social studies and science in the Republic of Korea (Lee et al., 2022); IT investment and recorded instruction in rural Chinese middle schools (Bianchi et al., 2022); recorded class videos for students at home during COVID-19-related school closures (Clark 2 We would like to thank David Evans and Fei Yuan for sharing their database, as well as Daniel Rodriguez-Segura for sharing his database. If you are aware of additional evaluations not included here, please let us know. 3 Computer assisted learning (CAL) refers to EdTech interventions in which students learn independently (i.e., engage in self-directed learning) with the assistance of a computer software program, though teachers or monitors are often present to provide technical support, encouragement and oversight. 3 Photo: imtmphoto / Alamy Stock Photo et al., 2021); and the “Dual–teacher program” in as “has tech” (rather than “EdTech”), since tech China for rural secondary students, allowing them is part of the intervention but not the focus of the to watch live lectures from urban elite schools treatment. We note that, given its role in society, while receiving guidance from local teachers it is likely that technology is present to some (Li et al., 2023). extent in all education interventions, even if not explicitly specified by the evaluation authors. The non-tech studies in the database evaluate a broad range of education interventions aimed Within the 52 education studies that measure at improving student learning outcomes, such as student learning impacts in EAP MICs, we ob- pre-primary reading instruction (Abeberese et al., serve a wide variation in the effect sizes of ed- 2011), community-based playgroups (Brinkman ucational interventions. EdTech studies exhibit et al., 2017a; Macdonald et al., 2017), teacher the most extensive variation in impact. Although training (Loyalka et al., 2013; Fuje and Tandon, EdTech interventions (depicted as light blue dots 2018; L. Zhang et al., 2013), student scholarships in Figure 1) have demonstrated negative, null, or financial incentives for teachers (Barrera-Os- and low-impact outcomes, they also account for orio & Filmer, 2016; X. Chen et al., 2013; Filmer the majority of impacts in the top quartile of the & Schady, 2009; Loyalka et al., 2019a; Yi et al., distribution. Furthermore, the largest effect size 2015), and interventions aimed at improving in our dataset is from an EdTech intervention student health outcomes linked to poor school (0.978 SD). At first glance, this visualization may performance, such as reducing rates of anemia lead one to believe that EdTech represents the or myopia (Du et al., 2022; Wong et al., 2014; most promising avenue for allocating marginal Kleiman-Weiner et al., 2013; Nie et al., 2020; investment dollars to enhance student learning. Sylvia et al., 2013). In addition, there is one study However, an in-depth examination of the charac- involving the use of a camera and community teristics of each study reveals a more complex engagement to improve teacher attendance reality. (Gaduh et al., 2020) and another employing a video-based teacher development tool (Chen et al., 2020). These types of studies are classified 4 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Figure 1 Comparison of effect sizes (adjusted) with confidence intervals for EdTech, Has Tech, and Non-Tech educational interventions in the EAP region across prepandemic, pandemic, and post-pandemic Has tech EdTech Non-tech -0.50 0.00 0.50 1.00 1.50 Effect size (adjusted, in S.D.) Source: World Bank, EAP Education Interventions, 2023. See Annex 1 for a complete list of studies and effect sizes (adjusted). 1.1 Intervention scale matters for drawing policy-relevant lessons Most studies in our sample are pilot studies, of 52) of the studies in our EAP sample report- meaning that they are not part of larger ed- ed the overall intervention size, focusing on the ucation interventions or policy actions, and effects of education programs at the national or therefore operate within only a small popu- regional level. Conversely, 81 percent (42 out of lation of students, teachers and schools. As a 52) did not disclose the total treatment size and result, the “total treatment size” is frequently not appear to be small-scale pilot studies, leading reported by researchers, since the “treatment to a significant research literature gap. sample size” (the number of students, for exam- ple, receiving the intervention) is the same as To account for the absence of complete inter- the total treatment size. Only 19 percent (10 out vention size data, we employ treatment sample 5 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations size as an imperfect proxy for total treatment size. The average treatment sample size in our dataset is only 2,145 students, and we have no treatment sample involving more than 25,000 students. While we recognize the costs and logistical challenges associated with conducting high-quality, large-scale studies, it is essential to provide data with high external validity and scalability potential to enable education policy makers to rely on data-driven insights. Effect size appears to decline with increasing intervention scale proxied by treatment sample size because implementation at scale is very different from implementing a pilot program (Fig- ure 2 and Table 1). Based on discussions with authors of some of the studies, the high levels of expertise and monitoring provided during research with studies involving hundreds or thousands of students, as well as technical support and encouragement to teachers and administrators, may lead to higher levels of treatment from teachers and students. When these smaller trials are ex- panded without the same quality and quantity of support and attention, competing priorities and business-as-usual practices may negatively affect both implementation and impact. This negative association holds regardless of whether the intervention is EdTech, non-tech, or involves varying degrees of technology in the learning environment. Figure 2 Relationships between treatment sample size (log) and effect size (adjusted) for EAP studies 1 Effect size (adjusted, in S.D.) .5 0 4 6 8 10 Treatment sample size (log) Effect size (adjusted) Fitted values Source: World Bank, EAP Education Interventions, 2023. 6 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Table 1 Continuum of technology from no/low to high tech (1) (2) (3) (4) Tech EAP Effect Size EAP Effect Size EAP Effect Size EAP Effect Size (SE) (SE) (SE) (SE) Log treatment -.0348521* -.050755 -.0369909*** -.1364512** sample size (.017592) (.1552634) (.0086817) (.0490839) Log treatment .0011235 .0076679* sample size ^2 (.0108972) (.0037248) .3843002 ** .4394985 .4053289*** .7122272*** Constant (.1280345) (.5506411) (.0611044) (.1610507) Adjusted 0.0264 0.0173 0.0299 0.0355 R-squared Note: *p<0.05, **p<0.01, ***p<0.001 Source: World Bank, EAP Education Interventions, 2023. Sample of global studies is drawn from David K. Evans and Fei Yuan, 2022, “How Big Are Effect Sizes in International Education Studies?,” Educational Evaluation and Policy Analysis 44(3), https://doi.org/10.3102/01623737221079646 Based on the evidence we have available, there implications for the advice we can confidently are few studies that would allow us to predict give to governments on the expected learning with confidence the impact of any nationwide impacts of implementing specific EdTech and education intervention, including an EdTech other learning programs nationally. In light of these intervention. We cannot confidently predict the findings, we believe it is crucial for researchers learning impact of even a subnational scale-up. in the EAP region to focus on conducting more This issue is not unique to EdTech or EAP and larger-scale effectiveness studies that provide is a major concern in the education sector and reliable insights into the real-world implications beyond.4 In this sense, many if not most of the of implementing specific EdTech and learning education studies in our EAP dataset are closer programs, ultimately guiding decision-making to studies of efficacy rather than effectiveness. for meaningful, nationwide policies in education. Efficacy trials (explanatory trials) determine whether an intervention produces the expect- ed result under ideal circumstances, whereas effectiveness trials (pragmatic trials) measure the degree of beneficial effect under “real world” clinical settings. This has large, and limiting, 4 See for example, Lant Pritchett and Justin Sandefur (2013) “Context Matters for Size: Why External Validity Claims and Development Practice Don’t Mix.” CGD Working Paper 336. Washington, DC: Center for Global Development ; Jonathan Stern, Matthew Jukes, Benjamin Piper et al. (2021), Learning At Scale Interim Report; Vivalt (2020), “How much can we generalize from impact evaluations?”. Specifically, in this second report from Stern et al. (2021), the authors propose a definition for working at scale in education, which is to implement in at least 500 schools and have universal or near-universal coverage in at least two sub-national areas. In their global review, they found eight programs that met this criteria and for which student-learning impact data were available based on an evaluation. 7 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations 1.2 “The Implementor Effect”: education interventions by NGOs versus governments The entity implementing the activity affects the effectiveness of studies (an “implementor effect”), in addition to the scale of intervention. In our EAP dataset, studies yielding the largest impacts tend to be conducted by NGOs or other non-state actors such as research institutions or thinktanks (Figure 3). Within the EAP sample, among the top 10 percent of effect sizes, 8 out of 11 (or 73 per- cent) are interventions implemented by NGOs, while only three of these large-effect interventions (27 percent) are carried out by governments. Considering effect sizes greater than or equal to 0.16 SD,5 27 out of 36 (or 75 percent) are associated with NGO implementation, as opposed to 25 per- cent with government implementation. This pattern is also observed in the global sample of Evans and Yuan (2020). Among the top 10 percent effect sizes, 62.50 percent are jointly implemented by NGOs and governments, 32.81 percent solely by NGOs, and 4.69 percent exclusively by govern- ments. Meanwhile, among the effect sizes equal to or greater than 0.16 SD, 45.19 percent are dual implementation, 39.90 percent with NGO implementation, and only 14.90 percent with government implementation. We call this the “implementor effect.” While non-governmental interventions tend to be more effective, they are also smaller in sample size. Therefore, it is difficult to say to what extent their higher effectiveness is related to the implementor or the treatment sample size. Figure 3 Comparison of effect sizes and treatment sample size (log) associated with NGOs versus government in EAP studies 1.00 Government NGO 0.80 Effect size (adjusted, in S.D.) 0.60 0.40 Top 10% 0.20 0.16 SD 0.00 -0.20 0.00 2.00 4.00 6.00 8.00 10.00 12.00 Treatment sample size (log) Source: World Bank, EAP Education Interventions, 2023. 5 Reported to be the median effect size of education interventions across low- and middle income countries in Evans and Yuan, 2022. 8 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations In our EAP dataset, a study conducted by Mo et al. (2020) provides a clear illustration of the “imple- mentor effect.” The study investigated the extent to which variations in implementation agencies influence program effectiveness. A total of 120 rural Chinese schools employed the same Com- puter Assisted Learning (CAL) treatment to facilitate English-language learning. The treatment was implemented by an NGO in one group of schools, by the government in another group, and a third group served as a control. The authors discovered that, in contrast to the control condition and unlike the NGO program, the government program did not improve student achievement in English. In fact, it led to a decrease in student performance on average (effect size = -0.07 SD), although the negative impact was not statistically significant (Figure 4). This outcome was attributed to the fact that teachers in the government arm were more likely to substitute the EdTech intervention for regular instruction, which was contrary to the required protocol. The authors hypothesized that this might be due to government officials being less likely than NGO implementers to directly monitor program progress (Mo et al., 2020). Figure 4 Impact of in-school CAL on English scores of grade-4 students in rural China with different implementation agencies from Mo et al., 2020 Implemented by NGO 0.16 Implemented by government -0.07 -0.10 -0.05 0.00 0.05 0.10 0.15 0.20 Effect size (adjusted, in S.D.) Source: Mo, Di, Bai, Yu, Shi, Yaojiang, Abbey, Cody, Zhang, Linxiu, Rozelle, Scott, & Loyalka, Prashant. “Institutions, imple- mentation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China,” Journal of Development Economics Vol. 146, 2020: 102487. .org/10.1016/j.jdeveco.2020.102487. Apart from the propensity to implement interventions on a smaller scale than governments, there are other factors that may contribute to NGOs and other non-state actors being more likely to implement interventions with large impacts. One plausible explanation is that, in certain instances, NGOs possess greater skill and experience in effectively implementing specific interventions. These organizations, which often devise the interventions that they carry out (such as the NGO respon- sible for the CAL curriculum implementation in China), typically have the technical expertise and practical knowledge required for designing evaluations and implementing EdTech interventions. Furthermore, NGOs, which generally manage a smaller number of externally-funded projects and thus have a high incentive for the success of their interventions in order to generate future finan- cial support, may exhibit a stronger commitment to high-quality implementation than government entities. The latter may oversee a number of projects simultaneously and may be less concerned about securing funding. 9 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations “The available evidence tends to come from smaller-size interventions conducted by NGOs, with impacts that governments are unlikely to achieve when implementing at national scale. These findings imply that the impact evaluation revolution of the past decades is not particularly helpful as a guide for addressing the learning inequalities in EAP.” Our findings imply that the impact evaluation tions such as “does EdTech work” or “what is the revolution of the past decades is not partic- most effective education intervention”, and the ularly helpful as a guide for addressing the answer in many cases is that we simply do not learning inequalities in EAP. Specifically, the know. The available evidence tends to come from limitations of scale and implementor effects are smaller-size interventions conducted by NGOs, essential to consider when selecting education with impacts that governments are unlikely to interventions. Decision makers may ask ques- achieve when implementing at national scale. 1.3 Taking implementation heterogeneity into account: the importance of uptake and dosage Average effect sizes do not convey information on the extent to which students or teachers participated in the intervention. Average effect sizes also hide whether the intervention’s magnitude of impact differed across students by their characteristics, which is crucial information for understanding whether a certain intervention increases or narrows the learning gap between students.” Heterogeneous effects (by uptake, dosage, stu- rows the learning gap between students. dent characteristics, etc.) also matter, but are rarely reported by both EAP or global studies. Another EdTech intervention from the region il- Average effect sizes which are most commonly lustrates the importance of uptake and dosage reported do not convey information on the extent when considering the effect size of interventions to which students or teachers participated in the (Ma et al., 2020). The average effect of this CAL intervention (uptake and dosage), particularly if intervention to support math learning in Taiwan, their participation was less or more than what China, was zero, as compliance was low overall. was specified in the study’s protocol. Average However, there was significant improvement for effect sizes also hide whether the intervention’s the most active students in the treatment group, magnitude of impact differed across students by with impacts increasing as usage increased up their characteristics (e.g., socio-economic status, to 23 minutes per week (Figure 5). Since many baseline academic performance, gender, etc.), education interventions, including EdTech ap- which is crucial information for understanding proaches, depend heavily on teacher adoption whether certain an intervention increases or nar- and integration, these issues of uptake and dos- 10 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations age are important to consider when thinking about average effect sizes. Reporting a “minimum dosage” required for impact in evaluations could be a helpful start. The same pattern in variable impacts based on use also applies to non-tech interventions. The study conducted by Du et al. (2022) shows how the effect size on student standardized math scores varies depending on whether or not they comply with wearing eyeglasses at the endline. While providing subsidized eyeglasses improved the average test score of treatment students by 0.062 SD, the math scores of those who actually wore the eyeglasses increased by 0.918 SD. Unfortunately, many studies do not report the actual compliance and dosage of an intervention and generally only report the planned compliance and dosage. Thus, important information about why an intervention may or may not have an impact (i.e., is it because the intervention itself is not effective, or because students/teachers did not comply with it?) is often unavailable. Figure 5 Impact of in-school CAL on math scores of grade 4 students in Taiwan, China, with different usage intensities 18 min/week 0.152 20 min/week 0.181 21 min/week 0.198 23 min/week 0.225 0 0.05 0.1 0.15 0.2 0.25 Effect size (adjusted, in S.D.) Source: Ma, Yue, Cody Abbey, Derek Hu, Oliver Lee, Weiting Hung, Xinwu Zhang, Chiayuan Chang, Chyi-In Wu, and Scott Rozelle. “The Impact of Computer Assisted Learning on Rural Taiwanese Children: Evidence from a Randomized Experiment.” REAP Working Paper, 2020. https://fsi-live.s3.us-west-1.amazonaws.com/s3fs-public/taiwan_o-cal_working_paper_2020oct.pdf. Taking effect heterogeneity into account based on student characteristics. Along with consid- ering uptake and dosage, a study by Ma et al. (2020) exemplifies the importance of assessing the heterogeneity in effect sizes across students with different characteristics (also known as heteroge- neous effects), as well as the mechanisms behind those impacts (i.e., are these differences due to different dosage or simply because the intervention worked better for some students than others?). This information is often masked behind average effect sizes in studies, as noted in Vivalt, 2020, Evans & Yuan, 2022, and Pritchett & Sandefur, 2013. In the Ma et al. (2020) study, students with lower baseline math scores and male students were less likely to use the software, while teacher characteristics also influenced usage, among other variables. 11 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations As interventions grow larger and barriers to implementation increase, their potential to have dif- ferential impacts across students—in addition to the average effect size of the intervention—also grows larger. These startling findings shed light on the danger of a “one-size-fits-all” education program that compromises the interest of subgroups of participants for the overall positive impact. One study that had a strong degree of heterogeneity evaluated a preschool education intervention in rural Indonesia, and it found positive impacts on the developmental outcomes for poor children (0.20 SD) but negative effects on the developmental outcomes for non-poor children (-0.15 SD) (Brinkman et al., 2017). There are also examples of studies in which only the more privileged or high-performing students gained, while their worse-off peers did not have improved outcomes, such as one intervention in Kenya that provided textbooks to students (Glewwe et al., 2009). Besides this caveat, another important caution about basing decisions to implement certain interventions solely on average effect sizes is that, while an intervention may result in overall positive impacts for the primary target outcome (i.e., academic performance), it may also have negative externalities on other important outcomes. There is little available empirical evidence on this, but a gamified CAL platform in Chile had positive impacts on student learning in math (0.27 SD) but resulted in more severe symptoms of math anxiety and a lower desire among students to collaborate (Araya et al., 2019). Several aspects deserve attention from researchers and policy makers in the field of EdTech to improve student learning. First and foremost, partnerships between NGOs, research institutions, and governments are crucial for leveraging the strengths of each implementor. These partnerships can start to address the challenges of scalability and effectiveness of EdTech for student learning. The technical expertise of NGOs and research partners, who often design and implement success- ful pilot programs, can be complemented at the initial design phase by government involvement. These partnership can help specify design approaches and costs that have the potential for future scale up. Second, future impact evaluations should report actual uptake and dosage of EdTech interventions. Implementers should use comparable measures to ensure comparability, and carefully measure compliance with the intervention protocol to improve insight into varying treatment impacts. Finally, there is a growing need to better understand the heterogeneity of effects. As interventions scale to broader populations, we should consider student characteristics such as socio-economic background, target outcomes, gender, disability etc. and the heterogeneous effects of different interventions on these subgroups. Future impact evaluations can be improved by including analyses of heterogeneity to inform more customized intervention designs, and a greater diversity of inter- vention programs to improve learning for all students and meet a broader range of student needs. 12 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Section 2 What happened? Pre-pandemic learning inequalities and learning losses during the COVID-19 pandemic 2.1 Overview of the learning crisis before and during the COVID-19 pandemic A learning crisis persists in middle-income of learning loss (Patrinos et al., 2022). This is countries (MICs) in the EAP region (Bridging significantly higher than the average learning the Learning Gap, World Bank, forthcoming). loss (0.17 SD) identified by a recent global sys- Multiple factors contribute to learning poverty, tematic review from the World Bank (Patrinos including lack of access to highly qualified and et al., 2022). A recent World Bank report on motivated teachers, inadequate school infrastruc- learning loss in Cambodia found that student ture including running water and electricity, and assessment scores declined by 42.8 points in poorly designed academic curricula (Bau & Das, Khmer and 56 points in math between 2016 and 2020; Chand et al., 2021; Duarte et al., 2011). Evi- 2021, representing substantial losses of 8.4 and dence from existing studies suggests substantial 11.3 percent, respectively (Bhatta et al., 2022). COVID-19-related learning losses across several In Riau province, Indonesia, there was a 40 per- MICs in the EAP region. For instance, researchers cent reduction in the number of Grade 2 and 3 found a learning loss of 0.22 standard deviations students who could read and comprehend text (SD) in China during a seven-week school clo- in 2021 compared with 2018 (Molato-Gayares, sure, which equates to over half a school year 2022), while nationally, Grade 4 students in 2023 13 Photo: imageBROKER.com GmbH & Co. KG / Alamy Stock Photo have lost 11.2 months equivalent of math skills and 10.8 months equivalent of language skills in comparison with Grade 4 students in 2019 (World Bank, forthcoming) In response to the pandemic school closures, many EAP countries deployed EdTech to support student learning and to promote continued student engagement with schooling and retain students once schools reopened. However, striking disparities existed in students’ access to technologies before the COVID-19 pandemic, especially among the poorest populations in low- and middle-in- come countries of EAP. In Indonesia, for example, less than 40 percent of the students from the bottom income group had access to an internet connection. Only around 20 percent of those from the bottom income group could access a computer that might be used for online learning (Figure 6). Figure 6 Important inequalities existed pre-COVID-19 pandemic in student access to internet and computers A. Percent of students that have access to an B. Percent of students that have internet link, by income group access to a computer they can use for online learning, by income group Brunei Darussalam China (B-S-J-Z) Indonesia Malaysia Philippines Thailand 0 20 40 60 80 100 0 20 40 60 80 100 Percent Percent Bottom Quintile 2nd Quintile Middle Quintle 4th Quintle Top Quintile Source: World Bank staff calculations, using 2018 PISA data. 14 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Low levels of engagement with learning during school closures overall were compounded by unequal access to EdTech. Surveyed EAP countries saw notable disparities between urban and rural regions in terms of the share of households with children participating in online or mobile learning (Figure 7). Vietnam was the only country where more than half of its students were engaged in remote learning, as 66 percent of urban and 53 percent of rural households reported their children’s participation in online or mobile learning activities. Slightly less than half of the urban respondents in Indonesia (44 percent) and the Philippines (41 percent) reported engagement in online or mobile learning activities when schools were closed, whereas their rural counterparts had significantly lower usage (36 percent for rural Indonesia, 25 percent for rural Philippines). In particular, Lao PDR (15 percent urban, 5 percent rural), Mongolia (12 percent urban, 10 percent rural), and Myanmar (6 percent urban, 1 percent rural) exhibited extremely low levels of online or mobile learning for all students during survey rounds in 2020 and 2021, though urban levels of engagement were much higher. These findings suggest that in addition to high levels of learning loss overall, unequal access to technologies translates into higher learning loss for students from poorer socio-economic backgrounds. Figure 7 Percent of households with children engaged in online or mobile learning activities during weeks of COVID-19-related school closures, by country and sub-region 100 80 66 60 53 Percent 44 41 40 36 25 22 20 15 12 12 10 5 6 1 0 May Oct Dec Mar May Jun Nov Jun Jul Apr May May Dec Oct Aug Dec Jun 2020 2020 2020 2020 2020 2020 2020 2021 2021 2021 2021 2020 2020 2020 2020 2020 2020 ia a r nes bod nes ia PDR goli nma ippi tnam Cam Indo Lao Mon Mya Phil Vie Source: World Bank staff calculations, based on HFPS data and UNICEF’s “COVID-19 and School Closures” report. Even for those who were able to access learning households were more likely to engage in little resources during school closures, the amount of or no online learning (31.4 percent) compared time spent learning was much higher for children with those from households in the top 20 per- in wealthier households compared with children cent income tercile (17.1 percent). Only 17 percent in lower-income households (Figure 8). In the of the wealthiest Indonesian children attended Philippines, the number of children attending one hour or less of online learning, while more 90 percent or more of online classes was twice than 30 percent of children from the middle- and as high (56.7 percent) for children in the highest lowest-income households attended one hour income level as in the lowest income level (27.1 or less per week. percent). In Indonesia, children in lower-income 15 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Figure 8 Children in wealthier households were more likely to be engaged in online or mobile learning A. Remote learning class attendance in the B. Hours spent on online learning Philippines, by income group activities in Indonesia, by income group 100% 100% 12.1 9.2 10.3 90% 27.1 8.0 15.7 8.7 80% 35.8 80% 56.7 15.5 70% 19.9 28.8 20.7 60% 60% 14.8 50% 34.2 19.5 23.6 40% 40% 12.2 20.5 34.0 30% 11.7 16.5 20% 15.0 20% 10.5 30.8 31.4 10% 13.9 17.1 17.1 0% 8.9 0% Top 20% Middle 40% Bottom 40% Top 20% Middle 40% Bottom 40% Attended 0-25% of classes Attended 25-50% of classes 1 hr or less 2 hrs 3 hrs 4 hrs 5 hrs or more Attended 50-75% of classes Attended 75-90% of classes Attended 90-100% of classes Source: World Bank staff calculations, using HFPS data. “The gap in remote learning participation and unequal dosage implies that, despite the efforts of governments, parents, teachers, and the private sector, inequity in student learning outcomes was exacerbated by the transition to online learning during COVID-19- related school closures.” The gap in remote learning participation and al., 2020; Grewenig et al., 2020; Biswas et al., unequal dosage (time spent on online learning) 2020). This global challenge is referred to as the implies that, despite the efforts of governments, “Matthew effect,” whereby those who already parents, teachers, and the private sector, inequity in enjoy wealth, education, and prior exposure to student learning outcomes was exacerbated by the technology are most likely to benefit from new transition to online learning during COVID-19-related technologies (Trucano, 2013). school closures. Following Yarrow et al. (2022), these mirror the findings from more extensively Disparities were also present in the strategies studied, high-income countries in other regions adopted by families in EAP countries to mitigate the during COVID-19-related school closures, as a learning losses of their children following school widening gap in learning outcomes emerged as closures. According to HFPS data, households a result of pre-existing inequalities in technology in wealthier developing countries (e.g., Malaysia, access and other forms of privileges (Clark et Indonesia) spent more on their children’s edu- al., 2021; Donnelly & Patrinos, 2020; Dorn et cation during school closures and were more 16 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations likely to prioritize internet connectivity and device-related improvements. In contrast, households in poorer countries (e.g., Myanmar, Lao PDR) were more likely to spend on tutoring services com- pared with devices or internet connectivity (Figure 9). One reason for this is the pre-existing gap in network infrastructure across EAP countries. Recent data from the International Telecommunication Union (ITU) reveal that, as of 2021, only 1.66 percent of Burmese and 2.03 percent of Laotians were subscribed to fixed broadband services. Conversely, upper middle-income countries in the EAP region, such as Malaysia, had a broadband penetration rate nearly six times higher (11.12 percent) in the same year. In this regard, pre-existing disparities in internet infrastructure not only influenced household decisions on education expenditures, but also exacerbated the gap in access to online learning resources across countries with different income levels.6 Figure 9 On average, households in wealthier countries spent more on their children’s education to compensate for school closures 70 Malaysia % HHs incurring additional expenses 60 50 Cambodia Indonesia 40 A. Share of households incurring Lao PDR additional education expenditures vs. 30 GDP per capita 20 Myanmar 10 0 100 1000 10000 100000 GDP per capita 35 30 30 26 25 23 22 21 21 20 19 B. Households incurring education Percent expenditures due to school closures 15 14 14 15 during the pandemic, by category 11 10 6 5 2 1 1 0 Cambodia Indonesia Lao PDR Malaysia Myanmar improved internet connection tutoring hardware/devices moving schools Source: World Bank staff calculations, using HFPS data. GDP per capita data are from OECD and World Bank national accounts data files, 2021. 6 Notably, Cambodia is an exception. ITU data show that only 2.03 percent of its populations had fixed broadband subscriptions, whereas 26 percent of its households spent more expenditures on improving internet connection during COVID. This is because fixed broadband services are not widely available in Cambodia. Based on pre-pan- demic data from 2017, there were 117,049 fixed broadband subscriptions in Cambodia, accounting for just over 1 percent of total internet subscriptions in the country (UN-OHRLLS, 2017). At the same time, there were 18,572,973 mobile cellular subscriptions in Cambodia in 2017 (International Telecommunication Union (ITU), 2017). This indicates that mobile connectivity was a much more prevalent mode of internet access in Cambodia than fixed broadband, even prior to the pandemic. 17 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations For students from socio-economically disad- Another complementary measure was to provide vantaged backgrounds, reducing disparities in more effective teacher training and orientation learning achievements can be supported by on different aspects of technology-enabled implementing more customized EdTech interven- distance learning. Based on teacher survey in tions, empowering these students to capitalize Cambodia, training on EdTech-based teaching on EdTech and online education opportunities. skills was inadequate, with only about 43 percent To counter the tendency of EdTech to increase of schools providing teacher training on teach- inequalities and enhance the effectiveness of Ed- ing specific subjects using EdTech, and even Tech interventions, researchers and implementors smaller percentages on preparing classes for can consider supplementary measures tailored EdTech-based learning (Bhatta et al., 2022). More to the needs of communities that they intend effective and targeted training on EdTech-spe- to serve. In a COVID-19-related investigation in cific teaching skills could have profound implica- Nepal, researchers utilized mobile phone text tions for teachers with limited prior use of ICT in and voice communications, and found positive teaching and learning contexts, and improve the impacts on mathematics learning for students effectiveness of distance learning in developing belonging to the poorest households and with countries of EAP for the next crisis. parents of low literacy level (Radhakrishnan et al., 2021). 2.2 There is a “perception gap” between decision makers and the experience of students and families with EdTech In addition to the expanding learning inequali- Among those who did not find EdTech effective, ties, we uncovered a discrepancy between the 37 percent cited a lack of access to electronic perceptions of decision makers in EAP and the devices (PC/tablet/laptop) as the primary cause. public regarding the effectiveness of EdTech in This is consistent with the results from the HFPS online learning during the pandemic. In a survey surveys in the Philippines, where 18.9 percent of conducted together with the Center for Global respondent households cited “lack of access to Development (CGD), on average, 51 percent of devices” as the number one barrier to learning the decision makers at the ministries of education during the COVID-19 pandemic. However, only and finance in Indonesia, Lao PDR, Mongolia, the 3.7 percent of the surveyed decision makers Philippines, and Vietnam acknowledged that EdTech mentioned limited internet access as a primary did not benefit all children equally. However, the reason for EdTech’s ineffectiveness (Figure 10B), majority (85 percent) believed that EdTech was which contrasts with the HFPS results that indicate effective in supporting student learning during limited internet access is, in fact, a significant COVID-19-related school closures (Figure 10A). obstacle, for example in Indonesia (Figure 11). 18 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Figure 10 Decision makers’ opinion on whether or not EdTech was effective and the reasons for ineffective EdTech usage A. Perceptions of EdTech effectiveness among EAP B. Decision makers' opinions on why decision makers during COVID-19-related school closures EdTech was not effective in supporting student learning during school closures "EdTech (e.g. Internet, TV, radio, or mobile) was effective in 40 supporting student learning during COVID-related school closures.” 37.0 100 100 30 90 94 92 80 85 Percent 70 74 76 70 20 60 18.5 50 14.8 40 11.1 10 30 3.7 20 0 Combination of all Children lacked access to internet Lack of parental support/supervision Teachers were unable to teach remotely Children lacked access to devices (pc/tablet/laptop) 10 0 lia s e sia R ne m ag AP ne PD go pi na er -E o o on lip et av n Ind La M hi Vi P No P EA Source: World Bank. Indonesia - High-Frequency Monitoring of COVID-19 Impacts 2020–2022, Rounds 7 (HIFY 2020–2022). Figure 11 Percentage of respondents citing constraints faced in learning from home in Indonesia, from highest to lowest Limited internet access 20.8 No constraints 12.9 Difficult to focus and concentrate on learning 12.7 No supporting devices (e.g. computer,laptop, tv, radio) 9.2 No guidance from parents/other adult household member 3.0 No internet access 2.6 Other constraints 1.6 No/limited space for study at house 1.5 No/limited electricity access 0.9 0 5 10 15 20 25 Percent Source: World Bank. Indonesia - High-Frequency Monitoring of COVID-19 Impacts 2020–2022, Rounds 7 (HIFY 2020–2022). 19 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Other questions in the same survey reveal that only 2.8 percent of decision-makers in five developing countries of the EAP region identified internet connectivity as the primary reason for low learning levels (Figure 12). In contrast, over half of the respondents (54 percent) considered implementation capacity the most crucial factor in enhancing student academic outcomes (see Yarrow et al. for a detailed discussion). It may be that officials perceive connectivity as a necessary but not sufficient prerequisite for supporting learning. Figure 12 Decision makers’ opinions on the main driver and solution to improving low student learning outcomes are poverty and implementation capacity, respectively A. Decision makers' opinions on what the most B. Decision makers' opinions on what the biggest important reason for low levels of learning are barriers to improving student learning outcomes are 60 54 50 40 38.3 40 Percent 30 30 27.0 27 Percent 20 20 10 10 10 7.8 2.1 2.8 2.8 5.0 5.7 1 1 4 4 0 0 Poor classroom instruction (teacher competency, classroom environment) Poor school facilities Malnutrition Lack of family support / involvement Lack of books and learning materials Poverty Political resistance from teachers' unions Lack of internet connectivity Combination of all Lack of interest from govt Implementation capacity Other Political resistance from others No answer Money Source: World Bank 2023, based on data collected together with the Center for Global Development (CGD). We do not yet have enough information to clearly untangle the apparent contradictions between officials belief that EdTech was effective during the COVID-19-related school closures and available data that it wasn’t effective in supporting learning for most students. Conclusions regarding the striking discrepancy between decision makers’ views and the experiences of students and families point to a clear need for further study and effort to understand and eventually reconcile this sub- stantial perception gap. Given the limited alternatives to tech-supported distance learning during pandemic-induced school closures, taking any action was often viewed as better than inaction, possibly creating a favorable impression for some decision-makers. 20 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Section 3 Moving forward: How EdTech can support learning for all in EAP Although large-scale evidence of student learning (see main EAP Education Flagship report for de- impacts from EdTech interventions by governments tailed discussion, Afkar et al., 2023). Going from in MICs in the EAP region is limited, countries and a low-performing teacher to a high-performing families continue to invest in EdTech. This makes teacher increases student learning dramatically. it essential to examine the evidence on how The effect has been measured from more than 0.2 technology can be utilized to enhance learning SD in Ecuador to more than 0.9 SD in India—the outcomes for all students. Drawing from our global equivalent of multiple years of business-as-usual knowledge base, we propose several potential schooling (Béteille & Evans, 2021). Technology applications of technology to improve student is therefore most effective when it is used to academic outcomes in developing countries in enhance the ability of teachers to ensure that the region and synthesize a range of EdTech every student learns (World Bank, 2020). We adoption models in EAP for the consideration therefore focus our recommendations on how of relevant stakeholders. EdTech can improve student learning through improved instruction that students receive in the Teachers are the primary actors to recover learn- classroom. ing in the aftermath of the COVID-19 pandemic 3.1 Recommendations There are no specific EdTech approaches list- without complementary investments in connec- ed in the “Great Buys” or “Good Buys” of the tivity and most importantly teacher training are Global Education Evidence Advisory Panel (World unlikely to improve learning. At the same time, Bank,2023). The evidence is clear that investing the distinction is becoming increasingly unclear in hardware like laptops, tablets and computers between an “EdTech intervention” and an ed- 21 Photo: World Bank Flckr ucation intervention that uses technology and decision makers select effective investments to interventions implemented in schools where tech- support student learning, which will differ based nology is frequently used. The recommendations on context, capacity and budget, as well as the below are presented with the aim of helping specific learning focus of each program. Recommendation 1: Consider using remote instruction, “the broadcast” or “dual teacher” model, where high-quality teachers are in short supply Remote instruction (i.e., “the broadcast” or “dual “dual–teacher model” has been used to refer to teacher” model), including both pre-recorded this novel form of instruction, though the term and livestream approaches, has shown signif- has normally been solely used to describe the icant positive impacts on learning in several livestream approach (He et al., 2020). studies conducted in rural areas of MIC con- texts where high-quality teachers are in short The interventions in Pakistan and Mexico are supply, including in Ghana (Johnston & Ksoll, examples of the pre-recorded (or asynchronous) 2017), India (Naik et al., 2020), Pakistan (Beg et approach of this remote instructional model. In al., 2019), Mexico (Borghesan & Vasey, 2021), both of these interventions, students watched a China (Bianchi et al., 2022; Li et al., 2023), and recording of a well-trained teacher who delivers Uruguay (British Council, n.d.). Four of these curriculum-aligned content tailored to the local studies were implemented by governments, and context. The videos lasted for a portion of class five had treatment samples greater than 4,000 time (about 10 to 15 minutes), while the remain- students (in fact, the studies in India, Mexico, and ing time involved instruction from the in-person China were evaluations of interventions scaled up teacher. The Pakistan intervention involved two to thousands of schools and in Uruguay the study days of in-service training to orient teachers on was part of a national program). It is important to how to use this blended learning approach, while note that, in contrast to the remote instruction in the model used in Mexico, known as telese- that occurred during COVID-19-related school cundaria, the teachers also received a detailed closures, these remote instruction interventions teaching guide with step-by-step instructions for all took place in the classroom during the regular how to teach each class. The Mexican model is school day, and they involved the presence of especially notable in that in these telesecundaria a teacher in the classroom with the students in schools, all school subjects are taught in this way, addition to the remote teacher (who transmits and in-class teachers are often responsible for the pre-recorded or livestreamed lecture). Thus, supervising classes in multiple subjects at a time. in some contexts (particularly China), the term 22 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations The interventions in Ghana, India and Uruguay the synchronous models require moderate to involved the livestream (or synchronous) approach high internet speeds to support the livestream, to remote instruction, while the study in China which in the Ghana and India studies were de- included both livestream and pre-recorded ap- livered through satellite technology (a satellite proaches, in addition to other hardware inputs. modem, solar panels) and projection equipment Compared with the asynchronous interventions, (a projector, webcam, etc.), the marginal cost of these studies tended to replace a greater amount adding additional students to the intervention of the normal in-person classroom instruction is still likely lower than interventions with high with the remote instruction: for each class in computer-to-student ratio requirements, such as the India study, students listened to 30 minutes computer-assisted learning. Another advantage of livestreamed lecture and then engaged in a is modest training requirements for classroom 10-minute Q&A with the remote teacher (which teachers compared with more complex forms replaced one-third of all normal class periods of technology or education interventions with per week in students’ main subjects). In the new content creation (Naik et al., 2020). Finally, Ghana study, the livestreamed lectures were classroom teachers’ skills may improve through one hour in length for each class (replacing all observation of the remote teacher (He et al., 2020; classes in English and math). In the Uruguayan Zhang, 2020; Li et al., 2023). As demonstrated study, English lessons were delivered weekly by Li et al. (2023), the indirect impact of remote to classrooms via video conference, while the instruction, which led to enhanced local teacher local Uruguayan classroom teachers delivered quality, resulted in a 0.343 SD increase in stu- the follow-up practice face-to-face. dents’ math scores. Similarly, in Uruguay’s Plan Ceibal remote teaching program, local teachers Compared with other EdTech approaches, the use participated in weekly coordination meetings of this form of blended remote instruction, either with the remote teacher. During these meetings, synchronous or asynchronous, has real potential the remote teacher would assist the Uruguay- benefits. One is low-technology infrastructure an classroom teacher in preparing the lessons requirements compared with some other EdTech and practicing the English content to be covered interventions, as these interventions general- (British Council, n.d.). In addition, some scholars ly require one computer or tablet per teacher contend that the dual–teacher model can enable and therefore have higher student-to-comput- the classroom teacher to concentrate on more er ratios than some other EdTech interventions student-centered instruction, while the remote (Borghesan & Vasey, 2021). The pre-recorded teacher focuses on delivering learning content approach might be particularly useful for rural to students (Johnston & Ksoll, 2017). areas with unstable internet connections. Although Recommendation 2: Explore more effective EdTech-supported teacher professional development with enhanced practicality, accessibility and teacher motivation Similar to the dual–teacher model and other places than the teachers who need to be trained, technologies to assist instruction, virtual teacher but current evidence is mixed-to-negative. training could allow teachers in low-resource settings to connect with expert instructors and Existing impact evaluations on teacher training receive long-term mentorship, but this approach programs have yielded mixed results. A recent requires further evaluation. Distance professional synthesis of 170 studies in 40 LMICs found some development is a common interest where expert evidence of the benefits of online teacher pro- trainers are few and often located in different fessional development for improving teachers’ 23 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations pedagogical practices, content knowledge, and motivation (Hennessy et al., 2022), though the review noted that very few studies have explored impacts on student learning outcomes. A study in Brazil involved a hybrid intervention consisting of several in-person classroom observations with feedback for teachers and Skype-mediated expert coaching sessions for pedagogical coordina- tors demonstrated small, positive improvements in Portuguese and math for Grade-10 students (0.05–0.09 SD) at a cost of US$2.40 per student per year (Bruns et al., 2018). In contrast, a virtual coaching intervention in South Africa that examined the impacts of virtual teacher training on student learning found their virtual professional development (PD) was less effective when compared with their in-person training and only marginally less expensive (Cilliers et al., 2021). The quality and format of PD programs (such as the practicality and accessibility of training content), along with teacher motivation, serve as key determinants for the success of in-person PD initiatives (Loyalka et al., 2019; Popova et al., 2016), which likely is also necessary for online PD programs to succeed. Poorly designed training programs that do not align with teacher needs and interests are unlikely to improve teaching or learning. In this case, evidence from large-scale online teacher training in Indonesia during COVID-19-related school closures can provide us with further insights into ways that online teacher training could be improved at scale (Yarrow et al., 2022), including continuous mentorship on their implementation and opportunities to apply what they learned during PD sessions (see text box, below). Teacher Training in Indonesia A World Bank team mapped the eight largest private and public providers of online teacher training in Indonesia, covering 25 programs delivered in 2021. Most of the programs were short in duration and focused on digital literacy skills and how to implement remote learning by teachers for students. Training programs were mostly provided using online lectures, few provided individual coaching, while none provided opportunities for personalized teacher learning. In parallel, a nationally rep- resentative phone survey of 435 primary and junior secondary teachers spanning 30 provinces across Indonesia between February and March 2021 found that 44 percent of teachers participated in online learning during the pandemic, and that three-quarters of these teachers had never par- ticipated in online training prior to the pandemic. Many training participants reported challenges in implementing what they learned from online training. Most of the teachers who participated (88 percent) would like to continue receiving training online even after the pandemic ends. These re- sults suggest that demand for online training is expected to persist post-pandemic, but more can be done to improve training quality. Source: Yarrow, Noah; Khairina, Noviandri; Cilliers, Jacobus; Dini, Indah. 2022. The Digital Future of Teacher Training in Indonesia: What’s Next?. World Bank. One potential model could be to use blended teacher training and dual-teacher classroom pro- gramming together, though this has not yet been evaluated: “The blended model” support for teacher training and classroom instruction 24 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Online teacher training and dual–teacher classroom programming could be used together to enable students to receive higher-quality instruction from remote teachers and support classroom teachers to improve their own instruction. In the dual–teacher model, observing how remote teachers instruct could be part of classroom teachers’ professional development, and other components (i.e., one- on-one training sessions over phone calls, in-person follow-up visits, online discussions with other teachers, etc.) could be integrated as teacher training. This blended model could be implemented differently based on the availability of resources in each context. Options to implement these blended approaches to improving instruction (i.e., remote instruction and virtual teacher professional development), depending on the technological resources available in the given context include: • High-tech: If schools are already equipped with internet and/or satellite transmission infra- structure, or they are able to invest in such resources, then school administrators may consider using livestream to connect students with high-quality remote teachers, while one-on-one teacher professional development sessions could be partially conducted via video-confer- encing platforms. In those contexts where it is feasible for teachers to record themselves teaching, video-based self-reflection, as well as virtual observation of lessons, can be a part of professional development support. • Low-tech: In low-resource settings where internet connections are not widely available, les- sons could be recorded and then shared. Simultaneously, the classroom teacher can interact with a remote teacher in training sessions conducted over the phone blended with in-person visits. If the recorded lectures only take up a portion of the class period, teachers should ideally receive a structured lesson guide describing how to utilize the materials in conjunction with normal classroom activities. For remote teacher training sessions, a higher number of in-per- son teachers (trainees) could also be assigned to one mentor. SMS text messages and social media could be utilized to facilitate mentor-trainee communication and inter-peer learning among teachers (Jukes et al., 2017). Recommendation 3: Don’t use E-readers as an anchor for reading and learning programs Having a library of hundreds or even thousands learning compared with paper technology, and of textbooks and works of fiction at students’ may be more costly. One meta-analysis using 17 fingertips sounds like an educator’s dream. When studies from high-income countries of children first introduced, it was hoped that digital books, and adults found that “reading on paper was “e-readers” and related tablet and mobile devices better than reading on screen in terms of read- would help eradicate illiteracy around the world. ing comprehension” (Kong et al., 2018). Another While helpful for learners with some disabilities, meta-analysis of 54 studies of school children the available evidence suggests e-readers are and undergraduates primarily in high-income less effective than high quality teacher training countries by Delgado et al. (2018) tried to ac- and regular textbooks and readers to support count for difficulties in comparing paper texts student learning. with digital texts, which include features such as hyperlinks, animations, or adaptive tests, The available evidence indicates that e-read- which may have different effects on learning. ers are often less effective in improving student They found advantages in paper-based books 25 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations compared with digital reading, both in the case of per dollar than the e-reader group (Piper et al., time-constrained reading and when using narrative 2016). Lastly, a recent empirical study found no texts (Delgado et al., 2018). A review of recent significant differences in reading comprehension studies also indicates that generally students and time spent on reading between groups using retain, remember, and comprehend information print text and those using iPads. Feedback from more thoroughly when reading on paper than students in the treatment group suggested that on a screen (Cohn, 2021), though we note the they struggled with using in-app task compo- increasing ubiquity of screens in classrooms, nents. They also found the reading platform to particularly in higher income countries. be unsupportive of their note-taking, reading, and highlighting behavior (Sheen & Luximon, 2021). Research on e-readers in the context of MICs These findings from outside the EAP context indicates that there are additional challenges. indicate that increased learning is connected In Lagos, Nigeria, World Bank colleagues found to the content on the device, the supporting that e-readers with material aligned with the cur- environment including basics such as electricity, riculum led to increased student learning only whether there are printed materials available as in the absence of textbooks, while e-readers an alternative, and the usability of e-readers, with non-curriculum reading material led to though this may improve with time as students decreases in student learning in both reading become more adept at using them. To combat and math (Habyarimana & Sabarwal, 2018). A learning poverty in EAP, our attention should study in Zambia found that girls in an e-reader be on encouraging and supporting children’s treatment arm scored significantly better on two reading activities regardless of the medium of of three basic literacy assessments than girls delivery, though caution should be exercised in the control arm. The study noted that there based on the above findings when contemplating was a general lack of access to paper books in an investment in e-readers. these communities and that the treatment was not simply providing e-readers but also included Despite these negative findings, e-readers and a broader intervention package, which involved other devices may be appropriate for some spe- training and compensating adults to support and cific learning challenges. We have much more troubleshoot technical issues (such as replacing experience and evidence about paper-basEd- chargers that were burnt out) and which was Technology than screen-basEdTechnology, and necessary for the smooth functioning of the the technology is changing constantly. Technology intervention. The comparison groups did not has the ability to make text larger, or provide the receive any reading material, printed or other- same text in many different languages, which wise (Mensch et al., 2021). One would hope that may be very helpful for certain learners and con- e-readers and adult support would show better texts. For students, e-books reduce the number results than nothing at all! of textbooks they have to carry to school and include attractive features. These features, such Piper and colleagues, working on multiple in- as user-friendly functions, engaging graphics, terventions in Kenya, discovered that students enlarged text size, and plug-in speakers, could using individual e-readers experienced smaller promote student learning autonomy and encour- gains in oral reading frequency compared with age creativity (Embong et al., 2012). For teachers, those in the non-e-reader treatment groups. e-books could also simplify the management Cost-effectiveness was the lowest for the e-reader process, as they allow concurrent monitoring of group, with the control group and the two oth- classroom activities by each student (Embong er interventions involving paper-based books et al., 2012). Evidence from a recent study of demonstrating significantly larger learning gains fewer than 800 students in the Republic of Korea 26 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations reported gains in learning (0.28 to 0.32 SD in a paper-based and a screen-based approach to social studies and science) associated with the improving a specific aspect of student learning?” introduction of digital textbooks as a supplement The alignment of the content with the curriculum to other, printed material (Lee et al., 2022). The is essential, and there is great variety in the quality question education decision makers may ask, and features of digital texts, as well as printed therefore, is less “do e-readers and digital text- texts. These need to be carefully considered for books work?” and more “what are the trade-offs each context and learning goal before choosing in terms of costs and needs for support between a specific approach. Recommendation 4: Develop and expand interventions using assistive technol- ogy for learners with disabilities In 2022, the World Bank estimated that over learn and stay enrolled (WHO & UNICEF, 2022). 1 billion people have a disability (World Bank, Even in China, one of the high performers in this 2022). About 80 percent of them live in devel- domain, schools for vision and hearing impaired oping countries, making them the world’s largest students tend to work regionally, and do not minority (World Health Organization, 2023). Of report student learning outcomes (Yarrow et al., these, there are an estimated 41 million children forthcoming). Second, recent reports on assistive with disabilities in EAP (UNICEF East Asia and technology have highlighted challenges such as Pacific, 2020), making the potential market for inadequate funding, lack of access and expertise, assistive technology very large. However, access and a lack of awareness among teachers, as lags far behind demand (World Bank, 2020), in- well as among education decision makers more dicating a generalized market failure. Learners broadly. More than 90 percent of policy leaders with disabilities are the most likely to be out of interviewed in EAP agreed with the statement school and at high risk of dropout and tend to that children with disabilities deserved the same have lower levels of learning than their peers level of access to public schools, and that ac- (UNICEF East Asia and Pacific, 2022). Meanwhile, commodations should be made to include them assistive technology has the potential to integrate (CGD forthcoming; Yarrow et al., forthcoming). students in the classroom, facilitating interactions However, around 10 percent of respondents with same age peers and building recreational in multiple countries did not agree with these skills (Zilz & Pang, 2021). This is a missed oppor- statements, indicating that even among people tunity both for the students themselves and the in leadership positions, the right of educational wider society and can be partially addressed inclusion is not universally supported. These with existing EdTech solutions. issues are hindering the development of learn- ing technologies for students with disabilities. Several challenges, however, need to be taken Furthermore, students may feel stigmatized and into consideration before implementing assistive choose not to use these technologies, as existing technologies at scale. First, existing evidence technologies have been found not to align ap- on the effectiveness of assistive technology is propriately with the students’ needs (Khazanchi limited. So far, services for vision and hearing-im- et al., 2022). paired students tend to be localized rather than national. They are often provided with support Moving forward, it would be helpful for govern- from NGOs and international donors to fill gaps ments and civil society to collaborate with the in provision from national governments. Devices private sector and research community in the and software such as screen readers, speech- EAP region. Such partnerships can facilitate the to-text tools, and hearing aids can help children adaptation, development, evaluation, and scal- 27 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations ing-up of assistive education technology that to address these issues include market shaping, caters to the needs of children with disabilities, policy development, and information sharing, particularly in low- and middle-income countries. which can support large-scale improvements in Engaging with systemic changes is essential to product offerings, service provision, personnel ensure the effective use of these technologies training, and policy for accessible, adaptable, and in enhancing student learning outcomes. As not- high-quality assistive technology (World Health ed above, the sector faces challenges such as Organization, 2022). Addressing current gaps insufficient information for producers and con- in access and quality is crucial to avoid leaving sumers, barriers to entry, and fragmentation both millions of children with disabilities in the EAP between and within countries. Promising strategies region behind. Recommendation 5: Explore and evaluate the use of AI-enabled interventions for the personalization of learning and empowerment of teachers The potential of artificial intelligence (AI) in ed- 2021; Sedlmeier, 2001). The system attempts to ucation has received much attention in recent create an individualized learning pathway for the years as the use of AI tools to support or en- student that is continuously updated. hance learning has grown (Holmes et al., 2019). AI proposes providing learners with access to In contexts where teachers are overextended or high-quality personalized learning, in addition to otherwise have difficulty providing students with facilitating new approaches to assessment and individualized instruction, these systems could reducing teacher workload. At the same time, potentially help aid teachers in personalizing there are still many concerns about AI educa- the students’ learning pathways and providing tional applications in terms of their efficacy, the teachers with the data and time to better under- tendency of some forms of AI to replace teach- stand how to engage students in differentiated ers, as well as other ethical challenges (Akgun instruction. One recent intervention in the region, & Greenhow, 2022). We provide an overview which is part of a global initiative called Hi-Tech of potential opportunities for AI in education, Hi-Touch, uses an ITS to support educators in while also highlighting some caveats regarding teaching foundational knowledge and enabling potential drawbacks and gaps where more re- students to progress at their own speed, which search is needed. helped teachers focus on individualized instruction and active learning experiences in the class- One of the most common and long-standing room (Education Commission, 2020). The report applications of AI for improving student learn- recommends that crucial factors to the model’s ing is intelligent tutoring systems (ITS). The way early signs of success are having a dedicated such systems work is by automatically adjusting ministry contact who supports the project, using learning activities, difficulty, and scaffolding con- the software to identify opportunities for teach- tent based on the student’s learning level. They er professional development, and establishing make adjustments using four integrated models alignment between the software content and the (i) expert domain knowledge about the subject national curriculum to increase buy-in. (the domain model); (ii) a diagnostic assessment of the student’s current level (the student model); In addition to ITS, AI can be used to automate (iii) adaptive feedback and hints to assist student clerical and administrative tasks to relieve teach- learning (the teaching model); and (iv) a user ers’ non-teaching workloads. A recent estimate interface that facilitates communication between of the activity composition of teacher working the user and the system (Mousavinasab et al., hours in multiple countries found that less than 28 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations half of teachers’ workday involves directly interacting with students, with the majority of their time being dedicated to tasks such as preparation, evaluation and feedback, professional development, and administration (Cardona et al., 2023). By handling lower-level details, including scoring exam- inations and essays, teachers would have more time to dedicate to direct interactions with students. A further application of AI includes improving professional development approaches. New products can allow teachers to record their classroom interactions and use AI algorithms to provide them with data on the ratio between the time that students and teachers spend talking or between the types of discourse most often used during class time, as well as help them select specific segments of the video recordings to review with coaches (Jensen et al., 2020). A randomized experiment that evaluated an AI-powered classroom discourse analyzer tool that helps select video record- ing content for professional development sessions in Shanghai, China, found significant positive impacts on a sixth- and seventh-grade students math performance (0.24 SD) (Chen et al., 2020). Despite the recent explosion in interest in applications of AI in education, there is still limited rigorous evidence on its effectiveness (UNESCO, 2021), especially when compared to EdTech that performs similar functions but does not employ AI technology (van Klaveren et al., 2017; Vanbecelaere et al., 2020). It is important for governments to pilot different technologies, comparing those powered by AI and to other options to understand what works best to cost-effectively support student learning. Recommendation 6: Improve and expand research on EdTech for student learning There are six ways EdTech pilots and evaluations can be improved: 1. Test at scale – more evaluations of EdTech should be conducted at scale. While small “pilot” evaluations can be tests of efficacy, they often give little information about impacts, costs and capacities required to implement for large numbers of students, teachers and schools. 1. Design for scale – where tests with large numbers of students and teachers are not possible due to funding or other constraints, design with scale in mind, for example by minimizing costs and simplifying implementation. If teachers can only implement the program with intense expert sup- port, the impact on student learning is unlikely to be scalable with existing government capacity. 1. Report learning impacts by type of sub-population (gender, income, baseline learning level etc.) in addition to the average effect. This heterogeneity becomes increasingly important at scale, especially if some effects are negative or some types of learners benefit much more than others. 1. Report the number of participants in the total treatment group, as well as the sample used for the study. If the sample is part of a much larger program, it is helpful to know that the inter- vention is already working at scale. 1. Measure and report costs, including training costs. When governments consider different ap- proaches to support student learning, it is helpful to know what they might cost to implement. 1. Measure uptake and dosage effects, and report them. Since variations in intervention com- pliance and intensity often lead to heterogeneity in learning impacts, future researchers and implementers should develop standard metrics for uptake and dosage for comparison across interventions and study contexts. 29 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Photo: World Bank Flckr 30 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Section 4 Conclusion Future pathways of the EdTech industry in EAP: the predominant position in the Indonesian K-12 balancing public and private interests EdTech market for several years (Omidyar Network, 2019). Another example in Indonesia is Google In July 2021, the Central Committee of the Com- Classroom. During the pandemic the ministry munist Party of China issued Decree No. 40, titled requested 26 million workspace accounts to “Opinions on Further Reducing the Burden of be established for students within 24 hours as Students’ Homework and Off-campus Training schools were being closed. Google Classroom in Compulsory Education.” The stated objective completed setting up these accounts in March of the reform was to achieve a “double reduc- of 2020; however, recent analysis indicates that tion” in the areas of homework and out-of-school only around 10 million of these accounts were tutoring, while alleviating parents’ anxiety and ever activated by students, and that account promoting students’ all-round development and activations were mostly in urban areas (Author healthy growth. The regulation also stated that Interview). In Malaysia, the Ministry of Education companies that engaged in “subject-based train- has partnered with Google, Microsoft and Apple ing” “will be uniformly registered as non-profit to form the DELIMa (Digital Education Learning institutions,” and will not be listed on stock ex- Initiative Malaysia) initiative. The aim is to provide changes or have foreign investors. a range of e-learning applications and resourc- es, harnessing a multi-technology ecosystem Governments of other countries in the region for greater accessibility, and averages 1.7 million have taken a more favorable stance toward active users monthly (Project ID, 2020). In Viet- for-profit EdTech, aimed more at supporting or nam, Galaxy Education is piloting a dual–teacher at least crowding-in private sector initiatives. A approach for English language teaching in 100 scaled example from Indonesia is the relation- schools, trying to provide high-quality language instruction to classrooms who otherwise would ship between EdTech startup Ruang Guru and not have access to this level of subject expertise 32 regional governments. The partnerships al- (Author interview, May 2022). In Cambodia, the low Ruang Guru to widely distribute vouchers government is supporting teachers to identify sold to the Government for their product and students with hearing and vision disabilities using also include a package of educational content, a phone based app. The program then helps virtual classes, the online-based test platform, connect students to services so they remain in and teacher training. As a result of this strategy school and continue learning. (Yarrow et al., forth- and substantial funding, Ruang Guru has held coming). Lastly, the Republic of Korea presents an 31 Photo: World Bank Flckr example of how public-private partnerships can ing the experiences of various stakeholders over enhance EdTech system resilience during public the past few years (World Bank, 2020). health crises. Capitalizing on the country’s ad- vanced ICT infrastructure, the Korean government While the worst stages of the COVID-19 pandemic rapidly expanded the capacity of its e-learning have subsided for most countries in the region, platforms to support students who experienced the future remains uncertain. Without a resilient school closures during the COVID-19 pandemic. education system, teachers and students cannot About 50,000 public learning resources were unlock remote learning opportunities during times added to these e-learning platforms, with much of crisis, leading to significant learning losses and of the free content becoming available through risking widening the gap in academic outcomes a joint effort between the Korean government across students with varying levels of access to and the private sector (Ministry of Education, technology. Although EdTech is often a source 2020). The Korean government also proactive- of increasing learning inequality, its potential to ly addressed the digital divide in collaboration improve student learning outcomes hinges on with major telecommunications companies such addressing the current limitations and inequalities as Samsung and LG. As of April 2020, national it exacerbates. As the role of EdTech expands statistics indicate that 280,000 students rented with new AI applications, it becomes increasingly digital devices free of charge, accounting for 5.3 vital for policy makers and the private sector to percent of the total student population in the collaborate in fostering investment, innovation, Republic of Korea (Ministry of Education, 2020). evaluation, and regulation of the EdTech eco- system in the EAP region. By ensuring equita- In sum, the contrasting approaches to regulating ble access to internet connectivity and tailoring and supporting EdTech between China and oth- EdTech solutions to cater to students’ diverse er countries in the EAP region underscore the diversity of strategies governments can adopt to needs, we can unleash its potential to not only address the challenges and opportunities arising enhance academic performance but also bridge from public health and other crises, including the gap in learning inequalities. Meanwhile, con- the learning crisis. The integration of technology ducting more rigorous impact evaluations with in education is likely to persist and expand, as larger intervention scales and higher impacts many educational institutions, governments, and but lower implementation costs will enable us other stakeholders have recognized its potential to better test the effectiveness of EdTech inter- benefits, especially in making education systems ventions. In this regard, the post-COVID-19 era more resilient. In countries such as Indonesia, presents a unique opportunity to build a more for instance, a steep trajectory in internet pene- inclusive and effective educational landscape, tration rates has coincided with the emergence with technology serving as a potential catalyst for of numerous EdTech startups, which offer a wide meaningful and lasting change in the education variety of products and services aimed at improv- systems of EAP countries. 32 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations References Abeberese, A.B., Kumler, T.J., & Linden, L.L. (2014). Journal of Human Resources, 49 (3) 611–633; DOI: https://doi.org/10.3368/jhr.49.3.611 Afkar, Rythia; Tara Beteille; Mary Breeding; Toby Linden; Andrew D. Mason; Aaditya Mattoo; Tobias Pfutze; Lars M. Sondergaard and Noah Yarrow. Fixing the Foundation: Teachers and Basic Education in East Asia and Pacific. Washington DC: World Bank Group Akgun, S., & Greenhow, C. (2022). Artificial intelligence in education: Addressing ethical challenges in K-12 settings. AI and Ethics, 2(3), 431–440. https://doi.org/10.1007/s43681-021-00096-7 Araya, R., Arias Ortiz, E., Bottan, N. L., & Cristia, J. (2019). Does gamification in education work? Experimental evidence from Chile (Working Paper IDB-WP-982). IDB Working Paper Series. https://doi.org/10.18235/0001777 Barrera-Osorio, F., & Filmer, D. (2016). Incentivizing Schooling for Learning: Evidence on the Impact of Alternative Targeting Approaches. The Journal of Human Resources, 51(2), 461–499. http://www.jstor.org/stable/24736029 Bau, N., & Das, J. (2020). Teacher Value Added in a Low-Income Country. American Economic Journal: Economic Policy, 12(1), 62–96. https://doi.org/10.1257/pol.20170243 Beg, S. A., Lucas, A. M., Halim, W., & Saif, U. (2019). Beyond the Basics: Improving Post-Primary Content Delivery through Classroom Technology (NBER Working Paper No. 25704, p. 42). http://www.nber.org/papers/w25704 Béteille, T., & Evans, D. K. (2021). Successful Teachers, Successful Students: Recruiting and Supporting Society’s Most Crucial Profession. https://policycommons.net/artifacts/2445925/successful-teachers-successful-students/3467666/ Bhardwaj, Riaz; Yarrow, Noah; Cali, Massimiliano. 2020. EdTech in Indonesia: Ready for Take-off?. © World Bank, Washington, DC. Retrieved April 13, 2023, from https://openknowledge​ .worldbank.org/entities/publication/3290e918-2c89-5a90-899f-bba5941eb6ca Bhatta, S. D., Katwal, S., Pfutze, T., Ros, V., Wong, Y. N., & No, F. (2022). Learning Loss in Cambodia and the Use of EdTech during COVID-19. http://hdl.handle.net/10986/38351 Bianchi, N., Lu, Y., & Song, H. (2022). The effect of computer-assisted learning on students’ long-term development. Journal of Development Economics, 158, 102919. https://doi.org/10.1016/j.jdeveco.2022.102919 Biswas, Alok Kumar; Asaduzzaman, T. M.; Evans, David K.; Fehrler, Sebastian; Lnu, Deepika Ramachandran; Sabarwal, Shwetlena. (2020). TV-Based Learning in Bangladesh: Is it Reaching Students? [Text/HTML]. World Bank. Retrieved April 24, 2023, from https://documents.worldbank.org/en/publication/documents-reports/documentdetail/​ 693481594961374723/TV-Based-Learning-in-Bangladesh-Is-it-Reaching-Students 33 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations British Council. (n.d.). Plan Ceibal—Remote Teaching into Uruguayan public schools. Retrieved June 13, 2023, from https://www.britishcouncil.uy/en/programmes/education/ceibal-en-ingles Borghesan, E., & Vasey, G. (2021). The Marginal Returns to Distance Education: Evidence from Mexico’s Telesecundarias (p. 68) [Working Paper]. https://gabriellevasey.github.io/MarginalReturns_BorghesanVasey.pdf Brinkman, S. A., Hasan, A., Jung, H., Kinnell, A., & Pradhan, M. (2017). The Impact of Expanding Access to Early Childhood Education Services in Rural Indonesia. Journal of Labor Economics, 35(S1), S305–S335. https://doi.org/10.1086/691278 Cardona, M. A., Rodríguez, R. J., & Ishmael, K. (2023). Artificial Intelligence and the Future of Teaching and Learning. U.S. Department of Education, Office of Educational Technology. https://www2.ed.gov/documents/ai-report/ai-report.pdf Central Committee of the Communist Party of China (2021), Decree No. 40, “Opinions on Further Reducing the Burden of Students’ Homework and Off-campus Training in Compulsory Education. 中共中央办公厅 国务院办公厅印发《关于进一步减轻义务教育阶段学生作业负担和校外培训负担 的意见》—中华人民共和国教育部政府门户网站, 2021). http://www.moe.gov.cn/jyb_xxgk/moe_1777/moe_1778/202107/t20210724_546576.html Chand, S., Chaudhary, K., Prasad, A., & Chand, V. (2021). Perceived Causes of Students’ Poor Performance in Mathematics: A Case Study at Ba and Tavua Secondary Schools. Frontiers in Applied Mathematics and Statistics, 7. https://www.frontiersin.org/articles/10.3389/fams.2021.614408 Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. B. (2020). Efficacy of video- based teacher professional development for increasing classroom discourse and student learning. Journal of the Learning Sciences, 29(4–5), 642–680. https://doi.org/10.1080/10508406.2020.1783269 Chen, X., Shi, Y., Yi, H., Zhang, L., Mo, D., Chu, J., Loyalka, P., & Rozelle, S. (2013). The Impact of a Senior High School Tuition Relief Program on Poor Junior High School Students in Rural China. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2374361 Cilliers, J., Fleisch, B., Kotze, J., Mohohlwane, N., Taylor, S., & Thulare, T. (2021). Can Virtual Replace In-person Coaching? Experimental Evidence on Teacher Professional Development and Student Learning. RISE Programme. https://riseprogramme.org/publications/can-virtual-replace​ -person-coaching-experimental-evidence-teacher-professional Clark, A. E., Nong, H., Zhu, H., & Zhu, R. (2021). Compensating for academic loss: Online learning and student performance during the COVID-19 pandemic. China Economic Review, 68, 101629. https://doi.org/10.1016/j.chieco.2021.101629 Cohn, J. (2021). Skim, Dive, Surface: Teaching Digital Reading. http://wvupressonline.com/node/865 Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research Review, 25, 23–38. https://doi.org/10.1016/j.edurev.2018.09.003 34 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Donnelly, R., & Patrinos, H. (2020, December 8). Is the COVID-19 slide in education real? https://blogs.worldbank.org/education/covid-19-slide-education-real Dorn, E., Hancock, B., Sarakatsannis, J., & Viruleg, E. (2020). COVID-19 and student learning in the United States: The hurt could last a lifetime. Retrieved April 24, 2023, from https://www.mckinsey.com/~/media/McKinsey/Industries/Public%20and%20Social%20Sector/ Our%20Insights/COVID-19%20and%20student%20learning%20in%20the%20United%​ 20States%20The%20hurt%20could%20last%20a%20lifetime/COVID-19-and-student-learning-in​ -the-United-States-FINAL.pdf Du, K., Wang, H., Ma, Y., Guan, H., & Rozelle, S. (2022). Effect of Eyeglasses on Student Academic Performance: What Matters? Evidence from a Randomized Controlled Trial in China. International Journal of Environmental Research and Public Health, 19 (17), Article 17. https://doi.org/10.3390/ijerph191710923 Duarte, J., Garguilo, C., & Moreno, M. (2011). School Infrastructure and Learning in Latin American Elementary Education: An Analysis Based on the SERCE. https://www.researchgate.net/publication/254422066_School_Infrastructure_and_Learning_in_ Latin_American_Elementary_Education_An_Analysis_Based_on_the_SERCE Education Commission. (2020). High Touch High Tech Learning Vietnam (Education Workforce Initiative: Country Report). from https://educationcommission.org/wp-content/uploads/2020/​ 07/2020-EC-Country-Report-Vietnam-ENG.pdf Embong, A. M., Noor, A. M., Hashim, H. M., Ali, R. M., & Shaari, Z. H. (2012). E-Books as Textbooks in the Classroom. Procedia - Social and Behavioral Sciences, 47, 1802–1809. https://doi.org/10.1016/j.sbspro.2012.06.903 Evans, D. K., & Yuan, F. (2022). How Big Are Effect Sizes in International Education Studies? Educational Evaluation and Policy Analysis, 44 (3), 532–540. https://doi.org/10.3102/01623737221079646 Filmer, D., & Schady, N. (2009). School Enrollment, Selection and Test Scores (No. 4998; Policy Research Working Paper). World Bank. https://openknowledge.worldbank.org/server/api/core/ bitstreams/7f82d893-62e6-5a42-a710-b6ed16592329/content Fuje, Habtamu; Tandon, Prateek, 2018. When do in-service teacher training and books improve student achievement? Experimental evidence from Mongolia 2018—Review of Development Economics—Wiley Online Library. Retrieved April 25, 2023, from https://onlinelibrary.wiley.com/doi/10.1111/rode.12387 Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Teachers in Remote Areas. Research on Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035 Glewwe, P., Kremer, M., & Moulin, S. (2009). Many Children Left Behind? Textbooks and Test Scores in Kenya. American Economic Journal: Applied Economics, 1(1), 112–135. https://doi.org/10.1257/app.1.1.112 35 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Grewenig, E., Lergetporer, P., Werner, K., Woessmann, L., & Zierow, L. (2020). Covid-19 and Educational Inequality: How School Closures Affect Low- and High-Achieving Students. Retrieved April 24, 2023, from https://www.cesifo.org/en/publications/2020/working-paper/ covid-19-and-educational-inequality-how-school-closures-affect-low Habyarimana, J., & Sabarwal, S. (2018). Re-Kindling Learning. World Bank Policy Research Working Paper. https://elibrary.worldbank.org/doi/abs/10.1596/1813-9450-8665 He, X., Yang, H. H., & Wu, S. (2020). Dual–Teaching Model: A Practical Approach for Students in Rural Areas. 2020 International Symposium on Educational Technology (ISET), 193–195. https://doi.org/10.1109/ISET49818.2020.00049 Hennessy, S., D’Angelo, S., McIntyre, N., Koomar, S., Kreimeia, A., Cao, L., Brugha, M., & Zubairi, A. (2022). Technology Use for Teacher Professional Development in Low- and Middle-Income Countries: A systematic review. Computers and Education Open, 3, 100080. https://doi.org/10.1016/j.caeo.2022.100080 Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial Intelligence In Education: Promises and Implications for Teaching and Learning. Center for Curriculum Redesign. https://curriculumredesign.org/wp-content/uploads/AIED-Book-Excerpt-CCR.pdf Jensen, E., Dale, M., Donnelly, P. J., Stone, C., Kelly, S., Godley, A., & D’Mello, S. K. (2020). Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3313831.3376418 Johnston, J., & Ksoll, C. (2017). Effectiveness of Interactive Satellite-Transmitted Instruction: Experimental Evidence from Ghanaian Primary Schools. CEPA Working Paper. https://cepa.stanford.edu/content/effectiveness-interactive-satellite-transmitted-instruction​ -experimental-evidence-ghanaian-primary-schools Khazanchi, Pankaj, and Rashmi Khazanchi. “Role of Assistive Technology in Teaching Students With Disabilities in K-12 Classrooms.” Technology-Supported Interventions for Students With Special Needs in the 21st Century. IGI Global, 2022. 149–176. https://www.igi-global.com/gateway/chapter/300026 Kleiman-Weiner, M., Luo, R., Zhang, L., Shi, Y., Medina, A., & Rozelle, S. (2013). Eggs versus chewable vitamins: Which intervention can increase nutrition and test scores in rural China? China Economic Review, 24, 165–176. https://doi.org/10.1016/j.chieco.2012.12.005 Kong, Y., Seo, Y. S., & Zhai, L. (2018). Comparison of reading performance on screen and on paper: A meta-analysis. Computers & Education, 123, 138–149. https://doi.org/10.1016/j.compedu.2018.05.005 Lee, S., Lee, J.-H., & Jeong, Y. (2022). The Effects of Digital Textbooks on Students’ Academic Performance, Academic Interest, and Learning Skills. Journal of Marketing Research. https://doi.org/10.1177/00222437221130712 Li, H., Liu, Z., Yang, F., & Yu, L. (2023). The Impact of Computer-Assisted Instruction on Student Performance: Evidence from the Dual-Teacher Program. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4360827 36 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Loyalka, P., Liu, C., Song, Y., Yi, H., Huang, X., Wei, J., Zhang, L., Shi, Y., Chu, J., & Rozelle, S. (2013). Can information and counseling help students from poor rural areas go to high school? Evidence from China. Journal of Comparative Economics, 41 (4), 1012–1025. https://doi.org/10.1016/j.jce.2013.06.004 Loyalka, P., Popova, A., Li, G., & Shi, Z. (2019). Does Teacher Training Actually Work? Evidence from a Large-Scale Randomized Evaluation of a National Teacher Training Program. American Economic Journal: Applied Economics, 11(3), 128–154. https://doi.org/10.1257/app.20170226 Loyalka, P., Sylvia, S., Liu, C., Chu, J., & Shi, Y. (2019). Pay by Design: Teacher Performance Pay Design and the Distribution of Student Achievement. Journal of Labor Economics, 37(3), 621–662. https://doi.org/10.1086/702625 Ma, Y., Abbey, C., Hu, D., Lee, O., Hung, W., Zhang, X., Chang, C., Wu, C.-I., & Rozelle, S. (2020). The Impact of Computer Assisted Learning on Rural Taiwanese Children: Evidence from a Randomized Experiment. https://sccei.fsi.stanford.edu/publication/impact-computer-assisted​ -learning-rural-taiwanese-children-evidence-randomized Macdonald, K. A. D., Brinkman, S. A., Jarvie, W., Machuca-Sierra, M., Mcdonall, K. A., Tapueluelu, S., & Thanh Vu, B. (2017). Pedagogy versus School Readiness: The Impact of a Randomized Reading Instruction Intervention and Community-Based Playgroup Intervention on Early Grade Reading Outcomes in Tonga (SSRN Scholarly Paper No. 2901756). Social Science Research Network. https://papers.ssrn.com/abstract=2901756 Mason, E. N., & Smith, R. A. (2020). Tracking Intervention Dosage to Inform Instructional Decision Making. Intervention in School and Clinic, 56(2), 92–98. https://doi.org/10.1177/1053451220914897 Mensch, B. S., Haberland, N., Soler-Hampejsek, E., Digitale, J., Jackson-Hachonda, N., Chelwa, N., Nyirenda, P., Chuang, E., Polen, L., Psaki, S. R., Kayeyi, N., & Mbizvo, M. T. (2021). Effects of an e-reader intervention on literacy, numeracy and non-verbal reasoning among adolescent girls in Zambia: Evidence from a randomised controlled trial. Journal of Development Effectiveness, 13(3), 247–275. https://doi.org/10.1080/19439342.2021.1953566 Ministry of Foreign Affairs, Republic of Korea (2020). Responding to COVID-19: Online Classes in Korea. Retrieved March 31, 2023, from https://www.mofa.go.kr/eng/brd/m_22747/view.do?seq=​ 15&srchFr=&%3bsrchTo=&%3bsrchWord=&%3bsr Mo, D., Bai, Y., Shi, Y., Abbey, C., Zhang, L., Rozelle, S., & Loyalka, P. (2020). Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China. Journal of Development Economics, 146, 102487. https://doi.org/10.1016/j.jdeveco.2020.102487 Molato-Gayares, R. (2022). How to Recover Learning Losses from COVID-19 School Closures in Asia and the Pacific (Issue 217). Asian Development Bank. https://www.adb.org/publications/learning-losses-covid-19-school-closures Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi, M. (2021). Intelligent tutoring systems: A systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1), 142–163. https://doi.org/10.1080/10494820.2018.1558257 37 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Naik, G., Chitre, C., Bhalla, M., & Rajan, J. (2020). Impact of use of technology on student learning outcomes: Evidence from a large-scale experiment in India. World Development, 127, 104736. https://doi.org/10.1016/j.worlddev.2019.104736 Nie, J., Pang, X., Wang, L., Rozelle, S., & Sylvia, S. (2020). Seeing Is Believing: Experimental Evidence on the Impact of Eyeglasses on Academic Performance, Aspirations, and Dropout among Junior High School Students in Rural China. Economic Development and Cultural Change, 68(2), 335–355. https://doi.org/10.1086/700631 Omidyar Network. (2019). Scaling Access and Impact Indonesia Report. https://assets.imaginablefutures.com/media/documents/Scaling_Access_and_Impact_ Indonesia_Report_vFinal_r.pdf Patrinos, H. A., Vegas, E., & Carter-Rau, R. (2022). An Analysis of COVID-19 Student Learning Loss. https://doi.org/10.1596/1813-9450-10033 Piper, B., Zuilkowski, S. S., Kwayumba, D., & Strigel, C. (2016). Does technology improve reading outcomes? Comparing the effectiveness and cost-effectiveness of ICT interventions for early grade reading in Kenya. International Journal of Educational Development, 49, 204–214. https://doi.org/10.1016/j.ijedudev.2016.03.006 Popova, A., Evans, D. K., & Arancibia, V. (2016). Training Teachers on the Job. World Bank Policy Research Working Paper. https://openknowledge.worldbank.org/server/api/core/ bitstreams/54a97576-f662-51c8-b702-e5a6be49f993/content Pritchett, L., & Sandefur, J. (2013). Context Matters for Size: Why External Validity Claims and Development Practice Don’t Mix (SSRN Scholarly Paper No. 2364580). https://doi.org/10.2139/ssrn.2364580 Project ID. What is DELIMa? Malaysia’s Rocketing Online Education. (2020, July 14). https://project-id.org/blog/2020/7/13/what-is-delima-malaysias-rocketing-online-education Radhakrishnan, K., Sabarwal, S., Sharma, U., Cullen, C., Crossley, C., Letsomo, T., & Angrist, N. (2021). Remote Learning: Evidence from Nepal during COVID-19. World Bank Policy Brief. https://www.researchgate.net/profile/Uttam-Sharma-4/publication/353549026_Remote_ Learning_Evidence_from_Nepal_during_COVID-19/links/610279e91e95fe241a96945c/​Remote​ -Learning-Evidence-from-Nepal-during-COVID-19.pdf Schady, N., Holla, A., Sabarwal, S., Silva, J., & Yi Chang, A. (2023). Collapse and Recovery: How the COVID-19 Pandemic Eroded Human Capital and What to Do about it. World Bank. https://doi.org/10.1596/978-1-4648-1901-8 Sedlmeier, P. (2001). Intelligent Tutoring Systems. In N. J. Smelser & P. B. Baltes (Eds.), International Encyclopedia of the Social & Behavioral Sciences (pp. 7674–7678). Pergamon. https://doi.org/10.1016/B0-08-043076-7/01618-1 Sheen, K. A., & Luximon, Y. (2021). Effect of in-app components, medium, and screen size of electronic textbooks on reading performance, behavior, and perception. Displays, 66, 101986. https://doi.org/10.1016/j.displa.2021.101986 38 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Sylvia, S., Luo, R., Zhang, L., Shi, Y., Medina, A., & Rozelle, S. (2013). Do you get what you pay for with school-based health programs? Evidence from a child nutrition experiment in rural China. Economics of Education Review, 37, 1–12. https://doi.org/10.1016/j.econedurev.2013.07.003 Trucano, M. (2013). The Matthew Effect in Educational Technology. https://blogs.worldbank.org/edutech/matthew-effect-educational-technology UNESCO (2021). AI and education: Guidance for policy-makers. Retrieved June 16, 2023, from https://unesdoc.unesco.org/ark:/48223/pf0000376709 UNICEF East Asia and Pacific. (n.d.). Education for Every Ability. Retrieved March 31, 2023, from https://www.unicef.org/eap/reports/education-every-ability UNICEF East Asia and Pacific. (2020). Regaining confidence after school closures. Retrieved March 31, 2023, from https://www.unicef.org/eap/stories/regaining-confidence-after-school-closures Van Klaveren, C., Vonk, S., & Cornelisz, I. (2017). The effect of adaptive versus static practicing on student learning—Evidence from a randomized field experiment. Economics of Education Review, 58, 175–187. https://doi.org/10.1016/j.econedurev.2017.04.003 Vanbecelaere, S., Van den Berghe, K., Cornillie, F., Sasanguie, D., Reynvoet, B., & Depaepe, F. (2020). The effectiveness of adaptive versus non-adaptive learning with digital educational games. Journal of Computer Assisted Learning, 36(4), 502–513. https://doi.org/10.1111/jcal.12416 Vivalt, E. (2020). How Much Can We Generalize From Impact Evaluations? Journal of the European Economic Association, 18(6), 3045–3089. https://doi.org/10.1093/jeea/jvaa019 WHO & UNICEF. (2022). Global Report on Assistive Technology. https://www.unicef.org/reports/global-report-assistive-technology Wilichowski, T., & Cobo, C. (2021). Considering an adaptive learning system? A roadmap for policymakers. https://blogs.worldbank.org/education/considering-adaptive-learning-system​ -roadmap-policymakers Wong, Ho Lun; Shi, Yaojiang; Luo, Renfu, Zhang, Linxiu and Rozelle, Scott (2014). Improving the Health and Education of Elementary Schoolchildren in Rural China: Iron Supplementation Versus Nutritional Training for Parents: The Journal of Development Studies: Vol 50, No 4. (n.d.). Retrieved April 25, 2023, from https://www.tandfonline.com/doi/abs/10.1080/00220388.2013.866223 World Bank. (2020). Realizing the Future of Learning: From Learning Poverty to Learning For Everyone, Everywhere. https://documents1.worldbank.org/curated/en/250981606928190510/pdf/Realizing-the-Future-of​ -Learning-From-Learning-Poverty-to-Learning-for-Everyone-Everywhere.pdf World Bank. (2020). Pivoting to Inclusion: Leveraging Lessons from the COVID-19 Crisis for Learners with Disabilities. Retrieved March 31, 2023, from https://www.worldbank.org/en/topic/disability/publication/pivoting-to-inclusion-leveraging​ -lessons-from-the-c-ovid-19-crisis-for-learners-with-disabilities 39 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations World Bank. (2021). Reimagining Human Connections: Technology and Innovation in Education at the World Bank. Retrieved April 25, 2023, from https://documents.worldbank.org/en/ publication/documents-reports/documentdetail/829491606860379513/reimagining-human​ -connections-technology-and-innovation-in-education-at-the-world-bank World Bank. (2023) Cost-Effective Approaches to Improve Global Learning. Retrieved June 16, 2023, from https://thedocs.worldbank.org/en/doc/231d98251cf326922518be0cbe306fdc​ -0200022023/related/GEEAP-Report-Smart-Buys-2023-final.pdf World Bank. (2023). Disability Inclusion Overview. Retrieved March 31, 2023, from https://www.worldbank.org/en/topic/disability World Bank. (2023). The Invisible Toll of COVID-19 on Learning. https://documents1.worldbank​ .org/curated/en/099062323023530087/pdf/P179556020fd80030087730cbc843df07de.pdf World Health Organization. (2022). Global Report on Assistive Technology. (n.d.). Retrieved March 31, 2023, from https://www.who.int/publications-detail-redirect/9789240049451 World Health Organization (2023). Disability. World Health Organization. https://www.who.int/news-room/fact-sheets/detail/disability-and-health Yarrow, N., Khairina, N., Cilliers, J., & Dini, I. (2022). The Digital Future of Teacher Training in Indonesia. http://hdl.handle.net/10986/37218 Yarrow, N., Pfutze, T., Yoo, J., & Kim, H. EdTech in COVID Korea: Learning with Inequality (World Bank Group Korea Office Innovation and Technology Notes). World Bank. https://thedocs.worldbank.org/en/doc/286e9d2e22dd4122f15249083ca84772-0200022022/ original/EdTech-paper-4-29.pdf Yarrow, N., Pynnonen, L., Song, C., Bhardwaj, R., & Spiezio, M. (forthcoming). Unmet Needs: Use of Assistive Education Technologies to Support Children with Visual and Hearing Difficulties in the East Asia and Pacific Region. Yi, H., Song, Y., Liu, C., Huang, X., Zhang, L., Bai, Y., Ren, B., Shi, Y., Loyalka, P., Chu, J., & Rozelle, S. (2015). Giving kids a head start: The impact and mechanisms of early commitment of financial aid on poor students in rural China. Journal of Development Economics, 113, 1–15. https://doi.org/10.1016/j.jdeveco.2014.11.002 Zhang, L., Lai, F., Pang, X., Yi, H., & Rozelle, S. (2013). The impact of teacher training on teacher and student outcomes: Evidence from a randomised experiment in Beijing migrant schools. Journal of Development Effectiveness, 5(3), 339–358. Zhang, S. (2020, October 27). GET2020. Breaking Down Dual–Teacher Model’s Prospect Driven by National Distance Learning. JMDEDU. https://en.jmdedu.com/Article/542 Zilz, W., & Pang, Y. (2021). Application of assistive technology in inclusive classrooms. Disability and Rehabilitation: Assistive Technology, 16(7), 684–686. https://doi.org/10.1080/17483107.2019.1695963 40 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Annex 1 Annex Table 1 List of Included Studies for Figure 1 Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Sean Sylvia, Renfu Luo, Linxiu Zhang, Yaojiang Shi, Alexis Medina, Scott Rozelle. 2013. Do you get what you pay Sean et al. for with school-based health programs? Evidence from Non-tech –0.11 0.16 (2013) a child nutrition experiment in rural China, Economics of Education Review, Volume 37, Pages 1–12, ISSN 0272- 7757, https://doi.org/10.1016/j.econedurev.2013.07.003. Xinxin Chen, Chengfang Liu, Linxiu Zhang, Yaojiang Shi, Scott Rozelle. 2010. Does taking one step back get you two steps forward? Grade retention and Chen et al. school performance in poor areas in rural China, Non-tech –0.10 0.05 (2010) International Journal of Educational Development, Volume 30, Issue 6, Pages 544–559, ISSN 0738-0593, https://doi.org/10.1016/j.ijedudev.2009.12.002. Prashant Loyalka, Chengfang Liu, Yingquan Song, Hongmei Yi, Xiaoting Huang, Jianguo Wei, Linxiu Zhang, Yaojiang Shi, James Chu, Scott Rozelle. 2013. Loyalka et al. Can information and counseling help students from Non-tech –0.07 0.09 (2013) poor rural areas go to high school? Evidence from China, Journal of Comparative Economics, Volume 41, Issue 4, Pages 1012–1025, ISSN 0147-5967, https://doi.org/10.1016/j.jce.2013.06.004. Mo, D., Bai, Y., Shi, Y., Abbey, C., Zhang, L., Rozelle, S., & Loyalka, P. (2020). Institutions, implementation, and Mo et al. program effectiveness: Evidence from a randomized EdTech –0.07 0.14 (2020) evaluation of computer-assisted learning in rural China. Journal of Development Economics, 146, 102487. https://doi.org/10.1016/j.jdeveco.2020.102487. Ma, Yue., Zhang, Markus., Rule, Andrew., Loyalka, Prashant., Wang, Min., Rozelle, Scott. (forthcoming). Ma et al. Adaptive versus Non-Adaptive Computer Assisted EdTech –0.05 0.07 (forthcoming) Learning: A Mixed-Methods Analysis in Rural China. REAP Working Paper. Felipe Barrera-Osorio and Deon Filmer. Barrera- 2015. Incentivizing schooling for learning: Osorio and Evidence on the impact of alternative targeting Non-tech –0.04 0.16 Filmer (2015) approaches. University of Wisconsin Press. DOI: https://doi.org/10.3368/jhr.51.2.0114-6118R1. Feng et al. (2021). The Effectiveness of adaptive Feng et al. computer assisted learning in rural China. REAP EdTech –0.04 0.53 (2021) Working Paper. Ma, Yue., Zhang, Markus., Rule, Andrew., Loyalka, Prashant., Wang, Min., Rozelle, Scott. (forthcoming). Ma et al. Adaptive versus Non-Adaptive Computer Assisted EdTech –0.03 0.07 (forthcoming) Learning: A Mixed-Methods Analysis in Rural China. REAP Working Paper. 41 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech –0.03 0.12 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Max Kleiman-Weiner, Renfu Luo, Linxiu Zhang, Yaojiang Shi, Alexis Medina, Scott Rozelle. 2013. Eggs versus Kleiman- chewable vitamins: Which intervention can increase Weiner et al. Non-tech –0.03 0.00 nutrition and test scores in rural China? China Economic (2013) Review, Volume 24, Pages 165–176, ISSN 1043-951X, https://doi.org/10.1016/j.chieco.2012.12.005. Hongmei Yi, Yingquan Song, Chengfang Liu, Xiaoting Huang, Linxiu Zhang, Yunli Bai, Baoping Ren, Yaojiang Shi, Prashant Loyalka, James Chu, Scott Rozelle, Yi et al. 2015. Giving kids a head start: The impact and Non-tech –0.02 0.12 (2015) mechanisms of early commitment of financial aid on poor students in rural China, Journal of Development Economics, Volume 113, Pages 1–15, ISSN 0304-3878, https://doi.org/10.1016/j.jdeveco.2014.11.002. Prashant Loyalka, Chengfang Liu, Yingquan Song, Hongmei Yi, Xiaoting Huang, Jianguo Wei, Linxiu Zhang, Yaojiang Shi, James Chu, Scott Rozelle. 2013. Loyalka et al. Can information and counseling help students from Non-tech –0.01 0.09 (2013) poor rural areas go to high school? Evidence from China, Journal of Comparative Economics, Volume 41, Issue 4, Pages 1012–1025, ISSN 0147-5967, https://doi.org/10.1016/j.jce.2013.06.004. Feng et al. (2021). The Effectiveness of adaptive Feng et al. computer assisted learning in rural China. REAP EdTech 0.00 0.49 (2021) Working Paper. Mo,D., Bai, Yu., Boswell, Matthew., Rozelle, Scott. 2016. Mo et al. Evaluating the Effectiveness of Computers as Tutors EdTech 0.00 0.12 (2016) in China, 3ie Impact Evaluation Report 41. New Delhi: International Initiative for Impact Evaluation (3ie). Filmer, Deon, and Norbert Schady. “School Enrollment, Filmer & Selection and Test Scores.” Policy Research Working Schady Paper. World Bank, 2009. Non-tech 0.00 0.12 (2009) https://openknowledge.worldbank.org/bitstream/ handle/10986/4190/WPS4998.pdf. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.00 0.10 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Filmer, Deon, and Norbert Schady. 2014. The Medium- Filmer & Term Effects of Scholarships in a Low-Income Country. Schady Non-tech 0.01 0.11 Journal of Human Resources, Vol. 49, Issue. 3: page: (2014) 663–694. DOI: https://doi.org/10.3368/jhr.49.3.663. 42 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Mo, Di, Swinnen, Johan, Zhang, Linxiu, Yi, Hongmei Qu, Qinghe, Boswell, Matthew, & Rozelle, Scott. (2013). Can Mo et al. one-to-one computing narrow the digital divide and the EdTech 0.01 0.24 (2013) educational gap in China? The case of Beijing migrant schools. World Development 46: 14–29. Lai, Fang, Luo, Renfu, Zhang, Linxiu, Huang, Xinzhe, & Rozelle, Scott (2015). Does computer-assisted Lai et al. learning improve learning outcomes? Evidence from a EdTech 0.01 0.10 (2015) randomized experiment in migrant schools in Beijing. Economics of Education Review 47: 34–48. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.01 0.08 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Ree, Joppe de, Karthik Muralidharan, Menno Pradhan, and Halsey Rogers. “Double for Nothing? de Ree Experimental Evidence on an Unconditional Teacher Non-tech 0.01 0.05 (2018) Salary Increase in Indonesia*.” The Quarterly Journal of Economics 133, no. 2 (May 1, 2018): 993–1039. https://doi.org/10.1093/qje/qjx040. Ma, Yue, Cody Abbey, Derek Hu, Oliver Lee, Weiting Hung, Xinwu Zhang, Chiayuan Chang, Chyi-In Wu, and Scott Rozelle. “The Impact of Computer Assisted Ma et al. Learning on Rural Taiwanese Children: Evidence from a EdTech 0.02 0.12 (2020b) Randomized Experiment.” REAP Working Paper, 2020. https://fsi-live.s3.us-west-1.amazonaws.com/s3fs-public/ taiwan_o-cal_working_paper_2020oct.pdf. Li, H., Liu, Z., Yang, F., & Yu, L. (2023). The Impact of Li et al. Computer-Assisted Instruction on Student Performance: EdTech 0.02 0.36 (2023) Evidence from the Dual-Teacher Program. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4360827. Ho Lun Wong, Yaojiang Shi, Renfu Luo, Linxiu Zhang & Scott Rozelle. 2014. Improving the Health and Education Wong et al. of Elementary Schoolchildren in Rural China: Iron Non-tech 0.02 0.10 (2014) Supplementation Versus Nutritional Training for Parents, The Journal of Development Studies, 50:4, 502–519, DOI: https://doi.org/10.1080/00220388.2013.866223. Filmer, Deon, and Norbert Schady. “School Enrollment, Filmer & Selection and Test Scores.” Policy Research Working Schady Paper. World Bank, 2009. Non-tech 0.02 0.12 (2009) https://openknowledge.worldbank.org/bitstream/ handle/10986/4190/WPS4998.pdf. Renfu Luo, Yaojiang Shi, Linxiu Zhang, Chengfang Liu, Scott Rozelle, Brian Sharbono, Ai Yue, Qiran Zhao and Reynaldo Martorell. 2012. Nutrition and Educational Luo et al. Performance in Rural China’s Elementary Schools: Non-tech 0.02 0.07 (2012) Results of a Randomized Control Trial in Shaanxi Province. Economic Development and Cultural Change, Vol. 60, No. 4 (July 2012), pp. 735–772. 43 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level FUTOSHI YAMAUCHI. 2014. An alternative estimate of school-based management impacts on students’ Yamauchi achievements: evidence from the Philippines. Journal Non-tech 0.03 0.14 (2014) of Development Effectiveness, Vol. 6, No. 2, 97–110, http://dx.doi.org/10.1080/19439342.2014.906485. Mo, Di, Renfu Luo, Chengfang Liu, Huiping Zhang, Linxiu Zhang, Alexis Medina, and Scott Rozelle. Text Messaging Mo et al. and Its Impacts on the Health and Education of the EdTech 0.03 0.16 (2014b) Poor: Evidence from a Field Experiment in Rural China. World Development 64 (December 1, 2014): 766–80. https://doi.org/10.1016/j.worlddev.2014.07.015. Mo,D., Bai, Yu., Boswell, Matthew., Rozelle, Scott. 2016. Mo et al. Evaluating the Effectiveness of Computers as Tutors EdTech 0.03 0.10 (2016) in China, 3ie Impact Evaluation Report 41. New Delhi: International Initiative for Impact Evaluation (3ie). Ma, Yue, Fairlie, Robert. W., Loyalka, Prashant, Rozelle, Scott, & National Bureau of Economic Research (2020). Ma et al. Isolating the “tech” from EdTech: Experimental evidence EdTech 0.03 0.08 (2020a) on computer assisted learning in China. NBER Working Paper No. 26953. National Bureau of Economic Research. Yi, Hongmei, Di Mo, Huan Wang, Qiufeng Gao, Yaojiang Shi, Paiou Wu, Cody Abbey, and Scott Rozelle. “Do Yi et al. Resources Matter? Effects of an In-Class Library Project Non-tech 0.03 0.84 (2018) on Student Independent Reading Habits in Primary Schools in Rural China.” Reading Research Quarterly 54, no. 3 (July 2019): 383–411. https://doi.org/10.1002/rrq.238. Filmer, Deon, and Norbert Schady. 2014. The Medium- Filmer & Term Effects of Scholarships in a Low-Income Country. Schady Non-tech 0.03 0.12 Journal of Human Resources, Vol. 49, Issue. 3: page: (2014) 663–694. DOI: https://doi.org/10.3368/jhr.49.3.663. Renfu Luo, Yaojiang Shi, Linxiu Zhang, Chengfang Liu, Scott Rozelle, Brian Sharbono, Ai Yue, Qiran Zhao and Reynaldo Martorell. 2012. Nutrition and Educational Luo et al. Performance in Rural China’s Elementary Schools: Non-tech 0.03 0.01 (2012) Results of a Randomized Control Trial in Shaanxi Province. Economic Development and Cultural Change, Vol. 60, No. 4 (July 2012), pp. 735–772. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.04 0.12 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Bai, Yu, Mo, Di, Zhang, Linxiu., Boswell, Matthew, & Bai et al. Rozelle, Scott. (2016). The impact of integrating ICT with EdTech 0.05 0.08 (2016) teaching: Evidence from a randomized controlled trial in rural schools in China. Computers & Education 96, 1–14. 44 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Macdonald, Kevin Alan David, Sally Ann Brinkman, Wendy Jarvie, Myrna Machuca-Sierra, Kristen Andrew Mcdonall, Siosiana Tapueluelu, and Binh Thanh Vu. “Pedagogy versus School Readiness: Macdonald et The Impact of a Randomized Reading Instruction Non-tech 0.05 0.06 al. (2017) Intervention and Community-Based Playgroup Intervention on Early Grade Reading Outcomes in Tonga.” SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, January 18, 2017. https://papers.ssrn.com/abstract=2901756. Sean Sylvia, Renfu Luo, Linxiu Zhang, Yaojiang Shi, Alexis Medina, Scott Rozelle. 2013. Do you get what you pay Sean et al. for with school-based health programs? Evidence from Non-tech 0.05 0.17 (2013) a child nutrition experiment in rural China, Economics of Education Review, Volume 37, Pages 1–12, ISSN 0272- 7757, https://doi.org/10.1016/j.econedurev.2013.07.003. Hasan, Amer, Haeil Jung, Angela Kinnell, Amelia Maika, Nozomi Nakajima, and Menno Pradhan. “Understanding Hasan et al. the Longer-Term Impact of Improving Access to Non-tech 0.05 0.12 (2019) Preschool Education in Rural Indonesia.” Policy Research Working Paper. World Bank, 2019. https://papers.ssrn​ .com/sol3/papers.cfm?abstract_id=3490314. FUTOSHI YAMAUCHI. 2014. An alternative estimate of school-based management impacts on students’ Yamauchi achievements: evidence from the Philippines. Journal Non-tech 0.05 0.13 (2014) of Development Effectiveness, Vol. 6, No. 2, 97–110, http://dx.doi.org/10.1080/19439342.2014.906485. FUTOSHI YAMAUCHI. 2014. An alternative estimate of school-based management impacts on students’ Yamauchi achievements: evidence from the Philippines. Journal Non-tech 0.06 0.12 (2014) of Development Effectiveness, Vol. 6, No. 2, 97–110, http://dx.doi.org/10.1080/19439342.2014.906485. Lai, Fang, Zhang, Linxiu, Bai, Yu, Liu, Chengfang, Shi, Yaojiang, Chang, Fang, & Rozelle, Scott (2016). More Lai et al. is not always better: evidence from a randomised EdTech 0.06 0.18 (2016) experiment of computer-assisted learning in rural minority schools in Qinghai. Journal of Development Effectiveness 8, no. 4: 449–472. Ma, Yue, Fairlie, Robert. W., Loyalka, Prashant, Rozelle, Scott, & National Bureau of Economic Research (2020). Ma et al. Isolating the “tech” from EdTech: Experimental evidence EdTech 0.06 0.08 (2020a) on computer assisted learning in China. NBER Working Paper No. 26953. National Bureau of Economic Research. Brinkman, Sally Anne, Amer Hasan, Haeil Jung, Angela Kinnell, and Menno Pradhan. “The Impact of Expanding Brinkman et Access to Early Childhood Education Services in Rural Non-tech 0.06 0.12 al. (2017) Indonesia.” Journal of Labor Economics 35, no. S1 (July 2017): S305–35. https://doi.org/10.1086/691278. 45 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Ama Baafra Abeberese, Todd J. Kumler, Leigh L. Linden. 2014. Improving Reading Skills by Encouraging Children Abeberese et to Read in School: A Randomized Evaluation of the Sa Non-tech 0.06 0.06 al. (2014) Aklat Sisikat Reading Program in the Philippines. NBER Working Paper. FUTOSHI YAMAUCHI. 2014. An alternative estimate of school-based management impacts on students’ Yamauchi achievements: evidence from the Philippines. Journal Non-tech 0.07 0.14 (2014) of Development Effectiveness, Vol. 6, No. 2, 97–110, http://dx.doi.org/10.1080/19439342.2014.906485. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.07 0.09 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Gaduh et al. Teachers in Remote Areas. Research on Non-tech 0.07 0.08 (2020) Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.07 0.09 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Mo,D., Bai, Yu., Boswell, Matthew., Rozelle, Scott. 2016. Mo et al. Evaluating the Effectiveness of Computers as Tutors EdTech 0.08 0.08 (2016) in China, 3ie Impact Evaluation Report 41. New Delhi: International Initiative for Impact Evaluation (3ie). Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.08 0.13 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.08 0.12 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.08 0.11 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.08 0.10 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. 46 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Gaduh et al. Teachers in Remote Areas. Research on Blended 0.09 0.08 (2020) Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035. Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Gaduh et al. Teachers in Remote Areas. Research on Non-tech 0.09 0.08 (2020) Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035. Lai, Fang, Zhang, Linxiu, Bai, Yu, Liu, Chengfang, Shi, Yaojiang, Chang, Fang, & Rozelle, Scott (2016). More Lai et al. is not always better: evidence from a randomised EdTech 0.10 0.14 (2016) experiment of computer-assisted learning in rural minority schools in Qinghai. Journal of Development Effectiveness 8, no. 4: 449–472. Ho Lun Wong, Yaojiang Shi, Renfu Luo, Linxiu Zhang & Scott Rozelle. 2014. Improving the Health and Education Wong et al. of Elementary Schoolchildren in Rural China: Iron Non-tech 0.10 0.10 (2014) Supplementation Versus Nutritional Training for Parents, The Journal of Development Studies, 50:4, 502–519, DOI: https://doi.org/10.1080/00220388.2013.866223. Hasan, Amer, Haeil Jung, Angela Kinnell, Amelia Maika, Nozomi Nakajima, and Menno Pradhan. “Understanding Hasan et al. the Longer-Term Impact of Improving Access to Non-tech 0.10 0.12 (2019) Preschool Education in Rural Indonesia.” Policy Research Working Paper. World Bank, 2019. https://papers.ssrn​ .com/sol3/papers.cfm?abstract_id=3490314. Li, H., Liu, Z., Yang, F., & Yu, L. (2023). The Impact of Li et al. Computer-Assisted Instruction on Student Performance: EdTech 0.10 0.28 (2023) Evidence from the Dual-Teacher Program. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4360827. Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Gaduh et al. Teachers in Remote Areas. Research on Blended 0.11 0.06 (2020) Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035. Fang Lai , Linxiu Zhang , Xiao Hu , Qinghe Qu , Yaojiang Shi , Yajie Qiao ,Matthew Boswell, Scott Rozelle (2013) Computer assisted learning as extracurricular Lai et al. tutor? Evidence from a randomised experiment EdTech 0.12 0.12 (2013) in rural boarding schools in Shaanxi, Journal of Development Effectiveness, 5:2, 208–231, DOI: https://doi.org/10.1080/19439342.2013.780089. Yihua Yang, Linxiu Zhang, Junxia Zeng, Xiaopeng Pang, Fang Lai, Scott Rozelle, Computers and the academic Yang et al. performance of elementary school-aged girls in China's EdTech 0.12 0.10 (2013) poor communities, Computers & Education, Volume 60, Issue 1, 2013, Pages 335–346, ISSN 0360-1315, https://doi.org/10.1016/j.compedu.2012.08.011. 47 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Max Kleiman-Weiner, Renfu Luo, Linxiu Zhang, Yaojiang Shi, Alexis Medina, Scott Rozelle. 2013. Eggs versus Kleiman- chewable vitamins: Which intervention can increase Weiner et al. Non-tech 0.12 0.00 nutrition and test scores in rural China? China Economic (2013) Review, Volume 24, Pages 165–176, ISSN 1043-951X, https://doi.org/10.1016/j.chieco.2012.12.005. Linxiu Zhang, Fang Lai, Xiaopeng Pang, Hongmei Yi & Scott Rozelle (2013). The impact of teacher training Zhang et al. on teacher and student outcomes: evidence from a Non-tech 0.12 0.41 (2013) randomised experiment in Beijing migrant schools, Journal of Development Effectiveness, 5:3, 339–358, DOI: https://doi.org/10.1080/19439342.2013.807862. FUTOSHI YAMAUCHI and YANYAN LIU. 2013. Impacts Yamauchi of an Early Stage Education Intervention on Students’ and Liu Learning Achievement: Evidence from the Philippines. Non-tech 0.12 0.04 (2013) Journal of Development Studies, Vol. 49, No. 2, 208–222. Brinkman, Sally Anne, Amer Hasan, Haeil Jung, Angela Kinnell, and Menno Pradhan. 2017. “The Impact of Brinkman et Expanding Access to Early Childhood Education Services Non-tech 0.13 0.14 al. (2017) in Rural Indonesia.” Journal of Labor Economics 35, no. S1: S305–35. https://doi.org/10.1086/691278. Ama Baafra Abeberese, Todd J. Kumler, Leigh L. Linden. 2014. Improving Reading Skills by Encouraging Children Abeberese et to Read in School: A Randomized Evaluation of the Sa Non-tech 0.13 0.10 al. (2014) Aklat Sisikat Reading Program in the Philippines. NBER Working Paper. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.13 0.20 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Mo, Di, Renfu Luo, Chengfang Liu, Huiping Zhang, Linxiu Zhang, Alexis Medina, and Scott Rozelle. Text Messaging Mo et al. and Its Impacts on the Health and Education of the EdTech 0.14 0.14 (2014b) Poor: Evidence from a Field Experiment in Rural China. World Development 64 (December 1, 2014): 766–80. https://doi.org/10.1016/j.worlddev.2014.07.015. Yihua Yang, Linxiu Zhang, Junxia Zeng, Xiaopeng Pang, Fang Lai, Scott Rozelle, Computers and the academic Yang et al. performance of elementary school-aged girls in China's EdTech 0.14 0.08 (2013) poor communities, Computers & Education, Volume 60, Issue 1, 2013, Pages 335–346, ISSN 0360-1315, https://doi.org/10.1016/j.compedu.2012.08.011. Nie, J., Pang, X., Wang, L., Rozelle, S., & Sylvia, S. (2020). Seeing Is Believing: Experimental Evidence on the Impact Nie et al. of Eyeglasses on Academic Performance, Aspirations, Non-tech 0.14 0.15 (2020) and Dropout among Junior High School Students in Rural China. Economic Development and Cultural Change, 68(2), 335–355. https://doi.org/10.1086/700631. 48 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.15 0.13 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Lai, Fang, Luo, Renfu, Zhang, Linxiu, Huang, Xinzhe, & Rozelle, Scott (2015). Does computer-assisted Lai et al. learning improve learning outcomes? Evidence from a EdTech 0.15 0.08 (2015) randomized experiment in migrant schools in Beijing. Economics of Education Review 47: 34–48. Mo, Di, Bai, Yu, Shi, Yaojiang, Abbey, Cody, Zhang, Linxiu, Rozelle, Scott, & Loyalka, Prashant (2020). Mo et al. Institutions, implementation, and program effectiveness: EdTech 0.16 0.14 (2020) Evidence from a randomized evaluation of computer- assisted learning in rural China. Journal of Development Economics 146: 102487. Mo, Di, Zhang, Linxiu, Luo, Renfu, Qu, Qinghe, Huang, Weiming, Wang, Jiafu, Qiao, Yajie, Boswell, Matthew, Mo et al. & Rozelle, Scott (2014). Integrating computer-assisted EdTech 0.16 0.12 (2014a) learning into a regular curriculum: Evidence from a randomised experiment in rural schools in Shaanxi. Journal of Development Effectiveness 6(3): 300–323. Prashant Loyalka, Sean Sylvia, Chengfang Liu, James Chu, Yaojiang Shi. 2019. Pay by Design: Teacher Loyalka et al. Performance Pay Design and the Distribution of Student Non-tech 0.17 0.12 (2019) Achievement. Journal of Labor Economics, Vol. 37, Number 3. DOI: https://doi.org/10.1086/702625. Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Gaduh et al. Teachers in Remote Areas. Research on Blended 0.17 0.08 (2020) Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035. Mo, Di, Swinnen, Johan, Zhang, Linxiu, Yi, Hongmei Qu, Qinghe, Boswell, Matthew, & Rozelle, Scott. (2013). Can Mo et al. one-to-one computing narrow the digital divide and the EdTech 0.17 0.20 (2013) educational gap in China? The case of Beijing migrant schools. World Development 46: 14–29. Fafchamps, Marcel & Di Mo (2018). Peer effects in Fafchamps & computer assisted learning: evidence from a randomized EdTech 0.17 0.18 Mo (2018) experiment. Experimental Economics 21: 355–382. https://doi.org/10.1007/s10683-017-9538-z. Felipe Barrera-Osorio and Deon Filmer. Barrera- 2015. Incentivizing schooling for learning: Osorio and Evidence on the impact of alternative targeting Non-tech 0.17 0.18 Filmer (2015) approaches. University of Wisconsin Press. DOI: https://doi.org/10.3368/jhr.51.2.0114-6118R1. 49 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Linxiu Zhang, Fang Lai, Xiaopeng Pang, Hongmei Yi & Scott Rozelle (2013). The impact of teacher training Zhang et al. on teacher and student outcomes: evidence from a Non-tech 0.17 0.45 (2013) randomised experiment in Beijing migrant schools, Journal of Development Effectiveness, 5:3, 339–358, DOI: https://doi.org/10.1080/19439342.2013.807862. Mo,D., Bai, Yu., Boswell, Matthew., Rozelle, Scott. 2016. Mo et al. Evaluating the Effectiveness of Computers as Tutors EdTech 0.18 0.12 (2016) in China, 3ie Impact Evaluation Report 41. New Delhi: International Initiative for Impact Evaluation (3ie). Nicola Bianchi, Yi Lu, Hong Song, The effect of computer-assisted learning on students’ long-term Bianchi et al. development, Journal of Development Economics, EdTech 0.18 0.16 (2022) Volume 158, 2022, 102919, ISSN 0304-3878, https://doi.org/10.1016/j.jdeveco.2022.102919. Yihua Yang, Linxiu Zhang, Junxia Zeng, Xiaopeng Pang, Fang Lai, Scott Rozelle, Computers and the academic Yang et al. performance of elementary school-aged girls in China's EdTech 0.19 0.14 (2013) poor communities, Computers & Education, Volume 60, Issue 1, 2013, Pages 335–346, ISSN 0360-1315, https://doi.org/10.1016/j.compedu.2012.08.011. Lai, Fang, Zhang, Linxiu, Bai, Yu, Liu, Chengfang, Shi, Yaojiang, Chang, Fang, & Rozelle, Scott (2016). More Lai et al. is not always better: evidence from a randomised EdTech 0.19 0.12 (2016) experiment of computer-assisted learning in rural minority schools in Qinghai. Journal of Development Effectiveness 8, no. 4: 449–472. Chen, X., Shi, Y., Yi, H., Zhang, L., Mo, D., Chu, J., Loyalka, P., & Rozelle, S. (2013). The Impact of a Senior High Chen et al. School Tuition Relief Program on Poor Junior High Non-tech 0.19 0.12 (2013) School Students in Rural China. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2374361. Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (2020). Scores, Camera, Action? Incentivizing Gaduh et al. Teachers in Remote Areas. Research on Blended 0.20 0.08 (2020) Improving Systems of Education (RISE). https://doi.org/10.35489/BSG-RISE-WP_2020/035. Tang, Bin, Te-Tien Ting, Chyi-In Wu, Yue Ma, Di Mo, Wei-Ting Hung, and Scott Rozelle. “The Impact of Online Tang et al. Computer Assisted Learning at Home for Disadvantaged EdTech 0.20 0.16 (2018) Children in Taiwan: Evidence from a Randomized Experiment.” Sustainability 12, no. 23 (January 2020): 10092. https://doi.org/10.3390/su122310092. Lai, Fang, Luo, Renfu, Zhang, Linxiu, Huang, Xinzhe, & Rozelle, Scott (2015). Does computer-assisted Lai et al. learning improve learning outcomes? Evidence from a EdTech 0.20 0.14 (2015) randomized experiment in migrant schools in Beijing. Economics of Education Review 47: 34–48. 50 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Fafchamps, Marcel & Di Mo (2018). Peer effects in Fafchamps & computer assisted learning: evidence from a randomized EdTech 0.20 0.25 Mo (2018) experiment. Experimental Economics 21: 355–382. https://doi.org/10.1007/s10683-017-9538-z. Lai, Fang, Luo, Renfu, Zhang, Linxiu, Huang, Xinzhe, & Rozelle, Scott (2015). Does computer-assisted Lai et al. learning improve learning outcomes? Evidence from a EdTech 0.22 0.14 (2015) randomized experiment in migrant schools in Beijing. Economics of Education Review 47: 34–48. Clark, Andrew E., Nong, Huifu, Zhu, Hongjia, & Zhu, Clark et al. Rong. (2021). Compensating for academic loss: Online EdTech 0.22 0.02 (2021) learning and student performance during the covid-19 pandemic. China Economic Review: 101629. Nicola Bianchi, Yi Lu, Hong Song, The effect of computer-assisted learning on students’ long-term Bianchi et al. development, Journal of Development Economics, EdTech 0.23 0.13 (2022) Volume 158, 2022, 102919, ISSN 0304-3878, https://doi.org/10.1016/j.jdeveco.2022.102919. Lai, Fang, Zhang, Linxiu, Bai, Yu, Liu, Chengfang, Shi, Yaojiang, Chang, Fang, & Rozelle, Scott (2016). More Lai et al. is not always better: evidence from a randomised EdTech 0.23 0.14 (2016) experiment of computer-assisted learning in rural minority schools in Qinghai. Journal of Development Effectiveness 8, no. 4: 449–472. Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. B. (2020). Efficacy of video-based Chen et al. teacher professional development for increasing Blended 0.24 0.03 (2020) classroom discourse and student learning. Journal of the Learning Sciences, 29(4–5), 642–680. https://doi.org/10.1080/10508406.2020.1783269. Paul Glewwe, Albert Park, Meng Zhao, 2016. A better vision for development: Eyeglasses and Glewwe et al. academic performance in rural primary schools Non-tech 0.26 0.22 (2016) in China, Journal of Development Economics, Volume 122, Pages 170–182, ISSN 0304-3878, https://doi.org/10.1016/j.jdeveco.2016.05.007. Mo, Di., Zhang, Linxiu, Wang, J., Huang, Wang, Shi, Mo et al. Yaojiang., Boswell, Matthew., & Rozelle, Scott. (2015). (2015) Persistence of learning gains from computer assisted EdTech 0.26 0.16 combined learning: Experimental evidence from China. Journal of Computer Assisted Learning 31(6), 562–581. Paul Glewwe, Albert Park, Meng Zhao, 2016. A better vision for development: Eyeglasses and Glewwe et al. academic performance in rural primary schools Non-tech 0.27 0.24 (2016) in China, Journal of Development Economics, Volume 122, Pages 170–182, ISSN 0304-3878, https://doi.org/10.1016/j.jdeveco.2016.05.007. 51 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Lee, S., Lee, J.-H., & Jeong, Y. (2023). The Effects of Digital Textbooks on Students’ Academic Lee et al. Performance, Academic Interest, and Learning EdTech 0.28 0.33 (2023) Skills. Journal of Marketing Research, 0(0). https://doi.org/10.1177/00222437221130712. Macdonald, Kevin Alan David, Sally Ann Brinkman, Wendy Jarvie, Myrna Machuca-Sierra, Kristen Andrew Mcdonall, Siosiana Tapueluelu, and Binh Thanh Vu. “Pedagogy versus School Readiness: Macdonald et The Impact of a Randomized Reading Instruction Non-tech 0.29 0.20 al. (2017) Intervention and Community-Based Playgroup Intervention on Early Grade Reading Outcomes in Tonga.” SSRN Scholarly Paper. Rochester, NY: Social Science Research Network, January 18, 2017. https://papers.ssrn.com/abstract=2901756. Lee, S., Lee, J.-H., & Jeong, Y. (2023). The Effects of Digital Textbooks on Students’ Academic Lee et al. Performance, Academic Interest, and Learning EdTech 0.32 0.37 (2023) Skills. Journal of Marketing Research, 0(0). https://doi.org/10.1177/00222437221130712. Zhang, Yiyun, & Zhou, Xinlin. (2016). Building knowledge Zhang et al. structures by testing helps children with mathematical EdTech 0.34 0.33 (2016) learning difficulty. Journal of Learning Disabilities 49(2), 166–175. Paul Glewwe, Albert Park, Meng Zhao, 2016. A better vision for development: Eyeglasses and academic Glewwe et al. performance in rural primary schools in China, Journal Non-tech 0.41 0.24 (2016) of Development Economics, Volume 122, Pages 170–182, ISSN 0304-3878, https://doi.org/10.1016/​ j.jdeveco.2016.05.007. Yu Bai, Bin Tang, Boya Wang, Di Mo, Linxiu Zhang, Scott Rozelle, Emma Auden, Blake Mandell, Impact of online computer assisted learning on education: Bai et al. Experimental evidence from economically vulnerable EdTech 0.48 0.24 (2023) areas of China, Economics of Education Review, Volume 94, 2023, 102385, ISSN 0272-7757, https://doi.org/10.1016/j.econedurev.2023.102385. Habtamu Fuje, Prateek Tandon. 2018. When do in- Habtamu service teacher training and books improve student and Prateek achievement? Experimental evidence from Mongolia. Non-tech 0.62 0.00 (2018) The World Bank Review of Development Economics. DOI: https://doi.org/10.1111/rode.12387. Ito, Hirotake, Keiko Kasai, Hiromu Nishiuchi, and Makiko Nakamuro. 2019. “Does Computer-Aided Instruction Ito et al. Improve Children’s Cognitive and Non-Cognitive Skills?: EdTech 0.68 0.19 (2019) Evidence from Cambodia.” Discussion papers 19040. Research Institute of Economy, Trade and Industry (RIETI). 52 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Effect size 95% Intervention Study Citation (adjusted, Confidence Mode in S.D.) Level Ito, Hirotake, Keiko Kasai, Hiromu Nishiuchi, and Makiko Nakamuro. 2019. “Does Computer-Aided Instruction Ito et al. Improve Children’s Cognitive and Non-Cognitive Skills?: EdTech 0.76 0.43 (2019) Evidence from Cambodia.” Discussion papers 19040. Research Institute of Economy, Trade and Industry (RIETI). Du, K.; Wang, H.; Ma, Y.; Guan, H.; Rozelle, S.2022. Effect of Eyeglasses on Student Academic Performance: What Du et al. Matters? Evidence from a Randomized Controlled Trial in Non-tech 0.92 0.67 (2022) China. Int. J. Environ. Res. Public Health 2022, 19, 10923. https://doi.org/10.3390/ijerph191710923. Li, Haizheng and Liu, Zhiqiang and Yang, Fanzheng and Yu, Li. 2023. The Impact of Computer-Assisted Li et al. Instruction on Student Performance: Evidence from the EdTech 0.98 0.28 (2023) Dual-Teacher Program. IZA Discussion Paper No. 15944, Available at SSRN: https://ssrn.com/abstract=4360827 or http://dx.doi.org/10.2139/ssrn.4360827. 53 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations Photo: World Bank Flckr 54 Using Education Technology to Improve Student Learning in East Asia Pacific: Promises and Limitations