Use of Information and Communication Technologies in Education Effectively integrating technology in under- resourced education systems Use of Information and Communication Technologies in Education Effectively integrating technology in under-resourced education systems Authors Kumar Vivek1 Pradyumna Bhattacharjee2 Disclaimer This work is a product of the staff of The World Bank, with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. All the errors are responsibility of the authors. Attribution - Please cite the work as follows: Vivek, Kumar and Bhattacharjee, Pradyumna. 2021. “Use of Information and Communication Technologies in Education”. Washington, D.C.: World Bank. 1 The World Bank. Email: kumarvivek@worldbank.org 2 The World Bank. Email: pbhattacharjee2@worldbank.org Contents Executive Summary................................................................................................................ 4 1. Introduction .................................................................................................................... 5 2. Technology for what ? ...................................................................................................... 5 3. Technology as a potent tool to address challenges in under-resourced environments . 6 3.1. Assessing Learning ..................................................................................................... 6 3.2. Resolving Barriers to Learning .................................................................................... 9 3.2.1. Prepared Learners .............................................................................................. 9 3.2.2. Learning Focused Inputs and Effective Teaching ..................................................12 3.2.3. Skilled Management and Governance .................................................................13 3.3. Aligning Actors ..........................................................................................................20 4. Conclusion and food-for-thought for policymakers .......................................................23 References ............................................................................................................................27 Executive Summary Education systems in under-resourced environments face several challenges, some of them exacerbated by the COVID-19 pandemic. A possible way to address some of the challenges is to apply information and communication technologies. However, effectivel y integrating technology into education systems is a complex task. In this paper, factors contributing to successful integration of technology in education are explored, with a focus on under-resourced contexts. Case studies of successful technology implementation in education systems are discussed and analyzed to identify the factors that d ri ve success. The analysis is framed using the reform strategy offered by The World Development Report 2018 (Learning to Realize Education’s Promise). This is expected to provi d e p ol icymakers and practitioners a way to align their education technology initiatives and strategies wi th the larger education reform agenda. Key lessons identified from the analysis are as follows. First, it is necessary to articulate what precisely does the technology intervention change/enable. Second, it is important to better understand the context to develop technologies and implementation strategies that fit the operating context. Third, it is essential to regularly monitor and evaluate programs and to feed that information into continuously improving design and implementation. Fourth, through the enti re cycle of technology implementation, stakeholders must be consulted, understood, and empowered. However small the intervention, realizing the potential of technology tool s i n education requires keeping in mind the big picture offered by these lessons. 4 1. Introduction Investments to introduce various forms of information and communicati on technol ogies (ICTs) in public school education systems around the world rapidly increased over the l ast two decades and witnessed significant expansion during the COVID-19 pandemic. The classification and magnitude of such investments vary based on what report one chooses to read. Nonetheless, most of these investments have attempted to reproduce hardware - centric educational technology projects that were conceived in, and hence better suited to, resource-rich environments. The evidence on their effectiveness is either limited or largel y mixed. What will it take to succeed, then, when thinking about ICTs i n ed ucation for the long run? 2. Technology for what? “If I were given one hour to save the planet, I would spend 59 minutes defining the problem and one minute resolving it.” – Anonymous Challenging the status quo of school systems is hard. Ideas to re-imagine the system, many involving the use of ICTs, have been proposed over decades. In 1913, Thomas Ed ison, an American inventor, predicted that books would become obsolete in schools and be replaced by motion pictures. Around the same time, Edward Thorndike, a Columbia University educational psychology professor, imagined ‘adaptive textbooks ’ that would change based on individual students’ achievements. Sidney Pressey, an Ohio State University professor in 1933, and later B.F. Skinner, a Harvard University psychology professor in the 1950s, posited and even modeled teaching machines. In 1972, comp uter scientist Alan Kay conceived the Dynabook, “A Personal Computer for Children of All Ages” as yet another idea of personalized learning using what we today know as tablet computers. A lot of these ideas came from the Western world, especially the United States, and constitute the rich and fascinating history of envisioning ICT use in education systems. It is pertinent, therefore, to highlight that ‘technology innovations’ have been a key factor in shaping education systems around the world. The fundamental approach to sol vi ng real- 5 world problems using technology requires sufficient rigor in defining the problem from an end-user perspective , articulating why it’s important, develop i ng i deas and re -i magined processes that may be enabled or expanded/scaled with the use of technology tool s , and ensuring that the end-users are part of the solutioning exercise. The ideas of Edison, Pressey, and Skinner highlighted in the previous paragraph perhaps reflected i ncreasi ng demand for education as the problem and ‘mechanization’ as a way of improving efficiency in an ever-expanding classroom. The ideas of Thorndike and Kay probably identifi ed how- to-teach-at-the-right-level and how to encourage individual exploration as the problem and offered ‘personalization tools ’ as a solution. Before envisioning an ICT-enabled model for addressing the problems facing school education today, it is worthwhile pausing and reflecting on what we are solving for in today’s regional context. Several regions around the world are faced with challenges of access, quality, and inclusion in education. In response, the Worl d Devel opment Report (WDR) 2018 proposed a reform agenda with three complimentary strategies to realize education’s promise. This paper attempts to articulate key problems that need to be solved in executing the three strategic areas of the WDR reform agenda, and how ICT can address some of those. 3. Technology as a potent tool to address challenges in under-resourced environments The World Development Report 2018 ’s proposed reform strategy involves the following: • assessing learning to understand the current state of the system as well as set direction for the reform agenda, • acting on evidence to resolve barriers to learning, and • aligning all actors to make the whole system work for learning. 3.1. Assessing Learning In the past decade and a half, assessments have been central to the discourse on education reform around the world and have been responsib le for b ri nging i nto p ub li c consciousness the issue of low levels of learning in most national education systems. A 6 2012 World Bank report3 highlighted three main drivers of the quality of information produced by an assessment system: the enabling context, system alignment, and assessment quality. Broadly speaking, the enabling context covers policies, resources, and implementation arrangements for assessment activities; system alignment covers the connection between assessments and other components of the education system such as learning goals, standards, curriculum, and teacher training; and assessment quality covers the psychometric quality of the instruments, processes, and procedures used for assessment. Analyzing a country’s assessmen t systems using such criteria offers interesting insights into specific problems/areas that need to be addressed/strengthened and if or how ICT tools might help. For instance, several countries grapple with issues of widespread cheating in high-stakes examinations. In other cases, there are issues with the alignment of national large-scale assessments to the constructs/content they were intended to measure, i.e., the curriculum or learning standards. Test-items/questions and proced ures may al so vary in quality across regions and years, making the results difficult to compare. The process of administering assessments and collecting and collating data may also be p rone to errors at various stages, resulting in unreliable data, ti me -consumi ng analys is, and a slow and ineffective assessment-to-action loop. Some of the efforts to improve assessment systems have used ICT tools to improve the fidelity of the data, including integration of portable devices like mobile phones and tablets into data collection procedures, and using downstream analytical tools to shorten the assessment-to-action cycle. One example of interventions that have targeted system-wid e changes is the ‘big- bang’ assessment reform in the country of Georgia in 2010, wherei n al l school graduation examinations moved to computer-adaptive testing (CAT), a type of testing that adapts to the ability level of the person taking the test. 4 There are several potential benefits of such a shift, ranging from a reduction in cheating (because test takers are shown different questions) and availability of more precise ability-estimates of candidates, to development of extensive item banks well-aligned with national stand ards that can lead to more reliable and comparable results over years. An evaluation by the Georgian Ministry of Education and Science concluded that the computer-adaptive testing in Georgia was more cost-effective compared to administration of the same tests using paper and pencil techniques, and that its introduction had met the goals of l aunching an efficient, fair, and objective approach to student assessment. 3 (Clarke, 2012) 4 (Bakker, 2014) 7 The success of this model in Georgia, however, owes much to a number of enabling factors specific to the ‘context’. The small size of Georgia, and the presence of a wel l -develop ed internet infrastructure in its schools served as key ingredients to think in the d i recti on of technologically advanced assessment delivery modes. A strong commitment from the government to introduce national tests, fund the recurrent costs, and guarantee continuity of operations underpinned the enabling context. The most important success factor, however, was the leadership of a highly competent and experienced testing institution (National Assessment and Examinations Center, NAEC), and its ability to bring school s on - board. NAEC developed the CAT tool using its existing staff, with some training by international consultants in areas such as item calibration and test analysis using software packages. NAEC then conducted extensive outreach campaigns – led by its Director, a wel l- respected and trusted public figure – by visiting all regions and meeting with representatives of virtually all schools involved . The meetings focused on sharing information about registration and score dissemination processes, demonstrating worki ng prototypes of the test delivery system and test items, and, most importantly, convi nc ing schools that the tests would not be used against them and that the outcomes would not be disastrous as many feared. This was complemented by visits of technol ogy sp eci al ists to each school for inspection of computers and connectivity. This spurt of activity was followed up with sharing of information about the test with the public and med ia , setting up of a dedicated Frequently Asked Questions section on the website and social media channels, an open-door policy of interaction with the media, and launching of a web-based practice test. It is worth stressing that while decisions around the ‘right technol ogy’ were important, overall success was rooted in a much more comprehensive anal ysi s of need s and goals, and a rigorous implementation and stakeholder consultation p rocess . In fact, the role of the NAEC in successful implementation cannot be overstated; although, i t was politically not responsible for the decision of administering computer -adapti ve testing, i t took on the onus of engaging the ecosystem through outreach, ensuring stakeholder b uy - in and successful implementation. Another example of efforts to improve nati onal assessment systems is Afghanistan’s Monitoring Standards in Educational Growth (MTEG) program, launched in 2013, whi ch i s designed to assess the learning progression and trends in achievement outcomes of 8 students at key stages of schooling (Classes 3, 6 and 9).5 The program has developed a computer-based assessment for its Class 3 assessments6. 3.2. Resolving Barriers to Learning While assessments tell where one currently is and how far one has moved, this information is only helpful when acted upon. Drawing from an ever-growing body of evidence on what works, the World Development Report 2018 highlighted four key school-level i ngredi ents for learning: prepared learners, effective teaching, learning focused inputs, and the ski l led management and governance that pulls them all together. 3.2.1. Prepared Learners The World Bank has laid out a three-pronged strategy to accelerate progress toward s the goal of ensuring all children have the opportunity to reach their full potential 7: establishi ng an enabling environment; ensuring that the portfolio of programs cover s healthcare, nutrition, child protection, social protection and early learning; and, monitoring and assuring quality through compliance with quality standards and data availability. Similar to the case of assessments in the previous section, using such a framework i n anal yzi ng a country’s Early Childhood Development (ECD) system ensures the primacy of the needs and aspirations of all stakeholders in designing and delivering solutions, surfaces gaps, and thereby, generates insights on whether and ways in which ICT can play a role in sol vi ng the identified challenges. For example, wide range of Government departments delivering similar servi ces d ue to a lack of inter-sectoral coordination mechanisms for designing, deli vering and moni toring services and outcomes may become constraints to an enabling environment. Lack of quality assurance mechanisms may lead to proliferation of poor - qual ity curricul um and learning resources, untrained professionals, and ill-equipped child-care centres that adversely affect children’s growth and learning. 5 (ACER Centre for Global Education Monitoring, 2016) 6 See Box 1 for a detailed description 7 (Neuman & Devercelli, 2013) 9 Use of technology can sometimes provide governments simpler ways to deliver quality ECD interventions at scale. For example, one of the most inexpensive and widely available technologies, the radio, has been used to deliver interactive radio instruction (IRI) targeting the most vulnerable children in remote communities, with very encouraging results. IRI is a distance education system that combines radio broadcasts with active learning to improve educational quality and teaching practices. The first phase of Radio Instruction to Strengthen Education (RISE) project in Zanzibar used IRI to deliver songs, stories and physical activiti es to a group of underserved young children (preschool to Grade 2) and a lightly-trained adult facilitator in Zanzibar’s two lowest -performing districts. The project was led b y Zanz ib ar’s Ministry of Education and Vocational Training (MoEVT). MoEVT started wi th empoweri ng educators through capacity building of its educators in developing interactive audio programs and accompanying instructional and learning materials, p rod ucing audi o and video programs, and conducting monitoring and evaluation activities. These educators could then integrate math, life skills, and English and Kiswahili literacy lessons into comprehensive 30-minute programs. The programs complemented the standard curriculum, engaged listeners through locally based stories and activities, and could support teachers with minimal ECD instructional experience. Early childhood learning centers (TuTu Centers) were then established in areas with poor access to ECD and education, and RISE staff trained local community members as mentors to gui de l earners through the TuTu broadcasts and post-broadcast activities in non-formal settings identified by the community. MoEVT also trained its district officials in IRI implementation, in addition to monitoring and evaluation. The programs were broadcast on the government- sponsored radio station as well as through portable media player technology. The initiative delivered deep impact, exhibited through significant test score gains across all three tested subjects (Kiswahili, Math and English) 8,9. The success of the project once again highlights that the ‘right technology’ i s the one that suits the context; in this case the one that can reach vulnerable children i n remote areas and is easy to use. It also brings to fore another critical element for successful technol ogy - driven interventions: robust change management and empowerment of the last-mile actors. Capacity building at various levels and engagement with the communities not onl y enabled alignment for easy delivery of the project, but also aided the MoEVT’s ability to take over implementation and institutionalize it in just a few years. 8 (World Bank Group, 2017) 9 (EDC, 2015) 10 Other noteworthy models of technology use in ECD are Storyweaver (India) and Ready4K ! (USA). Storyweaver (India) is an online platform that gives child ren and adult -caregivers opportunities to read, print and even discuss levelled stories written in multiple languages, and also allows illustrators, authors and translators to collaborate to create and translate stories. Ready4K!10, an initiative that was implemented in the San Francisco Unified School Di stri ct during 2013-14 and 2015-16, aimed to address the large variations in learning experiences for young children with varying family backgrounds. One study had found that b y the age of four, children in “welfare families” hear about 3 million fewer words than child ren i n “professional families”. Ready4K! intended to bridge this gap by supporting parents of young children to provide a rich home-based learning experience. However, most exi sting parenting based interventions suffered from challenges of low access, high cost and parent dropouts, and therefore, had shown very little success. Given this context, Ready4K! arrived at a smart design through a careful consideration of the context and factors that affect implementation of parental support programs. Firstly, it employed a widely avail ab le and inexpensive technology, i.e. of text messages, as the primary mode of delivery of the intervention. This ensured that the project reached all communities, includi ng the under- resourced Black and Hispanic communities which had the highest drop-out rates from parenting programs. Secondly, Ready4K! broke down the complexity of p arental support into an accessible package of three messages a week – one message wi th a si mpl e b i te- sized step for the parents to follow, along with two messages for encouragement and reinforcement. The program resulted in increased parental involvement at home and school, leading to gains in early literacy. This initiative exhibits the promise of smal l b its of information, and widely available, low-cost and scalable technology solutions i n changing entrenched and complex adult behaviors, echoing the importance of und erstandi ng the context and identifying the right technology for that context. Similar to other challenges in the education systems, technology may not be the answer to the challenges of ECD. However, some right approaches that can complement and enab le activities related to ECD are possible, provided a clear understanding of the context in which ICT tools can help. 10 (York, Loeb, & Doss, 2017) 11 3.2.2. Learning Focused Inputs and Effective Teaching While a major part of the strategy to prepare learners entails focus on early childhood care and education, the strategy includes directing attention to the value of increasing motivation and effort in school and the need for increased and improved opportu ni ties to learn at home. There is a plethora of solutions that seek to provide rich and engaging lessons, activities, resources and support for both the classroom and home, mostly on literacy and numeracy curricula for primary grades and above. They range from tool s that aid reading at home to those that enable sharper learning experiences in school . Some of these tools may be more resource-intensive than others, but there are many that make use of existing and easily accessible technologies. Available in India and Pakistan, Worldreader is a widely-available tool that allows chi ld ren and adults to read stories from mobile devices, thereby increasing the exposure to q ual ity content and support outside the classroom. There have been pilots i n Paki stan that u se SMS technology to deploy short daily assessments. Solutions using SMS and Interacti ve Radio Instruction (IRI) demonstrate the potential of well-designed technology- based tool s to deliver improved learning experiences both in and out of classrooms, parti cularly in low- cost and limited-mobility environments. Another example is that of Mindspark, a mathematics learning program that uses adaptive-learning technology. A study i n India 11 found that students who used Mindspark after-school for 4.5 months, showed gains that were greater than those observed in almost any evaluation of education i nterventions i n poor countries. Technology can also be used to support teachers in improving their instruction. In Uganda, UNICEF’s MobiStation – referred to as a digital ‘school in a box’ – is equipped to faci litate a richer learning experience in both formal classroom and informal communal settings . It does so through a combination of laptop, speaker, and projector rolled into one p ortabl e suitcase. While hardware-assisted systems and software preloaded with content constitute most technology-enabled tools, there are a few examples that go beyond to set up vi rtu al or physical platforms that support content creation, dissemination and consump ti on. The Digital Infrastructure for School Education (DIKSHA) platform in India is one of such platforms. It is an open learning platform with a collection of learning resources in literacy 11 (Muralidharan, Singh, & Ganimian, 2016) 12 and numeracy. The platform allows for creation and publishing of interactive teaching content for learners. The created stories, assessments, worksheets, simulations, games and activities can be accessed through a website and mobile app. All the education content developed on it is linked to a particular curriculum, and can be used by students, p arents, schools and tutors. The platform provides analytics and dashboards that help track ratings, engagement and learning levels of users. With availability in 12 languages and enablement of content creation by teachers, parents and content creators, it represents a step towards democratization of content creation and rating. ICTs and properly developed multimedia materials can also enhance the initial preparation of teachers by providing good training materials, facilitating si mul ati ons, capturi ng and analyzing practice-teaching, bringing world experience into the traini ng i ns tituti on, and training potential teachers in the use of technologies for teaching/learni ng. For examp l e, teachers in the Indian state of Karnataka convene under the umbrella of the Subject Teacher Forum (STF) to engage in discussions related to their subject on online communities and create and share open digital educational content. All of these are tool s that were built for the context, address both content and infrastructure issues, use l ocall y aligned or created content, and have a component of training teachers in using them well. 3.2.3. Skilled Management and Governance At the school/institution level, technologies are crucial in such areas as admissions, student tracking, personnel management, staff development, and facil ities management. At the system-wide level, technologies provide critical support in domains such as school mapping, automated personnel and payroll systems, management information systems, communications, and information gathering, analysis, and use. ICT can be valuable in storing and analyzing data on education indicators, student assessment, educational infrastructure (physical and human), cost, and finance. More importantly, they can assist in constructing and assessing policy scenarios around different intended policy options to determine requirements and consequences and to hel p sel ect those that are the most appropriate. An example of use of ICT for management and governance may be found in the Indian state of Karnataka. It is an example that i s equal ly important for the manner of its implementation, as it is for its scale and sustainability. 13 Karnataka has over 60,000 schools (almost three-fourth of them managed by the government) and more than 450,000 teachers (with more than 50 percent of them employed in government schools). Through the 1950s to early- 1990s, Karnataka’s teacher recruitment and transfer processes were largely decentralized, ad -hoc and d omin ated b y patronage networks. District and state level bod ies, comprising local members of the legislative assembly and other influential political leaders, conducted the recrui tment of primary and secondary school teachers. Teacher transfers were also largely undertaken as a means of punishing errant teachers and rewarding those with connections. Suscep tib le to corruption, the existing systems and practices invited widespread criticism and dissatisfaction from teachers12. Given this context, the Government of Karnataka made a series of key reforms in improving the teacher recruitment and transfer systems. Amongst all the changes made, centralization of teacher recruitment and transfer processes wi thi n the office of the Commissioner of Public Instruction (CPI) in Bangalore and laying out of transparent criteria for teacher selection and transfer were the two most important ones . Once the problem was well defined and solutions envisioned, technology played a crucial role in the actual implementation. The sheer scale of the implementation entail ed a l arge volume of administrative tasks, and this was facilitated through an iterative instal lation of technology-enabled systems and processes, often adapted from other sectors of education where they had already met with some success. As these rules and processes were continuously improved and several new acts were enacted in recognition of success of executive orders, the technology systems were also revamped, ensuring sustained use over time. Thus, clarity in problem and solution definition, evidence-based and iterative manner of implementation, and support from leadership were key to the success of using technology in reforming teacher recruitment and transfers at scale and in making the processes and systems more transparent, objective and fair. Similar to the system in Karnataka, the Sindh School Monitoring System (SSMS) in Paki stan is also an example of an expansive system-wide technology system that was created to improve governance, accountability and service delivery. It constitutes field based monitors who use smart phones and biometric devices to monitor schools on a regul ar b asi s on a variety of attributes such as school status, infrastructure, facilities, staff and st udent attendance, which are then transmitted in real-time to centralized dashboards in the Education and Literacy Departments and Directorate of Monitoring & Evaluation. This helps planning and informs decision-making by the Sindh Government through review meetings 12 (Ramachandran, et al., 2018) 14 at the district level. With an iterative design that has resulted in the system sp anni ng 29 districts, monitoring of more than 26,200 schools, and biometric verification of 210,000 education staff, this system has helped the Government identify and take acti on against 40,000 absentees and 6,000 absconders. This project, too, benefited from a detailed analysis of the existing condition of the public education system i n Sindh, al ong wi th an enabling environment that was provided by the reform agenda of Sindh Government’s education sector plan, rather than being a standalone initiative. Box 1: Showcase Stories from South Asia Monitoring Trends in Educational Growth (MTEG), Afghanistan The Monitoring Trends in Educational Growth (MTEG) Program was l aunched i n 2013 b y the Learning Assessment Unit of the Afghanistan Ministry of Education, in partnership with the Centre for Global Monitoring at the Australian Council for Educational Research (ACER - GEM). Operational till 2017, MTEG was conceptualized with the purpose of providing policy makers with information about the state of education outcomes in Afghanistan. It aimed to do so by measuring trends in achievement outcomes in single classes over time (e.g. Grade 3, year-on-year), and growth of achievement in cohorts throughout the school cycle (Grades 3, 6 & 9)13. MTEG also provided information about the contextual factors around teaching, learning and home environments in order to provide a more complete picture of the influences on learning outcomes14. Based on the results of the Grade 6 assessments that were conducted i n 2013, the ACER team managing the program anticipated that students in Grade 3 would be spread across the spectrum of reading literacy, with significant numbers at very bas ic proficiency l evel s. To ensure accurate measurements for all children in Grade 3, i ncl ud ing those with very basic levels of proficiency, the team arrived at a few innovations. Firstly, students were sorted into two groups, Emerging and Independent Readers, based on a short preliminary reading assessment that was conducted by the test faci li tator. These two groups were directed to two different assessments, with the Independent Readers being 13 (ACER Centre for Global Education Monitoring, 2016) 14 For further information about MTEG in Afghanistan see https://www.acer.org/gem/key- areas/system-strengthening/mteg. The Grade 3 MTEG report is forthcoming. 15 assigned a slightly more challenging test than the Emerging Readers. However, to accurately measure different levels even within these broad groups, both the assessments had i tems of a range of difficulty levels. Secondly, the Grade 3 assessments were conducted on tablets that allowed for interactive input methods. The questions used touch-enabled responses such as Hot spot, mul t ipl e choice and drag-and-drop question types. The assessments also used integrated audio fi l es for many of the questions, including for those assessing letter sounds, word recognition and listening comprehension. An important reason for students being read -aloud questions was that it allowed for more accurate testing of phonemic awareness, p honics, vocabulary, listening comprehension, and mathematical abilities of students who were not strong readers. Use of colourful interfaces, playful design and interacti ve i nput method s motivated students to participate in these assessments. Another very important advantage of conducting assessments on tablets was that a consistent assessment exp eri ence was provided to all students. The students also enjoyed the added support of accessi ng aud io prompts as many times as they needed without the hesitation of requesting repeated clarifications from a human facilitator. Before the assessment commenced, each participant required around 15-20 min to acclimatize herself/himself with the software through guided walk-throughs of the interface and practice questions. The ACER team claimed that this resulted in almost all students being able to take the test without any challenges. The tablet-based assessment was available for offline delivery, meaning that no internet connection was necessary while students completed the assessment. Along with the benefits indicated above, the use of technology to deploy these assessments resulted in greater efficiencies. Data-collection on a tablet meant that the data was securely transferred to ACER’s servers as soon as the tablets were connected to the internet, thereby reducing the fear of loss of data. Encryption and user passwords ensured that both assessments and test data would be secure even in the event of loss of tab l ets. Technology-enabled implementation also resulted in an ability to complete assessments i n a short period of time. Using this innovative approach to assessments had numerous benefits, including enhancing the efficiency and effectiveness of data collection in the MTEG program. This use of technology supports the collection of high quality and rel iabl e data that can be used to inform policy makers about the learning outcomes of Grade 3 students in Afghanistan. 16 Mindspark, India15 Educational Initiatives (EI), the firm that designed the Mindspark software, had been founded on the idea that educational change could and would be catalyzed by demonstrating gaps in students’ learning. Consequently, EI had been conducting assessments and educational research, and had assessed about 14 million students since their founding in 2001. However, the slow pace of change in the education sector, d esp ite EI’s efforts in highlighting and explaining problems throug h i ts assessments, al ong wi th having unraveled patterns in the misconceptions and challenges that children experience, inspired EI to look beyond assessments. In 2008, the firm started developing Mindspark, a computer-assisted learning (CAL) software that enables students to learn mathematics and language at their own level and pace, and is meant to supplement classroom instruction. A mathematics-only program was launched in 2009, which is aligned to the requirements of a range of state and international curriculum, and currently packs content for Grades 1 to 10. In 2016, an English reading and language program for Grades 4 to 9 was also introduced. The roll-out of the Mindspark program has been staggered over devi ce -types, starting off with a desktop/laptop version in 2009, a tablet version in 2010, followed b y the release of a device-agnostic version in 2013 that required only an internet browser. Final ly, in 2014, EI launched an offline version of Mindspark that would need only periodic internet connections, learning from the various cycles of its implementation. Consciously staying away from hardware innovations of digitized whiteboards and textbooks, the Mindspark system chose to aim for a cloud-based student-centric software, wherein the focus is on ‘absorption’ of learning rather than the simple delivery of curriculum. This is enabled through a multitude of features, such as the use of finely - graded questions, adaptive logic, and intelligent responses and modules. These ensure that students are on a very gradual learning trajectory, are directed to different questions based on whether their responses are correct or not, and are provided continuous feedback, remedial instruction and quizzes. To support students with a range of abilities in comprehension, it can provide voiceover support in mathematics, and has built-in graphics, animations and educational games for an extensive part of the content. Additionally, it implements a smart recognition system that not only rewards good performance, but also 15 (Muralidharan, Singh, & Ganimian, 2016) 17 encourages desirable behaviors such as persistence in completing topics, spending time to read through feedback, and consistent use of the software. While these features drive student engagement with learning, the system also integrates a feedback system that allows students to rate each question (Like/ Dislike/ Excited/ Bored/ Confused) as well as provide typed feedback to EI. In an illustration of good design, Mindspark implementation begins with a needs- assessment, infrastructure check, analyses, followed by dialogue among EI, teachers and Principal to arrive at a decision on the suitability of Mindspark for the present chall enges and circumstances. Only once Mindspark is found suitable, does the team be gin implementation either in a few classrooms as a pilot or throughout the school, and provide teachers and students with orientation and training to use the software. Further, the program gives significant control to individual teachers, i.e. teachers can ‘activate’ content based on what they have taught in the classroom, and then, inform their instruction usi ng reports generated by the program. Over the years, the program has been tried and tested in a variety of contexts, starting from private schools to public schools in the Indi an states of Madhya Pradesh, Gujarat and Rajasthan. In 2012, recognizing the need and potential for innovation in urban centres, the Mindspark program was implemented in after-school learning centres in the urban slums of Delhi. The implementation in Delhi has been rigorously tested by researchers associated with MIT’s J -PAL. The study found that students who used the Mindspark program showed significant gains compared to the control group. In addition, the evaluation also found great relative gains in the levels of academically-weaker students compared to the control, since the rate of learning in the control group was minimal. While the study did not experimental ly i solate the effects of the CAL system from the effects of group teaching and additional instructional time, it presents evidence that indicates that the most significant contributi on was from the software. The software’s abilities to instruct at the appropriate level and continually update the content for individual students based on their i ndivi dual l earni ng trajectories were found to be the primary drivers of success. On the matter of productivity, too, the research presents encouraging results. Estimates i n the study suggest that, at a scale of 1,000 schools, (steady-state) implementation of the program would increase the government’s annual per-child cost by less than 2% . In fact, even with high fixed costs, absence of economies of scale and low attendance, the program 18 delivered greater learning at lower costs than public spending. In the context of substantial percentages of children in the South Asia region severely lagging their l evel s, the study provides findings that could have remarkable implications. Using estimates, it hypothesizes that gains in learning levels by using the system regularly could lead to stud ents catching up with their appropriate grade-level standards. The study also presents some very compelling social and policy benefits of effectively implementing a system such as this. Complete personalization offered b y a CAL system allows for children to learn at their own pace, which doesn’t only mean that i t can d eli ver content at a lower level and/or pace; it can also accelerate growth of the same students when they start learning at a faster pace. These adaptations come a t no ad di ti onal social costs to the children, such as changing their peers. It effectively ensures that students are not ‘tagged’ or fixed to one track, while allowing them to learn at their own pace. Secondl y, refraining from commenting on the ‘ideal’ vision of how education systems shoul d group children, a solution such as this allows nations to improve student learning without incurring heavy policy costs such as shifting away from the age-cohort based social grouping of students. In a way, it allows for governments to prioritize the right set of reforms at different stages of their own journey towards a transformational education system. Before concluding, a fine reading of the study report also unpacks some very relevant lessons for governments thinking about implementing such systems. While the CAL system was pivotal in delivering improved learning, instructors may have a very important rol e to play in the overall implementation. For example, in the study, which was i mp lemented i n an after-school centre in Delhi, the EI instructor played a crucial role in encouraging regular student attendance, ensuring time on task, supervising school -homework completi on and exam preparation during group-instruction period. Since some of these activities were demanded by parents, thinking through a holistic design of the intervention and obtaining buy-in from parents would be critical to successful implementation. In addition, the study also notes that the very nature of teaching-learning requires complex contextual- awareness and communication; thus, technology may serve to complement teachers, rather than substitute them. Secondly, parental demand for the Mindspark centres of the study was low in the absence of fee-waivers; thus, low-resource communities may not b e able to access programs that charge a fee, and governments may need to think about delivering both hardware and software. 19 To summariz e, the software’s ability to instruct at the right level and pace, along with ti me- productivity considerations, and cost-effective and well-rounded i mpl ementation, make this a useful solution to test, implement, and iteratively improve i n contexts with severe levels of inequity and resource constraints. 3.3. Aligning Actors Reform components such as assessing learning and acting on available evid ence are not effective in and by themselves. Any reform initiative needs to gain traction and sustain within the systems it is implemented in. For this to successfully happen, it i s i mportant to identify and map different actors, and align and equip them with the necessary ski ll s and orientation to direct all inputs towards those outcomes. However, aligning actors and hence, systems, to absorb a new set of reforms is a challenging task. There are p rimari ly two ways in which alignment may fail to take place. Firstly, accountability fails when different actors hold the system accountable to different goals. For example, while students and parents may value learning, governments and school bodies may value only appropriate budgetary allocation. Secondly, while governments may task schools and education departments with the goal of learning, the funding, information and incentives in the system may not be aligned to learning. For example, while schools or education departments may very eas ily produce enrolment data, they won’t have information systems set up to produce such reliable learning data at high quality. Similarly, fund ing or retention of staff may not be linked to performance, but to other parameters, and thus, not incentivize learning. Thus, any reform initiative needs to be accompanied b y an anal ysis that seeks to answer the following questions: • Are all stakeholders aligned with the shifts that the reform seeks to p rod uce, and hence, aligned on the data that they will use to assess and improve the reform? • Is there adequate data on the outcomes that the reform seeks to achieve? Therefore, it is clear that the power of using crucial levers of change, i.e. goal s, fi nancing, and incentives, is predicated on the easy availability of high -quality information. Technology can play a vital role in setting up the infrastructure i n the form of Ed ucation Management Information Systems (EMIS) that can enable this. There are several 20 examples 16 of successful building or consolidation of Education Management Informati on Systems (EMIS), including in Guatemala where data was made open to al l through easi ly accessible website of the Ministry of Education; in Honduras where along with public access to data, relevant stakeholders were trained on using data; and Lithuania, where interest was generated among municipalities and school representatives, l eadi ng to the creation of local and school-level EMIS, and creation of an annual education report to measure student achievement. These are easily contrasted with examples that exhi bi ted barriers such as lack of clarity on the goals and purpose (Albania, Vietnam, Indi a), l ack of political will (Bulgaria, Latvia), not equipping key actors or involving them in design and dissemination of data (Bulgaria, Vietnam, Pakistan, Chad, Ghana), and lack of legal framework that supports the EMIS (Bolivia and Argentina). What these numerous examples illustrate is the criticality of the process of reform, of the efforts to align for accountab il ity, even in the creation of systems for tightening the very drivers of accountability. One of the most critical design elements that enables a successful EMIS is reformers/practitioners supporting all relevant actors in identifying, articulating and prioritizing a problem, and then, helping create safe and conducive environments for iterative solutioning and innovation 17. An example of this design principle is exhibited by an organization called Open Learning Exchange (OLE, Nepal) that has dedicated strategies for partnering with the community, government, schools and other expert organizations. These strategies aim to build alignment on the vision, strategy and design of the sol ution , promise and ensure commitment to sustain a high level of performance, and hold all parties involved accountable to outcomes. Therefore, to set in motion and sustain the virtuous cycle of assessing, acting and aligning actors towards learning, i t i s vi tal that al l appropriate inputs are supported through alignment of all relevant actors, most importantly the school management, community, the education department, and local government. Box 2: What about the ‘ Disruptive Technologies’? Disruptive innovation theory, first introduced in the mid-90s, describes i nnovati ons that originate in low-end footholds in existing markets (where incumbents typically focused on 16 (Abdul-Hamid, Saraogi, & Mintz, 2017) 17 (Andrews, Pritchett, Samji, & Woolcock, 2015) 21 their most profitable and demanding customers and paid less attention to less-demanding customers) or that create new-markets as footholds 18. Some of these innovations ‘sustai n’ if they continue delivering better products and services , generally at lower costs, than incumbents and become mainstream. It is rare that a technology or product i s i nher ent ly sustaining or disruptive; disruption is a process, and smart disrupters improve their products and drive upmarket. While analyzing industries, therefore, it is pertinent to discuss ‘new technologies’, moni tor their trajectory, and see if they emerge as successful disruptors over a period. Michael Porter, a Harvard Business School professor, analyzed the advent of n ew technologi es as consisting of three ‘waves’19. Wave 1 (1960s and 1970s) automated individual acti vi ties i n the value chain, such as bill payments, maintaining records, comp uter - ai ded d esi gn etc. They largely improved the periphery of the education system by enhancing the productivity of certain activities. Wave 2 (1980s and 1990s), powered by the rise of the internet, l ed to coordination and integration across individual activities with outside suppl iers, channel s, and customers; and across geography. It allowed universities, for example, to closely integrate research networks. Wave 3 of technology being unleashed now is about technology becoming an integral p art of the product or service itself. Embedded sensors and processors power products i n the augmented reality and virtual reality domains; cloud computing and internet of things connect products in never-imagined ways; big data powers dramatic software applicati ons in artificial intelligence and machine learning; and ledger technologies such as b l ockchain can fundamentally transform transactions. Very early experiments with some of these i n the education system also exist. Augmented reality headsets have p romised ‘i mmersive’ learning experiences, machine learning algorithms have been deployed to predict stud ent outcomes, and blockchain-based educational certificates are already being piloted. Has the education industry been ‘disrupted’ in the past? Perhaps history hol ds a l esson. Johann Gutenberg's invention of the printing press in mid 1400s disrup ted the access to information, which was earlier limited to copying and production of the written word i nto laborious manuscripts accessible to wealthy elites. Technology to create nearly i d entical 18 (Christensen, Raynor, & McDonald, 2015) 19 (Porter & Heppelmann, 2014) 22 books of quality at an economical price created a massive new-market foothold: the general public who were non-consumers of books. This disruption shifted the entire educational practice in the world: the relationship between the educator and student now had a third ingredient - technical texts as silent instructors. Will the onslaught of the third wave of technology disrupt educati on va l ue chai n (and at several points)? Like most cases of possible disruption, it is too early to say. Large -scale education has been an extremely resilient enterprise in that sense. Part of the reason i s that it is hard to find examples of disruptive innovati on in the public sector, which constitutes a major chunk of basic education delivery around the worl d. Several factors, such as profit motives and competitive pressures propel disruptive innovations in the private sector, but are largely absent in the public sector. Even when the p ubli c sector i s innovating, the most common types of innovation often drive costs up 20, as they tend to add increasingly complex and expensive features onto existing innovations (think of introducing smar tboards in classrooms). In such ‘traditional’ innovations, price vs performance is a trade-off: it is generally assumed that better performance requires more teachers, smaller class sizes, and better facilities (i.e. paying higher price). Disruptive innovations need to break such trade-offs and reduce costs by finding footholds, i.e. a population of underserved/unserved consumers, and then going mainstream. ‘O nline education’ models (such as Massive Open Online Courses, or MOOCs), mi ght have found such a new-market foothold by targeting non-consumers of education. If online education products continue to improve and get driven upmarket, we might have a case of successful disruption. Amidst the rapid technological progress of the present millennium, one thing is certain: the status quo of the education enterprise might, finally, be changing. 4. Conclusion and food-for-thought for policymakers “If all you have is a hammer, everything looks like a nail.” 20 (Eggers, Baker, & Vaughn, 2013) 23 – Abraham Maslow The challenges faced in education are multi-dimensional, and vary immensely from region to region, many a times within the same country. Over the past two decades, a number of ICT tools have been developed and implemented in the school systems around the worl d. However, technology – no matter how well designed – is only a magnifier of human i ntent and capacity; it is not a substitute21. It is often tempting to think of ICTs as a si l ver -b ul let solution, and jump to the debate on specific inputs (what kind of laptops should I buy?) that might have worked elsewhere (most likely the evidence being from highly de veloped learning environments). But acquiring the technologies themselves, no matter how hard and expensive, may be the easiest and cheapest element in a series of elements tha t ultimately could make these technologies sustainable or beneficial. Effectively integrating technology into educational systems is much more comp li cated . It involves a rigorous analysis of educational objectives and changes, a realistic understanding of the potential of technologies, a purposeful consideration of the pre - and co-requisites of effectiveness of ICTs for education, and an assessment of the prospects of this process within the dynamics of educational change and reform. The impact of ICTs for education depends to a large extent on the purpose for which ICTs are used. For examp l e, if technology is used to stream recorded lectures/videos of existing teachers , and software is used to merely provide digital copies of existing textbooks, it is far -fetched to expect learning results significantly different from classroom lecturing or textbook use. However, these instructional technologies may extend educational opportunities to situations where there is no lecturer or textbook. In simple terms, understanding what one wants to accomplish, understanding the users and beneficiaries, and understand ing the relevant "use cases", is the key. Development efforts utilizing ICT tools that are more likely to succeed are informed by co-creation and take place as close to the user as possible. Integrating technology into the educational process is an intricate, multi faceted p rocess that involves a series of deliberate decisions, plans, and measures. As evidence22 indicates , scaling any intervention from a ‘proof of concept’ stage requires multiple iterations of experimentation, in which underlying mechanisms are identified and the model i s refi ned 21 (Toyoma, 2015) 22 (Banerjee, et al., October 2017) 24 based on this understanding. The following 23,24 might serve as a step-by-step guide to policymakers when thinking about ICT-enabled improvement of school education systems: • Create a vision for ICT use in education, identifying educational areas for ICT intervention, formulating ICT-in-education policy, and articulating what precisely will the ICT intervention change/enable • Understand the current context, for all aspects identified in the vision for ICT use i n education , so as to identify/build the ‘right technology’, and then develop an implementation strategy that helps achieve the vision, by accounting for challenges and leveraging strengths of the current system o Consult a range of actors, such as students, parents, teachers, teacher trainers, administrative officials, and civil society organizations o Identify challenges and strengths related to the quality and relevance of content, ease and cost of set-up and maintenance of infrastructure, hardware and software, and knowledge, skills and mindsets of relevant actors o Empower all key actors associated with the education system towards performing their duties and realizing their aspirations o Train relevant actors, support them in deploying technology and create the necessary culture for the use of ICT through awareness drives across the system o Proactively create adequately contextualized and tested content/reports/outputs that will form the heart of the intervention o Allocate funds for maintenance and update of the infrastructure, i .e. b oth software and hardware o Prioritize pro-equality as a fundamental tenet of the strategy • Monitor, evaluate, continuously learn, and improve design and implementation o Allocate adequate funds and resources to the creation of an MIS that serves as a backbone, enabling the collection, aggregation, storage, analysis and synthesis of data towards improvement of the design and implementation of technological solutions, programs and policies; evaluate for impact 23 (Trucano, 2016) 24 (Information for Development Program (infoDev), 2007) 25 o Build sufficiently exhaustive practices that ensure data security and pack appropriate privacy provisions All the above steps might not be relevant in every context, but it is necessary to thi nk of a ‘big picture’ even with smaller interventions. Technologies have great potential for knowledge dissemination, effective learning, and efficient education services. However, to truly realize this potential, the educational policies and strategies need to be right, ICT-in-education policies need to be well thought out, and the prerequisite conditions for using these technologies must be met concurrently. 26 References Abdul-Hamid, H., Saraogi, N., & Mintz, S. (2017). Lessons Learned from World Bank Education Management Information System Operations. The World Bank. ACER Centre for Global Education Monitoring. (2016). Monitoring Trends in Educational Growth: Assessment Framework for Afghanistan. Melbourne Vic: Australian Council for Educational Research. Andrews, M., Pritchett, L., Samji, S., & Woolcock, M. (2015). Building capability by delivering results: Putting Problem-Driven Iterative Adaptation (PDIA) principles into practice. In A. G. Approaches, Whaites, Alan; Gonzalez, Eduardo; Fyson, Sara; Teskey, Graham (pp. 123-133). OECD. Bakker, S. (2014). The Introduction of Large-scale Computer Adaptive Testing in Georgia: Political context, capacity building, implementation, and lessons learned. Russia Education Aid for Development (READ). Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukherji, S., . . . Walton, M. (October 2017). From Proof of Concept to Scalable Policies: Challenges and Solutions, with An Application. National Bureau Of Economic Research. Christensen, C. M., Raynor, M. E., & McDonald, R. (2015). What is Disruptive Innovation? Harvard Business Review. Clarke, M. (2012). What Matters Most for Student Assessment Systems: A Framework Paper; Systems Approach for Better Education Results (SABER) Working Paper No. 1. Washington D.C.: World Bank. EDC. (2015). Radio Instruction to Strengthen Education (RISE) and Zanzibar Teacher Upgrading by Radio (ZTUR) Post-project Evaluation in Zanzibar; Final Report. Education Development Center. Eggers, W. D., Baker, L., & Vaughn, A. (2013). Public sector, disrupted: How disruptive innovation can help government achieve more for less. Deloitte University Press. Information for Development Program (infoDev). (2007). ICT-in-Education Toolkit for Policy Makers, Planners and Practitioners. Version 2.0. http://www.ictinedtoolkit.org/. 27 Muralidharan, K., Singh, A., & Ganimian, A. J. (2016). Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India. National Bureau of Economic Research. Working Paper 22923. Neuman, M. J., & Devercelli, A. E. (2013). What Matters Most for Early Childhood Development: A Framework Paper; Systems Approach for Better Education Results (SABER) Working Paper No. 5. Washington D.C.: The World Bank. Porter, M. E., & Heppelmann, J. E. (2014). How Smart, Connected Products Are Transforming Competition. Harvard Business Review. Pritchett, L. (2015, December). Creating Education Systems Coherent for Learning Outcomes: Making the Transition from Schooling to Learning. Research On Improving Systems of Education. Ramachandran, V., Béteille, T., Linden, T., Dey, S., Goyal, S., & Goel Chatterjee, P. (2018). In Getting the Right Teachers into the Right Schools : Managing India's Teacher Workforce (pp. 194-207). Washington, DC: World Bank. Toyoma, K. (2015). Geek Heresy: Rescuing Social Change from the Cult of Technology. New York: PublicAffairs, an imprint of the Perseus Books Group. Trucano, M. (2016). SABER-ICT Framework Paper for Policy Analysis: Documenting national educational technology policies around the world and their evolution over time. World Bank Education, Technology & Innovation: SABER-ICT Technical Paper Series (#01). Washington D.C.: The World Bank. World Bank Group. (2017). Promising Approaches in Early Childhood Development : Early Childhood Development Interventions from Around the World. Washington, DC: World Bank. York, B., Loeb, S., & Doss, C. (2017). One Step at a Time: The Effects of an Early Literacy Text Messaging Program for Parents of Preschoolers. 28 29