INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES KNOWLEDGE PACK Remote phone-based formative assessment EXPLORE INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES KNOWLEDGE PACK REMOTE PHONE-BASED FORMATIVE ASSESSMENT © 2022 International Bank for Reconstruction and Development / The World Bank INDEX 1818 H Street NW, Washington, DC 20433 INTRODUCTION Telephone: 202-473-1000; Internet: www.worldbank.org Problem statement WHO WHY License: Creative Commons Attribution CC BY 3.0 IGO This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, Use cases and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. This report was also supported with funding from the Global Partnership for Education. Structure of solutions The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the Challenges and trade-offs WHAT HOW part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such What has been done boundaries. in other countries Rights and Permissions CONCLUSION The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full TO GO FURTHER attribution to this work is given. Please cite the work as follows: EdTech and LEeP team. 2022. Knowledge Pack : Remote phone-based ANNEXES formative assessment. Washington, D.C.: World Bank Group Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; Acknowledgment: Much appreciation goes to Diego Armando Luna and Aishwarya Khurana from the e-mail: pubrights@worldbank.org LEAP team who took part in the development of this resource, in coordination with the EdTech Team, Design : Alejandro Scaff, Sarah Kleinmann and to Omar Arias and Jaime Saavedra for their overall support. Also we want to thank colleagues Marguerite Clarke and Elizabeth Sedmik for providing comments to enrich these resources. 2 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Introduction What is a KP? About this KP Knowledge Packages (KPs) are short, The use of technology has substantially pragmatic guides on individual topics increased in all stages of the learning within EdTech, meant to provide sufficient assessment process, from the planning knowledge and understanding so that and design of learning assessments to the non-technical stakeholders can make assessment scoring and assessment results key planning, design, and procurement dissemination. decisions for education. This KP is developed by the World Bank’s Learning Assessment Platform (LeAP) team They can be used as a starting point for and coordinated by the EdTech team. the planning of technology deployment to improve education, especially with Its purpose is to provide information on education ministries. how basic phones, specifically through Short Message Service (SMS), Interactive Voice Response (IVR), and direct phone calls, can support formative assessment activities and continuation of learning outside the classroom even in low-resource contexts. 3 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES WHO are the main stakeholders ? KPs are designed with a human-centered vision. Ministry of Education Task Team Leaders (TTL’s) & This knowledge pack is meant to provide officials Bank Project Managers sufficient knowledge and understanding to help decisionmakers make key planning, design, and procurement decisions Governmental agencies Researchers and technical of technologies for working on learning experts in learning assessment. assessment activities assessment Other governmental NGOs and private or regional institutions organizations working in specialized in curriculum the field of education and learning assessment 4 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW Problem statement | Use cases WHY is this KP designed ? PROBLEM STATEMENT WHAT ARE PHONE-BASED FORMATIVE ASSESSMENTS? WHY SHOULD WE FOCUS ON THEM? What are phone-based formative Why should we focus on phone-based formative assessments? assessments? Formative assessment consists of activities Before the COVID-19 pandemic, to provide real-time feedback to commonly implemented by teachers students typically learned in students. to monitor students’ progress towards classrooms and interacted with curriculum learning goals. It aims to teachers who assessed their As a response, expanding remote generate timely information that teachers learning and provided real-time learning and assessment became can use to shape instruction, help students feedback to support their learning common. While many countries understand concepts and reinforce progress. were able to quickly roll out remote learning content, and inform subsequent learning initiatives using various coursework. Moreover, formative But this connection broke during the web-based applications, not all assessment is a low-stakes, adaptive COVID-19 pandemic and it has had children were able to access such process centered on individualized support a significant impact on education resources, particularly when and constructive feedback. systems worldwide, affecting they required having internet whether and how students learn connectivity and digital devices Phone-Based Formative Assessment refers remotely. (such as computers, tablets, to the use of mobile phone technologies and smartphones) at home. to engage students outside the classroom, Since the beginning of 2020, the This situation reflects some of assess their learning in real time, and COVID-19 pandemic forced countries the challenges faced in low- provide timely, constructive feedback to around the world to close schools. resource contexts due to limited promote learning continuity. This impacted the ability of teachers infrastructure. 5 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW Problem statement | Use cases WHY is this KP designed ? PROBLEM STATEMENT PART OF THE SOLUTION FOR LEARNING CONTINUITY DURING FUTURE SHOCKS How do we ensure that students continue to learn during 1. Mobile cellular telephone subscriptions are subscriptions to a school closures and our education systems are resistant to public mobile telephone service that provide access to the PSTN similar shocks and disruptions in the future? using cellular technology. The indicator includes (and is split into) the number of postpaid subscriptions, and the number of active With a global penetration rate of 104 percent, the access prepaid accounts (i.e. that have been used during the last three to basic phones for learning assessment delivery can be months). The indicator applies to all mobile cellular subscriptions part of the answer. that offer voice communications. It excludes subscriptions via data cards or USB modems, subscriptions to public mobile data Worldwide, mobile phone subscription The purpose of this KP is to provide services, private trunked mobile radio, telepoint, radio paging and rates per 100 people1 have increased information on how basic phones, telemetry services. from 12 in 2000 to 109 in 2019. When specifically through Short Message 2. A ‘unique mobile subscriber’ is defined as an individual person who can own multiple mobile connections (i.e., SIM cards). looked at unique mobile subscribers2, Service (SMS), Interactive Voice Response 3. Basic phones are the most basic forms of mobile phones that the penetration rate remains small in (IVR), and direct phone calls, can support use second generation (2G) network and can be identified with low- and lower- middle-income countries. formative assessment activities and their small screen, a standard numeric keypad, long battery life For example, in Sub-Saharan Africa (SSA), continuation of learning outside the and no high-end features such as access to internet. Basic phones unique mobile subscribers stand at 46% classroom even in low-resource contexts. primarily perform two main functions: voice calls and sending/ of the population, and only 28% of the The three technology solutions are receiving text messages. These phones rely on mobile networks population in SSA have access to internet selected because they rely on texts and such as Vodafone for communication. One common example is on their mobile phone. calls which are the most basic features Nokia 1100. When compared with ‘feature phones’ and ‘feature Thus, more low-tech and readily available available and are compatible with all smart phones,’ basic phones are largely similar in their external modalities, such as basic phones3, kinds of mobile phones (including basic appearance but feature phones also come with minimum multimedia and internet capabilities. These were among the first can facilitate learning continuity when phones). phones to use third generation (3G) networks. students and teachers cannot be physically together. 6 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW Problem statement | Use cases WHY is this KP designed ? USE CASES RECOMMENDED USES OF PHONE-BASED ASSESSMENTS Phone-based assessment solutions can • natural disasters, student sampling, standardization of the be used for at least two purposes: • conflicts, and assessment administration, test security • other emergencies and potential malpractice such as item 1 To conduct formative assessment leaking, test-taker identity corroboration, to, for example, gauge the extent of 2 To conduct impact evaluations on, or cheating). ‘students’ absorption of content, identify for example, analyzing the effect of any misconceptions in understanding, interventions introduced in response provide constructive feedback to to the pandemic on student learning students or caregivers, and offer outcomes. additional learning resources and activities to support learning. Remote phone-based formative assessments can General Guidance: For the purposes be used when in-person teaching and of making high-stakes decisions or learning are not possible. Such as during monitoring learning at the system school closures as a result of: level, phone-based assessments are • pandemics, generally not a suitable option due • social unrest, to several constraints (e.g. limitations to 7 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? WHAT DOES PHONE-BASED FORMATIVE ASSESSMENT COVER? The table below describes the three phone-based technologies and their use for the delivery of formative assessments including their advantages and disadvantages. SMS IVR DIRECT PHONE CALLS Text messaging is a basic feature IVR works by inviting users to call a Direct phone calls allow teachers to included across all mobile phones. number (usually toll-free). An automated assess students’ engagement with Formative assessment through text system delivers pre-recorded audio remote learning resources, understand messaging is an asynchronous process, messages that may contain on-demand their absorption of learning content, during which students respond to the educational content or voice-based and respond to students’ questions. questions sent to them based on a quizzes. previously delivered content. The learning content delivery and assessment While using IVR, the users (child/caregiver) respond Direct phone calls serve as an opportunity to can occur either at different moments of time by pressing the number key or saying the number provide emotional support and encouragement, or presented together to acquire immediate that corresponds to their answer choice (i.e., “say which are particularly important when students information about students’ learning. The delivery yes or press 1, say no or press 2”). IVR also allows and teachers cannot be together physically in the of assessment content through SMS typically users to leave a recorded voice message with a classroom (Lainchaur, 2020, p. 16) requires the development of a tailored software detailed response or query. application that sends out SMS messages and receives responses, which can be done through a variety of mobile aggregator platforms, such as Twilio and Tatango. 8 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? STRUCTURE OF SOLUTIONS The following table summarizes the key features of each technology solution and their use for the delivery of formative assessments. SMS IVR DIRECT PHONE CALLS Users call a phone number to listen to learning content and respond to a Short text-based quizzes are sent to series of voice-based multiple-choice Teachers or instructors contact students students. questions. by phone to go over learning content, assess students’ knowledge, guide Assessment With two-way SMS, students can respond Answers are selected either by pressing students to appropriate learning content, procedure via SMS (like the learning assessment, corresponding keys or saying the correct and encourage their use of the learning learning content can also be delivered answer. content. Feedback can be provided during through SMS either before or after the The IVR system processes responses and the phone call. assessment). may provide voice-based feedback based on the result. Assessment Received regularly— i.e., once a week to Made sporadically or regularly —can also Accessed at students’ convenience. frequency several messages per day. be requested by parents if needed. 9 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? STRUCTURE OF SOLUTIONS SMS IVR DIRECT PHONE CALLS Cost: The most cost-effective of the Accessibility: Supports students with Accessibility: Supports students with three tools. visual disabilities as well as students/ visual disabilities as well as students/ parents with limited literacy. parents with limited literacy. Time flexibility: Users can respond at their convenience, given the Higher Respnose Rate: As a result of Synchronicity: Synchronous interaction Solution asynchronous nature. more inclusive sample, IVR leads to with students allows for a more advantages higher response rate compared to SMS. personalized approach, promotes active Reminders: Additional messages can be engagement of students in assessment and delivered to remind students to take the Language flexibility: Content can be learning process, builds rapport with the assessment. delivered in multiple languages, including assessor. those with no written form. Access: working basic mobile phone Access: any type of phone. Access: any type of phone. Free SMS: need to SMS credits or a free of charge alternative. Free calls: need to offer phone call Free calls: need to offer phone credits to credits or a toll-free number. teachers for making calls. Infrastructure Mobile aggregators: needed to link requirements different carrier networks with SMS Voice actors: needed to record the Digital infrastructure: needed to manage software / platform providers. content (in different languages). students’ data and feedback. Software: needed to administer content Software: needed to administer content Switchboard system: needed if hotlines and record students’ answers. and record students’ answers. are used. 10 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? CHALLENGES AND TRADE-OFFS SMS IVR DIRECT PHONE CALLS Parent authorization: needed to send Parent availability: may be needed for Community User familiarity: necessary for receiving them SMS and reception of SMS outside monitoring the call between the child and considerations and engaging with IVR. of school hours. the enumerator. 160-character restriction. Non-Latin characters might not display properly. Possible difficulties due lack of access to Possible additional burden on teachers to a personal phone and longer duration of make the calls (sometimes even outside the Phones with small screens may create engagement. school hours). Solution usability issues. limitations Students must be attentive and complete Calls need to be scheduled and Hard to track possible parental assessments in one sitting— saving isn’t rescheduled to ensure children’s availability interference (e.g. parents telling children an option. and access to phone at the time of call. the correct answers). Cannot be used with landline phones. 11 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? WHAT HAS BEEN DONE IN OTHER COUNTRIES USE OF SMS: MOBILE-BASED POST LITERACY PROGRAMME, PAKISTAN ABOUT THE PROGRAM CONTENT DELIVERY Implemented in Punjab province of Pakistan.The program The program spanned over six months and was divided into aimed at providing literacy teaching in Urdu to young and two stages: first stage comprised of in-person teaching that adult women aged 15-30 with limited or no literacy skills. lasted two months and second stage focused on mobile based Over three years, the program provided free mobile phones, literacy that lasted four months. During first two months, SIM cards and SMS services for four months to over 2,500 the participants met six times a week for two to three hours participants. to learn and write alphabet and read with emphasis on phonics. During the second stage, participants received SMS on their phones 6-8 times a day. After receiving the SMS, the participants were required to read the texts, write them in their notebooks and answer the questions asked. ASSESSMENT PROCEDURE To monitor the progress and participation, a web- RESULTS/OUTCOMES based system was used to send SMS to the participants. Assessments involved responding to questions/multiple- During the pilot phase with 250 learners, test results showed choice-questions or tests sent to them. that after 4 months of mobile based literacy instruction, only 14% of the participants fell into 0-50% score range compared to 90% during the first month. Source: U. Hanemann (Ed.). (2013). Mobile-Based Post Literacy Programme. Pakistan. UNESCO Institute for Lifelong Learning. Retrieved from: https://uil.unesco.org/case-study/effective-practices-database-litbase-0/mobile-based-post-literacy-programme-pakistan 12 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? WHAT HAS BEEN DONE IN OTHER COUNTRIES USE OF IVR: ALLO ALPHABET, CÔTE D’IVOIRE ABOUT THE PROGRAM CONTENT DELIVERY The study was first piloted in a rural village in Adzopé (Côte The content delivered focused on overall literacy skills such as d’Ivoire). The study included 38 students from grade 5 and phonological awareness and print-sound mapping. Overtime, spanned five weeks from October-December 2018. Later the difficulty of the content was increased from simple from 2019-2020, the program was scaled up to reach 1200 phoneme and syllable awareness to mapping between letters, children (800 in treatment and 400 in control) in remote words, and sounds. rural communities. Literacy curriculum in Attié and French The learner was required to call a specified number. Once the were designed and implemented on an interactive voice call was placed, it was automatically disconnected and the response (IVR) system named Allo Alphabet. system was programmed to call back the learner. At start of each call, the system played a welcome message, that updated the learner of their progress . ASSESSMENT PROCEDURE To assess the learners, the system played a pre-recorded RESULTS/OUTCOMES audio message containing a question. After selecting a response, the learner received feedback from the system. For correct answers, the system would prompt The analysis showed that, on average, learners called the the next question. For incorrect answers, the system first system 14.2 days out of 32 days and initiated 81.4 calls over provides a hint and gives another opportunity to answer. five weeks spending 6.2 minutes on calls. Source: Madaio, M.A., Kamath, V., Yarzebinski, E., Zasacky, S., Tanoh, F., Hannon-Cropp, J., Cassell, J., Jasińska, K. and Ogan, A. (2019). “You Give a Little of Yourself”: Family Support for Children’s Use of an IVR Literacy System. In ACM SIGCAS Conference on Computing and Sustainable Societies (COMPASS) (COMPASS ’19), July 3–5, 2019, Accra, Ghana. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3314344.3332504 13 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Structure of solutions | Challenges and trade-offs | What has been done in other countries HOW WHAT are the potential solutions? WHAT HAS BEEN DONE IN OTHER COUNTRIES USE OF PHONE CALLS: YOUNG LOVE, BOTSWANA ABOUT THE PROGRAM CONTENT DELIVERY Between February and March 2020, before many countries The sample of 4,500 students was randomly divided into closed schools due to the pandemic, a phone-based three sub-groups: first group that received weekly text assessment of basic numeracy skills was administered to message followed by phone call, second group that received over 4,500 students in grades 3 to 5. Over 70 assessors weekly text message only, and control group. The students in (former teacher aides) were trained remotely using voice treatment groups received targeted instruction in the form of notes and sharing resources via WhatsApp.  tailored messages based on their learning level.  ASSESSMENT PROCEDURE RESULTS/OUTCOMES The Annual Status of Education Report (ASER) survey instrument was adapted for phone-based administration. The assessment entailed the following The study suggested that SMS combined with phone calls tasks: a) Number operations; b) Timed word problem; can result in substantial learning gains and revealed that ‘text c) Explanation of the solution. On average, the phone message followed by phone call’ led to 31% score increase in calls with students lasted between 15 and 20 minutes. numeracy. ‘Text message followed by phone call’ had larger Facilitators presented to the students a the learning and more cost-effective effects on engagement in learning activities sent via SMS and check their understanding. when compared to ‘text message only’. Source: Angrist, N., Bergman, P., & Matsheng, M. (2021). School’s Out: Experimental Evidence on Limiting Learning Loss Using “Low-Tech” in a Pandemic. Working paper 28205. National Bureau of Economic Research 14 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW to implement next steps ? ENABLING CONDITIONS CONSIDERATIONS FOR IMPLEMENTING PHONE-BASED FRMATIVE ASSESSMENTS The LeAP team elaborated a full list of ess the prerequisites pre-conditions to ass​ and enabling conditions for successful implementation of each of the solutions EDUCATION SYSTEM CAPACITY (e.g. mobile penetration rates, existence of CONSIDERATIONS mobile aggregator in the country, average cost per SMS/1MB mobile data/call rates, #1 Learning content and its alignment with access to smartphones and internet by assessment: Existing distance learning teachers, MOE’s engagement, and local initiatives available and the type of implementor’s capacity among others)    technology used in distance learning are critical to define the feasibility and success of phone-based assessment. #2 Human resources: Engage teachers either in direct implementation of phone-based assessments or in using the data obtained from these assessments. 15 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW to implement next steps ? ENABLING CONDITIONS CONSIDERATIONS FOR IMPLEMENTING PHONE-BASED FRMATIVE ASSESSMENTS for financial transactions, access to mobile phone calls, average cost of TECHNOLOGICAL electricity, etc. mobile data for 1GB). AND LOGISTICAL CONSIDEREATIONS #3 Service providers: Service providers #5 Time: Considering the time frame and software platforms are key for implementation (e.g. access to a #1 Demographics: Consider the skills component to successful delivery of database of caregivers’ or students’ and competencies required to use these educational applications (assess the phone numbers). Duration of school devices . Adult literacy rates, urban context of usability, accessibility and closures, and implementation timeline population share or language diversity affordability). Aim for at least one are also important. in the country/region can help select the operational mobile aggregator capable modality (SMS, IVR or phone calls) for of implementing SMS/IVR/phone call #6 Local implementor capacity: A delivery of assessments. solution. Availability of or ease of setting- key factor will be selecting the right up toll-free numbers, short codes , two- implementing partner. It is critical #2 Technology: Mobile phone way SMS, and frequency of internet to consider whether the candidates penetration and network coverage outages/shutdowns in the country also have previous experience with are key to evaluate a country’s needs to be considered. implementing phone-based technological capacity. Phone-based interventions or assessments, assessments requires students to have #4 Financial: Consider the costs of experience delivering distance a mobile phone at home. Consider different phone-based technologies learning interventions and working the availability of connectivity, mobile (e.g. average cost per SMS, average with mobile network operators and cellular subscriptions, mobile accounts per-minute cost of different types of mobile service providers. 16 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW Conclusion WHAT WHO WHY The range of stakeholhers is wide: World   Bank SMS, IVR and direct phone calls are three staff (TTLs), Ministry of Education officials, technology solutions that can be used on all governmental agencies working on learning kinds of phones to conduct remote formative assessment activities, other governmental or assessments during the time of school regional institutions specialized in curriculum closures and impact evaluations. But for the and learning assessment, NGOs and private purposes of making high-stakes decisions organizations working in the field of education, or monitoring learning at the system level, researchers and technical experts in learning phone-based assessments are generally not assessment. a suitable option. Each technology solution has advantages Before implementing remote phone-based WHAT and limitations. For instance SMS are cost formative assessments, the following eight HOW effective but have 160 character-limit and does elements should be considered: not always support all language scripts hence it On education system capacity: may pose additional challenges when assessing (a) Alignment with existing learning content, (b) literacy. Such limitations should be carefully Human resources; considered before finalizing a technology On technological and logistical considerations solution for assessment. (c) Demographics, (d) Technology, (e) Service providers, (f) Financial, (g) Time, (h) Local implementor’s capacity. 17 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES To go further CLOUD OF KPs R E L AT E D S O U R C E S AI/ML Devices Cloud Adaptive Learning Connectivity Go to the next page for linked Digital Infrastructure blogs, podcasts, documents, Digital identity BE DATA Data visualization ENGAGE the events and tools. DRIVEN NRENs ECOSYSTEM Procurement Ecosystem Startups Data collection EMIS AssistiveTechnologies LMS LEARNER DESIGN and Mobile based S T AY C O N N E C T E D Computer based ACT AT SCALE, Digital ASK WHY? FOR ALL Assessment Literacies Follow us on Twitter Digital Content EMPOWERED TEACHERS Subscribe to our podcast channel Teachers Competencies Spotify & Anchor More updates on Medium Subscribe to our EduTech Newsletter OTHER EXISTING EdTech website RELATED KPs LMS Digital content Teachers’ Mobile distance & competencies Hybrid Ed Solutions 18 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES HOW Annexes WHAT REFERENCES AND LINKS OF INTEREST • Innovations in phone-based assessments to • Are students still learning during COVID-19? EVENTS support learning Formative assessment can provide the answer • Building and assessing youth skills remotely • Assessing outside of the “classroom box” while BLOGS during COVID-19: The experience of Educate! in schools are closed: The potential of phone-based East Africa formative assessments to support learning continuity • Is It Possible to Measure Learning by Phone? PODCAST • Adapting Assessments to the Remote Education Setting • How to Create Learning Content in 160 Characters DOCUMENTS (SMS) • Viamo: Share valuable information with anyone • Practical Lessons for Phone-Based Assessments of who owns a phone in their language of choice. Use TOOLS Learning a combination of channels (voice, SMS, USSD, web • Remote Learning: Evidence from Nepal during app, IM bot, etc). COVID-19 • Questbase: QuestBase is the tool to better and faster test and certify the knowledge of students. 19 INTRODUCTION WHO WHY WHAT HOW CONCLUSION ANNEXES Supported with funding from