WORLD BANK POLICY BRIEF Photo: NJCTL INNOVATION IN EDUCATION IMPROVING LEARNING OUTCOMES THROUGH ICT TECHNOLOGY Many countries in sub-Saharan Africa identify technology as a viable complementary tool to improve learning outcomes in primary and secondary education. New research evidence from The Gambia suggests that a student-centered computer-assisted learning (CAL) program significantly improves student performance and teaching e ectiveness in secondary schools. Policy action to complement traditional pedagogical methods with interactive technology in classrooms can help countries translate their gains in access to education into greater achievements in learning and skills. WHAT IS THE ISSUE? The remarkable progress toward universal schooling in primary and secondary education in many sub- Saharan African countries has not translated into expected gains in quality learning and skills. 1. HIGH ACCESS BUT LOW LEARNING Africa’s population is young (below 35) and the number is likely to double by 2055, generating Over the last two decades, countries in sub-Saharan potential economic opportunities associated Africa have dramatically improved educational with a growing working-age population. enrollment exceeding 97 percent enrollment rate However, a recent World Bank’s study suggests in primary education in 2017. The gross enrollment that the workforce in sub-Saharan Africa is still ratio in secondary education has also increased among the least skilled in the world, hindering from 13 percent to nearly 43 percent between policy efforts to reap off the benefits of a 1970 and 2014, according to World Bank data. Yet, demographic dividend. overall quality of learning and student performance in mathematics and science remain quite low in comparison with economically similar countries 3. TECHNOLOGY AS A PANACEA? around the world. Just as in other regions of the world, countries in sub-Saharan Africa are considering innovative 2. YOUTHFUL POPULATION BUT ways to match progress in universal schooling with the quality of learning. One area that is INADEQUATE SKILLS increasingly gaining attention is the use of ICT According to the United Nations, three-quarter of technology in the classroom environment. TECHNOLOGY IN THE CLASSROOM AND LEARNING: WHAT CAN WE LEARN FROM THE GAMBIA? In 2012, The Gambia pioneered in Africa the implementation of a new pedagogical innovation called the Progressive Science Initiative® (PSI®) and Progressive Mathematics Initiative® (PMI®), which incorporates technology directly into standard teaching and learning methods. The PSI-PMI program is gaining traction across the region, with other dozen countries having already adopted or lining up to adopt it. In order to guide policy for both the adoption and its implementation, World Bank researchers worked with the Ministry of Basic and Secondary Education of The Gambia to carefully evaluate the impact of the program. This Policy Brief presents the main finding of this research and discusses its implications for policy in The Gambia and beyond. BACKGROUND In 2012, the Ministry of Basic and Secondary Education of The Gambia, with the support of the World Bank, partnered with the New Jersey Center for Teaching and Learning (NJCTL) to implement PSI-PMI as a pilot project in 24 secondary schools in the country (8 Lower Secondary and 16 Upper Secondary). Amid concerns of poor student performance, the PSI-PMI program seeks to improve learning through increased interactions, effective teaching, active student participation, and better monitoring. The PSI-PMI technology integrates an interactive whiteboard (IWB) with smart responders into teaching and learning. The IWB allows teachers to develop digital course content and, with internet access, can foster teacher collaboration through peer review of teaching material. The smart responders are 1 Innovation in Education - World Bank Africa Brief wireless handheld devices that allow multiple students to simultaneously provide PSI-PMI IMPLEMENTATION TIMELINE responses that teachers can monitor AUGUST 2012 and track in via the IWB. This innovation Teacher training session 1 also allows teachers to track students’ DECEMBER 2012 comprehension in real-time, and enables Teacher training session 2 JANUARY 2013 them to make necessary adjustments, which PSI-PMI instruction in pilot is an innovation compared to the traditional schools APRIL 2013 setting where only one student at a time can respond to a question. By design, the Teacher training session 3 SPRING 2013 PSI-PMI program fosters a student-centered Roll-out expanded to learning experience with recurrent short additional schools assignments and a collaboration-friendly JUNE 2013 seating arrangement. Upper basic students in cohort 1 took a modified To study the impacts of the CAL program GABECE on student learning, researchers focused AUGUST 2013 on senior secondary pilot schools where Teacher training session 4 JANUARY 2014 students participated in the PSI-PMI One school from cohort 2 program (see Figure 1). More specifically, began instruction they assessed the performance of 12th APRIL 2014 grade students on the West African Training of cohort 2 by SPRING 2015 Senior School Certificate Examination cohort 1 teachers (session 5) Cohort 1 students took the (WASSCE), the compulsory high-school exit WASSCE examination. These students were exposed SPRING 2016 to the program for three years, starting Cohorts 1 and 2 students in the 10th grade. The WASSCE is taken took the WASSCE SPRING 2017 at grade 12 and determines placement in Cohort 2 students took tertiary institutions. Given that the selection 2018 the WASSCE of the pilot schools was not random, the Gambian trainers provided researchers relied on a non-experimental training in Rwanda research design approach to evaluate the (February), Nigeria (August), impact of the PSI-PMI program. and Niger (October) Present Pilot program continues and is ready to scale up. Map 1: PSI-PMI Program Schools (Senior Secondary Schools) in The Gambia Source: Blimpo et al. (2020) - Forthcoming Innovation in Education - World Bank Africa Brief 2 STUDY DESIGN To estimate the impacts of the PSI-PMI program on learning Figure 1: PSI-PMI Schools and Non-PSI-PMI outcomes, the research team aimed to examine variations in Schools before and after Matching: student performance that are credibly attributable to the PSI-PMI intervention. Focusing on the 12th grade students in the 16 pilot 4 secondary schools, the team created two control groups of a priori comparable students, one from within the same schools 3 and another from set of comparable schools. They applied Distribution a Propensity Score Matching (PSM) approach to school-level 2 administrative data and generated from a sample of 55 eligible secondary institutions a group of 16 schools that are comparable 1 to the PSI-PMI schools (see Figure 2). The PSM method was the best method the researchers could use given the circumstances. 0 While the researchers can ensure that the program students are 0 0.2 0.4 0.6 0.8 1.0 comparable to the non-program ones using the available data, School Quality Index the PSM provides no guarantee to account for unobservable PSI-PMI program school Non PSI-PMI school characteristics. 4 To attenuate this problem and identify the learning impacts of the 3 PSI-PMI program, the researchers proceeded in several steps. First, they used other survey data they collected, including student- Distribution 2 level data, as additional information in the matching process. This allowed them to create two control groups – one with students in the PSI-PMI schools who did not participate in the pilot and 1 another control with students from the non-program matched schools. The former addresses potential bias that may stem from 0 0 0.2 0.4 0.6 0.8 1.0 differences in school characteristics. Second the researchers School Quality Index conducted two separate but complementary evaluations. Before PSI-PMI program school Non PSI-PMI school students took the WASSCE, they did one set of evaluation using the researchers’ designed exam (pre analysis plan). Then, using Source: Blimpo et al. (2020) - Forthcoming the WASSCE exam scores, they replicated the analysis when it became available to check for consistency of program impacts and to contrast outcomes between treated and control group students. In both research designs, they control for student and school level covariates. Third and finally, they used two different statistical estimation methods to conduct the analysis. The results were consistent across these variations, raising confidence that the findings likely to be causal. Although the PSI-PMI program is exclusively designed for mathematics and science subjects (biology, chemistry, and physics), the program evaluation focused only on mathematics and English. One reason is that the two subjects are the only compulsory subjects for all students and are therefore less likely to suffer from the selection bias stemming from students’ preferences and abilities. Another reason is that student performance in these two subjects is the main determinant of admission into tertiary institutions. Also, looking at the impact of a STEM-focused program on a non-STEM subject such as English could be useful to further inform the role of technology in overall learning, cross-subject learning externalities, and teaching. To better understand the dynamics behind their quantitative analysis, the researchers conducted additional consultations though unstructured interviews and focus group interviews with different stakeholders including students, teachers, principals, and administrators at the ministry of education. 3 Innovation in Education - World Bank Africa Brief KEY FINDINGS The implementation of the PSI-PMI program, a Computer-Assisted-Learning technology for mathematics and science subjects, positively influenced students’ learning outcomes in secondary schools across The Gambia. 1. The PSI-PMI program significantly improved student performance. Æ The PSI-PMI program led to a 9.2 percentage points increases in students’ average score in mathematics (or a 46 percent increase); Æ An increase of 15 percentage points (a threefold increase) in the share of students who obtained credit in both mathematics and English, a criterion for college admission in The Gambia; Æ The PSI-PMI program, a STEM-designated technology, improved student’s English score by 3.87 percentage points, suggesting that IT technology influences overall learning. 2. But these gains in learning were largely driven by high-performing Students at the baseline. Æ The impact is positive and significant only for students who performed at the credit threshold and above in mathematics in their 9th grade certification exam (Figure 2); Æ This particular outcome holds is irrespective of gender and socioeconomic background of the students, provided that they have solid background at the baseline. The stakeholders were generally satisfied with the PSI-PMI program despite a number of implementation challenges and technical constraints. 3. School principals, teachers, and students have highly positive views about the program Æ Results from the qualitative survey and focus group discussions indicate that school administrators, teachers, students, and officials from the Ministry of education are highly welcoming with the program; Æ About 80 and 95 percent of math and science teachers, respectively, who used the PSI-PMI Figure 2: Impact of PSI-PMI on sub-groups of technology in their classrooms believe that it 12-grade students improved teaching effectiveness and learning. Impact of PSI-PMI Program on Students with Varied Baseline Performance 4. But a number of challenges prevent the PSI-PMI 2 Impact on 12th grade math exit exam (WASSCE) from being used in its full potential. Æ Gaps between the contents and the official curriculum, as well as the alignment with how 1 students are traditionally evaluated were reported as challenges; Æ Timely maintenance of equipment was lacking 0 as equipment breakdown persisted to such an extent as to disrupt program implementation in some schools; -1 Æ Few collaboration and peer review among Credit and Above Pass Fail teachers due to limited internet and reliable Math score at 9th grade exit exam (GABECE) electricity supply; Æ Most schools lacked collaboration-friendly round Source: Blimpo et al. (2020) - Forthcoming tables for students as the existing tables were designed for the traditional teaching approach Innovation in Education - World Bank Africa Brief 4 POLICY RECOMMENDATIONS The findings from this study are encouraging and point to several policy levers that can further amplify its effectiveness. 1. Program implementation should take precaution to allow for careful evaluation along the way to allow for timely adjustment and promote learning by doing. It is therefore critical to carefully document (with reliable data) the implementation process and, if possible, to consider a Randomized Control Trial design to allow for more rigorous evaluation to help adjust the program when needed and promote learning by doing. 2. The content of the PSI-PMI require continuous efforts to align with the existing curriculum and student evaluations. Many teachers reported that the content was not sufficiently adapted and there may be a need to adjust it to match the existing Gambian curriculum on which the WASSCE is based. 3. To address the variation of the impact on students and ensure that the program benefits all students, the implementers should monitor the implementation at the classroom level, set up classroom peer observations, exchange visits among teachers within and across schools. 4. More generally, experience sharing among countries (administrators and teachers) can help countries avoid challenges experienced elsewhere. 5. A highly responsive technical support unit per school (or group of schools) to handle technical glitches or replace defective equipment is needed to ensure implementation continuity. 6. The experience from The Gambia demonstrates that reliable electricity is essential for smooth implementation. Therefore, countries should consider school-level investments in redundancies in energy supply such as backup generators or solar panels. Photo: NJCTL 5 Innovation in Education - World Bank Africa Brief ABOUT THIS RESEARCH The background research paper for this policy brief was co-authored by Æ Moussa P. Blimpo, Senior Economist, Africa Chief Economist Office, World Bank Æ Ousman Gajigo, Manager, Research Department, African Development Bank Æ Ryoko Tomita, Senior Economist, Education Global Practice, World Bank Æ Solomon Owusu, PhD Fellow, United Nations University – MERIT Æ Yanbin Xu, PhD student, Georgetown University The Full Paper Is Available Under the Title “Technology in the Classroom and Learning Secondary Schools” and is forthcoming as a World Bank Policy Research Working Paper. For more information you contact the corresponding author Moussa P Blimpo at mblimpo@worldbank.org. ACKNOWLEDGMENT The authors would like to thank The Ministry of Basic and Secondary Education of the Gambia, Meskerem Mulatu and Tanya June Savrimootoo of the World Bank for facilitating this study; Robert Goodman and his colleagues at the New Jersey Center for Teaching and Learning (NJCTL) for the implementation; Souleymane Soumahoro, Beatrice A. Berman, and Rajesh Sharma for their support in producing this policy brief. All errors are those of the authors. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.