Policy Research Working Paper 9685 Scalable Early Warning Systems for School Dropout Prevention Evidence from a 4.000-School Randomized Controlled Trial Francisco Haimovich Emmanuel Vazquez Melissa Adelman Education Global Practice June 2021 Policy Research Working Paper 9685 Abstract Across many low- and middle-income countries, a sizable schools and largely leveraging existing government resources, share of young people drop out of school before completing reduced the dropout rate in the transition from primary a full course of basic education. Early warning systems that to lower secondary school by 4 percent (1.3 percentage accurately identify students at risk of dropout and support points) among schools assigned to the program, and by 9 them with targeted interventions have shown results and percent (3 percentage points) among program compliers. are in widespread use in high-income contexts. This paper Although the effect size is relatively modest, the low cost presents impact evaluation results from an early warning of the program (estimated at less than US$3 per student) system pilot program in Guatemala, a middle-income and successful implementation at scale make this a prom- country where nearly 40 percent of sixth graders drop out ising and cost-effective approach for reducing dropout in before completing ninth grade. The pilot program, which resource-constrained contexts like Guatemala. was implemented in 17 percent of Guatemala’s primary This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at fhaimovich@worldbank.org, evazquez@cedlas.org, and madelman@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Scalable Early Warning Systems for School Dropout prevention: Evidence from a 4.000-School Randomized Controlled Trial* AEA RCT ID: AEARCTR-0004091. Key words: Dropout prevention, Early Warning, Impact Evaluation, School Management, Guatemala JEL Classification: I2, I3, J24 * This study was possible thanks to the support of the Ministry of Education of Guatemala (Ministerio de Educación –MINEDUC). The authors want to also thank participants during seminars and presentations at World Bank and MINEDUC. The views expressed here are those of the authors alone and do not necessarily reflect the opinions of the World Bank. All errors are our own. Contents Introduction............................................................................................................................. 3 2. Context................................................................................................................................ 6 Intervention and study design................................................................................................... 7 The ENTRE program........................................................................................................... 7 Experimental design and data................................................................................................. 13 Eligibility .......................................................................................................................... 13 Experimental design and research questions........................................................................ 13 Data .................................................................................................................................. 14 Descriptive statistics and balancing tests............................................................................. 18 Results .................................................................................................................................. 20 Program effects (Intent-to-Treat Estimates)......................................................................... 20 Placebo tests...................................................................................................................... 24 Mechanisms ...................................................................................................................... 27 Qualitative results .............................................................................................................. 30 Program participation......................................................................................................... 31 LATE ................................................................................................................................ 33 Final Remarks ....................................................................................................................... 36 References ............................................................................................................................ 38 ANNEX ................................................................................................................................ 40 1. Introduction Across many low- and middle- income countries, young people’s educational attainment continues to fall short of aspirations, as high rates of enrollment in early grades quickly decline due to students dropping out before completing a full course of basic education. In Guatemala, the country of focus in this paper, education is de jure compulsory through ninth grade, but nearly 40% of sixth graders drop out before getting there. For youth who drop out prematurely, global evidence suggests that, on average, they will earn less and experience more social and economic challenges than their peers with more years of completed education (Patrinos and Psacharopoulos 2004; Oreopoulos and Salvanes 2011; Bentaouet-Kattan and Székely 2015; Cardenas, De Hoyos, and Székely 2015). Dropping out of school is often not a discrete decision, but the culmination of an ongoing process of disengagement. A growing body of research in education, psychology, and behavioral economics points to the importance of understanding the early origins of dropout and young people’s underlying decision-making processes (O’Donoghue and Rabin 2001; Hammond et al 2007; Mullainathan and Shafir 2013; Frazelle and Nagel 2015). These features suggest that the most effective way to prevent dropout from occurring in the long-term is to build human capital from early childhood onwards. At the same time, there is a pressing need to directly intervene among youth already at high risk of dropping out. Many interventions have shown some promise, depending on the country context and level of education targeted – including information provision on the value of education, remedial tutoring, socioemotional skills coaching, and scholarships (Dearden et al 2009; Di et al 2013; Chappell et al 2015; and Avitabile et al 2019 among many others). These interventions vary significantly in their targeting approaches and level of intensity, resulting in important differences in the resources (human, financial, and otherwise) required to implement them successfully. In countries like Guatemala with high rates of dropout and constrained resources, there is strong demand for accurate targeting and low-cost, context- appropriate interventions. In this paper, we report the outcomes of an early warning system pilot program in Guatemala aimed at reducing dropout in the transition from primary to lower secondary school. Early warning systems – which vary in their structure but usually include monitoring routinely collected 3 indicators of student risk and taking preventative actions tailored to students’ needs – are in place in at least 30 U.S. states and the majority of European countries (European Commission 2013; O’Cummings and Therriault 2015). As more low and middle-income countries invest in developing reliable education management information systems, student data that forms the foundation of an early warning system is becoming increasingly available. However, little evidence exists on whether and how such systems can effectively reduce dropout in countries with limited capacity to respond to students’ needs. In Guatemala, the pilot program, named ENTRE (Estrategia Nacional para la Transición Exitosa – National Strategy for Successful Transition), was designed to utilize recently developed data systems to identify students at risk and activate preventative, evidence-based measures at the school level. ENTRE focuses on three main factors identified as both particularly important for causing dropout in the primary to secondary transition in Guatemala and addressable through relatively low-cost interventions: a lack of knowledge among key primary school actors about effective measures to help students stay in school; a lack of knowledge among these same actors about which students are most at risk of dropping out; and a lack of prioritization of dropout as a problem to be addressed, as school actors have myriad responsibilities and dropout in the transition to lower secondary occurs after students have left their primary schools. To assess the effectiveness of this early warning system approach, implementation of ENTRE was designed as a four-arm randomized experiment (three treatment arms and one control arm) among one-fifth of all the public primary schools in the country. In the first treatment arm, school principals and sixth grade teachers receive a user-friendly guidance manual and half-day training on specific, evidence-based methods to help students make the transition to lower secondary school – in essence, providing them more information on how to help students stay in school. A letter signed by the Minister of Education accompanied the guidance manual, emphasizing the importance of every Guatemalan child completing at least nine years of basic education (primary and lower secondary) and the responsibility of schools to support them in doing so. The second treatment arm adds to the first a list of sixth-grade students at high risk of dropping out based on a prediction model using administrative data– providing them more accurate information on who needs the most help. The third treatment arm adds to the first two small behavioral nudges to try and keep dropout prevention top-of-mind for principals and teachers. A fourth, control arm included schools that received no intervention or information about ENTRE. 4 We find that assignment to the program significantly reduces dropout by 1.3 percentage points (3.6 percent of the baseline dropout rate) in the subsequent school year. The effect size is similar across the three treatment arms, ranging from 1.1 to 1.5 percentage points, and we cannot reject the hypothesis that the impacts are equal across treatment arms. This suggests that the core intervention of providing information on how dropout can be prevented is driving the results. Impacts are heterogeneous, with dropout reduction concentrated among higher-risk students, students attending larger primary schools (and therefore likely facing fewer supply constraints), and male students. Furthermore, imperfect compliance, both in terms of incomplete take-up of treatments and contamination of the control group, suggest that intent-to-treat (ITT) estimates are likely a lower bound of the effect of the program on the treated. We therefore also estimate impacts on program compliers and find that the local average treatment effect is a 3.1 percentage point reduction in dropout (about 8 percent of the baseline dropout rate), roughly double the ITT estimate but much less precisely estimated. To further understand the mechanisms driving the results, we collected data on several intermediate outcomes and conducted a series of focus groups with program actors. Across treatment arms, school principals report an increase in their own expectations of the returns to education for their students, in their beliefs about their own ability to change students’ outcomes, their self-reported prioritization of dropout prevention among their responsibilities, and a large increase in targeting their efforts to at-risk students. While the magnitude of the estimated impacts is modest, the program leverages the Ministry of Education’s existing administrative data structures, management systems, and personnel, with limited additional costs. Specifically, total costs per student are estimated at USD 2.91, 36% of which corresponds to the design of the program— i.e., the cost of implementing the program amounts to USD 1.85 per student— and 45% to the opportunity cost of principals and teachers. Estimates of the monetary benefits accruing to students who do not drop out from higher labor market earnings suggest an internal rate of return of 28% and a benefit-cost ratio of almost 19-to- 1, suggesting that ENTRE is a highly cost-effective, sustainable, and scalable approach to addressing short-run drivers of dropout. Moreover, ENTRE may be a platform program on which additional interventions to reduce dropout can be developed and rolled out, presenting a promising 5 opportunity for Guatemala and other countries interested in leveraging the basic preconditions to identify and support students at risk of dropout. The remainder of the paper is organized as follows. Section 2 describes the education context in Guatemala. Section 3 describes the ENTRE program, while Section 4 describes the experimental design and data. Section 5 presents the impact evaluation results and evidence on likely mechanisms. Section 6 concludes. 2. Context Over the past two decades, Guatemala has continued to make important progress in expanding access to primary and secondary education, as well as in improving student achievement. Primary and lower secondary school completion rates have increased by 43% and 112%, respectively, since 2000. At the same time, average learning outcomes have also improved – Guatemalan sixth graders’ performance in both reading and math improved much more between 2006 and 2013 than the LAC region overall (World Bank, 2019). Increasing average learning outcomes while expanding access is a significant accomplishment, yet much remains to be done. Despite this progress, early school dropout remains an important challenge to increasing educational attainment in Guatemala. School dropout occurs at all ages, but becomes widespread starting in early adolescence. Whereas net enrollment is over 85 percent in primary school, it falls to only 45 percent in lower secondary and 25 percent in upper secondary (UNESCO, 2020). Dropout is particularly concentrated in the transitions from primary to lower secondary (6th to 7th grade) and from lower to upper secondary (9th to 10th grade), where approximately one out of three students drops out in each of these transitions (Adelman et al, 2018). These high dropout rates reflect systemic issues that require medium to long term approaches and significant investment to comprehensively address. In Guatemala, children’s needs are immense. Over half the population is under the national poverty line, and the country has the world’s sixth highest rate of chronic malnutrition (World Bank, 2020). As a result, many children come to the school system suffering from multiple dimensions of deprivation, yet public education services are inadequate. The Government of Guatemala spends less than 3% of GDP on education, well below regional peers, primarily due to low revenues overall, as the government does devote over 20% of its budget to education. As a result, both the quantity and quality of education supply are 6 insufficient to meet existing needs. For example, preschool starts relatively late, typically around age 6, only lasts for one year, and covers less than half of the population. In addition, the supply of secondary education is also insufficient and skewed towards urban and more prosperous areas of the country. In the departments with the highest poverty rates, there is only one secondary school for every 400-500 secondary-school age youth, while in lower poverty departments, this ratio is closer to one school for every 100 youth. Yet there is a pressing need for targeted and scalable action in the short term among the large pool of youth already at high risk of dropping out. Evidence from across Latin America shows that greater shares of youth who are out of school and out of work contribute to a range of negative social and economic outcomes, including higher rates of crime, long-term reductions in labor productivity and growth, and increases in inequality (De Hoyos, Rogers, and Székely 2016). 3. Intervention and study design The ENTRE program In 2017, motivated by the challenges and constraints described above, the Government of Guatemala, with support from the World Bank, designed the ENTRE program as a scalable and low-cost early warning system. The program is aimed at reducing dropout in the transition from primary to lower secondary school, where almost a third of students stop their formal education. Like most early warning systems, the key goals of ENTRE are to provide targeted support to students at-risk and to take timely, preventive actions. At a minimum, this requires that school principals and teachers have a basic knowledge of effective dropout prevention strategies, are able to identify students at risk, and implement these strategies in a timely manner. However, quantitative and qualitative data—including focus groups with teachers and school principals, as well as interviews with public officials conducted to inform the design of ENTRE — showed important gaps in all these dimensions. First, school actors have limited knowledge of, or support on, dropout prevention strategies, and school principals—who play a leading role in these initiatives— are usually regular teachers 7 without any specific leadership or management preparation. Second, school actors have limited information on who is at risk of dropping out. Data on mandatory “dropout risk flags” reported by school principals and teachers show that they identify on average fewer than 5% of the students who end up dropping out in the transition from primary to lower secondary. 1 Third, there are several parallel demands and multiple tasks that compete for the attention of principals and teachers, which may prevent dropout prevention from being top-of-mind, particularly since dropout in the transition from primary to lower secondary occurs after students have left their primary schools and are no longer under their official responsibility. For instance, surveys of school principals show that reducing dropout is not a top priority for them, as they rank it third on average out of five options (below teaching/curriculum and internal administrative tasks). Considering these gaps, the government designed ENTRE to combine three key components: a brief training on actionable, evidence-based strategies to support students to stay in school; information on sixth-grade students at high risk of dropping out; and small behavioral nudges to try and keep dropout prevention top-of-mind for principals and teachers. Component 1: Short training on actionable strategies to prevent dropout The first component consists of a half-day training for school principals and sixth-grade teachers on specific, evidence-based methods to help students make the transition from primary to lower secondary – in essence, providing them more information on how to help students stay in school. The training was complemented by a short and user-friendly guidance manual (13 pages of text and informative illustrations in the main manual, plus 13 pages of additional resources in annex) that summarizes the training content, a copy of which was given to each participant at the beginning of the half-day training. Along with the guide, the school principals received a personalized letter from the Minister of Education that emphasized the importance of every Guatemalan child completing at least nine years of basic education and urged the school principal to lead her or his staff in implementing what they learned through the ENTRE program. 1 Part of this could be explained by perverse incentives to under-report students at risks. The data mentioned is collected by a program that asked school actors to identify students at risk. After identifying students at risk, several administrative tasks were triggered for each student, creating significant additional work for principals and teachers. 8 The training covered the objectives of ENTRE, some of the key barriers to staying in school (lack of motivation, financial constraints, lack of family support, poor academic performance, and inertia), and several methods and resources to address these barriers. For instance, to address the lack of motivation to continue studying, in line with international evidence on cost-effective strategies for this barrier, the training presented information on economic returns to education in Guatemala (which are among the largest in Latin America) and recommended a set of activities to effectively communicate these returns to students. The participants, divided into small groups, then had the opportunity to practice the approach with role-play exercises, and to brainstorm about the logistics of implementation. The sixth-grade teacher and the principal of the same school were intentionally placed together so they could start discussing implementation in their own school. To address financial constraints, the participants were informed about means and merit-based scholarships available through the Ministry of Education (MINEDUC) and provided with instructions on how to apply to them (including where to locate the required forms). As highlighted by empirical evidence (see Bettinger et al, 2018), the participants were also informed about the importance of helping disadvantaged families to fill out these forms, since many times despite qualifying for this type of benefit, low-income families do not apply to them. The training also included a section on the importance of focusing efforts on students at-risk of dropout. Participants were encouraged to use their own judgement to assess this risk, but they were also informed that some schools would receive complementary information on students at-risk as part of MINEDUC’s efforts to identify the most effective content for ENTRE (component 2). Component 2: Information on students at high risk of dropping out The second component of ENTRE is a list of sixth-grade students at high risk of dropping out in the transition to lower secondary, providing school actors with more information on who needs the most help. The list was estimated using routinely collected administrative panel data on students and combining linear regression models with a simple algorithm. These models are able to correctly identify 82% of the sixth-grade students who will drop out within the next year, performing better than other commonly used targeting approaches, and as well as models used in the U.S. (For more details on the methodology, please see Adelman et al., 2018). 9 The list was specific for each school, providing the name and identification number of each student, and was addressed to the school principal. To reduce any risk of negative labeling, both the letter from the Minister and the content of the training emphasized the fundamental importance of avoiding any type of stigmatization of students at risk of dropping out. The main objective of this component was to help school actors target their support to students most in need of it, given limited time and resources.2 Component 3: Behavioral nudges The third component consisted of five monthly reminders sent to school principals in order to keep dropout top-of-mind and motivate them to act. The messages were sent through SIRE, an online portal that school principals use regularly to conduct different administrative tasks and exchange information with MINEDUC. 3 Each month, between June 20 and October 1, these messages appeared on the homepage of the portal as soon as school principals logged in. The reminder messages utilize various behavioral insights to motivate action, namely: • Social recognition (message 1): Social incentives can exert a powerful effect on behavior given the human desire for status and recognition. Social rewards, such as status and recognition, can motivate people to exert effort and can even substitute for monetary rewards in some situations. The first message of SIRE was inspired by this theory, informing school principals about an official certificate to recognize the schools that made the most progress in terms of reducing dropout and the publication of the list of recognized schools in a popular newspaper in Guatemala. This information can act as an extrinsic incentive for school teams to act. • Salience (messages 2-4): Research shows that people are more likely to respond to stimuli that are timely and accessible, and are more likely to do something that their attention is drawn towards. Even though school principals are being provided with concrete ideas, 2 In addition, this information could have also created accountability incentives, given that the schools were informed in advance about the specific students at risk and their dropout status can be monitored easily through MINEDUC’s online portal, SIRE. 3 About 85% of school principals reported in the baseline questionnaire that they use SIRE either “much” or “to a certain extent”, while only 15% of them reported using it either “a little” or “never” (14.3% and 0.25%, respectively). 10 dropout prevention may not be their foremost priority, especially given the number of other tasks they are also responsible for. Salient reminders about their goal (helping at-risk students), and the tools and resources they have at their disposal can draw their attention to the task and be an impetus for action. Messages 2-4 are therefore structured to increase the salience of dropout prevention, including motivational phrases, reminders of key actions, and references to the guidance manual. • Loss frame (message 5): People sometimes put more weight on potential losses than on potential gains. This tendency can also affect people’s level of effort in response to various incentives. Message framing is based on prospect theory (Tversky & Kahneman, 1981), that posits that framing a behavior in terms of its prospective costs or benefits can have significantly different effects on individuals’ decision making. The final reminder message utilizes loss frame to communicate the urgency of acting now since failure to do so could result in losing something, namely the chance for social recognition. Figure 1: message example (first message) Short-message 1: 11 Long-message 1: Implementation In 2018, ENTRE was piloted as a four-arm randomized experiment (three treatment and one control arm) in 4,000 Guatemalan public primary schools. The training took place in two phases. During May 8 and 9, 2018, approximately 70 pedagogical support staff from MINEDUC received a two-day training delivered by core members of MINEDUC’s Technical Working Group on Dropout, with support from World Bank staff. Between May 29 and June 8, these 70 staff traveled around 21 of the 22 departments of Guatemala to train approximately 4,100 school teachers and school principals representing 3,000 schools. 4, 5 At the beginning of the training, participants received an envelope containing the guidance manual, the letter from the minister (if they were school principals), and the list of students at-risk (if they were school principals and were in the corresponding treatment arm). While the materials in the first and second components of ENTRE were distributed during the trainings, the third component of the program -the reminders- was delivered exclusively through SIRE. The first message announcing social recognition was activated about six weeks after the training, on June 20. From that date on, reminders were updated in SIRE on a monthly basis, on July 16, August 6, September 3 and October 1. School principals were informed by their 4 The average number of actors per school is well below two because in many small, rural schools the principal and the sixth grade teacher are the same person. 5 Due to a volcanic eruption, training for school actors in the 22 nd department was postponed and completed in the month of June. 12 supervisors about the presence of a new reminder in SIRE and were invited to enter the online portal each time. The platform was also used to capture baseline and follow-up information through online questionnaires, which complemented baseline administrative information available at the beginning of the school year and a follow-up survey implemented on paper after the school year came to an end. 4. Experimental design and data Eligibility As mentioned above, there are areas in Guatemala where children just do not have reasonable access to secondary schools. Given its features, ENTRE is targeted at schools and students where the supply-side constraints are less likely to be binding. Therefore, to minimize the role of these constraints, the universe of eligible schools was restricted to public schools in the top 30% of municipalities in terms of supply of secondary schools per student. In addition, within those municipalities, the very small schools likely to be located in remote rural areas (the schools belonging to the bottom 15% in terms of school size) were excluded from eligibility, as these schools tend to face binding local secondary school supply limitations. Finally, to target schools that could benefit more from the type of information provided by the program, we dropped schools that—according to our predictive model—had zero students at-risk of dropping out. As a result, the universe of eligible schools totals 6,037 public primary schools.6 Experimental design and research questions The program was designed as a multi-arm randomized experiment. Randomization was used to create four experimental groups of 1,000 schools each (three treatment arms and one control arm). Namely, from the 6,037 eligible schools, a simple random sample of 4,000 was drawn and then each was randomly assigned to one of the four experimental groups. The schools in the first treatment group receive component 1 (the half-day training and the user-friendly guidance manual, as well as a letter from the Minister of Education) which is aimed at improving knowledge on simple evidence-based methods to help students make the transition to lower secondary school – 6 In total, there are approximately 15,930 public and 2,699 private primary schools in Guatemala. 13 in essence, providing teachers and principals with more information on how to help students stay in school. The second treatment arm adds to the first arm the list of sixth-grade students at high risk of dropping out (component 2), providing school actors more accurate information on who needs the most help. The third treatment arm adds to the first two arms small behavioral nudges (component 3) to try and keep dropout prevention top-of-mind for principals and teachers. A fourth, control arm included schools that received no intervention. This multi-arm design, summarized in Diagram 1, allows us to disentangle the impact of each component on dropout. Diagram 1 : four-experimental groups TREATMENT 1 TREATMENT 2 TREATMENT 3 CONTROL Guide/training Guide/training Guide/training No intervention + List + List + Reminders 1000 schools 1000 schools 1000 schools 1000 schools Data Our main data source is the administrative school data set compiled by SIRE. This data set includes unique student identifiers that has enabled longitudinal tracking of student academic records since 2011. The academic records include basic demographic information and detailed grades earned by subject and year, as well as information on enrollment status, including whether the student dropped out and whether they passed the grade. This data was the main input for estimating the dropout prediction model, as well as the data source for the main outcome of interest (student level dropout). We also collected data on the quality of implementation of the program and several intermediate (self-reported) outcomes through a series of paper-based and online questionnaires integrated into the SIRE platform. All schools participating in the experiment were requested to complete an baseline questionnaire online, two follow-up online surveys, and a final, paper-based follow-up survey. In addition, the 3,000 treatment schools were also requested to complete an online 14 comprehension test for the guidance manual and paper-based questionnaires at the end of the half- day training. Table 1 summarizes the main questionnaires used to collect data. Table 1. Summary of data sources Data source/ Key information Response Date Sample Type Questionnaire collected rate (%) SIRE 2016 All Online Key administrative 100% student-level data administrative (placebo), schools (including Students’ IDs, data 2017 (4,000) detailed grades, age, gender, and dropout (placebo / status) prediction) ; 2018 (baseline), 2019 (follow- up) Complementary May 7 th- All Online Perceptions on school 70% dropout, returns to baseline data May 29 th schools education and influences (4,000) on the dropout decision, as well as targeted support and tasks prioritized by principals. Training May 29 th- All Paper- Feedback on the content 91% and quality of the feedback June 6 th treatment based training received and schools about the trainer assigned. Confirmation (3,000) of reception of materials (guide/list), and self- reported level of understanding and 15 satisfaction with the training. Comprehension July 16 th All Online Understanding of the 45% content in the Guide test treatment captured by 12 multiple schools choice questions. (3,000) First-follow up August All Online Perceptions on school 34% dropout, returns to 25 th schools education and influence (4,000) on dropout decision, as well as tasks prioritized by principals. Second follow- September All Online Guide and list take-up, 24% implementation details up-follow 25 th schools and degree of targeted (4,000) support. Third follow-up April 15 th- All Paper- Perceptions on school 85% dropout, returns to May15 th schools based education and influence (next year) (4,000) on dropout decision, as well as targeted support and tasks prioritized by principals + Guide and list take-up and implementation details. Below, we summarize the main data sources for different parts of the analysis, including program participation, outcomes, and potential mechanisms. Program Participation To assess the take-up of the program we utilize multiple data sources. For treatments 1 and 2, we use school principals’ self-reports on the reception of the dropout prevention guide (which is also 16 a proxy of participating in the training) and the list of students at risk, respectively. This information was collected in the last follow-up questionnaire, which was the only paper-based follow-up questionnaire and the one with the highest response rate. The proportion of schools with data on the reception of the guide was about 85 %, with no significant differences across control and treatment arms. For reception of the list of students at risk, the response rate was lower, with significant differences between the treatment and the control groups (the proportion of schools with data on this question was 77% and 55%, respectively). The difference may be driven by school principals in the control group not understanding the question (as they likely never heard about the list of students at-risk). The data on the take-up of treatment 3 comes directly from SIRE administrative records and is proxied by a dummy capturing whether the school principals logged into SIRE at least once in the month that followed the activation of the reminders on the platform. Outcome data The main outcome indicator is a dummy variable capturing whether individual students who were enrolled in primary school in April 2018 had dropped out by April 2019. To construct this variable, we exploit the panel structure of our data—using the students’ identification numbers to track their academic trajectories through different years—with a variable reporting the enrollment status of the students during any given academic year (i.e., currently enrolled or dropped out). If a student is not found in the next year’s panel, we consider that the student has dropped out. In addition, potential errors in the linkage of students’ administrative records across years would also be interpreted as dropouts in our dataset. We do not expect these challenges would affect our identification strategy for two main reasons. First, even with these limitations, the SIRE data replicates well the dropout patterns observed in Guatemalan household surveys (Adelman et al, 2018). Second, there is no reason to believe that this potential measurement error would be different across randomly selected control and treatment groups. In short, we expect relatively small and random measurement error in dropout, creating (if anything) a small downward bias to our estimates. Data on potential mechanisms The data on mechanisms was mostly collected via online questionnaires. There were some challenges with the response rates of the online questionnaires capturing the potential mechanisms, 17 so we consider analysis of this data as exploratory. In particular, 26% (22%) of the treatment (control) schools missed the baseline questionnaire, 13% (11%) the final paper-based questionnaire, and 59% (63%) the short-term online follow-up. However, the key results are robust to controlling for baseline responses and to dropping T3 schools, which had higher response rates.7 Finally, the attrition rate is not correlated with the baseline responses, nor with student dropout rates at baseline or follow-up. Qualitative analysis We also performed a series of focus groups (12 in total) to gather qualitative information on the challenges of program implementation. We interviewed almost 100 people involved in the program, including 63 teachers and principals from across treatment arms and 34 trainers in 4 different locations. 8 Short paper-based questionnaires were also completed during these focus group sessions. Descriptive statistics and balancing tests Table 2 reports the average baseline characteristics for the students and their schools according to the administrative data available.9 The table reports the averages for the control arm (column 1), the pooled treatment arms (column 2), and each treatment arm separately (columns 3-5). The administrative data includes detailed academic grades by subject (math, language, natural sciences, etc.), age, gender, school dropout rates, the average dropout probability for the cohort, and regional variables. Overall treatment and control groups are relatively balanced. Most of the differences in the variables considered are statistically insignificant. The chi squared test after running seemingly unrelated regressions of each of these baseline variables on treatment assignment shows that the 7 These schools had higher response rates in online questionnaires likely because the reminders made schools more prone to see the survey link; furthermore, they explain all the difference on the response rates between treatment and control schools in the online survey used. 8 The teachers and principals come from various departments, including Quetzaltenango, Quiché, Totonicapán, Chimaltenango, Sololá, Sacatepéquez, Jalapa, and Jutiapa. 9 We did not include in this analysis the complementary online data given that the take-up was roughly 70%. 18 differences in all the pre-treatment administrative variables are jointly insignificant (the lowest p- value is 0.22). However, the treatment groups perform worse than the control group in two key dimensions, namely dropout (36% for the pooled treatment vs. 34%), and school size (268 vs. 296, respectively), suggesting that the treatment schools are relatively more vulnerable than the control schools. In particular, the 2-percentage points gap in dropout rates is an issue because it creates a significant downward bias in the impact of the program (particularly taking into account that we are analyzing a very low-cost intervention). To correct for this bias, as will be discussed in the next section, we will include school-fixed effects in our estimations (exploiting the availability of repeated cross-sections from the administrative data). Table 2. Baseline average characteristics of the students in control and treatment groups. Variable Control ENTRE ENTRE 1 ENTRE 2 ENTRE 3 Age 12.91 12.94 12.92 12.95* 12.94 1 = Male 0.50 0.51 0.51 0.52 0.51 1 = Repeater 0.01 0.01 0.01 0.01 0.01 Grade in math (previous year) 7.25 7.25 7.25 7.25 7.25 Grade in language 1 (previous year) 7.42 7.41 7.39 7.44 7.41 Grade in language 2 (previous year) 7.45 7.43 7.42 7.45 7.43 Grade in language 3 (previous year) 7.48 7.43* 7.42 7.45 7.41* Grade in natural science (previous year) 7.47 7.44 7.44 7.44 7.45 Grade in social science (previous year) 7.42 7.42 7.41 7.43 7.41 Grade in artistic expression (previous year) 7.80 7.78 7.77 7.80 7.78 Grade in physical education (previous year) 8.07 8.04 8.03 8.05 8.04 Grade in citizenship education (previous year) 7.68 7.67 7.66 7.68 7.65 Grade in productivity and development (previous year) 7.84 7.82 7.81 7.84 7.81 Risk 0.34 0.36** 0.37* 0.36 0.36 Dropout rate in the transition from 6th to 7th grade in the school that the student attends 0.33 0.36** 0.36* 0.36 0.36 Average age in 6th grade in the school that the student attends 12.91 12.94 12.92 12.95* 12.94 Proportion of male students in 6th grade in the school that the student attends 0.50 0.51 0.51 0.52 0.51 Proportion of repeaters in 6th grade in the school that the student attends 0.01 0.01 0.01 0.01 0.01 Number of students at risk in the school that the student attends 11.47 12.58 12.82 13.05 11.86 Size of the school that the student attends 296.37 268.39* 268.77 261.83* 274.61 Average score in math in 6th grade (previous year) in the school that the student attends 7.30 7.31 7.30 7.27 7.36* Average score in language 1 in 6th grade (previous year) in the school that the student attends 7.47 7.49 7.48 7.47 7.52 Average score in language 2 in 6th grade (previous year) in the school that the student attends 7.53 7.50 7.50 7.50 7.51 Average score in language 3 in 6th grade (previous year) in the school that the student attends 7.52 7.50 7.49 7.51 7.51 Average score in natural sience in 6th grade (previous year) in the school that the student attends 7.51 7.52 7.53 7.51 7.54 Average score in social science in 6th grade (previous year) in the school that the student attends 7.46 7.50 7.49 7.47 7.54** Average score in artistic expression in 6th grade (previous year) in the school that the student attends 7.85 7.85 7.85 7.83 7.87 Average score in physical education in 6th grade (previous year) in the school that the student attends 8.10 8.07 8.04 8.08 8.08 Average score in citizenship education in 6th grade (previous year) in the school that the student 7.75 7.75 7.74 7.74 7.76 Average score in productivity and development in 6th grade (previous year) in the sch. the stud. attends 7.91 7.91 7.89 7.93 7.92 % of students in schools with bilingual modality 0.39 0.41 0.41 0.41 0.41 % of students in municipalities with SINAE 0.21 0.21 0.21 0.19 0.22 % of students in municipalities prioritized by poverty 0.53 0.54 0.53 0.56 0.53 % of students in Guatemala 0.18 0.19 0.19 0.17 0.20 % of students in El Progreso 0.01 0.01 0.01 0.01 0** % of students in Sacatepequez 0.01 0.01 0.01 0.01 0.01 % of students in Chimaltenango 0.05 0.05 0.05 0.04 0.05 % of students in Escuintla 0.06 0.06 0.05 0.06 0.06 % of students in Santa rosa 0.03 0.01 0.01* 0.02 0.01 % of students in Solola 0.04 0.03 0.03 0.05 0.02* % of students in Totonicapan 0.04 0.04 0.04 0.04 0.04 % of students in Quetzaltenango 0.03 0.04 0.05* 0.04 0.03 % of students in Suchitepequez 0.03 0.03 0.03 0.03 0.03 % of students in Retalhuleu 0.02 0.02 0.03 0.02 0.02 % of students in San marcos 0.09 0.09 0.09 0.08 0.09 % of students in Huehuetenango 0.04 0.04 0.04 0.06* 0.04 % of students in Quiche 0.07 0.07 0.07 0.08 0.07 % of students in Baja Verapaz 0.03 0.03 0.02 0.03 0.03 % of students in Alta Verapaz 0.10 0.10 0.09 0.10 0.11 % of students in Peten 0.05 0.04 0.05 0.03** 0.06 % of students in Izabal 0.04 0.04 0.04 0.04 0.05 % of students in Zacapa 0.00 0.00 0.00 0.00 0.00 % of students in Chiquimula 0.03 0.04 0.03 0.04 0.04 % of students in Jalapa 0.05 0.04* 0.03* 0.04 0.03* % of students in Jutiapa 0.03 0.02 0.03 0.03 0.02 Test of joint significance (p-value) 0.90 0.98 0.22 0.45 Number of observations 19940 57235 18974 19175 19086 Notes: (1) * Statistically significant difference with respect to the control group at 10%. ** Statistically significant difference with respect to the control group at 5%. *** Statistically significant difference with respect to the control group at 1%. (2) Standard errors are clustered at school level. 19 5. Results Program effects (Intent-to-Treat Estimates) We start by analyzing the intent-to-treat estimates on dropout. We first estimate the following “pooled” model –which maximizes our power to assess the impact—using a linear probability model (estimated with OLS): = + . ( . ( )) + . + + + (1) Where is the dropout status of student s in school j between year t and year t+1, Entre is a dummy capturing whether the school was (randomly) assigned to the program, is a dummy indicating whether the program is already being implemented (i.e., whether t=school year 2018- 19), represent the students characteristics, and and are year and school fixed-effects.10 Standard errors are clustered by school and year. In addition, we estimate the following model to capture the impact of the (random) assignment to the three alternative treatment arms: = + ∑3 =1 . ( ( ) . ( )) + . + + + (2) Table 3 reports the estimated models. In this table, columns 1-3 show the results for three different specifications: (i) no control variables, (ii) basic student-level characteristics, and (iii) student- level characteristics and school-fixed effects (the last one is our preferred specification and corresponds to equation (1)). The results for the pooled model are reported in the first panel of Table 3. The magnitude of the effect is quite similar in all three specifications, but we gain more precise estimates as we add more 10 Alternative specifications are considered in the robustness section. 20 controls. 11 According to our preferred specification (column 3), attending a school that was assigned to the program significantly reduces dropout by 1.3 percentage points (about 4 percent of the baseline dropout rate) and this effect is significant at the 1 percent level. In the second panel, we report the estimated ITT estimates for each arm. The results for our preferred specification (column 3) indicate that the effect of the treatment is similar across the different arms, ranging from 1.1 to 1.5 percentage points, with all the coefficients statically significant at the 5 or 10 percent level (depending on the arm considered). In fact, we cannot reject the hypothesis that the impact is the same across the three treatment arms, suggesting that the basic intervention on how dropout can be prevented is mostly driving the results. Table 3. Impact of the program on the probability of student dropout ITT (1) (2) (3) Panel (i): Pooled Model ENTRE j -0.010 -0.013** -0.013*** (0.018) (0.006) (0.004) Observations 152,673 145,628 145,628 R-squared 0.000 0.301 0.331 Panel (ii): Treatment arms ENTRE 1 j -0.013 -0.016** -0.015*** (0.022) (0.008) (0.005) ENTRE 2 j -0.011 -0.010 -0.011** (0.022) (0.007) (0.005) ENTRE 3 j -0.007 -0.013* -0.013** (0.023) (0.007) (0.005) Observations 152,673 145,628 145,628 R-squared 0.000 0.301 0.331 Controls at student level NO YES YES School fixed effects NO NO YES Notes: 1) Standard errors clustered at school-year level in parenthesis. 2) *** p<0.01, ** p<0.05, * p<0.1. 3) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1 ; 4) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year of the intervention; 0 = year before the intervention); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group. 5) All specifications in panel (i) include the corresponding Postt and Entrejt dummies, while those in panel (ii) include the Postt and Entre ijt dummies; 6) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System). 11 This is indeed a general pattern that we observe throughout the analysis conducted in this paper. For simplification purposes, in the rest of the paper we mostly report the equivalent to column (iii) –our preferred specification —but more detailed tables are available upon request. We also conduct additional checks in the robustness and placebo sections. 21 In Table 4, we report the same models but include interactions with variables to capture the presence of heterogeneous effects, namely: (i) the ex-ante probability of dropout of the students (based on the predictive model described above); (ii) the school size; and (iii) the gender of the student. As in the previous table, the first panel reports the pooled model, and the second panel the corresponding coefficients for the three arms. 12 Table 4. Heterogeneity in the impact of the program on the probability of student dropout. Interaction: (i) Pre-treatment (ii) School size (iii) Gender dropout probability (=1 if large school) (=1 if boy) Panel (i): Pooled Model ENTREj + (ENTREj ) x (Interaction j ) -0.029** -0.020*** -0.019*** (0.012) (0.006) (0.007) ENTRE j 0.001 -0.005 -0.007 (0.006) (0.006) (0.007) Observations 145,628 145,628 145,628 R-squared 0.333 0.331 0.331 Panel (ii): Treatment arms ENTRE 1j +(ENTRE 1j ) x (Interaction j ) -0.032** -0.026*** -0.032*** (0.015) (0.008) (0.009) ENTRE 2j +(ENTRE 2j ) x (Interaction j ) -0.036** -0.014** -0.011 (0.014) (0.007) (0.008) ENTRE 3j +(ENTRE 3j ) x (Interaction j ) -0.020 -0.021*** -0.013 (0.015) (0.007) (0.009) ENTRE 1 j 0.000 -0.003 0.003 (0.008) (0.008) (0.009) ENTRE 2 j 0.007 -0.007 -0.011 (0.007) (0.008) (0.009) ENTRE 3 j -0.005 -0.004 -0.013 (0.007) (0.008) (0.009) Observations 145,628 145,628 145,628 R-squared 0.333 0.331 0.332 Controls at student level YES YES YES School fixed effects YES YES YES Notes: 1) Standard errors clustered at school-year level in parenthesis; 2) *** p<0.01, ** p<0.05, * p<0.1. 3) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1. 4) ENTRE j represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year of the intervention; 0 = year before the intervention); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group. 5) Interaction j in column (i) is given by the variable Risk sjt (Probability of dropout for student s that attends school j in year t estimated in the Early Warning System); in column (ii) by Large school sjt: (1=Large school: size in 2017 > median), and in column (iii) by Man sjt: (1=Man). 6) All specifications in panel (i) include the corresponding Postt , Entrejt and Interaction j dummies, while those in panel (ii) include the Postt and Entre i jt dummies; 6) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System). 12 In all these cases, to avoid any potential bias, we included interactions not only with the variable capturing the impact of the assignment to the program ((Entre)x(Post)) but also with both the program (when relevant) and year dummy, and we included the variable of interest as a direct control as well. 22 First, we analyze whether the impact of the program varies according to students’ ex-ante probability of dropout. Targeting support to at-risk students is one of the cornerstones of the program’s design, and we would expect a larger impact for those identified as at-risk because they are the population whose outcomes may change due to intervention (versus low-risk students, who are likely to transition to secondary school regardless). Indeed, we find that the impact of the program is almost entirely driven by the students with higher risk of dropout. For the overall program effect, as shown in column 1 of table 4, while the impact of the program was negligible for students with a very low risk of dropout (0.1 percentage points), the program reduced dropout by almost 3 percentage points for the students with the highest risk. This pattern is similar when analyzing the different treatment arms (column 1, second panel), with the largest point estimate corresponding to treatment 2 with a 3.6 percentage point gap. For treatment 3, the pattern is quite similar, but the estimated impact is not statistically significant. Second, we analyze heterogeneity based on school size (second column of table 4). Students in larger schools could benefit more from the program for at least two reasons: first, the fact that supply constrains are less likely to be binding in the areas where they are located (as usually there is more supply of secondary education around larger schools), and second, targeting support is probably more relevant in larger schools, where teachers and principals may have less time for each student. As shown in the first panel, the impact is 1.5 percentage points bigger in large schools (schools above the median school size), adding to 2 percentage points of reduction in dropout. In small schools, the results suggest a (statistically insignificant) reduction in dropout of 0.5 percentage points. The pattern is quite similar across treatment arms (second panel), with the largest point estimate corresponding to treatments 1 and 3. Third, we analyze heterogeneity by gender (third column of table 4). Results could vary by gender for several reasons, including whether the treatment itself is better suited for boys or girls, or whether teachers or principals have biased perceptions on who needs the treatment more. The first panel show that the overall impact of the program tended to favor boys. The reduction in dropout was 1.9 percentage points for boys and only 0.7 percentages points for girls (and this coefficient is not statistically significant). However, this result masks important heterogeneity across treatment groups. In treatment 1—the only group that did not receive “objective” information on the dropout risk by student – basically all the effect of the program is explained by boys (3.2 percentage points), while girls did not benefit from the program at all (the coefficient is indeed 23 positive, but small and not statistically significant). For the other two treatments (both of which received the list of students at-risk) there is no difference between the point estimates for boys and girls, indicating that the benefits of the programs were evenly distributed by gender. Since the only common difference between T1 and T2-T3 is the provision of information on who is at risk, this provides suggestive evidence that the program might have corrected gender-biased perception on who is at risk of dropping out. Indeed, pre-program administrative data does suggest that these biases exist. While observed data shows that boys are 13% less likely to drop out, principals’ perceptions data on who is at-risk of dropout and who needs more support is biased towards boys, and the bias is quantitatively substantial (between 45% and 50% gender gaps in the risk perceptions). This is illustrated in Figure 2. Figure 2. Risk ratios: data vs. perceptions (boys/girls) 150.0% 145.7% 86.9% Observed: dropout risk Perception: dropout risk Perception: needs support Notes: The observed risk ratio measures the actual dropout rate of boys divided by the actual dropout rate of girls, based on observed pre-treatment data (2017). Perception risk ratios are based on the (pre-treatment) self-reported data by school teams on who is at- risk of dropout (or need support), diving the percentage of boys who are marked as at-risk of dropping out (or in need of support) by school principals by the corresponding percentage of girls marked as such by them. Placebo tests Since our identification strategy combines random treatment assignment of schools with school fixed effects, our main identification assumption is that there are no different pre-existing trends for treatment and control schools. To assess if this is a valid assumption, we conduct several placebo tests. While in our main experiment we compare treatment and control schools between 24 April 2018 and April 2019 (i.e., before and after the pilot was implemented), in our placebo experiments we compare them between April 2017 and April 2018 (i.e., one year before the implementation of the ENTRE pilot). If our main experiment is just picking up pre-existing trends, these trends should be also captured by our placebo experiments. Tables 5 and 6 show that the coefficients of the placebo regressions are all quite small in magnitude and statistically insignificant, providing strong evidence against pre-existing trends. This is further illustrated in figures 3 and 4. Table 5. Placebo test of the impact of the program on the probability of student dropout ITT Placebo Experiment Panel (i): Pooled Model ENTRE j -0.000 -0.013*** (0.000) (0.004) Observations 147,977 145,628 R-squared 0.351 0.331 Panel (ii): Treatment arms ENTRE 1 j -0.000 -0.015*** (0.000) (0.005) ENTRE 2 j -0.000 -0.011** (0.000) (0.005) ENTRE 3 j 0.000 -0.013** (0.000) (0.005) Observations 147,977 145,628 R-squared 0.351 0.331 Controls at student level YES YES School fixed effects YES YES Notes: 1) Placebo: estimates using data from 2016 and 2017, when the ENTRE program did not exist. Experiment: estimates using data from 2017 and 2018 (year of the intervention) and the preferred specification in Table 3. 2) Standard errors clustered at school- year level in parenthesis; 3) *** p<0.01, ** p<0.05, * p<0.1. 4) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1. 5) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year 2017 in Placebo; = year 2018 in Experiment; 0 otherwise); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment arm i instead of assignment to the general treatment group. 6) All specifications in panel (i) include the corresponding Postt and Entrejt dummies, while those in panel (ii) include the Postt and Entre ijt dummies; 7) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System). 25 Figure 3. Impact of ENTRE on students’ dropout rate in the experiment and the placebo Notes: The graph shows the ITT estimates of the impact of ENTRE and each of its treatment arms (named Entre 1, Entre 2 and Entre 3) on students’ dropout rate in the real experiment and in the placebo. The latter replicates the estimation using data from 2016 and 2017 (instead of 2017 and 2018), when the program did not exist. Table 6. Placebo test of the heterogeneity in the impact of ENTRE on students’ dropout rate Interaction: (i) Pre-treatment dropout probability (ii) Gender (=1 if boy) (iii) School size (=1 if large school) Placebo Experiment Placebo Experiment Placebo Experiment Panel (i): Pooled Model ENTREj + (ENTREj ) x (Interaction j ) 0.002 -0.029** 0.002 -0.019*** -0.000 -0.020*** (0.005) (0.012) (0.006) (0.007) (0.000) (0.006) ENTRE j -0.002 0.001 -0.002 -0.007 -0.000 -0.005 (0.003) (0.006) (0.006) (0.007) (0.000) (0.006) Observations 147,977 145,628 147,977 145,628 147,977 145,628 R-squared 0.351 0.333 0.351 0.331 0.351 0.331 Panel (ii): Treatment arms ENTRE 1j +(ENTRE 1j ) x (Interaction j ) -0.000 -0.032** 0.004 -0.032*** -0.000 -0.026*** (0.006) (0.015) (0.007) (0.009) (0.000) (0.008) ENTRE 2j +(ENTRE 2j ) x (Interaction j ) 0.007 -0.036** -0.006 -0.011 0.000 -0.014** (0.007) (0.014) (0.007) (0.008) (0.000) (0.007) ENTRE 3j +(ENTRE 3j ) x (Interaction j ) -0.000 -0.020 0.007 -0.013 0.000 -0.021*** (0.006) (0.015) (0.007) (0.009) (0.000) (0.007) ENTRE 1 j -0.001 0.000 -0.005 0.003 0.000 -0.003 (0.003) (0.008) (0.007) (0.009) (0.000) (0.008) ENTRE 2 j -0.005 0.007 0.007 -0.011 -0.000 -0.007 (0.004) (0.007) (0.007) (0.009) (0.000) (0.008) ENTRE 3 j -0.000 -0.005 -0.007 -0.013 -0.000 -0.004 (0.003) (0.007) (0.007) (0.009) (0.000) (0.008) Observations 147,977 145,628 147,977 145,628 147,977 145,628 R-squared 0.351 0.333 0.351 0.332 0.351 0.331 Controls at student level YES YES YES YES YES YES School fixed effects YES YES YES YES YES YES Notes: 1) Placebo: estimates using data from 2016 and 2017, when the ENTRE program did not exist. Experiment: estimates using data from 2017 and 2018 (year of the intervention) and the preferred specification in Table 4. 2) Standard errors clustered at school- year level in parenthesis; 3) *** p<0.01, ** p<0.05, * p<0.1. 4) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1. 5) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that is in the treatment group) and Postt: (1 = year 2017 in Placebo; = year 2018 in Experiment; 0 otherwise); ENTRE ij with i=1,2,3 captures the same interaction as ENTREj but using assignment to the treatment 26 arm i instead of assignment to the general treatment group. 6) Interaction j in columns (i) is given by the variable Risk sjt (Probability of dropout for student s that attends school j in year t estimated in the Early Warning System); in columns (ii) by Large school sjt: (1=Large school: size in 2017 > median), and in columns (iii) by Man sjt: (1=Man). 7) All specifications in panel (i) include the corresponding Postt , Entrejt and Interaction j dummies, while those in panel (ii) include the Postt and Entre i jt dummies; 8) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System). Figure 4. Heterogeneous impact of ENTRE on students’ dropout rate in the experiment and the placebo Notes: The graph shows the ITT estimates of the impact of ENTRE on students’ dropout rate in the real experiment and in the placebo. The latter replicates the estimation using data from 2016 and 2017 (instead of 2017 and 2018), when the program did not exist. Mechanisms Our multi-arm design sheds some light on the potential mechanisms driving the results. As discussed above, the bulk of the impact seems to be explained by the component providing information on how to prevent dropout. In addition, the benefits of the program were mostly captured by students at risk of dropping out. We also find strong suggestive evidence that the information on at-risk students might have contributed to correct school principals’ gender biased- perceptions of who is more likely to drop out. In this section we further explore the potential mechanisms by analyzing data self-reported data from school principals on relevant channels. The analysis suggests that the program had a short- term impact on two relevant factors that might be linked to the knowledge and motivation of school principals: the perceptions of returns to schooling and the dropout expectations for the current academic year. In addition, the program appears to have large and more sustained effects on both the provision of targeted support to students at-risk and the prioritization of dropout as an issue. 27 In the first panel of Table 7 we show the coefficient capturing the (pooled) impact of ENTRE on different self-reported indicators (each coefficient corresponds to a different regression). The second panel reports the number of observations (schools) corresponding to each regression. The outcome variables are expressed as changes between the baseline and follow-up surveys. We measure the impact at two different moments of follow-up: short-term (about 2 months after the training was implemented and about a month before the enrollment for the next academic year begins) and mid-term (in the middle of the next academic year, about 8 months after the training was conducted). Table 7. Impact of the program on different self-reported indicators at school level (1) Coefficient (2) Observations ΔY j : Change between baseline and follow-up in: Short-term Medium-term Short-term Medium-term Dropout expectations (standardized) -0.22*** 0.03 1,005 2,385 (0.08) (0.06) Expected Returns to education (standardized) 0.25** 0.02 1,005 2,255 (0.10) (0.07) Prioritized tasks (dropout) 0.45*** 0.27*** 1,005 2,385 (0.13) (0.08) Prioritized tasks (Authorities' requests) -0.22** -0.16** 1,005 2,373 (0.11) (0.07) Beliefs about principal's influence on the dropout decision (standardized index) 0.03 0.07 1,005 2,383 (0.08) (0.06) Beliefs about main influence on the dropout decision (Principal) -0.01 0.02* 1,005 2,381 (0.01) (0.01) Belief that all students should complete secondary education (standardized index) 0.01 -0.00 1,005 2,400 (0.10) (0.06) Growth mindset proxy (principal's perception) 0.03 0.03* 1,005 2,352 (0.02) (0.02) Knowledge of students' families (standardized index) -0.09 0.01 1,005 2,394 (0.08) (0.05) Frequency of meetings (standardized index) 0.10 0.04 1,005 2,396 (0.09) (0.06) Targeted support (standardized) 0.61*** 0.12** 970 2,387 (0.08) (0.05) Notes: (1) Each coefficient in panel (1) corresponds to a different regression of treatment assignment on the change between baseline and follow-up in the outcome variable mentioned in the name of each raw. (2) Short-term refers to the change between the baseline and the first follow-up and Medium-term corresponds to the change between the baseline and the third follow-up. (3) Observations in panel (2) corresponds to the number of schools without missing data who were therefore used in the regressions. (4) All regressions control for the average characteristics of the students in the school (proportion of men, average risk of dropout and proportion of students of age 11, 12, 13, 14, 15, 16, 17, 18, and 19 & more). (5) Standard errors in parenthesis. (6) *** p<0.01, ** p<0.05, * p<0.1. (7) Standardized variables were transformed to have a mean of 0 and a standard deviation of 1. The analysis shows some interesting findings. First, being assigned to ENTRE reduced principals’ dropout expectations in the short-term by 0.22 standard deviations and increased the perceived returns to education (one of the topics most highlighted in the ENTRE training) by 0.25 standard 28 deviations. However, both effects vanish by the next academic year. This may be capturing a short- term (but timely) impact on the knowledge/motivation of school principals in the few months that followed the training, right before the enrollment process for the next academic year started. We also analyze the impact of the program on a set of variables capturing principals’ perceptions of their role in preventing dropout and of the growth mindset of students. We find some evidence of a positive impact in these dimensions (e.g. the analysis suggests that the program increases the growth mindset mentality of students – as reported by the principal— by 3 percentage points), but the impacts are rather small and only significant at the 10 percent level. In addition, we do not find any impact on principals’ self-reports related to students’ families. The relevance of regular communication with families was one of the aspects highlighted in both the training and the guide. However, according to self-reported data, neither the frequency of meetings nor principals’ perceptions of knowledge of families was affected by ENTRE. Overall, the coefficients (mostly) have the expected signs, but they are statistically insignificant. Next, we look at the impact of the program on targeted support to students at-risk of dropping out. This is at the heart of the ENTRE program and highlighted in a special section of the training for all schools across the three treatment arms. The results show a strong effect in this dimension. According to our estimation, the program improved the provision of targeted support by more than 60% of a standard deviation. While the impact decreased with time, it remained statistically significant in the next academic year. Finally, we looked at the impact of ENTRE on principals’ self-reported priorities. One of the channels through which ENTRE was expected to work was through the higher prioritization of dropout prevention among competing tasks (a challenge identified in the focus groups that informed the design of the program). School principals were asked to rank priorities from 1 to 5, with 1 being the least important, and 5 the most important. Our estimation indicates that being assigned to ENTRE significantly improved the ranking of dropout prevention among competing tasks in both the short and medium-term. Two months after the training was conducted, the ranking of dropout prevention improved by almost 0.5 (about 34% of a standard deviation). In the next academic year, the effect was cut by almost half (but it was still statistically significant). 29 Qualitative results Results from the focus groups with teachers, principals and trainers are mostly in line with the main quantitative outcomes described above. In particular, the dropout prevention guide was the element in the intervention most frequently exalted, with an excellent reception among teachers and principals, which highlights the importance of the core intervention on how dropout can be prevented. School actors also declared an increase in their targeted efforts regardless of the treatment arm to which they belonged. Moreover, principals in the third treatment arm reported that the automated reminders were too frequent and confusing, possibly explaining why the third treatment arm was indistinguishable in its impact. The pattern of heterogeneous effects by dropout probability, school size, and gender is further confirmed by the anecdotical evidence provided in the interviews. Besides the above-mentioned general claim of a greater emphasis on those students who needed more support, principals and teachers in multi-grade schools 13 reported having no time for ENTRE activities, which is consistent with the program’s lack of effect on small schools. In addition, when asked about proposals to improve the program, many school actors suggested that the guide should include differentiated strategies for boys and girls. Self-reported data from open questions in the focus groups also provide information on mechanisms and compliance. Teachers and principals claimed to have focused their strategies on motivating students to pursue secondary. At the time the interviews were made (about 4 months after the training was implemented), school actors showed a strong confidence on the positive results from the program, consistent with the short-term impact on dropout expectations described in the subsection above. Finally, trainers reported some problems with the logistic during the delivery of the guide and the list of students at risk, which suggests that compliance was not perfect and justifies the analysis in the next subsection. 13 Multi-grade schools are schools where students from two or more grades are put together in a classroom. They are typically prevalent among semi-urban and rural areas, and their size is significantly lower than the average school size. 30 Program participation The previous section reports the intent-to-treat effects (ITT), which capture the impacts of being “offered” a treatment regardless of the take-up rate of the components of the program. Another parameter of interest is the Local Average Treatment Effect (LATE), which measures the impact of a program for those individuals who are offered the program and do participate (i.e., the compliers). When there is not full compliance, the ITT tends to underestimate the LATE. As we will discuss in this section, this is likely the case here. The available data indicates that the RCT faced compliance challenges both in terms of partial take-up of the program and contamination of the control group, and hence the ITT estimates are likely a lower-bound of the effect of the program on the treated. Table 8 and Figure 5 describe the take-up of the components by the different treatment arms and the control group. For component 1—which provided information on what to do to prevent dropout—we use as a proxy of participation whether the principals report receiving the dropout prevention guide that was delivered during the training session. As shown in columns (1) and (2) of Table 8, about 80 percent of the schools selected to participate in the ENTRE pilot report receiving the guide, and this percentage was roughly the same across treatment arms. In addition, this data also shows that there was some contamination in this dimension. About one-third of the schools in the control group also report receiving the guide. This could have happened either because they participated in the training or because they managed to get the materials afterwards. We then assess take-up of component 2, which provided information on who is at risk of dropping out. According to the design of the program, schools in treatment groups T2 and T3 should have received an envelope containing both the guide and the list during the training. There are different reasons why take-up of the list was incomplete, including school actors missing their training session, or receiving the wrong envelope (i.e., an incomplete envelope or the envelope for another school). We proxy the take-up of this component with a dummy capturing if the school principals report receiving the list of students at-risk. Columns (3) and (4) of the table shows the take-up rate of the list for each treatment arm. Roughly two-thirds of the schools that should have received the 31 list (T2 and T3) report that they actually received it. On the other hand, a fraction of the schools in the control group (11%) and the T1 group (25%) also report receiving the list. 14 Table 8. Regressions of take-up on treatment and control assignment GUIDE j LIST j REMINDERS j (1) (2) (3) (4) (5) (6) Assigned Treatment 1 j 0.81*** 0.81*** 0.25*** 0.25*** 0.00 -0.00 (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) Assigned Treatment 2 j 0.80*** 0.80*** 0.62*** 0.62*** 0.00 0.00 (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) Assigned Treatment 3 j 0.81*** 0.81*** 0.65*** 0.65*** 0.79*** 0.79*** (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) Assigned CONTROL j 0.32*** 0.32*** 0.11*** 0.11*** 0.00 -0.00 (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) Observations 3,350 3,350 3,350 3,350 3,350 3,350 R-squared 0.75 0.75 0.54 0.54 0.79 0.79 Controls at school level NO YES NO YES NO YES Notes: (1) Standard errors in parenthesis. (2) *** p<0.01, ** p<0.05, * p<0.1. (3) GUIDE j: 1 = School j declared having received the guide in any format; LISTj: 1 = School j declared having received the list of students at-risk of dropout; REMINDERS j: 1 = School j entered the system to see the first reminder. (4) Assigned Treatment i j: (1= School j was in the treatment group i). (5) Controls at school level: Average characteristics of the students in the school (proportion of men, average risk of dropout and proportion of students of age 11, 12, 13, 14, 15, 16, 17, 18, and 19 & more). (6) Regressions are without intercept and controls are centered at the mean. (7) Observations are weighted by school size. Figure 5. Take-up of ENTRE components by the different treatment arms and the control group Note: The graph shows the percentage of schools in each group who received each component of the program. 14 Since we have perfect control on the elaboration of the lists (and these lists were only produced for T2 and T3), we know that this is mostly capturing either measurement error, social desirability bias in the answers of school principals, or schools getting the list of other schools (it is unlikely than the list of students at-risk from another school would be useful in these schools). 32 Finally, we assess the take-up of the reminders sent through SIRE. These reminders were activated in a sample of schools by an experienced technical team of the government using a specific software used to send alerts to subsamples of schools. The team had access to three pieces of information regarding the implementation of this component. First, using a guest id, the team verified that the reminders were posted according to the planned calendar. Second, the team had access to the list of schools that according to the SIRE team received the reminders. This list perfectly matched the treatment assignment. Third, the team had access to schools’ SIRE login activity for the month that followed the activation of the reminders. We combined these last two data sets to proxy the (timely) take-up of the component. Specifically, we approximate the take- up of the reminders with a dummy that equals one if the school logged into SIRE within that month and the SIRE team reports that the school received the reminders. Based on this, almost 80 percent of the schools in T3 were exposed to the reminders in a timely manner, and there was no contamination to other arms. 15 LATE As shown in the previous section, the take-up and contamination rates differ across treatment arms. In this context, it is not straightforward to compute the LATE (Behaghel et al, 2014). To address this, we do a rough estimation of the LATE for the pooled model using participation in the training as a proxy for take-up (as described above). There are three main reasons to select this variable to capture overall participation in the program. First, the training/guide was probably the most relevant activity in terms of both budget and substance (as suggested also by the ITT estimates). Second, other key elements of the program were distributed in the training, namely the Minister’s letter and the list of students at-risk. Third, the quality of the data capturing participation in the training is more reliable than the data capturing participation in other components. 16 15 As mentioned above, it is important to note that the treatment receiving the reminders (T3) ended up having larger response rates in all the online follow-up questionnaires (but not the paper-based or administrative data sets). For instance, the response rate for a “knowledge test” was 40% larger for this group in comparison to T1 or T3. The reason for this was that the link to the questionnaires was placed next to the text of the reminders. This probably created an extra burden for this group of schools that might have had a negative impact on the perception of the program. 16 On the one hand, the response rate for the variable capturing participation in the training/guide is considerably larger than the one corresponding to the list of students at risk (and the response rate does not differ across treatment and control groups). On the other hand, the “take-up” of reminders was reported directly by SIRE officials (we do not have direct data from school principals). 33 Table 9 reports the LATE estimates for the pooled model. The results indicate that ENTRE reduces dropout by 3.1 percentage points for students in complier schools (column 3 of Table 9), roughly doubling the ITT estimates. As expected, the LATE estimations are significantly more imprecise, with the standard errors three times larger than those corresponding to the ITT. Table 10 shows a similar pattern of heterogeneous effects by dropout probability, school size, and gender. The results show that participating in the program reduces dropout by about 6 percentage points for the students with the highest risk, 4.5 percentage points for students in large schools, and approximately 4 percentage points for boys. Table 9. Impact of the program on the probability of student dropout: LATE estimates. LATE (1) (2) (3) ENTRE j -0.025 -0.030** -0.031** (0.040) (0.013) (0.013) Observations 130,762 124,626 124,626 R-squared 0.000 0.294 0.325 Controls at student level NO YES YES School fixed effects NO NO YES Notes: (1) Local Average Treatment effects were estimated by instrumenting treatment (proxied as the effective reception of the ENTRE guide according to self-reported data in follow-up 3) with treatment assignment. (2) Standard errors clustered at school- year level in parenthesis. (3) *** p<0.01, ** p<0.05, * p<0.1. (4) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1 . (5) ENTREj represents the interaction Entrejt x Postt where in this case Entrejt: (1= Student s attends a school j in year t that was treated) and Postt: (1 = year of the intervention; 0 = year before the intervention). (6) All specifications include the corresponding Postt and Entrejt dummies. (7) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System). 34 Table 10. Heterogeneity in the impact of the program on the probability of student dropout: LATE estimates. Interaction: (i) Pre-treatment (ii) School size (iii) Gender dropout probability (=1 if large school) (=1 if boy) ENTREj + (ENTREj ) x (Interaction j ) -0.061* -0.046*** -0.044*** (0.032) (0.018) (0.017) ENTRE j -0.007 -0.013 -0.018 (0.017) (0.020) (0.018) Observations 124,626 124,626 124,626 R-squared 0.327 0.325 0.325 Controls at student level YES YES YES School fixed effects YES YES YES Notes: (1) Local Average Treatment effects were estimated by instrumenting treatment (proxied as the effective reception of the ENTRE guide according to self-reported data in follow-up 3) with treatment assignment. (2) Standard errors clustered at school- year level in parenthesis. (3) *** p<0.01, ** p<0.05, * p<0.1. (4) The dependent variable is a dummy variable that is equal to 1 if student s in school j dropped out school between year t and year t+1. (5) ENTREj represents the interaction Entrejt x Postt where Entrejt: (1= Student s attends a school j in year t that was treated) and Postt: (1 = year of the intervention; 0 = year before the intervention). (6) Interaction j in column (i) is given by the variable Risk sjt (Probability of dropout for student s that attends school j in year t estimated in the Early Warning System); in column (ii) by Large school sjt: (1=Large school: size in 2017 > median), and in column (iii) by Man sjt: (1=Man). (7) All specifications include the corresponding Postt , Entrejt and Interaction j dummies. (8) Student characteristics used as control: age dummies, gender and risk (probability of dropout estimated in the Early Warning System). Scalability and cost-benefit analysis As the size of the RCT suggests, the program was designed to be implemented at scale by relying on existing MINEDUC staff, including trainers, teachers, and school principals, and at a very low cost per student benefited by the program (roughly USD 2 dollars). The training and overall implementation of the program was chiefly in the hands of MINEDUC, with minor support from the World Bank. Expanding ENTRE to additional grades would reduce the costs per student by approximately 76%, given that the training and materials for principals (which represent roughly half of the budget for an average school) are a “fixed cost” for the school, there are many multiple- grade schools and the number of teachers per student is substantially larger in the sixth grade than in the first five years of primary education. The cost-benefit analysis was conducted using the J-PAL tool and following its cost guidelines.17 As previously mentioned, the intervention is low-cost, with a total cost per student of USD 2.91, 36% of which corresponds to the design of the program and 45% to the opportunity cost of 17 See https://www.povertyactionlab.org/resource/conducting-cost-effectiveness-analysis-cea 35 principals and teachers involved in it. Setting aside the one-time-only expenditures in the design of the intervention, the cost of implementing the program amounts to USD 1.85 per student. Estimates of the monetary benefits of ENTRE suggest an internal rate of return of 28% and a benefit-cost ratio of almost 19 with a 5% discount rate over a 40-years span. Another interesting cost-effectiveness measure can be computed by multiplying the main ITT estimate of the program (0.013) by 100 and dividing it by the implementation cost per student (1.85). This measure of additional years of student participation per USD 100 spent allows us to compare ENTRE with other type of interventions. For example, in a recent review from J-PAL, the impact of all the scholarships, conditional cash transfers, and subsides analyzed range between 0.01 and 0.17 additional years of education per USD 100 spent (including the well-known PROGRESA in Mexico with an impact of 0.01 additional years).18 ENTRE is much more cost-effective based on this metric, since it provides 0.70 additional years of student participation per USD 100 spent. Moreover, and perhaps most importantly for scale and sustainability, for Guatemala where per- student government spending at the primary level was over USD 530 in 2018, the costs of a program like ENTRE are feasible to absorb into national public budgets. 19 6. Final Remarks This paper presents experimental evidence that a low-cost and scalable early warning system can significantly reduce school dropout rates in the transition from primary to lower secondary, in a middle-income country with high average dropout. Using a 4,000-school multi-arm randomized experiment, we show that the program reduced the probability of student dropout in treated schools by 1.3 percentage points, and we estimate that this effect grows to 3.1 percentage points when we look at the impact on compliers (roughly 3.6% and 8.5% of the baseline dropout rate). The low implementation cost of the program—about 2 USD per student—makes the intervention highly cost-effective. The (conservative) rate of return of the program is estimated to be almost 30%, with a cost-benefit ratio of approximately USD 19 per each 1 USD invested. The evaluation was embedded in a large-scale implementation including almost 17% of all primary schools in Guatemala, making the results more informative for understanding the impacts of the program in ongoing, “real world” conditions. 18 See Figure 4 in J-PAL Policy Bulletin (2017) for a cost-effectiveness comparison with these and other programs. 19 Data on public education spending from the World Development Indicators. 36 The impact of ENTRE is heterogeneous, with significantly larger effects for students at higher risk of dropout, students in large schools, and boys. In particular, ITT (LATE) estimates show that the program reduced dropout by about 2.9 (6.1) percentage points for the students with the highest risk, 2.1 (4.6) percentage points for students in large schools, and approximately 1.9 (4.4) percentage points for boys. Our multi-arm design sheds some light on the potential mechanisms driving the results. The results indicate that the bulk of the impact seems to be explained by the component providing information on how to prevent dropout (as opposed to providing information on who is at-risk of dropping out or small behavioral nudges to try and keep dropout prevention top-of-mind for principals). In a setting like Guatemala, where school actors have limited knowledge of and support on dropout prevention strategies, and school principals—who play a leading role in these initiatives— are usually regular teachers without any specific leadership or management preparation, the training and guidelines on simple strategies to prevent dropout appear to be highly impactful. This component was also the most intensive in terms of time and resources (representing approximately 92% of the cost of the program). Self-reported data from school principals also helps to further unpack the mechanisms driving the results. This data shows that about two months after the training, principals in treated schools perceived higher returns to schooling, ranked dropout as a higher priority, and provided more targeted support to at-risk students. The impact on targeted support and prioritization are not only large in quantitative terms but also persist in the next academic year (with a smaller but still economically and statistically significant impact). Finally, the results suggest that school actors have gender-biased perceptions of who is at risk of dropout (significantly underestimating the relative likelihood of dropout for girls) and that providing objective information on these probabilities seems to have corrected these biases (and distributed the benefits of the program more evenly across boys and girls). This is an important suggestive result for the dropout prevention literature that calls for further research. 37 References Adelman, Melissa, Francisco Haimovich, Andres Ham, and Emmanuel Vazquez. 2018. “Predicting school dropout with administrative data: new evidence from Guatemala and Honduras.” Education Economics 26(4): 356-372. Avitabile, Ciro, Janina Cuevas, Rafael De Hoyos, and Julian Jamison. 2019. “Addressing High School Dropouts with a Scalable Intervention: The Case of Poder.” World Bank Policy Research Working Paper 9085. Washington, D.C.: World Bank. Behaghel, Luc, Bruno Crépon, and Marc Gurgand. 2014. "Private and Public Provision of Counseling to Job Seekers: Evidence from a Large Controlled Experiment." American Economic Journal: Applied Economics 6 (4): 142-74. Bentaouet-Kattan, R., and Székely, M. 2015. “Patterns, Consequences and Possible Causes of Dropout in Upper Secondary Education in Mexico.” Education Research International, Volume 2015, Article ID 676472. Bettinger, Eric P, Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu. 2012. “The role of application assistance and information in college decisions: Results from the H&R Block FAFSA experiment.” Quarterly Journal of Economics 127: 1205-1242. Cardenas, Mauricio, Rafael de Hoyos, and Miguel Székely. 2015. “Out of School and Out of Work Youth in Latin America: a Persistent Problem in a Decade of Prosperity.” Economia 16 (1). Chappell, Shanan, Patrick O’Connor, Cairen Withington, and Dolores Stegelin. 2015. A Meta- analysis of Dropout Prevention Outcomes and Strategies: A Technical Report in Collaboration with The Center for Educational Partnerships at Old Dominion University). Clemson, SC: National Dropout Prevention Center/Network at Clemson University. Dearden, Lorraine, Carl Emmerson, Christine Frayne, and Costas Meghir. 2009. "Conditional Cash Transfers and School Dropout Rates." The Journal of Human Resources 44 (4): 827-57. De Hoyos, Rafael, Halsey Rogers, and Miguel Székely. 2016. Out of School and Out of Work: Risk and Opportunities for Latin America’s Ninis. Washington, D.C.: World Bank. 38 Di Mo, Linxiu Zhang, Hongmei Yi, Renfu Luo, Scott Rozelle, and Carl Brinton. 2013. “School Dropouts and Conditional Cash Transfers: Evidence from a Randomised Controlled Trial in Rural China's Junior High Schools.” The Journal of Development Studies 49 (2): 190-207. European Commission. 2013. Early Warning Systems in Europe: Practice, Methods, and Lessons. Thematic Working Group on Early School Leaving. Brussels: European Commission. Frazelle, Sarah and Aisling Nagel. 2015. A Practitioner’s Guide to Implementing Early Warning Systems. Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educaitonal Laboratory Northwest. Hammond, Cathy, Dan Linton, Jay Smink, and Sam Drew. 2007. Dropout Risk Factors and Exemplary Programs. Clemson, SC: National Dropout Prevention Center, Communities In Schools, Inc. J-PAL Policy Bulletin. 2017. “Roll Call: Getting Children Into School.” Cambridge, MA: Abdul Latif Jameel Poverty Action Lab. Mullainathan, Sendhil and Eldar Shafir. 2013. Scarcity: Why Having Too Little Means So Much. New York: Times Books / Henry Holt and Company. O’Cummings, Mindee and Susan Therriault. 2015. “From Accountability to Prevention: Early Warning Systems Put Data to Work for Struggling Students.” AIR Early Warning Systems in Education Program. Washington, D.C.: American Institutes for Research (AIR). Oreopoulos, Philip and Kjell Salvanes. 2011. “Priceless: The Nonpecuniary Benefits of Schooling.” Journal of Economic Perspectives 25(1): 159-184. Patrinos, Harry and George Psacharopoulos. 2004. “Returns to Investment in Education: a Further Update.” Education Economics 12(2): 111-134. Tversky, Amos and Daniel Kahneman. 1981. “The Framing of Decisions and the Psychology of Choice.” Science 30 (211): 453-458. 39 O'Donoghue, Ted and Matthew Rabin. 2001. “Choice and Procrastination.” The Quarterly Journal of Economics 116 (1): 121-160. ANNEX Figure A1: Intervention timeline and activities 40