A way out? Evidence from two Trials of the Mafita Apprenticeship and Community-Based Skills Training Programs in Northern Nigeria April 21, 2021 Endline Evaluation Report Prepared by the World Bank’s Development Impact Evaluation (DIME) Department Emily Crawford Ben Crost Oeindrila Dube Samih Ferrah Marcus Holmlund Nausheen Khan Eric Mvukiyehe Nuruddeen Sambo Umar 1 Acknowledgments This report is the result of a fruitful collaboration between FCDO Nigeria1 , FCDOs’s Evaluation Unit, Adam Smith International and the Mafita project team, and the World Bank’s Development Impact Evalu- ation Department (DIME). We are grateful to our FCDO colleagues, in particular to Ed Smithson, Richard Sandall, Chris Price, Ogechukwu Omeribe, Tonia Momodu, Kristen Hopkins, Conor Doyle, and Federica Di Battista. We would also like to express our deep gratitude to the entire Mafita team, led by Umar Mohammed and, previously, by Ana Vinambres and Erling Petersen, and to the broader ASI team including Alison Bell and Alex Elphinstone. We are also grateful to Hanovia for collecting the data used in this report, including to their manage- ment team led by Segun Oguntoyinbo, to the survey coordination team, to enumerators, and to back-office staff. We also thank our qualitative research team, led by Julia Bailey and Daniel Shephard and including also Marlana Salmon-Letelier and Tobore Egborge, and to the qualitative data collection team based at the University of Zaria, in particular Usman Isyaku and Faisal Umar. On our own end, we thank our Field Coordinators, Ijeoma Ukoko and Nuruddeen Sambo Umar, for their persistent efforts without which this project would not have been possible. Most importantly, we offer our sincere gratitude to our research participants including Mafita trainees and other northern Nigerian youth, caregivers, master crafts persons, and trainers. Their willingness to share information about themselves and their experience were essential inputs into this research and it is our goal to make sure their participation will contribute to increasingly effective future programs and policies. This impact evaluation study was funded by FCDO Nigeria under the World Bank-administered i2i Trust Fund. The study received IRB clearance from the National Health Research Ethics Committee of Nigeria (NHREC) on 11/11/2018.2 The photos in the report were taken by the quantitative and qualitative research teams. 1 OnSeptember 2, 2020, the UK Department for International Development (DFID) was merged with the Foreign and Commonwealth Office to create the new UK Foreign, Commonwealth, and Development Office (FCDO). 2 NHREC protocol number NHREC/01/01/2007-19/10/2018 and approval number NHREC/01/01/2007-11/11/2018. 2 Contents Executive Summary 8 1 Background 14 1.1 Context and rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2 The Mafita Skills program and impact evaluation objectives . . . . . . . . . . . . . . . . . . . 15 1.3 Evidence from previous empirical studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2 Evaluation Design 21 2.1 Evaluation questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2 Theory of Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.3 Key outcomes of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.4 Experimental IE study design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4.1 Apprenticeship IE design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4.2 COSDECs IE design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.5 Data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.5.1 Quantitative data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.5.2 Qualitative data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.6 Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3 Analytical Framework 31 3.1 Estimating the main effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Estimating heterogeneous effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.3 Potential threats to validity and extensions to the primary analysis . . . . . . . . . . . . . . . 32 3.4 Presentation and interpretation of results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4 Mafita program participation 34 5 Results 36 5.1 Mafita Apprenticeship Program effects on economic outcomes . . . . . . . . . . . . . . . . . . 36 5.1.1 Effects of the Apprenticeship program on income generating activities . . . . . . . . . 37 5.1.2 Effects of the Apprenticeship program on asset ownership and consumption expenditure 40 5.1.3 Effects of the Apprenticeship program on job search and entrepreneurship . . . . . . . 42 5.1.4 Effects of the Apprenticeship program on literacy and numeracy . . . . . . . . . . . . 44 5.1.5 Effects of the Apprenticeship program on savings and loans . . . . . . . . . . . . . . . 46 5.2 Mafita Apprenticeship program effects on non-material outcomes . . . . . . . . . . . . . . . . 47 5.2.1 Effects of the Apprenticeship program on subjective well-being and self esteem . . . . 48 5.2.2 Effects of the Apprenticeship program on attitudes toward female empowerment . . . 49 5.2.3 Effects of the Apprenticeship program on social networks . . . . . . . . . . . . . . . . 49 5.2.4 Effects of the Apprenticeship program on enforcement of religious norms and anti-social behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.3 Mafita COSDEC program effects on economic outcomes . . . . . . . . . . . . . . . . . . . . . 54 5.3.1 Effect of the COSDEC program on income generating activities . . . . . . . . . . . . . 55 5.3.2 Effect of the COSDEC program on asset ownership and consumption expenditure . . 58 3 5.3.3 Effect of the COSDEC program on job search and entrepreneurship behavior . . . . . 60 5.3.4 Effect of the COSDEC program on literacy and numeracy . . . . . . . . . . . . . . . . 61 5.3.5 Effect of the COSDEC program on savings and loans . . . . . . . . . . . . . . . . . . . 61 5.4 Mafita COSDEC program effects on non-material outcomes . . . . . . . . . . . . . . . . . . . 62 5.4.1 Effects of the COSDEC program on subjective well-being and self esteem . . . . . . . 63 5.4.2 Effects of the COSDEC program on attitudes toward female empowerment . . . . . . 64 5.4.3 Effects of the COSDEC program on social networks . . . . . . . . . . . . . . . . . . . 65 5.4.4 Effects of the COSDEC program on enforcement of religious norms and anti-social behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6 Discussion and Policy implications 68 6.1 Summary of results from Mafita apprenticeship and COSDEC programs . . . . . . . . . . . . 68 6.2 Cost-benefit analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 7 Conclusions, policy recommendations, and lessons learned 76 7.1 Conclusions and policy recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.2 Lessons for future program implementation and learning . . . . . . . . . . . . . . . . . . . . . 79 8 References 82 A Appendix 85 A.1 Balance Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 A.1.1 Apprenticeship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 A.1.2 COSDEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 A.2 Attrition from the study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 A.2.1 Apprenticeship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 A.2.2 COSDECs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 A.2.3 Lee Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 A.3 Heterogeneous treatment effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 A.3.1 Apprenticeship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 A.3.2 COSDEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4 List of Tables 1 Main Mafita IE Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2 Sample Result Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3 Apprenticeship effects on income generating activities . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4 Apprenticeship effects on asset pwnership and consumption . . . . . . . . . . . . . . . . . . . . . . . . 41 5 Apprenticeship effects on job search and entrepreneurship behavior . . . . . . . . . . . . . . . . . . . . 43 6 Apprenticeship effect on female self esteem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 7 Apprenticeship effects on professional social networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 8 Apprenticeship effects on religious social networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 9 Apprenticeship effects on religiosity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 10 COSDEC effects on income generating activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 11 COSDEC effects on asset ownership and consumption expenditure . . . . . . . . . . . . . . . . . . . . . 59 12 COSDEC effects on job search and entrepreneurship behavior . . . . . . . . . . . . . . . . . . . . . . . 60 13 COSDEC effects on professional social networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 14 COSDEC effects on religious social networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 15 COSDEC effects on religiosity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 16 Summary Results from the Mafita IEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 17 Cost-benefit comparison for Apprenticeship intervention . . . . . . . . . . . . . . . . . . . . . . . . . . 75 18 Cost-benefit comparison for COSDECs intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 A1 Apprenticeship IE balance test using pre-randomization data . . . . . . . . . . . . . . . . . . . . . . . 85 A2 Apprenticeship IE balance test using Mafita baseline data . . . . . . . . . . . . . . . . . . . . . . . . . 86 A3 COSDECs IE balance test using Mafita pre-randomization data . . . . . . . . . . . . . . . . . . . . . . 88 A4 COSDEC IE balance test using Mafita baseline data . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 A5 Apprenticeship IE attrition test using pre-randomization Mafita administrative data . . . . . . . . . . . . 91 A6 Attrition test using baseline data for Apprenticeship program . . . . . . . . . . . . . . . . . . . . . . . 92 A7 COSDECS IE attrition test using pre-randomization Mafita administrative data . . . . . . . . . . . . . . 94 A8 COSDEC IE attrition test using baseline Mafita data . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 A9 Heterogeneous treatment effects for male and female participants: primary outcomes . . . . . 98 A10 Heterogeneous treatment effects for male and female participants: secondary outcomes . . . . 98 A11 Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: primary outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 A12 Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: secondary outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 A13 Heterogeneous treatment effects for IQE and non-IQE female participants: primary outcomes 99 A14 Heterogeneous treatment effects for IQE and non-IQE female participants: secondary outcomes 99 A15 Heterogeneous treatment effects for male and female participants: primary outcomes . . . . . 100 A16 Heterogeneous treatment effects for male and female participants: secondary outcomes . . . . 100 A17 Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: primary outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 A18 Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: secondary outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 A19 Heterogeneous treatment effects for IQE and non-IQE female participants: primary outcomes 101 A20 Heterogeneous treatment effects for IQE and non-IQE female participants: secondary outcomes101 5 List of Figures 1 Mafita pilot states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2 Theory of Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3 Apprenticeship IE Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4 COSDEC IE Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5 Training for quantitative data collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 6 Qualitative fieldwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 7 Mafita IE Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 8 Sample Forest Plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 9 Mafita program participation for treatment and control groups . . . . . . . . . . . . . . . . . . . . . . . 35 10 Number of days of training attended and graduation rates . . . . . . . . . . . . . . . . . . . . . . . . . 35 11 Summary effects of the Apprenticeship program on economic welfare outcomes . . . . . . . . . . . . . . . 37 12 Apprenticeship effects on income generating activities across youth categories . . . . . . . . . . . . . . . . 39 13 Apprenticeship effects on self and wage employment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 14 Apprenticeship effects on asset ownership and consumption across youth categories . . . . . . . . . . . . . 42 15 Apprenticeship effects on entrepreneurship and job search behavior . . . . . . . . . . . . . . . . . . . . 44 16 Apprenticeship effects on job search and entrepreneurship behavior across youth categories . . . . . . . . . 44 17 Apprenticeship effects on savings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 18 Apprenticeship effects on loans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 19 Apprenticeship intervention effect on non-material outcomes . . . . . . . . . . . . . . . . . . . . . . . . 47 20 Distribution of MHI-scores for treatment and control youth group . . . . . . . . . . . . . . . . . . . . . 49 21 Apprenticeship effects on range of professional social network by youth category. . . . . . . . . . . . . . . 51 22 Apprenticeship effects on participation in anti-social behavior (in the past 30 days) . . . . . . . . . . . . . 53 23 Apprenticeship intervention effect on participation in political or religious violence (in the past 3 months) . . 53 24 Apprenticeship intervention effect on attitudes toward violence . . . . . . . . . . . . . . . . . . . . . . . 54 25 Summary effects of the COSDEC program on economic welfare outcomes . . . . . . . . . . . . . . . . . 55 26 COSDEC effects on income generating activities across youth categories . . . . . . . . . . . . . . . . . . 56 27 COSDEC effects on self employment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 28 COSDEC effects on profit from self employment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 29 COSDEC effects on wage employment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 30 COSDEC effects on job search and entrepreneurship . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 31 COSDEC effects on savings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 32 COSDEC effects on loans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 33 Summary effects of the COSDEC program on non-material outcomes . . . . . . . . . . . . . . . . . . . 63 34 COSDEC treatment and control groups’ life satisfaction now and in 5 years’ time . . . . . . . . . . . . . . 64 35 Distribution of MHI-scores for treatment and control youth group . . . . . . . . . . . . . . . . . . . . . 64 36 COSDEC effects on professional social networks by youth category. . . . . . . . . . . . . . . . . . . . . 66 37 COSDEC effects on participation in anti-social behavior (in the past 30 days) . . . . . . . . . . . . . . . 67 38 COSDEC intervention effects on participation in political or religious violence (in the past 3 months) . . . . 68 39 COSDEC intervention effects on attitudes toward violence . . . . . . . . . . . . . . . . . . . . . . . . . 68 A1 Apprenticeship program effect ranges on material outcomes, adjusted for attrition . . . . . . . . . . . . . 96 A2 Apprenticeship program effects on non-material outcomes, adjusted for attrition . . . . . . . . . . . . . . 97 A3 COSDEC program effect on material outcomes, adjusted for attrition . . . . . . . . . . . . . . . . . . . 97 6 A4 COSDEC program effect on non-material outcomes, adjusted for attrition . . . . . . . . . . . . . . . . . 98 7 Executive Summary A key policy priority across the world is reducing the gap between the number of young peo- ple looking for work and the limited employment opportunities that are available to them. Marginalized youth in particular are often trapped in a perpetual cycle of economic and social despair where poverty and the lack of opportunities constrains their economic and other life outcomes. This further en- trenches poverty and other negative outcomes. An additional concern, especially in settings affected by fragility, conflict, and violence, is that economic and social marginalization may push youth towards crimi- nality, violence, and radicalization. A frequent policy response is to provide training and skills development opportunities to youth. Increasingly, such programs aim to provide a more comprehensive package including, for example, access to finance, entrepreneurship training, or other forms of support that may be useful in entering the labor market. In recent years, there has been a growing number of rigorous evaluations of such programs. However, the evidence base remains limited and there are several important open questions concerning, for example, the effectiveness of different skills training models such as more traditional classroom based voca- tional training and “real-world” apprenticeships, and how to tailor such programs for at-risk youth. This report presents findings from two randomized controlled trials (RCTs) incorporated in the Mafita program, a 5-year, UK GBP 36 million program funded by the UK Foreign, Com- monwealth, and Development Office (FCDO) and implemented by Adam Smith International (ASI) in four Northern Nigerian states: Kaduna, Kano, Katsina and Jigawa. Mafita’s primary goal was to give marginalized youth a better chance of finding skilled work or productive self-employment with the ultimate aim of increasing youths’ incomes. The program was also seen as having the potential to reduce grievances and decrease youth propensity to engage in crime and violence. The program was designed to target three primary groups of youth: (i) Almajiri (20%); (ii) adolescent girls including those participating in Islamic Quranic education (IQE; 20%); and (iii) other youth, including persons with disabilities, orphans and vulnerable children, and early school leavers (60%). This study was led by the World Bank’s Development Impact Evaluation Department (DIME), in close partnership with FCDO and ASI, and focused on two key programs under Mafita: • Apprenticeships. Participants were assigned to work with clusters of master craft persons (MCPs) to develop trade-specific skills through a learning-by-doing approach. Apprenticeships took place in Kano State for the first cohort of training, which is the focus of this report. Apprenticeships were preceded by basic foundational skills training in literacy and numeracy and business-related soft skills. • Community Skills Development Centers (COSDECs). COSDECs provided vocational and technical training aimed at developing trade-specific skills through a classroom-based approach in which fully equipped workshops and production facilities were used to train students. The COSDEC model was implemented in Kaduna State and Katsina State for the first cohort of training and included basic foundational skills training in literacy, numeracy, and business-related soft skills. A key difference between these programs is thus that the former built on traditional apprenticeships with 8 MCPs through a less formal, learning-by-doing approach while the latter was implemented in government- operated vocational training centers in a classroom setting. Each program ran for 9-12 months. The Mafita impact evaluations (IEs) described in this report were designed to rigorously evaluate these two programs to generate evidence on effective mechanisms to create jobs and opportunities in very poor and challenging places. Both the Apprenticeship and COSDEC programs were conducted as separate randomized controlled trials (RCTs): for each program, a subset of eligible candidates was randomly assigned to the program while others were assigned to the control group for the duration of the study. In addition to labor market impacts, the RCTs also explore whether these programs altered social, political and religious attitudes, and tendencies to engage in criminal activity and violent behavior. Furthermore, the IEs investigate whether impacts vary by gender or category of youth. It should be noted that the IEs, as agreed with the partners at the outset, were not designed to identify the marginal contribution of different elements of the Mafita program package which, in addition to skills training, in- cluded entrepreneurship training, internship opportunities, and access to finance interventions. Instead, the focus is on the overall effects of the programs as a whole. The IEs investigate the impacts of the Mafita programs on a set of labor market and eco- nomic welfare outcomes and on non-material and social outcomes. Labor market and economic welfare outcomes include income-generating activities (employment and earnings), household asset owner- ship and individual-level consumption, job-search behavior, and literacy and numeracy skills. Non-material and social outcomes include psychological well-being and self-esteem, social networks, views on female em- powerment, attitudes towards and participation in anti-social behavior and different forms of violence, and religiosity. The IEs evaluate the first cohort of Mafita participants in both the Apprenticeship and COSDEC programs which was implemented in Kaduna, Kano, and Katsina States. In this cohort, between the two programs, Mafita delivered training to about 4,000 youth. Final quantitative data collection on treatment and control groups was conducted between December 2018 and March 2019, six-to- nine months after program completion. Results of the Apprenticeship program Overall, the Mafita apprenticeship program had strong and positive effects on participants’ employment and productivity, job search behavior, and economic welfare, with some outcomes displaying striking heterogeneity in the results. Relative to the control group, Mafita apprentices were 17% (8 percentage points) more likely to be employed in self- or family-owned businesses (henceforth, self- employed), and 10% (2 percentage points) more likely to be in wage employment; raised their profits from self-employment by 17% (though income from wage-employment did not increase); and increased the time they spent on self-employment activities by 9%. They were also 12% (7 percentage points) more likely to have attempted to start a business in the past 6 months and 27% (7 percentage points) more likely to have looked for a job in the past 30 days. Former apprentices also had higher household asset ownership and consumption expenditures than did members of the control group. In particular, their households were 5% more likely to own bank accounts and apprentices themselves spent 8% more on food and 59% more on other expenses (measured over 7 or 30 days preceding the survey, respectively). We do not find evidence of 9 impact on literacy and numeracy overall though complementary qualitative research suggests some Mafita participants perceived improvements in these areas, and also in trade-specific technical skills. The study finds mixed evidence of impact on non-material outcomes. We do not find evidence that the program increased participants’ psychological well-being on the whole, though we see a small but significant increase in self-esteem for female participants. We also find that the program expands partici- pants’ professional social networks, thought we do not find any changes in the composition of religious social networks. This former impact is driven by a 7% increase in the number of people in former apprentices’ networks who are employed. We do not find evidence of impact on attitudes towards female empowerment from the perspectives of the youth or their caregivers. We also find no evidence that the apprenticeship pro- gram affected participation in anti-social behavior including stealing, taking drugs, or working for criminal groups; participation in riots; or the use of violence for political or religious motives. It should be noted, however, that these behaviors are very limited in youth participating in Mafita to begin with. Results of the COSDEC program Overall, the Mafita COSDEC program had strong and positive effects on participants’ em- ployment, productivity, entrepreneurship behavior, and economic welfare, with some outcomes displaying striking heterogeneity in the results. Relative to the control group, COSDEC participants were 35% (14 percentage points) more likely to be employed in self- or family-owned businesses; 39% (4 percentage points) more likely to be in wage employment; increased profits from self-employment by 38% and income from wage employment by 54%; and increased the time spent on self- and wage-employment by 31% and 22%, respectively. They are also 20% and 22% more likely to have attempted to start a business in the past six months or one month, respectively. Former COSDEC participants also reported higher con- sumption expenditure and household asset ownership than did members of the control group. We do not find evidence of impact on literacy and numeracy, though complementary qualitative research suggests some Mafita participants perceived improvements in these areas also. The study finds mixed evidence of impact on non-material outcomes. To summarize, we find evidence that the intervention increased participants’ subjective well-being, driven by improvements for female participants and IQE girls in particular. We also find that the program expands participants’ professional social networks, though we do not find any changes in the composition of religious social net- works. Finally, we find strong evidence of impacts on attitudes towards female empowerment both among participants themselves (driven by non-IQE girls and non-Almajiri boys) and among caregivers. However, we do not find evidence of impacts on anti-social behavior including stealing, taking drugs, or working for criminal groups. It should be noted, however, that these behaviors are very limited in youth participating in Mafita to begin with. We also find no evidence that the COSDEC intervention affected participation in riots or the use of violence for political or religious motives. Cost-benefit analysis and qualitative evaluation results While the study was not designed to identify the relative effectiveness of the two Mafita programs, a preliminary cost-benefit exercise suggests that the COSDEC program was more cost-effective. Under a baseline scenario, which assumes that economic benefits to participants decrease 10 by 5% per year after the end of the training to reflect the depreciation of acquired skills and a discount (or time-preference) rate of 5%, the COSDEC program breaks even after 19 years and has a return on invest- ment of approximately 10% over 30 years. The Apprenticeship program, on the other hand, breaks even only under more optimistic assumptions, and has a substantially lower discounted net benefit. The cost-benefit analysis, while exploratory and based on limited data, highlights that it is important to understand the sources of variation of cost-effectiveness for the design of future programs. Results from a complementary qualitative evaluation corroborate the quantitative IE find- ings for both the apprenticeship and COSDEC programs. One area of disagreement, however, is with regards to foundational skills development: while the quantitative IE does not find evidence of consis- tent impacts, participants in the qualitative evaluation cite training in literacy and numeracy as a program benefit. This suggests there may have been sub-groups for whom the program did increase these basic skills, even if these increases where not sufficiently large or consistent to be detected through the quantitative anal- ysis. Furthermore, the qualitative evaluation provides insight on participants’ high valuation of the technical skills training provided by Mafita, especially for female participants. Conclusions and policy implications There are several important high-level findings. First, both the Apprenticeship and COSDEC pro- grams were successful in increasing participants’ engagement in income generating activities and earnings, which in turn contributed to the improvement of their economic welfare in terms of increased consumption and asset ownership. Second, for both programs, improvements in outcomes related to income-generating activities appear driven by female participants for whom we measure large increases in self-employment and wage employment, especially for IQE girls. Third, neither program appears to have had positive impacts on basic literacy and numeracy skills, though qualitative evidence suggests participants had positive per- ceptions of both the foundational and trade-specific skills training. Fourth, the effects of both programs on non-material outcomes are mixed, with COSDEC program showing better results on some non-material outcomes. A notable exception is the expansion of participants’ processional social networks, which shows strong positive impacts for both programs. Overall, the results are encouraging and suggest that Apprenticeship and COSDEC-type programs can be used to promote self-and wage employment and increase earnings for vulner- able youth in fragile and conflict-prone settings. The results are particularly relevant for projects and policies that aim to promote gender equality in economic or social development. Additionally, the findings have policy implications regarding targeting and highlight the potential of targeting particularly under- served and vulnerable groups: Mafita had strong impacts on female participants, in particular IQE girls, who experienced strong increases in self-employment and income. There is also evidence, from the COSDEC program, that women’s participation can improve attitudes towards female empowerment by participants and their caregivers, a striking result given strong gender norms in northern Nigeria. Future research should consider ways to expand impacts on such gender-related attitudes and ultimately on behaviors and norms. At the same time, it is important to recognize the limitations of programs such as Mafita and incorporate implications into future programming. Mafita contributed positively to self-employment for vulnerable youth, but the results also highlight that there are additional external constraints to wage- 11 employment which (i) Mafita may not have been equally successful at addressing and/or (ii) may require additional programs (outside of Mafita’s scope) or structural changes. Additionally, while impacts on in- come are large on a relative scale, in terms of absolute gains they remain modest at best. Furthermore, the lack of positive impact on improvement in foundational literacy and numeracy skills, which was a key component of the program spanning 3-6 months on average, is concerning and warrants further attention as low levels of these basic skills could constrain the growth of youth’s micro-enterprises and prohibit entry into higher-productivity activities and wage labor. This raises an important question about how to more effectively incorporate such foundational skills into training and jobs programs such as Mafita. The study also shows that improvements in economic outcomes do not necessarily translate automatically into better outcomes in other domains, such as reductions in violence and other anti-social behavior. To the extent that these are goals of future programs, they need to be explicitly factored into targeting and program design. Beyond improving youths’ economic prospects, a frequent justi- fication for programs such as Mafita is that they will reduce youth participation in crime, violence, and other risky or anti-social behaviors. This study, however, does not find evidence in support of this hypothesis. There are two plausible explanations: first, these prevalence of these behaviors in youth attracted to Mafita is generally low; second, to the extent that these behaviors do exist, they are not explicitly targeted by pro- gram activities. This suggests that, when reducing such behaviors is indeed an objective, future programs should explicitly target youth engaged in these behaviors and factor this into intervention design. Finally, it is important to keep in mind that the findings in this report capture the effects of the Mafita apprenticeship and COSDECs programs in the relatively short term, as they are based on data that was collected six to nine months after program completion. This study therefore cannot speak to the longer-term sustainability or trajectory of impacts. However, it is important to note that impacts on some of the key outcomes in this study, such as those related to the labor market, often take time to materialize. Hence, it is possible the impacts reported on some of the key outcomes could be on the lower bound of the spectrum of potential effects. The findings from the two trials of the Mafita Apprenticeship and COSDEC programs in Northern Nigeria suggest several policy recommendations for future programming and re- search. • Recommendation #1: Apprenticeship and vocational training programs such as Mafita can be used to promote labor market outcomes of vulnerable youth in fragile and conflict-prone settings. Such programs should build on this research to incorporate design features that have been shown or that are highly likely to be effective and achieve positive impacts. • Recommendation #2: Future evaluations of multi-faceted employment programs such as Mafita should be designed to evaluate the relative contributions of specific program components. Such IEs should focus both on the design of these components and on their value-added in achieving program outcomes. • Recommendation #3: Program components recommended as focus areas for future rigorous evalu- ations include foundational skills training, entrepreneurship training, and access to finance. 12 • Recommendation #4: Future programs that build on Mafita must carefully consider cost-drivers to improve the cost-benefit ratio. Management information systems (MIS) should be strengthened to record high quality implementation and cost data on different program activities, which is needed to be able to more accurately estimate costs and benefits of the program as a whole and of specific components. • Recommendation #5: Future programs should leverage the potential to promote female empower- ment and to deepen and expand impacts on gender-related attitudes, behaviors, and norms. Further experimental research would be critical to ascertain the sustainability and expansion of gender im- pacts from labor market programs such as Mafita, and crucially, to investigate potential channels of influence. • Recommendation #6: Future programs should put in place incentive structures that encourage and promote new social and professional networks that may increase participants’ access to more job opportunities and enhance their employability. • Recommendation #7: To the extent that non-material dimensions and outcomes, such as psycho- logical well-being and self-esteem or reduced participation in violence and anti-social behavior, are program objectives, they should be more explicitly incorporated in program design, through targeting and non-employment-related interventions. • Recommendation #8: Future collaborative efforts to embed rigorous evaluations into program de- sign should incorporate (i) a participatory program/evaluation design and management process for buy-in from all stakeholders; (ii) regular communications protocols; and (iii) a robust management information system (MIS) to carefully monitor and document program activities and record adminis- trative data. The findings from the two trials of the Mafita program add critical evidence to help ascertain the efficacy and effectiveness of strategies designed to improve life outcomes for at-risk youth living under difficult circumstances with limited prospects. This is noteworthy, especially given the fragile northern Nigeria context where these trials took place. FCDO and ASI are to be commended for incorporating a technically rigorous, independent impact evaluation into program design to build the evidence base and support future policies and programs in this critical area. 13 1 Background 1.1 Context and rationale Since the late 1970s, Northern Nigeria has experienced religious and ethnic conflicts that have led to substantial violence and instability. The most recent intensification of conflict began shortly after the restoration of democracy in Nigeria in 1999, when several state governments in Northern Nigeria introduced legal reforms based on Sharia Law. This resulted in several episodes of riots and clashes between Muslims and Christians, beginning with the Kaduna riots in 2000, which led to several thousands of deaths. More recently, the region has been further destabilized by the rise of the insurgent group Boko Haram, which has carried out several high-profile attacks on Christian and moderate Muslim establishments since 2009. From 2013 through 2018, there were over 33,000 conflict-related deaths although there has been a decrease since 2016.3 While the conflict has been most prevalent in the North East, particularly in Borno state where Boko Haram activities have been concentrated, both state and non-state related deaths have also been reported in the North West (Matfess, 2019). In addition to religiously motivated conflict, Northern Nigeria has also experienced violence arising from political or other secular motives, including post-election violence in 2011 and confrontations between rival youth gangs known as “Yandaba”. Conflict and instability are thought to be fueled in part by the region’s high rates of poverty and unemployment and its large population of marginalized urban youth. Lack of employment prospects and poverty make young men more likely to join violent and criminal groups and to mobilize polit- ically for rents (Herbert & Husaini, 2018). This becomes a perpetual cycle: poverty and lack of opportunities limit the ability of youth to improve their capacities and this absence of a skilled labor force constrains in- vestment in the region, which further decreases opportunity for youth and increases their exposure to various risks. Youth in Northern Nigeria are perhaps especially affected by the challenging economic situation given the relatively vulnerability of this part of the country as a whole: a recent World Bank report states that almost half of the poor in Nigeria lived in the North West of the country and fully 87% live in the North in 2016 (The World Bank, 2019).4 Unemployment and poverty are particularly high among youth. In 2016, youth aged 15 to 24 accounted for half of all unemployed or underemployed Nigerians (Ministry of Budget & National Planning, 2017). Furthermore, 21% of youth aged 15 to 24 were not in employment, education, or training (NEET), with young women more likely to fall in this category (25% of women compared to 18% of men). Among those working, most are still in poverty: The ILO estimates that, among those working in the 15+ age group, 36.6% are making less than $1.90 per day while, within the 15-24-year age group, this number jumps to 50.7%. If we break this down by gender in the 15-24 age group, 55.5% of working young men and 43.8% of working young women are making less than $1.90 per day.5 In the northern Nigerian context, so-called Almajiri boys, children of poor families from rural areas who migrate to cities to study in Qur’anic schools are notable. Observers of the region worry that, due to their lack of economic opportunities, these youths become easy targets for recruitment by extremist groups. 3 See UCDP/PRIO Armed Conflict Dataset available at https://ucdp.uu.se/country/475https://ucdp.uu.se/country/475. Accessed on April 1, 2020. 4 Nigeria’s Northern region and North West geopolitical zone account, respectively for about 50% and 20% of Nigeria’s population. Note that the North comprises three geopolitical zones: North West, North East, and North Central. 5 See ILO Stats. https://www.ilo.org/shinyapps/bulkexplorer53/?lang=en&segment=ref area&id=NGA A. Accessed on April 1, 2020. 14 Aside from economic factors, other dimensions such as health and education also play a key role in determining overall well-being. Educational attainment in Nigeria has improved over the past decade, but large regional and gender-based disparities remain.6 The North West has lower educational indicators than the rest of the country (except for the North East, which has comparably low indicators). In 2013, the North West region had a youth literacy rate of 58%, an upper secondary completion rate of 24%, and a primary completion rate of 48%.7 In addition, 45% of children ages 9 to 12 had never been to school. In the North West, women are doubly disadvantaged, having distinctly lower educational access and outcomes (Rustad & Ostby, 2017). The subnational human development index (SHDI) of the northwestern states of Kaduna, Kano, and Katsina increased between 2007 and 2017 but remains low. Of three states where the present study was carried out, Kaduna has the highest current SHDI at 0.58 followed by Kano at 0.48 and finally Katsina, which has one of the lowest subnational development indices in Nigeria at 0.38.8 Within this context, the international community has responded through investments in programs to address the related challenges of low human capital, unemployment, poverty and conflict in Northern Nigeria (World Bank 2014; USAID 2015).9 Mafita, funded by the United Kingdom’s Foreign, Commonwealth, and Development Office (FCDO) and implemented by Adam Smith International (ASI) in Northern Nigeria between 2015 and 2020, is one such program. Mafita was designed to build on an improve the quality and relevance of existing training models provided by both the public and private sectors and was implemented in collaboration and coordination with government agencies such as the National Board of Technical Education. It aimed to improve local economic development and the labor-market and social outcomes of marginalized youth in targeted states of the region. 1.2 The Mafita Skills program and impact evaluation objectives Mafita was a five-year, £36 million initiative initially implemented in three Northern Nigerian states: Kaduna, Kano and Katsina. A fourth state, Jigawa, was added later. These states share similar employment and poverty characteristics, and only half of the population in the North West geopolitical zone (which includes the Mafita program area) is empoyed, compared to about two-thirds in other regions.10 The program’s primary goal was to prepare marginalized youth for skilled work or produc- tive self-employment in order to increase youths’ earnings, with the secondary goal of reducing grievances and decreasing the propensity to engage in crime and other anti-social behaviors. The program targeted three categories of youth: (i) Almajiri boys (20%); (ii) adolescent girls, including 6 For specific statistics, see the World Inequality Database on Education (WIDE) presentation of DHS 2013 data for Nigeria. Accessed on January 1, 2020 at https://www.education-inequalities.org/countries/nigeria 7 Of the three states in the North West where this study was carried out, Kano has the highest literacy rate followed by Kaduna and Katsina. 8 See the Subnational Human Development Index (4.0) hosted by Radboud University. Accessed on January 1, 2020 at https://globaldatalab.org/shdi/shdi/NGA/ 9 These efforts are in alignment with Nigeria’s Federal Government’s National Economic Growth and Recovery Plan for 2017-2020. This recognizes the need to improve technical and vocational education and training, especially for young women, to address the joint challenges of low human capital, unemployment, and conflict. See Federal Ministry of Education (2019). Education for Change: A Ministerial Strategic Plan 2018 - 2022. Abuja, Nigeria: Nigeria Federal Ministry of Education. pp. 22-27. 10 See the Labor Force Statistics Unemployment and Underemployment Report: Q2 2020. Na- tional Bureau of Statistics, Government of Nigeria. Accessed on January 29, 2021 at https://www.nigerianstat.gov.ng/pdfuploads/Q22 020U nemploymentR eport.pdf. 15 those in Islamic and Qu’ranic education (20%); and (iii) other vulnerable youth, including orphans, early school leavers, and persons with disabilities (60%). Recruitment was based on two primary selection criteria: (i) being in the age bracket 15 to 24 years old, and (ii) living on less than N225.76 (about $0.65 in today’s exchange rate) per day. Youth were recruited through community outreach workers with the support of caregivers. These caregivers were youths’ parents, relatives, and teachers, including mallams (religious instructors) who teach at Quranic and Islamiyah schools. Caregivers also included guardians who cared for orphans or migrant youth, especially those who came from rural areas. During the application process, youth indicated their two top preferences for trade skills they wanted to acquire. Throughout the program, it was expected that youth would be paid a stipend to cover daily transportation and food costs and to offset potential lost income. Mafita was implemented in two stages: a two-year “pilot” phase and a scale-up phase for the remaining three years. The study described in this report focuses on the pilot phase. Figure 1: Mafita pilot states Mafita included a set of interventions intended to diminish three key barriers faced by its target groups. These are: (i) lack of marketable skills and insufficient foundational skills (e.g., numeracy and literacy) to enter into existing formal skills development systems; (ii) high economic marginalization linked to the lack of education and opportunities; and (iii) social marginalization and stigma. The program sought to diminish these barriers through increasing participants’ access to skills provision through both formal and informal delivery systems; improving quality of formal and informal skills delivery systems and their relevance to the local labor market; and improving labor market linkages for participants. 16 These interventions were delivered through two programs. One program was an upgraded and formalized version of traditional apprenticeship schemes while the other was a vocational training program delivered through Community Skills Development Centers (COSDECs). The two programs differed in their skills delivery model and locations of implementation. Both programs, however, provided both foundational skills instruction and trade-specific skills training in addition to complementary elements such as access- to-finance interventions or entrepreneurship support. As such, both programs were designed to develop necessary skills and to provide complementary tools and competencies to be able to translate these into economic success. Additionally, both programs were based on prevailing skills development models in the areas in which they were implemented. The Apprenticeship program offered diverse training options to Mafita participants by linking them to master craft persons (MCPs) with whom they completed a one-year appren- ticeship. Mafita recognized that the traditional apprenticeship system could be very attractive as a source of skills to marginalized youth because of its flexibility, affordability, and its connection to future employment. Moreover, entry requirements and qualifications are generally low, a factor which makes it accessible to the types of youth served by Mafita, many of whom have no formal education. Mafita conducted an assessment of the MCPs’ working infrastructure, equipment, and tools, and conducted trainings for trade associations to which the MCPs belonged. MCPs were grouped in trade-specific clusters to expose youth apprentices to a broader range of experience and expertise. In the pilot phase, the Apprenticeship program was delivered in Kano State, in and around Kano city. The Apprenticeship program comprised several months of foundational and technical train- ing and, for some participants, entrepreneurship training. This included six months of foundational training in basic literacy in English (alphabet, vowels, word formation, nouns, grammar), numeracy (num- ber identification, addition, subtraction, multiplication, division, basic geometry), basic science (branches of science, matter) and business skills (e.g., project funding), working in teams, and managing relationships, followed by six to nine months of trade-specific on-the-job technical training with MCPs. The trade-specific technical training was informed by the National Vocational Qualifications Framework standards and par- ticipants had to complete assessments under this framework in order to graduate from the program. After completion of the apprenticeship intervention, 900 participants (non-randomly selected by the Mafita pro- gram) were offered additional entrepreneurship training for two months. The program also included access- to-finance interventions, for example support with opening bank accounts and linking participants to loan providers. Additionally, youth received a stipend to support daily costs (e.g., food and transportation) and to offset potential lost income.11 Mafita also offered a home-based version of the Apprenticeship program that primarily targeted IQE girls12 who face specific cultural constraints and limited mobility. This version of the program was delivered through MCPs identified as growth-oriented female entrepreneurs who provided training in their homes.13 11 The qualitative evaluation suggests that stipends to Mafita trainees in the program’s first cohort, which this study focuses on, were not always paid on time or at all. Such program delivery challenges were resolved for subsequent cohorts. 12 Female students enrolled in Islamic Quranic Education 13 This intervention operated much like the standard apprenticeship intervention in terms of objectives and methods. Fur- thermore, as with the standard apprenticeship intervention, Mafita provided the growth-oriented female entrepreneurs with training to upgrade their skills in order to increase the quality of instruction. 17 The COSDEC program was designed as a “one-stop shop” skills development model com- prised of quality, market demand-driven training combined with information on access to finance, business incubation, guidance, and counseling. This program aimed to move participants from marginalization to self-employment, wage employment, or further education. COSDECs offered train- ing in vocational and technical skills and in entrepreneurship, marketing, business, and organizational and life skills, in addition to small and medium enterprise incubation and support services. The program also included access-to-finance interventions. Youth received a stipend to support daily costs and offset potential lost income. In Mafita’s pilot phase, this program was delivered through six centers in Kaduna and Katsina states in rural, peri-urban, and urban areas. The COSDEC training included several months of foundational and technical training and, for some participants, entrepreneurship training. The program comprised three months of foun- dational skills training in literacy, numeracy, basic science, and soft skills, similar to the Apprenticeship program. This was followed by six to nine months of trade-specific technical training in a classroom setting where students had access to full workshops. COSDEC skills training occurred under the National Voca- tional Qualifications Framework standards and program participants had to complete assessments in order to graduate from the program. After completion of the COSDEC intervention, 540 participants (non-randomly selected by the Mafita program) were offered additional entrepreneurship training for two months. Mafita’s pilot phase included two rigorous impact evaluations (IE) to measure the causal impacts of both the Apprenticeship and COSDEC programs. IE objectives and designs for both programs were developed through two workshops including representatives from local governments in the three focal states, ASI, DFID, and DIME.14 The pilot phase of each program was subsequently set up as a randomized controlled trial (RCT) in which a random subset of eligible participants were offered program activities (the so-called treatment group), while the other random subset served as a comparison (or control) group. Impacts of each program are estimated by comparing outcomes of interest between the treatment and control groups based on data collected several months after the completion of Mafita’s pilot phase. More details on the IE design are provided later in this report. 1.3 Evidence from previous empirical studies Skills training programs such as Mafita are one of the most widely used active labor market policies in the world and many rigorous evaluations estimate the effect of these programs on material and non-material outcomes (McKenzie, 2017).15 In general, existing studies find that such programs have had mixed success in improving labor market and non-material outcomes for youth in poor countries. In this section, we focus on randomized evaluations that measure the impact of apprenticeship programs and technical/vocational programs in poor and fragile contexts (Blattman and Ralston, 2015).16 Most of the studies in this literature focus on classroom-based programs, relatively more similar to Mafita’s COSDEC program than to the Apprenticeship program. Hirshleifer et al. 14 The first workshop took place in Abuja in October 2015. The second was held in Kano in August 2016. 15 See also Kluve et al. (2016) for a recent meta-analysis and Betcherman et al. (2004) for a review. 16 Several of these are discussed in Blattman and Ralston’s 2015 review on Generating Employment in Poor and Fragile States, which was commissioned by DIME. 18 (2016) find that Turkey’s vocational training program for the unemployed had positive effects on the proba- bility of formal employment, household durable assets, and subjective well-being one year after completion of the training. These positive effects, however, were small compared to the cost of training and the authors conclude that the program is unlikely to pass a cost-benefit test. Card et al. (2011) evaluate the impact of a vocational training program for youth in the Dominican Republic. They find no evidence that the program increased employment but some evidence that it led to higher wages for trainees, conditional on finding employment. A subset of studies has explored the impact of skills training programs on female economic empowerment. Attanasio et al. (2011) find that a skills training program in Colombia had large positive effects on employment in formal-sector jobs, 13 to 15 months after the end of training. For women, training also had a large effect on wages and salary earnings. A cost-benefit analysis suggests that the program had a substantial positive rate of return for women and generated much larger gains than similar programs in de- veloped countries. Similarly, Adoho et al. (2014) find that a vocational training program for girls in Liberia led to large increases in employment after six months. They further find substantial positive impacts on self-confidence as well as ownership and control over monetary resources and savings. The authors estimate that, based on the earnings effect alone (if it remained constant), the program would recoup its costs after three years. Maitra and Mani (2014) find that a vocational training program for poor women in India led to small increases in employment and large increases in earnings, which persisted for at least 18 months after the end of training. The authors estimate that the earnings effect is large enough to recover the program cost within four years. On the other hand, Groh et al. (2016) evaluate the impact of providing work-related soft skills training and a wage subsidy on female youth employment in Jordan. They do not find training in employability skills to have a general impact.17 Existing evidence on apprenticeship programs in developing countries is (even) more lim- ited. Current evidence suggests that programs that combine classroom training with apprenticeships or internships yield positive employment effects in the short term, though only a few programs show sustained epon and Premand (2019) study a positive impacts in the long run (Kluve et al. 2016). For instance, Cr´ ote d’Ivoire which combines on-the-job and theoretical training. They “dual-apprenticeship” program in Cˆ find evidence of improved earnings for participating youth 18-24 months after the end of the program. They also find that youth experience large short-run opportunity costs of participating in the program in the form of lost earnings, highlighting the potential importance of a subsidy or stipend to participants to improve access and reduce dropout. Hardy et al. (2019) examine the short-tun effects of a government-sponsored apprenticeship training program designed to address high levels of youth unemployment in Ghana. They find that apprenticeships shift youth out of wage work and into self-employment, though the loss of wage income is not necessarily offset by increases in self-employment profits in the short run. The study also exploited a randomized match between apprentices and training providers to isolate the effects of trainers’ characteris- tics on the outcomes for apprentices. They find that apprentices who trained with the most experienced or profitable training providers had higher earnings and this increase more than offset the program’s average 17 Groh et al. (2016) find that the wage subsidy led to a large increase in female employment (mostly informal) in the short-run. However, the average effect was much smaller and no longer statistically significant four months after the subsidy ended. 19 18 negative effect on earnings. Cho et al. (2013) find that an apprenticeship program for youth in Malawi improved the subjective well-being of male participants but had no discernible effect on employment and income for men or women four months after training was completed. The authors acknowledge, however, that the failure of the program may have been partially due to a high dropout rate that was exac- erbated by an administrative error in the enrollment process. Only one study that we are aware of directly compares vocational training to firm-provided apprenticeships in small and medium enterprises. Alfonsi et al. (2020) compare these two skills development models for disadvantaged youth in Uganda. Two to three years after the intervention, par- ticipants in both types of training accumulated a similar level of trade-specific skills. However, an index of labor market outcomes (including employment, labor supply, and earnings) increased more quickly for youth in apprenticeships, but in subsequent years this declined to control group levels. In contrast, youth who participated in vocational training did not show the same level of initial improvement, but the index climbed steadily over time. Alfonsi et al. conclude that this likely stems from the fact that skills acquired through vocational training, in established and reputable vocational training institutes, are certified, which helped youth that became unemployed find new work more quickly. Overall, the existing literature suggests mixed results on economic outcomes for classroom- based vocational training programs and encouraging results for apprenticeships. In the case of vocational training, some studies show modest effects and others show ambiguous effects on participants’ labor market outcomes.19 The evidence on apprenticeship programs is positive but not yet sufficient to offer conclusive evidence of effectiveness. Existing empirical studies suggest potential female empowerment effects of employment programs. Alfonsi et al. (2017) find that providing adolescent girls with a mix of vocational training and life skills leads to substantial advances in economic empowerment four years post-intervention, driven mainly by additional engagement in self-employment activities.20 Likewise, Field et al. (2019) find that depositing salaries from India’s National Rural Employment Guarantee Act (NREGA) into female-owned bank accounts shifted intra-household bargaining power and increased female labor supply, especially for women who had not worked before. There is some evidence that training/employment programs, when combined with comple- 18 In a related paper, Hardy and McCasland (2015) study the effect of an apprenticeship program on outcomes at the firm level in Ghana. They find that firms that were randomly assigned apprentices significantly increased their number of employees relative to firms without apprentices. Though they do not estimate the effect of the program on apprentices, their results suggest that apprenticeships can generate employment by reducing search costs and allowing firms to identify high-quality workers. 19 Recent reviews have speculated on reasons for these mixed results. In his critical review of active labor market policies, including vocational training programs, McKenzie (2017) argues that one reason for this is that labor markets, especially in urban settings, appear to have fewer market failures than it is often assumed, leaving little role for many traditional active labor market policies. Similarly, Blattman and Ralston (2015) point to a wide range of potential reasons for the mixed results of vocational training programs including high dropout rates, training approaches, targeting, skills mismatch, and underlying assumptions and conclude that the main factors may simply be that skills training programs are in general difficult to get right or too expansive to pass a cost-benefit test. 20 The paper also evaluates changes in girls’ expectations for ages at marriage, childbearing, and fertility, and aspirations for their own children. It finds that the program affected most dimensions in the short term though these effects tended to fade over time, except for girls’ views on ideal ages of marriage and fertility. 20 mentary interventions, can reduce support for and participation in violence. Lyall, Zhou, and Imai (2020) study the effects of vocational training and cash transfers on combatant support among at-risk youth in Afghanistan. They find modest impacts of training on economic livelihoods which, when combined with cash, increases support for the Afghan government in the short run. They conclude that this shift in attitudes occurs through the signal of government resolve and competence provided by the cash transfers rather than as a result of improved economic livelihoods per se. Blattman and Annan (2016) find that a combination of agricultural skills training and capital grants for Liberian ex-combatants decreased their interest in mercenary work but had no effect on their sociopolitical attitudes. 2 Evaluation Design 2.1 Evaluation questions The Mafita IEs were designed to test the effectiveness of the Apprenticeship and COSDEC skills development models and to build on the existing empirical evidence summarized above. This study builds on the limited evidence of alternative skills training models that specifically target highly marginalized youth in fragile contexts and contributes to our understanding of the material and non-material impacts of such programs. Evidence on the latter dimension is especially limited. The IEs address the following primary research questions for both the Mafita Apprentice- ship and COSDEC programs: 1. What is the impact of the Mafita programs on the employment, income, and entrepreneurship of marginalized youth as well as on their subjective well-being and religious and sociopolitical attitudes and behaviors? 2. Do programs impacts vary according to the characteristics of participants, specifically their gender or whether they are part of a specific population group? To answer the first question, both the Apprenticeship and COSDEC programs were conducted as separate RCTs, which allowed for each program to have a randomized group that received program activities and another randomized group that did not receive program activities for the duration of the study. To answer the second question, which addresses possible distributional impacts of the two Mafita programs, subgroup analyses was conducted on a set of key pre-specified dimensions, including gender and categories of targeted youth. 2.2 Theory of Change The design of the Mafita program was informed by a detailed study that identified key market failures underlying youth employment and assessed the efficacy and gaps of the current skills delivery system in Northern Nigeria. This analysis revealed that access to economic opportunities by youth was primarily constrained by the interplay of at least three main factors21 : (i) local economies in the three focal states have limited scope for creating new high value-added enterprises and micro-enterprises 21 These factors are discussed in detail in Mafita Inception Report and the Concept Note on “Economic Opportunities Improvement Strategy”. 21 with potential to generate more stable and higher income jobs for millions of youth, thereby making it difficult to overcome youth unemployment challenges; (ii) a deficient skills delivery system resulting from an underfunded and poor quality public education system which tends to produce graduates with skills and competencies that are not aligned with local labor market needs, thereby creating a situation of sectoral and spatial mismatches between skills supply and demand; and (iii) cultural and religious barriers, which are often compounded by a largely patriarchal system that has tended to exclude many from participating in local economic opportunities. Mafita ultimately decided to focus on the low level of relevant skills and competencies among youth. This issue was perceived as a key constraint to accessing better employment opportunities which would need to be resolved to overcome youth unemployment and underemployment.22 Mafita’s design therefore focused on addressing gaps in the current skills delivery system across the focal states and their responsiveness to local labor market needs. At the same time, the program explicitly sought to address cultural and religious barriers which tend to exclude some population groups from the labor market through targeting different categories of marginalized, socially and economically vulnerable youth and through pro- viding an alternative, home-based apprenticeship model for some female trainees. Still, the limited job and entrepreneurship opportunities provided by the local economy may pose a significant, or even binding, con- straint to improving labor market outcomes through skills training. Figure 2 below outlines the theory of change for both the Apprenticeship and COSDEC programs, focusing on skills development. It displays key program inputs or interventions as well as expected outputs and outcomes. Figure 2: Theory of Change The two programs were delivered differently but were expected to work through similar channels. The main activities undertaken in the apprenticeship program were workplace-based training 22 Blattman and Ralston (2015) note: “In low-income countries, unemployment is seldom the main problem. Few adults in developing countries are unemployed in the sense of having zero hours of work...It is more accurate to call them ”underemployed“, in that people would like to work more hours for the same (or higher) wage they earn at their current activities...Due to some set of constraints, the returns to more labor in their existing activities tend to be low or even zero.” 22 and work experience with the MCPs each apprentice was assigned to. The main activity in the COSDEC intervention was classroom instruction and technical skills training. Both programs also included founda- tional skills training lasting 3-6 months. The immediate desired “output” (or short-term outcome) of both interventions was to improve trainees’ foundational skills, job-specific technical skills, and soft or social skills such as self-management, teamwork, communication, etc. Furthermore, by introducing participants to other apprentices and MCPs (in the case of the Apprenticeship program), or other students and skills trainers (in the case of the COSDEC program), expanded social networks can be considered an additional program “output”/short-term outcome.23 The expected direct outcome of this skills training was an improvement in the labor mar- ket outcomes of the trainees. The hypothesis is that participants’ new technical and soft skills would facilitate their access to jobs that they were previously not fully qualified for or otherwise unable to access, or help to start their own businesses. Similarly, foundational skills training would provide basic skills such as literacy, numeracy, and knowledge of the metric system needed to access and perform in many jobs. Ex- panded social networks would increase participants’ knowledge of job opportunities and referrals and their access to formal and informal business loans and other forms of support. Both programs were therefore expected to lead to increases in employment, entrepreneurship, and income among trainees. The programs were also expected to have an indirect effect on several non-material out- comes. For example, the intervention might also improve subjective well-being by reducing poverty-related stress and conferring increased social status. Likewise, the interventions might lead to a change in religious and sociopolitical attitudes and behaviors through several mechanisms. For instance, if the intervention ex- panded participants’ social networks, this might expose them to differing viewpoints, perhaps by acquainting them with adherents of other religions and members of other ethnic groups. In addition, increased income and employment might reduce grievances against the government and against members of outgroups. By exposing female trainees to male-dominated trades and having male and female participants train alongside one-another, the program might change attitudes towards the role of women and female empowerment. From a theoretical standpoint, both the Apprenticeship and COSDEC programs operated primarily on the labor supply side and aimed to increase the skills and employability of prospec- tive young workers through vocational training. Such programs typically provide some key inputs or services, such as capital or skills training and competencies, to individuals or small enterprises on the basis that (i) the lack of specific inputs are a key binding constraint that hold back some unemployed (or under-employed) individuals from participating in economic opportunities and (ii) these skills can be taught and learned relatively quickly, within the timeframe of the program (McKenzie 2017)24 . In this sense, skills training programs such as Mafita’s Apprenticeship and COSDEC programs act as substitutes for formal schooling in building human capital. McKenzie (2017) suggests that the return (in terms of earnings) from schooling and vocational training interventions in developing countries tend to be similar. 23 It is important to note that, following extensive consultations with DFID and ASI, this study was designed to focus on the impacts of the full Mafita programs. We are therefore not able to rigorously answer the “how” question, i.e., what are the mechanisms through which the outcomes are realized, and whether these channels work in isolation or are complementary. 24 Vocational training programs have been the most common form of active labor market program employed by developing country governments. According to Blattman and Ralston (2015), from 2002 to 2012 the World Bank and its client governments invested nearly USD 9 billion in 93 skills training programs, averaging nearly USD 1 billion per year and USD 100 million per project. 23 The theory of change outlined above emphasizes skills and human capital accumulation as a likely channel through which Mafita may achieve its target outcomes; however, other non- skills channels such as access to credit or capital, networking opportunities, and employability signaling, among others, could also be at work. Indeed, while our primary focus in this study has been on the two labor supply-side programs, i.e., Apprenticeship and COSDEC, which were directly under Mafita’s programmatic control, we note that the program overall also supported economic development efforts by local governments and private actors. If successful, such efforts might also have contributed to a labor market outcomes by not only enhancing the current skills the improvement of Mafita participantsˆ supply/delivery systems across the focal states, but by also helping align the skills of Mafita participants with local labor market needs, i.e., with labor demand. 2.3 Key outcomes of interest Table 1 lists primary and secondary outcome of interest to this study on which program impact was measured. As noted above, the primary goals of the Mafita program were: (i) to improve participants’ material outcomes (i.e., improve their labor market outcomes, increase income, and so on); and (ii) to positively influence non-material outcomes including reducing the propensity to engage in violent crime, violence, and other antisocial behaviors, improve psychological well-being, etc. Primary outcomes are those which are directly linked to the Mafita program’s main objectives while secondary outcomes are those which, though not directly targeted, are plausibly linked to program objectives and may represent important additional program benefits. Both sets of primary and secondary outcomes are derived from the proposed theory of change presented above. 24 Table 1: Main Mafita IE Outcomes Outcome category Definition Primary / Secondary Material outcomes Wage or self-employment in the past month Income Generating Income from wage employment or profit from self-employment in Primary Activities the past month Days worked in wage or self-employment in the past month Asset ownership (e.g., bank account, cell phone, etc.) Assets and Consumption Secondary Weekly and monthly expenditure on essentials and non-essentials Efforts to search for a job or start a business in the past 6 months and 30 days Job Search Behavior 97 Efforts to search for a job in the past week Secondary Time spent on job search Literacy and numeracy Literacy and numeracy skills Secondary Non-material outcomes Desired/actual age of marriage and fertility Domestic violence Autonomy (decision-making regarding study, work, partner, finances Attitudes toward female expenditure, etc.) Primary empowerment Perceptions on women?s role in society Aspirations (goals for future) Confidence in intellectual ability Anti-social behavior (taking of drugs; stealing; physical fights; criminal work) Political and religious violence (use of force for political or religious Participation in anti-social causes; administration of justice) behavior, violence and religious Primary Religious enforcement (willingness to punish acts of blasphemy, alcohol enforcement consumption, or clothing considered inappropriate; enforcement of religious laws) Time spent on religious activities Religiosity Strictness of religious beliefs Secondary Trust and attitudes towards adherents of other religions Use of violence in protesting injustice, resolving problems, or for Attitudes toward political violence Secondary political causes Cantril’s ladder of life-satisfaction Subjective well being 5-item mental health index Secondary & self esteem 7-item self-esteem index including problem-solving, determination in achieving goals, and self-confidence Primary contacts who are employed or live in a different neighborhood Range of professional and Primary contacts that practice a different religion Secondary religious social networks Trust in adherents and leaders of other religions 2.4 Experimental IE study design The Mafita IEs were designed to generate statistically rigorous estimates of the causal effects of the Apprenticeship and COSDEC programs. We define causal effects as the change in measured outcomes that are directly attributable to Mafita, as distinct from changes that would have happened in the absence of the program. The latter includes changes produced by other factors such as overall changes in the general economic environment or other government or donor-funded programs. We use a randomized controlled trial (RCT) research design to estimate causal effects of the programs. RCTs randomly assign potential participants to either a “treatment” group that receives a program or intervention, or a “control” group that does not. The control group serves as a counterfactual of what would have happened to the treatment group in the absence of the program or intervention. When 25 appropriately conducted on a large pool of potential participants prior to the start of the program, random assignment generates statistically identical groups that allow us to isolate program impacts in a credible manner. The key identifying assumption is that the two groups would fare similarly on the key outcomes of interest were it not for the program or intervention. Since the Mafita Apprenticeship and COSDEC programs were offered to different target populations in geographically separate areas (Apprenticeship in Kano; COSDEC in Kaduna and Katsina), we conducted two separate RCTs to evaluate their effects. We discuss the respective evaluation designs in turn. 2.4.1 Apprenticeship IE design Candidates were randomly assigned to the Apprenticeship program from a pre-selected pool of eligible applicants in Kano. This is highlighted in Figure 3. 5,165 eligible applicants were identified based on the program’s eligibility criteria, and these were then randomly assigned to either the treatment/program group (3,066 persons) or the control/comparison group (2,099 persons). The treatment group started the program in April 2017 and the control group was enrolled in the program after final data collection for the IE had been completed in March 2019. Figure 3: Apprenticeship IE Design A key requirement for the Mafita program was representativeness by gender and applicant type. One of the program’s requirements was that at least 40% of the treated individuals would women, at least 20% be Almajiri (Quranic schoolboys), and at least 20% be girls in Islamic and Quranic Education (IQE). To ensure this, randomization was stratified by geographical location, applicant type, and gender. The geographical location variable used for this purpose was the local government area (LGA) of the applicant’s residence. Three applicant types were considered: (i) Almajiri boys, (ii) IQE girls, and (iii) other, including persons with disabilities, orphans, and other vulnerable youth. As shown in Appendix A.1, the randomization was successful in creating balanced treatment and control groups. 26 2.4.2 COSDECs IE design Candidates were randomly assigned to the COSDEC program from a pre-selected pool of el- igible applicants in the two states where COSDEC intervention took place. This is highlighted in Figure 4. There were six COSDECs training centers operating in Mafita’s pilot phase, three in Katsina and three in Kaduna. For each COSDEC center, Mafita pre-screened approximately twice as many eligible applicants as the targeted number of enrollees, for a total of 1824 individuals. Half (900) of the pre-screened applicants were randomly assigned to the treatment/COSDEC group and began the program in April 2017, and the remainder (924) were assigned to the control/comparison group whose enrollment was delayed until after final data collection for the IE was completed. Figure 4: COSDEC IE Design Randomization was stratified on state, COSDEC center, and applicants’ preferred trade. When applying to the program, individuals were asked to specify a desired trade. According to Mafita program requirements, a maximum of 30 persons in each COSDEC center could be trained in the same trade. To ensure balance, random assignment was stratified by (i) state (Kaduna and Katsina), (ii) center (the six COSDEC centers in Daura, Funtua, Katsina, Kagoro, Mando and Sabon Gari), and (iii) preferred trade (brick-laying, carpentry and joinery, electrical installation, fashion design, hospitality, office management, welding and fabrication). As shown in Appendix A.1, the randomization was successful in creating balanced treatment and control groups. 27 2.5 Data collection 2.5.1 Quantitative data collection The main source of data for the Mafita IE is an endline survey conducted between December 2018 and March 2019, six-to-nine months after program completion.25 This comprised a youth questionnaire and a caregiver questionnaire that gathered information on key outcomes including training history, labor market outcomes, socioeconomic well-being, psychological well-being, attitudes and participa- tion in crime and violence, women’s empowerment, and social networks, among others. The endline survey targeted 5,165 participants in the Apprenticeship study, of which 86% were successfully surveyed (4,460), and 1,824 participants in the COSDEC study, of which 94% were successfully surveyed (1,708). The study also covered 3,070 caregivers of participants in the apprenticeship study and 1019 caregivers of participants in the COSDEC study.26 In addition to the endline survey data, the analysis presented in this report is also based on Mafita administrative enrollment data and on a baseline survey collected by the program with technical assistance from the World Bank DIME team. Figure 5: Training for quantitative data collection Data collection was preceded by several rounds of pilot testing and field testing of the survey instruments. Data collection instruments were tested and piloted to confirm that they captured 25 The data collection lasted for a period of 3 months. Therefore, for those surveyed in December 2018, this was six months after the conclusion of the intervention while for those surveyed in March, it was 9 months. It is important to note that six months is a relatively short post-intervention period since it usually takes time for employment effects of such programs to fully materialize. 26 Caregivers were identified through the youth endline survey, which included a caregiver listing exercise to collect information about youths’ caregivers such as demographics and contact information. The information collected from the youth survey was used to generate a list of unique caregivers who were then interviewed. 28 the required information, were clear to all participants, and were sensitive to the Northern Nigerian con- text. The youth questionnaire underwent several months of final development and refinement and included input from Mafita program staff and from multiple rounds of pilots. Pilots of sensitive questions provided important feedback on how to ask these questions using methods that provide maximum security to respon- dents, including audio and self-administered methods. Survey tools and other research procedures used to collect data underwent ethical review and were approved by the Nigerian Health Research Ethics Committee. Data collection was carried out by a professional survey firm recruited by the World Bank. The survey firm was responsible for translation and adaptation of the survey instruments, recruiting and training enumerators, in-field quality control, and data collection logistics. The survey firm mobilized a large team to cover participants in both interventions across multiple geographic areas. This included 98 field personnel comprising 3 state team leaders, 13 super- visors, 75 enumerators, and 7 independent back-checkers. Field teams were supported by back-office staff, including IT specialists to ensure efficient functioning of equipment and data transfer processes. This entire team participated in a 9-day training in Kano facilitated by the World Bank DIME team. At the end of the training, an evaluation test was conducted among the enumerators. More than the required number of enumerators participated in the training and the best were retained for fieldwork. During fieldwork, a number of measures were taken to ensure high quality data collection. First, surveys were conducted using electronic tablets enabling enumerators to collect and transmit data from the field to a cloud-based server. Electronic data collection allowed the evaluation team to conduct quality control measures in real-time, checking data on the same day it was collected. Second, concerted efforts, including multiple interview attempts, were made to maximize coverage of the target sample.27 Third, DIME field coordinators attended selected interviews to check for any issues early in the data collection process and followed up with prompt feedback to field teams on any errors found. Finally, we further ensured data quality with back-checks on a randomly selected sample of approximately 5% of all completed surveys. Back- checks consisted of a 15-20-minute interview comprising selected time-invariant questions, conducted by an independent team that had not been involved in the initial interview. 2.5.2 Qualitative data collection A separate qualitative study was undertaken to shed light on some of the non-quantifiable and narrative aspects of the program and to learn about the mechanisms through which the quan- titative outcomes may have occurred. Fieldwork for the qualitative study was conducted in October 2019 and included 60 interviews (52% female) and 27 focus group discussions (44% female). The sample of study participants was divided almost equally between the Apprenticeship and the COSDEC programs. The study incorporated various data collection techniques, targeted different stakeholders, and covered five main discussion topics. Data was collected through semi-structured focus group 27 To minimize attrition, the endline survey involved a two-phased tracking approach. In the first phase, we attempted to interview every study participant at his or her last known location. If the individual had moved, we attempted to obtain information about his or her current location from neighbors, relatives, friends, former employers, and (if applicable) religious teachers. We also attempted to contact individuals located in the vicinity of the original location immediately for their interview. Individuals that had moved farther away were tracked and interviewed in a second phase. 29 discussions (FGDs) and interviews combined with informal conversations and unstructured observations to understand participants’ experience in the Mafita program, the perceived impacts of that experience, and the possible explanations for what worked and what did not work well. Participants were asked questions concerning their daily lives and routines (livelihoods, housing situations, and social lives), their experiences with the Mafita program, and their perceptions and experiences with political and religious violence. FGDs were conducted separately for women and men to capture a gendered perspective on the program. To triangulate findings from the youth interviews, the study also conducted interviews and FGDs with trainers, MCPs, caregivers, and Mafita staff members. For comparative and contextual purposes, the research staff conducted interviews and FGDs with youth who were not participants in the Mafita program. Figure 6 illustrates a female FGD and an interview with a male study participant. The design of the qualitative study was informed by previous qualitative fieldwork in 2018. Figure 6: Qualitative fieldwork Detailed qualitative findings are summarized and analyzed in a separate, standalone report. The present report draws on these findings as appropriate to further illuminate quantitative findings. 2.6 Timeline The Mafita IE includes three phases: design, implementation, and analysis. The design phase began with an IE workshop which took place in Abuja, Nigeria, in October 2015 and included government representatives from target states, members of the Mafita program team, DFID Nigeria, and the DIME team. The initial concepts from this workshop were further developed and the IE design was finalized in August 2016. Implementation began with baseline data collection and random assignment of eligible applicants to both the Apprenticeship and COSDEC programs, which took place between October and December 2016. The first cohort then entered the program in April 2017 and continued through March 2018. Final quan- titative data collection was carried out between December 2018 and March 2019, six-to-nine months after program completion (qualitative data collection took place some months later, in September and October 2019). Analysis began following the completion of data cleaning, and results were first disseminated on March 12, 2020 during Mafita’s Project Closing and Lessons Learned event held in Abuja. This report was subsequently drafted including several rounds of revision to incorporate useful feedback from DFID, ASI, and the World Bank. The IE timeline is summarized in Figure 7. 30 Figure 7: Mafita IE Timeline 3 Analytical Framework In this section, we outline our strategy to analyze outcome data on participants and non- participants in Mafita’s Apprenticeship and COSDEC programs. While each program is the sub- ject of a separate RCT, these are based on parallel designs and so we use the same estimation strategy to analyze impacts on key outcomes of interests. We are primarily concerned with estimating average effects of the two Mafita programs, though we also explore possible heterogeneous effects. To adhere to the highest research transparency and integrity standards, detailed pre- analysis plans were developed prior to accessing survey data. These include, among other things, a precise definition of each outcome indicator and the grouping of these indicators into “families” of out- comes as well as specifications of statistical procedures we planned to use to analyze the Mafita programs’ impacts. The respective pre-analysis plans for the Apprenticeship and COSDEC programs were registered in advance with the American Economic Association (AEA)28 and with Experiments in Governance and Politics (EGAP).29 The remainder of this section provides a summary of the content covered in more detail in the pre-analysis plans. 3.1 Estimating the main effects Since applicants to the Mafita interventions were randomly assigned to the treatment and control groups, any difference in outcomes between the groups can be attributed to the effect of the intervention. We separately estimate the intention-to-treat (ITT) effects of the two programs (Apprenticeship and COSDEC) on the outcomes of interest using the following linear regression: Yi = β0 + β1 ∗ Ti + γ ∗ Xi + i (1) Where Yi is the outcome of interest for individual i, Ti is an indicator for assignment to the treatment group, and Xi is a vector of pre-program covariates. This vector includes the following variables: age, gender, ethnicity, religion30 , and indicators for randomization strata. 28 Apprenticeship: https://www.socialscienceregistry.org/trials/4500; COSDEC: https://www.socialscienceregistry.org/trials/4637 29 Apprenticeship: http://egap.org/registration-details/5957; COSDEC: http://egap.org/registration-details/6032 30 We control for ethnicity using an indicator variable for being a member of the predominant ethnic group (Hausa), and for religion using an indicator for being Muslim. 31 The analysis accounts for the broad and multi-faceted outcomes of interest which are based on a large number of component indicators. For example, the survey module to measure attitudes to- wards female empowerment includes over 40 questions, including ones about marriage and fertility decisions, decision-making in the household, and attitudes about women’s role in society. To reduce the number of statistical tests and thus the probability of false positives (Type I errors), we aggregated relevant outcomes related to a particular family into a single index, following Kling et al. 2007).31 We further adjusted for multiple hypothesis testing by controlling the False Discovery Rate using the step-up procedure proposed by Benjamini, Krieger and Yekutieli (2006). In our case, we adjust p-values for multiple hypothesis testing by group of outcomes for primary outcomes. We report results on both individual outcomes and the overall index for each family of outcomes. 3.2 Estimating heterogeneous effects We also explore potential heterogeneous treatment effects with respect to gender and youth category. We begin by estimating treatment effects separately for female and male respondents and compar- ing effect sizes. We then restrict the sample to male respondents and compare treatment effects for Almajiri boys and other boys.32 Finally, we restrict the sample to female respondents and compare treatment effects for IQE girls and other girls. For each of these dimensions, we use the following equation to estimate the relevant heterogeneous effects: Yi = β0 + β1 ∗ T reatmenti + β2 ∗ Heterogeneity + β3 ∗ T reatment ∗ Heterogeneityi + γi + i (2) Where Heterogeneityi denotes the different indicators use to denote respondent’s gender and youth category. 3.3 Potential threats to validity and extensions to the primary analysis We identify three potential threats to the validity of our analysis: attrition, outliers, and missing observations or values. Attrition, at 6 and 13 percent, respectively, for the COSDEC and Ap- prenticeship studies, was modest. However, we conduct additional tests to verify whether our results are robust to differential attrition across the treatment and control group.33 These results are reported in the Appendix Section A.2. We deal with outliers by winsorizing unbounded variables at the 99th percentile. We deal with missing data on outcome variables by following Kling et al. (2007) and imputing missing values by setting them equal to the mean of the variable for the relevant treatment group.34 Finally, we deal with missing observations on control variables by setting the missing value to an arbitrary number (e.g., zero) 31 To do this, we first expressed responses in terms of standard deviations from the control group mean. We then summed all standardized responses related to an outcome into an index, switching signs if necessary, to ensure that the positive direction always indicated an increase in outcome. In addition to presenting estimates for the overall index, we also present results for the component indicators to better gauge how various indicators contribute to overall effects within families. 32 The “other” category includes orphans and vulnerable children, disabled persons, and early school leavers. 33 We use a linear probability model that regresses an attrition indicator on the treatment indicator and a set of strata fixed effects. We also report Lee bounds for all estimates, to account for possible effects of unbalanced attrition, and find that we cannot rule out the main results presented below though, overall, the results for the COSDEC program are more robust to this adjustment. 34 outcome values are missing for less than 1% of the sample for both interventions. 32 and including a missing value indicator for each control variable that has missing values. We also estimate alternative specifications. First, we estimate the local average treatment effect by instrumenting the treatment assignment variable with an indicator for participating in and completing the program. Second, we control for available baseline outcomes to increase the precision of our estimator. Third, we report the effect of being assigned to a preferred trade in the Apprenticeship program.35 Finally, we estimate impacts on our primary outcomes for a subsample that excludes individuals who received entrepreneurship training. Results of these analyses are available upon request. They do not affect the conclusions presented in this report. 3.4 Presentation and interpretation of results The IE estimates the effect of the Mafita programs on a set of seven primary and nine secondary outcomes. These results are presented in tables like Table 2 below, where each row reports coefficient esti- mates for each index or outcome. The first column displays the mean outcome of individuals in the control group, which can be interpreted as the average outcome in absence of the program. The second column reports the outcome’s standard deviation in the control group, measuring how much this outcome varies across the individuals in the control group. The third column reports the program’s estimated effect and the standard error, which captures the degree of uncertainty around the estimate and with larger standard errors reflecting a higher degree of uncertainty. In the fourth column, we report the p-value of the estimates, which can be interpreted as the likelihood that the estimates could have been generated by chance if the intervention had no true effect. For primary outcomes, we adjust p-values for multiple hypotheses testing as described by Benjamini, Kruger & Yekuteli (2006). The fifth column gives the sample size used to generate estimates in each row. Table 2: Sample Result Table Control Mean Control St.d Effect P > |t| N (Index) Income generating activities -0.000 0.975 0.106*** 0.002 4460 (0.029) In past 30 days: Wage employment (proportion) 0.211 0.408 0.021* 0.081 4460 (0.013) Self employment (proportion) 0.497 0.500 0.083*** 0.001 4460 (0.016) Wage employment (# of days) 2.985 7.232 0.090 0.299 4460 (0.226) Self-employment (# of days) 7.852 10.061 0.702** 0.039 4460 (0.316) Income from wage employment (Naira) 1636.146 4574.620 79.393 0.299 4460 (142.090) Profit from self-employment (Naira) 3229.717 6732.724 543.277** 0.033 4460 (217.232) 35 Participants in the apprenticeship program were asked to choose two preferred trades that they wanted to be trained in. Since training slots were limited for some trades, some beneficiaries had to be assigned to a non-preferred trade. These beneficiaries were randomly selected by the research team, which allows us to estimate the causal effect of being assigned to a preferred trade. This analysis is particularly valuable since the qualitative study identified frustrations among participants who were not placed in their preferred trade as a potential challenge that could have affected their engagement with Mafita. 33 Additionally, we present some results in forest plots displaying the intervention’s average standardized treatment effect for an index/outcome across the different population groups we consider in the study. As illustrated in 8, this also includes the 95% confidence interval around the estimate. We use forest plots in addition to tables as they provide an efficient way of displaying multiple coefficient estimates at once and allow readers to have a broad overview of results’ consistency across multiple sub-samples.36 Figure 8: Sample Forest Plot *** Full ** Female * Male Almajiri Non Almajiri *** IQE Non IQE -.1 0 .1 .2 .3 .4 Magnitude Stars display level of significance based on p-value adjusted for multiple hypothesis testing Statistical significance level is denoted by adding stars to estimates that are significant at the 90%, 95% and 99% levels, according to the estimated p-value. For primary outcomes, the p-value is adjusted for multiple hypothesis testing. For example, statistical significance at the 95% level implies that there is only a 5% probability that we would observe an estimate of this size in our data if the program’s true effect was zero. These cases are denoted with two stars (“**”). Estimates that are significant at the 90% and 99% levels are denoted with one star (“*”) and three stars (“***”), respectively. 4 Mafita program participation Based on administrative data provided by Mafita, about three-quarters of persons assigned to the first cohort of either the Apprenticeship or the COSDEC program participated in their respective program. Figure 9 displays the rate of participation for each program for both the group of individuals assigned to the treatment/program group and those assigned to the control/comparison group. Across both programs, participation is relatively high for persons assigned to the program (72% for Apprenticeship and 73% for COSDEC) and “contamination”, the number of persons assigned to the control/comparison group that still participated in the program, is low (0.5% in Apprenticeship and 7% in COSDEC). This indicates that the programs were largely implemented according to the study design. 36 Heterogeneous effects as displayed in forest plot should not be considered as a complete statistical analysis, which means that no strict conclusion should be drawn from them. They are only intended to suggest possible heterogeneity across different sample group. In the Appendix, we report a more comprehensive statistical analysis used to detect heterogeneous treatment effect between groups. 34 Figure 9: Mafita program participation for treatment and control groups 80 80 72.01 73.46 60 60 % of participants per group 40 40 20 20 7.18 Control Control 0.46 Apprenticeship COSDEC 0 0 Figure 10 displays the average number of days of training and the proportion of participants that graduated from the Apprenticeship and COSDEC programs for both the treatment and control groups. Persons assigned to the treatment group attended an average of 67 and 149 days of training, respectively, for the Apprenticeship and COSDEC programs.37 Corresponding figures for the control group are 0 days and 13 days.38 As also shown in Figure 10, 64% of participants assigned to the Apprenticeship treatment group who attended at least some training graduated. The corresponding figure for the COSDEC treatment group is 70%. Figure 10: Number of days of training attended and graduation rates # Days attended and % of participants whom graduated Apprenticeship intervention COSDEC intervention 64.10 149 80 150 80 60 70.08 67 Average no. of days training Average no. of days training 60 60 100 40 % graduated % graduated 40 40 50 20 20 20 13 7.06 0 0.23 0 0 0 0 Control Control Treatment Treatment 37 If we include only members of the treatment group that attended at least one day of training, these averages increase to 87 and 202, respectively, for the Apprenticeship and COSDEC programs. 38 If we consider only those members of the control group who attended at least one day of training, the average number of days rises to 55 and 177, respectively, for Apprenticeship and COSDEC 35 In sum, we conclude that take-up and participation in the Mafita programs by youth as- signed to either the Apprenticeship or COSDEC programs was relatively high , while “con- tamination” of the control group was relatively low. This provides a basis for the quantitative IE analysis, as it suggests that the study design was broadly adhered to in the implementation of the program. Consistent with this conclusion, the qualitative study shows that Mafita participants reported high levels of satisfaction with the training, which they found to be useful. 5 Results In this section, we present evidence of the effects of participation in Mafita’s Apprenticeship and COSDEC programs on two broad outcome families: (i) labor market and economic welfare outcomes; and (ii) non-material outcomes. Each family includes a set of pre-specified outcomes and outcome variable indicators, as described in more detail in our pre-analysis plans39 . As noted earlier, we measure impacts of both programs on the same set of outcomes. However, for clarity we present the results separately. For each program, we present the main impacts on outcomes in the two families and explore heterogeneity by gender and youth category, where applicable. We present so called “intention-to-treat” (ITT) effects, meaning that the results we present are average impacts across all youth assigned to participate in the program, net of those assigned to the control group, regardless of whether any individual actually participated in the program or not. This provides a more accurate estimate of actual program impacts across the target population as, in any training program, some of those that sign up will not actually participate or complete training. We then discuss these findings in more detail, complementing them with evidence from the qualitative evaluation. 5.1 Mafita Apprenticeship Program effects on economic outcomes This section reports the main findings on the effects of the Apprenticeship program on labor market and economic welfare outcomes. The specific outcomes investigated include income-generating activities (i.e., employment and earnings), assets and consumption, job-search behavior, and literacy and 40 numeracy skills. Figure 11 summarizes the effects of the Apprenticeship program on this set of outcomes which shows positive and significant effects (at the 1% level) for income generating activities, job-search behavior, and assets and consumption. No effects are observed for numeracy and literacy skills accumulation. 39 Crost, Benjamin et al. 2019. “Community-based Skills Training, Employment and Social Transformation: Evidence from a Field Experiment in Northern Nigeria.” AEA RCT Registry. September 06. https://doi.org/10.1257/rct.4637-1.0. Crost, Benjamin et al. 2019. “A Way Out? Apprenticeship Training, Employment and Social Transformation in Northern Nigeria.” AEA RCT Registry. August 02. https://doi.org/10.1257/rct.4500-1.0. 40 As noted in the previous section, we consider income-generating activities a “primary outcome” as this is an important final outcome for Mafita, whereas the other outcomes are considered “secondary” as these are either channels through which participants may have had more success in income generating activities (job-search behavior; literacy and numeracy) or addi- tional benefits (assets and consumption). However, this classification is primarily for analytical purposes and does not contain any value judgement over the relative merits or importance of the various outcomes. 36 Figure 11: Summary effects of the Apprenticeship program on economic welfare outcomes *** Income generating activities *** Job search behavior *** Assets and expenditures Literacy and numeracy -.05 0 .05 .1 .15 .2 Magnitude Stars display level of significance based on p-value, except for primary hypothesis as precised in the PAP Overall, the Mafita Apprenticeship program had strong and positive effects on participants’ employment and productivity, job search behavior, and economic welfare. Relative to the con- trol group, Mafita apprentices were 17% more likely to be employed in self- or family-owned businesses (henceforth, self-employed), and 10% more likely to be in wage employment; raised their profits from self- employment by 17% (though income from wage-employment didn’t increase); and increased the time they spent on self-employment activities by 9%. They are also 12% more likely to have attempted to start a business in the past 6 months; and 27% more likely to have looked for a job in the past 30 days. Former ap- prentices also owned more assets and had higher consumption expenditure than did members of the control group. In particular, they were 5% more likely to own bank accounts and spent 8% more on food and 59% more on other expenses during the 7 or 30 days preceding the survey, respectively. We do not find evidence of impact on literacy and numeracy. We discuss these results in more detail below. 5.1.1 Effects of the Apprenticeship program on income generating activities For income generating activities, we analyze several variables that were pre-specified in the PAP, including self/wage employment in the past 30 days, the number of days spent on these activities, income from wage employment, and profit from self-employment over this period. Table 3 below reports the effects of the Apprenticeship program on each of these individual outcome measures and on the overall standardized index that aggregates them. 37 Table 3: Apprenticeship effects on income generating activities Control Mean Control St.d Effect P > |t| N (Index) Income generating activities -0.000 0.975 0.106*** 0.002 4460 (0.029) In past 30 days: Wage employment (proportion) 0.211 0.408 0.021* 0.081 4460 (0.013) Self employment (proportion) 0.497 0.500 0.083*** 0.001 4460 (0.016) Wage employment (# of days) 2.985 7.232 0.090 0.299 4460 (0.226) Self-employment (# of days) 7.852 10.061 0.702** 0.039 4460 (0.316) Income from wage employment (Naira) 1636.146 4574.620 79.393 0.299 4460 (142.090) Profit from self-employment (Naira) 3229.717 6732.724 543.277** 0.033 4460 (217.232) The Apprenticeship program had strong and positive effects on the overall index of income- generating activities. As highlighted in Table 3, the treatment group of Mafita apprentices had on average 0.11 standard deviations higher values across all six outcomes compared to individuals in the control group. This is significant at the 1% level. The strong positive effects are driven by most indicators in this index, except for (i) number of days working in wage employment and (ii) income from wage employment, where we do not detect evidence of positive impacts. We find large positive and statistically significant impacts of the Apprenticeship program on self-employment and a modest increase in profits from self-employment. Relative to the control group, Mafita apprentices increased employment in self- or family-owned businesses by 17% (8 percentage points) and profit from self-employment by 17%, equivalent to 543 Naira. These are statistically significant at the 1% and 5% levels, respectively. In addition, apprentices in self- or family-owned businesses also worked more regularly, increasing the time they spend working in these activities by 9% (equal to 0.7 additional days over a 30-day period). Taken together, these are encouraging results. This is especially true considering that they reflect the economic situation of Mafita apprentices six to nine months after program completion in a difficult labor market context. The effect of apprenticeships on wage employment is modest (2 percentage-point increase), and we find no evidence of a general increase in income from wage employment or time spent on wage employment. Results from the control group provide perspective on the generally low levels of wage employment, in absolute terms and relative to self-employment: in the control group, almost half engage in self-employment while only one fifth engage in wage employment. Additionally, income from wage employment among the control group is 1,636 Naira for the past 30 days, whereas profits over this period is double this, at 3,230 Naira. These differences in self and wage employment are especially stark for women (the control group rates are 45% and 7%, respectively). Overall, these results suggest that the returns to self-employment are generally higher for the type of youth targeted by Mafita and/or that there are addi- tional constraints to wage employment for this population. 38 Effects on income-generating activities are largest for IQE girls. This is shown in Figure 12 and is sup- ported by Table A11 which show that impacts for IQE girls are statistically different from those of non-IQE girls. Figure 12: Apprenticeship effects on income generating activities across youth categories *** Full ** Female * Male Almajiri Non Almajiri *** IQE Non IQE -.1 0 .1 .2 .3 .4 Magnitude Stars display level of significance based on p-value adjusted for multiple hypothesis testing The striking result for IQE girls is driven by strong relative increases in self- and wage- employment, and in income earned from these activities. Figures 13 show employment results for the full sample, for male and female participants separately, and for IQE girls. This highlights strong and practically meaningful impacts on self-employment for both male and female participants (for female participants, self-employment in the past 30 days increases by 25% relative to the control group; the corre- sponding increase for male participants is 11%). For wage employment, we see that results are statistically significant for the full sample but not when we disaggregate by gender. However, the relative increase in self-employment is strikingly large for IQE girls (about 41% increase over comparison group) and even their rates of wage employment doubles from 5% to 10%. Estimated increases in income for IQE girls are also large and statistically significant, though these are relative to low values in the control group. As shown in Figure (a) and Figure (b), for IQE girls, profits from self-employment nearly double from the control group average of 869 Naira and we see a similarly large proportional (112%) increase in income from wage employment, though from a low starting point (187 Naira). IQE girls are likely to face several constraints to entering wage or self-employment, as reflected in the low levels observed in the control group, and these results show that Mafita was particularly successful in its goal of improving the economic prospects of this highly vulnerable group. 39 Figure 13: Apprenticeship effects on self and wage employment ((a)) Self employment ((b)) Wage employment 5.1.2 Effects of the Apprenticeship program on asset ownership and consumption expenditure For this family of outcomes, we analyze two sets of variables. The first set relates to household ownership of financial and durable assets, including owning a bank account and things like a cell phone, fan, television, and electric iron, etc. The second relates to respondents’ basic expenditures on food, transport, and airtime (over the seven days preceding the survey) and on medical expenses, clothes, leisure, and other expenses (over the 30 days preceding the survey). Table 4 below reports the effects of the apprenticeship intervention on outcomes in each of these sets and on the overall standardized index that aggregates individual outcomes. The apprenticeship program had strong and positive effects on the overall measure of asset ownership and consumption. As shown in Table 4, the overall index is 0.13 standard deviations higher for the treatment group of Mafita apprentices. This difference is significant at the 1% level. Table 4 also displays the effects of the apprenticeship program on specific outcome indicators. Compared to the households of their peers in the control group, Mafita apprentices’ households were 4 percentage point more likely to have a bank account, an increase of 5% over the control group. However, we did not find evidence of impact on ownership of other assets such as generators or satellite TVs. Mafita apprentices also spent 8% more on food (over the seven days preceding the survey) and 59% more on “other” expenses (over the 30 days preceding the survey).41 Total expenditures in the past week increase by 107 Naira, which represents an 8% increase above the control group average (1,348 Naira). 41 “Other” includes expenditure on items that do not fall into the following categories: food, transport, airtime, medical expenses, clothes, soap/detergent/cosmetics, or leisure. 40 Table 4: Apprenticeship effects on asset pwnership and consumption Control Mean Control St.d Effect P > |t| N (Index) Assets and consumption 0.000 0.963 0.130*** 0.000 4460 (0.030) Proportion of respondents whose household has at least one: Electric iron 0.671 0.470 0.031 0.125 4460 (0.014) Fan 0.725 0.447 0.019 0.290 4460 (0.014) Television 0.572 0.495 0.012 0.577 4460 (0.015) Refrigerator 0.241 0.427 0.003 0.862 4460 (0.014) Generating set 0.200 0.399 0.007 0.717 4460 (0.013) Satellite or cable TV 0.246 0.430 0.015 0.487 4460 (0.014) Electricity 0.930 0.255 0.003 0.777 4460 (0.008) Bank account 0.814 0.388 0.041*** 0.003 4460 (0.011) Cell phone 0.993 0.083 0.004 0.167 4460 (0.002) Respondents expenditure (in Naira), in past 7 days, on: Food 702.420 1004.726 52.891* 0.059 4460 (29.197) Transport 405.864 577.737 39.508 0.125 4460 (17.889) Airtime 239.890 365.358 14.867 0.258 4460 (11.331) Medical expenses 647.966 1338.170 106.373 0.356 4460 (43.918) Clothes 1644.387 2664.090 176.967 0.290 4460 (87.562) Soap/Detergent/Cosmetics 475.315 674.301 89.387 0.158 4460 (22.239) Leisure 299.957 805.561 55.432 0.328 4460 (25.978) Other 61.200 409.132 35.996*** 0.003 4460 (14.476) Total (last 7 days) 1348.175 1524.717 107.265* 0.077 4460 (43.916) As shown in Figure 14, we find statistically strong impacts on asset holdings and consump- tion for both male and female participants. The overall index increases by 0.12 and 0.14 standard deviations for female and male participants, respectively. In fact, we observe statistically significant impacts for both Almajiri and non-Almajiri boys, and for IQE girls and non-IQE girls. There are no statistically significant differences across these groups, however. 41 Figure 14: Apprenticeship effects on asset ownership and consumption across youth categories *** Full ** Female *** Male * Almajiri *** Non Almajiri ** IQE * Non IQE 0 .1 .2 .3 Magnitude Stars display level of significance based on p-value The increase in consumption expenditure is corroborated by evidence from the qualitative evaluation which finds that increases in income, though generally small, generated a certain level of financial independence which was valued by participants. Furthermore, the qualitative evaluation suggests that female participants benefited relatively more, either because they were more likely to lack personal sources of income prior to the program (as corroborated by the quantitative findings) or because they tended to participate in more lucrative trades, such as fashion design. Statistical analysis of the quantitative data, however, does not confirm the presence of any differential effect by gender on household asset ownership and consumption expenditure. 5.1.3 Effects of the Apprenticeship program on job search and entrepreneurship In addition to knowing whether Mafita had an impact on income-generating activities, we also want to understand some of the channels through which the program might improve these outcomes. One potential channel is through increasing efforts to find work or establish a business. To examine this, we consider a set of outcomes including (i) whether or not the respondent looked for work in the past six months, i.e., the 6 months preceding the follow-up survey; (ii) the number of months in which the person looked for work in the past six months; and (iii) whether or not the respondent tried to establish a business in the past six months. We also consider the same outcomes specifically for the last 30 days. Finally, we ask whether respondents searched for work in the past seven days and the number of days in which they did so. Table 5 below reports the effects of the apprenticeship program on each of these outcome indicators and on the overall standardized index that aggregates the individual outcomes. 42 Table 5: Apprenticeship effects on job search and entrepreneurship behavior Control Mean Control St.d Effect P > |t| N (Index) Job search behavior 0.000 0.934 0.105*** 0.001 4460 (0.031) Proportion of respondents that: Looked for work (past 6 months) 0.226 0.419 0.016* 0.050 4460 (0.013) Looked for work (past month) 0.122 0.328 0.033*** 0.003 4460 (0.011) Looked for work (past week) 0.070 0.254 0.019** 0.012 4460 (0.008) Tried to start a business (past 6 months) 0.579 0.494 0.067*** 0.001 4460 (0.015) Tried to start a business (past month) 0.464 0.499 0.067*** 0.001 4460 (0.016) Job search intensity: Months (in past 6 months) 0.630 1.399 0.031* 0.083 4460 (0.044) Days (in past month) 1.491 5.022 0.370*** 0.006 4460 (0.168) Days (in past week) 0.276 1.129 0.071** 0.029 4460 (0.038) The Mafita Apprenticeship program increased job search and entrepreneurship behavior. The summary index for this outcome family increases by 0.11 standard deviations for the treatment group. This is significant at the 1% level. Relative to the control group, Mafita apprentices are 12% and 14% more likely to try to start a business in the past 6 months and 30 days, respectively, corresponding to a 7 percentage point increase over the control group rate in both cases. They are also 27% more likely to look for a job in the past 30 days and 7 days, corresponding to increases of 3 percentage points and 2 percentage points, respectively. The results show that youth are generally more likely to seek opportunities in entrepreneur- ship than in wage-employment. As shown in Figure 15, youth targeted by Mafita are much more likely to attempt to start a business than to seek wage employment. For example, 46% of the control group report attempting to start a business in the past 30 days while only 12% report searching for wage-employment. Furthermore, while we observe a statistically significant increase in the number of days that youth spend looking for work in the past 30 days and 1 week, driven by female participants overall, this is starting from a low baseline: even with the program’s impact, Mafita apprentices report spending less than two of the last 30 days seeking employment. 43 Figure 15: Apprenticeship effects on entrepreneurship and job search behavior Positive effects on job search and entrepreneurship behavior are not consistent across all groups, however. As shown in Figure 16. We observe consistent positive impacts on job search behavior for males, females, Almajiri boys and non-IQE girls. We also find a statistically larger effects for Almajiri relative to other boys. Figure 16: Apprenticeship effects on job search and entrepreneurship behavior across youth categories *** Full *** Female ** Male *** Almajiri Non Almajiri * IQE ** Non IQE -.1 0 .1 .2 .3 .4 Magnitude Stars display level of significance based on p-value 5.1.4 Effects of the Apprenticeship program on literacy and numeracy Mafita’s Apprenticeship program included foundational skills training, including in basic lit- eracy and numeracy. The follow-up survey therefore assessed these foundational skills using a method developed by the Uwezo initiative, which was adapted for Mafita’s context.42 Participants were asked to complete an increasingly complex (but basic) set of tasks, such as letter, word, and reading comprehension 42 Uwezo, meaning “capability” in Kiswahili, is an initiative focusing of foundational competencies in literacy and numeracy among children aged 6-16 years in Kenya, Tanzania, and Uganda. 44 (for literacy) and counting, number recognition, values, and basic arithmetic including addition, subtraction, multiplication and division (for numeracy), which were related to the foundational skills taught by Mafita. We find no evidence of impact of the Mafita Apprenticeship program on basic literacy and numeracy. Participants received a score corresponding to the number of tasks they were able to complete correctly, which ranged from zero to five for the literacy assessment and zero to seven for the numeracy assessment. According to the Uwezo assessment training manual43 for Kenya, Tanzania and Uganda, a passing score in literacy (5 out of 5) entails answering correctly on a series of tasks up to simple reading comprehension. A passing score in numeracy (7 out of 7) entails answering correctly on a series of operations up to basic division. Based on this test, we do not detect any impacts on the program on basic literacy and numeracy. These results are consistent for the full sample and for some of the sub-groups examined in our analysis, including male participants; Almajiri and non-Almajiri boys; and IQE girls. For female participants and non-IQE girls, we find a small but statistically significant decrease in literacy in the treatment group. In absolute terms, this amounts to a decrease from an average score of 3.3 to an average score of 2.9. Scores on literacy and numeracy assessments are generally low. On average, members of the control group completed 3.1 out of 5 literacy tasks, with only 36% achieving a pass rate, i.e., answering all the literacy questions correctly. For numeracy, on average, members of the control group completed 5.4 out of 7 of the numeracy tasks, with 45% achieving a pass rate (for both literacy and numeracy, the treatment group as a whole neither performed better nor worse on either assessment).44 This raises im- portant questions. First, could limited foundational skills have been a constraint, for example in securing better wage-employment? Second, if foundational skills indeed are important, how can future programs be designed to better improve these? The qualitative evaluation provides additional insight on skills acquisition. With regards to foundational skills, participants in the Mafita program discussed being able to better read and write, i.e., improving their literacy levels, and being able to sign their names as a great benefit from the program. A number of participant noted learning to read and write in English but, again, any improvements in literacy were not large or consistent enough to emerge as significant on average through the quantitative evaluation. Some participants also expressed being better able to associate and communicate with others, i.e. improving their interpersonal skills, including with customers. This may relate to the finding that participants developed expanded social networks. Participants in the qualitative study also noted that they have developed new skills including new technical trade skills, have been exposed to new tools and equipment, and have developed newer ways of improving current work including learning the theory or more technical aspects of their jobs. For example, Musa, a COSDEC participant says, “I was only given minor jobs at my place of work before joining Mafita such as clothing cushions, but after I completed the program my boss now assigns more technical jobs to me like making an entire cushion frame or bed cabin.” Trade-specific skills were not measured through the quantitative evaluation.45 However, improvements in technical skills 43 http://www.uwezo.net/assessment/training/ 44 The Uwezo assessment/pass rates vary by country, region and by household income level (ultra-poor, poor, and non-poor). If we compare our findings with those of children 10+ in Uganda, Tanzania and Kenya, the average combined literacy and numeracy levels of Northern Nigerian youth, which stands at 41%, closely resembles that of the ultra-poor category in Kenya at 42% , poor category in Tanzania at 43% and non-poor category in Uganda at 45%. http://www.uwezo.net/about-us/uwezo- findings/ 45 Mafita included training in many different trades and each of these would have necessitated its own technical skills assess- ment, which would have added significant complexity to an already complex survey. From a methods perspective, it would have 45 are consistent with impacts on employment observed in both the quantitative and qualitative studies. 5.1.5 Effects of the Apprenticeship program on savings and loans Mafita apprentices save more relative to the control group. As shown in Figure 17, program participants are 18% (9 percentage points) more likely to have saved money in the past three months, 86% (6 percentage points) more likely to hold savings in a bank or cooperative, and 104% (29 percentage points) more likely to have a bank account. Corroborating these results, participants in the qualitative evaluation noted that Mafita encouraged and even facilitated participants’ opening of bank accounts. In general, Mafita included a number of access-to-finance interventions (e.g., support with bank account opening, linking participants to loan providers, etc.) which may have contributed to these encouraging findings. Figure 17: Apprenticeship effects on savings The Apprenticeship program has strong effects on participants’ likelihood of incurring business-related debt. As shown in Figure 18, Apprentices are 61% (11 percentage points) more likely to hold a business-related loan and 40% (10 percentage points) more likely to report planning to do so in the future. These results are consistent with the findings that the program increased the proportion of self-employed respondents. We do not see impacts on holding general-purpose loans, though apprentices are 26% (9 percentage points) more likely to report that they plan to do so in the future. further been difficult to identify which technical assessment to carry out on which members of the control group, so that any estimate of the impact of Mafita on trade-specific technical skills would have been suggestive at best. Given these difficulties, it was decided to prioritize the existing set of outcomes. 46 Figure 18: Apprenticeship effects on loans 5.2 Mafita Apprenticeship program effects on non-material outcomes The evaluation investigates the impacts of the Mafita Apprenticeship program on several categories of non-material outcomes. These include psychological well-being, social networks, views on female empowerment, attitudes towards and participation in anti-social behavior and different forms of violence, and religiosity. Figure 19 summarizes the effects of the program on this set of outcomes. Figure 19: Apprenticeship intervention effect on non-material outcomes Well being Self esteem Attitudes towards women's empowerment Caregiver att towards women's empowerment *** Range of social network Religious social network *** Religiosity Religious enforcement Participation in anti social behavior Participation in political or religious violence * Attitudes toward political violence -.2 -.1 0 .1 .2 Magnitude Stars display level of significance based on p-value, except for primary hypothesis as precised in the PAP Overall, we find mixed evidence of impact on non-material outcomes. To summarize, we do not find evidence that the intervention increased participants’ psychological well-being on the whole, though we see a small but significant increase in self-esteem for female participants. We also find that the program expands participants’ professional social networks, though we do not find any changes in the composition of religious social networks. This impact is driven by a 7% increase in the number of people in former apprentices’ networks who are employed. We do not find evidence of impact on attitudes towards female empowerment nor on anti-social behavior including stealing, taking drugs, or working for criminal groups. It should be noted, however, that participation in these behaviors is very low to begin with. We also find no 47 evidence that the Apprenticeship program affected participation in riots or the use of violence for political or religious motives but do see an increase in support for the use of violence, specifically to solve problems. We discuss these results in more detail in the remainder of this section. 5.2.1 Effects of the Apprenticeship program on subjective well-being and self esteem We use standard tools to measure subjective well-being and self-esteem. Subjective well-being is measured through an index comprising frequently used measures (Cantril’s ladder and an index of five ques- tions relating to mental-health known as the Mental Health Inventory-5). Our measurement of self-esteem comprises indicators for aspirations, resilience, problem-solving, and seeking to help others. We do not find evidence that Mafita’s Apprenticeship program increases these outcomes. While these results hold for the sample of study participants as whole, female participants do show a small but significant increase in the self-esteem index of 0.09 standard deviations, as shown in Table 6. This is significant at the 10% level. This increase in the index is driven by higher levels of confidence in one’s ability to solve problems and the belief that the future depends on oneself. Table 6: Apprenticeship effect on female self esteem Control Mean Control St.d Effect P > |t| N (Index) Self-esteem -0.000 0.987 0.086* 0.068 1916 (0.047) On a scale of 1 to 4 (highest), extent of confidence in/agreement with: Ability to to prevent bad things from happening 2.217 0.734 0.018 0.730 1916 (0.035) Ability to problems alone 2.197 0.788 0.106** 0.039 1916 (0.038) Ones future depending mainly on oneself 2.079 0.774 0.081* 0.093 1916 (0.037) Ability to deal with problems 2.828 0.686 -0.022 0.730 1916 (0.032) Not feeling exploited or cheated 2.368 0.839 0.018 0.730 1916 (0.039) Ability to share problems with friends, family, or community members 3.329 0.606 -0.038 0.432 1916 (0.029) Ability to achieve anything if dedicated 3.596 0.524 0.023 0.730 1916 (0.026) The mental health status of youth deserves some attention, as mental health plays a primary role in both personal and professional dimensions of everyday life. To measure this, we use a modified version of the Mental Health Inventory-5 (MHI-5), a brief and internationally validated tool for assessing mental health in adults which comprises five non-sensitive questions. These questions ask the respondents how often they feel (i) happy, (ii) peaceful (iii) nervous (iv) downhearted and (v) depressed. Based on their responses, participants are assigned a score, with higher values indicative of better mental health. Following convention from other studies, we can consider a score of less than 13 or 14 (out of 20) as indicative of depression, though it should be noted that this is indicative only and not a clinical diagnosis.46 Figure 20, shows the distribution of index scores for the control and treatment groups. We see that most study participants are at or near the threshold of 13/14, and an important proportion of respondents are 46 Typically, a 25-point scale is used MHI-5 developed by Veit and Ware (1983). We use a 20-point scale and adjust the depression “threshold” accordingly 48 below this threshold. As already noted, we do not find evidence of impact of the program on this mental health index. Figure 20: Distribution of MHI-scores for treatment and control youth group Mental Health Index 30 % of respondents 10 0 20 5 10 15 20 Control Treatment 5.2.2 Effects of the Apprenticeship program on attitudes toward female empowerment The Mafita Apprenticeship program included both male and female participants and may, consequently, have changed attitudes towards female empowerment among participants and their caregivers. We therefore test for impact on an index comprising a range of outcomes on women’s voice and agency, such as women’s right to make their own occupational choices or to disagree with younger male family members. The index also includes indicators on attitudes towards women’s handling of financial matters such as making large household purchases, about family planning including the decision to have children, and about women’s broader role in society. Indices were constructed differently based on the re- spondent’s gender and whether the respondent was a youth or a caregiver. Specifically, variables related to desired age of fertility, actual age of fertility, age of marriage, and experience of domestic violence were not included when constructing the index for male study participants. Additionally, for the caregiver analysis, a more restricted set of outcomes was used. We do not find evidence of impact on female empowerment. While there are statistically significant impacts on some individual items in the index, we cannot reject the possibility that these are due to chance alone and, in general, we do not observe any distinctive patterns. This result holds for male and female apprentices, and for their caregivers. 5.2.3 Effects of the Apprenticeship program on social networks Through exposing participants to a new set of peers, Mafita may have expanded participants social networks which may affect economic and non-material well-being. We focus specifically on whether participants developed more connections with people who are employed or who live in different neighborhoods as proxy indicators for expanded professional social networks. We also examine whether the Apprenticeship program affected the number of persons in participants’ networks that attend religious school 49 and that practice a different religion from the respondent as proxy indicators for religious social networks. It is important to understand changes in professional social networks as this may be one channel through which Mafita improves labor market outcomes such as employment and income. It is also important to understand changes along more social dimensions, such as religion, as exposure to a wider variety of perspectives may affect attitudes towards social phenomena and related behaviors. We find that the program expands participants’ professional social networks. Specifically, we ask about the five people each respondent spends the most time with and, of those, how many are employed and how many live in the same neighborhood.47 As shown in Table 7, the impact is driven by a 7% increase in the number of people in former apprentices’ networks who are employed. This is corroborated by the qualitative study, which also found increased social networks centered around participants’ trades as an important feature of their Mafita experience. This is illustrated by a female participant, Hafsa, who said, “Honestly, before when you were not doing anything, people would look down on you, but now as a result of your trade, people respect you and even people that you didn’t associate with prior to enrolling in Mafita will try to associate themselves with you.” Table 7: Apprenticeship effects on professional social networks Control Mean Control St.d Effect P > |t| N (Index) Range of social network 0.000 0.996 0.138*** 0.000 4460 (0.031) Among 5 main contacts, number that: Are employed 3.177 1.645 0.229*** 0.001 4460 (0.047) Live outside respondents neighborhood 0.840 1.164 0.055* 0.073 4460 (0.037) Effects are larger for female apprentices. Figure 21 summarizes the impact on this two-item index across different population sub-groups. Further statistical analysis confirms there is a significant difference in results between female and male apprentices with the former displaying greater impacts on both variables included in the index. For non-IQE girls, for example, we find that the number of people employed in their network increases by 15% (for the sample as a whole, it increases by 7%). It is noteworthy that Almajiri do not report an increase in their professional social networks, suggesting that this group may face additional social constraints. 47 Because we ask about the five persons the respondent spends the most time with, we might consider any changes to reflect an increase or a decrease in “diversity” of the social network rather than an expansion/reduction per se. For the purpose of this report, these interpretations have similar implications. 50 Figure 21: Apprenticeship effects on range of professional social network by youth category. *** Full *** Female * Male Almajiri *** Non Almajiri * IQE *** Non IQE -.2 0 .2 .4 Magnitude Stars display level of significance based on p-value We do not find any changes in the composition of participants’ religious social networks. Here, our index comprises indicators based on three questions. The first asks, for the five people each respondent spends the most time with, how many practice another religion. The second and third ask about respondents’ degree of trust toward people that practice another religion and toward leaders of other religions, respectively. Results show that study participants’ social networks are very concentrated within their own religion, with only 14 persons in the control group reporting that there is someone that practices a religion different from theirs among the five persons they spend the most time with (this translates to less than .01 such contacts on average, as shown in table 8). Table 8: Apprenticeship effects on religious social networks Control Mean Control St.d Effect P > |t| N (Index) Religious social network -0.000 1.022 0.026 0.397 4460 (0.031) Among 5 main contacts, number that: Practice other religion 0.009 0.098 -0.003 0.530 4460 (0.003) Trust in: Practicants of other religions 2.693 0.897 0.036 0.530 4460 (0.028) Leaders of other religions 2.769 0.955 0.033 0.530 4460 (0.030) We observe a significant decrease in an index of religiosity among Mafita apprentices. As shown in Table 9, this results from small but significant decreases in hours spent on religious activities and perceptions of the importance of religion and of God’s presence. Former Mafita apprentices report spending about 20 minutes less per week on religious activities, which represents a decrease of 2% from the control group average of 19.4 hours. 99% of the control group report feeling a strong sense of God’s presence, and this declines by only 1 percentage point for the treatment group. Declines in perceptions of the importance 51 of religion are similarly modest. We interpret these results as indicative of an expansion of the types of non-religious activities and people youth regularly engage with, rather than a change in religiosity per se. This interpretation is supported by the qualitative study, which found that time formerly spent on religious activities was re-allocated to trade-related activities. As such, these results can be seen as related to the observed expansion in professional social networks. Table 9: Apprenticeship effects on religiosity Control Mean Control St.d Effect P > |t| N (Index) Religiosity 0.000 0.779 -0.129*** 0.000 4460 (0.031) On a scale of 1 to 4 (highest): Hours spent on religious activities (past 7 days) 19.426 12.662 -0.361* 0.095 4460 (0.383) Importance of religion 0.999 0.034 -0.004** 0.040 4460 (0.002) Nothing is more important than religion 0.792 0.403 -0.030** 0.030 4460 (0.013) Feeling a strong sense of Gods presence 0.990 0.101 -0.012*** 0.007 4460 (0.004) 5.2.4 Effects of the Apprenticeship program on enforcement of religious norms and anti-social behavior In addition to improving participants’ economic outcomes, Mafita was seen as having the po- tential to reduce support for religious extremism and participation in violence and other risky behavior. Regarding the former, rather than ask such questions directly (which would be highly subject to social desirability bias and which may make respondents uncomfortable or suspicious of the survey’s motives), we use a more indirect form of questioning which employs audio vignettes on the punishment of strangers for engaging in behaviors not consistent with religious norms. These include consuming alcohol, blasphemy, and wearing revealing clothing such as a mini-skirt. We find that the Apprenticeship program has no effect on attitudes towards the enforcement of religious norms. The analysis also finds that the program had no overall effect on anti-social behavior in- cluding stealing, taking drugs, or working for criminal groups. This is measured through an index of six behaviors highlighted in Figure 22. Though we measure very small decreases in each of these behaviors for the treatment group, these are not large enough to be statistically significant either individually or as a group. It should be noted, however, that for female participants specifically, the program reduced this index, driven by a 59% (1.6 percentage points) decrease in the consumption Tramadol of Beneline, common pain killers. . It should be noted that participation in anti-social behavior is very low to begin with. As shown in Figure 22, in the control group less than 5% of respondents admit to behaviors such as taking drugs or participating in violence and crime. While such outcomes are sensitive and may be subject to under-reporting, these results are corroborated by the qualitative evaluation. This is contrary to popular perceptions that marginalized youth such as Almajiri are heavily involved with violence and crime in Northern Nigeria. The results should thus be interpreted as suggesting that apprenticeship training does not produce significant changes in participation in violence, starting from an already low base. 52 Figure 22: Apprenticeship effects on participation in anti-social behavior (in the past 30 days) We also find that the Mafita Apprenticeship program did not affect participation in riots or the use of violence for political or religious motives. However, one area of concern is that, as shown in Figure 23, 24% of participants (based on the control group average) report actually using force or violence for religious reasons. This stands in stark contrast to the reported incidence of other forms of violence Figure 23: Apprenticeship intervention effect on participation in political or religious violence (in the past 3 months) However, we do find significant effects on attitudes towards violence, with those in the program more likely to agree that violence should be used to resolve problems. This is shown in Figure 24, which shows the proportion of control group respondents agreeing with various statements related to the use of violence and the estimated treatment effect. We are unsure of potential mechanisms through which the program may have caused respondents to agree more with the statement that violence should be used to resolve problems and, as the relationship is only weakly statistically significant at the 10% level, 53 would caution against over-interpretation of this apparent anomaly. Also, while the estimated proportional increase, at 28%, is large, the actual increase is a modest 1.9 percentage points. Figure 24: Apprenticeship intervention effect on attitudes toward violence 5.3 Mafita COSDEC program effects on economic outcomes Overall, the Mafita COSDEC intervention had strong and positive effects on participants’ employment and productivity, job search behavior and economic welfare, though some out- comes display striking heterogeneity in the results. Relative to the control group, former COSDEC participants were 35% (14 percentage points), more likely to be employed in self- or family-owned businesses (henceforth, self-employed); 39% (4 percentage points) more likely to be in wage employment; raised their profits from self-employment by 38% and income from wage employment by 54% ; and increased the time they spent on self-employment activities by 31%. They were also 20% and 22% more likely to have attempted to start a business in the past 6 months and past month, respectively. Former COSDEC participants also owned more assets and had higher consumption expenditures than did members of the control group. In particular, they spent 23% more on clothes. We do not find evidence of impact on literacy and numeracy. The overall effects on economic outcome categories are summarized in Figure 25 and discussed in more detail below. 54 Figure 25: Summary effects of the COSDEC program on economic welfare outcomes *** Income generating activities * Job search behavior *** Assets and expenditures Literacy and numeracy -.1 0 .1 .2 .3 Magnitude Stars display level of significance based on p-value, except for primary hypothesis as precised in the PAP 5.3.1 Effect of the COSDEC program on income generating activities For income generating activities, we analyze several variables that were pre-specified in the pre-analysis plan, including self/wage employment in the past 30 days, the number of days spent on these activities, and income from wage employment or profit from self-employment over this period. Table 10 below reports the effects of the COSDEC program on each of these individual outcome measures and on the overall standardized index that aggregates individual outcomes. Table 10: COSDEC effects on income generating activities Control Mean Control St.d Effect P > |t| N (Index) Income generating activities -0.000 0.915 0.238*** 0.001 1708 (0.046) In past 30 days: Wage employment (proportion) 0.108 0.311 0.042*** 0.007 1708 (0.016) Self employment (proportion) 0.382 0.486 0.136*** 0.001 1708 (0.024) Wage employment (# of days) 1.541 5.358 0.344** 0.033 1708 (0.261) Self-employment (# of days) 6.206 9.484 1.898*** 0.001 1708 (0.473) Income from wage employment (Naira) 750.235 3038.836 406.362** 0.013 1708 (174.411) Profit from self-employment (Naira) 2578.147 6046.679 979.417*** 0.003 1708 (317.998) The COSDEC program had strong and positive effects on the overall index of income- generating activities. As highlighted in Table 10, the COSDEC participants had on average 0.24 stan- dard deviations higher values across all six outcomes compared to individuals in the control group. This is 55 significant at the 1% level. Individually, all outcomes show strongly significant positive results. We find large positive and statistically significant impacts of the COSDEC program on self- and wage-employment and productivity, with modest but economically meaningful increases in income. Relative to the control group, Mafita COSDEC participants increased employment in self- or family-owned businesses by 37% (14 percentage points) and wage employment (i.e., outside employment) by 36% (4 percentage points). Program participants also worked more regularly, increasing the time they spend on self- and wage employment by 31% (1.9 additional days over a 30-day period) and 22% (0.3 addi- tional days), respectively. Finally, profit from self-employment increases by 38% (979 Naira over a 30-day period) and income from wage-employment increases by 54% (406 Naira over the same period. These are very encouraging results, especially given that they reflect the economic situation of Mafita youth six to nine months after program completion in a difficult labor market context. As with apprenticeship participants, the results suggest that returns to self-employment are generally higher for COSDEC participants and/or that there are additional constraints to wage employment for this population. Results from the control group provide perspective on the generally low levels of wage employment, in absolute terms and relative to self-employment: in the con- trol group, 38% report self-employment while only 11% report wage employment. Also, income from wage employment in the control group is 750 Naira on average, whereas profit from self-employment is 2,578 Naira. Effects on income-generating activities appear largest for IQE girls. This is shown in Figure 26 and is supported by the heterogeneity regression Table A17 in the Appendix which show that impacts for IQE girls are statistically different from those of non-IQE girls. Figure 26: COSDEC effects on income generating activities across youth categories *** Full *** Female *** Male ** Almajiri * Non Almajiri *** IQE Non IQE -.2 0 .2 .4 .6 Magnitude Stars display level of significance based on p-value adjusted for multiple hypothesis testing The striking result for IQE girls is driven by strong increases in the rate of self-employment, time spent on self-employment activities, and in related profits. Figure 27 shows impact on self-employment for the full sample, for male and female participants, and for IQE girls. This highlights 56 statistically strong and economically meaningful impacts on self-employment for both male and female participants (for female participants, self-employment increases by 51%; the corresponding increase for male participants is 23%). The larger impact on female participants was corroborated by the qualitative study, in which young women were more likely than young men to discuss starting new productive activities aligned with their training and more likely to frame their former life as idle. For IQE girls, however, the rate of self-employment increases by 74% (26% points above the control group). IQE girls increase the number of days spent on self-employment activities by 83% (4.46 days more spent on such activities over the past 30 days). Figure 27: COSDEC effects on self employment Estimated increases in IQE girls’ income from self-employment are also large and statisti- cally significant, though these are relative to low values in the control group. As shown in Figure 28, IQE girls’ profit from self-employment more than doubles, increasing by 1,424 Naira over a control group average of 1,332 Naira. Consistent with results from the Apprenticeship IE, this shows that Mafita was particularly successful in its goal of increasing the economic prospects of this highly vulnerable group. In fact, the overall effects on income-generating activities observed for female participants are driven by IQE girls. Figure 28: COSDEC effects on profit from self employment 57 We do not observe similarly strong increases for wage employment-related outcomes among IQE girls, however. This highlights the constraints that such marginalized groups face in entering wage employment. IQE control group participation in wage employment is just 5%. Though we do not observe a significant difference in the overall income-generating activities index for male and female participants, it seems that the COSDEC program’s effect on wage employment for male participants were larger. As shown in Figure 29, increases in the rate of wage employment are statistically significant for the full sample and for male participants, but not for female participants. The proportion of male COSDEC participants engaged in wage employment activities increases by 46% while no effect is observed for female participants, even relative to a lower control group average. Similarly, a significant increase in income is observed for male participants (increase of 765 Naira, or 65% more than the control group). It is important to note that the null effects on wage employment for female participants may be a consequence of gender-specific labor market constraints, i.e., it may be harder for females to enter the labor market independent of their qualification, which Mafita was not designed to directly address. Furthermore, women may tend to naturally gravitate, or “self-select”, into self-employment activities knowing the greater barriers they face in the market for wage labor. Figure 29: COSDEC effects on wage employment 5.3.2 Effect of the COSDEC program on asset ownership and consumption expenditure For this family of outcomes, we analyze two sets of variables. The first set relates to household ownership of financial and durable assets, including owning a bank account and things like a cell phone, fan, television, and electric iron, etc. The second relates to respondents’ basic expendi- tures on food, transport, and airtime (over the seven days preceding the survey) and on medical expenses, clothes, leisure, and other expenses (over the 30 days preceding the survey). Table 11 below reports the effects of the COSDEC program on outcomes in each of these sets and on the overall standardized index that aggregates individual outcomes. The COSDEC program had strong and positive effects on the overall index of asset owner- ship and consumption. As shown in Table 11, the overall index is 0.18 standard deviations higher for the treatment group of Mafita COSDEC participants. This difference is significant at the 1% level. This table 58 also displays the effects of the COSDEC program on specific outcome indicators. Specifically, we see indi- vidually statistically significant effects on ownership of generators and cell phones, though in absolute terms these effects are small (4% and 1% increases, respectively). Former COSDEC participants also reported spending 23%, or 235 Naira, more on clothes in the past 30 days. Table 11: COSDEC effects on asset ownership and consumption expenditure Control Mean Control St.d Effect P > |t| N (Index) Assets and consumption -0.000 0.973 0.176*** 0.000 1708 (0.047) Proportion of respondents whose household has at least one: Electric iron 0.648 0.478 0.034 0.272 1708 (0.022) Fan 0.615 0.487 0.001 1.000 1708 (0.023) Television 0.602 0.490 0.023 0.448 1708 (0.023) Refrigerator 0.281 0.449 0.019 0.448 1708 (0.021) Generating set 0.156 0.364 0.045* 0.067 1708 (0.018) Satellite or cable TV 0.239 0.427 0.002 1.000 1708 (0.021) Electricity 0.881 0.324 0.011 0.486 1708 (0.015) Bank account 0.759 0.426 0.036 0.202 1708 (0.020) Cell phone 0.982 0.132 0.014* 0.067 1708 (0.005) Respondents expenditure (in Naira), in past 7 days, on: Food 501.012 909.565 68.216 0.125 1708 (44.068) Transport 292.765 518.864 51.266 0.609 1708 (25.358) Airtime 168.394 287.967 28.822 0.250 1708 (14.014) Medical expenses 483.441 1172.660 242.284 0.250 1708 (63.646) Clothes 1006.527 2190.506 234.742* 0.056 1708 (108.249) Soap/Detergent/Cosmetics 435.615 649.853 5.436 0.602 1708 (30.058) Leisure 155.626 586.661 108.712 0.272 1708 (33.137) Other 84.282 495.270 28.881 0.106 1708 (25.357) Total (last 7 days) 962.171 1309.727 148.305** 0.041 1708 (63.481) The increase in consumption expenditure is corroborated by evidence from the qualitative evaluation which finds that increases in income, though generally small, generated a certain level of financial independence which was valued by participants. Furthermore, the qualitative evaluation suggests that female participants benefited relatively more, either because they were more likely to lack personal sources of income prior to the program (as corroborated by the quantitative findings) or because they tended to participate in more lucrative trades, such as fashion design. Statistical analysis of the quantitative data, however, does not confirm the presence of any differential effect by gender on asset ownership and consumption expenditure. 59 5.3.3 Effect of the COSDEC program on job search and entrepreneurship behavior In addition to knowing whether Mafita had an impact on participation in wage and self- employment, we also want to understand some of the channels through which the program might improve these outcomes. One potential channel is through increasing efforts to find work or establish a business. To examine this, we consider a set of outcomes including (i) whether or not the respondent looked for work in the past six months, i.e., the 6 months preceding the follow-up survey; (ii) the number of months in which the person looked for work in the past six months; and (iii) whether or not the respondent tried to establish a business in the past six months. We also consider the same outcomes for the last 30 days. Finally, we ask whether respondents searched for work in the past seven days and the number of days in which they did so. Table 12 below reports the effects of the COSDEC program on each of these outcome indicators as well as on the overall standardized index that aggregates the individual outcomes. Table 12: COSDEC effects on job search and entrepreneurship behavior Control Mean Control St.d Effect P > |t| N (Index) Job search behavior -0.000 0.983 0.078* 0.099 1708 (0.048) Proportion of respondents that: Looked for work (past 6 months) 0.190 0.392 0.003 1.000 1708 (0.019) Looked for work (past month) 0.104 0.305 -0.001 1.000 1708 (0.015) Looked for work (past week) 0.059 0.235 0.007 1.000 1708 (0.012) Tried to start a business (past 6 months) 0.455 0.498 0.093*** 0.001 1708 (0.024) Tried to start a business (past month) 0.359 0.480 0.080*** 0.002 1708 (0.023) Job search intensity: Months (in past 6 months) 0.485 1.217 0.027 1.000 1708 (0.059) Days (in past month) 1.186 4.323 -0.031 1.000 1708 (0.209) Days (in past week) 0.207 0.951 0.044 1.000 1708 (0.049) The COSDEC program increased job search and entrepreneurship behavior. The summary index for this outcome family increases by 0.08 standard deviations for the treatment group, significant at the 10% level. This overall effect is driven entirely by attempts to start a business in the last 6 months and one month, which increase by 20% (9 percentage points) and 22% (8 percentage points), respectively. These results appear stronger for female participants, though statistical analysis of heterogeneity by gender do not confirm that the difference in impact between male and female participants is statistically significant. The results show that youth are generally more likely to seek opportunities in entrepreneur- ship than in wage-employment. As shown in figure 30, youth targeted by Mafita are much more likely to attempt to start a business than to seek wage employment. For example, 36% of the control group report attempting to start a business in the past month, while only 10% report searching for wage employment. 60 On average, youth in the control group report spending only 1.19 days in the past month seeking wage employment. Figure 30: COSDEC effects on job search and entrepreneurship 5.3.4 Effect of the COSDEC program on literacy and numeracy Mafita’s COSDEC program included foundational skills training, including basic literacy and numeracy. The follow-up survey therefore assessed these foundational skills using an assessment method developed by the Uwezo initiative, which was adapted for Mafita’s context.48 Participants were asked to complete an increasingly complex (but basic) set of tasks, such as letter, word, and reading comprehension (for literacy) and counting, number recognition, and basic arithmetic (for numeracy), which were related to the foundational skills taught in the Mafita curriculum. Basic literacy and numeracy tests were carried out on Mafita participants and members of their respective control group. As with the Apprenticeship program, we find no evidence of a general impact of the COS- DEC program on these foundational skills. Scores on literacy and numeracy assessments are generally low, at 55% question accuracy for Literacy and 72% question accuracy for Numeracy. On average, mem- bers of the control group completed 2.8 out of 5 literacy tasks, with only 29% of control group respondents achieving a pass rate, i.e answering all the literacy questions correctly and scoring 5 out of 5. In the case of numeracy, on average, members of the control group completed 5 out of 7 of the numeracy tasks, with only 35% of control group respondents achieving a pass rate, i.e answering all the literacy questions correctly and scoring 7 out of 7. Again, the treatment group neither performed better nor worse. 5.3.5 Effect of the COSDEC program on savings and loans COSDEC participants save more relative to the control group. As shown in Figure 31, program participants are 27% (12 percentage points) more likely to have saved money in the past three months, 66% (4 percentage points) more likely to hold savings in a bank or cooperative, and 96% (28 percentage points) more likely to have an individual bank account. Corroborating these results, participants in the qualitative 48 Uwezo, meaning capability in Kiswahili, is an initiative focusing of foundational competencies in literacy and numeracy among children aged 6-16 years in Kenya, Tanzania, and Uganda. 61 evaluation noted that Mafita encouraged and even facilitated participants’ opening of bank accounts. In gen- eral, Mafita included a number of access-to-finance interventions (e.g., support with bank account opening, linking participants to loan providers, etc.) which may be behind these encouraging findings. Figure 31: COSDEC effects on savings The COSDECs program also had strong effects on participants’ likelihood of incurring debt, particularly business-related debt. As shown in Figure 32, participants are 13% (4 percentage points) more likely to hold any debt and 46% (14 percentage points) more likely to hold a business-related loan. Furthermore, program participants are 26% (10 percentage points) more likely to plan to borrow for business purposes in the future, and 21% (10 percentage points) more likely to plan to borrow in general. These results are consistent with the positive impacts observed on entrepreneurship. Figure 32: COSDEC effects on loans 5.4 Mafita COSDEC program effects on non-material outcomes The Mafita impact evaluation also investigates the impacts of the Mafita COSDEC program on several categories of non-material outcomes. These include psychological well-being, social networks, 62 views on female empowerment, attitudes towards and participation in anti-social behavior and different forms of violence, and religiosity. Figure 33 summarizes the effects of the program on this set of outcomes. Figure 33: Summary effects of the COSDEC program on non-material outcomes ** Well being Self esteem *** Attitudes towards women's empowerment ** Caregiver att towards women's empowerment *** Range of social network Religious social network * Religiosity Religious enforcement Participation in anti social behavior Participation in political or religious violence Attitudes toward political violence -.2 -.1 0 .1 .2 .3 Magnitude Stars display level of significance based on p-value, except for primary hypothesis as precised in the PAP Overall, we find mixed evidence of impact on non-material outcomes. To summarize, we find evidence that the program increased participants’ subjective well-being, driven by improvements for female participants and IQE girls in particular. We also find that the program expands participants’ professional social networks, though we do not find any changes in the composition of religious social networks. Finally, we find strong evidence of impacts on attitudes towards female empowerment both among participants themselves (driven by non-IQE girls and non-Almajiri boys) and among caregivers. However, we do not find evidence of impacts on anti-social behavior including stealing, taking drugs, or working for criminal groups. It should be noted, however, that participation in these behaviors is very low also among the control group participants. We also find no evidence that the COSDEC program affected participation in riots or the use of violence for political or religious motives. We discuss these results in more detail below. 5.4.1 Effects of the COSDEC program on subjective well-being and self esteem The COSDEC program had a positive effect on an index of subjective well-being, though not on a separate index of self-esteem. The subjective well-being index increases by 0.11 standard deviations, significant at the 5% level, for COSDEC participants relative to the control group. This is driven by changes in current life satisfaction as reported through a commonly used measure known as Cantril’s ladder. Here, participants are asked to place themselves on a 10-step ladder, with the top of the ladder representing their best possible life and the bottom of the ladder representing their worst possible life. COSDEC participants place themselves 0.3 steps higher, an increase of 14% over the control group average of 2.26. Thus, though this change is statistically significant it appears to be a small change in real terms. We also do not find evidence of any change in anticipated life satisfaction in five years’ time. 63 Figure 34: COSDEC treatment and control groups’ life satisfaction now and in 5 years’ time Cantril's ladder (now) Cantril's ladder (5 years) 30 20 15 % of respondents % of respondents 20 10 10 5 0 0 0 2 4 6 8 2 4 6 8 10 Control Treatment Control Treatment As with youth that participated in the Apprenticeship program impact evaluation, the mental health of COSDEC youth deserves attention. We do not observe evidence of impact on mental health as measured through the Mental Health Inventory-5 (MHI-5), a brief and internationally validated too for assessing mental health in adults. Based on their responses, participants are assigned a score with higher values indicative of better mental health. Following convention from other studies, we can consider a score of less than 13 or 14 as suggestive of depression, though this should be interpreted with caution as indicative only and not as a clinical diagnosis. We see that an important proportion of study participants are at or near the threshold, and a significant minority are below the threshold. This is an important area for future study, as mental health plays a primary role in well-being and in achieving positive life outcomes. Figure 35: Distribution of MHI-scores for treatment and control youth group Mental Health Index 30 % of respondents 10 0 20 5 10 15 20 Control Treatment 5.4.2 Effects of the COSDEC program on attitudes toward female empowerment The COSDEC program had strong and statistically significant impacts on participants’ atti- tudes towards female empowerment. On average, participants scored 0.13 standard deviations higher on an index combining several indicators, which is significant at the 1% level. This index includes several indicators on women’s voice and agency, on attitudes towards women’s role in household decision-making, 64 and on their broader role in society. Impacts appear to be driven by improved attitudes towards women’s empowerment amongst female COSDEC participants, particularly non-IQE girls. Additionally, among male participants we see evidence of improvement for non-Almajiri only.49 This suggests that such attitudes may be more difficult to change for groups that are highly exposed to very traditional values. The program also improved caregiver attitudes towards female empowerment. On average, the index of attitudes towards women’s empowerment increases by 0.05 standard deviations for caregivers with a female ward in the program, relative to a control group of caregivers with female wards but none en- rolled in the program. This is significant at the 10% level.50 This is a very promising finding, both from the perspective of individual participants (since caregivers have an important influence on young women’s voice and agency, including key life decisions such as those concerning marriage or childbearing) and in terms of the ability of programs such as Mafita to promote positive social impacts beyond immediate program participants. The qualitative study finds that several female non-Mafita respondents explained that their parents or guardians would not allow them to open a shop or work in a shop; meanwhile the caregivers of Mafita respondents were often instrumental in recruiting participants. 5.4.3 Effects of the COSDEC program on social networks The COSDEC program expands participants’ professional social networks. As shown in Table 13, the index increases by 0.18 standard deviations, significant at the 1% level. This is due to both an increase, among the people participants spend the most time with, in the number that are employed (10% increase) and that live outside their neighborhood (3% increase). As shown in Figure 36, the increase appears to be driven by female participants and by non-IQE girls in particular, though the differences between the different groups are not large or consistent enough to be statistically significant. The qualitative study also finds that participants experienced expanded social networks that largely centered around their trade and friends they made in Mafita. Building friendships with employed individuals also appeared to be encouraged by some COSDEC trainers. For example, Maryam shared how she used to only spend time with her husband, but now has friends through Mafita and her work in carpentry. Tasi’u spoke about being an introvert without any friends but now having multiple good friends through Mafita. Table 13: COSDEC effects on professional social networks Control Mean Control St.d Effect P > |t| N (Index) Range of social network -0.000 0.989 0.182*** 0.000 1708 (0.048) Among 5 main contacts, number that: Are employed 2.818 1.621 0.266*** 0.001 1708 (0.076) Live outside respondents neighborhood 1.212 1.342 0.117** 0.038 1708 (0.065) 49 Differences in impact on the overall index between male and female participants, between IQE and non-IQE girls, and between Almajiri and non-Almajiri boys are not large or consistent enough to be statistically different, however. 50 Results on individual indicators are not statistically different from zero. When aggregated into the index, however, the general positive tendency is reinforced and becomes significant. 65 Figure 36: COSDEC effects on professional social networks by youth category. *** Full *** Female * Male Almajiri Non Almajiri * IQE *** Non IQE -.2 0 .2 .4 .6 Magnitude Stars display level of significance based on p-value We do not find evidence of overall changes in the composition of participants’ religious social networks. As shown in Table 14, however, we do observe impacts on individual items included in the index. Specifically, COSDEC participants report significant decreases in trust in adherents and leaders of religions other than their own, though in practical terms these decreases seem small (in both cases, the average reported decrease is 0.1 on a 5-point scale). Table 14: COSDEC effects on religious social networks Control Mean Control St.d Effect P > |t| N (Index) Religious social network 0.000 0.935 -0.064 0.173 1708 (0.047) Among 5 main contacts, number that: Practice other religion 0.028 0.231 0.020 0.156 1708 (0.014) Trust in: Practicants of other religions 2.636 0.886 -0.092** 0.048 1708 (0.040) Leaders of other religions 2.516 0.970 -0.110** 0.035 1708 (0.046) We do, however, observe a significant decrease in an index of religiosity among Mafita COS- DEC participants. Relative to the control group, this decreases by 0.09 standard deviations, significant at the 10% level (see Table 15). We interpret this as indicative of an expansion of the types of non-religious activities and people youth regularly engage with, rather than as a decrease in the importance of religion per se. As such, these results can be seen as related to the observed expansion in professional social networks. This interpretation is supported by the qualitative study. For example, a caregiver in Katsina explained how young women previously spent much of their time “pursuing Islamic knowledge” but, after Mafita, they spend “most of their time with other professionals.” 66 Table 15: COSDEC effects on religiosity Control Mean Control St.d Effect P > |t| N (Index) Religiosity 0.000 0.985 -0.086* 0.064 1708 (0.046) On a scale of 1 to 4 (highest): Hours spent on religious activities (past 7 days) 20.625 13.175 -0.595 0.484 1708 (0.605) Importance of religion 1.000 0.000 -0.001 0.484 1708 (0.001) Nothing is more important than religion 0.733 0.442 -0.042 0.247 1708 (0.022) Feeling a strong sense of Gods presence 0.978 0.148 -0.003 0.768 1708 (0.007) 5.4.4 Effects of the COSDEC program on enforcement of religious norms and anti-social behavior Similar to the Apprenticeship program, we find that the COSDEC program has no overall effect on attitudes towards the enforcement of religious norms or on anti-social behavior. As shown in Figure 37, no impacts are observed on a range of anti-social behaviors (though there are observed increases of decreases in some of these behaviors that appear large in relative terms, these are from very low baselines and in any case are not large enough to be statistically significant). As with Apprenticeship program participants, the general participation in these behaviors is low, with less than 5% of control group participants reporting having engaged in them in the past 30 days. The results should thus be interpreted as indicating that COSDEC participation did not reduce these behaviors, starting from an already low base. Similarly, we do not see any effects on attitudes towards the enforcement of religious norms. Figure 37: COSDEC effects on participation in anti-social behavior (in the past 30 days) COSDEC participation did not affect participation in riots or the use of violence for political or religious motives. However, one area of concern is that 15% of participants (based on the control group average) report actually using force or violence for religious reasons, as highlighted in Figure 38 below. These results are corroborated by the qualitative evaluation, which finds that while participants generally view violence as unacceptable an exception is with regards to violence to right a religious wrong, such as blasphemy. 67 Figure 38: COSDEC intervention effects on participation in political or religious violence (in the past 3 months) We also do not find evidence that COSDEC participation changed attitudes towards the use of violence. As shown in Figure 39, we do find, however, that just over 15% of participants (based on the control group average) report that use of violence is justified in protesting an injustice, similar to the proportion that report actually having used violence for religious reasons. The qualitative study also found that, in the rare cases where participants said violence was justified, it was to address religious violations or injustice. Figure 39: COSDEC intervention effects on attitudes toward violence 6 Discussion and Policy implications 6.1 Summary of results from Mafita apprenticeship and COSDEC programs Table 16 presents a summary of the main results from both the Apprenticeship and COSDEC impact evaluations across all outcome families. Mafita primarily targeted economic and labor market outcomes, where the results largely show significant positive impacts. At the same time, programs such as Mafita are often justified on the basis of positive effects on non-material outcomes. The impact evaluation found evidence that Mafita (in particular, the COSDEC program) led to positive changes in some non- material outcomes such as increases in female empowerment, subjective well-being, and professional social networks. However, we found no evidence for consistent improvements in other non-material outcomes, such 68 as reductions in anti-social behavior or reduced participation in or support for violence. Table 16: Summary Results from the Mafita IEs Evidence of positive Evidence of positive Outcomes impacts (Apprenticeship) impacts (COSDEC) Material outcomes Income generating *** *** activities Job search and entrepreneurship *** * behavior Consumption expenditures and *** *** assets ownership Literacy and numeracy *** *** Non material outcomes Self-esteem *** *** Subjective well-being *** *** Professional social networks *** *** Religious social networks *** *** Youth attitudes towards *** *** female empowerment Caregiver attitudes towards *** *** female empowerment Enforcement of religious norms *** *** Anti-social behavior *** *** Participation in political or *** *** religious violence Attitudes toward violence *** *** Note: For each program (Apprenticeship or COSDEC), an outcome family receives one star if the overall estimated effect is statistically significant at the 90% level, and two or three stars respectively if these effects are also significant at the 95% or 99% level. Significant outcomes with an unexpected direction are indicated in red This study provides robust evidence of the causal impacts of Mafita’s Apprenticeship and COSDEC programs, revealing four important high-level findings. First, both the Apprenticeship and COSDEC programs were successful in increasing participants’ engagement in income generating ac- tivities and earnings, which in turn contributed to the improvement of their economic welfare in terms of increased consumption expenditure and household asset ownership. Second, for both programs, improve- ments in outcomes related to income-generating activities appear to be larger for female participants, for whom we measure large increases in self-employment and wage employment, especially for IQE girls. Third, neither program appears to have had positive impacts on literacy and numeracy foundational skills, though qualitative evidence suggests participants had positive perceptions of the technical skills they acquired and of the foundational skills training component.51 Fourth, the effects of both programs on non-material outcomes are mixed, with the COSDEC program showing better results on these outcomes. A notable exception is 51 It is possible that sub-groups of Mafita participants benefitted in the form of increased literacy and numeracy, but these effects were not large or consistent enough to be detected through the quantitative analysis. 69 the expansion of participants’ social networks, which shows strong positive impacts for both programs. Below, we dissect these findings, comment on comparisons and contrasts between the two programs, and draw on both quantitative and qualitative evidence to highlight some of the mechanisms that may explain the results across the two Mafita programs evaluated. It should be noted, however, that the two programs were implemented in different locations and targeted populations who may have had important differences even if they conform to the same basic groups (e.g., Almajiri or IQE girls). Differences in results between the two programs should therefore be interpreted cautiously and should not be seen as providing evidence of the superiority of one model versus the other. Mafita was successful in increasing employment. Overall, we find strong evidence that the Ap- prenticeship and COSDEC programs improved employment outcomes, measured 6-9 months post-program, in comparison to the control group. Employment effects are stronger for self-employment than for wage employment for both programs. Furthermore, the rate of self-employment is higher than the rate of wage employment, even in the control group. This is consistent with the findings of the qualitative evaluation, with existing literature, and with the Northern Nigerian labor market context which highlights several structural constraints to wage employment. It suggests that youth face additional important obstacles to securing high-quality outside job opportunities that require other programs or structural changes beyond what programs like Mafita are designed for. This is further highlighted by the fact that, among youth in our control group, about twice as many report making attempts at starting a business as report looking for wage employment, and the same is true for Mafita program participants. In addition to higher rates of employment, the Mafita program provides important eco- nomic benefits to program participants in the form of increased earnings. For both programs, we find positive and significant effects on earnings from self-employment. Additionally, COSDEC participants, also report higher income from wage employment. Post-program earnings gains are mainly achieved through more small-scale entrepreneurial activities among youth that participated in Mafita’s programs. These ac- tivities may have been facilitated by increased savings or access to loans. Indeed, the results show that both programs increased the proportion of respondents that contracted a loan in order to start a business and that are planning on future borrowing for business purposes. We also find that both programs increased the proportion of respondents who saved money in the last three months, who used a formal institution to save money (a bank or cooperative society, and who have a bank account. Qualitative evidence suggests that Mafita youth were frustrated with the lack of start-up capital to set up micro-enterprises after the program. As a female COSDEC trainee states: “We need capital and working materials to actualize what we’ve learned and be independent.” Similarly, a male apprentice reported that: “We learned the skills taught to us really well, but we did not have tools and equipment to open our own places of trade.” The extent to which lack of capital constitutes a barrier that prevents Mafita youth from maximizing their full potential is an important subject for future research and experimentation. 70 Increases in income due to Mafita translated into greater household asset ownership and higher consumption expenditures on various essential and non-essential items. Participants in both programs report increases in their household’s ownership of assets such as bank accounts (Apprentice- ship) and generators and cell phones (COSDEC). They also report increased personal weekly expenditure relative to their respective control group. Additionally, monthly expenditure increases were recorded for Apprenticeship participants. While the effects on earnings are large in relative terms, they are moderate in absolute terms. Still, qualitative evidence shows that the increases in earnings, though small, enabled some financial independence which was valued especially by female participants. For example, Safara’u, an Apprenticeship participant, says, “My life has really changed in the sense that I save money now. I can buy for myself hijab, garments, and any other thing I want.” This includes investing in her business: “I get money when- ever I make and sell shoes and bags, from which I keep the profit and go to market to buy more production materials.” Fatima, another Apprenticeship participant, states, “Before joining Mafita, I asked my dad for money...Today, I cater for about 50% of my financial needs. My business really helps me a lot.” Find- ings from the qualitative evaluation suggest that, prior to Mafita, many participants were engaged in very low-productivity trades such as selling toffee/candy or mosquito coils, which many did not consider to be a proper job. For example, Maryam, a COSDEC program participant, used to sell water and juice, but now, after her training in carpentry, she sells chairs and drawers for more income. The study finds evidence suggesting larger impacts for female participants on outcomes re- lated to income-generating activities. For both Apprenticeship and COSDEC programs, improvements in this area appear driven by female participants for whom we measure large increases in self-employment and/or wage employment. For IQE girls specifically, program impacts are striking, with large proportional increases in income from both wage and self-employment, though this is relative to very low levels in the control group. The large impacts observed for IQE girls suggest that Mafita was particularly successful in its goal of improving the economic prospects of this highly vulnerable group. IQE girls are likely to face several constraints to entering wage or self-employment, as reflected in the low levels observed in the control group. Providing home-based Apprenticeships may have been an especially important innovation under the program for this group. Results from the qualitative evaluation corroborate the quantitative evidence that Mafita’s economic impacts were relatively greater for female program participants. Additionally, the qual- itative evaluation highlights how female participants expressed satisfaction at being productively engaged. For example, Khadija from the Apprenticeship program states, “Actually, before now, one sits and watches as the time passes while doing nothing, but now you go to work if you have any [or] to the market.” Another female participant from Kano, Fatima, notes, “Now that the organization took the responsibility to train us, we make things no matter how little, with this you have helped yourself. Our idle lives have changed. You have now got a job.” The quantitative and qualitative components of the study present mixed results on skills de- 71 velopment. The quantitative study does not find any impact on basic foundational literacy and numeracy skills.52 The qualitative evaluation, however, finds participants to have gained technical skills and literacy and numeracy skills and be broadly satisfied with the new skills acquired through the program. Mafita train- ing was rated positively by participants who found that it was useful and had given them more confidence, positivity, and networks, including developing the capacities to communicate in English. Qualitative study participants also note that they have developed new skills including new technical trade skills, have been exposed to new tools and equipment, and have developed newer ways of improving current work including learning theoretical and more technical aspects of their jobs. For example, Musa, a COSDEC participant, says, “I was only given minor jobs at my place of work before joining Mafita such as cushion covers, but after I completed the program my boss now assigns more technical jobs to me like making an entire sofa frame or bed frame.” He adds, “Things have surely changed. Unlike before, I now have a profession.” Safara’u, an Apprentice says, “We were recruited to be trained for a job skill. (...) And all thanks to God, we have learnt how to make handmade shoes and bags, while helping ourselves.” The effects of Mafita’s Apprenticeship and COSDEC programs on non-material outcomes are mixed but a notable exception is the expansion of professional social networks, where both programs show strong positive impacts. Specifically, participants in both Apprenticeship and COSDEC programs reported interacting with more people from other neighborhoods and more employed persons relative to their respective control group. The expanded social networks of trainees may expose them to differing viewpoints and members of other social groups. It may also increase their access to job opportunities (through better knowledge of opportunities or increased referrals, for example). These results are corroborated by the qualitative analysis which finds that participants made new friends and believe they achieved greater social status through Mafita. Participants in both programs also express a change in the nature of interactions with peers from “hanging out” to structuring these around “productive” activities related to work. For example, Hafiza, a female Apprenticeship participant, says, “When we met [with friends] in the past, we usu- ally engaged in less fruitful discussions and gossiping. But when we meet now [since joining the program], the focus of our discussion has changed to how we can improve our skills. We discuss about new styles and how we can make them better. We don’t talk about things that are not useful anymore.” Qualitative research participants also discussed helping each other, both with learning and financially, consistent with the development of professional social networks. We also observe a statistically significant decrease in the importance of religion in daily life among participants We interpret these results as indicative of an expansion in non- religious activities and interactions rather than as a decrease in the importance of religion per se. As such, this result can be related to the observed expansion in professional social networks. This interpretation is supported by the qualitative study. There is strong evidence of improvement in attitudes towards female empowerment for COSDEC participants and their caregivers, but not for Apprenticeship program participants. Among COSDEC participants, we find more favorable views on attitudes towards female empowerment 52 The quantitative study was not designed to measure changes in trade-specific skills. 72 across a range of topics including financial matters, views on domestic violence, and women’s voice and agency, including decisions about their fertility. Effects are driven by female participants. Movement in these indicators is very promising, since studies have shown that some of the most significant roadblocks to adolescent girls acquiring human capital and fully participating in labor markets include childbearing and early marriage. The qualitative study also finds evidence of young women being less reliant on parents and husbands for financial needs after participating in Mafita, although this finding emerged among both COSDEC and Apprenticeship participants. It is noteworthy that caregivers with female wards enrolled in a COSDEC also showed improvements in their attitudes towards female empowerment, compared to caregivers with female wards not enrolled in the program. These caregivers appear to have more progressive views on women’s role in society overall, though results on individual outcome indicators are not statistically dif- ferent from zero. This is a potentially promising finding since caregivers, which mostly comprise parents or guardians, have an important influence on young women’s ability to make key life decisions such as those concerning marriage or childbearing. While tentative, this finding suggests that programs such as Mafita can have positive effects beyond program participants and improve attitudes towards the role of women more generally. This change might also have a financial dimension. For example, in the qualitative study, one caregiver of COSDEC participants explained that parents benefit more financially from having female children enrolled in Mafita. The results do not indicate general improvements in subjective well-being and self-esteem for either program, though we do find small but significant improvements in self-esteem for female Apprenticeship participants and in subjective well-being for COSDEC participants. Specifically, female apprentices show a statistically significant increase in the overall self-esteem index driven by the belief that the future depends on themselves and confidence in their ability to solve problems alone. For COSDEC participants overall, the program improved subjective well-being, driven by improvements in current life satisfaction measured through Cantril’s ladder. The qualitative research corroborates the findings that female participants experienced im- provements in their overall mental well-being. Across both programs, female participants in the qualitative evaluation rated Mafita as very useful, stating that the program had given them more self- confidence (by helping them acquire new skills, in particular) and contributed to a more positive outlook. Female participants also tended to view their Mafita experience (even) more positively than their male counterparts, as they were able to achieve higher productivity and move into traditionally male-dominated trades. One such trade is carpentry, where Maryam, a female COSDEC participant from Katsina expresses that she is now able to make carpentry products that “impress her neighbors, including stools and toys for children.” This suggests that, beyond improving attitudes towards female empowerment, programs such as Mafita can directly expand women’s economic opportunities including through facilitating entry into male- dominated trades. We do not find evidence of impact for either the Apprenticeship or COSDEC programs on violence and other anti-social and risky behavior. The quantitative IE does not find evidence of overall reductions in stealing, taking drugs, or working for criminal groups, nor on the use of violence for 73 political or religious reasons. Because these types of behaviors are often difficult to measure through surveys due to reporting or social desirability bias, we incorporated alternative measures to increase the accuracy of data on these outcomes. These included, for example, the use of audio questions and self-administration, on the tablet, of some sensitive questions to enhance privacy, which allowed study participants to answer without revealing their responses to enumerators. Data from the control groups demonstrates that the type of youth attracted to Mafita do not generally engage in anti-social or risky behaviors. It is therefore not surprising that the Appren- ticeship and COSDEC programs did not have large or consistent impacts in this area. These findings are corroborated by qualitative evidence which reveals that respondents, in general, are not involved in criminal activities. While this was not an explicit dimension targeted by Mafita, training programs for vulnerable youth are often motivated, in part, on their potential to also reduce such behaviors. The results of this study show that, if this is indeed an objective, programs should be targeted towards youth that present greater baseline incidence of these behaviors; it may also be necessary to include program components that specifically target this dimension. At the same time, this result presents an important counterargument to popular narratives which may associate entire groups, such as Almajiri, with crime, violence, and other socially undesirable behaviors. Overall, the limited impacts of the Mafita Apprenticeship and COSDEC programs across several non-material domains could, at least in part, be explained by the fact these dimensions were not explicitly targeted or were only targeted as secondary objectives, without dedicated programmatic components or targeting toward individuals engaged in these behaviors. How- ever, many employment/training programs consider non-material outcomes to be intrinsically important in addition to serving as potential mechanisms through which effects on employment and other economic outcomes may be obtained, deepened, and/or sustained. In Mafita’s case, it is not clear whether the lack of consistent evidence on impacts across the wide-range of non-material outcomes investigated is because such a program is not the right tool to address these other outcomes, whether complementary interventions are needed to successfully address such dimensions, or whether targeting was an issue. Additional rigorous studies are needed to gain further insight into these possibilities. 6.2 Cost-benefit analysis One criticism often raised against skills training programs such as the Mafita Apprenticeship and COSDEC programs is that their relatively high costs compare unfavorably to their mod- erate effects on labor market outcomes (e.g. Blattman and Ralston, 2015). To explore whether this criticism also applies to the Mafita interventions, we compared program costs to the increase in incomes of beneficiaries estimated by the impact evaluation.53 According to information gathered by the implementing agency, the direct costs of training was NGN 100,888 for the apprenticeship intervention and NGN 154,817 for the COSDEC intervention.54 To this, we 53 We follow the literature discussed in Blattman and Ralston (2015) by focusing on income increases as the main measure of economic benefits for this comparison. Of course, our evidence suggests that the interventions led to improvements in other outcomes, such as female empowerment and social networks, which may be valued by policymakers. To keep our analysis as succinct as possible, we do not take these benefits into account here. 54 These costs include the cost of purchasing start-up tools and equipment for each beneficiary, estimated at NGN 70,000. 74 add the indirect opportunity cost of participating in the training, which amounts to the lost income that beneficiaries could have earned had they not participated. We estimate this by multiplying the monthly income of the control group by the duration of the training, twelve months for apprenticeship and nine months for COSDEC. This calculation estimates the indirect cost at NGN 58,390 for apprenticeship and NGN 39,940 for COSDEC. We then compare the total cost (direct + indirect) to the discounted stream of annual benefits, which we calculate by multiplying the estimated effect on total monthly income (from wage employment and self- employment) by twelve. According to this estimate, the annual benefit is NGN 7,472 for apprenticeship and 16,629 for COSDEC. Our baseline scenario assumes that this benefit decreases by 5% per year after the end of training, to reflect the depreciation of acquired skills. We also report results for scenarios that assume depreciation rate of 10% or 0%. We assume that the program cost is incurred in the program’s first year, and the benefit is incurred annually starting in the second year.55 Our baseline scenario assumes a discount rate (rate of time- preference) of 5% but we also report results for discount rates of 3% and 10%. The apprenticeship intervention is only cost-effective under optimistic assumptions. In our baseline scenario with 5% discount rate and 5% skill depreciation, the program has a negative net benefit over time horizons of more than 50 years. The table below shows that this is the case for all scenarios, except the most optimistic one with zero depreciation and a discount rate of 3%. In that scenario, the intervention breaks even after 23 years and has a net benefit of approximately NGN 25,000, a return on investment of over 25%. However, if either the discount rate or the depreciation rate are higher than 3% and 0% respectively, the intervention is not cost-effective even over very long time-horizons. Table 17: Cost-benefit comparison for Apprenticeship intervention Years to break-even Discounted net benefit after 30 years (NGN) 0% skills depreciation 5% skills depreciation 10% skills depreciation 0% skills depreciation 5% skills depreciation 10% skills depreciation Discount rate 3% 23 >50 >50 25263 -35071 -63356 Discount rate 5% 36 >50 >50 -7956 -50004 -70947 Discount rate 10% >50 >50 >50 -55057 -73500 -84126 Results are more positive for the COSDEC intervention. Under the baseline scenario, the intervention breaks even 23 years after the end of training and has a discounted net benefit of NGN 8,500 over a 30-year time-horizon, a return on investment of over 4%. However, the table below shows that the cost-effectiveness of the intervention is sensitive to the assumptions. Under the most optimistic scenario, with zero skill depreciation and a discount rate of 3%, the intervention breaks even after 11 years and has a net benefit of over NGN 176,005 over a 30-year time horizon, with a return investment of over 90%. However, under the less optimistic scenarios, with skill depreciation or discount rates of 10%, the intervention is not cost-effective even over very long time horizons. They also include the costs of renovating the COSDEC centers which we assumed to amortize over 30 cohorts of beneficiaries. The figures exclude the stipend paid to the beneficiaries, since this was a pure transfer. In other words, the recipient accrues the stipend as a benefit at the same time that the program incurs it as a cost, so that it cancels out of any cost-benefit comparison. 55 The estimated benefits are based on intent-to-treat effects, meaning average effects per individual randomized into the treatment group. Based on enrollment data, we estimate that only approximately 75% of individuals in the treatment group enrolled in the training. To make costs and benefits comparable, we thus assume that program costs were only incurred for 75% of individuals. Results are very similar when we compare the full program cost to estimated benefits based on the Local Average Treatment Effect, meaning the average effect per individual who was actually trained. 75 Table 18: Cost-benefit comparison for COSDECs intervention Years to break-even Discounted net benefit after 30 years (NGN) 0% skills depreciation 5% skills depreciation 10% skills depreciation 0% skills depreciation 5% skills depreciation 10% skills depreciation Discount rate 3% 11 16 >50 176005 41732 -21213 Discount rate 5% 13 23 >50 102007 8500 -38106 Discount rate 10% 36 >50 >50 -2746 -43788 -67437 In sum, the COSDEC program appears to have been more cost-effective than the Appren- ticeship. We stress that these results are indicative and based on very limited data. If confirmed, however, these results could provide Mafita with an opportunity to reshape the Apprenticeship and COSDEC pro- gram more cost-effectively. The results of the cost-benefit analysis highlight the importance of measuring the effects of different program components. Identifying components that had little benefit for participants would help reduce the program’s cost without compromising its effectiveness. 7 Conclusions, policy recommendations, and lessons learned 7.1 Conclusions and policy recommendations Mafita sought to address several challenges related to generating employment for some of the poorest and most vulnerable populations in the context of Northern Nigeria. As originally conceived, the program also aimed to reduce extremist views and these groups’ propensity to participate in violence and crime. In this evaluation, we study the impact of the program on outcomes related to the labor market and on social, political, religious, and anti-social attitudes and behaviors. We also examine the program’s impact on secondary outcomes that were not directly targeted by the program but were plausibly affected by it, including subjective well-being and self-esteem. Mafita’s Apprenticeship and COSDEC programs were designed to address some, but not all, constraints faced by marginalized youth in northern Nigeria in seeking employment and income-earning opportunities and in moving into more productive activities. Systemic issues including, for example, unreliable electricity networks, macroeconomic policies that limit foreign direct in- vestment, stigma and discrimination, and conflict and violence all contribute to an unfavorable labor market for the youth populations targeted by Mafita. It is important to consider this context to appreciate Mafita’s considerable achievements and its natural limits. With this in mind, below we highlight key findings and propose several recommendations for future programming and learning. It should be noted that the IEs were not designed to directly compare the effectiveness of the two Mafita programs. Such a rigorous comparison would have required randomly assigning Mafita participants to either the Apprenticeship program or COSDEC program, ideally with another randomly assigned group that received neither program. This was not possible, however, due to geographic and opera- tional constraints (the Apprenticeship program was implemented in Kano state, while the COSDEC program was implemented in Kaduna and Katsina states). We therefore cannot conclude with certainty whether any observed differences in impact between the two programs are due to differences between the interventions or differences in the target populations, local economic conditions, or other contextual factors. Overall, the results suggest that Apprenticeship and COSDEC-type programs can be used to promote self-and wage employment and increase earnings for vulnerable youth in fragile 76 and conflict-prone settings. This contrasts with much of the existing rigorous empirical evidence on skills training programs which shows limited effects. Mafita, however, incorporates several design features that previous research has highlighted as potentially important for the success of such programs. These include, for example, targeting more marginalized youth groups such as Almajiri boys or IQE girls; skills upgrading for trainers and master crafts-persons to increase the quality of training; provision of stipends (or wage sub- sidies) to replace loss of income from other sources; inclusion of programs that are complementary to skills training, such as entrepreneurship training/support or promoting access to finance; and the provision of tools. Policy recommendation #1: Apprenticeship and COSDEC-type programs can be used to promote self- and wage employment and increase earnings for vulnerable youth in fragile and conflict-prone settings. Such programs should build on this and other rigorous evaluations of skills and employment programs to incorporate design features that have been shown or that are highly likely to contribute to positive impacts. The results suggest that returns to self-employment are generally higher for the type of youth targeted by Mafita, suggesting that there are additional constraints to wage employ- ment for this population. Some of these constraints are structural (e.g., limited demand for labor) and are outside of the scope of interventions such as Mafita’s Apprenticeship and COSDEC programs. Others, however, relate to areas which Mafita could possibly have addressed more effectively, and which may limit youth’s prospects in the market for wage labor. To design more effective programs, further evidence is needed on the specific impacts of key program components, which would provide important insight on the channels through which programs like Mafita produce their impacts. It should be noted that this study, as agreed with FCDO and ASI from the outset, was not designed to identify the marginal contribution of different elements of the Mafita program package. This is an important limitation in the study’s ability to estimate marginal effects or contributions of specific components on the key outcomes. Policy recommendation #2: Future skills training and jobs programs should incor- porate rigorous evaluations which are designed to explicitly focus on key components as a complement to overall program impact. Such evaluations should focus both on the design of these components and on their value-added in achieving program outcomes. The Mafita IEs suggest foundational skills training to be one area that would benefit from further investigation. In particular, low levels of basic literacy and numeracy among Mafita’s target group and the lack of evidence of impact in these areas raises two important questions. First, to what extent do low levels of these foundational skills keep some youth from entering into higher-productivity activities and wage labor? Second, how can future programs be designed to more effectively equip youth with basic foundational skills? Answers to both issues are critical for our understanding of the role of foundational skills in active labor market interventions such as the Mafita programs. Our findings also suggest that access to capital may be a key constraint to youth employ- ment and earnings. Indeed, in a multi-study review of employment programs in developing countries, Blattman and Ralston (2015) find that capital-centric programs show the most promise in generating em- ployment and increasing incomes for the poor, especially in post-crisis and other settings where lack of capital might be a binding constraint that forces firms or entrepreneurs to operate below their optimal size. 77 Start-up capital to promote entrepreneurship was initially considered as part of Mafita but, ultimately, this was decided against for sustainability reasons. Policy recommendation #3: Program components recommended as focus areas for future rigorous evaluations include foundational skills training, entrepreneurship train- ing, and access to finance. While impacts on income are large on a relative scale, they are modest in absolute terms. A limited cost-benefit exercise raises concerns over the cost-effectiveness of both the Apprenticeship and COSDEC programs. This finds that the COSDEC program breaks even after 19 years and has a return on investment of about 10% after 30 years. However, this conclusion is sensitive to assumptions about the discount rate of skill depreciation.56 The Apprenticeship program breaks even only under optimistic assumptions and has a substantially lower discounted net benefit. Policy recommendation #4: Future programs that build on Mafita must carefully consider cost drivers to improve the cost:benefit ratio. Program delivery and scale- up through government systems may reduce some costs, though this may also affect program quality and effectiveness. Additional research on the impact of such programs delivered through government systems in fragile and conflict settings appears important. Management information systems (MIS) should be strengthened to record high quality implementation and cost data on program activities, which is needed to more accurately estimate costs and benefits of the program as a whole and of specific components. Findings from the Mafita IEs are particularly relevant for other projects, programs and policies that aim to promote gender equality and women’s empowerment, especially in fragile settings. Our results show that both Mafita programs had strong impacts on female participants’ economic outcomes, especially for IQE girls, not only raising their employment but also increasing their earning po- tential by almost double or more. Indeed, Mafita was particularly successful in improving the economic prospects of this highly vulnerable group, even if this was from a low baseline. There is also evidence, from the COSDEC program, that women’s participation can improve attitudes towards female empowerment by participants and their caregivers, a striking result given strong gender norms in northern Nigeria, though more evidence is needed to better understand how these effects are achieved. Policy recommendation #5: Future programs should leverage the potential to pro- mote gender equality and women’s empowerment through positive changes in gender- related attitudes, behaviors, and norms. Further experimental research would be critical to ascertain the sustainability, deepening, and expansion of gender impacts from labor market programs such as Mafita, and crucially, to investigate potential channels of influence. The positive effect on participants’ professional social networks suggests the programs such as Mafita can be used to promote the social and economic inclusion of marginalized groups. The Apprenticeship and COSDEC programs led to expanded professional networks, meaning that program 56 The baseline scenario in the cost-benefit analysis sets both the discount rate and skills depreciation at 5% 78 participants were more likely to interact with people who were employed and/or living outside of their neigh- borhood. The expanded social networks of trainees may expose them to differing viewpoints and members of other social groups. It may also increase their access to job opportunities (through better knowledge or increased referrals, for example). Understanding the spread of information and informal support structures created through such networks could be a promising area for future programming and research to deepen and sustain Mafita’s impacts. Policy recommendation #6: Future programs should put in place incentive struc- tures that encourage and promote new social and professional networks that may in- crease participants’ access to job opportunities and enhance their employability. This is also a promising area for future research. Overall mixed results on non-material outcomes suggest that, if such outcomes are an objective, training and employment programs need to be complemented with specialized in- terventions that target them more directly. Such non-material outcomes are often considered to be important in their own right and as potential mechanisms/channels to achieve, deepen, and sustain economic outcomes such as employment and income. The evidence from this and other studies suggests, however, that impacts on economic outcomes do not always or automatically translate into impacts in other areas, such as reductions in violence, crime, or other anti-social or risky behaviors.57 This suggests that, if such outcomes are indeed an objective, they need to be more explicitly factored into program design, either through tar- geting or the inclusion of complementary interventions specifically targeting these outcomes. Policy recommendation #7: To the extent that non-material dimensions and out- comes - such as psychological well-being and self-esteem or reduced participation in violence and anti-social behavior - are program objectives, these should be explicitly incorporated in program design including, potentially, through complementary inter- ventions that are not focused explicitly at income-generating activities. Alternatively, programs aiming to shift these outcomes by boosting income should explicitly target income generating activities toward youth who are at risk along these dimensions, or engaged in criminal and violent activities. 7.2 Lessons for future program implementation and learning We conclude this report with a brief discussion on practical lessons learned for future col- laborative efforts on rigorous, prospective impact evaluations embedded into program design. We emphasize the importance of: (i) an engaged participatory process that ensures buy-in from all the stakeholders and maintains close collaboration throughout the evaluation process; (ii) regular communica- tion protocols; and (iii) a robust management information system (MIS) to carefully monitor and document program activities and record administrative data essential to understanding program implementation. 57 The study shows that participation in anti-social behaviors by youth attracted to Mafita was very low to begin with. 79 Policy recommendation #8: Future collaborative efforts to embed rigorous evalu- ations into program design should incorporate (i) a participatory program/evaluation design and management process for buy-in from all stakeholders and to maintain a close collaboration throughout the evaluation process; (ii) regular communications pro- tocols; and (iii) a robust management information system (MIS) to carefully monitor and document program activities and record administrative data needed to understand implementation. To maximize knowledge generation and learning from prospective impact evaluations such as the Mafita IEs, it is critical that the evaluation be designed early and through a partic- ipatory process that ensures buy-in from all stakeholders and maintains close collaboration throughout the evaluation process. As noted earlier in this report, the initial evaluation design came out of a two-day workshop convened in Abuja with key stakeholders (DFID/FCDO, ASI, representatives from the three focal states targeted by Mafita, and the World Bank/DIME) during Mafita’s preparation phase. The workshop’s objectives were to identify and agree on evaluation priorities and key elements of the impact evaluation design. This early start allowed us to embed a rigorous impact evaluation into program activities that would not have been possible otherwise. The successful completion of the Mafita IEs is due, in large part, to the participatory nature of the evaluation process, whereby all the stakeholders were actively and collectively involved in decision-making from the onset and throughout the entire evaluation. Communication protocols, documentation, and institutional memory are vital to ensuring multi-year collaborative evaluation partnerships remain on track through minimizing infor- mation loss and ensuring continuity in evaluation plans. Successful implementation of prospective impact evaluations such as the Mafita IEs depends critically on collaboration between implementing agencies, donors, and the evaluation team, whereby all stakeholders are involved from the outset and regularly review evaluation and learning objectives throughout the duration of program implementation and the evaluation process. Clear communication protocols between the evaluation team and all key stakeholders should be established at the outset and adhered to by all partners, both to ensure that activities remain on track and to preserve institutional memory in case of staff turnover, which is to be expected over a multi-year multi-partner project. Such clear communication protocols are also important to ensure realistic expecta- tions regarding program benefits to participants. They may also be important in increasing both take-up and graduation rates among eligible youth. It is important to ensure proper identification of program participants. One issue identified during the implementation of this study (and also, in parallel, by Mafita program staff) was some instances of replacement/impersonation of youth officially enrolled in the program, meaning that the program was not always delivered to those persons that were enrolled. This is a serious challenge to both the program and the impact evaluation. Solutions to this problem range from introducing stricter controls for partici- pant identification through simple and low-cost (e.g., issuing photo ID cards to confirmed participants) to technology-enabled solutions such as biometric identification (e.g., fingerprint scanners). The latter would also have the additional important benefit of providing an automated source of data on program participation. A robust management information system (MIS) to facilitate program implementation and monitoring and provide data to conduct process, impact, and cost-effectiveness analyses is 80 essential. This should include unique identification codes to identify participants and facilitate the tracking of attendance in specific activities and in the program overall. The systematic use of such codes would also facilitate the comparison of information across different documents and databases.58 58 An important challenge for the impact evaluation described in this report was to merge data from program enrollment lists, baseline surveys, and from the final survey without a unique code which unambiguously identifies each Mafita program participant. While other forms of identification, e.g. first and last name, may appear sufficient, often this is not the case in practice. For example, participants may use different names, or inconsistent spellings may be used, or participants may share the same or very similar names, all of which complicate the process of tracking participants and combining data for analysis from multiple sources. 81 8 References Adoho, F., S. Chakravarty, D. T. Korkoyah Jr., M. Lundberg and A. Tasneem. 2014. The Impact of an Adolescent Girls Employment Program: The EPAG Project in Liberia. World Bank Policy Research Work- ing Paper 6832. Alfonsi, L., Bandiera, O., Bassi, V., Burgess, R., Rasul, I., Sulaiman, M. and Vitali, A. (2020), Tackling Youth Unemployment: Evidence From a Labor Market Experiment in Uganda. Econometrica, 88: 2369- 2414. Anderson, M. L. 2008. Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects. Journal of the American statistical Association, 103(484), 1481-1495. Attanasio, O., A. Kugler and C. Meghir. 2011. Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial. American Economic Journal: Applied Economics 3: 188- 220. Benjamini, Y., A. M. Krieger and D. Yekutieli. 2006. Adaptive Linear Step-up Procedures that Control the False Discovery rate. Biometrika 93: 491-507. Betcherman, G., K Olivas and A Dar. 2004. Impacts of Active Labor Market Programs: New evidence from Evaluations with Particular Attention to Developing and Transition countries. World Bank Social Protection Discussion Paper 0402. Blattman, C., & Annan, J. 2016. Can employment reduce lawlessness and rebellion? A field experiment with high-risk men in a fragile state. American Political Science Review, 110(1), 1-17. Blattman, C., & Ralston, L. (2015). Generating employment in poor and fragile states: Evidence from labor market and entrepreneurship programs. Available at SSRN 2622220. Card, D., P. Ibarran, F. Regalia, D. Rosas-Shady and Y. Soares. 2011. The Labor Market Impacts of Youth Training in the Dominican Republic. Journal of Labor Economics 29: 267-300. Cho, Y., Kalomba, D., Mobarak, A. M., & Orozco, V. 2013. Gender differences in the effects of vocational training: Constraints on women and drop-out behavior. The World Bank. epon, B., & Premand, P. 2019. Direct and Indirect Effects of Subsidized Dual Apprenticeships. IZA Cr´ Discussion Paper No. 12793. Available at SSRN 3495770 Crost, B., Dube, O., Holmlund, M., and E. Mvukiyehe. 2019. A Way Out? Apprenticeship Training, Employment and Social Transformation in Northern Nigeria. AEA RCT Registry. August 02. Available at https://doi.org/10.1257/rct.4500-1.0. 82 Crost, B., Dube, O., Holmlund, M., and E. Mvukiyehe. 2019. Community-based Skills Training, Em- ployment and Social Transformation: Evidence from a Field Experiment in Northern Nigeria. AEA RCT Registry. September 06. Available at https://doi.org/10.1257/rct.4637-1.0. ˆ (No. 436). Com- Fetzer, T. 2019. Can Workfare Programs Moderate Conflict? Evidence from IndiaA petitive Advantage in the Global Economy (CAGE). Field, E. M., Pande, R., Rigol, N., Schaner, S. G., & Moore, C. T. 2019. On Her Own Account: How Strengthening Women’s Financial Control Affects Labor Supply and Gender Norms (No. w26294). National Bureau of Economic Research. Groh, M., Krishnan, N., McKenzie, D., & Vishwanath, T. 2016. The impact of soft skills training on female youth employment: evidence from a randomized experiment in Jordan. IZA Journal of Labor & Development, 5(1), 9. Hardy, M. and J. McCasland. 2015. Are Small Firms Labor Constrained: Experimental Evidence from Ghana. Working Paper. Hardy, M. L., Mbiti, I. M., Mccasland, J. L., & Salcher, I. 2019. The Apprenticeship-to-Work Transition: Experimental Evidence from Ghana. The World Bank. Herbert, S. & Husaini, S. 2018. Conflict, instability and resilience in Nigeria. Rapid Literature Review 1427. Birmingham, UK: GSDRC, University of Birmingham. Hirshleifer, S., D. McKenzie, R. Almeida and C. Ridao-Cano. Forthcoming. The Impact of Vocational Training for the Unemployed: Experimental Evidence from Turkey. Forthcoming in The Economic Journal. Kling, J. R., J. B. Liebman and L. F. Katz. 2007. Experimental Analysis of Neighborhood Effects. Econometrica 75: 83-119. Kluve, J., S. Puerto, D. Robalino, J. M. Romero, F. Rother, J. Stoterau, F. Weidenkaff and M. Witte. 2016. Do Youth Employment Programs Improve Labor Market Outcomes: A Systematic Review. IZA Dis- cussion Paper No. 10263. Lee, D. S. Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects, The a1102, https://doi.org/10.1111/j.1467- Review of Economic Studies, Volume 76, Issue 3, July 2009, Pages 1071ˆ 937X.2009.00536. Lyall, J., Zhou, Y. Y., & Imai, K. 2020. Can economic assistance shape combatant support in wartime? Experimental evidence from Afghanistan. American Political Science Review, 114(1), 126-143. Maitra, P. and S. Mani. 2014. Learning and Earning: Evidence from a Randomized Evaluation in India. IZA Discussion Paper No. 8552. 83 Matfess, H. 2019. The New Normal: Continuity and Boko Haram’s Violence in North East Nigeria. https://www.acleddata.com/2019/02/11/the-new-normal-continuity-and-boko-harams-violence-in-north-east McKenzie, D. 2017. How effective are active labor market policies in developing countries? A critical review of recent evidence. The World Bank Research Observer, 32(2), 127-154. Ministry of Budget & National Planning. 2017. Economic Recovery and Growth Plan 2017-2020: Federal Republic of Nigeria. p. 92. Rustad, S. A., & Ostby, G. 2017. Education and systematic group inequalities in Nigeria. Oslo: Peace Research Institute Oslo. Uwezo assessment training manual http://www.uwezo.net/assessment/training/ Wager, S., & Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523), 1228-1242. The World Bank (2019). Advancing Social Protection in a Dynamic Nigeria. ASA P165426. (formerly “Adapting Social Protection in a Changing Nigeria”) Main Report http://documents.worldbank.org/ curated/en/612461580272758131/pdf/Advancing-Social-Protection-in-a-Dynamic-Nigeria.pdf. Ac- cessed on April 1, 2020 84 A Appendix A.1 Balance Tests Balance tests are used to assess whether the randomization process was successful, that is, whether the resulting treatment and control groups were statistically identical prior to the start of Mafita. Using the database of eligible participants included in the randomization process and Mafita’s own baseline data, we test for statistically significant differences between control and treatment on a subset of pre-treatment covariate indicators. To do so we estimate the following equation: Yi = β0 + β1 ∗ T reatmenti + Xi + i (A1) where Yi denotes a pre-treatment covariate for each study participant collected either before the ran- domization procedure or at baseline, T reatmenti denotes whether they were assigned to the treatment or control group, Xi is a fixed effect which accounts for the randomization strata, and i is the individual error term. Table A1, A2, A3 and A4 present, for each indicator included in the balance test, the control group mean, the coefficient for the difference between treatment and control groups (and, in parentheses below this, the standard error of this estimate), and the p-value indicating whether the estimated difference is statistically significant at standard levels. A.1.1 Apprenticeship Table A1 presents the results of the balance tests for the Apprenticeship program using data from the pre-randomization database. No statistical difference is found between the treatment and control groups. Table A1: Apprenticeship IE balance test using pre-randomization data Control Mean T-C P < |t| N Age 19.065 0.048 0.446 5145 (0.063) Orphan Vulnerable Children 0.528 -0.009 0.291 5165 (0.009) Early School Leaver 0.126 0.008 0.332 5165 (0.008) Person with Disability 0.006 0.000 0.856 5165 (0.002) First choice of trade Carpentry and Joinery 0.022 0.008 0.104 5165 (0.005) Electrical Installation 0.050 -0.005 0.390 5165 (0.006) GSM Repairs 0.044 0.004 0.497 5165 (0.006) Masonry 0.012 -0.003 0.320 5165 Continued on next page 85 (0.003) Refrigerator repair 0.022 0.000 0.904 5165 (0.003) Satelite Installation 0.024 0.004 0.436 5165 (0.005) Welding and Fabrication 0.019 0.002 0.710 5165 (0.004) Second choice of trade Carpentry and Joinery 0.032 -0.000 0.965 5165 (0.005) Electrical Installation 0.061 -0.001 0.912 5165 (0.007) GSM Repairs 0.089 -0.004 0.607 5165 (0.008) Masonry 0.012 -0.001 0.713 5165 (0.003) Refrigerator repair 0.044 -0.000 0.980 5165 (0.005) Satelite Installation 0.040 0.007 0.250 5165 (0.006) Welding and Fabrication 0.022 -0.000 0.972 5165 (0.004) Notes : All specification include strata fixed effect and robust standard errors. Column 1 displays the control group mean, Column 2 shows the difference between treatment and control group, Column 3 displays the p-value for the coefficient on the treatment indicator. *: Significant at 10%; **: 5%; ***: 1% Table A2 presents the results of the balance tests for the Apprenticeship program using baseline data. In terms of statistically significant differences, we find that persons assigned to the treatment group are less likely to have performed some forms of skilled work in the past seven days, to report having previously been taught skills, to be currently working as an apprentice, or to interact with Muslims. As we see statistically significant differences on a small set of variables only and as, in all cases, the absolute differences in proportions are small, we conclude that the randomization process was successful in creating balanced treatment and control groups. To ensure that our results are robust to pre-program differences, we run a specification including baseline control variables and find that this does not alter our findings. Table A2: Apprenticeship IE balance test using Mafita baseline data Control Mean T-C P < |t | N Completed primary school 0.903 0.013 0.157 4023 (0.009) No. of parents still alive 1.604 -0.004 0.822 4023 (0.019) Continued on next page 86 No. of times slept hungry (last 7 days) 0.714 -0.029 0.488 4023 (0.041) Sleep outside (y/n) 0.015 0.002 0.698 4023 (0.004) Hours of working to earn food (last 7 days) 1.772 0.324 0.103 4023 (0.199) Hours of participating in an apprenticeship (last 7 days) 1.340 0.184 0.421 4023 (0.228) Hours of articipating in other skills building program (last 7 days) 0.315 0.111 0.287 4023 (0.105) Hours of studying religious texts (last 7 days) 13.251 0.245 0.543 4023 (0.404) Hours of attending a place of worship (last 7 days) 6.611 -0.416 0.131 4023 (0.276) Hours of socializing (last 7 days) 10.140 -0.187 0.523 4023 (0.292) Proportion hawking to earn money (last 7 days) 0.086 0.001 0.876 4023 (0.009) Proportion hustling to earn money (last 7 days) 0.136 -0.002 0.851 4023 (0.011) Proportion working as house help to earn money (last 7 days) 0.059 -0.000 0.969 4023 (0.008) Proportion teaching Quranic school to earn money (last 7 days) 0.044 -0.003 0.662 4023 (0.007) Proportion performing other skilled work (last 7 days) 0.132 -0.018* 0.094 4023 (0.011) Has been taught a trade (proportion) 0.169 -0.007 0.555 4023 (0.012) Has been taught skills (proportion) 0.104 -0.020** 0.037 4023 (0.010) Currently looking for a job (proportion) 0.407 0.011 0.504 4023 (0.016) Currently working (proportion) 0.025 0.004 0.480 4023 (0.006) Currently an apprentice (proportion) 0.112 -0.025** 0.014 4023 (0.010) Has applied for a loan proportion) 0.015 0.001 0.818 4023 (0.004) Has male friends (proportion) 0.793 0.013 0.238 4023 (0.011) Has female friends (proportion) 0.676 0.008 0.530 4023 (0.013) Has Christian friends (proportion) 0.107 -0.011 0.261 4023 (0.010) Has Muslim friends (proportion) 0.991 -0.000 0.933 4023 Continued on next page 87 (0.003) Interacts with males (proportion) 0.823 -0.010 0.358 4023 (0.011) Interacts with females (proportion) 0.698 0.008 0.548 4023 (0.013) Interacts with Muslims (proportion) 0.986 -0.009** 0.039 4023 (0.004) Interacts with Christians (proportion) 0.158 -0.002 0.894 4023 (0.012) Agrees youth unemployment is source of depression (proportion) 0.937 -0.002 0.852 4023 (0.008) Agrees violence is justified against injustice (proportion) 0.130 0.006 0.612 4023 (0.011) Agrees government will change if people protest peacefully (proportion) 0.731 0.009 0.509 4023 (0.014) Agrees violence is justified to resolve problems (proportion) 0.089 0.013 0.184 4023 (0.010) Agrees walking away from a fight is cowardice (proportion) 0.149 -0.002 0.843 4023 (0.012) Agrees it is ok to hit someone back (proportion) 0.178 -0.006 0.652 4023 (0.012) Felt discriminated 0.108 0.015 0.157 4023 (0.011) Agrees women should have own opinions (proportion) 0.437 -0.014 0.407 4023 (0.016) Agrees woman should have the right to decide over HH money use (proportion) 0.730 -0.017 0.234 4023 (0.015) Notes : All specification include strata fixed effect and robust standard errors. Column 1 displays the control group mean, Column 2 shows the difference between treatment and control group, Column 3 displays the p-value for the coefficient on the treatment indicator. *: Significant at 10%; **: 5%; ***: 1% A.1.2 COSDEC Table A3 presents the results of the balance tests for the COSDEC program using data from the pre- randomization database. No statistical difference is found between the treatment and control groups. Table A3: COSDECs IE balance test using Mafita pre-randomization data Control Mean T-C P < |t| N Age 19.139 -0.047 0.688 1810 (0.117) Youth category: orphan or vulnerable child (proportion) 0.271 -0.024 0.232 1824 (0.020) Early school leaver (proportion) 0.225 -0.013 0.480 1824 (0.019) Youth category: person with disability (proportion) 0.050 -0.007 0.496 1824 Continued on next page 88 (0.010) Notes : All specification include strata fixed effect and robust standard errors. Column 1 displays the control group mean, Column 2 shows the difference between treatment and control group, Column 3 displays the p-value for the coefficient on the treatment indicator. *: Significant at 10%; **: 5%; ***: 1% Table A4 presents the results of the balance tests for the COSDEC program using baseline data. In terms of statistically significant differences, we find that persons assigned to the treatment group report spending more time studying religious texts, are less likely to agree that graduates are likely to find jobs, and are more likely to have worked. As we see statistically significant differences on a small set of variables only and as the absolute differences appear small (with the exception of the proportion that have previously worked, where the treatment group proportion is 4 percentage points greater than the control group mean of 9%), we conclude that the randomization process was successful in creating balanced treatment and control groups. To ensure that our results are robust to pre-program differences, we run a specification including baseline control variables and find that this does not alter our findings. Table A4: COSDEC IE balance test using Mafita baseline data Control Mean T-C P < |t| N Hours of studying religious texts (in normal day) 3.982 0.274** 0.045 1555 (0.137) Has been taught a trade (proportion) 0.158 0.026 0.177 1555 (0.019) Has attended a skills training (proportion) 0.135 -0.011 0.523 1555 (0.017) Currently working (proportion) 0.123 -0.003 0.832 1555 (0.016) Looking for a job (proportion) 0.652 0.005 0.830 1555 (0.024) Currently an apprentice (proportion) 0.089 -0.006 0.682 1555 (0.014) Has applied for a loan (proportion) 0.012 0.003 0.612 1555 (0.006) Agrees most graduates find jobs (proportion) 2.557 -0.066* 0.081 1555 (0.038) Agrees relationship between gov’t and the unemployed is good 1.726 -0.002 0.953 1555 (0.035) Agrees job opportunities available for youth 2.000 0.002 0.960 1555 (0.040) Has felt discriminated against (proportion) 0.259 -0.002 0.925 1555 (0.022) Has felt discriminated against (proportion) 0.143 -0.018 0.274 1555 (0.017) Participated in a protest in last year (proportion) 0.026 -0.005 0.498 1555 (0.008) Continued on next page 89 Agrees unemployed are cared for by government (proportion) 0.976 0.021 0.558 1555 (0.036) Agrees gov’t actions are responsible for economic problems 1.417 0.007 0.866 1555 (0.042) Has been arrested (proportion) 0.014 -0.007 0.163 1555 (0.005) No. of meals in a normal day 2.479 -0.006 0.860 1555 (0.034) Married (proportion) 0.140 0.003 0.849 1555 (0.017) Has worked (proportion) 0.086 0.038** 0.010 1555 (0.015) Notes : All specification include strata fixed effect and robust standard errors. Column 1 displays the control group mean, Column 2 shows the difference between treatment and control group, Column 3 displays the p-value for the coefficient on the treatment indicator. *: Significant at 10%; **: 5%; ***: 1% 90 A.2 Attrition from the study Attrition refers to the loss of data from the study sample because individuals cannot be surveyed. The rate and reasons for this may vary across the treatment and control groups, and this may have implications for the results drawn from the quantitative analysis. To assess this, we run a series of tests to quantify whether being assigned treatment group impacts the likelihood that an individual does not participate in the endline survey. For this analysis, we define a “dropout” as a person who is included in the randomization (i.e., is assigned to either the treatment or the control group) but whose data is not captured in subsequent surveys.59 We test whether being assigned to the treatment group increases or decreases the likelihood of dropping out. We measure attrition at two stages of the project, after random assignment (baseline survey) and after program completion during endline data collection. We run three types of test to capture the potential effect of being assigned to the treatment group on dropout and attempt to identify attrition predictors. Equation A2 estimates the effect of treatment assign- ment on the likelihood of dropping out, controlling for strata fixed effects Xi . Equation A3 estimates the effect of treatment assignment on the likelihood of dropping out, controlling also for pre-treatment covariates Zi . Finally, equation A4 includes interaction terms between pre-treatment covariates and treatment status to estimate heterogeneous effects of treatment assignment on attrition by baseline characteristics). The results of these analyses are in Table A5, A6, A7, and A8. Attritioni = β0 + β1 ∗ T reatmenti + Xi + i (A2) Attritioni = β0 + β1 ∗ T reatmenti + β2 ∗ Zi + Xi + i (A3) Attritioni = β0 + β1 ∗ T reatmenti + β2 ∗ Zi + β 3 ∗ T reatment ∗ Zi + Xi + i (A4) Additionally, to correct for possible unbalanced attrition bias, we use Lee bounds to adjust for attrition and report the results of the main index. As described in more detail below, this method consists of trimming the tails of the sample proportionally to equalize the proportion of the lost sample in the treatment and control groups. A.2.1 Apprenticeship As shown in tables A5 and A6, we observe differential attrition between the treatment and control groups. However, this is balanced when accounting for the interaction of treatment with observable characteristics from Mafita pre-randomization administrative data and the program’s baseline dataset. Table A5: Apprenticeship IE attrition test using pre-randomization Mafita administrative data (1) (2) (3) Treatment -0.061 -0.061*** 0.125 (0.010) (0.010) (0.091) Age 0.001 0.007* (0.002) (0.004) Continued on next page 59 In other words, for the purpose of this analysis “dropout” has nothing to do with actual participation in the program. 91 Age X Treatment -0.009** (0.005) IQE Girls -0.005 0.008 (0.015) (0.022) IQE Girls X Treatment -0.021 (0.025) Observations 5165 5145 5145 Notes : Column 1 provides the estimated differen- tial attrition for the treatment group controlling for randomization strata fixed effects. Column 2 includes additional control variables. Column 3 tests for differential attrition by each control vari- able. *: Significant at 10%; **: 5%; ***: 1% Table A6: Attrition test using baseline data for Apprenticeship program (1) (2) (3) Treatment -0.057*** -0.058*** -0.023 (0.011) (0.010) (0.095) Completed primary school 0.004 0.007* (0.017) (0.030) Completed primary school X Treatment -0.003** (0.036) No. of parents still alive -0.010* -0.022 (0.009) (0.016) No. of parents still alive X Treatment 0.006 (0.019) No. of times slept hungry (last 7 days) -0.001*** -0.004** (0.004) (0.007) No. of times slept hungry (last 7 days) X Treatment 0.001* (0.009) Sleep outside (y/n) 0.007 0.019 (0.044) (0.082) Sleep outside (y/n) X Treatment -0.032 (0.095) Hours of working to earn food (last 7 days) 0.001 0.002 (0.001) (0.002) Hours of working to earn food (last 7 days) X Treatment -0.002 (0.002) Hours of participating in an apprenticeship (last 7 days) 0.000 -0.001 (0.001) (0.001) Hours of participating in an apprenticeship (last 7 days) X Treatment 0.001 (0.002) Hours of articipating in other skills building program (last 7 days) 0.001 0.002 (0.002) (0.003) Continued on next page 92 Hours of articipating in other skills building program (last 7 days) X Treatment -0.002 (0.004) Hours of studying religious texts (last 7 days) 0.001 -0.001 (0.000) (0.001) Hours of studying religious texts (last 7 days) X Treatment 0.001 (0.001) Hours of attending a place of worship (last 7 days) 0.000 0.001 (0.001) (0.001) Hours of attending a place of worship (last 7 days) X Treatment -0.002 (0.001) Hours of socializing (last 7 days) -0.000 0.000 (0.001) (0.001) Hours of socializing (last 7 days) X Treatment 0.000 (0.001) Proportion hawking to earn money (last 7 days) 0.014 0.040 (0.019) (0.036) Proportion hawking to earn money (last 7 days) X Treatment -0.044 (0.042) Proportion hustling to earn money (last 7 days) 0.014 0.009 (0.016) (0.028) Proportion hustling to earn money (last 7 days) X Treatment 0.004 (0.033) Proportion working as house help to earn money (last 7 days) 0.030 0.003 (0.025) (0.046) Proportion working as house help to earn money (last 7 days) X Treatment 0.013 (0.055) Proportion teaching Quranic school to earn money (last 7 days) -0.036* -0.010 (0.022) (0.043) Proportion teaching Quranic school to earn money (last 7 days) X Treatment -0.048 (0.049) Proportion performing other skilled work (last 7 days) -0.008 -0.032 (0.016) (0.027) Proportion performing other skilled work (last 7 days) X Treatment 0.038 (0.033) Has been taught a trade (proportion) -0.006 -0.005 (0.015) (0.024) Has been taught a trade (proportion) X Treatment 0.009 (0.032) Has been taught skills (proportion) -0.009 0.010 (0.019) (0.031) Has been taught skills (proportion) X Treatment -0.029 (0.039) Currently looking for a job (proportion) 0.016 -0.009 (0.011) (0.019) Currently looking for a job (proportion) X Treatment 0.034 Continued on next page 93 (0.023) Currently working (proportion) -0.033 -0.011 (0.029) (0.059) Currently working (proportion) X Treatment -0.038 (0.067) Observations 4023 4023 4023 Notes : Column 1 provides the estimated differential attrition for the treatment group controlling for random- ization strata fixed effects. Column 2 includes additional control variables. Column 3 tests for differential attrition by each control variable. *: Significant at 10%; **: 5%; ***: 1% A.2.2 COSDECs As shown in Table A7, we observe differential attrition between the treatment and control groups. However, this is balanced when accounting for the interaction of treatment with observable characteristics from Mafita pre-randomization administrative data. When performing the analysis using baseline data (Table A8), we find that persons in the treatment group who (i) report having previously been taught a trade are 9 percentage points more likely to attrit, (ii) report having attended a previous skills training are 5 percentage points more likely to attrit, and (iii) report currently working are 7 percentage points more likely to attrit. Table A7: COSDECS IE attrition test using pre-randomization Mafita administrative data (1) (2) (3) Treatment -0.034 -0.033*** -0.111 (0.011) (0.011) (0.085) Age 0.002 -0.000* (0.002) (0.003) Age X Treatment 0.003** (0.004) IQE Girls -0.028* -0.025 (0.014) (0.023) IQE Girls X Treatment -0.004 (0.025) Observations 1824 1810 1810 Notes : Column 1 provides the estimated differen- tial attrition for the treatment group controlling for randomization strata fixed effects. Column 2 includes additional control variables. Column 3 tests for differential attrition by each control vari- able. *: Significant at 10%; **: 5%; ***: 1% 94 Table A8: COSDEC IE attrition test using baseline Mafita data (1) (2) (3) Treatment -0.033*** -0.034*** -0.183** (0.011) (0.011) (0.071) Hours of studying religious texts (in normal day) -0.001 -0.002* (0.002) (0.003) Hours of studying religious texts (in normal day) X Treatment 0.003** (0.004) Has been taught a trade (proportion) -0.024* -0.066** (0.018) (0.032) Has been taught a trade (proportion) X Treatment 0.087** (0.037) Has attended a skills training (proportion) 0.049** 0.070* (0.023) (0.037) Has attended a skills training (proportion) X Treatment -0.047* (0.045) Currently working (proportion) -0.013 -0.049** (0.016) (0.024) Currently working (proportion) X Treatment 0.064** (0.031) Looking for a job (proportion) 0.010 0.020 (0.012) (0.020) Looking for a job (proportion) X Treatment -0.018 (0.025) Currently an apprentice (proportion) -0.032 -0.020 (0.021) (0.033) Currently an apprentice (proportion) X Treatment -0.036 (0.044) Has applied for a loan (proportion) 0.005 -0.041 (0.050) (0.040) Has applied for a loan (proportion) X Treatment 0.110 (0.090) Agrees most graduates find jobs (proportion) 0.003 -0.002 (0.008) (0.012) Agrees most graduates find jobs (proportion) X Treatment 0.010 (0.015) Agrees relationship between gov’t and the unemployed is good 0.006 0.003 (0.008) (0.012) Agrees relationship between gov’t and the unemployed is good X Treatment 0.011 (0.016) Observations 1555 1555 1555 Notes : Column 1 provides the estimated differential attrition for the treatment group controlling for randomization strata fixed effects. Column 2 includes additional control variables. Column 3 tests for differential attrition by each control variable. *: Significant at 10%; **: 5%; ***: 1% 95 A.2.3 Lee Bounds To measure the extent by which our results might be driven by sample selection due to unbalanced attrition, we implement Lee bounds (Lee, 2009). This method corrects for unbalanced attrition in the treatment and control groups by trimming the tail of the group-specific sample with less attrition according to the differ- ential rates of attrition between the two groups. Thus, the share of observed individuals is equalized across the two groups. Figures A1 and A2 display, respectively, the possible range of Apprenticeship treatment effects for the material and non-material outcomes adjusted for differential attrition rate using Lee’s method. The left hand end-point of each line shows the estimated coefficient obtained after trimming the upper end (right-hand side; i.e., the highest perfomers in the treatment group) of the distribution for each outcome family and the right hand extremity of the line show the estimated coefficient obtained after trimming the lower end (left-hand side; ie., the lowest performers in the treatment group) of the distribution. We find that, of our four primary outcome indices, only the assets and expenditure index is fully robust to this adjustment. However, consistent with our main results, we cannot rule out a positive effect for the income generating or job search behavior indices. Figure A1: Apprenticeship program effect ranges on material outcomes, adjusted for attrition Lee Bounds - APPRENTICESHIP Material outcomes Income generating activity Job search behavior Assets and expenditures Literay and numeracy -.1 0 .1 .2 Magnitude Concerning the non-material outcomes, as shown in Figure A2, our results on expansion of professional social networks is fully robust to this adjustment, and we cannot rule out a reduction in religiosity as found in our main results. Similarly, we cannot rule out more favorable attitudes toward political violence as found in our main results; however, results from this analysis suggest that, when corrected for attrition, this index is more likely to be negative. Interestingly, when adjusted for differential attrition, the range for the index for participation in anti-social behavior becomes fully negative. 96 Figure A2: Apprenticeship program effects on non-material outcomes, adjusted for attrition Lee Bounds - APPRENTICESHIP Non material outcomes Attitudes toward political violence Participation in political or religious violence Participation in anti social behavior Religious enforcement Religiosity Religious social network Range of social network Caregiver att towards women's empowerment Attitudes towards women's empowerment Self esteem Well being -.2 -.1 0 .1 .2 .3 Magnitude Figures A3 and A4 display, respectively, the possible range of COSDEC treatment effects for material and non-material outcomes adjusted for differential attrition using Lee’s method. We find that all or our main results on material and non-material outcomes are robust to this adjustment. Furthermore, we find that, when adjusting for differential attrition, the range for the religious social networks index becomes fully negative. Figure A3: COSDEC program effect on material outcomes, adjusted for attrition Lee Bounds - COSDEC Material outcomes Income generating activity Job search behavior Assets and expenditures Literay and numeracy 0 .1 .2 .3 Magnitude 97 Figure A4: COSDEC program effect on non-material outcomes, adjusted for attrition Lee Bounds - COSDEC Non material outcomes Attitudes toward political violence Participation in political or religious violence Participation in anti social behavior Religious enforcement Religiosity Religious social network Range of social network Caregiver att towards women's empowerment Attitudes towards women's empowerment Self esteem Well being -.2 -.1 0 .1 .2 .3 Magnitude A.3 Heterogeneous treatment effects A.3.1 Apprenticeship The following tables show heterogeneous treatment effects for the Apprenticeship program when comparing male and female participants (table A9 for primary outcomes families; table A10 for secondary outcome families), Almajiri and non-Almajiri male participants (tables A11 and A12) and IQE and non-IQE female participants (tables A13 and A14). Table A9: Heterogeneous treatment effects for male and female participants: primary outcomes Income Generating Religious Attitudes toward Participation in Participation in Activities Enforcement women anti social political or empowerment behavior religious violence Treatment 0.1118*** -0.0437 0.0467 -0.1094** 0.0698 (0.0337) (0.0462) (0.0461) (0.0449) (0.0443) Male X Treatment -0.0099 0.0447 0.0270 0.1113* -0.0486 (0.0561) (0.0639) (0.0626) (0.0650) (0.0637) Male 0.8740*** 0.3033** -0.2975 0.0720 0.4089*** (0.1020) (0.1412) (0.5700) (0.1034) (0.1074) Observations 4460 4460 4460 4460 4460 Notes :Apprenticeship program heterogeneous impacts by gender for primary outcome indices controlling for age, re- ligion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A10: Heterogeneous treatment effects for male and female participants: secondary outcomes Attitudes Religious Religiosity Well being Self esteem Literacy and Assets and Range of Job search toward social numeracy expenditures social network behavior political network violence Treatment 0.0077 0.0707 -0.1545*** -0.0111 0.0815* -0.0805* 0.0945** 0.2174*** 0.1165*** (0.0446) (0.0433) (0.0487) (0.0482) (0.0464) (0.0441) (0.0401) (0.0505) (0.0361) Male X Treatment 0.0914 -0.0805 0.0455 0.0716 -0.0908 0.0862 0.0648 -0.1433** -0.0205 (0.0628) (0.0618) (0.0621) (0.0648) (0.0637) (0.0593) (0.0590) (0.0637) (0.0594) Male 0.6346*** 0.0093 0.3693 -1.1769 -0.0672 -0.2903 -0.2354 0.2989 0.0657 (0.1474) (0.6140) (0.2850) (0.7580) (0.6252) (0.6558) (0.7794) (0.5992) (0.1087) Observations 4460 4460 4460 4460 4460 4460 4460 4460 4460 Continued on next page 98 Notes :Apprenticeship program heterogeneous impacts by gender for secondary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A11: Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: primary outcomes Income Generating Religious Attitudes toward Participation in Participation in Activities Enforcement women anti social political or empowerment behavior religious violence Treatment 0.0795 0.0171 0.0338 0.0203 0.0114 (0.0526) (0.0509) (0.0519) (0.0481) (0.0492) Almajiri X Treatment 0.0651 -0.0464 0.0928 -0.0471 0.0229 (0.0880) (0.0943) (0.0887) (0.1041) (0.0964) Almajiri -1.1047*** 0.4712* -0.0640 0.2065** -0.0890 (0.3654) (0.2743) (0.4054) (0.1002) (0.1898) Observations 2544 2544 2544 2544 2544 Notes : Apprenticeship program heterogeneous impacts for Almajiri and non-Almajiri male participants on primary out- come indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of signifi- cance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A12: Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: secondary outcomes Attitudes Religious Religiosity Well being Self esteem Literacy and Assets and Range of Job search toward social numeracy expenditures social network behavior political network violence Treatment 0.0910* 0.0114 -0.1087** 0.1104** -0.0031 -0.0336 0.1609*** 0.1305*** 0.0249 (0.0526) (0.0544) (0.0497) (0.0533) (0.0537) (0.0461) (0.0486) (0.0498) (0.0545) Almajiri X Treatment 0.0149 -0.0502 -0.0068 -0.1422 -0.0237 0.1114 -0.0347 -0.1410 0.1788** (0.0880) (0.0874) (0.0794) (0.0934) (0.0914) (0.0853) (0.0878) (0.0942) (0.0827) Almajiri -0.0842 0.0402 0.5565*** 0.2558 -0.0466 -0.8344*** -0.4647 -0.8717*** -0.5067 (0.3300) (0.2578) (0.1922) (0.4082) (0.2873) (0.2245) (0.4097) (0.2818) (0.3154) Observations 2544 2544 2544 2544 2544 2544 2544 2544 2544 Notes : Apprenticeship program heterogeneous impacts for Almajiri and non-Almajiri male participants on secondary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A13: Heterogeneous treatment effects for IQE and non-IQE female participants: primary outcomes Income Generating Religious Attitudes toward Participation in Participation in Activities Enforcement women anti social political or empowerment behavior religious violence Treatment 0.0476 -0.0398 -0.0133 -0.1267** 0.0665 (0.0617) (0.0587) (0.0592) (0.0578) (0.0598) IQE X Treatment 0.2344** -0.0098 0.1366 0.0138 0.0224 (0.0937) (0.1016) (0.0970) (0.1059) (0.1055) IQE 0.4516*** -0.0708 -0.2941 0.0157 0.1029 (0.1663) (0.1963) (0.6558) (0.1209) (0.1287) Observations 1916 1916 1916 1916 1916 Notes :Apprenticeship program heterogeneous impacts for IQE and non-IQE female participants on primary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A14: Heterogeneous treatment effects for IQE and non-IQE female participants: secondary outcomes Attitudes Religious Religiosity Well being Self esteem Literacy and Assets and Range of Job search toward social numeracy expenditures social network behavior political network violence Treatment 0.0734 0.1226** -0.1915*** 0.0295 0.0729 -0.1061* 0.1035* 0.2381*** 0.1444** (0.0618) (0.0567) (0.0690) (0.0612) (0.0601) (0.0557) (0.0597) (0.0584) (0.0637) IQE X Treatment -0.1692* -0.1289 0.1057 -0.1139 0.0341 0.0570 0.0331 -0.0850 -0.0115 (0.0989) (0.0928) (0.0920) (0.0976) (0.0960) (0.0951) (0.0920) (0.0970) (0.0934) IQE 0.8987*** -0.0366 -0.1699 -0.9438 -0.1552 -0.5865 -0.4095 0.9867* 0.1494 (0.2499) (0.6321) (0.2606) (0.8082) (0.6497) (0.6989) (0.8764) (0.5858) (0.2085) Observations 1916 1916 1916 1916 1916 1916 1916 1916 1916 Notes : Apprenticeship program heterogeneous impacts for IQE and non-IQE female participants on secondary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. 99 A.3.2 COSDEC The following tables show heterogeneous treatment effects for the COSDEC program when comparing male and female participants (table A15 for primary outcome families; table A16 for secondary outcome families), Almajiri and non-Almajiri male participants (tables A17 and A18, and IQE and non-IQE female participants (tables A19 and A20. Table A15: Heterogeneous treatment effects for male and female participants: primary outcomes Income Generating Religious Attitudes toward Participation in Participation in Activities Enforcement women anti social political or empowerment behavior religious violence Treatment 0.2064*** 0.0214 0.1402 0.0819 0.1025 (0.0537) (0.0613) (0.0554) (0.0758) (0.0680) Male X Treatment 0.0639 0.0154 -0.0402 -0.0224 -0.1347 (0.0922) (0.0890) (0.0864) (0.0956) (0.0951) Male 0.2535*** 0.0688 -0.1947** 0.0809 0.0755 (0.0808) (0.0755) (0.0803) (0.0845) (0.0873) Observations 1708 1708 1708 1708 1708 Notes :COSDECs program heterogeneous impacts by gender for primary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A16: Heterogeneous treatment effects for male and female participants: secondary outcomes Attitudes Religious Religiosity Well being Self esteem Literacy and Assets and Range of Job search toward social numeracy expenditures social network behavior political network violence Treatment 0.0333 -0.0472 -0.1277* 0.1337** -0.0502 -0.0494 0.1325** 0.2344*** 0.1443*** (0.0714) (0.0724) (0.0692) (0.0677) (0.0679) (0.0680) (0.0562) (0.0709) (0.0551) Male X Treatment -0.0808 -0.0344 0.0847 -0.0399 0.0402 0.1328 0.0936 -0.1039 -0.1237 (0.0972) (0.0961) (0.0938) (0.0974) (0.0965) (0.0951) (0.0935) (0.0955) (0.0934) Male 0.0411 0.0197 0.2561*** 0.0514 0.1832** -0.0905 0.0186 0.0858 0.1684** (0.0918) (0.0853) (0.0831) (0.0776) (0.0856) (0.0884) (0.0823) (0.0868) (0.0852) Observations 1708 1708 1708 1708 1708 1708 1708 1708 1708 Notes :COSDECs program heterogeneous impacts by gender for secondary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A17: Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: primary outcomes Income Generating Religious Attitudes toward Participation in Participation in Activities Enforcement women anti social political or empowerment behavior religious violence Treatment 0.2123** -0.0796 0.2020 0.2066* -0.0189 (0.0970) (0.0865) (0.0876) (0.1244) (0.1162) Almajiri X Treatment 0.0425 0.2464* -0.1819 -0.2005 -0.0123 (0.1341) (0.1294) (0.1307) (0.1353) (0.1367) Almajiri -0.0474 -0.0895 -0.2124** -0.0077 -0.1091 (0.0963) (0.1038) (0.1060) (0.0839) (0.1159) Observations 861 861 861 861 861 Notes : COSDECs program heterogeneous impacts for Almajiri and non-Almajiri male participants on primary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A18: Heterogeneous treatment effects for Almajiri and non-Almajiri male participants: secondary outcomes Attitudes Religious Religiosity Well being Self esteem Literacy and Assets and Range of Job search toward social numeracy expenditures social network behavior political network violence Treatment -0.0499 -0.0198 0.0136 0.0712 0.0032 0.0690 0.1180 0.1387 0.0488 (0.0993) (0.1039) (0.0927) (0.0872) (0.0980) (0.0935) (0.0967) (0.0957) (0.1016) Almajiri X Treatment -0.0175 -0.1495 -0.1775 0.0204 -0.0357 0.0681 0.1684 -0.0153 -0.0229 (0.1367) (0.1363) (0.1329) (0.1412) (0.1429) (0.1295) (0.1355) (0.1365) (0.1352) Continued on next page 100 Almajiri 0.0001 0.2057* 0.3702*** -0.1371 0.0247 -0.6655*** -0.3095*** -0.1324 -0.0798 (0.1103) (0.1097) (0.1017) (0.1111) (0.1152) (0.1055) (0.0982) (0.1033) (0.1094) Observations 861 861 861 861 861 861 861 861 861 Notes : COSDECs program heterogeneous impacts for Almajiri and non-Almajiri male participants on secondary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A19: Heterogeneous treatment effects for IQE and non-IQE female participants: primary outcomes Income Generating Religious Attitudes toward Participation in Participation in Activities Enforcement women anti social political or empowerment behavior religious violence Treatment 0.0759 0.0411 0.2414 0.1489 -0.0060 (0.0969) (0.0852) (0.0894) (0.1113) (0.0852) IQE X Treatment 0.3398** -0.0842 -0.1977 -0.1629 0.1977 (0.1412) (0.1269) (0.1247) (0.1512) (0.1374) IQE -0.0910 0.0444 -0.0018 0.1835* -0.0004 (0.1048) (0.1065) (0.0940) (0.0966) (0.1011) Observations 847 847 847 847 847 Notes : COSDECs program heterogeneous impacts for IQE and non-IQE female participants on primary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. Table A20: Heterogeneous treatment effects for IQE and non-IQE female participants: secondary outcomes Attitudes Religious Religiosity Well being Self esteem Literacy and Assets and Range of Job search toward social numeracy expenditures social network behavior political network violence Treatment -0.0245 0.0343 -0.1659* 0.0529 -0.1066 -0.0935 0.1130 0.2812*** 0.1271 (0.0947) (0.1081) (0.0982) (0.0973) (0.0966) (0.0973) (0.0979) (0.0951) (0.0917) IQE X Treatment 0.1059 -0.1631 0.0649 0.1320 0.1247 0.1076 0.0829 -0.1118 0.0198 (0.1447) (0.1319) (0.1405) (0.1433) (0.1408) (0.1385) (0.1386) (0.1396) (0.1285) IQE -0.1297 -0.1015 0.0389 0.0650 -0.0594 -0.1375 -0.0123 0.0047 -0.0522 (0.1218) (0.1052) (0.1181) (0.1121) (0.1108) (0.1086) (0.1086) (0.1075) (0.1016) Observations 847 847 847 847 847 847 847 847 847 Notes : COSDECs program heterogeneous impacts for IQE and non-IQE female participants on secondary outcome indices controlling for age, religion, ethnicity, and strata fixed effects with robust standard errors. Level of significance is based on the p-value adjusted for multiple hypothesis testing using the method described in Benjamini, Krueger amd Yekutieli (2006). *, **, and *** denote statistical significance at 10%, 5%, and 1%, respectively. 101