Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Report Number: ICRR0022927 1. Project Data Project ID Project Name P131394 Reaching Out of School Children II Country Practice Area(Lead) Bangladesh Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51660,IDA-63210,IDA-68010,IDA- 31-Dec-2017 148,814,922.81 D3830 Bank Approval Date Closing Date (Actual) 02-Oct-2012 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 130,000,000.00 0.00 Revised Commitment 159,711,191.70 0.00 Actual 151,975,527.33 0.00 Prepared by Reviewed by ICR Review Coordinator Group Denise A. Vaillancourt Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As stated in the January 10, 2013 Financing Agreement (Schedule 1,p. 4) the project’s development objective (PDO) was "to improve equitable access, retention and completion in quality primary education for out-of- school children in selected underserved areas of the Recipient's territory.” PDO-level indicators were slightly fine-tuned, as a part of the project’s two restructurings in 2017, and then added to with the introduction of a new component supporting new activities aimed at a new target group Page 1 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) under the first Additional Financing (AF1), approved in September 2018. A split rating methodology will thus be undertaken to assess the project against the original PDO indicators and targets, and then against the PDO indicators and targets (a combination of original and new ones) in place at the time of the AF1 approval. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 18-Sep-2018 c. Will a split evaluation be undertaken? Yes d. Components Original Components: Component 1: Increasing Equitable Access (original estimate of $113.5 million + $2.5 million under AF2, amounting to a revised component estimate of $116.0 million; actual cost: $76.93 million) to reduce number of out-of-school children in selected under-served areas. (i) Grants and allowances subcomponent aimed to support the establishment, operation, and management of Learning Centers (LCs), providing grants to LCs and education allowances for students. Activities included enlistment of Partner Organizations, identification and mobilization of communities, enlistment of LC teachers, formation of Center Management Committees (CMCs), selection of eligible students, and approval of LC applications meeting agreed criteria and procedures. CMCs would manage eligible LCs with project-supported grants and financing of student uniforms, stationery, rental and maintenance of LC venues, and salaries for teachers and upazila-level training coordinators (TCs). Education allowances to incite enrollment and retention of out-of-school children in the primary cycle were to be provided to children: (a) from disadvantaged households; (b) between 8 and 14 years of age at time of enrollment; (c) with a project ID card; (d) with 80 percent attendance; and (e) passing semester exams. (ii) Pilot subcomponent aimed to finance LC interventions in selected urban slums and a voucher program for children working as domestic workers, whose education had suffered long-standing neglect. It aimed to support 100 LCs in 15 slums in Dhaka during the first two years, and 200 additional LCs based on the pilot’s evaluation. The voucher scheme was to finance student education allowances and tuition fees for 700 child domestics. Technical assistance was to support the pilot’s design, implementation, and evaluation. Page 2 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Component 2: Enhancing Education Quality (original estimate of $11.3 million + $0.8 million under AF2, amounting to a revised component estimate of $12.1 million; actual cost: $16.27 million) to improve retention in, and completion of, the primary education cycle. (i) Teacher development subcomponent designated Dhaka University’s Institute of Education Research (IER) as training agency to lead foundation training, refresher training, and subject-based training, focused on English and Math. A Quality Working Committee was to guide and oversee activities, and a TC was responsible for coordinating teacher training in upazilas. The National Academy for Primary Education was expected to provide inputs on LC teacher training modules (but only after a proper assessment of its capacity), and technical assistance was also envisaged. (ii) Classroom support subcomponent would mobilize Assistant Upazila Education Officers, head teachers from nearby public primary schools, and upazila TCs to guide LC teachers on the use of subject plans, teaching-learning materials, and student assessments. Textbooks were to be provided free by the government of Bangladesh. Supplementary teaching-learning materials were to be identified and made available to project LCs. (iii) Pre-vocational skills training pilot sub-component was to target project students who completed at least grade 3 and were aged 15 and above. Vouchers (covering education allowances and tuition) would enable 5,000 eligible students to participate in existing (public and non-public) skills training programs, including those under International Development Association (IDA)-financed projects. Technical assistance was to support the pilot’s design, implementation, and evaluation to inform possible program expansion. Component 3: Improving Project Management and Capacity (original estimate of $7.7 million + $2.0 million under AF2, amounting to a revised component estimate of $9.7 million; actual cost: $22.25 million) to establish an effective implementation structure and capacity. (i) Project management and capacity building subcomponent envisaged a continuation of the arrangements under the first project, Reaching Out of School Children I (ROSC I), with the Directorate of Primary Education (DPE) of the Ministry of Primary and Mass Education (MOPME) serving as implementing agency, the ROSC Steering Committee overseeing implementation, carrying out joint annual reviews, and resolving implementation issues, and the project implementation unit (PIU) responsible for day-to-day implementation. A new quality working committee was to guide and oversee the quality component, and a project implementation committee, chaired by the Director-General, DPE, and composed of other key actors, was responsible for facilitating implementation and reporting to the Steering Committee. Technical assistance was to be provided by specialized consulting firms and by IDA. The Upazila Education Committee was to continue to support the Upazila Education Officer (UEO) on the coordination of activities. UEO capacity was to be enhanced through deployment of TCs. Assistant UEOs and head teachers would be active in classroom support at the field level. Planned capacity building activities included: (i) in-country training in management, monitoring and evaluation (M&E), educational development, procurement and financial management, and Right to Information compliance; (ii) international training on best practices in relevant education programs; and (iii) community-level training for CMCs. (ii) Social awareness and advocacy sub-component sought to increase social awareness and build advocacy on key project features, including: (i) development of a communications strategy and Page 3 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) implementation plan; (ii) preparation and dissemination of the project operations manual, brochures, posters, and audio-visual materials; (iii) central and upazila-level conferences, workshops, and community- level meetings facilitating peer learning; (iv) training of CMC members; and (v) mobilization of the media (newspapers, TV, online posts). Component 4: Monitoring and Evaluation (original estimate of $5.0 million + $1.2 million under AF2, amounting to a revised component estimate of $6.2 million; actual cost: $6.43 million) to establish an effective M&E system to monitor and evaluate the project’s performance and efficacy. (i) Monitoring of inputs, processes, and outputs subcomponent aimed to support: preparation of databases; validation of LCs by monitoring officers; preparation of quarterly self-reported LC monitoring forms; sample-based monitoring including compliance verification of education criteria and procedures by monitoring officers; public expenditure tracking surveys; internal monitoring of LCs; and regular management reviews. Use of SMS monitoring and data validation through smart phones was an innovation that was to benefit from Bank technical assistance. ii) Evaluation sub-component was to support a learning assessment pilot (two rounds of learning assessments for grades 3 and 5) and an evaluation of the impact of interventions on schooling outcomes. Activities included a baseline and follow-up surveys, and impact evaluations (IEs) of the various pilots. Revisions to components Component 5: Provide Safe and Equitable Learning Opportunities to Children and Adolescents of Displaced Rohingya Population (DRP) (estimate at time of AF1: $25.0 million; actual cost: $25.0 million), added in 2018 to address education needs of 300,000 crisis-affected DRP children and adolescents and to strengthen service delivery in Cox’s Bazar. Four subcomponents aimed to: (1) provide informal education to DRP children and adolescents; (2) enhance capacity of government and local administration; (3) improve DRP LC teacher training facilities; and (4) undertake psycho-social activities and awareness building. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost. The total project cost was originally estimated at $137.5 million. The estimate was revised upward to $169.0 million reflecting the cost of a fifth component to address the education needs of the DRP, added in 2018, and revised cost estimates for completing project activities, assessed in 2020, resulting from exchange rate fluctuations. Financing. IDA financing was comprised of (1) an original credit of SDR 86.2 million (US$130.0 million equivalent); (2) a first Additional Financing (AF1) of SDR 17.9 million (US$25.0 million equivalent), of which SDR 14.9 million ($20.84 million equivalent) was a grant and SDR 3 million ($4.16 million equivalent) was a credit; and (3) a second Additional Financing (AF2) of SDR 4.7 million ($6.5 million equivalent). A few days before project closing (June 21, 2021), the Bank approved the cancellation of 41 percent of the above-cited Page 4 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) SDR 3.0 million grant, in the amount of SDR 1.23 million. A total of SDR 85.69 million was disbursed against the original credit amount of SDR 86.2 million, achieving a disbursement rate of 86 percent. Of the SDR 14.9 million grant approved under AF1, SDR 1.23 million was cancelled and SDR 12.00 was disbursed (81 percent of the original amount and 88 percent of the amended total, net of cancelled funds), while the SDR 3.0 million credit, also approved under AF1, was 96 percent disbursed (SDR 2.89 million disbursed). Eighty-six percent of the SDR 4.70 million credit approved under AF2 (or SDR 4.04 million) was disbursed. Borrower Contribution. Under the original project, government counterpart financing in the amount of $7.5 million (or 5 percent of the original estimated cost) was anticipated, with IDA financing 95 percent. No additional government counterpart financing was factored into either of the two Additional Financings. Actual data in the ICR show no indication of any government counterpart financing being provided. The TTL and ICR Main Contributor, during a September 7, 2022 meeting with IEG, noted that the government did, indeed, provide 95 percent of its counterpart commitment (about $7.1 million equivalent). These funds (not reflected in ICR cost tables) financed staff salaries, office maintenance, and almost half of associated transportation costs. Key Dates. The project was approved on October 2, 2012 and became effective on January 30, 2013. Its mid-term review was undertaken in October 2015, and it closed on June 30, 2021, 3.5 years after the original closing date of December 2017. Restructuring and Additional Financing. The project was restructured four times and benefited from two Additional Financings. Two restructurings in 2017 (April and December) introduced refinements to the results framework and to cost estimates and allocations across components reflecting, among other things, the sudden, large influx of the Rohingya population in Bangladesh. The December 2017 restructuring also extended the original closing date by one year to December 31, 2018. AF1, approved in September 2018, supported a new component to address the education needs of Rohingya children, added new indicators to the results framework, and extended the closing date by two additional years to December 31, 2020. AF2 was approved in December 2020 to cover a funding gap caused by expected currency exchange fluctuations and extended the closing date by another six months (to June 30, 2021) to make up for time lost during a temporary suspension due to COVID-19. Two other restructurings (approved in September 2019 and June 24, 2021) reallocated funds, the latter cancelling a part of the IDA grant under AF1. 3. Relevance of Objectives Rationale The PDO is highly relevant to current country conditions. While the project made considerable headway in developing and supporting interventions and services to attract and retain out-of-school children and to facilitate their completion rates, the ICR notes (pp. 7 and 38-39) that there still remain many more deprived and marginalized out-of-school children residing in other under-served areas (remote upazilas, disaster-prone areas, urban slums), who would benefit from similar support. The PDO is highly relevant to the current development priorities of the country. Bangladesh’s Education Sector Plan (ESP) for FYs 2020/21 – 2024/25, issued on December 10, 2020, supports three goals: (1) improved access and equity for all, irrespective of gender, age, religion, ethnicity, and Page 5 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) disadvantaged groups; (2) improved quality and relevance of education; and (3) strengthened governance and system management. These goals are fully aligned with the PDO and the project’s four components, which support, respectively, improvements in: equitable access; retention; completion rates; quality (and relevance to labor market needs); management capacity building; and M&E for an improved evidence base. The project’s focus on out-of-school children is fully aligned with the objectives under the primary education segment of Bangladesh’s ESP, which emphasizes the need to reach this target group. The strong relevance of this focus was further enhanced with the addition in 2018 of a fifth component (under AF1) extending coverage to displaced and vulnerable Rohingya children. The PDO is also highly relevant to the Bank’s current Country Partnership Framework (CPF) for Bangladesh. The current CPF initially covered the period FY16-20. It was extended to FY21 following the Program Learning Review, which reaffirmed the continued relevance of its three pillars: growth and competitiveness; social inclusion; and climate and environmental management. (A new CPF for FY23-27 is currently under preparation, its initial preparation work delayed as a result of the COVID-19 epidemic.) Under its social inclusion pillar, the FY16-21 CPF supports the objective of improved equity in access and quality of education. The project’s focus on out-of-school children is highly supportive of this objective, as these children come from deprived circumstances, suffering from poverty, marginalization, residence in deprived areas, and employment as domestics, all of which undercut their access to quality primary education and their levels of achievement. The targeting of Rohingya children, under AF1, further increased project relevance on this front. In response to the CPF’s assessment of the inadequacy of the quality and relevance of education and the consequent mismatch of skills to the demands of the labor market, the project also focused on education quality, devoting a full component to teacher development and classroom support, and supporting a pre-vocational skills training pilot for targeted students who completed grade 3 (at least) and are aged 15 years and above. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve equitable access for out-of-school children in selected under-served areas (original outcome targets) Rationale The PAD did not present a Theory of Change. The one presented in the ICR (Annex 9, p. 68) captures well the thrust of the original project (Components 1-4), and the fifth component introduced under AF1. This rationale underpins all four Objectives, which are interlinked. Component 1 of the project sought to increase equitable access by: (a) providing grants to rural LCs supporting their establishment, operation, and maintenance; and (b) providing education allowances to students enrolled in project LCs who (i) belonged to disadvantaged households; (ii) were between the ages of Page 6 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) 8 and 14 years at the time of enrollment; (iii) had a project ID; (iv) attended 84 percent of classes; and (v) passed semester grade exams. A total of 720,000 out-of-school children and 21,700 rural LCs in 148 upazilas were targeted for this support. Of these, about one third (260,000 students and 8,400 LCs in 48 upazilas) had received support under the previous project (first ROSC, P086791) and would continue to receive support under this follow-on project; and the remaining two-thirds (approximately 450,000 out-of-school children and13,000 rural LCs in 100 new upazilas) would receive first-time support. This component also aimed to support a pilot in select urban slums. These interventions were expected to incite increases in the supply of LCs and in the enrollments of disadvantaged children, thus having a direct impact on the project’s objective of improved access among out-of-school children, especially poor, female, child domestics, and other disadvantaged children. Outputs and intermediate results: Rural LCs:  A total of 20,209 LCs were supported by the project, almost achieving the original, unchanged target of 21,700 (93% achieved). Selection criteria included upazilas where (i) enrollment rates were 80 percent or less; (ii) the gender gap in net enrollment was at least two percentage points; (iii) the primary completion rate was 50 percent or less; and (iv) headcount poverty was about 30 percent. The IE confirmed the effectiveness of these criteria, given that the poorest districts had the highest number of LCs.  Student attendance rate increased from a baseline of 75 percent to 85 percent by the project’s end, fully achieving the original, unchanged target of 85 percent. Urban Slum LCs:  The urban slum pilot (under the leadership of Save the Children as Specialized Agency) developed operational manuals and teaching-learning materials to support the establishment and operation of 113 LCs in 21 compounds in Dhaka city, under a successful, accelerated basic non-formal education program.  Under the scale-up of this program, 1,534 additional LCs in an additional 329 compounds were established at the community level in ten city corporation areas. Pre-vocational training:  A pre-vocational training pilot to deliver mobile vocational training to the rural poor and low-education adolescents in 13 market-driven trades was designed, implemented, and subsequently scaled up, based on its success. This activity included: o Identification and training of training providers; o Identification by training providers and community management committees of eligible trainees, who had passed the Primary Education Completion Examination, and their enrollment; o Provision of skills training vouchers to 25,000 enrolled students, covering covered education allowances and tuition; and Page 7 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) o Identification of suitable locations and training centers and the establishment and equipment of mobile vocational training centers. Outcomes: Rural LCs:  The number of out-of-school children enrolled in rural LCs rose from the baseline of 260,000 to 687,556 by the project’s closing, reflecting a substantial (95 percent) achievement of the (original, unchanged) target of 720,000.  Of these out-of-school children enrolled in LCs: o 49 percent were female, substantially achieving the target of 50 percent, thus preserving the baseline level of 50 percent; and o 87 percent were disadvantaged students, slightly exceeding the target of 85 percent. o An average of almost 7 percent of students were considered “specially abled children,” and about 2.5 percent were from indigenous communities.  The IE found that: o 97 percent of students came from families that were considered extremely poor; o the vast majority of their parents’ educational qualifications were below the secondary level; and o almost 20 percent of fathers and 19 percent of mothers were illiterate.  The IE reports that before enrolling in LCs, about 75 percent of children were out of school, 11 percent had gone to other schools but dropped out, and about 12 percent migrated from other schools.  The number of student years supported increased from a baseline of 226,991 to 1,678,177, exceeding the target of 1,343,000. This indicator encompasses student years supported at rural LCs and urban slum LCs, alike. While total student enrollments at rural and urban slum LCs fell slightly short of targets (albeit both substantially achieved), the significant surpassing of student years targeted was attributable to the underestimation of the initial target. At the project’s outset, it was assumed that some students would start in Grade 1, while others (dropouts) would start in other Grades (2, 3, 4, or 5) depending on when they left school. However, the actual placement of these students had many more starting in Grade 1 (and other lower-than-anticipated grades) because they did not have the basic skills and knowledge to start at more advanced grades, thus creating more student years supported by the project than originally planned.  Students’ parents reported that monthly allowances, textbooks, stationery, and uniforms lessened the economic burdens of families and incited attendance at project-supported LCs. Additional factors included the shorter distance to LCs (presumably in rural areas) and their flexible timing, suited to students’ needs. Teachers and communities’ improved perception of quality was also a likely factor. Urban Slum LCs  The urban slum pilot served about 3,200 students in Dhaka City.  Under the scale-up of this program, in ten city corporation areas, where about 55,000 eligible out-of- school children aged 8-14 years were identified, 46,318 were enrolled, of which half were female.  In total (pilot and scale-up combined), about 48,000 students participated in urban slum pilots, surpassing the original target of 12,000 and substantially achieving the adjusted target (under Page 8 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) the first restructuring) of 50,000 students. While the first two bullets add up to approximately 49,000 students, the total of about 48,000 (47,957, to be exact) was confirmed by the TTL and ICR Main Contributor, during a September 7, 2022 with IEG, as a more accurate figure, updated at the time of ICR finalization. Pre-vocational training:  The pilot and its subsequent scale-up phase culminated in 25,376 students from hard-to-reach regions, of whom 8,500 were from Cox’s Bazar, receiving 360 hours of skills development training and follow-up enterprise development support, by 12 training providers in 13 market-driven trades.  The IE revealed that the largest proportion of students enrolled in the pre-vocational training program (39 percent) came from moderately poor families, followed by 31 percent from extremely poor families. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective Improve equitable access for out-of-school children in selected under-served areas (revised outcome targets) Revised Rationale Same rationale as above, with Component 5 extending support to improve equitable access to DRP children. Outputs and intermediate results: Rural LCs, Urban Slum LCs, Pre-vocational Education:  Outputs for rural LCs, urban slum LCs, and pre-vocational education are all the same as those reported against the original Objective 1 outcome targets. In addition, this section reports on the complement of outputs of the expanded project scope (Component 5, financed under AF1), which aimed to provide safe and equitable learning opportunities to children and adolescents of the DRP.  At the onset of COVID, toward the end of the project, the rural LC activities had largely been completed, so there were no significant COVID-specific supports provided. The urban slum students had completed their studies at that point and were working from home, preparing for their final exams. They benefited from home visits of their teachers, who helped them to prepare for exams and ensured that those exams were properly administered. Pre-vocational training centers were allowed by the government to re-open very rapidly (on a much more compressed timeframe than other schools), thus minimizing the effect of COVID on their studies. DRP Camp Information LCs: Page 9 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394)  A total of 1,331 information LCs were supported for the DRP in Cox’s Bazar District, almost achieving the target of 1,500 approved in 2018 under AF1 (89 percent achieved).  A total of 1,196 new teachers were recruited from the Cox’s Bazar local community (of which 86 percent were female), and 3,000 teachers were trained on the use of teachers’ guides, student workbooks, and supplementary subject learning materials.  Of all teachers receiving training provided by IER under the project, 1313 were assigned to LCs in DRP camps.  A total of 3,000 teachers received special training in Cox's Bazar District, serving DRP children and adolescents, exceeding the target of 2,000 set under AF1.  A total of 7,473 CMC members (42 percent female) received orientation on roles and responsibilities, including community sensitization on the importance of education for boys and girls, and support for DRP activities.  A total of 1,331 consultations were held, one in each of the LCs in Cox’s Bazar District receiving project support.  When in-class learning was suspended due to COVID-19, the project provided 93,229 DRP children (of whom 50 percent were girls) with packages supporting distance learning with caregiver support, and hygiene and sanitation supplies to help protect them from infection. Students continued to access education through home-based modalities led by 41,497 child protection focal points and volunteer community-based social workers, trained to provide children with psycho-social support.  The project also helped develop, roll out, and train teachers and master trainers under a new e- learning platform. Outcomes: Rural LCs, Urban Slum LCs, Pre-vocational Education:  Outcomes for rural LCs, urban slum LCs, and pre-vocational education are all the same as those reported against the original Objective 1 outcome targets. In addition, this section reports on the complement of outcomes achieved under Component 5 (financed under AF1). DRP Information LCs:  Under AF1, focused on DRP children in Cox's Bazar, 112,000 children were enrolled in 1,331 LCs, not achieving the targets of 150,000 enrolled students in 1,500 LCs. The 1,331 LCs were being monitored and supported by 1,061 local center management committees following a cluster approach in the target district.  Of these enrolled DRP children, half (48 percent) were female, and 0.4 percent were children with disabilities.  At project completion, 234,400 DRP student years were supported, achieving two-thirds of the target of 350,000, established in late 2018 under AF1. Student years supported included 1.2 years of in-LC class learning for the 112,000 enrolled DRP students, plus one year of caregiver-led home- based learning for the 100,000 DRP students, who continued their studies after the onset of COVID. Page 10 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394)  Of the 25,376 students receiving vocational skills development training and follow-up enterprise support under the pre-vocational training program, 8,500 were from Cox's Bazar. Revised Rating Substantial OBJECTIVE 2 Objective Improve retention rates in the primary cycle of (formerly) out-of-school children in selected under-served areas Rationale Over and above its aim to improve equitable access to education, the provision under Component 1 of education allowances to out-of-school children in disadvantaged households was expected to improve their retention rates in primary school. These allowances covered incidental expenses associated with their education as well as the payment of their tuition, lifting financial barriers to education. Also expected to contribute to improved retention rates were activities supporting management strengthening under Component 3, especially the establishment and training of CMCs, which strengthened community involvement in school management and undertook awareness raising of parents about the importance of education. Outputs and intermediate results:  Outputs and intermediate results shown under Objectives 1 (equitable access) and 4 (quality) also supported the achievement of this objective. Outcomes: Rural LCs:  The average grade retention rate increased from a baseline of 60 percent to 83 percent at completion, exceeding the target of 75 percent.  Notwithstanding a still high dropout rate, exacerbated by COVID-19, the IE revealed a significantly lower rate in the program area (about 10.5 percent of children dropped out of project-supported LCs), compared with other schools (where about 25 percent dropped out). Urban Slum LCs Page 11 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394)  The IE also revealed that the 18 percent dropout rate in LCs supported under the Urban Slum program was about half of the 35 percent rate for the control group, confirming the significant contribution of the project in urban slums.  Moreover, the IE found that a relatively higher proportion of target children (61 percent) are currently studying, compared with 53 percent of children in the same age group from comparison villages and communities. The IE notes that this is a statistically significant difference that can be attributed to the project.  To mitigate potential learning losses due to the closure of LCs during the pandemic, the project adopted home-based learning with practice sheets and home visits by the LC teachers and organized parent supervised tests for 31,000 slum LC students, to facilitate preparation for the primary completion exam. Rating High OBJECTIVE 3 Objective Improve (primary cycle) completion rates of (formerly) out-of-school children in selected under-served areas Rationale Primary completion rates were expected to increase through a combination of financial incentives provided under the project (student allowances and payment of tuition fees), investments in quality improvements, and strengthening of community-based management, with increased parental information and involvement. Outputs and intermediate results:  Outputs and intermediate results shown under Objective 1 (equitable access) and Objective 4 (quality, and improved management and M&E) also supported the achievement of this objective. Outcomes: Rural LCs:  Completion rate for LC students (as measured by successful passing of Grade 5 exams) increased from a baseline of 73 percent to 84 percent, exceeding both the original target of 75 percent, and the slightly increased target of 78 percent, revised under the first (April 2017) restructuring. Of the enrolled students, 16 percent did not complete the exam.  The project was successful in preparing students to continue their studies at the junior secondary level or in pre-vocational training programs: 63.25 percent of project students were continuing their studies, Page 12 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) of which 53 percent were at the junior secondary level, 32 percent were in secondary, and 15 percent were at higher secondary and above.  Of those not continuing their studies (about 37 percent of all enrolled students), the majority (about 20 percent) completed the Grade 5 exam, and the rest (about 16 percent of all enrolled students) did not. Parents reported that the greatest impediment to these students continuing their studies was affordability. Still, 40 percent of LC students who did not continue their studies beyond primary level were employed outside of the home, about five percent in agriculture, and two percent in business. Correlates of schooling outcomes in surveyed communities revealed in the IE included: o Girls are more likely to continue their studies (66 percent), compared to boys (51 percent). o Children living in poor families are more likely to drop out. o The father’s occupation impacts the likelihood that children will continue to be enrolled. o Dropout rates are higher for children with illiterate mothers and mothers with less than primary education.  The IE found that the share of project students (sample) enrolled in schools (60 percent) compared favorably with those in the control group (52 percent). Urban Slum LCs:  Almost 70 percent of urban slum students completed the primary school cycle. Of the 27,017 students tracked out of 28,257 students of the urban slum program who participated in the Primary Education Completion Exam in 2020, 21,904 were enrolled in secondary education in 10 city corporations. The IE found that 65 percent of LC graduates were enrolled in secondary schools, compared with 52 percent in the control area. Rating High OBJECTIVE 4 Objective Improve the quality of primary education in under-served areas Rationale Component 2 sought to enhance education quality through teacher development, classroom support, and a pre-vocational skills training pilot. Outputs would include: teachers having received initial foundation, annual refresher, and specialized training; mobilized upazila-level technical resource persons providing technical support and guidance to teachers; textbooks and learning materials; and a pilot providing pre-vocational skills. Also contributing to improvements in education quality were activities under Components 3 and 4 aimed at strengthening management capacity and the implementation and use of learning assessments and impact evaluations for continued improvements to education for out-of-school children. Component 3 was expected to culminate in a working committee to guide and oversee quality and to mobilize agencies and communities to help implementation; and trained center management committees. Component 4 was expected to culminate in improved data quality and M&E. Page 13 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Outputs and intermediate results:  Four IE studies were completed by the project’s closing, not achieving the PAD target of three baseline and four follow-up IE studies, but fully achieving the revised target of four, established under the first (April 2017) restructuring. These studies provided insightful information on the extent to which the project’s support and pilots reached the targeted population, achieved the PDOs (equitable access, retention, completion, and quality), and impacted the lives of the target populations.  Teaching and learning materials were provided to project LCs and LC students through grants, in addition to government provision of textbooks.  M&E capacity was strengthened with the implementation of learning assessments, impact evaluations, the increased use of information technology for data collection and reporting, and enhanced use of data and evidence for decision-making (see M&E section).  Communities were trained and supported to be more involved in the management of urban and rural LCs. Rural LCs:  Several activities were implemented to improve education quality, including teacher training, carrying out of student learning assessments, and provision of pre-vocational skills-based training responding to market demands.  The IER of the University of Dhaka provided a 15-day foundation training, six-day annual refresher training, and six-day subject-based training focused on English and Math for teachers, and a two-day foundation training for CMCs. o A total of 20,934 teachers were trained against a target of 20,000 (including 1,313 assigned to LCs in DRP camps), substantially achieving the original target of 21,700 teachers trained under the project, and the revised target under AF2, approved in 2021, of 20,000. o . Of the teachers trained:  11,743 attended foundation training for LCs; most teachers reported participating in 15- day training programs, but some expressed concern that the duration was too short;  25,316 attended refresher training (encompassing teachers who were initially recruited/identified and additional ones recruited to replace some lost through attrition); and  247 attended subject-based training. o The IER and TCs, together with the head teachers of the closest government primary schools (all trained under the project), visited LCs regularly to provide feedback to head teachers for improving their skills. IER trained about 900 pool teachers to help supervise the LCs’ academic activities. o The IER trained the principal, vice-principal and teachers of the 21,000 CMCs established under the project on regulation writing, motivation, and classroom monitoring to strengthen supervision, support, and oversight of LCs. More than 60 percent of CMC foundational training participants were women. Pre-vocational training:  Under the pre-vocational training pilot and its subsequent scale-up: Page 14 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) o Twelve training providers were selected and received training provided by Save the Children on soft skills, life skills, and apprenticeship and entrepreneurship development.  25,376 students received 360 hours of skills development training by 12 training providers in 13 market-driven trades.  Skills training vouchers covering education allowances, as well as tuition fees in training institutes in which children were enrolled, provided to 25,000 enrolled students. o These 25,000 youth from hard-to-reach regions (of whom 8,500 were from Cox’s Bazar) received vocational training and enterprise development support, with a special focus on female youth, through mobile training centers. o Training providers also provided follow-up support to graduates, including technical guidance, apprenticeship arrangements, entrepreneurship training, and linkages with financial institutes to help graduates start their own businesses. Outcomes: Rural and Urban LCs:  Systematic learning assessments were carried out at Grade 3 and Grade 5 levels. Project students participated in the Primary Education Completion Examination – a prerequisite for advancing from Grade 5 to the secondary level – in 2014, 2015, 2017, 2018, and 2019. During the last year (2019) the pass rate for project students was 87 percent, a notably higher rate for students from disadvantaged backgrounds. The Bank team, during a September 7, 2022 meeting with IEG, noted that one learning assessment was carried out during the project, at Grade 3 and Grade 5 levels. It was administered to students of rural LCs only, as they were at advanced stages of implementation at that time. (Urban slum LCs, on the other hand, were just starting implementation.) While the single learning assessment cannot provide insights or trends for the quality objective, the Bank team noted that these assessments will continue under the country-wide scale-up of the program, which should facilitate the documentation and analysis of future trends and use of these results for evaluating program performance and informing further improvements.  Community members interviewed assessed rural and urban LCs’ education quality to be strong, with 50 percent assessing it to be good and 20 percent to be very good. Seventy percent of focus group discussion participants and 80 percent of head teachers expressed satisfaction with LCs’ infrastructure and learning environment.  Most teachers reported that rural and urban CMCs met regularly and played a central role in school performance, with committee members communicating consistently with teachers and parents to ensure attendance, support exams, ensure availability of materials, etc. Pre-vocational education:  70 percent of the 1,250 students in the pilot phase were employed after graduation.  A tracer study of pre-vocational training program graduates, carried out by Save the Children, found that: o 54 percent had outside employment (of which 61 percent were in the formal sector and 39 percent in the informal sector). Page 15 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) o 45 percent were self-employed; and only five graduates were not engaged in income- generating activities. o A higher percentage of men held outside employment (68 percent), while the majority of women (63 percent) were self-employed. o Almost all of the study’s respondents reported that: (i) they could utilize the skills acquired in their current profession (98 percent); and (ii) their family situation had improved as a result of their vocational training.  The project’s IE found that around 72 percent of the project-supported vocational training graduates had either outside employment or were self-employed, despite the economic downturn due to COVID- 19. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project was successful in reaching out-of-school children in different environments (rural, urban, and older children in need of pre-vocational training) and in retaining them in school. Over two-thirds of children enrolled in LCs completed the primary school cycle. Over half of LC graduates are continuing their education at the secondary level. Although implementation was impacted by several exogenous events, especially the influx of Rohingya children and the onset of the COVID-19 pandemic, the project was flexible and proactive in adjusting and adapting its activities to face emerging needs. The project exceeded expectations with respect to objectives 2 (retention) and 3 (completion), and substantially met expectations for objectives 1 (equitable access)and 4 (quality). The majority of those receiving pre-vocational training were gainfully employed. During a September 7, 2022 meeting with IEG, the Bank team provided the following assessment of the project’s attribution and counterfactual. Attribution of the outcomes to the project (and its predecessor, the first ROSC project (P086791)) is very strong. The Bank designed and developed, with the government, the innovative, community-based model for reaching out-of-school children and supported a learning-by-doing process for ongoing improvements to effectiveness, which has culminated into a government-owned program that is an integral part of its current Education Sector Plan and being scaled-up country-wide. The project worked in partnership with, and drew on the expertise and comparative advantages of, various development partners to ensure a well-coordinated, single, coherent approach and framework. Contributing partners included Save the Children, UNICEF, the United States Agency for International Development, and the Government of Canada. In the absence of this project (and its predecessor), many fewer (if any) out-of-school children (urban and rural) would have been given access to basic and pre-vocational education. There would have been no innovative, effective, community-based model that would have been tested, adapted, evaluated, and refined. As a consequence, the government’s adoption of a sector-wide program for reaching out-of-school children, based on this model, and its financing by the Bank and other development partners, would have Page 16 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) been highly unlikely. The two million children targeted under this program would likely not have been reached, nor their lives enriched by the benefits of returning to school. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale The rationale for the efficacy rating of the original project also applies here. Under the revised project, Component 5’s contributions to Objective 1 (targeting children in underserved areas) were mixed. Activities targeted at the DRP fell slightly short of expectations despite important progress, especially in view of the exogenous factors that affected implementation in its later stages, primarily COVID-19. Nevertheless, despite minor shortfalls in reaching the outcome targets under AF1, the strong performance of the rest of the project in achieving Objective 1 remains Substantial, and the efficacy ratings of the three other objectives (not supported under AF1) also remain unchanged. Building on the assessment of the project’s attribution and counterfactual assessed above under the original design, an additional observation can be made, taking into account the new activities introduced under AF1. In the absence of the experience and model provided under the project, the emergency provision of education to some 93,000 Rohingya children in refugee camps would likely not have been as rapid, efficient, or expansive in its coverage. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency Economic Efficiency. At appraisal, the discounted value of benefits of the original project (Components 1-4) for the base-case scenario was estimated at $340 million, and the present discounted value of costs was estimated at $167 million (based on a discount rate of 12 percent for the benefit and cost streams). The net present value (NPV) of program benefits was estimated at $173 million, with an associated internal rate of return (IRR) of 22 percent. The discounted value of benefits and costs for the new Component 5, financed under AF1, was estimated at $28.22 million and $18.99 million, respectively, while its NPV was estimated at $11-16 million over five years, based on conservative estimates regarding repatriation efforts over two years, with an estimated IRR of 17.5 percent. The ex-post analysis for the original project estimated a discounted value of $336.36 million in benefits and $255 million in costs. The ex-post NPV was estimated at $80.58 million (based on a discount rate of 12 percent for the benefit and cost streams), and the IRR was estimated as 18 percent. The ex-post discounted value of benefits and costs for the first AF are estimated at $19.35 million and US$16.95 million, respectively, with an estimated NPV of program benefits of US$2.40 million and an associated IRR of 16.72 percent. Only the “no-repatriation” scenario was calculated, since no repatriation has taken place. Further, the ex-post analysis assumed that Page 17 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Rohingya youths’ five years of increased earnings would be attributed to the education they received in the LCs under the project (appraisal assumption), and that they would earn half of that of a local youth. Thus, the benefit stream calculation is a conservative one and assumes the low-case scenario. No opportunity cost was included for the Rohingya children, as they had no earning opportunity during the project period inside the DRP camps. The quality premium is calculated as only half of that of the local children. As expected, a large part of the benefits accrues from the wage premium for project-supported primary school completers. Actual program costs do not constitute the largest category as they did at appraisal, in view of the huge increase in income and associated opportunity cost. In short, the project’s economic efficiency was Substantial. Although revised estimates generated less than the ex-ante IRR, this was due mostly to significantly increased opportunity costs according to the Household Income and Expenditure Survey (HIES) 2017, compared with the HIES 2010 used at appraisal. The updated data showed a significant rise in earnings and expenditures, and in the opportunity cost of not entering the labor force. The NPV of total program costs of the program was significantly higher ex-post than at appraisal. The NPV of benefit streams has also increased due to higher expected income/wages. Ex-post estimates are conservative lower-bound estimates, and do not account for externality benefits arising from healthier, more educated workers and a more equitable, inclusive society. Neither did the analysis assign additional benefits to primary level completers, who would have gone on to complete secondary or higher levels of education and receive higher wages. Additionally, increased income of the graduates is assumed to continue for 20 years (only five years for DRP), though typically a primary school completer is likely to remain active much longer and to receive increases in income beyond the projected 20 years. If the out-of-pocket cost (original financing) was doubled (20 percent as opposed to 10 percent as originally assumed), the IRR (original) decreases to 17.01 percent. A similar exercise brings the IRR of the AF1 to 15.97 percent. The IRR remains significantly positive under these scenarios, although below original expectations. For the AF1, if earnings of a graduate are calculated as equal (as opposed to half, as assumed) to those of a local graduate, the IRR increases to as high as 44 percent (optimistic). Implementation efficiency. The project’s implementation efficiency was substantial. While its implementation period of 9.5 years may seem long, this was due to two factors. First, AF1 expanded the project's scope not only in terms of the number of beneficiaries, but also through a separate, new component to help address the DRP crisis in one of the poorest areas supported by the project. The targeted number of children increased from 12,000 to 50,000 for urban slums and by 150,000 for DRP children. At the time of this AF in 2018 (about three years after effectiveness), the project had already disbursed US$102.24 million, or approximately 79 percent, of the original credit. More importantly, the project had either nearly achieved or surpassed original end-of-project PDO targets. The Rohingya crisis resulted first in a one-year closing date extension to process the AF, and then a two-year closing date extension to implement mostly the newly introduced activities. Subsequently, COVID-19 impacted implementation when schools and LCs were closed, disrupting ongoing activities at a time of accelerated implementation during the project's final phase. The project, nevertheless, produced important results at a lower cost than originally estimated. Moreover, the project’s delivery model was effective in providing education to out-of-school children through an informal, cost-effective mechanism, using established agencies in coordination with communities and effective partnerships with international partners. Other factors that contributed to implementation efficiency included the strong division of labor with other donors and organizations, the use of existing institutional arrangements for implementation, and good adjustments to changing circumstances during implementation. Page 18 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  22.00  Not Applicable 100.00 ICR Estimate  18.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the project’s original outcome targets: The relevance of the PDO is rated high; efficacy in achieving the PDO is rated substantial for the first objective (equitable access) and fourth objective (quality), and high for Objectives 2 (retention) and 3 (completion); and efficiency is rated substantial. These ratings produce an Outcome rating under the original outcome targets of Satisfactory. Under the project’s revised outcome targets: The relevance of the PDO is rated high; achievement of the first objective (equitable access) remains substantial, taking into account both the PDO indicators and targets from the original design and the two additional PDO indicators and targets under AF1, and achievement of the fourth objective (quality) was also substantial, while the ratings for the two remaining Objectives 2 and 3 (and associated targets) remain high; and efficiency is rated substantial. These ratings also produce an Outcome rating under the revised outcome targets of Satisfactory. The project's overall Outcome rating is therefore Satisfactory, reflecting only minor shortcomings in the project's preparation, implementation, and achievements. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Building on the first ROSC project, this second operation has established a successful system for reaching vulnerable, out-of-school children. Throughout implementation, the sustainability of this program received continuous attention and remains a high priority for the government’s goal of universal basic education, as articulated in its National Education Plan. The government’s Primary Education Development Program 4 is currently co-financed by a US$700 million IDA Credit and by nine other development partners (Quality Learning for All Program/P162619). This program aims to provide quality education to all children from pre- Page 19 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) primary to Grade 5 through an efficient, inclusive, and equitable system. This results-focused operation includes a focus on the learning needs of out-of-school children, extending the delivery model supported under this project to the country as a whole. The specific objective of the ongoing Program is to reduce the number of 8- to 14-year-old children who have either never enrolled in school or have dropped out of school by offering alternative education modalities up to Grade 5. It envisages the enrollment of up to one million children in LCs following the formal primary curriculum, supporting their eligibility to take the Completion Exam, and enabling them to continue their studies in the formal education system. Quality assurance will involve verification of LCs, monitoring and ensuring student and teacher attendance, and adherence to service standards. The Bureau for Non-Formal Education under the MOPME is responsible for implementing these activities. In short, strong government ownership of this approach (reflected in its prominence in the government’s Education Plan and Program, and in the fact that a Bureau within MOPME is assigned responsibility for its continued implementation), combined with the substantial resources devoted to the implementation of the Program (both domestic and international), are strong indicators of its sustainability. Moreover, the government has prepared a Position Paper on education support for DRPs and host communities, providing the basis for preparation of a new project to ensure continuing support to this target group. The positive feedback from parents and communities on their satisfaction with the project’s support, and the strong grounding of the approach by and for communities, with support from non-governmental organizations (NGOs), bode well for continued stakeholder and civil society support. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank’s preparation team helped design the project on the basis of an efficient model developed under the first operation, which had produced results both satisfactory and holding greater promise, all with a view to help the government respond effectively to one of its top priorities: improving access of out- of-school children to a quality education. During the design phase, the team reviewed and incorporated lessons from the earlier phase, while also gradually expanding its reach and potential through pilots of new initiatives (urban areas, domestic workers, pre-vocational education), which were expected to further enhance its impact. Risks were appropriately assessed. Indeed, the two major risks that affected the project (the Rohingya crisis and the pandemic) could not have been predicted at appraisal. The project’s M&E design was strong, especially its innovative and efficient monitoring arrangements; its implementation arrangements through a management information systems (MIS) Cell; a concise and well designed results framework; and the impact evaluations of the project and various pilots, designed to supplement project reporting. While not articulated in the PDO, the goals of poverty reduction and enhancing gender and social development were supported through the targeting of deprived, vulnerable populations, and the prospects of improving their success in school and in the workforce, thus potentially improving their incomes and potential. Fiduciary aspects were well prepared. The local-level delivery arrangements, complementing NGO support with strong community organization, management, and oversight of training, were yet another notable feature of project design, which enhanced management and contributed to better results. Page 20 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank’s joint country office- and headquarters-based team provided implementation support, which was timely, hands-on, and consistent. Reporting in the 16 (internal) Implementation Status and Results Reports and mission aides-memoire (translating into an average of one supervision mission every six months) was thorough, candid, and well justified. The sound implementation of M&E arrangements, supervised and guided by regular supervisions, ensured a strong and continued focus on development impact, with the generation of a rich array of relevant data through monitoring activities and impact evaluations on the project and its various pilots, all the while building and strengthening local capacity to collect and use quantitative and qualitative data to assess performance and outcomes, learn, and improve the effectiveness of interventions. Supervision of fiduciary and safeguard aspects was proactive, supporting corrective measures when minor issues arose. The Bank coordinated closely with teams supporting the design and implementation of the government’s third and fourth Primary Education Development Plans to ensure consistency, complementarity, and sustainability. Sustainability of interventions was addressed throughout the project’s implementation, and, as noted in Section 7 (Risk to Development Outcome), transition arrangements were well thought out and bode well for sustainability and further institutionalization of interventions. The Bank team was quick to respond to the government’s requests for support to address the needs of DRP children under a timely and well-designed AF1, and, under AF2, to respond to expected exchange rate losses in financing and provide proactive support to the government to provide learning continuity in the face of school closures due to COVID-19. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design M&E design built on the experience and capacities of the first project and aimed to further strengthen data collection, validation, and utilization. A separate project component supported these activities. The results framework was well articulated, underpinning a solid theory of change, and the project monitoring system was designed to track progress of well-chosen indicators, focused on inputs, processes, outputs and intermediate outcomes, and outcomes. Outcome indicators, along with planned impact evaluations, were well poised to measure the project’s outcome. The results framework indicators were feasible to collect and monitor. Page 21 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) Monitoring activities were to include: (i) preparation of databases and validation of LCs for the first year of LC establishment; (iii) the preparation by LCs of quarterly self-reported monitoring forms to upazila-level TCs for their certification and submission to the PIU; (iv) quarterly sample-based monitoring of LCs’ adherence to education criteria and procedures; (v) public expenditure tracking surveys; (vi) monitoring of LCs based on classroom visits; and (vii) regular management reviews of all aspects of project performance, including the mid-term review. The design envisaged innovative use of information technology for monitoring and service delivery, including SMS monitoring of grants and allowances disbursements; SMS monitoring of teacher attendance; smart phone use for validation of LC self-reported data; GPS mapping of LCs; LC report cards and websites; and post office cash cards for beneficiaries. Evaluation activities were to include: baseline surveys for IEs; follow-up surveys to assess impact, intermediate outcomes, and outcomes; IEs of pilot interventions; and a pilot of the DPE learning assessment system for two rounds of project student assessments for Grades 3 and 5. M&E implementation responsibilities were assigned to monitoring officers; a data processing and monitoring agency (project MIS Cell at the Local Government Engineering (LGE) Department); an SMS monitoring unit at the Project MIS Cell; and an independent evaluation survey firm and learning assessment agency. b. M&E Implementation M&E activities were carried out in a continuous manner, with proactive resolution of issues that arose through the strengthening of institutional arrangements, both in the PIU and in the field. Minor adjustments to the results framework for its improved streamlining were introduced in a timely manner. The monitoring unit under the MIS Cell provided invaluable service to help collect and monitor data covering over 20,000 LCs. The MIS Cell’s performance was comprehensive and timely, providing much data on students, teachers, LCs, and other information on a regular basis, and carrying out validation and periodic monitoring. The smartphone-based mobile application developed by the MIS Cell monitored attendance of field workers, collected monitoring reports, tracked and monitored teachers' daily work plans through government primary school coordination, and collated attendance summaries, among other aspects. Supervision and classroom observations also provided feedback on performance. Save the Children developed a tracer system to track progress of pre-vocational students and job placements. The Bank’s proactivity refined the results framework to improve monitoring. The pandemic delayed somewhat data collection for the IE, but the IE nevertheless provided useful information on the project’s impact. Methods applied included: (i) quasi-experimental, comparing project and comparison populations; (ii) a regression-based approach; and (iii) qualitative assessment, including focus group discussions, interviews, consultative meetings, and case studies. Households of project and comparison groups were interviewed through structured questionnaires. The Bank team conveyed to IEG during their September 7, 2022 meeting that M&E functions and processing are being fully sustained, in the context of the government’s adoption of a country-wide program based on the project model. Under the project, the MIS was established in the LGE Department, which had the technical expertise to carry it out. Under the government program, the LGE Department is providing technical support and training to hand this function over to Ministry of Education’s Bureau of Non-Formal Education. Page 22 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) c. M&E Utilization Data and information collected through M&E activities allowed for sound monitoring of the project’s progress and informed decisions to adjust the project’s implementation and priorities. Project monitoring data were used for results-based planning and implementation and for removing duplicate or other inappropriate entries for students from the system. LC monitoring and supervision supported the teaching-learning process as a means of tracking and enhancing education quality. LC report cards were prepared by the PIU, presented on its website, and disseminated through local education offices to LCs, CMCs, and parents to inform their decisions and priorities. The IE team found the data collected by the MIS Cell to be extremely helpful. M&E Quality Rating High 10. Other Issues a. Safeguards The Bank’s task team confirmed to IEG (during their September 7, 2022 meeting) that the government complied with the Bank’s safeguards policies and guidelines. The project triggered OP/BP 4.01 (Environmental Assessment) and was classified as Category B (partial assessment). With no new construction/renovation of LCs anticipated, the project was not expected to create any significant long-term environmental impact. LCs were instead to be housed in rental rooms, with special attention to ensure safe drinking water and acceptable sanitation and drainage facilities. An Environmental Management Framework (EMF) was prepared and disclosed by the implementing agency, addressing issues for mitigation and monitoring, and also defining the roles of the CMCs, NGOs, and teachers in selecting LCs, screening/assessment procedures, and standard mitigation and operational health and safety measures. The Indigenous Peoples policy (OP/BP 4.10) was triggered as the project targeted disadvantaged groups, including children from minority races and tribal groups. A Social Inclusion and Management Framework (SIMF) was thus prepared and disclosed to guide implementation to assess presence and characteristics of indigenous peoples and ensure their participation in decisions in a socially and culturally appropriate manner. The SIMF included a Grievance Redress Mechanism (GRM) for ensuring community engagement, transparency, and accountability in implementation. AF1 triggered two additional environmental safeguard policies (OP 4.04 Natural Habitat and OP 4.36 Forests), as activities were to take place within an important forest landscape. It also triggered OP 4.10 given the presence of small ethnic communities (tribal people) in Cox’s Bazar. Because AF1 was processed on an urgent basis, the completion of environmental and social safeguards compliance plans was deferred until after AF1 approval. The Safeguards Action Plan, developed during AF1 preparation, required revision and disclosure of the EMF and SIMF as a condition of effectiveness. LCs under the original design were established in rented spaces, as planned, while those in the Rohingya camps were constructed on allocated sites in compliance with the requirements of the EMF and SIMF. The environmental and social management performance of the government and project partners (Save the Children and UNICEF) was satisfactory. Page 23 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) The GRM, already in place in the DPE, was used for the project. Quarterly discussions at the upazila level, and quarterly meetings between technical coordinators and community management committees, facilitated exchanges and communication of issues and queries to the PIU. Multiple channels (dedicated phone number, Facebook account, project website) were available to receive complaints. Five grievance cases were received during implementation, all of which were investigated and resolved. UNICEF’s prevention of sexual exploitation and abuse was integrated, including guidelines and training. Two incidents of gender- based violence and sexual harassment were recorded in Cox’s Bazar, which triggered the firing of the perpetrators and the provision to the victims of psychosocial support and protection against retaliation. No third-party assessment was carried out. b. Fiduciary Compliance Financial Management The project’s Financial Management (FM) arrangements largely followed existing government policy and procedures, and benefited from the institutional arrangements established for the project. Given the project’s delivery of grants and allowances to a large number of LCs and students, the FM risk was assessed as substantial. Mitigation and actions to improve capacity, agreed at appraisal, included: appointment of a qualified FM consultant and two account staff, procurement of an off-the-shelf computerized accounting system; and training of project finance staff on FM-related aspects. Additional FM staff were appointed following AF1 approval. FM arrangements were satisfactory, despite minor issues related to documentation and disbursement processing. The Bank task team (in its September 7, 2022 meeting with IEG) confirmed that the government complied with the Bank’s FM policies and guidelines. A local bank contracted under a participation arrangement channeled funds for grants to LCs and stipends to enrolled students. All disbursements (grants, allowances, vouchers) were made based on verified data on LCs and institutes in urban areas and on individual students, and were remitted through the designated banking system. Submission of external audits was timely, and none of the audits was qualified. The last project audit had four minor observations that the government acknowledged and was addressing at the time of ICR preparation. Procurement The Procurement Capacity Assessment carried out at appraisal identified several issues, notably a shortage of staff with procurement knowledge in the PIU, and issues with internal control, documentation, information dissemination, contract administration, payments, and handling of complaints, among other things. Consequently, the risk for procurement operation and contract administration was assessed to be substantial, and a number of measures were agreed to help mitigate these risks, including identification of procurement focal points, deployment of a national procurement consultant, use of a Bid Evaluation Committee, establishment of a project webpage with procurement-related information accessible to the public, and a system for handling complaints. Additional procurement staff were appointed following the approval of AF1. The main procurement issue occurred early in the implementation period. The Bank cancelled an amount of SDR 20,309 from the credit in March 2015 following the misprocurement of four contracts slated to be Page 24 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) 100 percent financed under Category 2. The issue was inappropriate bidding practices by the bidders, not identified by the implementing agency. Project staff noted that the mistake was a result of weak capacity in the early years of the project. The government agreed to provide three weeks of procurement training to all staff working on procurement and was requested to refund the cancelled amount to IDA by June 30, 2015. It duly complied. Moreover, a strong procurement team was maintained in the PIU through the end of the project. Strengthening of procurement led to the project’s successful implementation and completion, with all procurement completed mostly in a timely fashion, including the $14 million procurement of a UN agency. The task team confirmed to IEG (in their September 7, 2022 meeting) that the government complied with the Bank’s procurement guidelines. c. Unintended impacts (Positive or Negative) IE findings, based on interviews exploring respondents’ perspectives, revealed that 3 percent of respondents thought drug use among children had decreased; 16 percent thought that school brought back enthusiasm and joy in learning; and 4 percent thought that project schools had a positive impact on the communities’ economic activities. The project also rose to the challenges posed by the COVID-19 pandemic, unanticipated at the time of design and AF1 approval, adapting to help build communities’ awareness and protection against the virus. LC teachers visited households to raise awareness on hygiene, distribute masks and sanitation materials, and collect donations to distribute food and medicine to those in need. Likewise, the pre-vocational training program raised awareness on COVID-related safety issues, and hand-washing facilities, thermal scanners, disinfectant materials, and masks were distributed to pre-vocational training centers. Save the Children adapted the urban slum program to include health and safety information and distributed preventive materials. It also facilitated student families’ access to local safety nets, food assistance, and protective equipment, all facilitating completion of children’s education. d. Other Over and above the outcomes for girls in the communities sampled in the IE, the project had an important impact on women’s empowerment. First, the overwhelming majority (80 percent) of LC teachers were female. These positions increased teachers' social standing in the community, prompting vulnerable community women to seek their advice, an increase in marriage proposals for single women, and the enrollment of a higher number of female students. Second, children’s mothers were involved in the CMCs' and LCs’ management committees. Close to 90 percent of all LC management committee heads, elected by their members, were female. This has enhanced women’s capacity to interact with local actors, government, and other stakeholders, developed their leadership experience, and raised their status in their communities. Third, the IE reports that rates of child marriage fell among young female students. Fourth, female participants in the pre-vocational training program used newly acquired skills to enter the work force (as paid workers or self employed), boosting their social and economic empowerment. According to the Save the Children Completion Report, participation of females in pre-vocational training is increasing due to effective community mobilization. Fifth, the IE’s qualitative survey revealed that 9 percent of respondents thought that child marriage had decreased, 4 percent believed that sexual harassment had decreased, and 6 percent believed that social mobility of girls had increased. On the negative side, it was not possible to carry out the child domestic worker pilot, in view of the lack of cooperation of those who engaged them in this work. Page 25 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E High High Quality of ICR --- Substantial 12. Lessons The following lessons are a subset of the seven relevant and insightful lessons presented in the ICR, slightly recast by IEG:  Non-formal education, delivered at the community level with support from communities and NGOs, offers a useful model for improving access and retention of out-of-school children, while also improving quality, even in the face of exogenous challenges. This delivery model can be adapted to various settings (deprived rural and urban areas and camps for displaced populations) and to different types of education (basic and pre-vocational). The project demonstrated the value of informal education as an effective, low-cost mechanism for service delivery to groups that would otherwise not have access to education, and in a way that is responsive to local context.  Successful adaptation of the (original) rural model of service delivery to accommodate urban slums can be achieved by clustering several (four to eight) LCs in one compound, providing a near-complete school environment, as opposed to individual, more dispersed LCs in rural settings. The urban slum experience has also shown that an accelerated model, facilitating the completion of the five-year primary cycle in three years, can be accommodating of the needs and constraints of older (eight- to 14-year old) students from poor and vulnerable families. Urban settings also present the challenges of frequent family migrations, an important cause of dropouts.  Mobile vocational training centers provide an effective model for delivering skills training in hard-to-reach areas. They help overcome barriers to participation including safety, cost, and travel time, especially for marginalized beneficiaries, thus prompting increased participation of vulnerable adolescents, especially girls.  The contracting of a specialized agency for teacher training can be effective and efficient, and experience points to the importance of enhanced monitoring of training. Experience and feedback also suggest room for further strengthening of teacher training, especially supplementing foundation training with two days of teacher observation and feedback, and providing more support to trainees (increased budgets for expenses, caregivers during their absence, and training materials). Page 26 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) 13. Assessment Recommended? No 14. Comments on Quality of ICR Quality of Evidence was strong. First, the collection, review, and analysis of project data tracking outputs, intermediate outcomes, and outcomes was carried out in a systematic fashion and prepared, reported, and validated on a regular basis, under the overall coordination and technical oversight of a capable MIS Cell. Second, impact evaluations on the project and on various pilots were carried out on the basis of sound methodologies, producing viable results – both qualitative and quantitative – which complemented project data and provided additional insights and lessons for further improvement. There were a few incidences of information and data provided in the efficacy section not being cited or not specifying which target group(s) it applied to (rural LCs/students, urban slum LCs/students, pre-vocational training centers/students, and/or DRP camps/children). In a context of four restructurings and two AFs, Annex 8 provided a clear, succinct account of the project’s outcome and intermediate outcome indicators and targets, as they evolved across these changes, also including baselines and end-of-project achievements. Quality of Analysis was also strong. The efficacy section was well organized around the various project objectives and effectively reported on and analyzed the two main sources of data and information (project data, various impact evaluations) plus others (including Save the Children’s Completion Report on the urban slum pilot and program) to weave together a narrative of the project’s accomplishments, shortcomings, and evolutions, which distilled salient messages and provided a strong evidence base for documenting and rating performance. Quality of Lessons. The lessons were clearly drawn from the evidence and analysis presented in the ICR and linked to its narrative and ratings. While specific to this project’s unique experience, these lessons were articulated in a way that would be relevant and insightful to other countries embarking on similar endeavors to find creative, effective, community-based approaches to extend basic education services to poor, marginalized, out-of-school children. The ICR’s Results Orientation was strong. The evolution of the results framework (and of the project indicators and targets) was clearly articulated, and the efficacy section was well organized around the PDOs and effectively drew on the rich array of project data. Each objective was assessed in terms of the entire results framework (inputs, outputs, intermediate outcomes, and outcomes), providing a clear understanding of the links between the project’s support and its intermediate and ultimate effects. While the PAD did not present a theory of change, the ICR prepared one, which provided an accurate and helpful synopsis of the project. Internal Consistency/Adherence to Guidelines. The various parts of the ICR were internally consistent, providing a robust narrative of this creative, ambitious, and adaptive project. The ICR was respectful of ICR guidelines, with a couple of minor exceptions. First, there was no unequivocal confirmation of government compliance with the relevant fiduciary policies, guidelines, and covenants. This was provided by the Bank team during a follow-up meeting with IEG, held on September 7, 2022. Second, there were some inconsistencies/lack of clarity on the project costs and financing. Total project cost at completion is presented in Text Table 2 (p. 25) and in Annex 3 (p. 55) as $146.88 million equivalent. Annex 3 shows “0s” in the two columns entitled “Amount at Appraisal” and “Actual as Percentage of Approval.” Text Table 2 seems to show Page 27 of 28 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Reaching Out of School Children II (P131394) the original IDA financing and AF1 amounts, but not the AF2 amounts, or cancellations. Text Table 2 shows the planned government financing at appraisal ($7.5 million equivalent or 5 percent of the total original cost) and leaves government financing blank in the “Actual at Completion Column,” with no explanation in the table or in the text of whether or why the government provided (or did not provide) any counterpart financing. The Bank team did confirm in its September 7, 2022 meeting with IEG that the government provided 95 percent of its counterpart commitment. The presentation of original and cancelled amounts of each of the IDA credits and grants supporting the project in terms of SDRs would have helped explain/reconcile the discrepancy in US dollar amounts of IDA financing and disbursements found within the ICR, some downloaded automatically from the Bank’s system, some presented in cost tables, likely a result of different exchange rates used. Reporting use of funds in terms of SDR would also have provided more accurate rates of disbursements of each of these IDA credits and grants. This was ultimately accessed and assessed by IEG.. a. Quality of ICR Rating Substantial Page 28 of 28