Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) Report Number: ICRR0021363 1. Project Data Project ID Project Name P106259 AF: Edu. Qlty. Improvement Program II Country Practice Area(Lead) Afghanistan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H3540,TF-91943,TF-93962 01-Sep-2012 412,517,086.85 Bank Approval Date Closing Date (Actual) 31-Jan-2008 01-Sep-2012 IBRD/IDA (USD) Grants (USD) Original Commitment 30,000,000.00 408,123,582.00 Revised Commitment 29,243,701.70 383,660,094.77 Actual 28,856,992.08 383,660,094.77 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (p. 4), the project's objective was "to assist the Recipient in increasing equitable access to quality basic education, especially for girls." Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) The project received Additional Financing (AF) in 2012, and some outcome targets were revised at a 2015 restructuring. As the targets were revised upward, a split rating is not warranted. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 13-Aug-2015 c. Will a split evaluation be undertaken? No d. Components The project contained three components: 1. School grants (appraisal: US$ 21.8 million, of which US$ 8.7 million from the International Development Association (IDA) and US$ 13.1 million from the Afghanistan Reconstruction Trust Fund (ARTF); AF, US$ 69.3 million; actual, US$ 205.45 million). This component was to support the improvement of teaching and learning by facilitating the creation of enabling school environments, and support the improvement of basic school facilities at existing government-registered primary, middle, and secondary schools with teachers on payroll. It was to finance school grants for quality enhancement (developing linkages between school communities and local government, empowering schools and communities as local education managers through the development of School Improvement Plans), school grants for infrastructure (rehabilitating and constructing schools using community participation and community-based decision making), and social mobilization and awareness (mobilizing and strengthening school Shuras to manage schools based on local priorities). 2. Teacher and principal training and education (appraisal: US$ 48.7 million, of which US$ 16.7 million from IDA and US$ 32.0 million from ARTF; AF, US$ 90.1 million; actual, US$ 140.69 million). This component was to create sustainable systems that would increase the level of professional knowledge and skills of educators throughout the country. It was to provide training to teachers and principals, with increasing focus over time on the creation and development of local professional networks to allow educators to share ideas, problems, and solutions with their peers. The component was to be delivered in partnership with the United States Agency for International Development (USAID)'s Building Education Support to Teacher Training (BESTT) program and have national coverage, with USAID covering eleven provinces and the Bank covering the remaining 23 provinces. Teacher training and follow-on sessions were to be delivered at the district level by newly-established District Teacher Training Teams (DT3s), offering three types of Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) training: pedagogy and basic knowledge (INSET-I), content-knowledge training (INSET-II), and training to educate under-qualified female teachers (Accelerated Learning). Training for principals was to be delivered by non-governmental organizations (NGOs) that were also to contribute to the development of professional principals' networks. The component was also to support increases in the number of women entering Teacher Training Colleges (TTCs, through a scholarship program for girls entering the program and committing to becoming teachers upon completion), teaching in TTCs (through a nation-wide recruiting drive targeting women with Bachelor's degrees), and working as high school teachers in under-served high schools across the country. 3. Monitoring and evaluation (M&E) (appraisal: US$ 9.6 million, of which US$ 4.6 million from IDA and US$ 5.0 million from ARTF; AF, US$ 90.6 million; actual, US$ 76.2 million). This component was to build on and support existing Ministry of Education (MoE) project management structures created under the predecessor project, Education Improvement Program I (EQUIP), and to establish and implement a practical M&E system so that lessons could be drawn in a timely fashion to facilitate project improvement and policy making. As indicator targets were on a path to being achieved or surpassed after the first few years of implementation, the government requested AF in 2012. At that AF, the eleven provinces previously covered by the USAID BESTT project were absorbed into this project (under component 2); three pilots were added on strengthening sub-national planning and budgeting, conditional cash transfers (CCTs) for low-income households with girls in grades 6-12, and introduction of a school operations and maintenance process (under component 3); and support was provided for the MoE's development of a strategy for implementation in insecure provinces. At the August 2015 restructuring, 70 Model School grants were added, an evaluation process to learn lessons from quality enhancement grants was put in place, and a needs assessment and strategy for human resources was developed (under component 1); a teacher management information system (TMIS), TTC curriculum and textbooks for pedagogy courses, and standardized assessment and evaluation system for TTC were developed (component 2); and training, capacity-building, and resource management systems for human resources, infrastructure services, contract management, and an Education Management Information System (EMIS) were developed (component 3). Also under component 3, activities were added to deliver 3.7 million textbooks, develop a web-based geo-referencing information system to function as an on-line educational atlas of the country, establish a gender unit in MoE, and develop a learning assessment covering grades 3, 6, and 9. In addition, all three of the planned pilots that had been added under component 3 at the 2012 restructuring were dropped, to be considered for a follow-on project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) Project cost and financing: The project was originally to be financed by a US$ 30 million IDA Grant, US$ 50.1 million from the ARTF, and US$ 106.6 million from other donors (US$ 22 million from USAID, and US$ 84.6 million in expected financing from "potential development partners and other ARTF contributors," Project Paper, p. 10), for total planned project financing of US$ 186.7 million. The ARTF was a pooled source of on-budget financing for the education sector. In March 2009, the ARTF agreed to cover the entire financing gap of US$ 158 million (the initial ARTF allocation, plus the planned US$ 106.6 million that was to come from USAID and other donors); this was largely in response to the MoE's strategy of having donors channel funds through the ARTF into Bank-administered projects (Project Paper, p. 5). AF of US$ 250 million from the ARTF, to be allocated in two tranches of US$ 125 million each, was approved in 2012. US$ 28.9 million of the original IDA grant and US$ 393.4 million of ARTF and other donor financing were disbursed, such that actual total project cost was US$ 422.4 million. The remaining balance (US$ 14 3 million difference between planned and actual costs, though the undisbursed amount was reported as approximately US$ 16 million in the ICR, p. 67) was due to funds not used as a result of construction delays. The project team later clarified that the unused amount was US$ 6.4 million. Borrower contribution: No Recipient contribution was planned or made. Dates: The project was approved on January 31, 2008 and became effective on March 20, 2008. On March 10, 2009, the ARTF agreed to cover all funds required to support project activities through the initially planned closing date of September 1, 2012. The mid-term review took place in July 2011. On June 12, 2012, AF was approved, some project activities were modified as outlined in Section 2d, and the closing date was extended to August 15, 2014. On September 1, 2012, the IDA project closed. On March 6, 2014, the closing date was again extended to August 15, 2015, to allow implementation of activities supported by the remaining ARTF financing. On August 13, 2015, a level-2 restructuring was approved that modified some project indicators and activities (as outlined in Section 2d) and extended the closing date to December 31, 2016. In December 2016, the closing date was extended a final time to December 31, 2017. 3. Relevance of Objectives Rationale At the time of appraisal, Afghanistan had not fully emerged from a state of conflict. Despite challenging security and poverty contexts, since 2002 the country had succeeded in increasing school enrollment from one million to nearly six million children in grades 1-12, and girls' access from practically zero to nearly 35% of total enrollment (Project Paper, p. 1). Significant shortcomings remained, however, related to the still- large out-of-school population of students, with high gender and provincial disparities; low qualification and uneven deployment of teachers; shortfalls of textbooks and other materials; lack of usable school buildings, exacerbated by continued terrorist attacks on schools; low levels of high school graduation and admission to university; overall high illiteracy levels (28-37% of the population); and urgent need for civil service reform Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) of MoE personnel, including principals and teachers. The project's objectives were highly relevant to this country context. This project was a follow-on to the Bank's EQUIP project (P083964, US$ 35 million, 2004-2010), whose objective was to improve the quality of educational inputs and processes. The objectives of this subsequent project were a logical next phase, given country conditions and the number of children, especially girls, who remained out of school. The project widened the scope of the Bank's intervention to a nation-wide, multi-donor supported effort fully aligned with the government's first National Education Strategy Plan (NSEP 2007-2010), a five-year roadmap developed in 2006 prioritized into eight programs: literacy, curriculum, teacher education, construction, public administration and management, vocational education, and general education. Specific government goals for the education sector, aligned with Afghanistan's Millennium Development Goals, included net enrollment in primary school of 60% for girls and 75% for boys, operationalization of a new secondary school curriculum, a 50% in the number of female teachers, a 70% competency test pass rate for all teachers, and the establishment of a national system for assessing learning achievement. The objectives were also aligned with the Bank's Interim Strategy Note at appraisal (FY 2007-08), which had a pillar to build the capacity of the state and its accountability to citizens to ensure the provision of services that are affordable, accessible, and of adequate quality. The objectives remained highly relevant to the Bank's Country Partnership Framework (CPF, 2017-2021) at closing, which contains an objective on improved human development through increased access to quality education. The CPF has outcome indicators on primary school completion rates, ratio of girls to boys attending primary school, and improvement in the quality of primary school graduates, all aligned with this project's objectives. Similarly, the objectives remain relevant to the country's second and third NSEP (2010-2014 and 2017-2021), as well as with the National Priority Program -- Education for All, which had priority areas around providing adequate learning environments (especially for girls), improving access and retention, upgrading teachers' skills and qualifications and the quality of classroom teaching and learning, and strengthening the institutional capacity to develop and implement sound policies and management practices in the sector. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase ACCESS to quality basic education Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) Rationale The project's theory of change held that increased access to basic education, measured by school enrollment, would be improved through the construction and rehabilitation of school facilities and through the staffing of those schools with reasonably qualified teachers and principals. The project aimed to support facilities through the school grants program, which supported infrastructure construction and rehabilitation, and to fulfill staffing needs through teacher and principal training and through in-service education for teachers. The ICR (p. 19) noted that the project was one of the main contributors to school construction, but that UNICEF, USAID, Denmark, non-specified NGOs, and international military forces through Provincial Reconstruction Teams were also active. The ICR did not provide data on enrollment in areas specific to schools constructed by the project, and therefore precise attribution of observed outcomes to the project is not possible. Outputs 1602 schools were built under the project, not reaching the target of 1758. An additional 114 school buildings were under construction but incomplete as of project closing; the project team provided additional information that the number of incomplete buildings, as of January 2019, was 112, of which 109 are school buildings and three are administrative buildings. 8541 additional classrooms were built, essentially meeting the target of 8571. It is not clear how the project determined where to build new schools and classrooms. The ICR (p. 21) noted that this construction represented an "increase in rural schools," and that "many of the poorest areas were targeted" (p. 30), but no specific information was provided. The project team added that the project did not create any new schools, but instead provided infrastructure to existing schools that were located in open areas or under tents, focusing on the most rural or remote locations. 115,000 teachers were recruited or trained under the project, far exceeding the target of 30,000. 11,436 teachers received scholarships to TTCs, exceeding the target of 9,500. 14,992 schools with school Shuras were established, exceeding the target of 13,969. 13,000 Shura members were trained on development of School Improvement Plans and on social mobilization to promote awareness of the importance of education, especially for girls. The project's Social Mobilization Unit (SMU), with teams at the national, province, and district levels, provided cascade-style training on completion of forms, development of school maintenance plans, data collection and management, administration of learning assessments, and other areas. The ICR (p. 22) noted that, between 2009 and 2017, the SMU was able to Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) open 258 closed schools in insecure areas that catered to more than 13,336 students, 35% of them girls, but it did not explain the specific mechanisms by which the SMU opened closed schools. The project team clarified that the SMU was able to re-open schools through mobilizing community members to negotiate with opposition groups in insecure areas, and that social mobilization supported by the project was instrumental in making connections between provincial education directorates, members of the community, and, in some cases opposition groups in districts controlled by anti-government elements. The project provided extensive technical assistance to MoE across departments including teacher education, procurement, planning, learning assessment, audit, information technology, finance/accounting, social mobilization, and curriculum. A needs assessment for the human resources required to deliver education services was completed, as was a strategy for delivery of these services. 26 district-level education offices were rehabilitated. There were 8.9 million project beneficiaries, almost meeting the target of 9 million. Outcomes The number of boys enrolled in grades 1-3 increased from 1.28 million in 2008 to 2.0 million in 2017, meeting the target of 2.0 million. The number of boys enrolled in grades 4-6 increased from 1.24 million in 2008 to 1.73 million in 2017, exceeding the target of 1.50 million. The number of boys enrolled in grades 7-9 increased from 0.65 million in 2008 to 1.15 million in 2017, exceeding the target of 1.0 million. The number of boys enrolled in grades 10-12 increased from 0.22 million in 2008 to 0.63 million in 2017, not meeting the target of 0.7 million. Overall, the number of boys enrolled in grades 1-12 increased from 2.1 million in 2008 to 5.5 million in 2017, exceeding the target of 5.2 million. The number of girls enrolled in grades 1-3 increased from 0.88 million in 2008 to 1.3 million in 2017, not meeting the target of 1.6 million. Enrollment of girls in grades 4-6 increased from 0.69 million in 2008 to 1.1 million in 2017, meeting the target of 1.1 million. Enrollment of girls in grades 7-9 increased from 0.28 million in 2008 to 0.7 million in 2017, meeting the target of 0.7 million. Enrollment of girls in grades 10-12 increased from 0.08 million in 2008 to 0.3 million in 2017, not meeting the target of 0.4 million. Overall enrollment of girls in grades 1-12 increased from 1.9 million in 2008 to 3.4 million in 2012, not meeting the target of 3.8 million. The ICR (p. 19, fn 11) noted that enrollment in Afghanistan is defined as children who are registered for school; these data therefore do not necessarily mean that the students captured by these indicators were regularly attending school. Because of improvements in data quality, however, the MoE was able to Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) distinguish between students who were enrolled and those who actually attended regularly. The ICR did not report attendance data. Rating Substantial OBJECTIVE 2 Objective Increase EQUITABLE access to quality basic education, especially for girls Rationale The project's theory of change defined equity in terms of gender. It held that girls' enrollment in school would increase if facilities were more girl-friendly, including provision of gender-segregated latrines; if there were more female teachers; and if communities and families were sensitized to the importance of educating their girls. To this end, the project was to support social mobilization and awareness efforts to encourage communities and families to send girls to school, teacher training grants specifically for women, and teacher recruitment efforts aimed at qualified women. Outputs The project established a gender unit in MoE in 2012, providing support to curriculum development, teacher training, infrastructure services, and social mobilization. The project itself had a gender coordinator. The MoE's gender unit/team supported gender training of project provincial officers, resulting in the hiring of 500 female teachers in three provinces (though the ICR, p. 28, did not explain the mechanism through which the gender training produced increase hiring of female teachers), installed a helpline for recording harassment cases (though an analysis of the helpline has not been completed), and used EMIS data to assess barriers to education for girls. The project team clarified that the training increased hiring of female teachers by raising awareness of the importance of female teachers for girls' schools. According to the ICR (p. 28), school construction grants were prioritized to girls' schools, schools with both boys' and girls' sections/shifts, and boys' schools that planned to open sections/shifts for girls. The ICR also stated that the project "increased the focus of construction on the inclusion of boundary walls, water, and latrines for schools to provide a safe environment for girls and encourage parents to send their girls to school." The project team clarified that the project financed the construction specifically of boundary walls, water, and latrines in schools that already had a functioning building but were lacking these elements. Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) 11,436 scholarships were provided to females to attend TTCs, exceeding the target of 9,500. Of the scholarship recipients, 5,298 (46.3%) were tracked through a survey conducted by the MoE's Teacher Education Directorate. Of those who were tracked, 46% were hired as public school teachers, 3% took other government jobs, and the remainder continued their studies or are unemployed. It is noteworthy that over half of the scholarship recipients were not tracked (it is not known whether they completed the scholarship- financed training, or where/whether they are employed). In addition, gender grants were provided to TTCs based on the number of female students enrolled, to incentivize them to enroll more females. These gender grants were to be used to provide more facilities and a better learning environment for females studying to become teachers. The ICR (p. 28) also stated that quality grants in the amount of US$ 10,000 were provided annually to every TTC through 2015, regardless of female enrollment; the ICR did not state what number or percentage of TTCs received gender-specific grants, and therefore what the level of incentive was for TTCs to prioritize training of female teachers. The project team added that 41 TTCs and 111 Teacher Development Centers received gender grants. TTCs reaching more than 500 female students received US$ 10,000; those reaching 201-500 female students received US$ 8,000; and those reaching 10-200 female students received US$ 6,000. According to the ICR (p. 28), school management training focused on increasing female participation in Shuras. Outcomes The ratio of girls to boys in grades 1-3 declined from 69 in 2008 to 66 in 2017, not meeting the target of an increase to 79. The ratio of girls to boys in grades 4-6 increased from 56 in 2008 to 66 in 2017, not meeting the target of 69. The ratio of girls to boys in grades 7-9 increased from 45 in 2008 to 60.9 in 2017, not meeting the target of 62. The ratio of girls to boys in grades 10-12 increased from 37 in 2008 to 47.6 in 2017, not meeting the target of 57. The proportion of teachers who are female increased from 30% in 2008 to 33% in 2017, not reaching the target of 47%. Between 2008 and 2016, the completion rate for grade 3 improved by 28.8% for girls and 33.9% for boys; for grade 6, by 34.8% for girls and 28.1% for boys; for grade 9, by 72.4% for girls and 62.4% for boys; and for grade 12, by 79.3% for girls and 70.4% for boys. For grades 6, 9, and 12, the growth in completion rate was higher for girls than for boys (ICR, p. 22). The ICR did not provide the data points from which these increases in completion rates by gender were derived. Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) Of the total number of project beneficiaries, 39% were girls, not meeting the target of 42%. Rating Modest OBJECTIVE 3 Objective Increase equitable access to QUALITY basic education, especially for girls Rationale The project's theory of change held that community involvement, teacher training, and assessment/validation of school activities and learning outcomes would improve the quality of education. The project therefore supported quality enhancement grants to schools, teacher training, the establishment of school Shuras, and the development and implementation of management information systems and assessments of student learning outcomes. Outputs 16,588 schools received Quality Enhancement Grants (QEGs) during the project period. The proportion of public schools receiving QEGs increased from 26% in 2008 to 99% in 2017, essentially meeting the target of 100%. With the QEG funds, 8,466 schools established libraries (with an estimated utilization rate of 83% in 2015); 5,732 established laboratories (with an estimated utilization rate of 86%); and 6,275 purchased computers (with an estimated utilization rate of 80%). The ICR did not define "utilization rate" or explain how it was measured; the project team clarified that utilization rate referred to use by students. The ICR (p. 21, fn 13) cited quality issues with school construction, including variance in completion of the 1602 project-constructed schools. Most of the project-supported schools were complete (with boundary walls, latrines, and furniture). 424 cases were defined as having "missing elements," meaning that "only a school building was provided and the remaining components were categorized as missing components or annexes." 415 of the schools were defined as "incomplete," meaning that they were functional but not completed according to the terms of the contract, with missing elements including galvanized iron sheets or furniture (ICR, p. 31). Overall, 162 schools (about 10% of the total) were categorized as both incomplete and with missing elements. Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) The project provided grants of US$ 15,000 to each of 73 model schools. These schools were intended to reinforce an active learning environment, with the hope of producing a demonstration effect for other schools in their communities According to the ICR (p. 22), an MoE analysis indicated that these schools had improved school environments, more effective discipline, increased student enrollment, enhanced teacher performance, better physical facilities (toilets, drinking water, playgrounds, laboratories, libraries, and computer labs), and higher levels of student satisfaction, but no detail of this analysis was provided. The ICR also did not indicate how many students were covered by these project-supported model schools, or whether the desired demonstration effect was realized. The project team added that these schools covered almost 178,000 students, 42% of whom were female, in 34 provinces, and that a study of 24 of these schools (involving 288 student interviews) compared with 24 regular, non-model schools found the model schools to have an improved environment, better facilities, higher student satisfaction, more discipline, and higher- quality teaching (see also ICR, p. 22). A system for learning assessment was put in place at the primary level (grades 3 and 6). The ICR (p. 22) stated that assessment "is the engine that drives student learning because it encourages active learning," but no detail on how this was the case in Afghanistan was provided. Although the original results framework included learning outcomes as a key project outcome indicator, and the first-ever national learning assessment for grades 3 and 6 was implemented, the ICR (p. 10) cited learning outcomes only as a longer- term benefit, and the results of the implemented learning assessments were not reported in the ICR. A 5% sample check of the school EMIS was implemented with third-party verification, meeting the target. However, a planned school information/report card pilot was not carried out. The proportion of teachers who have professional qualifications increased from 28% in 2008 to 58% in 2017, exceeding the target of 48%. 100,833 teachers completed in-service teacher education (INSET-I, pedagogy and basic knowledge), exceeding the target of 100,000. 93,985 teachers completed INSET-II (content knowledge), not reaching the target of 100,000. 115,814 teachers completed INSET-III, exceeding the target of 100,000. 154,699 teachers completed INSET-IV, exceeding the target of 100,000. 154,811 teachers completed INSET-V, exceeding the target of 100,000. 84,831 teachers completed INSET-VI, exceeding the target of 73,000. The ICR did not define INSET III through INSET VI; the project team clarified that the numerical designations referred to modules covering different areas, and that they were sequential in scope. 21,277 school administrators and principals completed a management training program, exceeding the target of 14,000. Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) 99% of schools received textbooks as planned, exceeding the target of 90%. Outcomes The proportion of secondary schools with science labs reached 32%, essentially meeting the target of 33%. The percentage of all schools with drinking water facilities increased from 50% in 2008 to 65% in 2017, not reaching the target of 80%. The percentage of all schools with latrines, toilets, or urinals increased from 49% in 2008 to 68% in 2017, almost meeting the target of 70%. A teacher training review conducted by the Pakistan Institute of Social and Policy Sciences found that "there had been considerable progress toward improving the quality of teachers and education management from a very low baseline" (ICR, p. 24). An evaluation of the project-supported Diploma in Teacher Education (DTE) showed that it had advantages when compared to similar pre-service teacher education programs in the South Asia region (India and Pakistan), specifically that it allocated more time to practicum, gave attention to subject knowledge deficit, and included an action research component to build capacity for reflective engagement with teaching practice. The evaluation found that the DTE program covered 78% of required competencies under the teacher competency framework. It also found that the INSET programs provided "adequate coverage" of required competencies. However, the ICR (p. 32) also cited an Early Grade Reading Assessment conducted for grades 2 and 4 indicating that shortcomings remain in teacher competencies and qualifications, that a relatively low percentage of TTC graduates have been hired as teachers, and that "teacher recruitment is often prone to corruption and nepotism" (ICR, p. 32). The student-to-classroom ratio decreased from 74 in 2008 to 60.2 in 2017, exceeding the target of 70. The number of students completing the last stage of a grade increased from 2008 to 2017 as follows: grade 12, from 0.06 million to 0.21 million, exceeding the target of 0.20; grade 9, from 0.11 million to 0.32 million, almost meeting the target of 0.34 million; grade 6, from 0.4 million to 0.56 million, exceeding the target of 0.50 million; and grade 3, from 0.51 million to 0.75 million, almost meeting the target of 0.78 million. Rating Substantial OVERALL EFF TBL OLD Rationale Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) The ICR (p. 19) noted that Afghanistan experienced a resurgence of conflict beginning in late 2014, such that progress toward outcomes was uneven, with drops especially in 2016. The ICR (p. 28) also acknowledged that girls' attendance is still low relative to boys due to security concerns, cultural issues of families not allowing girls to attend, and lack of a school within a reasonable distance of girls' homes. Despite not reaching targets for girls' access to education and therefore for equity, the project substantially achieved objectives related to overall access and to quality. Overall efficacy is therefore rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency At the project fell under OP 8.0, economic and financial analysis at appraisal was not feasible. Given the number of children, especially girls, out of school, the proposed investment was "clearly justified in light of the potential high returns to basic education, as universally acknowledged" (Project Paper, p. 7). The ICR (pp. 62-68) assessed efficiency along three dimensions: the economic benefits of increased education attainment due to increased access; the benefits from improved education quality; and the economic and social returns to increased equity for girls. Cost estimations were limited to the first and second components, plus costs of the third component that were likely to have contributed directly to improved education benefits (15% of the total component costs, based on the percentage of staff supported by the component who were directly involved in project coordination). Drawing from international literature, the ICR (p. 63) acknowledged that the returns on education in Afghanistan are low compared to other countries and economies, likely due to the slowdown in economic growth and conflict situation in recent decades. Despite these low returns, however, education remains a key determinant of participation in wage employment. Using the mean rate of return in Afghanistan to an additional year of schooling of 1.6 found by Montenegro and Patrinos (World Bank Policy Research Working Paper 7020, 2014), enrollment rates throughout the project's lifetime, and reported average annual income for unskilled workers as the wage base, the ICR found a net present value ranging from US$ 414.1 million (worst-case scenario) to US$ 2.9 billion (best-case scenario), with an internal rate of return ranging from 31% to 43%. The worst-case scenario assumed that all project schools would close operations immediately after project closure. The best-case scenario assumed that schools would continue to enroll students for another ten years following project closure. The assumptions used in the analysis were quite conservative, not capturing students who were already enrolled in schools prior to the project start date of 2008, and using a 5% discount rate. The ICR did not conduct a sensitivity analysis under different discount rates. Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) It is likely that efficiencies were realized through the development of the ARTF, which moved the country's education sector from donor-driven and fragmented to a situation in which MoE could take control of the sector dialogue with donors, "an important achievement" especially in a fragile environment (ICR, p. 30). The ICR, however, noted several important elements of implementation inefficiency. The project faced "severe challenges with the construction component" due to data misreporting and limited oversight of community contracts, resulting in a presidential decree mandating that construction activities be shifted away from the MoE to ministries having core responsibility for construction (Ministry of Urban Development and Housing for urban areas; Ministry of Rural Rehabilitation and Development for rural areas) (ICR, p. 21). Issues of insecurity and the resulting lack of mobility for both the Bank and MoE to monitor progress impacted supervision of construction throughout the project's lifetime, though overall low capacity and lack of coordination among MoE departments also impacted efficiency of construction (ICR, p. 31). A tracking unit formed in late 2015 to monitor implementation identified problematic construction sites, but "senior management did not act on its recommendations" (ICR, p. 32). Overall, the unit cost of building a classroom for 40 students increased by approximately 140% over the project's lifetime due to the unstable security context and low fiduciary management capacity of the MoE (ICR, p. 67). The ICR (p. 26) noted that project extensions were due to delays in school construction and "weaknesses in financial management and procurement" (see also Section 10b), and that construction delays accounted for 73% of the remaining project balance (approximately US$ 16 million) at the end of the implementation period. There were also significant shortcomings in procurement and financial management (see Section 10b). Due to the extent, causes, and impact of the construction delays, as well as shortcomings in fiduciary compliance, project efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 100.00 ICR Estimate  43.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) The project's objectives were highly relevant to country context, government strategy, and current Bank strategy. Achievement of the project's objectives to increase access to quality education was substantial, but equity was only modestly achieved, given failure to reach targets for girls' enrollment and boy-girl ratios. Efficiency was modest, given delays and other challenges around school construction and financial management. These ratings indicate moderate shortcomings in the project's preparation and implementation, and therefore overall Outcome is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The follow-on project ("Eqra (Read) Project," P159378, 2018-2023, US$ 280 million), with objectives to increase equitable access to primary and secondary education in selected lagging provinces, especially for girls, and to improve learning conditions, will continue many of this project's activities. However, substantial risks remain. MoE continues to face cumbersome internal processes, lack of departmental coordination, and limited procurement and financial management capacity. It continues to rely heavily on technical assistance. Corrupt practices are a challenge, particularly with regard to teacher recruitment (ICR, p. 29). Most importantly, growing insecurity and recurrent security incidents restrict mobility throughout the country, limiting MoE's capacity to deliver services and the overall functioning of school and government services. 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared as a rapid response operation under OP/BP 8.00, Rapid Response to Crises and Emergencies, as was the case for all IDA grants for Afghanistan at the time due to the continued conflict and persistent fragility of state institutions. According to the ICR (p. 30), the Bank played a significant role in increasing the credibility of the MoE and its capacity to coordinate donors through use of the ARTF. The Project Paper envisioned supervision and implementation support missions at least three times/year. The results framework logically and plausibly linked planned activities and outputs with expected outcomes, and the indicators were appropriate measures of progress toward the development objectives. The planned third-party verification and evaluation process was an important step toward ensuring that data and results were accurately reported. Though the project was a direct follow-on to a first education quality improvement project, the appraisal document provided no Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) focused, specific discussion of lessons learned from prior experience. The ICR (p. 30), however, cited an unnamed study specifically linking lessons from the first project to design of this one, including the need to strengthen community participation processes, the importance of supporting principals and teachers as leading players in school management committees, the ways in which local communities can participate in monitoring processes through School Improvement Plans, the need for planning and documenting (in a qualitative and quantitative manner) of principal and teacher training, and the importance of capacity development spanning national, provincial, district, and community levels. Risk analysis was cursory but appears to have been accurate (Project Paper, pp. 12-13), with a number of risks rated High (overall inherent country risk, ongoing security risks, project financial management risk, and the overall project risk rating), and mitigation measures bringing these risks down to Substantial. In hindsight, it appears that more intensive planning and capacity development might have strengthened the MoE's ability to construct schools more cost-effectively. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Security challenges, heightened since late 2014, resulted in fewer supervision missions than originally planned, lowered MoE capacity to monitor service delivery, and relocation of Bank staff from Kabul to Dubai in 2014. Third-party monitoring helped ensure completion of most projects. The Bank team consistently documented progress toward development objectives, as well as financial management and procurement progress. The ICR (p. 37) reported on "consistent supervision" by the Bank team, both in- country and from Dubai, as key to project support. The task team was heavily involved "on a daily basis" in seeking solutions to situations where school construction had halted or was incomplete, including guidance to the MoE to improve basic drawings and cost estimates. The team provided a high level of supervision under extremely challenging circumstances. Safeguards compliance was problematic (see Section 10a), but it appears that the Bank took all available measures under highly challenging conditions. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) a. M&E Design The MoE was to monitor progress against the indicators included in the results framework, with the project management unit (EQUIP Coordination Unit) responsible for creating timely and comprehensive reports. The third component included detailed provision for both monitoring and evaluation (Project Paper, pp. 20-21), including checklists to ensure that school grants were being used for appropriate purposes; NGO/community/Shura/DT3 monitoring and reporting of student enrollment as well as teacher and principal training; support for female teachers being tracked by the MoE's Teacher Education Department; and the EQUIP M&E team collecting data, entering information into the EMIS, and analyzing results to highlight shortcomings, successes, and lessons. Also as part of the third component, independent evaluators were to assess achievement across all components and the project as a whole, focusing on thorough evaluation of a select sample of districts. Indicators and baselines were well established in the results framework (Project Paper, pp. 24-25), with access to be measured by percentage increase in enrollment for both girls and boys, equity to be measured by percentage increase in enrollment for girls, and quality to be indicated through the establishment and administration twice (year 2 and year 4) of student learning outcome assessments (reading and numeracy tests). These indicators were simple and measurable, and they adequately captured achievement of the objectives. However, that there was inconsistency in the planned use of learning outcome data. While results of reading and numeracy tests were originally included as key project indicators, it was also noted that quality was not to be measured by the results of learning assessments, but instead by the fact of their administration; the latter approach was considered to be appropriate, given government and sector capacity, data availability, and the country's development status. b. M&E Implementation Reporting of data on indicators in the results framework was done systematically throughout the project's lifetime. Third-party monitors visited nearly 80% of all school sites supported by the project, some more than once. The ICR (pp. 32-33) reported that the third-party monitoring system "was instrumental in reporting progress and problems to the Bank so that implementation concerns could be addressed on an ongoing basis." Data were verified using a 5% sample. Two overall project evaluation reports were produced by the independent monitors (in 2013 and 2015). The I-SAPS study independently assessed training activities under the second component. The project conducted the first-ever national learning assessment for grades 3 and 6 (though the ICR did not report the results of this assessment). It also developed a web-based atlas of schools based on Geographic Information System data. Education statistical yearbooks were published, but not on an annual basis. c. M&E Utilization Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) According to the ICR (p. 34), EMIS data were used by the MoE, the Bank, and other donors for monitoring the education system and making investment decisions.The ICR (p. 33) reported that studies and lessons from this project have been "instrumental" in informing the design of the follow-on project. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was environmental assessment category B. The Project Paper did not contain a specific list of triggered safeguard policies, but the discussion in Annex 8 (pp. 54-76) implied that the project triggered OP/BP 4.01, Environmental Assessment; OP/BP 4.10, Indigenous Peoples; OP/BP 4.11, Physical Cultural Resources; and OP/BP 4.12, Involuntary Resettlement. The Environmental and Social Safeguards Framework (ESSF) developed for Afghanistan was to be applied to this project, with the infrastructure grants to use approved standards, designs, and community-based participatory approaches. The ESSF also contained guidelines for documentation and procedural requirements in case of land acquisition or voluntary donations of land. Independent third-party evaluation of civil works was to be carried out in selected provinces at least once during the project period. According to the ICR (p. 35), the project faced challenges with safeguards, and ratings fluctuated throughout implementation. Safeguard ratings were downgraded to unsatisfactory in 2011 after a thorough review by a safeguards team, due to failure to print and distribute local-language versions of the ESSF, conduct planned awareness-raising programs, properly document and legally transfer community land donations for schools, ensure that community school construction met basic quality requirements, and formalize grievance systems. Adjustments were made, and the rating was subsequently upgraded to moderately satisfactory. In 2014, a grievance redress mechanism was developed, but by the following year this mechanism was "showing signs of weakness" related to land acquisition documentation and school construction quality. By project closing, the safeguard rating was downgraded to moderately unsatisfactory because grievances were not being logged, there was no follow-up to complaints, and land donations were still not being properly documented (ICR, p. 35). b. Fiduciary Compliance According to the ICR (p. 35), financial management performance fluctuated between moderately satisfactory and unsatisfactory throughout the life of the project. The unsatisfactory ratings during implementation stemmed from lack of responsiveness to audit observations, failure to share 2014 and Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) 2015 internal audit reports with the Bank, an overdue refund of an ineligible expense, and outstanding advances to provinces. At closing, the rating remained unsatisfactory due to a "significant number" of outstanding advances, lack of recording of fixed assets, and disbursed funds that exceeded physical progress and were likely to be declared ineligible. There were no outstanding quarterly unaudited interim financial reports at closing. The final audit report was due on June 30, 2018. The project team added that the final audit report has been completed, and MoE has responded to its observations. The ICR (p. 35) stated that the Bank was conducting an in-depth fiduciary review of the project. The project team added that this review has been completed, but the report is not available as of January 2019. During implementation, procurement was rated moderately unsatisfactory to unsatisfactory (ICR, p. 36). Issues related to overall limited capacity, an inadequate contract management system, weak tracking monitoring and reporting, inadequate coordination among units, and non-compliance with Bank procurement guidelines. As a result, full payment of contracts did not correspond to physical progress, there was non-compliance with Bank procedures in the recruitment of consultants, procurement methods were used without the Bank's no objection, and supporting documents were not provided for post procurement reviews. These challenges produced school construction delays. Over time, the project developed a database for contract management, but "there was limited use of the system." At project closure, procurement capacity "remained low" (ICR, p. 36). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory More consideration at entry could have been given to Bank Performance Satisfactory Moderately Satisfactory development of Ministry of Education capacity to plan and monitor school construction. Quality of M&E Substantial Substantial Quality of ICR --- Substantial Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) 12. Lessons The ICR (pp. 38-40) presented several insightful lessons, including: Heavy reliance on technical assistance with a separate coordination unit does not effectively build institutional capacity. Although the use of a dedicated project coordination unit was seen as necessary because of the project's emergency nature and weak existing capacity within MoE, it will be important moving forward to develop a permanent cadre of middle-level managers within MoE (a lesson that has been incorporated into the design of the follow-on project). Third-party monitoring using local organizations and NGOs can be effective in conflict areas. In this case, when Bank staff were unable to continue regular site visits due to the security situation, third- party monitoring enabled continued reporting on project activities and outcomes. Strong management and supervision, attention to capacity development, and detailed prior planning are essential to the success of large construction components in low-capacity, fragile environments. In this case, construction was initiated without sufficient planning and precaution, resulting in the design of schools not suitable to local conditions, insufficiently detailed school drawings and technical specifications, substantial delays due to procurement challenges, and poor and/or incomplete construction. Increasing girls' enrollment requires a balanced strategy incorporating both supply and demand factors, and especially careful, deliberate attention to cultural and security concerns. In this case, infrastructure investments (boundary walls, separate toilets, water supplies) and scholarships for girls to attend teachers' training colleges were not sufficient for reaching gender targets. Although the project included innovative social mobilization strategies, girls' enrollment remained suppressed due to family pressures for girls not to attend school. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Edu. Qlty. Improvement Program II (P106259) This evidence-based ICR presented an in-depth narrative of the project's challenges and achievements. It situated the project effectively within a challenging security and governance context. However, it had several shortcomings. The ICR (p. 27) conducted a split rating based not on revisions in the project's objectives or outcome targets, but instead on "worsening country conditions due to increased conflict and security challenges"; this methodology did not follow established guidelines. The ICR made several general statements about project activities without providing supporting detail (for example, the targeting of school construction to poor and rural areas). It was lengthy (40 pp.), perhaps justified by the project's length, complexity, and changes during implementation. a. Quality of ICR Rating Substantial Page 21 of 21