Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Report Number : ICRR0020103 1. Project Data Project ID Project Name P089761 RY Sec. Educ. Dev. and Girls Access Country Practice Area(Lead) Yemen, Republic of Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-44010,TF-94223 31-Jan-2015 103,400,000.00 Bank Approval Date Closing Date (Actual) 18-Mar-2008 31-May-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 20,000,000.00 14,026,750.00 Revised Commitment 20,000,000.00 6,616,750.00 Actual 19,125,488.09 6,615,000.00 Sector(s) Secondary education(88%):Central government administration(6%):Sub-national government administration(6%) Theme(s) Education for all(67%):Gender(33%) Prepared by Reviewed by ICR Review Coordinator Group Estelle Rosine Raimondo Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives Project Development Objectives The objective of the project is to support the Recipient's Program for improving gender equity, quality, and efficiency of secondary education in selected districts with a particular focus on girls in rural areas within the Project Area (Financing Agreement, 2008, p. 5). The Project Development Objective was identical in the Project Appraisal Document (PAD, p. 5). The project was supported through a multi-donor trust fund. The objective of the grant was to support the implementation of the Secondary Education Development and Girls Access Project (SEDGAP), which was designed to support the achievement of the strategic objective of the Government of Yemen's National General Secondary Education Strategy (NGSES). The latter aimed at reducing poverty through social and economic development (Project Grant No. TF094223, p. 5). As the key outcome targets were revised at a May 2014 restructuring, a split Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) rating will be used. At the time of project restructuring, the IDA credit has disbursed US$10.96 million (ICR, p xiii, Table H), or 57.3% of the US$19.12 million in actual IDA financing. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 11-May-2014 c. Components Original components (as approved; the ICR does not provide actual costs by component at the time of restructuring): The original proposal consisted of financing a series of interventions in 9 Governorates (Abyan, Aden City, Dhamar, Hadramout, Hodaidah, Ibb, Lahej, Sana’a city and Taiz). Following a phased approach, the project intended to first target 5 governorates in phase 1 (to last about 18 months after the start of the project). The project was to benefit approximately 90 schools in 30 districts, 6,800 students through school community grants, and 30,000 teachers through training. The four original components were as follows: Component 1: Improving equity and reducing gender gaps through demand-and supply-side interventions (US$74 million total; IDA $15 million). This first component had four sub-components that aimed to improve school facilities and learning equipment, as well as build schools' capacity to manage and ultimately implement school-community grants. More specifically: • Sub-component 1.1 sought to upgrade and furnish school facilities (US $53.7 million total; IDA US$11.0 million) by extending and rehabilitating existing facilities. This sub-component also included the supervision of civil work sites during the construction phase. • Sub-component 1.2 was to focus on the provision of learning equipment and materials (US $12.3 million total; IDA US$2.4 million) comprising of, inter-alia, science and computer laboratories, as well as materials in line with the existing curriculum • Sub-component 1.3 sought to develop operating procedures and build the capacity for managing school-community grants (US$ 7.0 million total; IDA: US$1.4 million) which included financing technical assistance to develop manuals on school-community grant scheme implementation; as well as intensive training to mothers’ and fathers’ councils and school management. • Sub-component 1.4 consisted of the provision of school-community grants (US$7.0 million total; IDA: US$1.4 million). Under this sub- component, a mix of subsidies and conditional cash transfers (CCTs) were to be financed as targeted incentives to students through school community allocations. These included (i) subsidies to arrange transportation to school; (ii) subsidies to contract female staff in schools; and (iii) CCTs to girls in grades 4-12 and boys in grades 10-12 specifically in districts where more than 50% of the population was below the national poverty line. Component 2: Enhancing the quality of service delivery (US$23.4 million total; IDA US$4.0 million) comprised three sub-components • Sub-component 2.1 focused on building the capacity of teachers to improve their teaching and learning practices in the classroom (US$ 22.0 million total; IDA: US$3.7 million). This comprised the development of upgraded teaching and learning materials aligned with the existing curriculum, and with an ongoing revision of textbooks; the delivery of training to support teachers and school teams; and strengthening of managerial and leadership capacity of school principals. • Sub-component 2.2 sought to strengthen the learning assessment system (US$0.8 million total; IDA: US$0.2 million) once the Center of Measurement and Evaluation was established. The project aimed to introduce new examinations in the school systems and strengthen the capacity of staff in schools to assess student performance based on standards in syllabi and learning materials. • Sub-component 2.3 pursued the improvement of planning, programming, and data collection and analysis (US$0.6 million total; IDA: US$0.1 million) through training of central and district level staff of in each governorate who were responsible for planning, programming and data analysis in preparing projection reports. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Component 3: Supporting secondary education policy development (US$1.5 million total; IDA: US$0.3 million) This component was to be achieved through a number of studies, to include inter alia: reviews of in-service and pre-service training of teachers; preparation of curriculum framework policy documents; feasibility assessments for a regulatory framework that would seek private sector investment in secondary education; and tracer studies on secondary education graduates. Component 4: Project management and monitoring (US$4.4 million total; IDA: US$0.8 million) This component aimed to support the Project Administration Unit (PAU) to carry out the fiduciary and monitoring and reporting responsibility ascribed to it in a timely and effective manner. It also financed M&E-related activities. Under the project restructuring the following modifications were made to the components: Component 1: Improving equity and reducing gender gaps: (planned, US$33.65 million; actual, US$31.93 million) The scale of the restructured project was limited to 50 schools and the five governorates identified for the first phase of the project implementation, the scale-up to 4 other governorates was abandoned. Component 2: Enhancing the quality of service delivery (planned, US$14.10 million; actual, US$11.64 million) The restructured project had a narrower scope and focused on provision of learning equipment and material, and capacity building of teachers, principals and supervisors in school. The sub-components on strengthening the learning assessment system and developing staff capacity at the central governorate levels to improve planning, programming, data collection and analysis were dropped Component 3 : Project Management and Monitoring (planned,US$2.43 million; actual, US$2.19 million) The restructured project dropped the original third component on supporting secondary education policy development entirely and thus the original component 4 became component 3. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: The original estimated project cost was $103.4 million (PAD, Annex 5). After the eruption of a conflict in the country, development partners withdrew a significant amount of their funding, which led to a restructuring of the project and a decrease of the scale from 9 to 5 governorates. The revised appraised cost of the project was US$50.22 million. The final project cost was US$45.76 million, which is 91% of the appraisal amount. A number of planned activities (including trainings and assessments) did not take place, which reduced the overall cost. Financing: The project was financed via an IDA credit, a multi-donor trust fund (TF-94233) with the Netherlands and the UK Department for International Development (DfID) as co-financiers , as well as parallel financing by Germany (GTZ-KFW). • The final IDA credit was US$19.12 million, which is close to the appraisal estimate of US$20 million. IDA is one of two partners that did not drastically decreased its contribution to the project in 2012. • GTZ had committed US$3 million and disbursed the anticipated amount. • KfW had originally planned to finance US$ 11.80 million, which ended up representing US$ 9.50 million due to exchange rate fluctuations between the Euro and the USD and almost entirely disbursed (96%). • DFID had planned to finance US$ 41.80 million, which was revised to US$ 2.40 million in 2012. The agency actually provided US$ 2.33 million, which represents 97% of the revised appraisal amount. • The Netherlands' original planned contribution amounted to US$ 14.80 million, subsequently revised to US$ 6.60 million, which was fully disbursed. Borrower contribution • The Government had originally committed US$12 million. At the time of restructuring this amount was reduced to US$8.70 million. The government ended up disbursingUS$5.60 million (64% of appraisal amount) Dates (ICR p. xiii) Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) • April 2012: the project was restructured to reflect the reduced geographic coverage and reduced scope of the project. One component and two sub-components were dropped. Instead of the 9 original governorates, 5 were to be covered. A number of intermediate outcome targets in the results framework were modified but the PDO and the PDO-level targets were left unchanged. Financing was more than halved from an original plan of $103.40 million to $50.20 million. • June 2013: Reallocation of funds to the civil works component • May 2014: The project was restructured to change baseline and annual/end targets of the PDO indicators, using data from the 5 selected and retained governorates alone, instead of the 9 originally envisioned. • November 2014: a 4 months extension of the project closing date was agreed upon, from January 31, 2015 to May 31, 2015. 3. Relevance of Objectives & Design a. Relevance of Objectives Original: The project’s objective of strengthening secondary education and promoting gender equity remained highly relevant at closing. In a situation of conflict, maintaining basic services and promoting a quality and equitable education system are both important in the short and the medium term to ensure that youth exposed to ongoing conflict can be part of a peaceful transition and reconstruction. Moreover, the Gender Parity Index for the Gross Enrollment Ratio in primary and secondary education remained low and stable over the period of implementation at 0.79 (World Bank data). Moreover, the project was in line with the Bank's Interim Strategy Note for Yemen (FY2013-14), which identifies improved access to basic social services, with an increased focus on gender inclusion, as a key pillar. The objectives were well aligned with the Bank's corporate priorities as reflected in the Millennium Development Goals, its overarching goal of promoting shared prosperity, and its cross- sectoral priority of promoting gender equality. Finally, the project was a relevant piece in a comprehensive approach that the Bank developed in the education sector of Yemen with complementarities with other projects supporting basic and higher education respectively. Revised: While the revision of the PDO targets towards a reduced scope and scale meant a lower potential for impact, the nature of the PDO did not change and thus remained highly relevant. Rating Revised Rating High High b. Relevance of Design Original: The project components and activities sought to address a number of constraints bearing on the functioning of schools within the targeted districts, chosen for their substantially lower Gross Enrollment Ratio for both males and females compared to the national level and to non- project districts. The project also astutely used both supply and demand-side interventions. Gender Equity: the project usefully relied on gender-targeted strategies to address the specific needs of boys and girls (e.g., targeting CCT to girls in grades 4-12 vs. boys in grades 10-12 only, incentivizing the retention of girls through additional resources to schools); and attempting to overcome a key cultural barrier to girls access and retention by hiring and training female teaching assistants. The successful experience of a similar scheme at the basic education level informed this scheme. Quality: the project adequately sought to tackle the multidimensional issue of weak education quality via a multi-pronged approach (e.g., improving facility and teaching materials, training teachers and school management, and strengthening assessments). Efficiency: the project aimed to improve the internal efficiency of the system by reducing dropout and improving retention of students in schools. In addition, the original sub-components related to improving the sector’s capacity to plan, program, collect and analyze data had the potential to improve efficiency by keeping track of progress and costs. Finally, the planned activities under the original component 3 (supporting policy development) sought to address inefficiencies by envisioning the potential for private sector investment in the education sector. Quality of results framework: The results framework and the accompanying description of activities (PAD Annex 3 and 4) do not clearly discuss the underlying project logic and the linkages between activities, outputs and intended outcomes, especially with regards to enhancing efficiency. The PAD and the ICR use causal language with some inconsistency (e.g., impact, intermediate output, intermediate outcome e.g., p. 24). Finally, there is little discussion of assumptions or risks to consider, for example the possibility of attrition of trained teachers (as highlighted in the borrower’s ICR summary Annex 7, p. 40). Revised: Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Given the important unexpected obstacles (reduction of development partner financing and outbreak of armed conflicts) that the project had to face, the simplification and reduction in scale and scope allowed the project to refocus on its core components, and was an important strategic move. A number of activities that focused on quality improvements (teacher training and classroom observation studies to assess the outcome of teacher training activities, and the assessments of Grade 11 mathematics and science) were not completed by the time of project closure. However, these activities were continued via the support of KfW and performed through the involvement of the Education Research and Development Center (additional information provided by the team after the ICR was completed). The project ensured continuity of the activities despite ongoing conflict and the suspension of funding and thereby justifies the validity of the design choices. With regards to the results framework however, it is important to note that the project did not adjust its main PDO targets at the time of the April 2012 restructuring to match the revised scope and scale of the project but only did so towards the end of the project (in May 2014). Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Improving gender equity of secondary education in selected districts with a particular focus on girls in rural areas. Rationale Is rated modest: the project did not achieve its original targets under this objective but made some marginal progress on the core indicator despite the situation of conflict. Outputs • Civil works program was completed in 43 schools • The number of students attending project schools using school community grants was 8,141 which is above the original target of 6,800 students. However this indicator is not sex disaggregated which would have given a better measure of the outputs' contribution to outcome. • 89 female staff were financed by the project. Outcome • The Gender Parity Index of Secondary Education Access Rate in Project Districts increased from 0.42 to 0.63. This fell slightly short of the original target of 0.66. • The Grade 10 (G10) to grade 12 (G12) male retention rate in project districts decreased by 1.9 percentage points from its 80.6% at baseline to 78.7% falling short of its original target of 85.4%. • The G10 to G12 female retention rate in project districts slightly increased from its original baseline of 84.2% to 84.5%, which is 4.6 percentage points below its original target. Improving gender equity means, in part, providing both female and male students equal opportunities to access and attend secondary school. Limited improvements in both male and female retention rates is thus an issue for the achievement of the objective. The ICR discusses the difference in retention rate between boys and girls, in particular, that the retention rate for boys (78.7%) was lower than the retention rate for girls (84.5%). While no further explanation of this discrepancy is provided, the ICR hypothesizes that there might be a differential effect of conflict and crisis on boys and girls. Another hypothesis that could warrant further exploration might be that the decrease in boys' retention rate might represent an unintended adverse effect on boys' motivation because of the project's emphasis on girls' access and retention. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Rating Modest Revised Objective Improving gender equity of secondary education in selected districts with a particular focus on girls in rural areas. Revised Rationale Is rated substantial: the project achieved its main revised targets under this outcome Outputs See above Outcome • The Gender Parity Index of Secondary Education Access Rate in Project Districts exceeded the revised target, which was set to 0.60. • The G10 to G12 male retention rate in project districts increased by 5.7 percentage points from its revised baseline of 73.0% to 78.7% falling short of its revised target of 81% by a couple of percentage points. • The G10 to G12 female retention rate in project districts increased from its revised baseline of 78.3% to 84.5%, which is slightly above the revised target of 84.3%. Improving gender equity means in part, providing both female and male students equal opportunities to access and attend secondary school. Discussion of contribution claims The ICR provides limited information that helps assess the validity of the contribution claims. For example, the PAD (p.28) highlights that the enrollment growth increased by an annual rate of 6.8% for females and 0.9% for males in project districts up to 2004. The ICR or the indicators selected do not show whether there has been an acceleration of enrollment growth after the project became effective, which would be useful information to show the contribution of the project. Similarly, a measure of PDO indicators in the four districts that ended up not receiving project funding and support would have also provided useful information to assess trends in similar districts. On the other hand, positive feedback from the scheme was received through CCT beneficiaries interviews and video recording, where some parents, students and teachers provide strong testimonies of the positive impact of the scheme on student retention. Little information was provided on the the ways these students had been sampled to participate in interviews. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective Improving quality of secondary education in selected districts with a particular focus on girls in rural areas. Rationale Is rated modest: the project made progress in delivering the outputs related to this outcome but there is little evidence that quality has been enhanced given the fact that information on the main indicators measuring outcomes was not available. Outputs • There were 14,349 teachers trained in subject modules, which was significantly below the original target of 30,000 • The teachers were trained in pedagogy training certification as originally planned but with a postponing of the training activities beyond project closure. • There were 50 schools equipped with science labs, computer labs, and libraries • The assessment and observation of teachers in classroom observation took place after the official project closure Outcome Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) • Learning outcome in mathematics in G12: The achievement test results improved from 19.3/100 in year 3 of the project (when the baseline became available) to 31.2/100 at mid-term, which is below the original target of 40 that was anticipated for the end of the project. No later value is available. • Trained teachers using curriculum: the teacher assessments in classrooms were postponed and took place after official project closure. • Average performance of teachers from the schools in observation: the teacher assessments in classrooms were postponed and took place after official project closure. Rating Modest Revised Objective Improving quality of secondary education in selected districts with a particular focus on girls in rural areas. Revised Rationale Is rated substantial: the project made progress in delivering the outputs related to this outcome and met the revised targets Outputs The activities regarding the assessments of students and the classroom observations of teachers after their training were not completed by the time of official project closure but were delivered after through the partner organization, Education Research and Development Center. Outcome • Learning outcome in mathematics in G12: the achievement test results at mid-term (31.2/100) were above the revised target value of 19.3/100. Revised Rating Substantial PHEFFICACYTBL Objective 3 Objective Improving efficiency of secondary education in selected districts with a particular focus on girls in rural areas. Rationale Is rated modest: the project did not meet its original retention target and dropped some activities contributing to this outcome early on. Outputs • The school transportation subsidies and CCT contributed to the outcomes of increasing retention and decreasing repetition and dropout: The number of students attending project schools using school community grants was 8,141 which is above the original target of 6,800 students. • Curriculum framework policy document prepared: the related activities were dropped. No output was produced. • Feasibility assessment for private sector regulatory framework undertaken: the related activities were dropped. No output was produced. Outcome The sex-disaggregated measures of retention also capture internal efficiency results, as they show the number of non-repeaters in G12 as a share of G10 students 2 years before. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) • The G10 to G12 male retention rate in project districts decreased by 1.9 percentage points from 80.6% at baseline to 78.7%, falling short of the original target of 85.4%. • The G10 to G12 female retention rate in project districts slightly increased from the original baseline of 84.2% to 84.5%, which is 4.6 percentage points below the original target. Rating Modest Revised Objective Improving efficiency of secondary education in selected districts with a particular focus on girls in rural areas. Revised Rationale Is rated substantial: the project made significant progress towards achieving its revised targets. Outputs: (see above) Outcome: • The G10 to G12 male retention rate in project districts increased by 5.7 percentage points from the revised baseline of 73.0% to 78.7%, falling short of the revised target of 81% by a couple of percentage points. • The G10 to G12 female retention rate in project districts increased from the revised baseline of 78.3% to 84.5%, which is slightly above the revised target of 84.3% Revised Rating Substantial 5. Efficiency The PAD (Annex 9) provided a detailed economic analysis of the project, estimating an EIRR of 13.7% assuming the retention rates and quality improvements objectives were met. In addition, the economic analysis of the PAD presented some further benefits based on positive externalities, notably in terms of improved health outcomes and intergenerational benefits to basic education. The sensitivity analysis in the PAD (p.63) noted the high sensitivity of the EIRR to the retention rate. Neither the PAD nor the ICR economic analysis provide an estimate of the number of students (boys and girls) who are expected to benefit from the intervention, making it difficult to assess the soundness of the model. The ICR (Annex 3) updated this analysis based on some partially updated assumptions and actual project data. The updated EIRR was 17%. This improvement in EIRR can largely be explained by the restructuring of the project, which almost halved the estimated project costs, from an initial cost of US$ 103.4 million to US$ 47.2 million. The benefits stream was also slightly revised to reflect the reduction in the scale of the project (5 directorates instead of 9) and to incorporate the revised retention rates. However, enhanced quality of education was to be attained through a number of project interventions to improve teacher training, education planning, school management, and curriculum. Several of these key activities did not take place, and therefore the expected benefits from improved quality of education were only partially fulfilled. The pass rate was not updated and reflects the national average, not the specific rate in the project district, which is likely to be lower. Finally the benefit streams over the period of 20 years were not updated to take into account the high risk that the outcome would not be sustained, especially in the absence of further project investments (as stated in the ICR p. 16). On the other hand, other measures of efficiency were quite positive. For example the Project Administration unit (PAU) and Public Works Project (PWP) managed to use project funds efficiently, to negotiate effectively with contractors during the periods of conflict and turmoil to postpone some of the planned activities, and to overcome important implementation constraints. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  13.70 Not Applicable 100.00 ICR Estimate  17.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original: Moderately Unsatisfactory Under the original targets, project outcome is rated moderately unsatisfactory. Relevance of the original project objective is rated high. Relevance of design is rated substantial, and efficiency is rated substantial. However, achievement of all three objectives under the original targets is rated modest, with limited progress on core activities and none of the original outcome targets being met. Revised: Satisfactory Under the revised targets, project outcome is rated satisfactory. Relevance of the revised project objective is rated high and the relevance of design is rated substantial. Important progress in output delivery and revision of targets to a more appropriate level enabled the project to meet its key revised PDO targets, resulting in substantial achievement of the gender equity, quality and efficiency objectives. According to OPCS/IEG harmonized guidelines, the final outcome rating of a restructured project is determined by weighting the amount of the Credit that disbursed before ("original objectives/targets") and after ("revised objectives/targets") the project restructuring. The Credit had disbursed US$ 10.96 million out of US$ 19.12 million, or 57.3%, at the time it changed its PDO targets. Therefore, the combined outcome rating is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating A number of factors coalesced to support the view that the risks to outcome are high. First, the ICR notes the high risk that the outcome of the project may not be sustained because of the renewal of armed conflict, security turmoil and political interference (p. 16). A number of additional threats to the outcomes should be highlighted, including the limited capacity of the government to fund civil work activities and to support ongoing costs (ICR, p.18), the drop in male retention from 79.8% in 2013 to 78.7% in 2014 (which may be due to the impact of the armed conflict) and the Government’s incapacity to hire the 89 female teachers as civil servants, which threatens their status and the sustainability of project outcome (ICR p. 18). a. Risk to Development Outcome Rating High Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) 8. Assessment of Bank Performance a. Quality-at-Entry Quality at entry combines the assessments of (i) strategic relevance; (ii) organizational arrangements including M&E arrangements; and (iii) institutional arrangements with other donors and the government. The project design was well aligned with national priorities as expressed in the National General Secondary Education Strategy (NGSES) and based on a sound diagnostic that informed the selection of governorates. In addition, the approach of combining supply and demand side interventions was evidence-informed. On the other hand, the choice of organizational arrangements did not sufficiently take into account the capacity weaknesses of the PAU, despite the warning of a Quality Enhancement Review (QER) at the inception of the project, especially regarding the government’s capability to deal with donors’ pulled funding. Moreover there were some shortcomings in the design of the M&E framework. For example, given the high-level PDO indicators, the results framework would have benefitted from also including some indicators that captured an intermediary level of outcomes, that were more directly attributable to project interventions. For example, the QER had suggested a number of options (p. 3), including years of school per graduate, repetition rates, drop out rates, and pass rates. Instead, the results framework contained primarily output indicators. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision For the first two years of the project, disbursement was very slow and important activities were delayed. The ICR (p. 17) also notes that the Implementation Status Reports (ISR) ratings at the beginning of the project did not reflect accurately the limited progress taking place in implementation. Later during the implementation cycle, the Bank team demonstrated proactivity in adjusting the focus of the project in light of the slow start and amidst the important implementation challenges linked to the armed conflicts and the significant reduction in funding of a number of partners. Furthermore, the continuity of Bank's support to the PAU and PWP allowed the intervention to start again swiftly after the respite of 2012. Although the Bank had to relocate supervision to Amman and was unable to conduct supervision missions, it prepared and implemented an emergency plan to sustain activities despite reduced and suspended disbursements. It also strategically resorted to a third party monitoring mechanism, which allowed for monitoring activities to continue despite the conflict. Notwithstanding the above, the failure to adjust targets and baseline at the time of restructuring is a clear oversight of the Bank. Finally, the Bank remained overoptimistic about its capacity to deliver on key activities during restructuring, including with regards to important training activities for teachers. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The ICR praises the government’s contribution in providing an institutional framework for the project notably through the policy framework (NGSES) that guided the design of the project, and the proactive coordination of multiple units under the Ministry of Education that were key stakeholders in the project (teacher training, CCTs, girls' education). Despite the crisis, the government showed some commitment to continue with program activities, including after the official closure of the project. However, several ISRs (up to 2013) noted a limited collective vision, some dialogue problems, and a low level of ownership by the government as a continuous issue (ICR, Annex 11). Moreover, the government did not provide the necessary funding to make a number of key payments, most notably for the civil works activities. This can be explained in part by the difficult fiscal situation in Yemen during the escalated political unrest. Government Performance Rating Moderately Unsatisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) b. Implementing Agency Performance The ICR commends the implementing agencies (PAU, PWP and sub-national implementing stakeholders) for their resilience during project implementations amidst multiple challenges. Most notably, the implementing agencies were able to implement the emergency plans during the 2011 crisis. They demonstrated grit in resolving implementation delays in the school designs, training modules and implementation of civil work schemes despite the conflict, important staff turnover, slow or suspended disbursements, and difficult harmonization among development partners. Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The indicator framework was parsimonious with a sufficient number of indicators and usefully incorporated education statistics at the governorate level to be able to compute the retention rate for both males and females. Moreover, the institutional arrangements for M&E originally relied on an integrated structure within the PWP that was in charge of supervising multiple Bank-supported programs, including a series of projects in basic education, with the potential for an integrated monitoring platform for all school infrastructure contracts and their implementation. However, a number of shortcomings existed: • The main PDO indicator that captured learning outcome (improved learning outcome in mathematics in grade 12) was not sex-disaggregated. • The indicators that captured quality of service delivery relied on project activities to go forward (e.g., implementation of teacher assessment to assess teacher training). • Indicators that measured the results of the subsidy and CCT scheme only measured coverage and not sex-disaggregated uptake. b. M&E Implementation One of the main shortcomings in the implementation of M&E is that the results framework, performance indicators, and baselines were adjusted only 2 years after the project was restructured with a more limited scope and scale. In addition, given that a number of outcome indicators, including the fourth PDO indicator (learning outcome in mathematics), relied on the project's capacity to conduct some key activities (e.g., assessment of students and assessment of teachers) that ended up not taking place, a number of key outcome measures were not available at closing of the project, due to either cancellation of assessments or delays in implementation. The summary of ISRs usefully provided in the ICR (Annex 11) is testament to the recurrent issues with limited data for project monitoring. Notwithstanding the above, the choice of implementing third party monitoring during the period of conflict was judicious and enabled data collection at the output level to take place. Moreover, arrangements for monitoring the progress of the project were affected by PAU staff rotation and limited access to sources of data, especially in the areas most affected by the ongoing conflict (ICR p. 10). Issues of coordination between the development partners and slow project startup, and the need for project supervision to move to Amman, affected the original plan. The Bank and the government thus judiciously decided that a third party monitoring should be hired during the 1st half of 2012 (ICR p. 49), which enabled some monitoring activities to take place despite the instability in the country. c. M&E Utilization Given the late adjustment of project indicators, targets and baselines, it is unlikely that the monitoring system could have provided tight and accurate feedback loops to the implementation team. Nevertheless, the ICR points to the benefits of adding new output-level indicators after the restructuring which enabled the PAU and PWP to more closely monitor implementation and disbursement progress (ICR p. 10). Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Use of M&E information to inform continuation of engagement is uncertain. M&E Quality Rating Modest 11. Other Issues a. Safeguards Environmental: The project environmental category was rated B due to school rehabilitation and extension and replacement of existing schools. An Environmental Management Plan (EMP) was prepared for the project based on lessons learned from a similar intervention taking place in the Basic Education sector. Additional provisions were envisioned in the PAD to further improve supervision (PAD, p. 70), such as regular and random visits by trained engineers within the Governorates Education Offices (GEO) and District Education Offices (DEO), training and awareness campaigns for school managers, teachers and social works including on environmental aspects and safety measures, and an active participation of mothers’ and fathers’ councils in the management of schools and utilization of resources by the community. While the ICR does not provide details as to whether these activities have taken place in full or in part, the summary of the Borrowers’ ICR (Annex 7, pp. 39 and 41) points to some issues in the planned supervision of civil works due to the fact that no funds were allocated to the transportation of GEO/DEO and MOE engineers to the civil work sites. b. Fiduciary Compliance Financial management: The ICR reports (p.11) that the project management unit maintained an adequate financial management during the project, which was rated moderately-satisfactory throughout the project cycle. Reporting of project expenditures was adequate, withdrawal applications were processed without hurdles and audit reports were on time, except for the last report which was delayed by one week due to the conflict.The ICR notes that in 2013 the auditor identified ineligible monthly payments made to the members of the steering committee (civil servants) that were not part of the list of eligible expenses set out in the Financial Agreement. In late 2014, the Bank advised the PAU to refund US$ 88,800 to IDA and other development partners. Given the suspension of Bank’s disbursement in January 2015, the letter requesting the Government of Yemen to refund the ineligible amount was put on hold and had not been submitted formally by the time of the ICR submission. The TTL later confirmed that a substantial portion of the amount was later refunded or settled. Procurement: Procurement processes were rated moderately satisfactory throughout the project. Disbursement delays at the outset of the project, as well as during the period of conflict (2011), created important delays in project implementation. Moreover, the government did not pay a significant portion of its planned contribution, which created procurement issues as reported in the summary of the Borrower’s ICR (ICR, Annex 7). For example, a deficit of US $1 million in the civil works operation, due to currency exchange differences and non-payment of the government’s contribution, led to the PWP's inability to pay in full some contractors and suppliers. c. Unintended impacts (Positive or Negative) None noted. d. Other N/A 12. Ratings Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome High High --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Several interesting lessons are provided in the ICR (pp.19-22 and Annex 10), including the following three summarized by IEG: • The juxtaposition of supply-side (access and quality inputs such as a curriculum reforms, teacher training, and pedagodical materials) and demand- side interventions (incentives, school-community grants, participatory processes) is a particularly relevant approach to increase the access and retention of young women in secondary education where significant socio-cultural obstacles to their education exist, such as in the rural areas of Yemen. • Assessing the capacity of both development partners and country clients to achieve donor harmonization and absorb pulled funding is necessary, but challenging. Pooled financing mechanisms and joint governance frameworks can be complex and difficult to put in motion both for development partners and for the country client. • During a conflict or political crisis that triggers the suspension of the Bank’s and other development partners’ disbursements, a proactive emergency implementation plan that identifies transition mechanisms, essential activities, negotiations with active contractors, and support to DPs that remain in the country is an important tool that enables swifter revamping of activities in a post-crisis period. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is analytically sound and makes appropriate use of the existing evidence in assessing outcomes. The quality and thoroughness of the lessons section and annex must be commended, especially in identifying possible future courses of action for the Bank’s contribution. Similarly, the ICR provides useful additional information through its systematic and organized analysis of the ISRs (Annex 11). However, the ICR could have been more precise in its assessment of the results chain, including by providing more specific information on how the outputs delivered contributed to achieving the outcomes (e.g., explaining the linkage between the achievements within the CCT scheme and retention rate), especially in view of the shortcomings of the formal results framework at the outcome and intermediary outcome level. Moreover, the ICR contains some inconsistencies on the date when the PDO targets were revised, and consequently does not adequately apply the guidelines with regards to split rating. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY Sec. Educ. Dev. and Girls Access (P089761) a. Quality of ICR Rating Substantial