Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) Report Number: ICRR0023002 1. Project Data Project ID Project Name P132742 Second Skills Development Project Country Practice Area(Lead) Afghanistan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H8340 38,850,263.65 Bank Approval Date Closing Date (Actual) 19-May-2013 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 45,136,192.60 0.00 Actual 38,850,263.65 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (p. 4), the project's objective was to "increase the potential for employment and higher earnings of graduates from technical and vocational education and training (TVET) schools and institutes through improvements in the skills delivery system." At a June 2017 restructuring, the objective was revised to "improve TVET teacher competencies and curriculum in selected priority trades." Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) This ICRR will therefore perform a split rating assessment. At the time of restructuring, US$30.29 million, or 78 percent of Bank financing, had been disbursed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 07-Jun-2017 c. Will a split evaluation be undertaken? Yes d. Components The project was approved with four components: 1. Strengthening of the TVET institutional system (appraisal: US$8.5 million; actual US$3.4 million) was to provide technical assistance for the formulation of a detailed TVET strategy, plan, and implementation program, engaging a wide range of relevant stakeholders; development of quality standards for skills delivery, accreditation, and certification of student skills; design and implementation of a labor market information system and an overall management information system (MIS) for TVET; building of capacity among monitors and inspectors of schools and institutes accredited by the DMTVET (Deputy Ministry of TVET within the Ministry of Education); and conducting of certification examinations for TVET graduates. This component was dropped at the June 2017 restructuring, as it received little political support, the technical skills to implement it were lacking, and there was poor coordination with the International Labor Organization, which was to have assisted with implementation. 2. Improving the performance of TVET schools and institutes (appraisal: US$26.8 million; actual: US$16.43 million), was to be implemented in two steps. The first would recognize schools and institutes that had already improved academic management, reformed school administration, improved student placement, established linkages with local industry, reformed curriculum, or taken other steps to meet student needs, with "recognition grants" of US$30,000. The second would provide development grants of up to US$400,000 to approximately 15 institutes that had produced credible development plans to sustain and further extend reform efforts. All public and private institutions accredited by DMTVET (210 public and 47 private), plus any that were accredited during the project's lifetime, were to be eligible for support under this component. Each institute that received a development grant was required to set up a governing council, with representatives from the private sector, as a condition for disbursement. The component was also to fund a voucher program for about 1,000 exceptional students to pursue further professional certification. This component was partially dropped at the June 2017 restructuring, but its activities that were already under design, procurement, or construction to support seven selected institutes' infrastructure continued under the original project envelope, and the voucher program was continued to permit a small number of students at two institutes to complete their programs and graduate. Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) 3. Improving teacher competencies (appraisal: US$6.9 million) was to develop an infrastructure for in- service training for technical teachers and for building a pool of master trainers. In this area, the project was to collaborate closely with the German Agency for Technical Cooperation, which was already involved in pre-service and in-service technical teacher training. By preparing approximately 750 teachers and 60 master trainers, it was envisioned that the component would enable the Technical Teachers Training Institute (TTTI) to become sustainable in terms of faculty resources by the end of the project period. This component was dropped at the June 2017 restructuring, with its activities redesigned under a new component. 4. Project management, monitoring, and evaluation (appraisal: US$5.5 million; actual: US$7.26 million) was to finance third-party monitoring and evaluation (M&E) of student learning outcomes, teacher performance, and implementation of institutional development plans under the second component; satisfaction surveys of private sector partners; tracer studies of graduates from project- supported schools; and a public awareness campaign to sensitize high school student on TVET opportunities. External impact evaluation studies were also to be financed by this component. At the June 2017 restructuring, the component was renamed to "project management and technical assistance," primarily to mobilize additional technical and operational staff to support the delivery of new disbursement- linked indicator (DLI)-based activities (detailed below). The 2017 restructuring also added a fifth component: 5. TVET sector reform (original: US$15 million; actual: US$4.7 million), to support delivery of priority trades and upgrade TVET teacher competencies. Planned activities included the operationalization of Lead Institutes that would support (i) TVET teacher assignments in priority trades, (ii) domestic in-service teacher training, (iii) implementation of revised curricula in partner institutes in priority trades, (iv) increases in the share of practical training in the teaching-learning process in partner schools, and (v) strengthening of industry linkages and engagement of practitioners in the classroom. The component was also to support teacher policy development and implementation, curriculum reform for grades 13-14 in priority trades, and teacher assessment. A 2018 restructuring added a sixth component: 6. Financing of teacher salaries at the National Institute of Management and Administration (NIMA) and Afghanistan National Institute of Music (ANIM) (original: US$1.57 million; actual: US$1.57 million) was to finance teacher salaries at the two specified institutes on a declining basis, as the government progressively scaled up civil service staff in each institute. A 2019 restructuring added a seventh component: 7. Infrastructure upgrades at the Kabul Auto Mechanic Institute (KAMI) and three lead institutes and measures to mitigate gender-based violence (GBV) risks (original: US$2.5 million; actual: US$2.5 million) was to support safety and security upgrades at KAMI and Lead Institutes, and to add mitigation measures for GBV. A 2020 restructuring added a COVID-19 response component: 8. Supporting COVID-19 response through an Alternative Learning Program (ALP) (US$7.8 million; actual: US$7.8 million) was to establish a distance learning platform, develop and deliver multi-channel Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) content to reach students via a variety of methods during pandemic-related restrictions and lockdowns, and disseminate digital content. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was to be financed by a US$55 million Grant from the International Development Association. The government was to contribute US$5 million for salaries of some staff and technical teacher trainees, and the original budget included US$4.1 million in incremental operating costs and US$3.2 million in contingencies, bringing the total planned costs at appraisal to US$60 million. Actual disbursement was US$38.85 million. US$9.98 million was cancelled. The project was approved on May 19, 2013 and became effective on June 12, 2013. Its mid-term review took place in October 2015. The project closed on June 30, 2021, three years later than its original closing date of June 30, 2018. It was restructured five times:  June 7, 2017: This restructuring responded to the new human capital priorities of a new government, focusing on a critical set of outputs included in the original project that were anticipated to contribute most directly to employment-focused outcomes: teacher training, strengthening of select TVET institutes (Lead Institutes), and enhancing linkages with the private sector. It introduced a results- based financing approach leveraging DLIs.  October 4, 2018: This restructuring updated the implementing agency to the newly established TVET Authority (TVETA), introduced by government decree on April 21, 2018. In addition, a new component was introduced to reallocate unused funds to finance salaries for NIMA and ANIM, as well as other activities geared toward strengthening TVETA capacity.  December 18, 2019: This restructuring addressed social and environmental safeguards issues that had arisen at NIMA and KAMI. It also expanded the voucher program to include 50-80 additional students, and established a Youth Connecting Hub at TVETA. Financing was adjusted for some DLIs.  July 25, 2020: This restructuring responded to the COVID-19 pandemic by introducing condensed procedures for implementation of distance learning plans. It also reduced the DLI envelope from US$10.7 million to US$4.7 million and dropped several DLIs, allocating that US$6 million to the ALP, given that several of the anticipated training and assessment activities were no longer possible under lockdown restrictions.  June 28, 2021: US$9.98 million in project financing was cancelled because of activities that could not be completed due to COVID-19-related delays. 3. Relevance of Objectives Rationale The project's objectives were responsive to country context at appraisal. In the decade preceding the project, Afghanistan had experienced rapid economic growth and steady growth of the formal sector. Even Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) though the official unemployment rate was low (8.2 percent in 2012), around 77 percent of workers were underemployed and earned low salaries. There was a growing need for skilled labor. In a 2012 survey of small and medium enterprises in six major cities of Afghanistan, 53 percent of employers said they wanted to hire new employees, but 36 percent viewed lack of skills as a bottleneck, and 26 percent said they were never able to find employees with the right set of skills and adequate training. The TVET system was enrolling a growing number of students -- from 20,000 in 2007-08 to 70,000 in 2012-13 -- but a large proportion of the 10,000 graduates annually were not able to find jobs. This mismatch between supply and demand was due to the low quality of TVET delivery stemming from systemic deficiencies and low capacity of TVET service providers. According to the Emergency Project Paper (EPP, pp. 5-6), the critical barriers to reform of the TVET system were the absence of a robust institutional system for the sector, which was excessively fragmented; poor quality of education, including lack of standardized curricula and training materials; the absence of a reliable labor market information system; and poor linkages between the labor market and the TVET sector, with no mechanisms available for TVET providers to interact with industry. The project was intended to address this circular problem, where policies for economic development could not be implemented fully because the needed skills were nonexistent, and there was little investment in skills training because of insufficient motivation and pathways for industry to engage. At the same time, Afghanistan was and remains a highly volatile political and security environment. At the time of project preparation, the country was anticipating presidential election and a security transition when international forces were scheduled to depart in 2014. With potential economic turbulence expected as part of that transition, the project aimed to focus on intensive capacity-building measures, especially for civil servants and technical teachers, so that these human resources would be available in the post-transition period. This project's objectives were considerably less ambitious than those of its predecessor project. The Afghanistan Skills Development Project (US$15.25 million, 2008-2014, P102573) aimed to increase the number of immediately employable graduates by building, in stages, a high-quality TVET system that is equitable, market-responsive, and cost-effective. That project's outcome was rated Moderately Satisfactory by its ICR and by IEG, as it reached downwardly revised targets for graduates and their employment, but key legislation was not passed during project implementation to enable planned institutional reforms in the sector. It was therefore reasonable for this project to have crafted objectives that focused on more fundamental elements of institutional reform. The project was relevant to government strategy across its lifetime. The Afghanistan National Development Strategy at appraisal (2009-2013), if successfully implemented, would have led to a significant increase in local demand for skilled labor. The project aimed to enable the TVET system to meet this demand through institutional reforms. Among other national education plans, the National Priority Program One, under its Labor Program (Sustainable Decent Work Through Skills Development and Employment Policies for Job Rich Growth), focused on development of human resource capacity to meet the skill demands of the market; it emphasized that the government, private sector, and civil society should work together to form a coherent approach to the development of human resources capacity. The National Education Interim Plan (2011-2013) proposed increased access to TVET through the opening of new TVET centers in districts, as well as quality improvement in TVET delivery. Overall, the project was intended to improve the institutional environment in the institutes providing skills training, filling a critical link between supply and demand for labor market-relevant skills. The project was also relevant to Bank strategy. One of the three pillars of the Bank's Afghanistan Interim Strategy Note (ISN, 2012-2014) focused on inclusive growth and jobs. The project also supported the ISN's Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) cross-cutting agenda of gender mainstreaming by targeting government-run schools and institutes, which not only catered to economically disadvantaged youth from distant and insecure provinces by giving them free education, but also operated under a government policy that required at least 30 percent female representation. The objectives continued to be relevant to the Bank's most recent Country Partnership Framework (2017-2020), which contained pillars on (i) building strong and accountable institutions to enable the state to fulfill its core mandate to deliver basic services to its citizens, (ii) supporting inclusive growth, and (iii) deepening social inclusion. The Bank's forward-looking "2020 Priorities for Inclusive Development in Afghanistan" called for increasing attention to TVET and skills development, with a focus on demand-driven skills that are targeted to emerging needs in the private sector. The objectives as revised in 2017 remained relevant to country context and Bank strategy, and they responded directly to an evolving government strategy that increased focus on teacher competencies and strengthening of TVET institutional capacity. However, the move away from engagement with the private sector, prompted by lack of systematic information on potential employers (ICR, p. 25), decreased the relevance of the revised project. The EPP (p. 6) noted that formal and informal TVET systems in Afghanistan fell under two different institutional jurisdictions: the former with DMTVET and the Ministry of Education, and the latter with the Ministry of Labor, Social Affairs and Martyred and Disabled. Given the high transaction costs and potential coordination issues of working with two different ministries, the Bank prepared a separate project for informal TVET. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase the potential for employment and higher earnings of graduates from technical and vocational education and training (TVET) schools and institutes through improvements in the skills delivery system Rationale The theory of change for this objective held that formulation and implementation of a sectoral strategy, a teacher policy, and information systems covering both the labor market and the overall TVET system would strengthen the institutional capacity of TVET. The provision of grants to TVET institutes and schools to update and implement their development plans, supported by upgraded infrastructure at some institutes, the involvement of private sector representatives in TVET institute governance, capacity-building for monitors, inspectors, and administrators, certification of graduates' competencies against international standards, and provision of vouchers for students would lead to improved performance of TVET schools and institutes. In- service training for TVET teachers, rehabilitation and equipping of the Teacher Training Institute, and regular assessment of teacher competencies would lead to improvement in TVET teacher competencies. Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) These intermediate outcomes -- strengthened institutional capacity of the overall TVET system, improved performance of TVET schools and institutes, and improved teacher competencies -- would constitute improvements in the skills delivery system, which would increase the potential for employment and higher earnings of TVET graduates. Critical assumptions underlying this theory of change included the feasibility of engaging representatives from the private sector, the utility of international benchmarking for occupational skills standards, the extent to which institutional development grants would facilitate change and reform in TVET institutes, the continuity of government support for reforms, and ongoing demand for skilled workers. Outputs Outputs related to teacher training: A draft teacher management policy framework was approved by the government's Human Capital Committee and disseminated (a DLI), meeting the target. The framework outlined recruitment criteria for TVET teachers, ensuring that prior industry expertise and experience would be valued in the hiring process, and specified supervision and mentoring modalities and performance assessment processes. The policy was under implementation by the end of the project period. Three institutes were designated as Lead Institutes to support delivery of priority trades and met all the criteria for Lead Institute Operationalization, meeting the target. These institutes reformed their curricula and piloted competency-based modules, increased the share of practical training and participation of private- sector representation in their governance councils, and completed GBV and safety assessments (with mitigation measures). The number of TVET institutes implementing revised competency-based curricula for priority trades (a DLI) increased from one in 2017 to 17 in 2021, surpassing the target of nine institutes. The project team noted that 1,763 teachers participated in various forms of teacher training as part of the piloting and rollout of the new curricula or through overseas training; 319 of those teachers were female.1,189 registered TVET teachers were assessed to evaluate minimum competencies, not reaching the target of 1,500. The assessments were interrupted by the COVID-19 pandemic. The percentage of TVET teachers who attained minimum competencies in priority trades increased from 15 percent in 2017 to 100 percent in 2021, surpassing the target of 70 percent. This teacher competency result is based on the 42 teachers who completed international training at two institutes in India (see below); it is unlikely that this was a representative sample of teachers. A planned, more comprehensive assessment of teacher competence (a DLI that was eventually dropped) was not undertaken. 42 teachers received international training in priority trades, meeting the target (this was initially a DLI with a target of 300 teachers, revised to 200 teachers; the DLI was eventually dropped). These 42 teachers had passed their initial assessments and enrolled in teacher training institutes at two Indian universities to further upgrade their technical competencies. No teachers who had passed their initial assessments furthered their training at domestic training institutes (a DLI), not reaching the target of 400 teachers furthering their training in-country, because of COVID-related delays and lockdown of the TTTI training site. The share of female teachers was 43 percent in 2021, not achieving the target of 50 percent. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) 206 teachers were engaged in distance learning activities at project closure, exceeding the target of 200. These teachers produced 550 online practical training videos. 41 percent of teachers who supported distance learning activities were female. 79 percent of teacher grievances were responded to or resolved within the stipulated time, exceeding the target of 70 percent. Other outputs: A TVET Strategy and Plan for 2013-2018 was developed in 2013 and approved by DMTVET, meeting the target; however, DMTVET did not incorporate suggestions made by the Bank. The strategy was under implementation until the establishment of TVETA in 2018. The project team later added that the project also supported the development of a comprehensive national TVET sector strategy (2019-2024); a TVET institutional reform plan (2019-2021), from which three-quarters of recommendations were implemented by the end of 2021; and a new TVET law that was endorsed by parliament in 2021 and is in the process of finalization. Recognition grants worth US$30,000 were provided for 15 selected schools and institutes in a first selection round. However, further scheduled rounds were not implemented due to lack of necessary technical capacity at the institutes, causing delays in development of business plans, and poor financial management capacity of the government, which delayed disbursements. 100 National Occupational Skills Standards were benchmarked to international standards, not meeting the target of 140 standards. 336 students were awarded vouchers to cover TVET course fees and reading materials, not reaching the target of 1,200 students. 4,169 total students benefited from direct interventions to enhance learning, not reaching the target of 5,000 students. This result represents students who benefited from new curricula piloted in six trades. Of these, 1,732 were female, exceeding the target of 1,500 female students. 1,216 students and 85 teachers were trained in a code of conduct, not reaching the target of 1,400 students and 100 teachers. A final scheduled round of training was not possible due to pandemic-related school closures. In response to the COVID-19 pandemic, the project assisted with the development of an Alternative Learning Plan that encompassed a wide range of digital and distance education services tailored to Afghanistan's context. Activities included preparation and distribution of 2,064 sets of chapter notes for the project-focused priority trades; production of 7,550 video tutorials illustrating practical elements of TVET course work; and development of the country's first-ever online skills-related portal to act as a repository for the public on the newly developed skills-related curriculum. TVET awareness campaigns were conducted in 11 provinces, not meeting the target of 20 provinces. These campaigns included job fairs, voucher programs, and recognition grant workshops. Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) There were 42,724 direct project beneficiaries, exceeding the target of 40,500 beneficiaries. Of those, 788 were female, exceeding the target of 220. Outcomes The number of students who sat for certification examination administered by international certification providers increased from 1,058 in 2015 to 1,609 in 2016, not reaching the target of 18,000 students. The share of students who passed the certification examination remained stagnant at 68 percent, not reaching the target of 75 percent. Learning outcome data is presented for only one institute, NIMA, where scores improved at a level of 5 (on a scale of 1-10), not meeting the target of 7. The share of students from supported TVET institutions who were employed within six months after graduation increased from 52 percent in 2015 to 60 percent in 2016, not reaching the target of 62 percent. No more recent data or relevant earnings data are available. Achievement of the original objective is rated Modest. Although the project's interventions plausibly improved the skills delivery system through its interventions related to teacher training and curriculum, data on outcomes related to student skills (a proxy for potential for employability) and wages were unavailable beyond 2016. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective Improve TVET teacher competencies and curriculum in selected priority trades Revised Rationale The "Outputs Related to Teacher Training" listed under the first objective are the outputs and outcomes relevant to this objective, and the elements of the original theory of change related to teacher training and curriculum development apply here. The key outcomes related to improved TVET teacher competencies and curriculum were:  The number of TVET institutes implementing revised competency-based curricula for priority trades (a DLI) increased from one in 2017 to 17 in 2021, surpassing the target of nine institutes. The 17 institutes that piloted and/or adopted the new curricula were the three Lead Institutes and 14 others.  The percentage of TVET teachers who attained minimum competencies in priority trades increased from 15 percent in 2017 to 100 percent in 2021, surpassing the target of 70 percent. However, the teacher competency result is based on 42 teachers who graduated from two institutes in India; this is likely not a representative sample of all teachers trained under the project. Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) Achievement of the revised objective is rated Modest, based on the information on improved teacher competencies being taken from only a small number of teachers who attended two international training institutes. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale With achievement of the single original objective rated Modest, overall efficacy under the original objectives is also rated Modest. Overall Efficacy Rating Primary Reason Modest Insufficient evidence OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale With achievement of the single revised objective rated Modest, overall efficacy under the revised objectives is also rated Modest. Overall Efficacy Revision 1 Rating Primary Reason Modest Insufficient evidence 5. Efficiency The cost-benefit analysis at appraisal (EPP, Annex 10, pp. 98-103) estimated direct and indirect benefits through improvements in the internal efficiency of TVET schools, producing increased graduation rates and trade certification, and in external efficiency through increased skilled employment and better earnings of graduates as a result of higher productivity. It assumed a 30 percent earnings premium and 20 percent higher probability of employment for TVET graduates with trade certification relative to other students in the TVET system. The project was also assumed to create positive externalities, with more firms interested in hiring newly trained students with relevant skills. Based on a ten percent discount rate and 25-year time horizon, the analysis found a net benefit of US$51.7 million and an internal rate of return (IRR) of 23 percent. The analysis at closing found a net present value of US$2 million and IRR of 14 percent. The lower IRR at closing is not surprising, given the project's reduced ambition at restructuring, though it still exceeded the opportunity cost of capital. A 14 percent IRR is comparable with other skills development projects; a Bank- Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) financed project in North Macedonia was rated as substantially efficient with an IRR ranging from seven to ten percent, and one in Punjab was rated as substantially efficient with an IRR of 16-20 percent (ICR, p. 47). The ICR (p. 22) notes that implementation was "inefficient" and project management "less than effective" prior to the 2017 restructuring, after which the project "adapted and adjusted," increasing the efficiency of implementation. Inefficiency prior to restructuring was driven by weak leadership and ownership by DMTVET, problematic implementation arrangements, high costs of engaging international experts, and lack of effective strategy and incentives to encourage teachers to attend TVET training. Although there were improvements from 2017 forward, stemming from TVETA's strong commitment to overall sector reform and strengthening of the project management unit, there were still shortcomings that affected achievement of outcomes. The TTTI could not fully resume teacher training programs because of the COVID-19 lockdown and delays in allocating training budgets, which "negatively affected the project's progress in teacher training-related indicators" (ICR, p. 24). Financial management capacity was weak, largely due to inadequate staffing and difficulty in adjusting to results-based financing (which was new to the Ministry of Finance and other counterpart entities), leading to ineligible expenditures that were unresolved until just before project closing. The ICR describes TVETA leaders and project management staff as inadequately informed about the results-based financing modality, "confused about the status of the project's available financing for different components, which delayed several critical project activities and complicated the clearing of ineligible expenses" (ICR, p. 24). Funds were not fully reconciled until June 2021, the month the project closed, at which point the financial management rating was upgraded from Moderately Unsatisfactory to Moderately Satisfactory. Similarly, procurement performance, although improved in the spring of 2018, experienced delays in preparing technical specifications and processing during and following the pandemic. In March 2020, due to the COVID-19 pandemic, more than 300 TVET institutions involving 60,000 students and 3,800 teachers were closed. The emergency response plan, the ALP, was efficiently implemented, including preparation of chapter-note learning materials and their wide digital distribution; deployment of teachers to develop self-study guides; production of 7,500 competency-based video lectures; and establishment of an on- line portal to serve as a repository for curriculum and content. However, all of the allocated ALP budget could not be spent because of uncertainties around the duration of school closures, the need to revise and adapt implementation plans to rapidly evolving circumstances, and delays due to government-required mandatory secondary review of all COVID-19 response projects. Given significant implementation inefficiencies, some of which persisted following the 2017 restructuring and impacted achievement of results, project efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  23.00  Not Applicable Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) 100.00 ICR Estimate  14.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives were substantially relevant, with the revised objectives remaining relevant to country context and Bank strategy, and adapting appropriately to shifting government priorities; however, relevance decreased somewhat at the 2017 restructuring with the move away from engagement with the private sector. The original objective to increase the potential for employment and higher earnings of graduates was modestly achieved, due to a combination of poor outcomes and lack of data on improved skills, employment, or earnings. The revised objective to improve teacher competencies and curriculum was also modestly achieved, with the number of institutes implementing revised competency-based curricula surpassing targets, but teacher competency data provided for only a small, non-representative sample of teachers. Efficiency is rated modest due to evidence of significant implementation inefficiency. With substantial relevance, modest efficacy, and modest efficiency, outcome under both the original and revised objectives is rated Moderately Unsatisfactory. Overall outcome is therefore Moderately Unsatisfactory, reflecting significant shortcomings in the project's preparation, implementation, and achievement. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The project supported important investments in institutional strengthening and capacity building for TVETA, as well as revised curricula for priority trades, systems for teacher assessment, and GBV measures that had sufficient support to be sustained. Under the ALP, the developed capacity for distance learning could be an important resource to meet the needs of remote and underserved populations. However, following the August 15, 2021 political crisis and change in government, the project's outcomes are at high risk. Key TVETA senior leaders and technical specialists fled the country. There is no information on how many trained teachers remained in the country, especially considering the new administration's difficulties paying civil service salaries. The project's efforts to train female teachers and students are especially at risk, given new gender segregation policies and norms that limit women's participation in the job market and in daily life. In addition, considering the suspension of most development assistance, the economic situation in the country has degraded significantly; the resulting closure of small and medium enterprises will reduce the demand for skilled TVET graduates. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared on an emergency basis given the imminent transition in 2014. It addressed key development challenges. Lessons learned from the predecessor project included the need to focus on the market for skills, the importance of developing a strategic vision for TVET, the need to decentralize school administration, the key role of training for technical teachers, the likelihood that the private sector would respond positively to opportunities for involvement with governance of TVET institutions, and the importance of certifying TVET institutions. No new Program Implementation Unit (PMU) was initially set up for the project. The project team later confirmed that this decision was in response to a government-wide goal to avoid parallel structures. Given the need to move forward quickly, the existing Skills Team within the DMTVET was to be responsible for overall implementation, including administration of sub-grants to schools and institutes. Financial management and procurement were also to be carried out through the Skills Team, with capacity assessed in December 2012 and found to be adequate. An Implementation Support Plan was put in place (EPP, pp. 23-24). Overall implementation risk was rated Substantial, based on uncertainty surrounding the upcoming 2014 transition and the possibility that a new government might de-prioritize or change the focus of the project; governance issues related to transparency in personnel management and rent-seeking; and weak implementation and monitoring capacity across the TVET system, creating openings for poor governance and fraud. Mitigation measures included the appointment of a Project Steering Committee to oversee and monitor progress, as well as close supervision by the Bank team (especially of personnel recruitment). Ultimately, implementation arrangements were found to be inadequate, as the counterpart lacked adequate procurement and financial management capacity. The results framework lacked indicators with adequate baselines and realistic targets. Under unusually difficult fragility, conflict & violence (FCV) circumstances where some preparatory and readiness steps cannot be fulfilled by task teams, these are considered to be moderate shortcomings. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Despite ongoing security concerns and restrictions posted by the COVID-19 pandemic, the Bank team held in-person or virtual supervision missions twice annually. The 2017 restructuring kept the project aligned with the government's policy priorities and introduced the results-based financing modality. Further restructurings addressed other issues as they arose, including safeguards and GBV issues. According to the project team, a stand-alone PMU was eventually created prior to 2018, functioning within and at the direction of TVETA. The ICR does not mention whether the project's planned steering committee was ever constituted or met, but the project team confirmed that it met on an as-needed basis when significant decisions were required. Technical supervision and support was provided to TVETA on its own internal Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) institutional reforms, based on a Bank-supported institutional assessment and subsequent development of a comprehensive institutional reform plan. However, as the ICR (p. 27) notes, the DMTVET, TVETA, and PMU lacked the necessary capacity to handle financial management and procurement responsibilities as well as the results-based financing modality, and safeguards compliance fell short for the majority of the project period. It was incumbent on the Bank team to provide additional support for counterparts, from the senior level to line staff, to enable compliance with Bank policies and smoother implementation. For example, it is an important shortcoming that the "task team found about TVETA's misunderstanding of the results-based financing modality towards the end of the project, which limited its ability to react effectively" (ICR, p. 28); appropriately attentive supervision would have detected and dealt with this issue in a more timely manner. Nonetheless, in a difficult FCV context, the Bank team pushed back on overambitious goals set by TVETA during the second half of the project (ICR, p. 28) and continued to address weaknesses in implementation capacity. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The DMTVET Skills Team was responsible for project M&E. The initial M&E system was set up around monitoring of physical and financial progress; monitoring of indicators through major surveys at baseline, mid-term, and completion, and smaller annual surveys in interim years; participatory monitoring; and impact evaluation studies and special thematic studies. The project planned to develop a digital MIS, in collaboration with UNESCO, to cover all aspects of the project, and also to develop a separate Labor Market Information System to strengthen the links between skills training and the market for graduates. The initial results framework specified outcome, intermediate outcome, and output indicators that were logically linked and effective measures of achievement of the objectives. Most baselines were to be determined during the first phases of implementation, once participating schools and institutes were selected. A number of the originally-set output and outcome indicator baselines (for percentage of graduates employed within six months and share of female beneficiaries) that had been based on administrative data were later found to have been inaccurate during the mid-term review; the targets were re-set accordingly, retaining the originally targeted percentage change. b. M&E Implementation The results framework was modified during each restructuring (except the final restructuring in 2021) to adjust output indicators as activities evolved. Outcome indicators were revised at the June 2017 restructuring to accommodate the project's narrowed focus and reduced scope. A comprehensive, user- Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) friendly MIS was developed as planned. Prior to restructuring, the project did not provide regular updates on some key outcome indicators, including student placement, and learning outcome data were collected only once. Surveys were limited to NIMA and did not include other participating institutes, and no data were collected on the earnings of graduates. After 2017, data collection improved significantly, with regular updating of key indicators and external verification of most DLIs in 2017-2020. The ICR does not clearly explain the impact of the adoption of results-based financing or the DLIs on the measurement and interpretation of results. c. M&E Utilization The ICR (p. 26) notes that M&E data were "underutilized" to inform management decisions prior to the 2017 restructuring. The ICR reports that, after 2017, TVETA used available M&E data to inform the development and implementation of the national TVET strategy, TVETA institutional reform plan, and the APL. However, the extent to which this was actually the case is uncertain, given the ICR's other statements about the extent to which counterparts struggled to understand the results-based financing modality. Overall, in view of substantial shortcomings in M&E implementation and utilization, overall M&E Quality is rated Modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified under Environmental Assessment category "B" and triggered OP 4.01 on Environmental Assessment. Civil works were expected to have limited environmental impacts because they entailed renovation of existing schools. Any new small buildings were to be constructed on land already owned by the Ministry of Education. An Environmental and Social Management Framework was developed, with environmental guidelines and checklists for contractors. Six site-specific environmental and social management plans (ESMPs) were prepared for sub-projects at each of the six institutes originally envisaged to participate in the project. An environmental focal person was to be recruited by the Skills Team to ensure implementation of safeguards. Safeguards performance was rated satisfactory by Implementation Status Reports (ISRs) until September 2017, when the rating was downgraded to moderately satisfactory due to general low capacity and the lack of a dedicated safeguards specialist. The ISR in April 2018 further downgraded safeguards performance to moderately unsatisfactory because of delays in the recruitment process for this specialist and lack of attention to social safeguards-related requirements. The project had issued contracts for the Blind School without appending cleared ESMPs, a clear instance of non-compliance with the Bank's safeguards policies. The ICR did not state whether or how this issue was resolved, and the project team could not provide further information. Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) In 2019, an assessment of the physical and institutional capacity of the thee Lead Institutes and KAMI flagged two critical safeguards issues: allegations of 3-4 fatal incidents at KAMI and of GBV at NIMA. At KAMI, the project conducted extensive technical assessments, developed initial specifications, and prepared initial procurement plans to rectify shortcomings. However, given the unexpected scope of these works, there was not sufficient time to identify contractors and initiate construction under the auspices of the project. GBV risks were proactively addressed. An Action Plan was developed, and all of its provisions were implemented: formal GBV-related policies, procedures, and codes of conduct were prepared; 1,300 students, teachers, and staff were trained on these new standards; formal, safe, confidential reporting mechanisms were developed; relevant staff supervision and management was strengthened; and infrastructure was improved to address student well-being and safety. The project's safeguards rating was upgraded to moderately satisfactory in the October 2020 ISR. The ICR did not provide a statement on compliance with the Bank's safeguard policies. The project team later confirmed that the project was in compliance at closure. b. Fiduciary Compliance As described in Section 5, financial management capacity was weak. After the 2017 restructuring, the financial management rating was downgraded from moderately satisfactory to moderately unsatisfactory because of ineligible expenditures. The rating did not improve until project closing in June 2021, when it was raised to moderately satisfactory. The ICR does not report on audits. The project team later confirmed that, by project closing, clean audits were completed and accepted by the Bank's financial management team. As described in Section 5, low procurement capacity led to significant implementation delays. The procurement rating was satisfactory until February of 2015, at which time it was downgraded to moderately satisfactory; it was again downgraded to moderately unsatisfactory in June 2016 and to unsatisfactory in December 2016. The downgrades were primarily related to significant wait times for Ministry of Education finalization and approval of contracts. The rating was upgraded to moderately satisfactory in April 2018 after the project awarded three contracts and hired two competent procurement staff. The rating remained at moderately satisfactory through project closing. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) Reason for Ratings ICR IEG Disagreements/Comment Both the original and revised Moderately Moderately development objectives were Outcome Satisfactory Unsatisfactory modestly achieved, and efficiency was modest. There were moderate shortcomings with the project's initial results framework and implementation arrangements. The Client lacked understanding and readiness to undertake Bank Performance Satisfactory Moderately Satisfactory results-based financing. As implementation progressed, the Bank team did not provide adequate support to ensure compliance with safeguard and other key policies, and to facilitate smooth implementation. Quality of M&E Modest Modest Quality of ICR --- Modest 12. Lessons The ICR (pp. 27-28) offered several insightful lessons, including: Successful implementation of technical and vocational education projects hinges on design that matches ambition with realities of project context. In this case, even though the Ministry of Education had limited capacity and the DMTVET was struggling to deliver basic services, the original project set higher-level goals for accreditation, certification, and demonstrating increased earnings of TVET graduates. Setting the bar too high can dilute a project's focus on more fundamental, achievable results. Introduction of results-based financing can produce desired results only when the counterpart fully understands the modality. In this case, capacity at all levels was insufficient to benefit from the RBF incentive structure, a situation that deteriorated even further when the project came under new institutional arrangements and leadership. Institutional capacity building can be usefully incorporated as a TVET project objective in contexts where the skills development sector is relatively new and the foundations for sectoral development are not yet set. In this case, capacity was overestimated at the outset, and the project struggled throughout its lifetime to match capacity with planned activities and outcomes. Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Skills Development Project (P132742) 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was prepared after the August 15, 2021 political crisis. Given the authorizing environment at the time, the ICR team could not interact with the counterpart, project management unit, or other partners involved in project implementation. As a result, the TVETA could not develop its own Borrower's Report or provide comments on the draft ICR. The ICR was therefore based on a review of project documents and relevant Bank staff and TVET development partners. The ICR was clear, internally consistent, and results-driven. It carefully laid out the project's restructurings, the critical assumptions underlying the original project's theory of change, and the evolution of the project across its lifetime. It provided useful tables with additional detail on outcome indicators to supplement the information presented in the regular data sheet. However, key information and analysis were missing. The relationships between the planned outputs and intended outcomes were not always made clear. The ICR did not provide a complete list of DLIs, making it difficult to understand the planned or actual functioning and impact of results-based financing after the 2017 restructuring. There was insufficient detail on implementation efficiency, safeguards, financial management, and procurement. Annex 3 on Project Costs did not provide amounts at approval. The ICR appropriately opted for a split rating methodology, given the revision of the objectives, but it applied that methodology incorrectly, assessing the project in phases rather than against the original and the revised objectives across the project's entire lifetime. a. Quality of ICR Rating Modest Page 18 of 18