Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) Report Number: ICRR0022889 1. Project Data Project ID Project Name P133129 AR - Youth Employment Support Project Country Practice Area(Lead) Argentina Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-84640 31-May-2018 425,000,000.00 Bank Approval Date Closing Date (Actual) 15-Jan-2015 31-Jul-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 425,000,000.00 0.00 Revised Commitment 425,000,000.00 0.00 Actual 425,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project's objective, as stated in the Loan Agreement, was "to improve access of vulnerable youth populations to labor markets, increasing their employability by supporting the expansion and strengthening of Government employment programs." This ICRR concurs with the ICR's assessment (p. 15) that this statement contains a single evaluable objective, as project documents and indicators treat improved access to labor markets and increasing employability as the same thing, and support for the expansion and strengthening of Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) government employment programs is viewed as an output in support of that outcome. Section 4 will therefore assess achievement of one objective. Outcome targets were revised at two project restructurings. A 2018 restructuring made one target more ambitious, which does not require a split rating for assessment purposes. However, a 2020 restructuring revised two outcome targets downward. This ICRR will therefore conduct a split rating based on the 2020 restructuring, at which point US$328.57 million, or 77.2% of total Bank financing, had been disbursed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 28-Mar-2020 c. Will a split evaluation be undertaken? Yes d. Components The project contained three components: 1. Improvement and provision of orientation and training services offered to JMyMT program participants (appraisal: US$190.33 million, of which IBRD US$169.01 million; closing: IBRD US$100.94 million) was to improve the quality and relevance of services offered to participants in the "Youth with More and Better Jobs" (JMyMT) program, ensuring that an integrated package of training services (basic skills and technical training) is offered by providers that are independently assessed and certified, following clear curricula and objectives. It had four subcomponents:  Labor orientation workshops and training courses was to provide required workshops and development of a work program for each participant entering the program.  Expansion and strengthening of a job competency-based certification system was to finance the development of labor competency-based standards, the design of curricula and corresponding teaching materials, training of evaluators, assessment of participants' labor competencies, accreditation and quality certification of professional training institutions, teacher training at those institutions, and strengthening of Sector Councils responsible for overseeing training programs. These activities were to be financed on an output-based model.  Strengthening of capacity to provide training and employment services was to provide information technology (IT) equipment and related furniture for about 150 municipalities for use of program participants in orientation and job search workshops, job clubs, and IT courses.  Technical coordination was to strengthen project implementation. 2. Coverage consolidation of the JMyMT program (appraisal: US$475.69 million, of which IBRD US$169.9 million; closing: IBRD US$262.77 million) was to maintain the program's coverage at around Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) 200,000 youth per year, reaching about 35% of potential beneficiaries. It was to finance cash transfers to eligible participants enrolled in the program, including monthly stipends (for those participating in program activities and on-the-job training) and incentives linked to completion of specific program activities. 3. Strengthening and improving the performance of the network of Employment Offices (EOs) (appraisal: US$100.21 million, of which IBRD US$85 million; closing: IBRD US$60.23 million) was to incorporate new methodologies and management practices in existing and new EOs. The Ministry of Labor, Employment and Social Security (MTESS) was to enter into an agreement with each participating province, municipality, or non-governmental organization specifying the services to be delivered by the EO, including targets on performance indicators and the administrative procedures to be followed to monitor implementation. It had two subcomponents:  Strengthening the employment services network was to finance activities such as regional and national workshops and provision of technical assistance to participating EOs.  Output-based financing for eligible EOs was to provide support linked to outputs that MTESS considers most critical: placements in formal jobs, enrollment in/provision of services of employment clubs, placement in private sector on-the-job training projects, and enrollment in professional training courses. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates At appraisal, total project costs were US$767.29 million, with US$425 million from the International Bank for Reconstruction and Development (IBRD) and US$342.29 million from the Borrower. The project fully disbursed as planned. The project was approved on January 15, 2015 and became effective on May 7, 2015. It was restructured three times:  On October 27, 2016, to change institutional arrangements and reallocate proceeds among disbursement categories, and to revise one output target. Specifically, to reflect new government priorities following the 2015 presidential elections, new survey and evaluation activities were introduced, and IT training activities were cancelled as the MTESS decided to proceed with its own funds.  On February 6, 2018, to revise components and costs, adjust the results framework, and extend the closing date. Following a 2015 change in presidential administration and the June 2017 project mid- term review, it was decided to prioritize private-sector employment opportunities for program beneficiaries (through increased emphasis on on-the-job training), and funds were shifted from training to stipends. The results framework was modified, with several output targets adjusted (targets on placement in formal employment were revised downward to take into account more challenging labor market conditions), and one outcome target was made more ambitious. The closing date was extended due to slow progress during the first two years of implementation.  On March 28, 2020, to revise components and costs, adjust the results framework, and extend the closing date. Following presidential elections in 2019, the project was amended to ensure continuous support for the JMyMT program under difficult macroeconomic conditions as well as full disbursement of loan proceeds. The closing date was extended for another 16 months; technical assistance activities were added to assess the effectiveness of employment policies; and funds were Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) added for both training and stipends. Two outcome targets on formal employment of graduates were made less ambitious because of unfavorable economic conditions, and two new outcome indicators were added for better measurement of labor market insertion rates (comparing program participants with a comparison group, to control for the effects of the overall labor market). The project closed almost three years later than originally scheduled, on July 31, 2021. 3. Relevance of Objectives Rationale The objectives were strongly relevant to Argentina's country conditions. While there had been significant gains since the 2002 economic crisis in overall growth and poverty reduction, pronounced inequalities between urban and rural areas remained. A growth slowdown starting in 2012 had created challenges to employment generation. After reaching its lowest level of 6.7% in 2011, the unemployment rate had started to increase gradually and was 7.5% in the second quarter of 2014. There was also a decline in labor force participation rates, producing an overall decrease in the percentage of the population employed by nearly 1.5 points from 2011 to 2014. Access to formal, stable jobs remained out of reach for many workers, especially for youth, and even more so for young women. At the time of appraisal, government policy had been shifting away from income-support programs toward active labor market interventions. Responding to the need to structure pathways into work for unemployed youth, the government launched an employment and skills training program (JMyMT) for the 18- to 24-year- old population in 2008, targeting unemployed youth with incomplete secondary education. It covered interventions in labor orientation, finishing secondary school, accumulating experience and skills through internships, providing specific training, and finding productive employment either as salaried workers or through self-employment. By 2014, the program covered all provinces and municipalities containing 85% of the youth population, and about 200,000 youth had participated in one or more program activities each year. In 2011-2012, the formal employment rate of former JMYMT participants was 20.8%, compared to 9.1% in a control group. Building on that success, in January 2014 the government launched a complementary initiative, "Support to Argentina's Students" (PROGRESAR), aimed at completion of formal education by youth with a family income below minimum wage, as well as additional training opportunities. The government recognized that, while these programs had been successful, there was a need to make them even more effective and inclusive by increasing Employment Offices' (EOs') engagement with and registration of participants in training activities, and by guaranteeing acceptable standards of services throughout the country, including professional training and opportunities for work experience. The objectives remained highly relevant to government strategy through successive presidential administrations during the project's lifetime; while the 2015-2019 center-right government prioritized on-the-job training and the participation of employers, it remained committed to improving the employability of vulnerable youth. The objectives were also highly relevant to Bank strategy at appraisal and closing. The FY2015-18 Country Partnership Strategy contained a thematic engagement area of "Asset Availability of Households and Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) People," under which the results area "Improving Employability of Argentina's Youth" contributed to a specific outcome of doubling the relative chance of labor market insertion for 400,000 young people. The project also employed an output-based approach, which the CPS identified as a key lesson to guide the design of new operations. The FY2019-22 Country Partnership Framework had an objective on "Building Skills for the Future," with a focus on increasing training and retraining programs, including those who have left the formal education system without the necessary skills to find high-quality and productive jobs. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve access of vulnerable youth populations to labor markets, increasing their employability by supporting the expansion and strengthening of Government employment programs (original outcome targets) Rationale The theory of change underpinning both the project and the JMyMT program it supported held that improving and expanding the supply of orientation and training services, expanding the demand for these services among vulnerable youth through stipends, and strengthening and expanding the intermediation activities of EOs to connect vulnerable youth to both orientation and training services and employment opportunities, would increase vulnerable youth access to labor markets and increase their employability. Outputs The number of certified professional training institutions increased from 150 in 2014 to 250 in 2021, meeting the target of certifying 100 institutions.  163 labor competency norms were developed and formally registered.  1,225 individuals were trained as professional training instructors.  26 Sector Councils for Lifelong Learning and Labor Competency Certification were strengthened. The ICR did not specify the characteristics of this strengthening. 93,185 program participants took a professional training course under the project, not reaching the original target of 170,000, but exceeding the revised target of 91,500 participants. 162,294 program participants joined an on-the-job training activity in the private sector under the project, exceeding both the original target of 78,000 and the revised target of 85,000 participants. Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) 481,235 program participants joined a labor orientation workshop under the project, exceeding both the original target of 270,000 and the revised target of 380,000 participants. The project provided financial support to the MTESS for technical assistance activities aimed at strengthening the EO network, as well as output-based financial incentives to EOs for the delivery of training and labor mediation services.  32,045 program participants were placed in a formal job by an EO under the project, exceeding both the original target of 30,000 and the revised target of 25,000 participants.  11,616 female program participants were placed in a formal job by an EO under the project, not reaching the original target of 12,000, but exceeding the revised target of 10,000 participants.  131,012 program participants were enrolled in an employment club by an EO under the project, exceeding both the original target of 75,000 and the revised target of 70,000 participants.  53,056 program participants were enrolled in a professional training course by the EO and placed in on-the-job training in the private sector under the project, not reaching the original target of 80,000, but exceeding the revised target of 46,500 participants. Outcomes The number of beneficiaries of labor market programs increased from 660,000 in 2014 to 1,471,462 in 2021, exceeding the target of 930,000 beneficiaries.  Of those, the number who accessed the services of EOs increased from 630,000 in 2014 to 1,389,000 in 2021, exceeding the target of 965,000 beneficiaries. (Note: This metric was duplicative. If a participant received more than one service, they were counted more than once.)  The number of beneficiaries of training and retraining programs rose from 330,000 in 2014 to 685,446 in 2021, not reaching the target of 780,000 beneficiaries (note: as above).  The number of female beneficiaries of labor market programs increase from 347,160 in 2014 to 777,966 in 2021, exceeding the target of 489,180 beneficiaries. The percentage of JMyMT program participants who have left the program and are employed in the formal market was 25.2% at baseline in 2014, but fell to 20.6% in 2021 (see reasons below), not reaching the original target of 27%.  The percentage of female JMyMT program participants who have left the program and are employed in the formal market was 16% at baseline in 2014, but fell to 13.5% in 2021, not reaching the original target of 22%. The number of EOs obtaining at least 150 outputs in the preceding 12 months rose from 60 in 2014 to 159 in 2021, exceeding both the original target of 120 and the revised target of 140 EOs. The formal labor market insertion rate of participants immediately after completing the first training activity was 25.8% in 2021, while it was only 4.9% in 2021 for a comparison group. For females, the rate of formal labor market insertion for program participants immediately after completing the first training activity was 17.3% in 2021, while it was only 2.6% for a comparison group. This indicator was added at the Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) 2020 restructuring to take into account macroeconomic and labor market conditions. The target, a 20% difference between the treatment and comparison groups, was far exceeded. However, as the ICR (p. 17) notes, non-randomized selection criteria for the two groups may have produced systematic bias that obscured other factors, including a higher rate of informal employment in the comparison group; the comparison group was chosen ex post, and it included enrollees who did not participate in any program activities. In addition, the PAD (p. 4) had noted significant differences between a treatment and comparison group on this same indicator in 2011-2012, calling into question the project's contribution to improving the relative situation of the treatment group. Despite achievement of most outputs, targets for employment of program participants in the formal labor market were not met, likely because of the highly adverse macroeconomic context (ongoing recession and a 10% drop in GDP in 2020 due to the COVID-19 pandemic, causing an increase in the unemployment rate from 6.5% in 2015 to 11.4% in 2020). Achievement of the objective under the original outcome targets is therefore rated Modest. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective Improve access of vulnerable youth populations to labor markets, increasing their employability by supporting the expansion and strengthening of Government employment programs (revised outcome targets) Revised Rationale The percentage of JMyMT program participants who have left the program and are employed in the formal market was 25.2% at baseline in 2014, but fell to 20.6% in 2021, not reaching the revised target of 24%. The percentage of female JMyMT program participants who have left the program and are employed in the formal market was 16% at baseline in 2014, but fell to 13.5% in 2021, not reaching the revised target of 19%. With revised targets not met, achievement of the objective under the revised outcome targets is rated Modest. Revised Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) Rationale Although there was strong achievement of intermediate results, these did not fully translate into gains in formal employment among program participants. The ICR speculated convincingly that the shortfall was due to the unfavorable macroeconomic environment, especially from 2018 forward, further exacerbated by the COVID-19 pandemic in 2020. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency At appraisal, a cost-benefit analysis found an internal rate of return (IRR) of 23.1%, acknowledging that many of the project's benefits -- increase in human capital accumulation and social cohesion, reduction in inequality and risk of crime and violence -- could not be quantified. The analysis (PAD, pp. 59-67) compared the costs of the project with the expected flow of additional income to be received by participants over their lifetimes. It was reasonably assumed that program participation would positively affect a young person's probability of being employed, formality in employment, and salary, and that these differences would continue throughout their economically active life. The number of beneficiaries was estimated using historical JMyMT data and trends. Estimations for wages, employment status, and formality for the treatment and control groups were drawn from the 2013 urban household survey. The ICR's ex post analysis took into consideration the actual costs associated with the program and anticipated economic benefits based on the actual number of beneficiaries and increase in the likelihood of being formally employed relative to the comparison group. The results were based on a finding that, on average, the program produced US$5,087 in net salary gains per participant over their economically active lifetimes. Using a discount rate of 3%, the assessment found a net present value of US$4.1 billion and an IRR of 22.2%. Operational efficiency was adequate. The project's closing date was extended twice for a total of 28 months, but these extensions contributed to full disbursement of the loan during a period of time when the Argentine Peso underwent significant devaluation. Project outputs exceeded what was planned, and unitary costs per participant and unit of service were lower than anticipated (ICR, p. 47). However, there were some shortcomings. Monetary amounts assigned to the output-based financing mechanisms sometime lagged behind inflation, eroding their incentive potential. Early in the project period, shortages of training services meant that participants often experienced "significant delays" after enrolling in JMyMT or completing an orientation activity (ICR, p. 25). The incompatibility of the Universal Child Allowance program (in effect through 2016) with JMyMT discouraged participation in the latter; beneficiaries were not permitted to enroll in both programs simultaneously. Similarly, changes in 2015 to the eligibility criteria and payment for PROGRESAR participants weakened incentives to participate in JMyMT, resulting in migration of over 100,000 people to PROGRESAR that year (ICR, p. 24). Finally, while the project's objectives remained a priority among successive presidential administrations, shifts in ministerial arrangements and personnel slowed project implementation. Efficiency Rating Substantial Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  23.10  Not Applicable 100.00 ICR Estimate  22.20  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project was highly relevant to country conditions, government strategy, and Bank strategy across its lifetime. Relevance is rated High. Although it achieved or overachieved most of its planned outputs, the project did not reach either its original or revised targets for formal labor market insertion of participants, likely due to unfavorable macroeconomic conditions. The outcome indicators added at the last project restructuring to control for the poor labor market were not adequate indicators of achievement. Efficacy is therefore rated Modest under both the original and revised outcome targets. Efficiency is rated Substantial due to a favorable economic analysis and indications of adequate implementation efficiency. Taken together, these ratings indicate that there were significant shortcomings in the project's preparation, implementation, and results, producing an Outcome rating of Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome According to the ICR (p. 28), transition after project closing should be smooth, as the project was implemented using existing MTESS organizational and management structures. The JMyMT program continues to receive both political support and federal budget funding, and a follow-on project is under preparation by the Bank. Institutional strengthening achieved under the project, including improvement in competency-based training and certification programs, the incorporation of output-based financing mechanisms for service providers and EOs, strengthening of the MTESS management information system, and expansion and strengthening of the EO network, is likely to be sustained. The International Monetary Fund is projecting improved economic growth for Argentina in the near term, which should reactivate labor markets and therefore enhance potential labor market outcomes among former project beneficiaries. Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) 8. Assessment of Bank Performance a. Quality-at-Entry Project design drew from the Bank's extensive prior experience in active labor market programs in Argentina over the preceding two decades (PAD, p. 11), including a seasoned task team. Lessons learned from predecessor projects included the adequate capacity of the government to deploy income transfer and employment programs, but also the importance of transparency and accountability in the administration of benefits, and the need for sound monitoring systems to track funds and results and to ensure compliance with eligibility criteria and program regulation. Previous experience also drew attention to the need for: (a) emphasis on outcomes, rather than inputs or processes, to maintain focus on results; (b) use of output-based financing mechanisms wherever possible; (c) recognition of the multi- dimensional challenges of youth employment, calling for multi-pronged project design; (d) development of a strong monitoring system at the outset against agreed standards; (e) adequate budget availability for activities related to the indigenous peoples safeguard; (f) anchoring the competency-based training system in a demand-driven approach determined by the needs, interests, and characteristics of employers; (g) flexible and modular provision of services; and (h) focus on the performance of EOs. Based on these lessons and experiences, project design developed multiple strategies to reach different segments of the target population, relying on a wide network of service providers that included non- governmental organizations, labor unions, private training providers, public universities, EOs, and employers to deliver orientation and training services. Risk assessment at entry was rated substantial overall due to the potential impact that macroeconomic trends might exert on the labor market (PAD, p. 13). Individual risks were rated low to moderate, with specific attention to the complex financial structure for disbursement of loan proceeds; mitigation involved careful design of disbursement mechanisms and strong planned implementation support from the Bank, especially over the project's first few months. There was also political risk stemming from planned national elections in October 2015, though support for youth employment policies across the political spectrum combined with strong engagement of the Bank team with policy makers across all sectors were mitigating factors. Risk related to stakeholder coordination was identified at appraisal (PAD, p. 56), but with no substantive risk management measures. Weak coordination at the local level materialized during the early stages of implementation and resulted in a limited supply of training services (ICR, p. 25). Participants experienced significant delays after enrollment. The limited supply of training offerings also hindered the implementation of skills development pathways tailored to the individual needs of participants, who had to settle for whatever training was available. It was also a shortcoming that the results framework did not contain outcome indicators that could adequately measure the project's contribution to achievement of the objectives. As explained earlier, this issue was caused by the severe negative impact of the economic downturn on employment opportunities, which could not have been fully anticipated by the task team at appraisal. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) The task team was based in the Bank's Buenos Aires office, contributing to close and ongoing support for project implementation. During multiple restructurings, the Bank team kept up with changes in the priorities of various presidential administrations as well as the overall implementation environment. Bank guidance and support in the areas of financial management, procurement, and safeguard policies were effective. Changes to the results framework at the 2020 restructuring partly addressed shortcomings, though the definition of the comparison group weakened the utility of the added outcome indicators. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Results monitoring, and evaluation were to be based on three sources of data: the management information system (MIS) of the Employment Secretariat, where detailed data on activities related to JMyMT were collected; the database for social security contributions (SIPA), managed by the Federal Tax Authority; and the information system that integrates performance data from all participating EOs. All training activities were to be registered on the Register of Training and Employment Agencies, an MIS used to analyze the program's performance. Given that the project used an output-based model, the monitoring system for outputs was considered critical, and its performance was to be monitored carefully by the Bank team, particularly in the first year of project implementation. MTESS was also to continue with the regular impact evaluation strategy that had been in place for MJyMT since its creation, which combined data from the program's registry, the SIPA database, and the Permanent Household Survey. The results framework contained adequate indicators to measure intermediate results related to beneficiaries taking advantage of services and institutional strengthening. However, the potential impact of a highly negative macroeconomic environment was not anticipated during preparation, leaving the two most important outcome indicators inadequate to measure project achievement, as formal employment levels were negatively affected by labor market conditions outside the project's control. b. M&E Implementation During implementation, the MTESS published monthly reports that covered project indicators, and the Bank also monitored the EO MIS for data on outputs. External technical audits were conducted regularly to ensure the internal consistency of the various databases and registries on project participants and activities, including stipend payments, registration of trainees with the social security agency, and formal labor market insertion of program participants. At the 2020 project restructuring, two additional outcome indicators were added to single out the results attributable to the program. However, those indicators -- on differences in labor market insertion rates for Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) participants who completed at least one orientation or training activity versus those who had not -- may have contained systematic bias in the definition of the comparison group, limiting their utility. In addition, the planned impact evaluation strategy was not implemented due to loss of key staff and internal reorganization at MTESS. c. M&E Utilization The Employment Service's MIS was a critical tool available to inform program implementation, but the ICR reported that "this potential remained largely untapped." Furthermore, "MIS interfaces at the EO level were conceived largely as a data collection tool rather than as management tools that could be used by EOs to improve their performance" (ICR, p. 26). Findings based on MIS data were used, however, to inform a 2019 labor market and social protection diagnostic that is feeding into the design of a follow-on operation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was rated Environmental Assessment category "C." The nature and scope of the project were not expected to generate adverse environmental impacts. During implementation, the environmental risk rating and environmental safeguard performance were defined as low and satisfactory, respectively (ICR, p. 27). The project triggered OP/BP 4.10, Indigenous Peoples (IP). JMyMT beneficiaries included indigenous people in 20 of Argentina's 24 provinces. A National Indigenous Peoples Plan (NIPP) was prepared to ensure that their rights and interests were adequately protected during project preparation and implementation. The NIPP was based on existing analogous plans implemented under previous Bank- financed projects. It was disclosed through the MTESS website on November 18, 2014 and on the Bank's website on November 13, 2014. During implementation, the social risk was rated low and social performance was considered satisfactory (ICR, p. 27). The Indigenous Peoples Planning Framework set a goal of 6,500 IP youth participating in the project, which was exceeded with actual participation of 7,167 IP youth. The project also contributed to the institutionalization of an intercultural approach within the MTESS, encompassing a new Gender and Intercultural Guide for EOs and participation of the IP area in the preparation of a follow-on project. b. Fiduciary Compliance A Financial Management (FM) Assessment carried out during appraisal found arrangements to be satisfactory to support project implementation. Administrative and financial management matters were handled by the Technical Coordination Unit of the MTESS, including collecting required information on Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) outputs and financial flows, ensuring that necessary audit reports were prepared on time, requesting disbursements as necessary, and preparing reports and submitting Statements of Expenses according to the procedures established in the Operational Manual. FM was rated as either Satisfactory or Moderately Satisfactory and the FM risk rating was moderate throughout the project's lifetime. All financial statement audit reports were received on time or with a delay of less than four months, and all expressed clean opinions. All interim financial reports were considered acceptable, though some were received with delays. Project planning, budgeting, accounting, internal controls, funds flow, financial reporting, and auditing arrangement were adequate (ICR, p. 27). A procurement assessment at appraisal identified several factors of concern: the risk of delays in processing procurement activities to set up 150 classrooms in a timely manner, and lack of staff at MTESS with procurement experience. According to the ICR (pp. 27-28), there were procurement delays, and some weaknesses were identified during implementation in relation to the updating of information in the Bank's STEP (Systematic Tracking of Exchanges in Procurement) system and filing of documentation of procurement processes. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR (pp. 29-30) offered several insightful lessons, some of which are restated here: Output-based financing for skills development and training programs can be fine-tuned by project design teams to incentivize not just production of desired outputs, but also their relevance and quality -- in other words, not just "more" outputs but also "better" ones. In this case, payments to training providers rewarded not just enrollment but also completion. Following this model, incentives Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) to employers could reward aspects such as tutoring and mentoring, as well as permanent hiring. Payments to Employment Offices could go beyond enrollment to focus on beneficiaries' eventual insertion into the labor market. Project design teams targeting vulnerable youth for a skills development and training program can be maximally inclusive by addressing the diversity of sources of vulnerability within the target population. Vulnerability can stem from gender, indigeneity, socio-economic characteristics, and geographic location, all of which might require different strategies for outreach, intervention, and service provision. In an uncertain macroeconomic environment, the usefulness of outcome indicators related to skills training and active labor market interventions can be negatively impacted if those indicators depend on fluctuating overall employment levels. In this case, key outcome indicators initially focused on changes in employment outcomes without considering the possible influence of macroeconomic factors outside the project's control. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, concise, and candid, with high-quality evidence presented consistently throughout the document. It interrogated the evidence well, linking available data to intended outcomes and highlighting shortcomings in the project's formal results framework. Its lessons were insightful and well grounded in the project's experience presented in the main narrative. Although the ICR's rationale for performing a split rating was well reasoned, the efficacy assessment was based on project periods (before and after restructuring) rather than on achievement across the project's entire lifetime against the original and then revised outcome targets (per IEG/OPCS harmonized guidelines). a. Quality of ICR Rating Substantial Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR - Youth Employment Support Project (P133129) Page 15 of 15