Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Report Number: ICRR0021883 1. Project Data Project ID Project Name P145782 CO Access with Quality to Higher Educa. Country Practice Area(Lead) Colombia Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-83540 30-Jun-2019 200,000,000.00 Bank Approval Date Closing Date (Actual) 01-Apr-2014 15-Mar-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 0.00 Revised Commitment 200,000,000.00 0.00 Actual 200,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Salim J. Habayeb Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (page 5), the objective of the Project was “to increase student enrollment, graduation and equity in higher education, by: (a) increasing the number of ACCES Student Loans and the ACCES Program's focus on students from disadvantaged socioeconomic backgrounds and in quality higher education institutions and programs; and (b) enhancing the Borrower's institutional capacity. Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) The Borrower’s institutional capacity refers to Colombia’s student loan agency, Instituto Colombiano de Credito Educativo y Estudios Tecnicos en el Exterior (ICETEX). Achievement of project objectives will be assessed separately by “increasing the number of ACCES Student Loans and the ACCES Program's focus on students from disadvantaged socioeconomic backgrounds and in quality higher education institutions and programs” applied separately to student enrollment, graduation, and equity in higher education. “Enhancing the Borrower’s institutional capacity” will be assessed as an output supporting each of the three objectives of enrollment, graduation, and equity. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project had two components: Component 1: Student Loans (estimated cost at appraisal and actual cost: total cost – US$424 million; IBRD - US$200 million; ICETEX – US$135 million; Government – US$89 million). The component was to increase student enrollment and graduation and improve equity in higher education by focusing ACCES loans on students from socioeconomically disadvantaged backgrounds and quality higher education institutions and programs. It would finance provision of ACCES student loans to eligible students to finance their tuition and/or subsistence in eligible Higher Education Institutions to pursue undergraduate degrees through both new loans and/or loan renewals for continuing students. Component 2: Institutional Strengthening (estimated cost at appraisal and actual cost: total cost US$12 million; ICETEX – US$12 million. IBRD did not participate in the financing of this component). The component was to strengthen ICETEX's management practices, explore further diversification of financing sources and carry out an impact evaluation in order to enhance ICETEX's ability to increase student enrollment and graduation and improve equity in higher education. It had four sub-components: Subcomponent 2.1: Strengthening ICETEX's loan administration and portfolio management by (i) developing differentiated collection strategies based on portfolio risks, and (ii) upgrading of information systems (hardware and software). Subcomponent 2.2: Strengthening ICETEX's capacity to manage its ACCES program through using scorecards and supervising the implementation of contracts with higher education institution. Subcomponent 2.3: Provision of support for ICETEX's diversification of financial resources through, inter alia, the carrying out of: (i) studies, including on alternatives to expand its funding sources and to develop an endowment fund, and on partnerships with the private sector; and (ii) fundraising activities. Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Subcomponent 2.4: (i) The carrying out of: (i) impact evaluations of ICETEX's student loans; (ii) tracer studies to track employment of graduates of higher education institutions that had received student loans provided by the Borrower, including ACCES student loans; and (ii) the provision of support for ICETEX's overall Project coordination, implementation, and monitoring including, inter alia, the carrying out of audits. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost: The original and actual total project costs were US$436 million. Financing: The project was financed by a US$200 million IBRD Loan; and by ICETEX with a contribution of US$147 million, as originally planned. Borrower contribution: The Borrower provided US$89 million. Dates: The project was approved on April 1, 2014, with a Closing Date of June 30, 2019. It became effective on January 28, 2015. With the Bank loan fully disbursed and project-related activities completed ahead of schedule (the last disbursement on the IBRD Loan was made in January 2017, and the completion and delivery of the project’s impact evaluation was made in May 2018), the project was closed on March 15, 2019. 3. Relevance of Objectives Rationale Project objectives are relevant to the country situation, government policy and Bank strategy. The emphasis on closing access and quality gaps addresses existing inequalities in a higher education system with high enrollment and graduation rates but strong distinctions between regions and among ethnic groups; and that emphasis is aligned with Colombia’s national development strategy. The project also continues a long engagement in higher education by the Bank with ICETEX and the government: starting in 2002 with an Access to Higher Education Loan, and continuing in 2008 with phase one of an Adaptable Program Loan and now the project under review. The project objectives are aligned with the Country Partnership Framework (CPF) FY16-FY21, where it contributes to the second out of three pillars - Enhancing Social Inclusion and Mobility through Improved Service Delivery. Specifically, project objectives remained generally consistent with CPF Objective 4 "Improved Access and Quality of Education". In terms of improved quality, the CPS noted that the Bank will shift its support towards education quality and integration into labor markets, and emphasized education quality from early childhood through tertiary levels. The project did not explicitly include a quality improvement objective, but it had reasonably assumed that favoring enrollment in accredited Higher Education Institutions (HEI) would lead to improved education for enrollees, although the stipulation linking student loans to accredited HEIs was abolished in July, 2018. The ICR itself stated that future projects should explicitly include quality as a key development objective (ICR, p. 25). It also noted (ICR, p.11) that the additional financing (P166177) which was effective since 2019 under PACES Project provided US$320 million to improve the quality of tertiary education. Rating Relevance TBL Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase STUDENT ENROLLMENT in higher education, by (a) increasing the number of ACCES Student Loans and the ACCES Program's focus on students from disadvantaged socioeconomic backgrounds and in quality higher education institutions and programs; and (b) enhancing ICETEX's institutional capacity. Rationale This objective was to be achieved by expanding tertiary education for vulnerable students and reducing their financial barriers to entry by increasing the number of ACCES student loans and lowering the cost of attending and remaining in higher education institution. Supporting achievement of this objective (as well as achievement of Objectives 2 and 3 below) were measures to improve ICETEX’s institutional knowledge base, modernize information technologies, and improve its financial management and administration of loans. The ICR reported the following achievements regarding the institutional capacity of ICETEX: Financial sustainability. ICETEX’s assets under management increased, achieving all targets; and a AAA credit rating was maintained. Financial innovation. ICETEX pursued legislation and regulatory changes to introduce new financial products: (i) successfully introducing policy for investing resources in dormant accounts; (ii) successfully pursued the approval of the regulatory framework for income-contingent loans for financing higher education; (iii) advanced regulations for opening of tax-incentivized savings accounts for children’s university studies; and (iv) advanced the setting up a fund to collect private sector donations to facilitate access to higher education. Loan administration. With adjustments to billing and collection procedures, the non-performing loan ratio went from 16.1 percent in 2012 to 8.17 percent in 2017. Information systems upgrade. The implementation of a strategy to improve information technologies in ICETEX, which should allow for more agile internal operations, better information management, and more responsive service to students. Corporate governance. In collaboration with the International Finance Corporation, a diagnosis and an action plan was developed to improve ICETEX’s corporate governance structure, and is currently being implemented. Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Higher education institutions’ support. ICETEX undertakes regular visits to 178 universities to support students in managing their loans, compared to a target of 90 universities. Drawing on an improved ICETEX, Objective 1 was to be achieved by increasing the number of ACCESS student loans and decreasing the financial burden of attending and remaining in higher education. Outcome: The number of students enrolled in undergraduate degree programs and professional technical ones rose from a baseline of 1.84 million students in 2012 to 2.24 million students in 2017, almost achieving the target of 2.28 million students. The ICR reports the following intermediate results related to this outcome:  Achievements regarding ICETEX institutional capacity (listed above).  The ACCES program improved the likelihood that a beneficiary student would enroll in college by 20 percent (exceeding more modest performance in other Latin America countries).  The overall number of ACCES loans increase by less than targeted: the number of new loans decreased from an average of 40,000 at project preparation to about 20,300 in 2018, compared to an expected 44,000; while the number of ACCES student loan renewals rose from 158,000 to 186,000, compared to a target of 218,000, i.e. reaching only 47 percent of the target. The ICR suggests this was related to an increase in the average cost of loans. The cost of loans per student rose both because a higher proportion of loans was going to students accepted to expensive, high-quality institutions, as well as because of increases in tuition costs in Colombia across educational institutions. Rating Substantial OBJECTIVE 2 Objective To increase STUDENT GRADUATION from higher education, by (a) increasing the number of ACCES Student Loans and the ACCES Program's focus on students from disadvantaged socioeconomic backgrounds and in quality higher education institutions and programs; and (b) enhancing ICETEX's institutional capacity. Rationale This objective was to be achieved by lowering the cost of attending and remaining in higher education; increasing enrollment in accredited Higher Education Institutions and programs, and increasing the proportion Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) of high-ability students. Supporting achievement of this objective were the measures to improve ICETEX’s institutional knowledge base, modernize information technologies, and improve its financial management and administration of loans listed under Objective 1 above. Outcome: Graduation numbers grew from 241,000 in 2012 to 365,000 in 2017, almost achieving the target of 375,000. The ICR reports the following intermediate results related to this outcome:  Achievements regarding institutional capacity of ICETEX, listed under Objective 1 above.  The average graduation rate rose from 4.7 percent during the decade preceding the project to 6.7 percent during the life of the project, from 2012 to 2017.  Dropout rates, a key factor in influencing graduation, decreased by about 7 percent.  Improvement in student performance reflected in an increased share of college subjects passed or approved (by 3 to 4 percent) compared to non-ACCES students, and maintenance subsidies reducing drop-out rates from 39 percent to 16.5 percent for students who receive them. Rating Substantial OBJECTIVE 3 Objective To increase EQUITY in higher education, by (a) increasing the number of ACCES Student Loans and the ACCES Program's focus on students from disadvantaged socioeconomic backgrounds and in quality higher education institutions and programs; and (b) enhancing ICETEX's institutional capacity. Rationale This objective was to be achieved by increasing access to higher education (i) regionally, by increasing the number of students from disadvantaged zones; (ii) socioeconomically, by increasing the number of students from socioeconomically vulnerable households; and (iii) ethnically, by increasing the number of students from vulnerable minorities. Supporting achievement of this objective were the measures to improve ICETEX’s institutional knowledge base, modernize information technologies, and improve its financial management and administration of loans, listed under Objective 1 above. Outcomes: The proportion of new higher education students from families earning less than two minimum salaries rose from a baseline of 58 percent in 2012 to 64 percent in 2017, achieving the target. Other poverty measures also indicated improvements related to equity: using Colombia’s multivariate poverty index, enrollments from households at the lowest level increased from 23 percent in 2012 to 31 percent in 2016; Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) using quintiles, enrollments from households in the bottom two quintiles rose from 21 percent in 2013 to 27 percent in 2017. The project’s impact evaluation indicated that ACCES loans had a larger positive impact on enrollment and academic performance for students from the lowest socio-economic strata. Targeted only at the poorest, they reduced drop-out rates by 23 percentage points and increased graduation rates by 13 percentage points. In its support for vulnerable minorities, 50 percent of ACCES loans were allocated to vulnerable minorities compared to 9 percent in 2012. Improvements in equity between regions were more modest. While favoring remote areas, the proportion of students from remote areas fell from a baseline of 4 percentage points in 2012 to 2.9 percentage points in 2018, compared to a target of 6.5 percentage points. This reflects a small universe of candidates, few institutions of higher education, and low academic achievement. As in the case with Objectives 1 and 2, achievements regarding institutional capacity in ICETEX also contributed to the outcomes noted under this Objective 3. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The undergraduate degree programs and professional technical programs covered by this student-loan project saw increases in enrollment numbers and graduation rates increased over the project period (almost meeting targets), and the proportion of poor students increased, suggesting improved equity. These achievements were (arguably) also supported by improvements in the institutional capacity of ICETEX (Colombia's student loan agency). Overall Efficacy Rating Substantial 5. Efficiency Economic analysis was performed on project components 1 (student loans) and 2 (institutional strengthening) at appraisal and for the ICR. Both analyses used the same assumptions, albeit in the case of the ICR the analysis was updated with the latest data. The assumptions are judged reasonable – a working life of 40 years, wage increases adjusted for graduate degree as the main source of benefits, effects of unemployment, etc. At appraisal, the project was expected to yield an overall Internal Rate of Return (IERR) ranging from 16 percent to Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) 29 percent, depending on degree levels and subsidies received. The internal rate of return for the investment was also positive for Higher Education Institutions (15-24 percent, depending on the degree and stratum) and ICETEX (0.2 percent as a whole. The rate of return to society estimated at the appraisal cost-benefit analysis was 18 percent. The ICR calculates that the project yielded an overall IERR of 13 percent. The rate of return for students was estimated at 5.3 percent to 14.5 percent, depending on degree level and subsidies received. The rate of return for ICETEX was estimated at 7 percent to 14 percent, depending on the type of student. Sensitivity analysis indicated these results held under various scenarios. The IERRs overall and for students were marginally lower than at appraisal due to cost increases, while the IERR for ICETEX improved substantially from the appraisal estimate of 0.2 percent, due to the way interest on subsidies was calculated and the decrease in drop-out rates. Other factors also suggested strong efficiency – a dynamic project execution with on-time signing, as well as accelerated disbursements and implementation. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  18.00  Not Applicable 100.00 ICR Estimate  13.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated substantial, as the PDO was relevant to the country situation, government policy and Bank strategy. Objectives were all rated substantial, having moderate shortfalls, and efficiency was rated substantial. a. Outcome Rating Satisfactory Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) 7. Risk to Development Outcome The project’s objectives were strongly anchored in the government’s higher education improvement policy. It was preceded by two Bank projects, one focused on improving access, and the other a first phase of the current project. The Bank’s relationship with the education sector in Colombia is continuing through the Access and Quality in Higher Education Project that was approved by the Board in January 2017, and an additional financing of another US$160 million for this operation became effective on January 2019. This new operation, also in partnership with the government and ICETEX for the higher education sector, seems likely to ensure that key outcomes and priorities of this project will continue to be monitored far beyond the original closing date. At the same time, changes in political leadership -- and, possibly, in priorities -- introduce a degree of uncertainty into a program that has its budget renegotiated annually. Already, ICETEX itself has introduced a change in emphasis of lending towards students in (more expensive) accredited HEIs, which is progressively increasing program costs and may in the future strain budgets and ICETEX’s lending capacity if not carefully monitored. Continued Bank involvement does bring some stability into this environment. 8. Assessment of Bank Performance a. Quality-at-Entry Project preparation benefited from the long-term engagement (since 2002) of the Bank with ICETEX in providing student loans (see also Section 3),and built on the experience of several operations that preceded this one, including the first phase of this project (see Section 7). The project was successful in increasing equity, albeit less so in remote areas (where structural factors – a weak student base and few accredited HEIs – rather than project design would constrain success). The project had a well-structured theory of change that mapped the logical chain of how inputs would achieve changes in outcomes through activities and outputs. M&E design allowed monitoring of project progress and outcomes; risk mitigation measures were appropriate, as were procurement and financial arrangements. Consequently, the project was successfully implemented (timely effectiveness, successful implementation, and early disbursement. Key indicators displayed moderate issues, discussed in Section 10a. Quality-at-Entry Rating Satisfactory b. Quality of supervision Project implementation was monitored through regular supervision missions twice a year for a total of eight supervision missions. Effectiveness was timely, and the project was implemented and funds disbursed ahead of schedule. Implementation also included a smooth task team leader transition. Progress reports were produced on time and provided feedback to the counterparts. Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The results framework was coherent, with key indicators linked to outcomes. However, indicators provided national-level enrollment and graduation figures, while ICETEX’s influence extended to only a tenth of the population. This issue was addressed through impact evaluations that confirmed the results of other indicators. b. M&E Implementation ICETEX provided regular monitoring of project progress, implementation support missions did data reporting, and the Education Ministry provided bi-annual reports. Beneficiary feedback was provided through ICETEX field offices. c. M&E Utilization The information flows described under Section 9b were used by ICETEX as learning experiences – in particular, they generated remedial activities to improve results among vulnerable minorities and students from remote areas. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Social safeguards. The implementation of the Operational Policy (OP 4.10) of indigenous peoples for the ACCES Project was triggered with a focus on inclusiveness across gender, culture, and on an intergenerational dimension. The supervision carried out for social safeguards aspects of the project showed a progressive effort to improve user-assistance mechanisms, dissemination of information, and targeting of potential beneficiary populations, especially in remote areas of the country where indicators remain below target. The project supported improvements to enrollment and permanence strategies for vulnerable populations and pertinent evaluation instruments such as the development of Affirmative Action Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Plans with HEIs, focus groups with beneficiaries, and verification of the grievances and customer service system. Environmental safeguards. The project is classified as category C. Therefore, the OP/BP 4.01 on Environmental Assessment is not triggered. b. Fiduciary Compliance Financial management. The project had adequate FM arrangements, which provided reasonable assurance that loan proceeds were used for the intended purposes and FM requirements were complied with throughout the implementation period. ICETEX, as the implementing entity, was in charge of all FM aspects. There was an adequate control framework for the project, which included internal reviews performed by the Office of Internal Control; also, the entity was audited by the Office of the Comptroller General of Colombia. Interim Financial Reports, on a biannual basis, were reported to the Bank in a satisfactory manner. The auditors issued unmodified opinions, and these were acceptable to the Bank. FM performance was consistently Satisfactory and financial management requirements were complied with throughout the implementation period. Procurement. The project had no Bank-financed procurement, as Bank financing of Component 1 (Student Loans) was not subject to the World Bank’s procurement guidelines, and all contracts for procurement of goods and services were completely financed by ICETEX. c. Unintended impacts (Positive or Negative) None noted. d. Other None noted. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) 12. Lessons The following lessons are drawn from the ICR. Demand-driven higher-education financing mechanisms such as the ACCES student loan program in Colombia can lead to positive development outcomes. Emphasizing information outreach (presence on the ground) can reduce the knowledge gap and encourage participation by economically and geographically disadvantaged students. Enhancing financial literacy among loan recipients can help students manage their financial obligations under student-loan programs such as ACCES. The majority of individuals receiving student loans lack financial literacy and may face critical debt burdens over time, indicating that remedial courses in personal finance can be crucial for the success of a program such as ACCES. The very poorest and students from remote areas may need more specific or more intensive attention. While ICETEX has meaningfully helped improve education opportunities for vulnerable students, the very poorest ones or those from remote areas have made relatively few gains. These students need more robust and strategic interventions, such as remedial academic programs and coaching, incentives towards graduation, targeted financial products, and support to regional institutions for accreditation. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is evidence-based, drawing on field discussions and available documentation (essentially ISRs) with reasonable sourcing. The logic of the ICR builds on the logic in the theory of change and in the results framework, and the narrative is generally clear in explaining achievements. In some instances, especially in the efficacy analysis, the analysis could have been more systematic and could have paid greater attention to questions of attribution. The document is consistent with the guidelines. a. Quality of ICR Rating Substantial Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Access with Quality to Higher Educa. (P145782) Page 13 of 13