Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) Report Number: ICRR0022897 1. Project Data Project ID Project Name P151318 Higher Education Support Project Country Practice Area(Lead) Mali Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-56060,IDA-D0430 31-Dec-2020 31,429,045.60 Bank Approval Date Closing Date (Actual) 07-May-2015 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 33,000,000.00 0.00 Revised Commitment 33,000,000.00 0.00 Actual 31,431,332.40 0.00 Prepared by Reviewed by ICR Review Coordinator Group Denise A. Vaillancourt Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As stated in the May 30, 2015 Financing Agreement between the Government and IDA (Schedule 1, p. 5), the project’s development objective (PDO) was “…to improve the relevance of selected higher education programs and the stewardship of the higher education system in the Recipient’s territory.” The Project Appraisal Document (PAD) of April16, 2015 presented the same statement (PAD, pp. viii, 8, and 27). The PDO did not change throughout the project’s life, nor did key outcome targets or indicators. Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Original Components: Project support was delivered through three components. Component 1: Support to Selected Public Higher Education Institutions (HEIs) (original estimate: $22.00 million; actual cost: $19.58 million) was to improve the relevance of selected higher education programs, while contributing to improving stewardship by supporting governance improvements and quality- related interventions in selected existing HEIs demonstrating a strong commitment to reform and offering programs in agriculture, livestock, mining and technology. Subcomponent 1.1: Support to selected existing public HEIs was to focus on four institutions (with particular areas of specialty): (i) the Rural Polytechnic Institute of Katibougou (IPRK) (agriculture, research, and outreach); (ii) the University of Segou (UOS) (livestock, agriculture/water management, irrigation); (iii) the University of Science and Technology of Bamako (applied sciences); and (iv) the National School of Engineering (ENI) (engineering programs, including mining). The project was to support the implementation of five-year (2015-2020) Institutional Development Plans (IDPs) and Annual Work Plans (AWPs) for each of these HEIs, approved by their respective Boards of Trustees prior to effectiveness. Each AWP, specifying annual objectives and indicators, was to serve as the basis for contracts to be cosigned by the HEI rector, the Ministry of Higher Education and Scientific Research (MESRS), and the Ministry of Economy and Finance (MEF). The AWPs were to include all activities and funds allocated to the institutions (domestic and external), with clear roles, responsibilities and mechanisms for monitoring and evaluation (M&E) of annual performance. Annual AWP reports and audits were to complement the report of the Quality Assurance (QA) unit on HEI performance, used for monitoring of disbursement-linked indicators (DLIs). Two DLIs, with seven disbursement-linked results (DLRs) focused on governance, relevance, and quality, with relevant targets set for each HEI, were incorporated into performance-based contracts. Subcomponent 1.2: Support to the UOS livestock department: This HEI, Mali’s first public regional university, required more in-depth support, given that it was still in the process of development, while other selected HEIs were already established. This subcomponent was to support the development of a new livestock and animal health department, particularly the construction and equipment of new teaching facilities, goods, and technical assistance required to set up a modern, functional facility. Subcomponent 1.3: Support to the development of the new Bamako Africa School of Mines: Also in support of a new HEI under development, set up to address the skills gap in the country’s mining sector, this subcomponent aimed to support the development of a business model for the school, incorporating a public-private partnership (PPP) framework to involve interested mining companies. To this end, the Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) subcomponent was to support a feasibility study, covering financing options, architectural plans, curriculum and program development, and equipment specifications. Component 2: Higher Education System Support (original estimate: $8 million; actual cost: $8.67 million) was to improve the stewardship of the higher education system by strengthening the government’s capacity to articulate and implement reforms; play an oversight and regulatory role, including for the private sector; and manage information. To this end, this component was to support:  Upgrading of the Directorate General for Higher Education and Scientific Research (DGESRS), including: building of new premises, procurement of office furniture and information technology (IT) equipment, and staff training.  Support to the Planning and Prospective Unit (PPU) and the development of the sub-sector plan, including: procurement of equipment, furniture, consultants, and training of PPU and HEI staff on the production of annual statistics in coordination with the management information system (MIS) and annual tracer studies. The PPU was to assist HEIs not participating in Component 1 to develop and monitor their respective five-year IDPs, with the support of a Dutch-financed project.  Support to the Quality Assurance (QA) Unit: consultant services and workshops to promote and facilitate QA implementation in HEIs, assist selected academic units to undertake self assessments and guide implementation of the bachelor's – master's – doctorate system, and provide equipment and furniture.  Management and Information System (MIS), as an integral part of the West and Central African Research and Education Network, complementing existing interventions, including technical assistance (TA) to enhance the MIS and upgrade its software based on specifications outlined in a 2011 preliminary study. This included: (i) the development of a standard MIS for HEIs (accounting, budgeting, academic and student management, facilities management); and (ii) the implementation of a Higher Education Management Information System (SIGES) at the levels of the MESRS and HEIs.  Virtual library and upgrading libraries: complementing UNESCO’s financing of a virtual library, acquisition of eBooks and complementary equipment for HEIs as well as three videoconferencing rooms.  Development of private sector providers, including: studies and related consultancy services to support the development and expansion of private HEIs; and TA and equipment upgrading for the MESRS unit responsible for private HEIs. Component 3: Project Management and Technical Assistance (original estimate: $3.00 million; actual cost: $3.11 million) to provide fiduciary support and studies and analytical work (improvement of higher education M&E, a new financing system that would incentivize a more competitive higher education system, exchange programs for HEIs with internationally renowned institutions, and innovation to promote learning outcomes, academic excellence, and responsiveness to market demands). TA was also envisaged to support selectively services of HEIs not included under Component 1 to compete for partnerships with African Centers of Excellence in West and Central Africa. Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) There were no revisions to components during the life of the project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost. The total cost of the project was $31.36 million equivalent, or 95 percent of the original estimate of $33.00 million. Financing and Borrower Contribution. The project was fully financed by IDA, through a credit (IDA- 56060) in the amount of SDR 10 million ($14 million equivalent) and a grant (IDA-D0430) in the amount of SDR 13.5 million ($19 million equivalent). No Borrower contribution was planned or provided. A total of 9.22 million SDRs was disbursed against the SDR 10 million IDA credit (92 percent disbursed), while 100 percent of the SDR 13.5 million SDR IDA grant was disbursed. Of the total IDA allocation of SDR 23.5 million (credit and grant combined), a total of SDR 22.72 million was disbursed, translating into a 97 percent disbursement rate. The Investment Project Financing instrument employed a "mixed modality," with approximately $3.2 million of the project's total financial resources allocated to DLIs and the balance of resources allocated to items- based disbursements. The "mixed modality" approach sought to demonstrate incentive-based improvements at the HEI level. Key Dates. The project was approved on May 7, 2015, became effective on September 23, 2015, and closed on June 30, 2021. The project was restructured once to extend the original closing date of December 31, 2020 by six months to allow for the completion of a small number of activities, already programmed but not completed, due to delays caused by the COVID-19 pandemic and the August 18, 2020 coup d’etat. 3. Relevance of Objectives Rationale The PDO is highly relevant to current country conditions. Mali’s National Higher Education Policy (PNES) 2021-2030 and its Strategic Plan for Higher Education (2021-2026) are clear in articulating chronic issues and challenges facing the sector. Notwithstanding some improvements in sector performance, the PNES points to remaining challenges, including shortcomings in the quality of higher education training and research and the extent to which they are responsive to labor market needs, especially in priority sectors. The national policy and strategy documents also highlight shortcomings in higher education sector governance, which undermine sector quality, effectiveness, and efficiency. The project's explicit focus on priority sectors (agriculture, livestock, mining, engineering, and technology) was highly appropriate given the very large share (80 percent) of all higher education students in Mali enrolled in the humanities and social sciences disciplines (ICR p. 6). While quality of tertiary education was not explicit in the PDO statement, it was well integrated into the stewardship objective (and corresponding results framework and DLIs/DLRs), including the establishment and strengthening of quality assurance institutions, capacities, Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) processes, and partnerships with the private sector (the future employers of program graduates) at the national level and in each of the target HEIs, as well as their monitoring and ongoing improvement. The PDO is highly relevant to the current development priorities of the country. Building on the foundation laid by this project, Mali’s new 10-year policy (PNES) seeks to: (i) improve the quality of programs to meet labor market needs; (ii) improve the quality of the faculty and teaching staff; (iii) address the shortage of teaching staff by increasing their numbers; (iv) increase student enrollment; (v) further integrate higher education and scientific research; and (vi) improve governance within the system. Implementation of the first five years of this policy, as articulated in the Strategic Plan for Higher Education, converges around five axes aiming to: (i) strengthen the quality, effectiveness, and efficiency of governance; (ii) optimize equitable and inclusive access to higher education; (iii) improve the quality of training and learning environments; (iv) increase graduates’ employability; and (v) deploy the Information Technology Services Policy through strengthened links with HEIs. The PDO is highly relevant to the Bank’s most recent strategy for Mali. The World Bank’s Country Partnership Framework (CPF) for Mali was initially conceived to cover the period FY2016-2019 and was subsequently extended to include FY2020. In a February 24, 2022 meeting, the Bank’s Mali team (co-task team leader and other team members) informed IEG that a short-term Country Engagement Note (CEN) is currently under preparation but not yet available. The CEN's short-term horizon is driven by currently high political uncertainty in Mali. The team also noted that this CEN is expected to continue the strong focus on human capital development to achieve economic reform and resilience. In the meantime, the 2016-2020 CPF supports three areas of focus, all relevant to the project. The first area of focus (Improve Governance) aims to improve the allocation and efficient use of public expenditures, enhance public sector accountability, and improve service delivery. The second area of focus (Improve Infrastructure and Connectivity for all Malians) includes an emphasis on improving skills required for the development of agriculture, livestock, and other priority sectors and subsectors, as well as the integration of young people, especially girls, in the labor market. The third area of focus (Build Resilience) includes an objective to build human capacity, with an emphasis on strengthening the quality and relevance of education services, both basic and higher education. The project was included in the CPF as direct support to the objective of human capacity development. The project was also consistent with the World Bank’s twin goals of eliminating extreme poverty and promoting shared prosperity. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Improve the relevance of selected higher education programs Rationale Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) The PAD did not include an explicit theory of change. The ICR (p. 5) provides a good synopsis of the program's logic in the form of a results chain, reflective of the project design as described in the PAD (p. 10). In support of Objective 1 (improve the relevance of selected higher education programs), the project financed a number of interventions to help four public HEIs develop new programs, improve existing programs, license these programs, and attract new students in three priority sectors (agriculture, livestock, and mining). The relevance and quality of these programs were to be enhanced through the establishment and implementation of partnerships with employers, including the private sector, through which priorities would be set, existing programs revamped, and new ones developed so that graduates’ skills would be better aligned with priority sectors’ needs. Supplemental support (civil works, equipment, TA) provided to two new universities, still in the process of developing programs, to fill skills gaps in critical sectors included: (i) the development of a new livestock and animal health department at UOS; and (ii) the development of a business model for the Bamako Africa School of Mines, incorporating a PPP framework. This support was expected to culminate in intermediate results: an increased number of national students enrolled in licensed programs at participating HEIs; an increased share of the HEIs’ budgets allocated to pedagogical inputs; an increase in the amount of cash generated internally at participating HEIs; and ultimately enhanced relevance, as measured by employer satisfaction with graduates of priority programs, number of programs developed with private sector participation, and tracer studies documenting experiences and successes of HEI graduates in priority sectors. It was expected that achievement of these outcomes (and intermediate outcomes) would be facilitated by a set of DLIs and DLRs embedded in performance contracts and implemented within the four participating HEIs: DLI #2 (improved relevance of selected programs) and four supporting DLRs:  DLR #4: number of training programs licensed in line with agreed guidelines  DLR #5: number of new students enrolled in licensed programs developed with the private sector and approved by the Board of Trustees  DLR #6: number of teachers trained in the licensed courses approved by the Board of Trustees  DLR #7: percentage increase in financial resources generated internally Outputs and intermediate results  A total of 36,392 students were enrolled in licensed programs at participating HEIs, up from a baseline of 10,000 and exceeding the target of 18,900. o Of these students enrolled in licensed programs, 12,744 were female, up from a baseline of 2,500 and exceeding the target of 5,000. Outreach, affirmative action, scholarships, and financial incentives supported the achievement of this intermediate result (DLR #5).  A total of 1,403 teachers were trained in the licensed programs, up from a baseline of 120 and exceeding the target of 780. This includes teachers receiving in-service or pre-service training, which enabled them to teach the revised program content in a satisfactory manner. The tracer study has documented administrators’ satisfaction with the quality of these teachers, post-training (DLR #6).  By the project’s completion, 15 percent of budgets in participating HEIs were allocated to pedagogical inputs, up from a baseline of 10 percent, but falling short of the target of 30 percent. This indicator refers to non-capital allocations, which improve teaching-learning processes with the exclusion of Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) salaries, additional hours, and scholarships. Although the target was missed partly due to the indicator's definition, and even though the focus was on the project-supported HEIs, the project did achieve a substantial system-wide improvement in the quality of teaching and learning through the provision of pedagogical resources nationwide. The ICR notes that, in retrospect, the target may have been ambitious.  The amount of cash generated internally at participating HEIs increased by 216 percent over the baseline year (2015), exceeding the target of a 160 percent increase. This increase is based on the financial resources generated by the university from a combination of fees, donations, and services rendered using the specialized equipment provided through the project (DLR #7).  Six public HEIs prepared proposals for acquiring support from the Africa Centers of Excellence (ACE) in West and Central Africa Project and were selected by the project to support development of partnerships with ACE institutions, achieving the target of six. These proposals were competitively selected by a committee of national and international experts to support public HEIs other than the four project-supported HEIs in implementing innovative proposals for human resource development and scientific research with ACE institutions. This was the first time Malian HEIs had the opportunity to participate in a regional higher education competition. The six participating public HEIs had a total of 15 research proposals that support research and training for 13 Master’s degree students, 31 Ph.D. students, and 15 post doctorates. The programs have been important for maintaining professional connections and research collaboration with other regional higher education institutions.  The project supported the creation of the Department of Livestock and Animal Health at the UOS and the development of a new African School of Mines to increase the quality of graduates with skills aligned to these priority sectors. Supported included construction, equipment, curriculum development, new training programs, and training of faculty to deliver these programs. Outcomes  From a baseline of 45, a total of 197 programs in participating HEIs were proposed and developed with private sector participation by the project’s end, exceeding the target of 135. These programs were licensed and approved by the QA unit within the MESRS. Prior to their approval by MESRS, these programs underwent a process internal to the HEIs including recommendation by the universities and approval by a council in charge of reviewing programs on offer, made up of faculty and private sector representatives, and/or a committee in charge of scientific and pedagogical perspectives. The establishment and application of this QA capacity and process for all new and improved courses, both within HEIs and at the national level, is indicative of improved quality of training and program graduates. The factoring into this process of private sector participation is indicative of improved relevance, since the private sector represents the future employers of program graduates, and their role was to ensure the matching of training to their labor market needs (DLR #4). This being said, the 2019 tracer study (noted below) does point to the opportunity to further strengthen HEI-private sector partnerships.  A number of project findings and outcomes were documented in three tracer studies conducted during project implementation: two in 2017, and one in 2019. The 2019 (closest to project-end) study, conducted by the National Observatory for Employment and Training, provided data and information on 12,508 students from seven (project and non-project) HEIs who were enrolled in Bachelor's degree programs during 2015-2018, with follow-up interviews of a random sample of these students. As such, it is not well suited to assess the project's impact on students' quality and employability, since this Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) particular cohort of students was enrolled during the first half of the six-year project, when the project's activities to improve relevance and stewardship were being launched and implemented. The ICR's economic analysis (Annex 4, p. 56) also points out that "...the study was limited to the first cohort of beneficiary students that completed their Bachelor's at the end of 2017-18, and it could take longer for the impact of quality improvements to take full effect and translate into higher employability rates." This being said, survey results were factored into the preparation of the new 10-year higher education policy (2021-2030) and Strategic Plan for Higher Education (2021-2026) and point to both accomplishments and remaining challenges, including the following: o Employers’ satisfaction with graduates in priority sectors (agriculture, livestock, mining) achieved a level of 87 percent, exceeding the target of 40 percent (employer satisfaction surveys). Neither the PAD nor the ICR indicates any baseline value for this indicator, but the ICR (paras. 4-5) does document a considerable gap at this project’s outset between the needs of employers in priority sectors and the number and quality of HEI graduates. It would be reasonable to assume that the project’s target of 40 percent is likely indicative of the low level of employers’ satisfaction with the workforce they hired from HEIs at the time of project design. It is also plausible to assume that employers’ increased involvement in designing and improving new and improved programs was also a factor, and that (as is noted in the ICR’s economic analysis (Annex 4)), employers’ satisfaction is likely to increase further as newer cohorts of graduates, fully benefiting from project interventions, enter the workforce post- project. o Administrators in the project HEIs were satisfied with the quality of the professionally licensed teaching staff. o University staff interviewed would like to see a greater focus (up to 60 percent of hours) devoted to practical skills. o The large majority of graduates benefiting from internships at the time of training were enrolled in project-supported universities, and internship opportunities were largely based on personal relationships. o The insertion rates of project-supported and non-project-supported HEI graduates was roughly the same, at 6 out of 10 graduates, However, the share of project graduates working in their primary sector is higher among project graduates than non-project graduates, as are earnings The ICR does not quantify these differences. o Six out of 10 jobs held by project graduates are formal jobs. o Employability of both project and non-project graduates was problematic, with 30 percent of graduates persistently unemployed, 28 percent in paid employment, four percent on a paid internship, 15 percent self-employed, and 23 percent continuing training. Again, this study documents outcomes for students graduating in the early years of the project (2015-2018), prior to the achievement of project outcomes. Impacts are expected to be more robust for graduates of project HEIs, who graduated at the project’s end, or post-project. o A total of 37,795 students benefited from the project in the four selected HEIs and other HEIs benefiting from the project, slightly exceeding the target of 35,000. The ICR (p. 14 and Annex 1) documents a target of 18,900, but this ICRR uses the target of 35,000 stated in the PAD. There were no formal revisions of targets. Females made up 35 percent of these student beneficiaries, achieving the target of 35 percent. However, the tracer study found that male graduates had advantages over their female counterparts in terms of access to sustainable employment, average time to find employment, and wages. This finding applies to both project and non-project cohorts, and the ICR did not provide specific data on these differentials from the tracer studies. Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) Rating Substantial OBJECTIVE 2 Objective Improve the stewardship of the higher education system Rationale Investments in the licensing of programs, in the generation and use of various sources of information (tracer studies, employer satisfaction surveys, a higher education information system including a virtual library and online courses, and studies), in forging partnerships with the private sector, and in quality assurance were, together, expected to culminate in enhanced sector management capacity. Specifically, these investments were expected to result in: an increased number of public HEIs equipped with a management information system (MIS) with operational resource management capacity, the establishment of a semi-autonomous QA Unit; an increased number of users accessing the digital library system; and the annual publication of statistical reports with relevant data from HEIs, both public and private. These, in turn, were expected to lead to improved stewardship of the higher education system in Mali, as measured by: the number of public HEIs with a functional Board of Trustees (or University Council) with substantial representation of the private sector and other priority sector employers, and the number of institutional development plans approved at public HEIs. It was expected that achievement of these outcomes (and intermediate outcomes) would be facilitated by a set of three DLRs implemented within the four participating HEIs, which provided focus and incentives for achieving: DLI #1 (improved institutional governance) and three supporting DLRs:  DLR #1 Functional Board of Trustees (or University Council) of HEIs in place at four HEIs  DLR #2: Adoption and implementation of management tools and procedures of HEIs  DLR #3: Approval of HEIs IDPs and yearly operational budget with indicators Outputs and intermediate results  Each year the annual report on achievements against DLIs was produced on time (before end-March), achieving the target. This report, supervised by MESRS (DGESRS) and the QA unit, assessed progress made by each participating HEI on each DLI and DLR. The achievement of all the DLIs and the full disbursement of funds against these DLIs are considerable achievements for the MESRS, especially since this was the first higher education project to implement results-based financing.  A total of nine public HEIs have an MIS with operational resource management, up from a baseline of 0 and achieving the target of 9. o Thanks to the project’s support, SIGES is now able to provide at the level of each of the nine universities accurate statistics, online registration, human resource management, and financial information, all critical inputs for the higher education statistical report. Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) o The project supported the Mali Research and Educational Network, which allowed for the interconnectivity of Malian HEIs with other African HEIs. o The project helped establish videoconferencing centers in seven sites, facilitating virtual meetings with other institutions related to teaching and research. This strengthened connectivity facilitated the launch of online courses at basic, secondary, and tertiary levels when schools closed due to the COVID-19 pandemic. o With project support, an online learning platform (Moodle) was established, and teachers and researchers were trained on the preparation and delivery of courses. Overall, 625 videos were prepared, 126 higher education teachers were trained in online course preparation and delivery, and 123 online courses were developed by teachers and researchers. The Moodle platform is envisaged by MESRS to be used for the creation of a future virtual university in Mali.  MIS statistical reports are published annually, providing relevant data on HEIs, both public and private, achieving the target. Support for the SIGES system has allowed for the collection of standardized data that can be used to compare data and publish the statistical yearbook.  A total of 84,341 users (students and faculty members) have accessed the digital library system, up from a baseline of 0 and far exceeding the target of 10,000. The virtual library is fully established, and access was given to all students and teachers of HEIs registered in the MIS. With project support, a contract with CAIRN.INFO (a French language web portal containing scholarly materials in the humanities and social sciences) provided more than 10,000 online resources to all professors and students in public HEIs identified through SIGES.  Quality assurance was substantially strengthened under the project. o A quality assurance agency (the Malian Agency for Quality Assurance of Higher Education and Scientific Research, AMAQ-SUP), responsible for program regulation, accreditation, and evaluation, was established in June 2018, accorded semi-autonomous status, and became operational, achieving the target. o Internal quality assurance units (CIAQs) were also developed within the HEIs. To date, the four selected public HEIs, the vast majority of other public HEIs, and most private HEIs have a CIAQ. After establishing their CIAQs, HEIs conducted self assessments to improve their programs based on education standards. QA support included: study tours, development of evaluation standards, development of administrative and financial operating procedures for the AMAQ-SUP, and awareness raising on the importance of quality assurance.  Capacity of the National Directorate of Higher Education and Scientific Research has been enhanced. It is now able to carry out studies, collect and analyze data provided by SIGES, support planning with the HEIs, provide statistical reports, monitor performance-based conditions of the HEIs, and provide support and training related to QA. Moreover, MESRS has given this National Directorate overall responsibility for developing a common vision for HEIs as well as the broader sub-sector program to improve the quality and relevance of higher education. Its evolution from a National Directorate to a General Directorate provides it with an increased stature and leadership role to facilitate assumption of its expanded responsibilities.  The project was successful in introducing DLIs, achieving its goal of developing the four project HEIs into new models of institutional and pedagogical governance. The four HEIs have embedded the project-supported governance and operational models into their institutions and now serve as governance models for MESRS and other HEIs in Mali. In addition, the MEF, based on project experience, has entered into performance-based contracts with other HEIs, for which the DGESRS is responsible. While not part of the original project design, the government has drafted a new regulatory Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) framework conferring autonomy to HEIs based on the project model universities, which was reported to be undergoing a review and consultation process at the time of ICR preparation (ICR, p. 18). Outcomes  Nine public HEI IDPs (five-year strategic plans specifying priorities, planning, budget, indicators, and implementation formation) were approved by the HEIs’ Boards of Trustees and found satisfactory to the Bank, up from a baseline of 0 and achieving the target of nine. Together, these nine institutions cover the full range of disciplines offered at the bachelor's, master's and doctorate levels, and account for 97 percent of students enrolled in public HEIs.  Nine public HEIs had a functional Board of Trustees, up from a baseline of 0 and achieving the target of nine. Functional boards are defined as having at least 30 percent of non-civil servants/private sector representatives from sectors relevant to programs developed at the HEI, and meeting at least twice annually to approve the HEI budget and relevant documents, thus ensuring HEI autonomy. These Boards deliberate on key elements of HEI management, such as strategic plans, budget, appointment of key staff, QA, and institutional rules, regulations, and procedures, in conjunction with the AMAQ-SUP. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale All of the PDO indicators were met or exceeded and the DLIs were all met. The project substantially improved the quality and relevance of the higher education system in Mali. Employers’ satisfaction with project-supported graduates they hired is high. Over 150 new programs were developed in partnership with the private sector and licensed. The quality of professionally licensed teaching staff has risen. More project graduates are working in their primary sector (compared to non-project graduates); and the number of project beneficiaries slightly exceeded targets. The project also made substantial contributions to improvements in sector stewardship. Performance-based contracting has been incorporated into the new governance structures of project-supported HEIs and this model will be extended to other HEIs. Investments in the MIS, the higher education information system (SIGES), and IT/digital capacity have strengthened the generation and use of information for sector tracking and management and the expansion of access to digital information, resources, and learning. Well- functioning quality assurance structures have been established both within MESRS and within HEIs, substantially boosting the capacity for quality assurance. The establishment of five-year institutional development plans and budgets and functional Boards of Trustees in each of nine HEIs has substantially strengthened HEI’s strategic management and oversight capacity. Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) Attribution The Mali team shared their perspectives on the project’s attribution/contribution to outcomes during a 2-24- 2022 meeting. While many development partners support basic education, very few support the higher education sector. The outcomes reported by the ICR can be largely attributed to the project, although two projects also contributed. The Netherlands Development Cooperation has contributed to project outcomes through the Netherlands Post-Secondary Education Capacity Building Project (NICHE), which focused on the professionalization of higher education programs. The Bank and the Netherlands worked closely together, including occasional joint supervisions, to ensure continued coordination and full complementarity of efforts. In addition, the Canadian International Development Agency, through the implementation of the Project for Agricultural Training for Food Security in Mali (Formation agricole pour la sécurité alimentaire au Mali, FASAM) has also contributed to outcomes. Katibougou and Segou HEIs (both supported under the Bank’s project) benefitted from lessons learned and collaboration with this project for the design and implementation of their study programs on agriculture and farm breeding. The institutional strengthening and sector reform outcomes are strongly attributed to the Bank’s project and dialogue. Project objectives (aimed at improving sector governance and stewardship) and its approach (performance-based contracting, increasing responsibilities and autonomy of HE institutions) were critical in providing proof of concept and in supporting reforms in project-supported HEIs, which are widely acknowledged and appreciated by national actors and stakeholders and now being extended to the entire sector. Counterfactual. During the 2-24-2022 meeting with IEG, the Mali team also shared their perspectives on the counterfactual of no project.  The DGESRS used to be a National Directorate. Now it is a General Directorate, with more responsibility, an increased steering role for the higher education sector. It undertakes data collection, analysis, and reporting, with a view to informing evidence-based decision-making. Project activities to strengthen its capacity have equipped it to play its role effectively. Without project support sector stewardship would not be as strong.  The Management Information System established with project support has facilitated and rendered much more efficient the management of information and the easier access of students to information and scholarships. Among its key features is the institutionalization of a unique identifier for the student population (some 120,000 students), facilitating their online registration, and linking them to scholarship opportunities. Sector efficiency would have been much weaker in the absence of the project.  Quality assurance and regulation of the higher education sector would have been limited, less independent and weaker, had the project not supported the establishment of a semi-autonomous quality assurance function at the national level (AMAQ-SUP) and had there been no establishment of quality unites in project HEIs.  In the absence of performance-based contracting, introduced under the project, financing, management and oversight of higher education programs and institutions would have been less focused on results and reform and more on inputs.  In the absence of the project’s significant contribution to the strengthened digitalization capacity of the education sector, Mali would not have been equipped and able to take the quick action it did to extend Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) access to education (including basic, secondary and higher education) during the time of school closures due to the COVID-19 pandemic. (See also Section 10.c on unintended impacts.) Overall Efficacy Rating Substantial 5. Efficiency An end-of-project cost benefit analysis (CBA) showed that the project is likely to have generated substantial benefits and reasonably high rates of return. Both the internal rate of return (IRR) and net present value (NPV) estimates show that the project was economically viable at the end of the project implementation period based on actual project beneficiaries. These estimates were calculated for sub-components 1.1 (support to existing public HEIs) and 1.2 (support to the UOS livestock department) because the benefits for these interventions were quantifiable. Calculations culminated in an estimated benefits/cost ratio of 1.84, a NPV of benefits of US$228.9 million, and a strong IRR of 12.5 percent. To obtain a lower-bound of the benefits/cost ratio, the economic analysis factored in the costs of the remaining components, which are not easily quantifiable, and assumed no associated benefits. Even with such an assumption the benefit/cost ratio was found to be 1.77. The results of the economic analysis strongly support the assertion that the project offered good value for money for the beneficiaries (ICR, Table 4.2, p. 58). Sensitivity analyses, which varied the project’s impact on employment outcomes for graduates, reaffirmed the project’s efficiency. The economic analysis, described above, was based on the assumption that the employment rate of graduates would correspond to the employer satisfaction rate, as measured through the 2019 tracer study. Two sensitivity analyses were undertaken, varying the impact of the project on employment outcomes for graduates. The first (low case scenario) assumed no difference in the long-term employment rate of student beneficiaries compared to the overall employment rate for graduates of tertiary education in Mali, and still culminated in a benefit/cost ratio of at least 1.1. The second (high case scenario) assumed a 10-percentage- point gain above Mali’s overall employment rate for tertiary graduates, culminating in a benefit/cost ratio of at least 2.1. The non-quantifiable benefits are also likely to have been significant. The costs of component 2, which aimed to build institutional capacity and strengthen sector stewardship, represented a much smaller share of project financing than those of component 1. The project developed a proof-of-concept framework to improve the governance of the higher education sector in Mali. If there were accompanying reforms to extend this model sector-wide, all higher-level institutions would benefit from a mechanism to improve program quality; justify, mobilize, and prioritize investments in infrastructure and equipment; and achieve greater efficacy, efficiency, and accountability through a results-based approach fostering improved management and accountabilities. Implementation was efficient overall. The project achieved its objectives, with only a six-month extension of the original implementation timeframe. This is noteworthy, considering that the extension was made to compensate for delays caused by two major events that occurred during the project’s final year: the onset of the COVID-19 epidemic and the August 2020 coup d’etat. Together, the IDA Credit and Grant were 97 percent disbursed (see also Section 2e). The project’s implementation arrangements, its interventions aimed at strengthening institutions and capacity for sector stewardship, its results-based financing approach, and the granting of Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) increased autonomy to HEIs all contributed to its smooth and timely implementation. Some delays in the implementation of QA activities/regulations at the central level, reported in 2017, negatively impacted the ability of the ENI and the IPRK to attain their DLI targets, but these issues were subsequently resolved. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the PDO is rated high, as the objectives were highly responsive to: (i) current country conditions (shortcomings in the quality and relevance of higher education training and research and in governance capacity, notwithstanding recent improvements on these fronts); and (ii) current development priorities of the country (continued improvements in the quality of programs and their relevance to market needs, enhanced employability of graduates, and strengthened sector governance). Additionally, the PDO were substantially relevant to the World Bank’s most recent CPF (FY2016-2019, extended to 2020), with the preparation of a new CEN under way. Under its “Build Resilience” focus, the CPF includes a Human Capital Development objective that specifies the need to improve the quality and relevance of education services, encompassing all levels of schooling and training. Efficacy of both objectives is rated substantial. Objective 1 (improve the relevance of selected higher education programs) was substantially achieved with a mix of targets fully achieved (tracer surveys conducted, highlighting both successes and areas for improvement) and exceeded (employers’ satisfaction with graduates and the number of licensed programs developed with the private sector). Objective 2 (improve the stewardship of the higher education system) was substantially achieved, with HEIs' structures, capacities, and autonomy strengthened. Efficiency is rated substantial, with the end-of-project CBA (1.8), IRR (12.5%), and NPV ($288.9 million) showing that the project was economically viable. Implementation was smooth overall and use of project resources efficient. Taken together, these ratings indicate that there were only minor shortcomings with the project's preparation, implementation, and achievement, producing an Outcome rating of Satisfactory. Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) a. Outcome Rating Satisfactory 7. Risk to Development Outcome Project activities have been institutionalized by MESRS and the four project-supported HEIs, which now serve as models for improved governance and program relevance. Institutional strengthening at the central and institutional levels was effective in mitigating original capacity risks and culminated in improvements in data systems and in the use of data to develop employment policies. QA capacity and processes have been developed and institutionalized, contributing to improved program quality and relevance. This capacity strengthening and institutionalization of new processes enhances the likelihood of project interventions being sustained. Nevertheless, three overarching risks to development outcomes were identified in the ICR (p. 28). First, relatively poor education outcomes, combined with relatively high unemployment and poverty, have made Mali’s large youth population susceptible to ongoing political tensions. Mali will need to encourage and nurture more productive employment throughout the economy, including for self-employed and informal workers in agriculture, livestock, and mining sectors. Second, project-financed equipment and construction will require government provision of regular maintenance and renewal to ensure continued impact of investments. Third, performance-based contracts signed between selected HEIs and MESRS to achieve DLIs and DLRs prior to disbursement of funds boosted quality and institutional autonomy and strengthened governance. Mali’s adoption of a draft regulatory framework that seeks to extend this model sector-wide, including conferring of institutional autonomy to HEIs, is critical for ensuring the sustainability and scale-up of this reform. In addition, going forward, the AMAQ-SUP team will need to increase its capacity to respond to increased demand for accreditation of HEI programs and institutions. 8. Assessment of Bank Performance a. Quality-at-Entry The project’s design was closely aligned with Mali’s National Policy and Strategy for Higher Education Development (2003-2013) and with the Bank’s Interim Strategy Note for Mali, which was focused on long-term economic growth and human capital development and in effect at the time of preparation. The design and its underlying theory of change were straightforward and focused, with a clear PDO, key indicators, and DLIs, which were appropriate for tracking progress towards achieving intermediate and final outcomes. The design reflected peer reviewers' recommendations and lessons from other relevant projects, with a particular focus on performance-based contracts. The use of results-based financing was a new, innovative approach for Mali, but at the same time appropriate to the project’s aim to improve HEI governance and the quality and relevance of HEI programs. The risk assessment identified the salient Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) risks and included appropriate design features to mitigate them. The Bank’s preparation team included the relevant technical expertise and knowledge of reform and institutional strengthening to underpin a sound, well-informed design. Preparation was appropriately consultative, including faculty, staff, and students (and their parents) of HEIs, as well as MESRS and MEF staff. The Bank’s preparation team undertook due diligence in its preparation of fiduciary aspects, conducting assessments of financial management and procurement capacities, building implementation arrangements, and ensuring adequate capacity based on strengths and constraints identified. Quality-at-Entry Rating Satisfactory b. Quality of supervision Supervision missions were conducted on a semi-annual basis, properly staffed with higher education specialists and fiduciary and safeguards staff and consultants. Especially given the innovative nature of the project, with its results-based approach and the PDO goal of improving sector stewardship, the project team was actively engaged in supporting the government in its implementation efforts. Supervision teams provided consistent focus and support to financial management, procurement, and safeguards progress, and worked with DGESRS, the Project Implementation Unit (PIU), and other implementation agencies to build their capacities on these fronts. The missions were systematic in documenting progress and challenges in aides-memoire and internal reporting. Aides-memoire provided a clear record for the government on performance, challenges, and next steps, which provided clarity of guidance and a vehicle for learning and dialogue as reforms were implemented. By nature of the project’s results-based design and strong M&E, supervision and aides-memoire maintained a strong results focus. Challenges were addressed effectively without compromising the project’s design integrity. Supervision reporting also kept World Bank management informed of progress and was an important input to the ICR process. The Bank was effective in responding to the government’s request for a six-month extension due to the COVID-19 pandemic and a coup d’etat during the project’s final year, which allowed for the completion of the project, as designed, and the institutionalization of new governance and quality assurance structures. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The theory of change, as presented in Figure 1 (ICR, p. 5) was logically sound, with clear links between inputs, outputs, intermediate outcomes, and the PDO. The PDO were clearly articulated at the outset and did not change. The six PDO-level indicators and eleven intermediate outcome indicators were clearly Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) defined and provided good measures of progress and outcomes. The design envisaged routine monitoring and reporting by the PIU and DGESRS, supported by the development of a computerized MIS that would integrate university subsystems into the national higher education system. The aim was to capture needed data for reporting on the project indicators, statistical yearbooks, and progress against HEIs’ development plans. The project’s results-based approach included DLIs and DLRs for the four project-supported HEIs, which were coherently linked to the HEIs' development plans and to the performance-based contracts. The DLIs and DLRs paid specific attention to new program introductions, teacher training, IT updates, and the monitoring and implementation of HEI development plans. As such, the M&E design was supportive of the government’s overall HEI reform agenda and the establishment of a QA process for institutions and HEI programs. Another strong feature of M&E design was its support of critical studies and evaluation, designed to provide complementary information, data, and insight on higher education financing, employment and quality of program graduates, and employers’ satisfaction with these graduates. b. M&E Implementation M&E was rated satisfactory in the Bank’s internal reporting throughout project implementation. Reporting of data on the project indicators was systematically carried out during the entire life of the project. The DGESRS, PIU, and HEIs analyzed and reported on achievements against indicators on a regular (semiannual) basis. HEIs also issued progress reports on DLI achievement, linked to the performance- based contracting. Performance contracts were monitored by the DGESRS and PIU to ensure targets were being met. The AMAQ-SUP (fully institutionalized with project support) undertook M&E of a higher education QA mechanism, and continues to fulfill this function post-project. QA units were established within each project-supported institution and institutional self-assessments conducted. The project helped establish a robust MIS for the higher education sector. The DGESRS also conducted studies, with the support of technical assistance, including: a report on the challenges and constraints of private higher education in Mali; a feasibility study for the African School of Mines – Mali; an environmental and social impact study for the New School of Mines; an a study on the curriculum and programs for the New School of Mines. The project also conducted tracer studies and an employer satisfaction survey that were instrumental in assessing project achievements and impact, as well as a study on financing of higher education and feasibility of a student loan system. c. M&E Utilization MESRS centralized all data on a single platform and regularly monitored the workforce by university, faculty, and sector. The data has also informed the publication of an annual statistical yearbook. IT systems and capacities established within project-supported HEIs facilitated the systematic collection and reporting of performance contracting data, helping MESRS and DGESRS in their overall monitoring responsibilities and HEIs in tracking their own progress against performance indicators. The establishment of QA units within each project-support HEI and their conduct of self evaluations are expected to be institutionalized, with the establishment of a strong and functioning AMAQ-SUP. MEF has expanded the use of performance-based contracts, whose results will (together with HEI budgets) provide the basis for allocating university budgets. The higher education MIS also institutionalized a unique identifier linked to a digitized student card, access portal, digital working place, online learning, online registration, and integrated management of payments. Studies undertaken with project Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) support and DGESRS’s strengthened capacity to conduct studies and analyze data enhanced the preparation of the new Strategic Plan for Higher Education (2021-2026). M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was assigned Category B status, since no activities were expected to have a negative impact on environmental resources or quality of life. OP/BP 4.01 “Environmental Assessment” was triggered due to potential production of solid waste with the refurbishment and construction of facilities (DGESRS and UOS). OP/BP 4.12 “Involuntary Resettlement" was also triggered. An Environmental Social Management Framework and a Resettlement Policy Framework from a previous education project (P123503) were updated and disclosed in-country (March 30, 2015) and at the World Bank InfoShop (April 2, 2015). Safeguards were consistently monitored, and internal ratings remained in the satisfactory (including moderately satisfactory) range. In 2019, the PIU completed the Environmental and Social Impact Assessment, which assessed limited impact and scale of key construction activities. Moreover, construction sites’ Environmental and Social Management Plans were prepared with a view to mitigate the potential risks. In 2019, MESRS established a project-specific grievance redress mechanism (GRM), replacing the project’s previous reliance on the GRM in MESRS, which was not project-specific. During the project period, the GRM received and satisfactorily processed two complaints related to construction in Segou. b. Fiduciary Compliance Financial Management (FM). FM performance received satisfactory ratings except for one moderately satisfactory rating in April 2016, due to issues associated with FM staff changes that were quickly resolved. A strong FM team and sound FM arrangements underpinned proper financial management, including: regular steering committee meetings for oversight; timely and satisfactory submission of interim financial reports; submission of required audits to the Bank, all unqualified; and effective implementation of recommendations emanating from external audit and Bank supervisions. The audit report for CY 2020 issued an unqualified opinion and noted the effective implementation of all FM recommendations. This evidence indicates compliance with the Bank’s exigencies. Procurement. The consistent rating of moderately satisfactory in the Bank’s internal reporting system reflected sustained delays in processing, approval, and signing of procurement documents. Bank procurement staff provided training and technical support to the PIU to ensure compliance with procurement guidelines and improve procurement efficiency. The Bank team also conducted annual procurement reviews, which recommended improvements to contract processing. In 2020 the PIU migrated to the Bank’s Systematic Tracking of Exchanges in Procurement (STEP) system, including Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) procurement activities in the STEP platform. Procurement plans were updated and submitted to the Bank on a regular basis and always of good quality. c. Unintended impacts (Positive or Negative) The project’s key digital achievements, including the establishment of a robust MIS in the higher education sub-sector and the establishment of a distance learning platform, rapidly informed the government’s decision to introduce the use of alternative education delivery channels during the COVID-19 pandemic with a focus on the use of radio for primary and lower secondary education provision, television for upper secondary, and distance learning (internet) for higher education. The project team took the lead in facilitating learning continuity during the pandemic, producing notable results: 625 instructional videos for national education; training of 126 teachers on distance learning; and delivery of 123 online courses. The project also facilitated open access by students, teachers, and researchers to digital resources through the virtual library, which bodes well for future partnerships with universities at the global level. d. Other Gender. The project fully achieved the target for women’s enrollment in licensed programs (Section 4), but the tracer study points to the continued need to focus on closing the gender gap within both HEIs and the labor market, especially with regard to women's access to sustainable employment, average time to find employment, and wage differentials. The new Strategic Plan for Higher Education (2021-2026) envisages a review of the scholarship program with specific focus on prioritizing women’s applications. Moreover, MESRS intends to increase the presence of women in fields of technological science, medicine, pharmacy, agronomy, and animal medicine. Higher Education Governance. The project’s second objective to improve the stewardship of the higher education system in Mali was expressly focused on improving higher education governance; and it was substantially achieved. Associated outputs and outcomes are documented in Section 4. Additionally, the performance-based contracts introduced under the project have now been institutionalized and have been an important driver of change among project HEIs, strengthening their ownership and autonomy around clearly defined objectives, activities, and results. This project-demonstrated effect has caused MEF to institutionalize performance-based contracts and budgets across all HEIs in Mali, with a view to enhancing accountabilities sector-wide. Poverty Reduction and Shared Prosperity. The ICR (p. 21) noted that the provision of quality training to a growing number of students in priority economic sectors has already resulted in improved employment opportunities for graduates as evidenced by the tracer studies. These opportunities have the potential to contribute to the twin goals of poverty reduction and shared prosperity, with new, increased cohorts of graduates benefiting from the full project's effects and facing improved prospects for employment. At the same time, the ICR (p. 21) cautions that unemployment and under-employment are still a concern for HEI graduates, factors that are cited in studies as facilitating the radicalization and violent extremism of individuals, which is a high risk in the Malian context. There is scope for improving the relevance of HEI programs, continuing to connect with the private sector to this end, and encouraging more productive employment throughout the economy. It is also important to take note of the ICR's observation (p. 2) that Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) equity in access to secondary and tertiary education remains a challenge, given that the upper income quintiles of the population make up the largest share of beneficiaries of these levels of education. This points to the need for continued efforts to attract and retain more equitable intake of tertiary-level students through scholarships and other incentives and supports. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The following lessons, drawn from the ICR and slightly consolidated by IEG, are well-grounded in the ICR’s storyline and relevant to other countries undertaking higher education reform. Establishing a solid governance system and strong ownership are critical for ensuring implementation, institutionalization, and sustainability of reforms. In the case of Mali, the project helped support the establishment of the Directorate General for Higher Education and Scientific Research, responsible for overseeing and guiding the development and reform of the higher education sector and the implementation of performance-based contracts. Instrumental in achieving this outcome was the commitment of the actors who facilitated reform implementation, leading to the autonomy of the project-supported higher education institutions. Realization of the government’s intention to adopt the draft higher education regulatory framework will be important for institutionalizing autonomy for HEIs. The establishment of a strong and viable M&E system enables the full documentation and analysis of program achievements and challenges, which, in turn, enable decision-makers and implementers to learn and enhance their effectiveness and impact. The project successfully established a sound M&E system and supported numerous studies and evaluations that were undertaken during the project’s life and proved to be essential for informing policy decisions in the higher education sector. Effective accreditation mechanisms are essential for ensuring quality of higher education programs and institutions. In the case of Mali, the establishment of two complementary quality assurance entities has substantially strengthened capacity on this front. The Malian Agency for Quality Assurance of Higher Education and Scientific Research has evolved into an effective regulation and accreditation body. In addition, quality assurance units, established in higher Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) education institutions, are essential for the preparation of their self-evaluations (which are linked to performance-based contracts) and quality criteria. Performance-based contracts can be an important vehicle for strengthening dialogue, transparency, monitoring, and increased autonomy of higher education institutions. The use of performance-based contracts at the HEI level has helped HEIs negotiate annual programs and budgets based on targets and results rather than on inputs. The Ministry of Economy and Finance has recognized these contracts as best practice and recommends expanding this approach to improve the oversight and efficiency of other HEIs. Senior management of the HEIs have supported the reforms enthusiastically and restructured internal processes in keeping with the spirit of these reforms. They have also been strong proponents of the continued use and expansion of the reform system-wide. Government strategic thinking and agility on the use of information technology can lead to a more expansive use of the technology than originally designed or envisioned. Project investments in an HEI information system allowed for considerable connectivity with other teaching and research institutions. These initial investments in information technology enabled the Ministry of Higher Education and Scientific Research and universities to adapt quickly to COVID-19 pandemic- related school closures by launching online courses, not only at the HEI level, but also at the primary and secondary levels, when and where possible. This led to much broader use of the information systems and technology than originally envisioned and enhances potential for the creation of a future virtual university. 13. Assessment Recommended? No 14. Comments on Quality of ICR Quality of Evidence. Evidence provided in the ICR was drawn from project data collected and analyzed at the HEI and ministry levels. Project data facilitated the reporting of progress against relevant and well-defined indicators, and the baselines and targets established for them. They were systematically documented in the ICR’s Annex 1 (Results Framework and Key Indicators) and appropriately reflected in the Efficacy discussion. This data was complemented by data and findings from tracer studies, which assessed the quality and employability of graduates and the satisfaction of their employers, among other aspects. The ICR also explained and drew on the links between the HEI-level DLIs and DLRs and the intermediate outcome and outcome indicators. The ICR provided additional information and insights, gleaned from the tracer studies, on a number of shortcomings and challenges to achieving further improvements in the quality and relevance of training programs. These findings and insights were presented in general statements and did not systematically cite data from the studies to back up statements (e.g., comparisons of outcomes between project and non- project graduates, and comparisons of wage data between project and non-project graduates and between male and female graduates). Quality of Analysis. The ICR reflected a solid analysis of the evidence, especially the use and triangulation of project data, study results, and achievements against DLIs and DLRs. The ICR’s analysis also integrated Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Higher Education Support Project (P151318) qualitative results and findings to make a convincing case for real achievements in sector reform and stewardship. The ICR did not directly assess attribution/contribution of the project, or the counterfactual of no project. A February 24, 2022 meeting with the task team provided good input to these analyses. Quality of Lessons. Lessons presented in the ICR were based on the evidence, analysis, and findings presented in earlier sections of the document and were well linked to the ICR’s narrative and ratings. They are relevant to other countries that may be attempting to improve the quality, relevance, and stewardship of the higher education sector. Results Orientation of the ICR was strong. The efficacy discussion was oriented around the project’s two development objectives, and was systematic in providing evidence for each element of the results chain in terms of how it facilitated PDO achievement. Internal Consistency/Adherence to Guidelines. The ICR was internally consistent and adhered to the guidelines, with a couple of minor caveats.  The ICR did not provide the IDA Credit and Grant amounts (original and actually disbursed) in SDRs, which is necessary to calculate the real disbursement rates. The ICR presented these only in terms of US$.  The majority of the discussion of relevance of objectives focused on relevance at the time of project development. The assessment of current relevance did not squarely address relevance to country needs, but did address relevance to new government policy and plans. The assessment of relevance to the Bank’s current strategy referred to a CPF that seems to be out of date – FY2016-2019, extended to 2020 -- and did not indicate whether there is a new CPF under preparation, or any other (updated) indication of the Bank’s strategy in Mali in the interim. The task team clarified in its February 24, 2022 meeting with IEG that a short-term Country Engagement Note is under preparation but not yet available. a. Quality of ICR Rating Substantial Page 22 of 22