Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) Report Number: ICRR0021775 1. Project Data Project ID Project Name P111592 MZ Higher Educ Science & Techn. (FY10) Country Practice Area(Lead) Mozambique Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46840,IDA-D0510 31-Dec-2015 79,706,558.13 Bank Approval Date Closing Date (Actual) 25-Feb-2010 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 40,000,000.00 0.00 Revised Commitment 82,309,106.00 0.00 Actual 79,706,558.13 0.00 Prepared by Reviewed by ICR Review Coordinator Group Antonieta Romero- Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Follette 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the original financing agreement, the Higher Education Science and Technology (HEST) project had the following objectives: (a) increase the number and raise the quality of graduates at the undergraduate and graduate levels, and (b) strengthen the national research capacities to produce research outputs of relevance to the Recipient’s strategic economic sectors. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) A third objective was added at a 2015 Additional Financing (AF): (c) strengthen the institutional framework for Technical and Vocational Education and Training (TVET). Because of the revised objectives, a split rating will be performed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 02-Jul-2015 c. Will a split evaluation be undertaken? Yes d. Components The structure of the project included three components: Component A: System governance, quality and management strengthening (appraisal, US$14.10 million; AF, US$9.15 million; actual, US$14.11 million and AF US$17.45 million). The HEST project was to provide capacity building of system oversight and quality assurance institutions, implementation of key quality assurance policies, and piloting of the higher education financing reform. Through this component the HEST was to provide training, technical assistance and/or goods to: (i) regulatory institutions and the Instituto Nacional de Ensino à Distância (National Institute of Distance Education, NED) at the Ministry of Education and Culture (MEC); (ii) institutions in charge of developing new financing mechanisms for scholarships and training; (iii) the National Directorate of Higher Education; (iv) the authorities in charge of creating a third provincial learning resource center and implementing distance learning activities; (v) the scientific councils and the Ministry of Science and Technology (MCT); and (vi) the Executive Secretariat of the Commission for Professional Education Reform (ES-COREP), enhancing the TVET system. At the AF, the original PDO activities were retained with scaled-up targets resulting in higher numbers of students graduating from higher education institutions; of Master's degree graduates in math, sciences, engineering and agriculture; of programs of higher education institutions (HEIs) accredited by the National Accreditation and Quality Council (CNAQ); and of project beneficiaries. More activities were added (creating certification and accreditation mechanisms, technical assistance to the new National TVET Authority (Autoridade Nationale do Ensimo Professional, ANEP), and a decentralized management model for five institutions) to support the development of qualifications systems connecting medium-level TVET with tertiary education. Funds were also provided to establish a management information system for higher education. Component B: Improving quality of teaching, learning and research through competitive funds (appraisal, US$15.8 million; AF, US$13.0 million; actual, US$15.41 million and US$11.84 million). This component included five activities: (i) technical assistance, training, and goods to the MEC’s institutional development fund to develop information communication applications for education; (ii) loans to Mozambique’s private HEIs to enhance the quality of their teaching and learning processes; (iii) grants to Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) Mozambique’s public HEIs to enhance the quality of their teaching and learning processes; (iv) grants to Mozambique postgraduate students, to improve the quality, relevance, and production of applied research; and (v) advisory services to the ES-COREP in delivering TVET grants and scholarships. With the AF, the National Training Fund for Professional Education was established to improve teaching and learning in technical and vocational education. Component C: Equity and competitive scholarships (appraisal, US$10.1 million; AF, US$20.1 million; actual, US$8.08 million and US$12.81 million). Through this component, scholarships were to be provided for: (i) approximately 350 poor/disadvantaged undergraduate students, in particular to females from the provinces; (ii) existing faculty and researchers acquiring master’s and PhD degrees, with emphasis on science, engineering and technology; and (iii) teachers participating in TVET Lecturer Training Scholarships. The AF provided support for 40 additional TVET Lecturer Training Scholarships. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Total project costs were estimated at appraisal at US$40 million. With the AF and exchange rate losses, total actual costs were US$79.7 million. Financing: The project was initially to be financed by a US$40 million Bank Credit. AF was provided in the form of a US$45 million grant. Unallocated funds in the amount of $2.75 million were used for Mozambique’s participation in the Africa Higher Education Centres of Excellence (ACE) Program (P151847, 2016-2022, US$148 million). These funds were cancelled from the HEST project and are not considered in the overall HEST project funds. The HEST project’s total planned financing, including the AF, was therefore US$82.25 million. Total disbursement was US$79.7 million, with losses of about US$ 3.24 million (4% of total) due to SDR/US$ exchange rate changes. Borrower contribution: No Borrower contribution was planned or made. Dates: The project was approved on February 25, 2010 and became effective on November 11, 2010. It underwent a mid-term review on May 15, 2013 and had one restructuring and two fund reallocations:  July 2, 2015: A level 2 restructuring authorized AF to: scale up each component of the original PDOs (dropping two fulfilled activities, scaling up targets, adding a new objective and new activities) and extend the original closing date by three years from December 31, 2015 to December 31, 2018. The AF also provided US$2.75 million to finance Mozambique’s participation in the regional ACE project for the Center of Excellence of Oil and Gas in the Eduardo Mondlane University.  June 25, 2016: A funds reallocation between disbursement categories took place: SDR 1,900,000 (US$2.75 million equivalent) were cancelled from the HEST project and transferred to the ACE project.  March 19, 2019: Another funds reallocation between disbursement categories was authorized. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) The project closed on December 31, 2018. 3. Relevance of Objectives Rationale The objectives were relevant at appraisal. In 2010, Mozambique was undergoing an unprecedented swell of its higher education system; however, constraints remained. The number of quality graduates (notably from STEM areas) was still insufficient to fully support the country’s economic growth. A lack of funds for research and other operational areas hindered the engagement of research institutions in meaningful research (PAD, para 28). The limited capacity (in some technical areas and the TVET curriculum) of teachers from intermediate-level education and TVET affected the government’s ability to geographically enlarge the enrollment of more distance learners, especially of females. The government also faced the need to assure the TVET skills curriculum’s harmonization with other HEIs and employment market requirements. The project’s objectives – of enlarging the numbers of quality graduates (from HEI and the TVET), producing more robust and relevant research products for strategic sectors, and having a TVET system creating a labor force with market-relevant skills – were relevant to the 10-year Strategic Plan for Higher Education approved in 2000, and to the Education Strategy (PEE 2012-2016). The HEST PDOs were also consistent with the Poverty Reduction Strategic Plan (adopted in 2000), which sought to have a better educated and skilled work force entering the labor market each year. The project’s emphasis on boosting distance learning was also aligned with the government’s efforts to transform Mozambique into a more technology-driven and knowledge-based modern industrial country (ICR, para 22). The project was also relevant to Bank priorities. At appraisal, the PDOs reflected priorities of Pillar 2 of the World Bank 2008-2010 Country Partnership Strategy (CPS), which was to expand the skills, technologies and knowledge in key economic growth areas. In the same manner, the HEST targeted also the expansion of technologies and the number of undergraduates and graduates with skills and knowledge in science, technology and engineering. Similarly, the HEST was aligned with the 2011-2015 CPS (endorsed by the Board on February 8, 2012), in particular with Pillar 1, which aimed at supporting a better educated and skilled workforce. That goal was also a priority of the HEST. After appraisal, the PDOs remained relevant for supporting the government’s Higher Education Strategic Plan (PEES 2012-2020), and continued to be linked to the Bank’s Country Partnership Framework at closing (2017-2021), under which Area 2 focuses on investing in strengthening the human capital skills base. In particular, the HEST supported activities for the development of a national policy and regulatory framework for higher education, which included the design of a 2012-2020 strategic plan. Rating Relevance TBL Rating High Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase the number of graduates at the undergraduate and graduate levels Rationale The project’s theory of change presupposed that the increased number of undergraduate and graduate students was to be supported by the project’s Equity and Competitive Scholarships Program, via two funds, namely the Institutional Development Fund (IDF) and the national research fund (FNI). Outputs: increased number of scholarships granted to undergraduates and graduate students  2,110 needy students received scholarships to pursue a first degree at a Mozambique HEI, surpassing the original target of 1,900. About 34.6 percent of the scholarships were given to women. The project developed and implemented targeting mechanisms to assess scholarship eligibility of students based on their social and economic circumstances.  345 scholarships were awarded to students in the 3rd and 4th years of STEM courses at public and private HEIs, exceeding the original target of 340.  146 master’s graduates from post-graduate courses were supported by the Institutional Development Fund (IDF), of which 133 were in sciences and agriculture and 22 in education; and 176 master's students, 45 PhD students, and 1 postdoctoral student graduated under the scholarships program. No targets were provided.  A loan scheme option was implemented as an alternative scholarship financing modality.  Standardization and verification mechanisms for the selection of scholarship recipients were enhanced.  21 internships of medical doctors outside the country in areas that were lacking expertise in the country were supported. Outcomes  19,197 students (8,995 females and 10,202 males) graduated from HEIs, surpassing the revised target of 14,700 graduates (4,998 females and 9,702 males). The original target was 10,000 graduates (4,000 females and 6,000 males).  605 master’s degree students graduated from areas relevant to economic growth, i.e., math, science, engineering, and agriculture programs. This exceeded the revised target of 340 graduates (original target of 40 graduates). Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592)  The number of male master's degree graduates increased from 17 in 2010 to 407 in 2018, and female master’s degree graduates increased from 17 to 198, both exceeding the revised targets of 221 and 119 respectively (original targets: total 40 graduates, of which female 20 and male 20). The evidence shows that the number of graduates increased from both the undergraduate and graduate levels at HEIs (especially master’s degree students graduating in math, science, engineering, and agriculture) and from the TVET programs. Rating Substantial OBJECTIVE 2 Objective Increase the quality of graduates at the undergraduate and graduate levels Rationale The theory of change assumed that the rise in quality of HEI graduates was to result from: (i) operationalization of the accreditation body, the CNAQ, and (ii) the design of the National Qualification Framework for Higher Education assuring the basic regulatory structure for HEI institutions (ICR, para 25). Outputs  The CNAQ was established and became operational in 2017 (a year before conclusion of the project).  The CNAQ accredited 97 courses/programs, surpassing the revised target of 45 programs academic courses/programs (original target 20). CNAQ also provided the enabling environment for the adoption of quality assurance in major institutions.  9,432 HEI graduates (49% of the 19,197) were from newly accredited HEIs programs. Prior to the project, the quality of HEI programs was not assessed or measured.  By September 2019, 61% of university graduates from the 2017-2018 academic year had found employment, and of these, 40% had found work immediately on completing their studies (Survey on Education, Employment and Transition by University Graduates, in the Study on "Inclusive Growth in Mozambique, Scaling up Research and Transition by University Graduates.")  The higher education and science and technology regulatory frameworks were revised and led to the design of the 2014 National Higher Education Qualifications Framework in agreement with changes in the Law of Higher Education. Outcomes  In principle, the quality of students’ education was raised with CNAQ applying the accreditation system of standards and vouching for the quality-trained graduates. Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592)  However, the project's results framework was not designed to offer data on whether the new system and standards actually translated into the outcome of enhanced quality of graduates. The absence of students' exam scores or placements prevents measurement of a rise in the number of students who graduated from programs with enhanced quality. Due to absence of convincing outcome data on the quality of graduates, achievement of this objective is rated Modest. Rating Modest OBJECTIVE 3 Objective Strengthen the national research capacities to produce research outputs of relevance to the country's strategic economic sectors Rationale The theory of change for this objective was based on the premise that a larger number of funded research funded projects would build research groups and outputs of national excellence and relevance. Change was to happen with support for the national research fund (FNI) to finance researchers from HEIs and research institutions inside and outside the capital, young researchers, and research partnerships with industry. The premise for change was also grounded on the provision of capacity development to consolidate research findings and disseminate results (ICR, para 28). Outputs  88 research institutes received funding from the FNI, exceeding the revised target of 70 (original target was 25). The FNI became a fully institutionalized provider of research funding.  215 proposals were funded by the FNI (no target was provided).  176 sub-projects supported public and private HEIs to improve learning conditions and teaching and to support research, and the four of the research sub-projects were in partnerships with organizations or companies (reaching the target of four projects).  64 IDF-supported proposals to improve teaching and learning were approved, exceeding the revised target of 42 (original target of 31) proposals.  275 reports and research papers (citable and non-citable, out of 1,060 submitted) were published in domestic and international journals, surpassing the maintained original target of 220.  6 higher education statistical yearbooks and 3 publications with regular data on STI indicators were published, meeting the revised target (original target of 3 yearbooks). Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) Outcomes  The ICR (para 29) reports that 75 percent of research projects financed under this operation fully met their intended outcomes (and that an additional 11 percent of the research projects achieved a majority of their intended outcomes), thus meeting the target of 75 percent of research projects achieving their intended outcomes. Although the ICR explains that "this was validated during joint field visits that included the technical evaluators of the proposals and the FNI team" and includes (in Annex 7, Table 9) a list of some of the research projects and their achievements, the information provided on the validation's coverage and methodology (as well as on the underlying reporting of achievements of the research projects) is insufficient to support a conclusion that the observations about these achievements are reasonably reliable and accurate. This lack of sufficient information is considered a moderate shortcoming in achievement of this objective.  Evidence from surveys administered to the Mozambique Research and Education Network (MoReNet) in 2016 and 2017 indicated that 75 percent of the beneficiaries felt that the project's investments reflected their needs. This achievement is below the maintained original target of 85 percent of survey participants offering that response. Rating Substantial OBJECTIVE 4 Objective There was no fourth objective under the original project development objective statement. Rationale There was no fourth objective under the original project development objective statement. Rating Not Rated/Not Applicable OBJECTIVE 4 REVISION 1 Revised Objective Strengthening the institutional framework for TVET Revised Rationale The theory of change assumed that the development of a qualification system that linked medium-level education to TVET tertiary education would reasonably contribute to strengthening the TVET institutional framework. Such reinforcement was presumed to stem from support to the Executive Committee for Professional Education Reform (COREP) and funding from the FNI to TVET institutions, teachers’ learning, and COREP managers' capacity building. The expectation was that scholarships for teachers would enhance Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) their technical knowledge and abilities to implement a new TVET competency-based curriculum (CBT). In addition, the establishment of the first-time technical committees was at the basis of the expected actualization of technical fields in the TVET. Outputs  After the Manica Provincial Distance Learning Centre (CPEDM) was inaugurated on March 28, 2017 in the provincial capital, Chimoio, 3,075 students of higher education (particularly those that became teachers in secondary and primary schools), intermediate-level students originally without access to HEI conventional classrooms, and teachers enrolled in undergraduate courses.  116 scholarships were granted to TVET teachers who furthered their training in technical subjects, exceeding the maintained original target of 40 scholarships.  1,093 teachers received training on the new CBT, surpassing the maintained original target of 200. Based on external verifications, over 80% of teachers apply the new teaching methodology. The teachers with upgraded training contributed to an observed increase in 2018 in the students' pass rate of 73.3 percent compared to 67.9 percent in 2014.  66 TVET managers were trained on decentralized management institutions (no target was provided).  20 Sectorial Technical Committees were established (no target was provided).  34 TVET institutions were accredited and provided the CBTs. 11 projects were funded to develop vocational education institutions that implement the new CBT (no target was provided).  35 new TVET qualifications were developed. Qualifications of eight higher education fields were updated based on the revised system: health sciences (medicine); engineering (undergraduate degree in civil engineering, computer engineering and telecommunications, electrical engineering, chemical engineering); law (degree in law); education (teacher education); and management (degree in accounting and auditing). Outcomes  A qualification system was put in place connecting medium-level TVET with tertiary education. The TVET and the HEI qualification systems were synchronized at project completion, which in itself is evidence of enhancement of the TVET’s institutional framework.  Two tracer studies to students provided further evidence of achievement of this outcome: students not only indicated that the new competency-based curriculum (CBT) used in TVET institutions (supported by the project) was useful, but also that they prefer the CBT compared to the classical system, finding it more challenging and more uniformly accepted and recognized by employers (ICR, paras 33 and 34). Revised Rating Substantial Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall efficacy is rated Substantial under both the original and revised objectives. The majority of targets of outputs and outcomes were met or surpassed, and there is evidence that project interventions contributed to observable results. Among these, the project delivered a more than anticipated number of undergraduate and graduate students, the intended number of research publications with relevance for economic growth and supporting policy formulation, and a TVET-HEI synchronized system, both producing a larger number of graduates and many from enhanced programs. With respect to quality of education, although there is not concrete evidence of achieved outcomes, the CNAQ introduced quality assurance mechanisms whose impetus is likely to sustain the process of upgrading quality enhancement in major HEIs. Overall Efficacy Rating Substantial 5. Efficiency Cost-effectiveness Cost-benefit analyses based on a human capital approach were conducted at appraisal and completion by the ICR. At the basis of the approach is the assumption that the benefits of more years of education are higher productivity and higher future labor market earnings among graduates. The project also took into account the experience of prior Bank-financed higher education projects in Mozambique and other countries. Ex-ante analysis in the PAD considered as benefit the incremental salary, estimated as the difference between the salary of a worker with a higher degree and the salary of a worker with a secondary education degree (both in public service careers). Other social benefits (i.e., increased tax revenue) were not included. Cost estimates included: (a) costs of the intervention, (b) the social unit costs for undergraduate and postgraduate students, which includes the direct cost of schooling (registration and tuition) and other costs accrued by society (e.g. donation to universities), and (c) the opportunity cost of schooling. An estimated annual social unit cost of US$3,479 per student was used (PAD, para 260). The ex-post analysis in the ICR assessed direct contributions of the project in supporting the creation of a larger number of graduates. Project costs included the investment implementation cost of the three components. To assess scholarships, the estimated costs per student were taken from the UIS-UNESCO database, which corresponded to US$878 per student in tertiary education in 2013. According to the ICR (para 4), “benefits and costs of Components A and B were analyzed together, as these were assumed to be multi-level interventions to strengthen the TVET and higher education system as a whole. Finally, the impact of Component C, related to the higher education and postgraduate scholarships, was estimated." Costs and benefits were estimated using the actual number of students observed in TVET and HEIs, compared to a counterfactual scenario, where numbers were estimated in the case of the absence of the project. The project yielded an internal rate of return (IRR) of 12.1 percent, with a Net Present Value (NPV) of US$4,223 million. The IRR was in line with the IRR Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) estimated during the project’s appraisal (11.5 percent). Best practices and results from other recent projects supporting the development of TVET, higher education, or research were also used for comparison. Assessment by this review: The human capital approach, the estimated costs per student from UIS-UNESCO and the Internal Rate of Return of 11.5% used for economic analysis (taken from prior World Bank projects) are reasonable parameters to ex-ante determine the opportunity cost of capital invested in strengthening tertiary education and boosting the numbers of undergraduates and graduates. Similarly, findings from studies of scholarship impacts on student enrollment are a reasonable basis to establish a comparison of enrollment with intervention versus counterfactual scenarios to estimate cost and benefits among low-income students. Implementation efficiency The quality of data from regular monitoring enabled accomplishment of milestones, scaling-up/expanding the project's scope, and timely delivery of outputs. There were initial delays by the FNI (due to a slow process for opening bank accounts to receive money), and later, in 2014, by the CNAQ (due to changes in CNAQ management and longer-than-estimated time for HEIs to complete accreditation self-evaluations). Nevertheless, by 2013, the mid-term review found that outcome targets were going to be achieved prior to original closing date. In 2018, all implemented activities were completed and targets for all indicators were met or exceeded (ICR, para 37). At closure, all allocated project financing was completely disbursed and all planned activities executed. Assessment by this Review: The overall efficiency of the project is rated Substantial, as positive impacts (mainly increased graduation rate, enhanced scholarship mechanisms, a newly established CNAQ that accredited the quality of 97 programs, and a synchronized qualifications system of secondary education institutions with TVET) yielded economic returns (12.10%) higher than the estimated cost of capital (11.5%). The project was also diligent in its management of project funds. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  11.50  Not Applicable 0 ICR Estimate  12.10  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) 6. Outcome Relevance of the objectives is rated Substantial, as the three PDOs continue to be consistent with the development priorities of the Bank and the Government of Mozambique. As noted in section 4 above, the project performed with seemingly substantial efficacy, generating a larger number of graduates from undergraduate and graduate program than planned, and also operationalizing the CNAQ (although the theory of change and the ICR did not provide data to verify improved quality of graduates). The project was also effective in enhancing Mozambique’s research capacity and dissemination of research products, and it achieved the synchronization of TVET and secondary education frameworks with HEI programs. In the process, the TVET system was reinforced. Efficiency is rated substantial considering that, despite a few implementation inefficiencies, target levels were met or exceeded, and that the project yielded greater-than-anticipated value for money without budget overruns. However, moderate shortcomings were noted in achievement of some of the objectives/outcomes -- in particular, related to assessment of the quality of graduates at the undergraduate and graduate levels and assessment of achievement of intended outcomes by research projects financed. The substantial relevance, efficacy, and efficiency, along with moderate shortcomings in achievement of two of the objectives/outcomes used to assess overall efficacy, are consistent with an Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The increased flow of quality students was supported with a scholarship program and the creation of the quality assurance body. Post-project, the sustainability of this development outcome is potentially at risk due to Mozambique’s public spending constraints since 2016. Cuts in tertiary education budget allocations were not originally presumed. Without scholarships, the deficit of graduates is likely to persist, and the limited budget may hamper the consolidation of the CNAQ and of the ANEP. In addition, CNAQ may be underequipped to handle the increased demand for accreditation created by the project. However, quality assurance frameworks and standards in STEM and other industry-relevant areas, as well as mechanisms to develop research, have been mainstreamed and should be sustained. The continuation of private sector financing is another factor affecting sustainability; the project mobilized private sector financing in joint research and technology transfer activities (FNI), IDF loans to private institutions, and partnerships with the private sector for income generation (ICR, para 49). Going forward, this Review notes that without the Bank and private sector interventions, the sustainability of the PDO may be substantially compromised. Acknowledging this, the Bank is preparing a new project focusing on HEST sub-sectors, i.e., training, funding and scholarships, TVET support, and cementing the institutionalization of the ANEP (ICR, para 75). 8. Assessment of Bank Performance Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) a. Quality-at-Entry Data on private and public returns to education studies, as well as economic, sector, cost-benefit and financial analyses, and best practices (PAD, paras 242-247) formed the basis for the development of the project’s theory of change. The team aligned the theory of change with the country's Higher Education Plan and Bank priorities as outlined in the CPS at appraisal. Targets for the increase in quantity and quality of undergraduate and graduate students, strengthened research, scholarship awards, and funding mechanisms, as well as TVET frameworks, were set according to Mozambique's Higher Education Plan and the Bank's CPS. According to the ICR (para 55), the monitoring plan was suitable for the participating institutions and project units to carry out. However, an important shortcoming to quality at entry was that the project’s M&E design lacked indicators that would adequately measure the extent to which higher education institutions “raised the quality of graduates,” which is a notable absence given this goal was part of the PDO. Although the M&E design made provisions to measure the number of higher education institutions with accredited academic programs, apparently no provision was made to provide a metric of students’ performance (for example, demonstrating enhanced knowledge of specific subjects or enhanced skills for specific jobs), or to define what share of a cohort would be considered “a high number of graduates.” Without these parameters, it would not be possible to make a valid judgment of whether there was an increase in the number of high performing or “quality,” graduates. Except for this lack of specification about the number of students graduating from programs with enhanced quality, this review considers that the PDO statement was mostly well formulated and outcome-oriented. Other than the caveats and shortcomings noted above, many indicators were measurable and fit for tracking progress towards milestones as outlined in the results chain. However, there appear to have been opportunities for improvement in defining the methods for assessing and validating achievements of research projects. The preparation team identified risks congruously with the current situation, incorporated features from other Bank-supported higher education projects to mitigate those risks, and included relevant technical experts and Bank peer reviewers to assess the project's expected economic returns. Of particular relevance was that the design was calibrated with the Bank's know-how from the first higher education project in Mozambique (which focused on the quality of graduates and researchers), and it also prioritized gender and regional equality. Overall, project preparation appears to have assessed sustainability challenges; however, it missed anticipating potential post-project cuts in government allocations to tertiary education linked to the entry in office of a newly-elected administration. Such an event did not receive consideration as a potential deterrent for outcome sustainability. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Throughout the life of the project, the bi-annual Implementation Status and Results Reports rating on implementation supervision were always either satisfactory (early years of the project) or moderately satisfactory. Between November 2011 and June 2014, and again in June 2015 and in June 2018, HEST Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) implementation performance was rated moderately satisfactory, mostly due to challenges experienced by government counterparts in finding staff for the involved implementing agencies. The Bank showed responsiveness in supporting the AF, and skillful management of the project through challenging macroeconomic conditions resulting from external shocks, e.g., the global drop in commodity prices, weaker export revenues, devaluation of the metical against the dollar, and inflation, to cite a few (ICR, para 61). The Bank team closely coordinated with MEC, MTC, and ES-COREP/ANEP, reported on financial management and procurement progress, and worked with the project management unit team to build their capacity in these areas. Bank experts also supported various implementation units. In addition, experts participated in review panels for research proposals and the award of research grants (ICR, para 73). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E system design made it possible to track progress towards most of the PDOs. The system enabled the quantification of the increase in numbers of graduated students, recipients of scholarships and grants, participants in training programs disaggregated for TVET and HEIs, and the delivery of quality assurance outputs, e.g., the regulation for the inspection of HEIs and regulations for the procedures for licensing and functioning of HEIs. Similarly, the M&E system verified that the synchronization of a qualification system between medium-level TVET and tertiary education took place. Concerning improvement in quality of graduates, the ICR (para 9) reported that the objective was operationalized by the institutionalization of accreditation programs and the certifications of HEI programs by the QNAC. However, the M&E system lacked the ability to determine change, i.e., the change in quality of technical knowledge of students and TVET teachers who received training, and of students who graduated before CNAQ began its quality assurance functions. A measure for learning outcomes is unavailable, making it problematic to assess outcomes from knowledge-enhancement activities and the overall contribution of the project to improved quality of graduates. Responsibility for data collection initially sat with the National Directorate of Higher Education (DNES) in MEC and the National Directorate of Planning, Statistics, and Cooperation in MCT. After government reorganization, it was transferred to ES-COREP/ANEP and DNES in the Ministry of Science and Technology, Higher, Professional and Technical Education (MCTESP). b. M&E Implementation M&E activities included systematic routine monitoring, a mid-term review, and tracer surveys. Overall, M&E data was available to determine achievement of milestones, identifying implementation successes Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) or the need for corrective strategies to assure timely progress. For instance, the ICR indicated that the 2014 Implementation Supervision and Results Report (ISR) identified delays in CNAQ´s accreditation process (due to changes in its management staff and the underestimation of the time needed for HEIs to complete their self-evaluations). As a result, CNAQ was pulled back on track and was institutionalized before closure as the active education quality assurance player. In another example, the mid-term revealed the potential early achievement of targets, and on this basis the AF in 2015 took place. Tracer surveys showed the usefulness of the new CBT curriculum for TVET students and of the IDF grants. The ICR reported that field visits and assessments validated the finding that 75% of research projects had fully met their intended outcomes, although detail on the method and coverage of these validations was not provided. In addition, the project supported the design, development, and completion of a higher education management system, and the development of Science, Technology and Innovation indicators, which now inform on science and technology needs in Mozambique. A project-supported digitalized management and monitoring system helped the National Scholarship Institute in the selection of student scholarship recipients. c. M&E Utilization M&E findings were effectively used to monitor the status of implementation and progress on outcome indicators. Bi-annual M&E findings were shared with the MEC, MTC, MCTESTP, ES-COREP and ANEP, as well as with the project administrator. The 2015 mid-term review´s finding of the early reaching of targets was used by the government to ask for the introduction of a third PDO and the upscaling of targets. These data also were used by the Bank to approve the additional funds to restructure the project accordingly. Project-financed tracer studies and post-intervention analyses were used to corroborate evidence of impact, and to support the introduction of a functioning Board of Administration to institutionalize the FNI before closure. The project also used the FNI management and fiduciary monitoring system to ensure that funds were used for the intended research purposes. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was rated environmental assessment Category "B" (partial assessment), and OP/BP 4.01 "Environment Assessment" was triggered due to the construction of one distance learning resource center. Throughout the project, safeguards were monitored and were consistently rated Satisfactory in ISRs (ICR, para 69). b. Fiduciary Compliance Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) Evidence reported by the ICR (para 70) supported the observation that the project made proper budget execution and fund/grant disbursements as stated in the Financial Agreements. Financial management performance was rated satisfactory in ISRs throughout the life of the project for all relevant implementation agencies, i.e., MCT and MEC (original project design) and MCTESTP and ES- COREP (AF design). Ex-ante, the PAD stated that the Bank team carried out a financial management capacity assessment. Workshops were conducted by Bank financial management specialists after the Bank introduced its new Client Connection. All of the implementation agencies accomplished the financial covenants of the Financing Agreements, including agencies disbursing scholarships and research grants. Interim financial reports were submitted to the World Bank in a timely manner, and ES-COREP budget execution was consistently rated satisfactory. Auditors’ opinions throughout the project were unqualified (clean). The final audit report took place on June 30, 2019. Procurement was consistently rated satisfactory in ISRs except for one assessment of MEC performance rated moderately satisfactory due to recruitment delays of a procurement specialist. The Bank team ensured compliance with procurement guidelines through training of procurement staff in implementing agencies and sub-project teams. No deviations from the Bank’s procurement guidelines were reported. HEST migration to the Bank’s Systematic Tracking of Exchanges in Procurement (STEP) took place in June 2017, with procurement activities being included in the STEP platform. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome rating of Moderately Satisfactory, based on substantial relevance, efficacy, and efficiency, along with moderate shortcomings in achievement of some of the objectives/outcomes used in the Outcome Satisfactory Moderately Satisfactory assessment of overall efficacy (in particular, related to assessment of the quality of graduates at the undergraduate and graduate levels and assessment/validation of the achievement of intended Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) outcomes by research projects financed). Quality at Entry is rated Moderately Satisfactory because the M&E framework is Bank Performance Satisfactory Moderately Satisfactory considered to have been inadequate to measure achievement of all the objectives. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR (paras 77 to 81) offered several useful lessons, including the following adapted by IEG: 1. Establishing a National Qualification Framework requires strong commitment and ownership from all stakeholders. An inter-ministerial governmental structure can ensure commitment, framework ownership, and synchronization of qualifications of tertiary education institutions and TVET. In this case, the incorporation of TVET as part of the MCTESTP, along with the participation of area-specific experts from HEIs, helped ease the transition of students from TVET institutions to HEIs as both systems became synchronized. 2. Effective accreditation mechanisms are essential to ensuring quality of higher education programs and institutions. In the case of the CNAQ, the use of digital platforms to support data management introduced more flexibility to the accreditation process for courses and programs. External evaluation of the CNAQ's performance helped the government gain insight on how to improve HEI's own processes and procedures. 3. Anticipating the capacity needs for an accreditation body to meet a potential post-project increase in the demand for HEI accreditation enables newly created bodies, such as the CNAQ, to solidify after closure. This was a key point in the case of the HSET, as compulsory accreditation and a pre-accreditation for all new HEI programs became a requirement. The HEST did not take this factor into account, and consequently the QNAC faced challenges in its ability to meet increased demand for its accreditation services, putting the sustainability of the outcome at risk. 4. The following considerations can support the enhancement of research capacity and its relevance:  Early development of guidelines with clear selection and award criteria are key for research grants implementation.  Training on proposal writing and presentation as well as feedback to researchers can accelerate readiness and quality of research proposals.  Site visits to research groups by area-specific specialists, followed by capacity building in fiduciary matters, can help ensure compliance with research fund procedures. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592)  Partnerships and knowledge networks with international institutions can enhance research project funding, harmonization of methodologies for data collection, and strengthening of higher education quality assurance mechanisms. 5. Scholarships for equity and competitiveness programs can benefit from:  M&E processes with beneficiaries and students involved can improve the quality and relevance of information.  Clear eligibility and selection criteria of scholarship candidates help achieve objectives.  A shift from a demand-driven model to integrated supply-side scholarship management is eased when clear planning and monitoring mechanisms and procedures are applied to scholarship holders. Similarly, the shift is facilitated when the rights and duties of scholarship holders, host institutions, and scholarship granting institutions are clearly defined in the scholarship’s handbook. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided thorough information on the project's scope and implementation. Overall analyses and descriptions were presented with candor. Evidence from the M&E system and post-training surveys formed the basis of the ICR’s logical explanation of results, supported by analyses of data referenced in tables in the main text and annexes. The lessons section provided coherent insights on relevant factors to assure effectiveness. However, the ICR had room for added clarity on the analysis of results achieved. For instance, the ICR did not discuss the framing of the first objective, which encapsulated multiple dimensions of change in one phrase (such as quantity, quality, and two levels of education) and how this posed challenges for assessing achievement of each individual aspect of the PDO. In particular, the inclusion in the objective of two distinct elements (quality and quantity) was not acknowledged. The narrative of achievement related to the research outcomes objective contained shortcomings in terms of its fluidity and organization. a. Quality of ICR Rating Substantial Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ Higher Educ Science & Techn. (FY10) (P111592) Page 19 of 19