ROMANIA Reimbursable Advisory Services Agreement on Technical Assistance to Enhance Quality Assurance in Higher Education System in Romania (P157508) Output 7. Training and dissemination report May 2021 Project co-financed from the European Social Fund through the Administrative Capacity Operational Programme 2014-2020! www.poca.ro This report was delivered under the Reimbursable Advisory Services Agreement on Technical Assistance to Enhance Quality Assurance in Higher Education System in Romania signed between the Romanian Agency for Quality Assurance in Higher Education and the International Bank for Reconstruction and Development on July 24, 2017. The present document corresponds to Output No. 7 under the above-mentioned Agreement. Disclaimer This report is a product of the International Bank for Reconstruction and Development/the World Bank. The findings, interpretation, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. This report does not necessarily represent the position of the European Union or the Romanian Government. Copyright statement The material in this publication is copyrighted. Copying and/or transmitting this work or portions of it without permission may be a violation of applicable laws. For permission to photocopy or reprint any part of this work, please send a request with the complete information to either: (i) the Romanian Agency for Quality Assurance in Higher Education (59, Mărăști Blvd, Bucharest, Romania); or (ii) the World Bank Group Romania (31, Vasile Lascăr Street, Et 6, Sector 2, Bucharest, Romania). Acknowledgment This report was prepared by Alina Sava (Education Specialist, TTL), Manuela Rădulescu (Education Specialist), and Iulia Mărieș (Education Specialist), with inputs from Romina Miorelli and Rafael Llavori-de Micheo (International Experts). The Task Team would like to thank ARACIS representatives, especially Mr. Iordan Petrescu (President) and Mrs. Emilia Gogu (Project Coordinator). Contents Abbreviations and Acronyms ..................................................................................................................... 4 Executive Summary.................................................................................................................................... 5 Introduction ............................................................................................................................................... 6 I. Project background ............................................................................................................................ 8 The methodology for the classification of HEIs and ranking of study programs ................................... 9 The methodology for external assessment of HEIs ............................................................................... 9 II. Building Capacity .............................................................................................................................. 11 The concept in this context.................................................................................................................. 11 The approach ....................................................................................................................................... 11 III. Public Consultations ......................................................................................................................... 13 Public consultations on the methodology for external assessment of HEIs........................................ 13 Public consultations on the methodology for the classification of HEIs and ranking of study programs ............................................................................................................................................................. 15 Interviews with ARACIS representatives and administrative staff ...................................................... 16 IV. Training ............................................................................................................................................ 18 Testing and piloting the methodology for the classification of HEIs and ranking of study programs . 18 Training for the use of the IT platform ................................................................................................ 18 Final Training ........................................................................................................................................ 19 V. Communication and Dissemination Activities ................................................................................. 20 VI. Addressing the sustainable development, equal opportunities, non-discrimination, and gender equality aspects ....................................................................................................................................... 22 VII. Lessons learned and recommendations .......................................................................................... 25 Annexes (included as a separate document) ....................................................................................... 27 3 Abbreviations and Acronyms ARACIS Romanian Agency for Quality Assurance in Higher Education BEX Executive Board of ARACIS Council ECA European and Central Asia EHEA European Higher Education Area ENQA European Association for Quality Assurance in Higher Education EQA External Quality Assurance EQAR European Quality Assurance Register for Higher Education EU European Union EUA European University Association HE Higher Education HEI Higher Education Institution IM Internal Manual IT Information Technology IQA Internal Quality Assurance LEN Law of National Education MoE Ministry of Education QA Quality Assurance QAFIN Improving Public Policies in Higher Education and Enhancing the Quality of Regulations by Updating Quality Standards Project QMS Quality Management System RAS Reimbursable Advisory Services ROF Regulation of Organization and Functioning UEFISCDI Executive Unit for Financing Higher Education, Research, Development, and Innovation WB World Bank 4 Executive Summary 1. This document presents an overview of capacity building and dissemination activities, training, workshops, etc., that aimed to strengthen the Romanian Agency for Quality Assurance in Higher Education (ARACIS) and the Ministry of Education (MoE)'s capacity to implement the external assessment and classification methodologies. 2. The activities summarized under three main sections of the Report – (i) public consultations, (ii) training, and (iii) dissemination activities – have been carried out throughout the project and included a series of training sessions on the utilization of the information platforms, and capacity building seminars for the ARACIS and MoE staff, among other activities. The report also incorporates a section dedicated to sustainable development, equal opportunities, non-discrimination, and gender equality to respond to ARACIS's specific request. 3. Updating national frameworks, methodologies, guidelines, and procedures aligned with European standards should improve learning outcomes, resilience, inclusion, and sustainability of the Romanian higher education ecosystem. Creating ownership for results, public consultations, and a permanent strive for adopting modern instruments will increase Romania's quality culture. The methodologies, guidelines, and procedure manuals developed within the project will serve as working instruments for professionals involved in public policymaking and allow implementing quality assurance processes and classification and ranking exercises in the following years. 4. The project also created instruments to gather data and build databases to perform future exercises, develop transparency tools and quality policies. These databases will be used and supplemented in the long-term by decision-makers in the higher education sector, while the results will be used by relevant stakeholders, particularly students and employers. The developed platforms will transfer knowledge to relevant actors in the higher education system, establishing a high-performance management approach. 5. The training and information sessions organized during the project implementation have been an essential resource for internal and external stakeholders, including HEIs representatives. Subsequently, they will disseminate the information at institutional, local, regional, and even sectoral levels. The experience of the project can be replicated to other levels, such as HEIs, or departments of institutions with responsibilities in higher education. The project provides the expertise and instruments to ensure a coherent and integrated approach of all results over an extended period. 6. Through the project, the quality assurance in Romanian Higher Education System marked a significant step forward. ARACIS must enhance its role as a critical active player in the current debate on quality assurance trends within the European Association for Quality Assurance in Higher Education (ENQA) and European and Central Asia (ECA) region. More active Romanian higher education and quality assurance sectors will ensure a more diverse higher education landscape in the European Higher Education Area (EHEA). 5 Introduction The Reimbursable Advisory Services Agreement on Technical Assistance to Enhance Quality Assurance in Higher Education System in Romania aims to improve the capacity of the Ministry of National Education1 and the Romanian Agency for Quality Assurance in Higher Education (ARACIS) to govern the quality assurance and to support increased attainment, quality, and efficiency of the Romanian higher education system. This Agreement is designed to support the improvement of the administrative capacity of ARACIS by developing procedures for the internal organization of activities, training of staff, and developing and implementing consultation mechanisms to ensure that all stakeholders are participating in the decision-making process. The Agreement is part of a broader project, "Improving Public Policies in Higher Education and Enhancing the Quality of Regulations by Updating Quality Standards" (QAFIN). The Project beneficiary is the Ministry of Education (MoE), in partnership with ARACIS. The World Bank (WB) was subcontracted to provide technical assistance in the implementation of several activities. ARACIS, empowered by the beneficiary, has taken the lead for most project activities. World Bank ARACIS Ministry of Education This Reimbursable Advisory Services (RAS) Agreement comprises two components: (a) Component A – Improve Management and Quality Assurance Practices in the Romanian Higher Education System, and (b) Component B – Capacity Building for the Recipient and the MoE Staff. Improve Management and Quality Assurance Practices in the Romanian Component A Higher Education System Component B Capacity Building for the ARACIS and the MoE Staff Under the first component, two important documents presenting recommendations for preparing MoE and ARACIS methodologies have been delivered, respectively: (1) a new methodology for the 1 The name of the Ministry was changed from Ministry of National Education (MoNE) to Ministry of Education and Research (MoER) in November 2019 and Ministry of Education (MoE) in December 2020. 6 classification of higher education institutions (HEIs) and ranking of study programs – to MoE, and (2) a new methodology for external assessment of HEIs and associated management instruments – to ARACIS. The second component aims to strengthen ARACIS and MoE's capacity through training activities and informative workshops on the assessment and classification methodologies developed under this Agreement. The capacity-building process started at the project launching and continued throughout the implementation activities carried out during the joint preparation of external assessment and classification methodologies. In the process, the international and local team shared knowledge and expertise to build solid outputs and create ownership from the beginning. As such, the capacity building consisted of a sum of activities (e.g., revision of internal procedures and regulations) and complemented with consistent and regular training and workshops to strengthen the capacity of ARACIS and MoE, specifically to implement the external assessment and classification methodologies, while also creating a modern European Quality Assurance Agency. This report presents an overview of capacity building and dissemination activities, training, workshops, etc. The activities summarized under three main sections – (i) public consultations, (ii) training, and (iii) dissemination activities – have been carried out throughout the project and included a series of training sessions on the utilization of the information platforms, and capacity building seminars for the ARACIS and MoE staff, among other activities. In addition, training materials, such as presentations, agenda of meetings, surveys, etc. are attached to this report for further dissemination. Disclaimer: The report incorporates a dedicated section on sustainable development, equal opportunities, non-discrimination, and gender equality aspects, as per the Client's specific request. More details are included in Chapter VI. Phase I. Phase V. Methodologies Phase II. Public Phase III. Testing Phase IV. Communication & Platforms Consultation & Piloting Training & Dissemination Development Figure 1. The flow of communication and dissemination activities The outbreak of the COVID-19 pandemic in early 2020 impacted the entire society, including the HEIs. It showed that technology would be the primary resilience instrument for the higher education sector. At the same time, HEIs will need to operate more strategically as teaching, learning, and research embrace and adapt to remote delivery and online settings. ARACIS and MoE also need to adapt their processes and procedures to this context. Thus, the entire higher education system should invest in developing digital infrastructure toward building more agile and flexible systems. 7 I. Project background Scope of the Project. The project aims to achieve and implement evidence-based public policies, specifically the correlation of strategic planning with program budgeting in financing HEIs, through enhancement of external quality assurance, classification of HEIs, and ranking of study programs. The project also focuses on improving the institutional capacity of MoE and ARACIS by adapting structures, optimizing processes, training human resources, and enhancing the use of transparent public consultation mechanisms to ensure stakeholders' participation in decision-making and application of European quality standards. Among the specific objectives of the project are: A. Optimization of strategic planning and program budgeting by: (i) providing the necessary tools, methodologies, and guidelines for external quality assurance in higher education, respectively classification of HEIs and ranking of study programs, and informing decisions regarding how funds are allocated to HEIs and their study programs; (ii) clarifying the role and mandates of institutions involved in quality assurance processes in higher education. B. Implementation of the methodology and guidelines for classification of HEIs and ranking of study programs based on clear and transparent sets of indicators and data. C. Enhancement of ARACIS institutional capacity by revising internal procedures and regulations, developing manuals and guidelines, and training its staff, as well as staff of MoE and HEIs. Box 1. Main project results • Methodology and guidelines for external assessment of HEIs. • Methodology for the classification of HEIs and ranking of study programs. • Revised Regulation of Organization and Functioning (ROF) and Internal Manual (IM) of ARACIS. • Procedures manual on the role and competencies of MoE, subordinated councils, and ARACIS, in quality assurance, ranking, and classification processes. The COVID-19 pandemic impacted the project activities planned in the last year of implementation, particularly the final training and dissemination activities. However, the COVID-19 crisis presents an opportunity to restructure practices to incorporate valuable lessons from online activities and develop new strategies for accelerating the pace of the project implementation. Most of the activities have shifted to online, benefiting from the recent developments of online platforms and video conferencing tools. Physical meetings have become virtual, while large events have been canceled or postponed. The QAFIN project was suspended for two months, during April and June 2020. After resuming the activity, the RAS was amended to extend its implementation until June 29, 2021. Several measures have been identified by MoE, ARACIS, and WB teams to overcome the lack of traditional dissemination activities: - Organizing virtual meetings and seminars to disseminate the project's activities to a large audience, internal and external stakeholders, and experts [see chapter V for details]; 8 - Organizing virtual interviews with members of the Executive Board (BEX) and administrative staff of ARACIS to substantiate recommendations on improving the internal manual of procedures, regulations of organization and functioning (ROF), job descriptions, and alignment of the departments' functions [see chapter III for details]; - Increasing visibility and transparency of ARACIS activities and project results through regular updates of the website (www.aracis.ro) [see chapter V for details]; - Reaching out to academics through the National Council of Rectors [see chapter V for details]; - Organizing virtual training to increase MoE and ARACIS institutional capacity [see chapter IV for details]. In the current COVID-19 crisis, the main implementation challenge relates to the virtual training delivery by a pair of international and local experts for approximately 225 participants. The methodology for the classification of HEIs and ranking of study programs The methodology for the classification of HEIs and ranking of study programs (bachelor level) complies with the requirements of the Law of National Education (LEN) no. 1/ 2011. The Classification of HEIs component is focused on grouping universities in categories A, B, and C as stipulated in the LEN. According to this Law, class A is for universities mainly focused on education; class B, for universities focused on education and scientific research and artistic creation; and class C, for universities of advanced research and education2. The allocation of universities to the categories A, B, and C seeks to group them according to the different focus on education (teaching and learning) and scientific research. For each fundamental field, the categories are defined, considering mainly activity and demographic indicators rather than performance indicators. The Ranking component seeks to measure the quality of individual study programs. The ranking is a straightforward assessment of quality across several dimensions, including research. It allows institutional program offerings to be compared and put in a hierarchy. The ranking of study programs is envisaged to be implemented in clusters at the level of bachelor study fields. The evaluation for classification of HEIs and ranking of study programs is performed based on the following criteria: (a) teaching and learning; (b) scientific research; (c) internationalization; (d) regional and social engagement; (e) knowledge transfer; (f) students and graduates' satisfaction. A set of indicators is assigned for each criterion. The indicators used for the classification of HEIs differ from those used for ranking of study programs. According to the international practice in higher education, European Standard Guidelines 2015, recommendations of the Bologna Process in terms of transparency, the classification and ranking exercises are carried out to provide a clear picture of the diversity in the HE system. The methodology for external assessment of HEIs The main goals for the external assessment of HEIs are: (a) strengthening the quality and relevance of learning and teaching; (b) strengthening the employability of graduates throughout their entire working life; (c) increasing the degree of inclusion of the national higher education system; and (d) implementing the structural reforms. 2 Law of National Education (LEN) no. 1/ 2011, art. 193, para 4. 9 The proposed methodology has moved from an input-based approach to an output- and outcome-based approach, with indicators defined to enhance accountability and quality assurance in higher education. It aimed to focus on the mechanism and core framework to support ARACIS to develop and implement its external assessment activities in an increasingly demanding European context. Two new QA processes have been taken into considerations. One is the Audit, which is to be implemented to evaluate HEIs with mature internal quality assurance systems, and the other is the evaluation of joint programs. Overall, these developments aim to enhance the HE system and align it with current trends in the European Higher Education Area (EHEA). More specifically, the introduction of Audits is driven towards strengthening the quality culture across and within HEIs that has been developed in the last decade in the country. Audits' use also allows for a more flexible and adaptable framework to the increasingly changing nature of the HE sector. Audits will be available to HEIs that obtained the highest results qualifications in the accreditation or periodical evaluation processes. In these cases, HEIs will be entitled to carry out internally these evaluation processes. Regarding the assessment of the Joint Programs, the proposal is to adopt the European Framework. The standards are based on the agreed tools of the EHEA without applying additional national criteria. These have been developed to facilitate an integrated approach to quality assurance of Joint Programs that genuinely reflect their joint character. According to the European Framework, these standards should be applied depending on the needs of the cooperating HEIs and the requirements of their national frameworks. 10 II. Building Capacity The concept in this context The report refers to the conceptual definition of "institutions" from a broad perspective. The World Development Report 2002 defines institutions as rules, enforcement mechanisms, and organizations 3. The concept of institutions in the context of this report refers to the pair of organizations and rules, whether formal or informal. Formal rules include laws, regulations, procedures, or rules enforced by legal authorities, while informal rules reflect unwritten codes of social conduct. There are various ways of defining institutional capacity building. The WB 4 defined capacity development (or capacity building) as a locally driven process of learning by leaders, coalitions, and other agents of change that brings about changes in socio-political, policy-related, and organizational factors to enhance local ownership for and the effectiveness and efficiency of efforts to achieve a development goal. There are debates in the literature on whether the capacity building is different from capacity development/ enhancement (considering that institutional capacity building deals with a new structure or rules, while development/ enhancement is focused on improving existing ones). The WB has generally accepted that institutional capacity building refers to all stages in the process; hence this report will use the terms interchangeably. Institutional capacity building interventions refer to technical assistance activities focused on enhancing institutions (e.g., support improvements in the effectiveness of institutions by revising and improving their structure, management, and procedures), and changes in regulations (supporting the development of new or updated policies, rules or regulations applied across the entire institution). The goals of any policy, program, or project will only succeed if the recipients improve their ability to engage, benefit from the provided support, and take ownership. Effective changes related to policy instruments (e.g., methodologies, procedures) will not be achieved beyond the RAS project without strong stakeholder ownership. When change is targeted in an institution, the newly adopted measures require time, space, and ownership. Additional support seems to act as an essential lever for institutional enhancement and strengthening of the demand-side capacity. Such change also depends on the institutional enabling environment to foster commitment, motivation, authority, and leadership to implement the proposed recommendations. Both high-level and technical-level commitments are needed to address the institutional capacity constraints effectively. The approach One of the critical goals of the World Bank in providing advisory services is to enhance capacity in implementing institutions using the WB's expertise and acting as the catalyst, facilitator, adviser, and knowledge provider. The capacity-building process started at the project launching and continued throughout the implementation activities carried out during the joint preparation of the methodology for external assessment and the methodology for classification of HEIs and ranking of study programs. 3 World Development Report 2002, Building Institutions for Markets, World Bank, 2002. 4 The Capacity Development Results Framework, World Bank, 2009. 11 A capacity development approach has been taken to ensure ARACIS gained the knowledge, skills, and experience to lead to sustainable change. The partnership has been characterized by critical reflection and, wherever possible, research evidence. Active steps have been taken to support a process of continuous improvement. Our capacity-building approach aimed to build a sustainable future for quality assurance education reform. The WB team always looked to work jointly with ARACIS as a partner to build capacity, support knowledge transfer, and secure engagement with the higher education community through well-planned communications. In addition, the WB and ARACIS worked closely with stakeholders to ensure a clear sense of needs and priorities to maximize the impact of the work for the benefit of stakeholders. In the process, the international and local WB team shared knowledge and expertise, organized mutual working sessions with ARACIS and MoE experts, and public consultations to build solid outputs and create ownership from the beginning through the end, over four years of implementation. ARACIS and MoE experts provided feedback on multiple iterations of the methodologies for external assessment, and classification and ranking, and incorporated comments, observations, and recommendations. The methodology for the classification of HEIs and ranking of study programs was finalized and published in October 2020. In February 2021, the final version of the methodology for external assessment of HEIs was published on the ARACIS website and submitted to MoE for official approval. The World Bank team highly appreciates the excellent cooperation, availability, and involvement of the ARACIS representatives and local experts and is grateful for their continued support and feedback. To support both methodologies, the WB team helped ARACIS develop two IT modules as tools to inform results on the classification, ranking, or external assessment. Currently, HEIs are introducing data as a first exercise of classification and ranking to test the relevance, reliability of indicators and data proposed, as well as applicability. These tools are used to facilitate communication with HEIs and provide templates to support conducting the QA evaluations, classification of universities, or ranking of study programs. Simultaneously, ARACIS provides HEIs with access to the IT module developed and there is a clear breakdown of tasks in the process, as illustrated by the picture below. MoE ARACIS HEIs inform HEIs populate the IT module cu provide acces to IT module provide acces to databases data Figure 2. Key tasks of the institutions involved in the classification and ranking exercise Finally, higher education must invest more in monitoring and evaluation systems to analyze results, interventions, or policy deployment, leading to evidence-based decisions and equally build the institutional capacity to collect and analyze data at the system level. Addressing institutional challenges and continue to enhance institutional capacities remain key to improve resilience to future shocks and the entire educational ecosystem. Resilience is built with a commitment to policy action implementation, is enabled by adequate learning and research environment and efficient allocation of resources, only when supported by enhanced capacity to design 12 and implement policies, prioritize resources, plan and execute investments. In addition, resilience needs greater coordination between ministry, central agencies, and HEIs to improve emergency response capacity; the potential impact of future shocks remains high and calls for urgent action and flexible systems. III. Public Consultations Public consultations on the methodology for external assessment of HEIs The adjustments to the ARACIS QA methodology proposed by the WB team draw on good international practice and inputs provided by ARACIS and other relevant stakeholders. Insights from the international experiences draw heavily on the findings of background research conducted by the WB team to develop the methodology 5 and on the extensive experience on QA systems in Europe and beyond of the international experts involved in the team. In addition to the literature review and review of institutional practices based on publicly available documents for QA agencies around the world such as analytical works, annual reports, and self- assessment reports – the latter often prepared as part of a review conducted by the European Association for Quality Assurance in Higher Education (ENQA) – as well as documents prepared for quality procedures conducted by the QA agencies, independent QA experts from across the European Union and other comparative contexts were interviewed. The experts interviewed were Ellen Hazelkorn (Dublin Institute of Technology, Ireland), Rick Hopper (President, Kennebec Valley Community College, USA), and María-José Lemaitre (Executive Director, Centro Interuniversitario de Desarrollo, Chile). These interviews helped to shape the guides prepared for interviews with representatives of the QA agencies. Subsequently, the WB team interviewed representatives of QA agencies from the selected countries and representatives of European QA agencies 6. AQ Austria: The Agency for Quality Assurance and Accreditation Austria based in Vienna, Austria AQU: The Catalan University Quality Assurance Agency based in Barcelona, Spain ASIIN: The Accreditation Agency for Study Programs of Engineering, Information QA Agencies Science, Natural Sciences and Mathematics based in Düsseldorf, Germany EVALAG: Evaluation Agency Baden-Württemberg based in Mannheim, Germany NOKUT: The Norwegian Agency for Quality Assurance in Education based in Oslo, Norway NVAO: The Accreditation Organization of the Netherlands and Flanders based in the Hague, the Netherlands 5 World Bank, 2018, Quality Assurance in European Higher Education: Using Polarities to Compare Sound Practices in External Quality Assurance in Select Systems. 6 Interviews were held between September to November 2017. 13 QQI: Quality and Qualifications Ireland based in Dublin, Ireland UKA: The Swedish Higher Education Authority based in Stockholm, Sweden EUA: European University Association European Agencies EQAR: European Quality Assurance Register for Higher Education EURASHE: European Association of Institutions in Higher Education ESU: European Students' Union Organizations Student ANOSR: National Alliance of Student Organizations in Romania The WB team reviewed ARACIS's QA methodologies and corresponding legislative framework and discussed different proposal drafts in several consultation instances. The latter included meetings held between September 2018 to March 2019 to prepare the first draft and two workshops, as pre- consultation events on the QA framework on the external assessment of HEIs, organized in February 2019 for feedback on subsequent iterations of this proposal. ARACIS representatives and its main stakeholders – QA external expert evaluators (registered in the ARACIS' Evaluators Registry), internal QA experts, HEIs representatives, and students – participated in the events. The workshops were conducted by Fabrice Henard and Romina Miorelli, WB international experts. The draft methodology, delivered by the WB team in August 2019, was consulted with relevant actors and stakeholders in the Romanian Higher Education System. The public consultation process took place between October 2019 and February 2020. ARACIS and MoE have organized five regional conferences in the main university cities: Bucharest (October 2019), Timișoara (November 2019), Iași (December 2019), Cluj-Napoca (January 2020), and Brașov (February 2020). Over 250 participants attended the public consultation events. The WB international experts Romina Miorelli and Rafael Llavori-de Micheo presented new proposed elements of the methodology. The presentation is enclosed as Annex 6. The public consultations aimed at identifying innovative aspects and the appropriate mechanisms related to the quality assessment of HEIs. ARACIS organized this ample process with the participation of the National Council of Rectors, student associations, trade, and employers' unions. The public consultations validated the external assessment methodology, standards, criteria, and indicators proposed by the World Bank and revised by the experts involved by ARACIS in project implementation. The consultations focused on two main areas, largely covered by the two questionnaires applied during the five regional conferences: an analysis of the perception on the role of quality assurance process of HEIs, and ensuring transparency in the process of revising the methodology and tools used in external assessment: - Ensure consensus on the proposed standards, criteria, and indicators for the assessment of doctoral programs and joint programs. 14 - Establish the opportunity to introduce external audits instead of external institutional evaluations. - Validate the proposed criteria. - Validation of the alignment with other legislative regulations and public policies. The methodology for external assessment of HEIs was finalized and published on the ARACIS website in February 2021 and submitted to MoE for official approval. Public consultations on the methodology for the classification of HEIs and ranking of study programs In December 2017, a first draft of the methodology was published on the MoE website to be publicly consulted by various stakeholders. The methodology was presented at the National Council of Rectors in February 2018 and an updated version in June 2018. Public consultations on the methodology for the classification of HEIs and ranking of study programs in Romania were conducted in June 2018. This joint consultative process was undertaken as part of the WB and ARACIS's work to develop the methodology for the classification of HEIs and ranking of study programs. It underlined emerging themes and recommendations that served as inputs to the final version of the methodology. The WB team consulted various stakeholders and policymakers throughout the consultation process to ensure a comprehensive representation of ideas and concerns. These consultations were conducted with the following objectives guiding the process: • present the proposed methodology for the classification of HEIs and ranking of study programs in Romania; and • organize working groups to obtain feedback on the proposed methodology and indicators. The public consultations were held in five (5) cities across Romania, in Bucharest, Timișoara, Iași, Cluj-Napoca, and Brașov, from June 11 to June 15, 2018. Participants and stakeholders joined the consultation process from neighboring cities, in addition to the five cities listed above. The consultations were conducted by the WB experts – Romina Miorelli, Alex Usher, and Jeremie Amoroso – and attended by representatives of universities, the business sector, students, and experts of the MoE and ARACIS. An online survey was launched in July 2018 to consolidate stakeholders' opinions for improving and completing the methodology on the classification of HEIs and ranking of study programs. The survey was designed to collect feedback and suggestions to improve the methodology and structured as follows: • Section I: respondents' opinions and proposals for completing the methodology – allowing to understand their perspective on the current legislative framework, the impact of the classification and ranking exercise, and their priorities in terms of key indicators; and 15 • Section II: respondents' profile – allowing to group the respondents and to compare answers based on their characteristics if needed. Over 260 participants attended the public consultation events and working groups discussions, and 413 stakeholders answered the online survey. An advanced version of the methodology was presented at the National Council of Rectors in February 2019 and the final version in October 2019. The methodology for the classification of HEIs and ranking of study programs and the list of indicators were finalized and published in October 2020 on the ARACIS website. Interviews with ARACIS representatives and administrative staff In July 2020, qualitative interviews were carried out with the Executive Board (BEX) and administrative staff representatives of ARACIS to identify procedural gaps in the current organizational flow and areas for improvement. The primary goal was to receive insights to substantiate proposals on improving the internal manual of procedures, ROF, job descriptions, and alignment of the departments' functions. Due to the COVID-19 pandemic that required a switch from physical to virtual meetings, the interviews were also conducted online. Training needs analysis. The immediate purpose of the report was to strengthen the organization, modernize it, and create an enabling environment for productivity and performance. The analysis focused on identifying any deficiency or discrepancy in the institution that could hinder the coherent and smooth implementation of activities. It aimed at improving organizational performance and stimulating the effectiveness and efficiency of its operations to improve the delivery of services. The scope of the training needs analysis was to ultimately create a strong organization with a good reputation and a well-defined footprint to be recognized as such in the EHEA. Internal Manual (IM) of ARACIS. The WB team presented recommendations to revise the Internal Manual of Quality Procedures and Activities of ARACIS to fit in the new proposed external evaluation framework. The report described the quality management system (QMS) of ARACIS, identifying principles related to organization and resources, allowing the ARACIS staff to understand better and implement the quality assurance policy. The IM is a methodological and operational reference for the ARACIS management and staff involved at various evaluation and QA services levels. It presented a set of concepts and methods for the Agency to develop and maintain a QMS. Regulation of Organization and Functioning (ROF) of the ARACIS. The WB team presented recommendations to revise the ROF of ARACIS to better align with the new external evaluation framework and better address the needs of the Romanian HE system while also complying with the regulations in place. 16 Therefore, the WB proposal aimed to support the strengthening of the organizational structure of ARACIS and recommended standardized job descriptions concerning this structure. The proposed job descriptions were built on the following principles: (a) clear distinction between the functional responsibilities, (b) clear hierarchical relations, and (c) well-defined roles and tasks for each position so that ARACIS can develop job descriptions for each specific job based on the selected examples. 17 IV. Training Testing and piloting the methodology for the classification of HEIs and ranking of study programs The methodology for the classification of HEIs and ranking of study programs was tested and piloted in different stages of development for refining the indicators and allowing universities to provide constant feedback on the IT platform design. The WB IT experts developed the IT platform to support implementing the classification of HEIs and ranking of study programs exercises. The platform was tested during the piloting phase and updated accordingly. The teams agreed to simplify the data collection from universities and streamline the reports for classification and ranking purposes. The system can provide templates of reports per university, fields of study, and indicators tested during the piloting phase. In April 2019, the WB IT experts presented the IT platform during the first Regional Conference, organized by the MoE in Bucharest. The IT platform will be used on the one hand by universities to introduce/upload data, and on the other hand, the ARACIS and MoE will regularly aggregate data for the classification of HEIs and ranking of study programs. First Phase: the first pilot project was implemented between June and July 2019 and involved six universities that voluntarily accepted to participate in the pilot, verify the data gathered from various MoE databases, and fill in the missing data. ARACIS and WB teams jointly prepared a piloting report. Second Phase: the nationwide pilot project was implemented between May and June 2021. A number of 70 universities with more than 2150 study programs (BA level) have agreed to participate in the exercise. This pilot project represents a considerable effort coordinated by the ARACIS and WB IT experts, involving over 200 university representatives. A follow-up report will be prepared in July 2021. Training for the use of the IT platform The first Regional Conference, organized by the MoE in Bucharest, was followed by a training on data collection conducted by the WB team in May 2019. The "Training for data collection in the IT platform" was dedicated to the representatives of the six universities selected for the pilot phase (Politehnica University of Bucharest, University of Craiova, Bucharest University of Economic Studies, Al. I Cuza University of Iași, West University of Timișoara, Transilvania University of Brasov). A number of 32 participants, including WB, MoE, and ARACIS representatives, debated the availability and accuracy of data collected to compute indicators and generate the classification of HEIs and ranking of study programs. ARACIS and MoE representatives 18 collected a set of comments and suggestions that were incorporated into the platform before initiating the data collection process. In April 2021, an online training session was organized to prepare the second phase of testing and piloting the methodology and the IT platform. Final Training Training sessions on the provisions and implementation of the developed methodologies are dedicated mainly to the MoE staff in the subordinated councils (CNFIS, CNCS, CNADTCU), ARACIS staff, and HEIs representatives. ARACIS subcontracted the WB to provide technical assistance services and support during the training, ensuring virtual participation of international lecturers to present recent developments and perspectives regarding the classification of HEIs and ranking of study programs, and quality assurance practices in the EHEA, as well as preparation of a training and dissemination report. The training would also address sustainable development, equal opportunities and non-discrimination, and gender equality topics. The trainees would receive a certificate of participation upon completing the training sessions and final evaluation. The training sessions are planned to take place between late May and June 2021, with around 225 participants. The QAFIN project representatives, Mrs. Cristina Icociu, and Mr. Iordan Petrescu, will open the training, presenting the scope, activities, and project results during the four years of implementation. The topics covered in training include the methodology for the classification of HEIs and ranking of study programs, the methodology for external assessment of HEIs and the associated guidelines, IT platforms, and equal opportunities, non-discrimination, and gender equality themes. The training agenda and presentations are enclosed in Annex 9 and Annex 10, respectively. The speakers and trainers: Cristina Icociu (Project Manager) Iordan Petrescu (President of ARACIS) Romina Miorelli (WB International Expert) Rafael Llavori-de Micheo (WB International Expert) Alina Sava (WB Education Specialist) Ștefan Stanciu (Local Expert) Alexandru Chiș (Local Expert) Mircea Neagoe (Local Expert) Mircea Popa (Local Expert) Luminița Moraru (Local Expert) Florin Mihai (Local Expert) 19 V. Communication and Dissemination Activities Communication. The purpose of a coherent communication procedure is to implement internal and external institutional communication, identify expectations, collect basic information, determine targets, develop content, and disseminate messages on ad hoc supports. It describes how the institutions define their communication policy and select various communication channels with partners and various stakeholders. Above all, communication is an action of sharing information between people having common interests. The effectiveness of results depends mainly on the quality of communication. Effective sharing of information promotes everyone's understanding of the institution's policy and objectives. Continuously improving communication with stakeholders, especially with HEIs, is a priority of MoE and ARACIS's activity. A special section on the ARACIS website is dedicated to projects implemented by the Agency, including the QAFIN project. Therefore, all project activities and results have been made available to the interested stakeholders. Information on the QAFIN project has also been made available on the MoE website. ARACIS website is a valuable instrument and a public communication channel that the Agency uses to make information available more quickly to a broader audience. Most importantly, ARACIS publishes on its website all significant results of the evaluation process, such as validation and evaluation reports endorsed by the Council. ARACIS also publishes working documents, methodologies, criteria, guidelines, overall higher education system and thematic analyses (e.g., the "Quality Barometer"), brochures relevant to stakeholders, and annual activity reports, including self-assessment reports. These documents are published in the Romanian language. ARACIS manages to translate the most important documents in English and publish both versions on the website. ARACIS publishes and disseminates the "Quality Assurance Review for Higher Education" (QAR). This academic journal focuses on QA in higher education, aiming to transfer and share good practices, support communication between internal and external experts, and develop a quality assurance culture. ARACIS distributes the journal to all HEIs. All ARACIS important news and press releases are also published under the News section of the ARACIS website. Dissemination. The primary objectives of the dissemination activities are to (a) effectively and efficiently raise awareness and understanding of the project, from concept to final outcomes, (b) to maximize the "footprint" of the project, and (c) to promote adoption of the final project outcomes. Moreover, the objectives include producing outreach materials published in various forms and formats, e.g., policy statements, manuals, guidelines, brochures, etc. The following audience groups are considered as target audiences for the dissemination of project activities: • Internal audience: The ARACIS and MoE staff need to stay well informed about all project activities and outcomes. Internal communication is vital to minimize resistance to change. This 20 is addressed by organizing regular meetings to review and analyze challenges and the implementation progress of activities. • External stakeholders: ARACIS and MoE regularly communicate with various stakeholders, from HEIs and students to national authorities (line ministries, National Institute of Statistics, etc.) and international institutions (ENQA, EQAR, peer agencies from other countries) to raise public awareness, reach consensus, receive scientific support, and ultimately, build trust. Other dissemination activities included: Informative sessions: • To introduce the purpose and the critical aspects of the survey dedicated to the ARACIS staff in June 2020. • To present and collect feedback on the IT platform developed to support the implementation of the methodology for external assessment of HEIs, sessions were organized with the ARACIS staff members in July 2020 and August 2020. • Other informative meetings to disseminate project activities and results. Conferences. The MoE organized the second round of public consultations on the methodology for the classification of HEIs and ranking of study programs April 11-13, April 17-18, May 30- Timisoara Iasi Bucharest during April and June 2019. Representatives of 2019 2019 June 1, MoE, ARACIS, WB, HEIs, and student organizations 2019 have participated in three consultation events in Bucharest, Iași, and Timișoara. The methodologies, standards, criteria, indicators, and platforms developed in the project were presented and frequently consulted with the National Council of Rectors. Meetings. Informative meetings of the WB team with the project management representatives, especially with Mr. Iordan Petrescu, President of ARACIS, Mrs. Cristina Icociu, Project Manager, Mrs. Cristina Ghițulică, Vice-President of ARACIS, and Mrs. Emilia Gogu, ARACIS Project Coordinator were held regularly to ensure smooth implementation of project activities. Publication of press releases on https://www.aracis.ro/comunicate-qafin/. Distribution of electronic and printed materials. ARACIS and MoE prepared and disseminated the project's outputs in electronic format through the ARACIS website. The final forms of the methodologies, guidelines, and procedures were printed as brochures and distributed to external stakeholders, including HEIs, student associations, etc. Examples of brochure covers are enclosed in Annex 11. 21 VI. Addressing the sustainable development, equal opportunities, non-discrimination, and gender equality aspects The 2030 Agenda for Sustainable Development 7 provides a blueprint for peace and prosperity for people and the planet, now and into the future. There are 17 Sustainable Development Goals (SDGs) that call for action by all UN member states in a global partnership. They recognize that ending poverty and other deprivations must go together with strategies that improve health and education, reduce inequality, and stimulate economic growth. Sustainability. Through its objectives, the project is a model of good practice, generating sustainability by enhancing the external evaluation system and creating a quality culture in higher education. The methodologies developed within the project – for classification of HEIs and ranking of study programs and for external assessment of HEIs – are expected to evolve and mature over time. Simultaneously, the project provides a starting point for several activities that will be explored in the future through these methodologies. The methodologies, guidelines, and procedure manuals will be used to implement quality assurance processes and classification and ranking exercises in the following years. Moreover, they could be used as transparency tools at the systemic level. The methodology for ranking the study programs constituted a conceptual effort that provided the prerequisites for expanding the ranking criteria and indicators in the future. The project also created instruments to gather data and build databases to perform future ranking exercises, develop transparency tools and quality policies. These databases will be used and supplemented in the long- term by decision-makers in the higher education sector, while the results will be used by relevant stakeholders, particularly students and employers. At the same time, ARACIS, using modular platforms, could ensure the transfer of data collected to other peer institutions (e.g., UEFISCDI, CNCS, CNATDCU, etc.) empowered to work with this data. The developed platforms will transfer knowledge to relevant actors in the higher education system, establishing a high-performance management approach. The methodologies and guidelines will serve as working instruments for professionals involved in public policymaking. The training and information sessions organized during the project's implementation period have been an essential resource for internal and external stakeholders, including HEIs representatives. Subsequently, they will disseminate the information at institutional, local, regional, and even sectoral levels. The experience of the project can be replicated to other levels, such as HEIs, related departments of institutions with responsibilities in higher education, etc. The project provides the expertise and instruments to ensure a coherent and integrated approach of all results, beyond its completion time, over a longer time horizon. 7 Transforming our world: the 2030 Agenda for Sustainable Development, United Nation, 2015. 22 Inclusive education. Sustainable Development Goal 4 ("Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all") refers to persons with disabilities. It calls for equal access to all levels of education and inclusive, accessible, and effective learning environments for all students, including those with disabilities. It also requires the design of education facilities to be responsive to the needs of students with disabilities. The Convention on the Rights of Persons with Disabilities requires states to ensure that persons with disabilities have access to inclusive education at all levels, including lifelong learning, equal to their peers without disabilities. The Convention on the Rights of the Child reinforces this position. The 2011 WHO–World Bank World Report on Disability discusses the available data on the educational participation gaps between children with and without disabilities, barriers to participation, and good practices in promoting an inclusive learning environment for students with disabilities. The World Development Report 2018, Learning to Realize Education's Promise (World Bank 2018), also offers guidance on tackling the learning crisis and promoting learning and skills development for all. The WB Inclusive Education Resource Guide 8 provides guidance on inclusive education for persons with disabilities, sexual and gender minorities, and special education needs. In addition, the guide also suggests considering any other minority groups (such as ethnic minorities, refugees, etc.) in the specific countries. From exclusion to inclusion and diversity. Exclusion occurs when students are directly or indirectly prevented from or denied access to education in any form. Moreover, when the education of students is provided in separate environments, in isolation from other students, segregation occurs. Inclusion involves a process of systemic reforms embodying changes and modifications in content, teaching methods, approaches, structures, and strategies in education to overcome barriers with a vision serving to provide all students with an equitable and participatory learning experience and environment that best corresponds to their expectations, requirements, and preferences. Diversity ensures the inclusion of students and strengthens their knowledge, skills, and behaviors while supporting and promoting the differences in society. Inclusive education refers to strengthening the capacity of the whole education system to reach out to all learners. Increasing access and ensuring learning and achievement for all students is critical to developing human capital and economic growth. National-level implementation requires enabling policy to articulate and support inclusive education, substantial data collection and management systems, flexibility in curriculum, and coordinating with other societal aspects, such as employment. The principles of inclusion and equity are about ensuring access to education and having quality learning spaces and pedagogies that enable students to thrive, understand their realities, and work for a more 8 Inclusive Education Resource Guide: Ensuring Inclusion and Equity in Education, World Bank, 2020. 23 just society. The QAFIN project tackled inclusion aspects in implementation, particularly in training activities and in developing the methodologies. In this respect, the methodology for ranking study programs considered the indicator – Graduation rate of students from disadvantaged categories, benefiting from scholarships from HEI's funds. The methodology for external assessment of HEIs considered three indicators: (i) HEI organizes tutoring activities and ensures the development of programs for remedial seminars, laboratories, and other didactic obligations for students with learning difficulties; (ii) Equal opportunities policy and measures; and (iii) Opportunities for student educational needs to pursue, interrupt, dropout, and resume its studies. The importance of these indicators has been confirmed during the public consultations. Gender equality. Sustainable Development Goal 5 refers to gender equality and empowering all women and girls. Gender equality is not only a fundamental human right but a necessary foundation for a peaceful, prosperous, and sustainable world. The Gender Equality Strategy 2020-2025 9 also considers gender equality a fundamental right that could bring more jobs and higher productivity, a potential that needs to be realized as we embrace the green and digital transitions and face up to our demographic challenges. Equal access to education and fair and high-quality educational processes are essential for gender equality and Europe's future economic prosperity. The Report on Gender Equality 10 looks at gender segregation in higher education, graduates of tertiary education, and participation in adult learning. The European Pillar of Social Rights emphasizes the importance of education and training and lifelong learning to ensure that women and men acquire and maintain the skills they need to participate fully in society and successfully manage transitions in the labor market. The Council recommendation on key competencies for lifelong learning also encourages the Member States to foster efforts to involve more women and men in lifelong learning activities. The same Report on Gender Equality 11 highlights that around half of research institutions in the EU have adopted a gender equality plan, representation of women in decision-making positions in research still shows room for improvement. Men account for 78 percent of heads of institutions, while boards of publicly funded research organizations have only 38 percent of women members. The QAFIN project implemented its activities to ensure that women and men can equally access project resources and services, participate in project activities and decision-making processes, and equally benefit from training and other capacity-building activities. 9 Striving for a Union of Equality, The Gender Equality Strategy 2020-2025, European Commission, 2020. 10 Gender Equality Index 2020: Digitalisation and the future of work, European Institute for Gender Equality, European Commission, 2019. 11 idem. 24 VII. Lessons learned and recommendations Over the past four years, the project outputs have led to knowledge transfer in key areas of higher education, such as quality assurance, classification of HEIs, and ranking of study programs. These gains have been achieved in a challenging environment, and progress toward institutional reforms is at risk due to substantial political volatility and turnover. Despite this turnover at the political level, the continuity of technical-level personnel across the World Bank and ARACIS teams allowed for crosscutting considerations on factors helping facilitate knowledge transfer and increase sustainability. Further support through coaching and mentoring is essential for enhancing capacity at HEIs level. A good project requires a mix of international experts to bring in good practices from other countries and local experts to understand the local context and support adaptive collaboration for delivering project outputs. This new approach to building capacity has been applied throughout the project implementation. ARACIS staff and experts were directly involved in working with the RAS team to develop methodologies, guidelines, and procedures. The World Bank team members participated in the expert groups and served as members and coaches to guide the collaborative process. ARACIS perceived this "learning by doing" process as a practical and innovative approach to maximize knowledge transfer. Further activities provided by ARACIS should consist of regular training, coaching, and mentoring program in order to enhance capacity of HEIs. Building a higher education ecosystem. Updating national frameworks, methodologies, guidelines, and procedures aligned with European standards should improve learning outcomes, resilience, inclusion, and sustainability of the higher education ecosystem. Creating ownership for results, public consultations, and a permanent strive for adopting modern instruments will increase Romania's quality culture. Quality also means improved equity and inclusion to increased access in higher education. Creating operational partnerships. In Romania, the WB team worked closely, consistently, and at the same level of expertise as counterparts. Fostering a positive climate and providing timely response and a consistent approach are keys for a new generation of projects. The WB team had a close collaboration with the QAFIN management team, particularly with ARACIS, an excellent and reliable partner - receiving regular, consistent, and constructive feedback, which has been instrumental in delivering high- quality reports and successful implementation of activities. Scale-up and replicating experience in European and Central Asia (ECA) countries. The technical expertise supported informed analysis, dialogue, and evidence-based decision-making by implementing instruments like methodologies, ROFs, etc. In addition, lessons learned during implementation identified innovations and solutions which can be scaled up or replicated to strengthen institutions in other ECA countries. ARACIS should continue to support the dialogue between Internal Quality Assurance (IQA) and External Quality Assurance (EQA). The proposed QA methodology has moved from an input-based approach to an output- and outcome-based approach, with indicators defined to enhance accountability and quality assurance in higher education. It aimed to focus on the mechanism and core framework to support ARACIS to develop and implement its external assessment activities in an increasingly demanding European context. The purpose of this "dialogue" is twofold: (i) to reinforce the IQA of HEI to assume more responsibilities in the management of their QA policy; and (ii) to focus the 25 ARACIS' efforts in a more enhancement-led accreditation procedure taking the HEI holistically as the independent object of analysis. Implementing QA of short educational packages (SEP) and the recognition of their (micro-) credentials or (micro-)qualifications. Universities, as lifelong learning institutions, should focus on (i) implementing robust methodologies, but also flexible and adaptive; (ii) fostering the consolidation of quality culture to encourage the diversity and dynamism of the system; (iii) enhancing permeability and learning pathways; and (iv) ensuring quality as motivation to learn and re-learn. The massive pressure on HEIs and the HE sector is holding on the impact on employability, particularly in a worldwide economic crisis and a post-pandemic situation. HEIs should commit to achieving better mechanisms to improve professional profiles to graduates by strengthening their relationships with relevant stakeholders. QA procedures, particularly program accreditation, should refine the guidelines directed to measure and merit successful performance indicators related to employability. Through the QAFIN project, the QA in Romanian HE System marked a significant step forward. ARACIS must enhance its role as a key active player in the current debate on QA trends within ENQA as well as in ECA. ARACIS and Romanian HEIs should agree on a shared international strategy on QA to promote the national IQA and EQA framework as a good practice at the regional level. A more active Romanian HE and QA sector will result in more balanced and diverse HE landscapes in the EHEA. 26 Annexes (included as a separate document) 27 Competence makes a difference! Project co-financed from the European Social Fund through the Administrative Capacity Operational Programme 2014-2020! 28