ROMANIA Advisory Services Agreement on Assistance to MNESR for Capacity Development for Monitoring and Evaluating the Implementation of Education Strategies OUTPUT 1 A UNITARY FRAMEWORK FOR MONITORING AND EVALUATING THE IMPLEMENTATION OF EDUCATION STRATEGIES November 2016 Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European This report is delivered under the Advisory Services Agreement on “Assistance to MNESR for Capacity Development for Monitoring and Evaluating the Implementation of Education Strategies” between the Ministry of National Education and Scientific Research and the International Bank for Reconstruction and Development, signed on June 29, 2016. It corresponds to Output 1 “M&E Unitary Framework” as set forth in the Schedule – Timetable under the above-mentioned agreement. This report was prepared by a World Bank team of experts including: Mariana Moarcas, Eliezer Orbach, Alina Sava, and Kevin Alan David Macdonald. The SABER EMIS Country Report, annexed to this report, was developed by Hussein Abdul-Hamid and Diana Mayrhofer with contributions from Valentin Bucur. The team would like to thank Cristian Aedo, Elisabetta Capannelli, and the Romania Portfolio Team for the continuous advice and support provided. The team would also like to thank the counterparts within the Ministry of National Education and Scientific Research, especially the Strategic Management and Public Policies Directorate led by Valentin Popescu, and Daniel Bojte, former EMIS coordinator, for their excellent collaboration. Disclaimer This report is a product of the International Bank for Reconstruction and Development/the World Bank. The findings, interpretation, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of the World Bank or the Governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. This report does not necessarily represent the position of the European Union or the Romanian Government. Copyright Statement The material in this publication is copyrighted. Copying and/or transmitting portions of this work without permission may be a violation of applicable laws. For permission to photocopy or reprint any part of this work, please send a request with the complete information to either: (i) the Ministry of National Education and Scientific Research (G-ral H.M. Berthelot Street No. 28-30, Bucharest, Romania) or (ii) the World Bank Group Romania (Vasile Lascăr Street, No 31, Et 6, Sector 2, Bucharest, Romania). Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 1 TABLE OF CONTENTS ABBREVIATIONS AND ACRONYMS ......................................................................................................................... 3 1. EXECUTIVE SUMMARY .................................................................................................................................. 4 2. CONTEXT AND BACKGROUND INFORMATION ............................................................................................... 7 3. THE OBJECTIVES AND FOCUSES OF THE UNITARY FRAMEWORK ................................................................... 9 4. A CONCEPTUAL OVERVIEW OF THE FUNCTIONS OF M&E ............................................................................ 11 3.1 WHAT IS THE DIFFERENCE BETWEEN ‘MONITORING’ AND ‘EVALUATION’ AND WHAT LINKS THEM TOGETHER? ............................ 11 3.2 WHAT ARE THE POTENTIAL USES OF THE INFORMATION OBTAINED IN M&E AND WHO ARE THE POTENTIAL USERS?.................... 14 3.3 WHAT ARE GOOD INDICATORS?................................................................................................................................. 16 5. BASELINE ASSESSMENT OF THE MNESR’S CAPACITY TO M&E THE IMPLEMENTATION OF STRATEGIES ....... 18 4.1 CURRENT DATA COLLECTION PROCESS IN THE MINISTRY ................................................................................................. 18 4.2 OTHER SOURCES OF INFORMATION, SURVEYS AND STUDIES AND THEIR SUITABILITY FOR M&E THE IMPLEMENTATION OF THE STRATEGIES ........................................................................................................................................................... 20 6. MANAGEMENT ISSUES AND TECHNICAL BOTTLENECKS............................................................................... 21 7. KEY ORGANIZATIONAL ARRANGEMENTS FOR THE DEVELOPMENT AND OPERATION OF THE M&E SYSTEM 23 6.1. WHAT KEY TASKS ARE INVOLVED IN THE OPERATION OF AN M&E SYSTEM? ....................................................................... 23 6.2. ORGANIZATIONAL ARRANGEMENTS FOR THE DEVELOPMENT OF THE M&E SYSTEM ............................................................. 25 6.3. ORGANIZATIONAL ARRANGEMENTS FOR THE OPERATION OF THE M&E SYSTEM ................................................................. 29 Annex SABER EMIS Country Report Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 2 ABBREVIATIONS AND ACRONYMS EC European Commission ECD Early Childhood Development ECE Early Childhood Education ECEC Early Childhood Education and Care EMIS Education Management Information System ESIF European Structural and Investment Funds EU European Union ESL Early School Leaving EEA European Economic Area EUR Euro IES Institute of Education Sciences IT Information Technology M&E Monitoring and Evaluation MNESR Ministry of National Education and Scientific Research MOPF Ministry of Public Finance NAQ National Authority for Qualifications NGO Non-Governmental Organization NIS National Institute of Statistics OECD Organization for Economic Co-operation and Development PISA Programme for International Student Assessment RAS Reimbursable Advisory Services RBM Results Based Management SABER Systems Approach for Better Education Results SIIIR Integrated IT System for Education in Romania SOS Secretary of State TE Tertiary Education TG Technical Group VET Vocational Education and Training WB World Bank Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 3 1. EXECUTIVE SUMMARY This document represents the first output under the Reimbursable Advisory Services (RAS) Agreement on Assistance to the Ministry of National Education and Scientific Research (MNESR) for Monitoring and Evaluating (M&E) the Implementation of Education Strategies aimed at building the capacity of the MNESR to (i) monitor and evaluate the implementation of the Strategy to Reduce Early School Leaving (ESL), Strategy for Increasing Tertiary Education (TE) Attainment, Quality and Efficiency, Strategy for Lifelong Learning (LLL) and Strategy for Vocational Education and Training (VET); and (ii) apply an evidence-based tool --the Systems Approach for Better Education Results (SABER) --to key policy areas with the purpose of fostering a structured policy dialogue with decision makers and stakeholders. The M&E activity to be undertaken by the MNESR is a huge endeavor given that each of the four high-level strategies consists of a distinct statement of direction for action and a complex set of strategic objectives and plans of action. In total, in the four high-level strategies, there are seventeen strategic objectives, encompassing a large range of measures and activities, and more than 200 indicators to be tracked. The Ministry’s challenge is to monitor and evaluate these strategies in a systematic, efficient and unitary way. To meet this challenge, all M&E activities need to be well- planned and organized given the multitude of players involved in implementation. The roles of the MNESR and its subordinated and coordinated bodies must be clearly defined, and the responsibilities implied by these roles must be assigned to specific organizational units and managers within each of them. The “Unitary Framework for M&E of the Implementation of the Strategies” was developed by the WB team to support the MNESR address the above-mentioned challenge now, but also to build capacity for M&E the implementation of future strategies through training and knowledge sharing. The proposed framework consists of the following sections: A conceptual overview of the functions of M&E (Section 3 in this document) – This overview clarifies the difference between ’monitoring’ and ‘evaluation’, in terms of their objectives, timing and the types of data they require. Monitoring is defined firstly as operationally-oriented, focusing on the conduct and control of operations, while “evaluation”, is defined as results-oriented, focusing on the outputs and outcomes of these operations. Monitoring is defined secondly as an on-going activity carried out throughout implementation, while evaluation is defined as an intermittent activity carried out when it is possible to see the results – always and at the end of action. The overview draws the Ministry’s attention to the reality that there are many potential users, of the information produced by M&E and that these users have different uses for it. Users include not only the managers and implementers of the strategies, but also the students and their parents, the international agencies that provide funds for implementation and the national agencies that oversee the MESR. Uses include not only the control of operations and the evaluation of results, but also the exercise of transparency and accountability. Finally, the overview also describes and illustrates the characteristics of good indicators, pointing at the type of data needed to produce them. Importantly, as part of the Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 4 development of action plans for 2017, the MNESR Technical Groups set up in the context of this M&E exercise are currently reviewing the indicators in each strategy, keeping in mind the distinctions between M&E, the different users and uses and the characteristics of good indicators. A baseline assessment of the MESR’s capacity to monitor and evaluate the implemen tation of its strategies (Section 4 in this document) – This assessment, undertaken by the Bank team, consisted of (i) a review of the use and collection of education data including their enabling factors using the Bank’s SABER EMIS1 tool; and (ii) a review of the suitability of existing data collection in Romania by other agencies (e.g.: household survey data, tracer studies, OECD PISA) to provide data for the indicators in the four strategies. The findings that are relevant to the M&E of the strategies, based on the application of the SABER EMIS tool,2 highlight the potential of the Integrated IT System for Education in Romania (SIIIR) in terms of coverage, access, usage, and dissemination, but also points out at data shortages (e.g. real time tracking and recording of student performance, teacher performance, professional development and trainings, etc.). The Bank team concluded that most of the data collected by other agencies is not suitable for the evaluation of the strategies and that the MNESR will have to collect it using tailor made instruments. A description of these instruments will be included in the upcoming work on methodology and instruments. Management issues and technical bottlenecks (Section 5 in this document) – This section focuses on three key factors that constrain the capacity of the MNESR to effectively monitor and evaluate the implementation of the strategies. these include the following: (i) the SIIIR is currently housed in a small organizational unit, the IT Office, with very small staff to service and maintain the system technically, and to address the large number of data issues that will arise while the four strategies are monitored and evaluated ; (ii) the staffing of the Strategic Management and Public Policies Directorate is insufficient to cover all the regular work in this unit and the work added with the launching of the four strategies and other strategies expected to be approved in the future. Both the IT Office and the Strategic Management and Public Policies Directorate need to have not only the right number of positions and staff to match their workloads, but also the right mixes of skill and expertise. For example, education statistics specialists or staff that could do or oversee advanced work (i.e. integration of data, relationship between multiple indicators or the projection of future trends) would be need in the near future. An additional constraint has to do with the financial capacity of the Institute of Education Sciences (IES) to carry out its evaluation role. The four strategies have many evaluation indicators requiring data that is not available from SIIIR (e.g. on Roma). The IES will have to develop and conduct tailor-made surveys for these indicators. However, while its staff can design the surveys as part of their regular jobs, they will not be able to undertake the data collection. This will require external data collectors and will incur related operational costs. What the Ministry can 1 EMIS = Education Management Information System 2 The full specialized SABER EMIS report is attached to this document. This report examines EMIS and its component policy domains much more broadly (beyond the M&E of the strategies) against global standards and best practices. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 5 do to overcome these constraints, is (a) to provide professional development and training to staff in the IT and the Strategic Management and Public Policies Directorate, (b) hire and retain in these two units additional staff who are highly-skilled, and/or sub-contract a private-sector agency to do some of the work, and (c) ensure an adequate operational budget for these units, as well as the IES. Key organizational arrangements that must be in place (Section 6 of this document) – the organizational arrangements for the development of the M&E system (which are different from the arrangements for its operation) are outlined in a formal Concept Note approved at Minister level in November 2015. The WB team worked with the Ministry to further specify and fine-tune these arrangements, clarifying the eight distinctive tasks that make up a working M&E system and specifying the responsibilities of all key players in carrying out these tasks. The tasks include data collection, data aggregation, data processing, data storage, data dissemination, IT maintenance and servicing, user education/promotion, and user support. The key organizational players and their responsibilities are summarized below: A Monitoring Committee – This committee, headed by a Secretary of State, has two roles: (a) to help the Secretary of State monitor and control the implementation of all strategies; and (b) to coordinate action among them. The Committee will review and resolve all implementation issues resulting from overlaps and inter-dependencies. A technical secretariat – The role of this secretariat is to provide administrative and technical support to the Secretary of State, the Monitoring Committee as well as the Technical Groups. It is housed in the Strategic Management and Public Policy Directorate. General directors – These directors are the ‘owners/managers’ of the strategies assigned to them. Their role is to manage the strategy’s implementation, as well as its monitoring. As the owners/managers of the strategies, each General Director reports to the Secretary of State and is accountable to him/her for the implementation of the strategy. Directors and/or senior experts - Since each of the four high-level strategies consists of many strategic objectives, measures, and operations, the general directors will allocate specific objectives and operations to specific directors/experts. Their role will be to manage, and be accountable for, these operations. Technical groups – For each strategy there is a technical group consisting of functionaries representing the Ministry and agencies involved in the implementation of the strategy. Each is headed by the general director in charge of the strategy, and its role is to advise him/her. The IES – The Institute is in charge of, and accountable for, evaluating all the strategies. It will have four evaluation teams headed each by a team manager who will report to the head of the Institute. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 6 A UNITARY FRAMEWORK FOR MONITORING AND EVALUATING THE IMPLEMENTATION OF EDUCATION STRATEGIES BY THE MINISTRY OF NATIONAL EDUCATION AND SCIENTIFIC RESEARCH 2. CONTEXT AND BACKGROUND INFORMATION 1. This document represents the first output under the Reimbursable Advisory Services (RAS) Agreement on Assistance to the Ministry of National Education and Scientific Research (MNESR) for Monitoring and Evaluating (M&E) the Implementation of Education Strategies. More specifically, the objective of the RAS is to build the capacity of the MNESR to (i) monitor and evaluate the implementation of strategies for reducing early school leaving, and improving tertiary education, life- long learning and vocational education and training, and (ii) apply an evidence-based tool--the Systems Approach for Better Education Results (SABER) --to key policy areas with the purpose of fostering a structured policy dialogue with decision makers and stakeholders. 2. The education sector is integral to the country’s efforts to achieve Europe 2020 targets. These include a reduction in the share of school levers to under 10 percent (Romania target: 11.3 percent; Romania status quo: very alarming 19.1 percent basically having increased every year by 1 percent since 2013); at least 40 percent of 30-40 year-olds having completed tertiary education (Romania target: 26.7 percent; Romania status quo: 25.6 percent), and at least 15 percent of the adult population (aged 25-64) having participated in lifelong learning activities (Romania target: 10. percent; Romania status quo: 1.3 percent). This also comes in the context of current demographic projections showing a decline in the number of students by 40 percent in 2025 and having immediate and long-term implications for Romania’s human capital and macroeconomic agenda. 3. In June and July 2015, the Government of Romania approved three education strategies representing ex-ante conditionalities for Romania’s access to European Funds under the Programming Period 2014-2020: Strategy to Reduce Early School Leaving (ESL), Strategy for Increasing Tertiary Education (TE) Attainment, Quality and Efficiency, and Strategy for Lifelong Learning (LLL). These three strategies were prepared by the MNESR with technical assistance from the World Bank during 2013-2014 and were subject to a consultation and internal Government processing through 2015. The fourth strategy on Vocational Education and Training (VET) was prepared by the MNESR separately and also approved by the Government in 2016. According to procedures requiring the endorsement of these strategies by the European Commission (EC) through the end of 2016, the EC has endorsed the strategies on Reducing ESL and TE earlier this year, while the strategies for Lifelong Learning and VET are in the process. 4. Funding is expected to be utilized under the European Structural and Investment Funds (ESIF), as well as from other EU (Erasmus+, EEA and Norway Grants), and national budget resources for the implementation of all of these strategies by a variety of public and private institutions, NGOs, etc. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 7 5. It is important to note that the implementation of the above mentioned strategies and utilization of European funds is not overlapping with the entire Programming Period 2014-2020; it needs to be actually considered from late 2015 and 2016, respectively, given that three strategies were approved in June-July 2015 and the fourth strategy (VET) was approved in August 2016. Also, resources from the ESIF have not yet been accessed; call for proposals have been launched in September-October 2016. Importantly, measures addressing ESL and VET have allocations amounting to EUR 200M and EUR 48 M, respectively. Project proposals are expected to be submitted in December 2016 and implementation will probably start in the first semester of 2017. 6. M&E are intrinsic management functions. The effective performance measurement of enterprises as large and complex as the education strategies is expected to be a process of M&E, learning and adjusting where necessary to improve the performance of initiatives leading to achieving the strategies’ objectives, EU2020 targets and the improvement of education in general. In this context, the M&E exercise will have to cover the implementation of measures under each strategy taking into account all financial resources (from EU funds, national budget, and other source, the diversity of stakeholders, and the timeframe. 7. In this context, at the time of the preparation of the above mentioned strategies, a Results Based Management (RBM) approach was proposed by the WB for M&E as a model most commonly used by OECD countries and the EU Commission to assess the effectiveness and efficiency of public sector policies, programme and projects. The RBM approach was adopted by all above mentioned strategies since it is not simply used for M&E – it is a full management cycle: (i) it is a design tool that permits policy, programme and project designers to spell out clearly why the initiative is needed, what it is expected to accomplish and how it will recognize its success; (ii) it is an implementation process tool that gives managers access to regular performance data allowing them to adjust activities and allocate resources for optimal results achievement; and (iii) it is an M&E mechanism that provides evidence of progress and, through effective analysis of performance data, draws important lessons for future targeted policies, programs and projects. 8. However, the M&E activity is a huge endeavor given that each of the four strategies is a complex document with its own specificity in terms of vision and strategic objectives/thrusts to be evaluated. There are in total in the four strategies seventeen (17) strategic objectives encompassing an impressive range of measures, actions, and indicators. For example, there are 59, 46, 40 and 77 indicators under the measures proposed in the strategies to Reduce ESL, improve TE, LLL and VET, respectively, to be monitored through the M&E exercise. 9. Therefore, the challenge faced by MNESR is to undertake the M&E of the implementation of the interventions promoted in the strategies in a coordinated and unitary way with clear roles and responsibilities established for the MNESR, its subordinated and coordinated bodies so as to ensure the relevant targets assumed by Romania in these documents, and ultimately the commitments under the EU 2020 are timely met. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 8 10. As a first step, the MNESR has established an outline for the coordination arrangement in view of the M&E of the implementation of the strategies (approved by the minister of education on November 12, 2015) and including: (i) technical groups within the MNESR (one for each strategy) responsible for the monitoring activities; (ii) a Monitoring Committee expected to oversee the M&E process for all strategies headed by the MNESR secretary of state and including high level representatives of other relevant institutions; and a Technical Secretariat (represented by the Strategic Management and Public Policies Directorate) assigned to assist the Monitoring Committee; and (iii) an evaluation group responsible for the analysis and evaluation of the strategies. 11. The above mentioned coordination arrangement has been recently activated at the level of technical groups, while the Monitoring Committee is still to be formalized by the MNESR. 12. The WB’s role is to provide technical assistance to enhance the capacity of the MNESR in the context of the M&E of the implementation of the above mentioned strategies, while simultaneously building up the ability of MNESR staff to carry out such activities on their own in the future through knowledge transfer activities. 13. Therefore, according to the RAS Agreement, the WB team developed this Unitary Framework for M&E of the implementation of the strategies with the main objective to clarify the roles and responsibilities of the MESR, its subordinated and coordinated entities, as well as the reporting and working relationships, the interactions with all relevant stakeholders. As part of this work, a baseline assessment of the capacity of the MNESR to M&E this process was also undertaken through (i) a review of the the use and collection of primary and secondary level education data using the World Bank’s SABER EMIS; and (ii) a review of the suitability of existing data collection in Romania (e.g.: household survey data, tracer studies, OECD PISA) to provide measures of indicators for the strategies. 14. A series of workshops and coordination meetings were organized in September and October 2016 with all four Technical Groups set up by the MNESR, as well as with the Strategic Management and Public Policies Directorate, and the Institute of Education Sciences. Such workshops also represented training opportunities for the staff of the MNESR and its coordinated and subordinated bodies. Detailed discussions were held on the main topics covered in this document. In addition, a dedicated workshop for the above mentioned SABER EMIS activity is currently being planned for the next period. 3. THE OBJECTIVES AND FOCUSES OF THE UNITARY FRAMEWORK 15. Broadly speaking, the goal of the unitary framework is to facilitate the monitoring and evaluation (M&E) of four key national strategies implemented by the MNESR (also referred to in this document as “the Ministry”) with funding from the European Union as well as other resources including from the national budget. As already mentioned, the strategies are in the areas of ESL, LLL, Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 9 TE, and VET. The framework is expected, however, to facilitate the monitoring and evaluation of other future strategies as well. 16. More specifically, the proposed unitary framework has the following four main objectives: Objective 1: Provide a conceptual overview of the functions of M&E, propose a unitary approach for M&E in the MNESR and help the Ministry apply the key points in this approach to the four strategies. In addressing this objective, the report focuses on the following questions: (a) What is the difference between monitoring and evaluation and what links them together? (b) What are the potential uses of the information obtained in M&E and who are the potential users? (c) What are good indicators? Objective 2: Provide a baseline assessment of the current monitoring and evaluation activities in the Ministry, including particularly the extent to which data that is already being collected, both by the Ministry and by other agencies, is suitable to produce the output and outcome indicators proposed in each of the four strategies. In addressing this objective, the report focuses on the following questions: (a) Is the primary-level and secondary-level data that is currently being collected by the Ministry suitable for the monitoring of the four strategies, and, if necessary, the development of additional indicators that can effectively reflect the implementation progress, as well as the success, of the four strategies? To answer this question, the report team used the World Bank’s SABER EMIS instrument. (b) Are there in Romania other sources of information that can be used, either alone or in combination with information available in the Ministry, to produce some of the needed indicators? To answer this question, the report team reviewed existing household surveys, tracer studies, OECD surveys and PISA tests. Objective 3: Identify management issues and technical bottlenecks that constrain the capacity of the Ministry to effectively monitor and evaluate the implementation of strategies, and suggest measures that can be taken by the Ministry in the coming three years to build this capacity. In addressing this objective, the report focuses on the question whether there are management issues and/or technical bottlenecks that constrain the capacity of the Ministry to effectively monitor and evaluate the implementation of the four strategies, and, if yes, what does the Ministry need to do right away to address them. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 10 Objective 4: Specify key organizational arrangements that must be in place for the development and, later, operation of the M&E system. In addressing this objective, the report focuses on the following questions: (a) What key tasks are involved in the operation of an M&E system and who should play, what role, in these tasks; and (b) How best is it to organize for the development and operation of the system? The following sections of this report detail the unitary framework by addressing the above mentioned objectives and clarifying questions. 4. A CONCEPTUAL OVERVIEW OF THE FUNCTIONS OF M&E 3.1 What is the difference between ‘monitoring’ and ‘evaluation’ and what links them together? 17. ‘Monitoring’ is operationally-oriented. In relation to strategy, it is defined in this report as the systematic observation of the strategy’s implementation and the recording of data on it for the purpose of control. The idea behind monitoring is to check that each strategy is carried out as planned – i.e., that all inputs needed for implementation are provided, all operations and activities are carried out and all required outputs are produced – all on time, at the anticipated cost, and as per the designed specifications. The idea behind control, is that if these are not, if the data suggests that certain things are not happening as planned, e.g., promised resources are not being delivered, or promised activities are not being carried out, then corrective action must be taken. This corrective action focuses on improving implementation measures, ensuring that the resources will be delivered and the activities will take place, but it does not focus on changing the strategy itself. 18. Evaluation, on the other hand, is results-oriented. It is defined in this context as the systematic observation and recording of the results of the strategy, for the purpose of verification. The idea is to check that the actual results of the strategy – both the immediate results and the longer- term result – match the intended results, i.e., that the objectives of the strategy are being achieved and that there are no unintended consequences. Like monitoring, evaluation too is often followed by corrective action, but in this case the corrective action focuses typically on modifying, changing or replacing the strategy itself, because the data collected and analyzed suggests that, while the strategy is implemented reasonably well, the intended results are not being achieved, i.e., the strategy is not working, or while working, it is too costly, or is also leading to too serious unintended consequences. 19. When applied to strategies, both monitoring and evaluation deal with three broad categories of objects: the inputs that go into the implementation of the strategies, the operations that make up the strategies, and the results of these operations. The results fall into three sub-categories: immediate results, or ‘outputs’ that are visible at end of action, short to medium-term results that are results of the results, i.e., of the outputs, and are typically called ‘outcomes’ and then longer-term, often broader, Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 11 results that are referred to as ‘impact’. Monitoring that is carried out for operational control, focuses on data related to inputs, operations and outputs. Monitoring that is carried out for evaluation focuses on outcomes and impact. The difference between the two is, however, not just in the types of data that needs to be collected. It is also in the types of questions that are being asked. Below is a highly- simplified systems’-based diagram of a primary and lower secondary education system. It represents the identical types of objects dealt with both in monitoring and in evaluation. It is followed by a table containing examples of the very different types of questions focused on in monitoring and in evaluation: Figure 1: What do Monitoring and Evaluation Focus on Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 12 Table 1. Types of Questions Asked in Monitoring and in Evaluation Object Monitoring Evaluation Inputs Are all students expected to enter If not, and in retrospect, what were the reasons? primary + lower secondary schools indeed entering the system? Are all financial, human and other In retrospect, have the mix, quality and quantity resources that were planned and of resources of all types allocated to the budgeted for the implementation of this implementation of this strategy been right? Was strategy being delivered to the primary the mix optimal? Was the delivery of the and lower secondary schools? As resources optimal? In other words, could have specified? On time? more/better outputs and outcomes been achieved, had the mix, the quality, the quantity, or the timing of delivery been planned differently? Are funds related to the In retrospect, have mechanisms for the implementation of this strategy being disbursement of funds slowed procurement and disbursed in line with the disbursement implementation down? Were there other factors plans? at play? Activities/ Are all activities that were planned for In retrospect, were the planned pace of Operations this strategy being carried out? As implementation, and the sequence activities, specified? On time? right? Did the pace and the sequence impact the results of the strategy? Did the actual pace of implementation result in cost escalation or had any other unintended consequences? Are all resources made available for the In retrospect, were all resources fully and implementation of this strategy being efficiently utilized? Could they be better utilized fully as planned and utilized? What factors impacted the utilization efficiently? and efficiency of the resources? Output & How many of those who entered lower How many of the student who graduated from Outcomes primary schools have completed it and lower secondary have entered high schools? moved to lower secondary schools? How well are they doing in high schools? How many of the entrants to lower In retrospect, have the number of graduates who secondary schools graduated from entered high schools as expected? them? Have they been performing as expected? Strategy as In retrospect, was the strategy financially, a Whole technically, logistically well designed? What were the benefits of the strategy relative to its costs? Has the strategy been realistic, feasible and cost effective? Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 13 20. Finally, the operational monitoring of strategies takes place from day one throughout the implementation period, because of the need to control action, while the evaluation of strategies takes place later, when it is possible to see results. Evaluation is often summative in nature, coming at the end of all operations that were part of a strategy. However, it is possible, and often desirable, to have also formative evaluation, which is an assessment of early signals of outcomes and impact, while the strategy-related operations are still being carried out. Such an assessment may help refine strategic action and, in some cases, may lead to significant changes in strategies while they are being implemented. 3.2 What are the potential uses of the information obtained in M&E and who are the potential users? 21. Before designing an M&E system for any strategy, it is important to clarify and decide what are the intended uses, and who are the intended users, of the information that is to be produced by the system, because the specific design of the system depends on both. The intended uses influence decisions on what needs to be monitored, i.e., what indicators need to be developed and what data needs to be collected for the development of these indicators. The uses also influence decisions on the frequency of data collection, as well as the types of analyses that will be carried out on the data. The users, on the other hand, influence decisions on how will the information be presented, how will it be disseminated, where will it be located and who will have what type of access to it. 22. Below is a general list of potential uses and users: (a) To provide direction for action – The most important tasks of organizational leaders are to provide direction for action to the organizations they lead and to mobilize for the organizations the resources with which to go in the given direction. Effective M&E systems are typically designed to provide organizational leaders the information – the data and analyses – that they will need in the future to review their organizations’ performance, to carry out the required situation analyses, assess progress and success, determine priorities and then formulate, or re-formulate the desirable direction. The tools that leaders use for this purpose are visions, policies, strategies and high-level strategic plans. If the information required for the development, review and updating of these tools is identified ahead of time, then it can be built into the M&E system, and if leaders then use it, then their visions, policies, strategies and plans will be much more effective, realistic and feasible than otherwise. Moreover, the information will help them not only validate a need for change when it arises, but also convince people that the change is needed. They will be able to better explain and justify strategic change not only to their staff internally, but also to their clients (e.g., parents) and to other stakeholders (e.g., the public) externally. (b) To help mobilize resources – As stated above, the mobilization of resources is a second most important task of organizational leaders. It makes little sense for organizational leaders and top managers to provide their staff direction for action if the resources to take the action are not there. M&E systems can be designed to provide the data and analyses needed by leaders/managers to Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 14 present convincing cases to the providers of funds. These can be cases to justify their budget request to government, or aid requests to donor agencies. (c) To help allocate mobilized resources internally – Closely linked to the mobilization of resources is the internal allocation of resources once mobilized. This is yet another challenging task for leaders as well as top and lower-level managers under them – in this case a triple- challenge task: to link resource allocation well to policy and strategy priorities, then to link it to clearly quantified operational requirements and then to convince staff, as well as often internal and external stakeholders, that the allocation is justified. Much of the data they need to do this well can be built into their M&E systems and then provided to them. (d) To manage and control daily operations – M&E systems can be designed to help managers manage specific plans, projects and programs and to exercise accountability. The information in these systems can be used to help develop better, clearer, more realistic operational plans in the first place – whether these are general periodical plans, budget-related plans or project-related plans – and then it can be used to monitor the implementation of these plans, focusing on matters such as those presented in the previous section: the delivery of inputs, the progress of planned activities, the utilization of all types of resources in these activities, the efficiency of the resources when they are utilized, the outputs produced and delivered, and (where possible and relevant) the outcomes, as well as the cost at which these outcomes are achieved. (e) To provide external parties information for review – Parties such as parliament and the Ministry of Public Finance (MOPF), as well as international aid agencies, including the EC and the WB, are all users of information. They require the Ministry to report to them and they use the information firstly to see if the Ministry is moving in the ‘right’ direction and how well; then they use it to assess and decide on the Ministry’s budget and aid requests and, finally, they use it also to monitor the performance of their funds by the Ministry. M&E systems can be designed to ensure that the data and analyses required to meet the needs of these multiple users is collected and analyzed proactively. (f) To enhance transparency to clients and the public, and improve communication with them – Clients, i.e., the students and their parents, are major users of information on what is taking place in the education system. The public, represented not only by elected officials in legislative bodies, but also by the press, is increasingly inquisitive about the system. Both are demanding higher levels of transparency to better understand what is going on. M&E systems need to be designed to meet these demands. Data can be collected and analyzed with the objective of being ready to provide the information to these clients and stakeholders not only reactively on demand, but in some cases, also proactively in advance of such demands to help develop and maintain effective communication and public relations. (g) To provide internal and external bodies information for research purposes – Finally, most organizations have pending policy and strategy issues that need to be dealt with. Often, these are just emerging, but are not yet evident. They are likely to come to the fore in some time and therefore are not demanding immediate attention. In well-managed organizations, forward-looking managers Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 15 often identify such policy and strategy issues and create active agendas for them. They charge internal policy units with the job of searching for answers before these issues come to the front. Often, the issues require serious research work, including the observation of long-term trends in the demands facing them, in the resources available to them and in the options that will be open to them. Where such research is undertaken, M&E systems can be designed to collect valuable data and produce key information that will be needed by the researchers. 23. As can be seen from the discussion so far, the combined number of potential users and uses that can be served by an M&E system is very large. It is important to keep in mind, however, that the larger the numbers selected for inclusion in an M&E system, the more complex and the costlier the system becomes. The M&E system to be developed by the Ministry for the four strategies is no exception. Each of the strategies already specifies many indicators and many data to be monitored. Thus, adding users and uses can become a complex and costly exercise. It is important, therefore, for the Ministry to carefully consider the potential users and uses of information for each of the four strategies, prioritize them, assess the potential costs and benefits of accommodating each one in the system, and then decide how to proceed. Not all users and uses need to be added at once; some can be added over time, as needed. 3.3 What are good indicators? 24. In the context of strategy implementation, indicators are observable and measurable descriptors of the state, as well as the results, of implementation. They focus on the factors discussed earlier: the inputs, activities and processes, outputs, outcomes and impact. Below is a table with examples of such indicators relating to three activities found in the VET and ESL strategies. Not all these indicators are included in the two strategies. Some were formulated by the report author to illustrate the different types of indicators: Table 2: Example of Types of Indicators Input/Resources Process/Activity Output: Outcome: Impact: Immediate following result Ultimate Result Result Funds for training Training courses No. of teachers No of students Labor market of teachers to being delivered trained as who received insertion rate of provide career career career guidance students (15-24 guidance to counselors years) receiving students career counseling Financial No. of activities Number of No. of children Allocation efficiency incentive engaging parents parents school ready of available provided to who received engaged in enrolled in first resources (cost per parents to help incentives in ECEC grade with child) engage them and ECEC related activities; no. previous ECE increase activities of parents Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 16 Input/Resources Process/Activity Output: Outcome: Impact: Immediate following result Ultimate Result Result enrolment in receiving the experience Attitude change ECEC incentive versus without toward childcare and Percentage of development first graders Impact of ECD who have program on received a children’s form of ECD development program Impact of preschool on child adaptation and performance in the first grade. Funds for the Early warning Number of No of children Changes in the work development of system is being children at risk at risk treated career trajectory of early warning developed and identified and who have not children at risk who system for introduced treated left school were identified, children at risk of Decrease in treated and leaving school dropout rate and completed school early school leaving rate 25. The most common characteristics of good indicators include the following: (a) They are clear and specific; their meaning is not subject to interpretation. Words such as ‘improved’ or ‘increased’ need to be accompanied by references to numbers or rates. (b) They can be measured precisely. They state clearly not only what is being measured, but also what specific data needs to be collected for the measurement. (c) They are valid, i.e. they adequately represent what they say they represent. This is particularly important in the context of outcomes’ or results’ measurement: the results they measure can be fully, or largely, attributed to the action that preceded them. (d) They are reliable, i.e., they measure consistently across time and across different data collectors. If two people measure the same indicator using the same tool, they would get the same result. (e) They are time-bound, i.e. there is a clear indication of the time limit within which action needs to take place and results are expected to be visible and measurable. (f) They are practical and feasible, i.e., the data needed for their production is available and is not too burdensome or too expensive to collect. The total number of indicators, even those that are based on easy-to-collect data can turn a feasible M&E system into an infeasible one. In many cases, there is a need to limit the number of indicators. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 17 (g) There is full clarity as to who is responsible for each of the tasks involved in M&E (which will be discussed next): collecting, aggregating, storing, analyzing and disseminating the necessary data for their production. (h) There is also full clarity on the frequency and times by when they must be delivered and to whom they must be delivered. 26. The Ministry is now reviewing the indicators in each of its four strategies with the aim of ensuring that they are in line as much as possible with these eight characteristics. 5. BASELINE ASSESSMENT OF THE MNESR’S CAPACITY TO M&E THE IMPLEMENTATION OF STRATEGIES 27. There are in Romania currently three major sources of education data: The MNESR, the MOPF and the National Institute of Statistics (NIS). The Ministry collects data relevant to schools, teachers and students, while the MOPF collects financial data. Both share the data to the NIS for statistics purposes. But their two databases are not linked. The MOPF collects data through its own mechanisms at a very detailed (budgetary) level. However, based on an agreement between the Ministry and the NIS, which is currently being finalized, the Ministry will soon become the sole provider of the data needed for official statistics on education, with responsibility for the collection, validation, processing, storage and dissemination. The Ministry already has a highly developed Education Management Information System called the Integrated IT System for Education in Romania or SIIIR in which all data will be integrated. 28. A baseline assessment of the MESR’s capacity to monitor and evaluate the implementation of its strategies was performed by the Bank team and consists of (i) a review of the use and collection of primary and secondary level education data including their enabling factors using the World Bank’s SABER EMIS tool; and (ii) a review of the suitability of existing data collection in Romania by other agencies (e.g.: household survey data, tracer studies, OECD PISA) to provide measures of indicators for the strategies. 4.1 Current data collection process in the Ministry 29. To review the current M&E activities in the Ministry, the team used the above mentioned SABER tool, developed by the Bank to support countries in systematically examining and strengthening the performance of their education systems. More specifically, the application of the SABER EMIS tool can identify important bottlenecks at both management and technical levels that could hinder the MNESR’s ability to monitor and evaluate its education strategies; as an externality, it also provides an important assessment for the use of data in education system management more broadly. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 18 30. The SABER EMIS tool was used to examine EMIS and its component policy domains against global standards and best practices and is focused on four key areas: (a) the enabling environment, which refers to the laws, policies, structure, resources, and culture that surround SIIIR, (b) the soundness of the key processes, structures and integration capabilities in the SIIIR, (c) the coverage and quality of the data, looking at the mechanisms required to collect, save, produce, and utilize the information in SIIIR in an accurate, secure, and timely manner, and (d) the utilization of the data in SIIIR in decision making by all users (parents, students, teachers, principals and policy makers) across the education system. The SABER methodology consists of a review of written policies and technical documents as well as interviews with key stakeholders across the education system. The results of applying the SABER tool in Romania are discussed in a specialized SABER EMIS report and presented together with this document as a background report. Below is a summary of the findings that are relevant to the monitoring and evaluation of the four strategies:  SIIIR covers data on students, teachers, school facilities, public financing, and national assessments; however, the coverage of each category is not full.  Student level data includes basic demographics such as age and gender as well as data on socio-economic status and ethnicity. Data on student transition rates into higher education is also recorded. Data on learning and physical disabilities too is collected, but no other data on the health of students is collected. There is no real-time procedure to monitor, track and record student learning.  Teacher data is not fully covered. Data on teacher salaries, experience and qualifications is covered, but there is no real-time procedure to monitor and record teacher performance and there is no data on professional development and trainings.  SIIIR does not yet collect data on grades obtained by students (Electronic National Grade- Book System) and on attendance of teachers and students.  A wide range of data on school facilities and infrastructure (e.g. furniture, internet connectivity, IT equipment, utilities, permits, etc.) are collected.  Data on the implementation school development plans is not collected.  Financial data is collected only on government spending; data on private expenditure including tuition, grants, donations and other school revenue are not collected.  Data on disadvantaged social groups such as Roma is not collected. Given that 95 percent of Roma leave school early, this is important data and needs to be accounted for in the system architecture.  SIIIR has capacity to compute and provide descriptive analysis, such as means and totals, cross tabulations to summarize indicators across different values of other indicators. But, because it tracks schools, teachers and students simultaneously, it can identify potential correlations between student learning and school or teacher performance. Currently, however, there are no processes to analyze associations or correlations between indicators.  SIIIR is methodologically sound with well documented and defined structure and concepts adhering to international standards and classifications.  All schools and school inspectorates have access to SIIIR. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 19  Data is provided by SIIIR internal users timely in high frequency. Data collection from schools is conducted at least twice a year. Final education statistics are disseminated within 6 to 12 months of the start of the year. The periodicity and timeliness of data appears to be sufficient for the needs of data users.  Ministry managers and staff get access to some data, in some cases directly on line and in other cases on demand, but the available data is not used widely by them.  Data is disseminated to the public only in the form of official reports and national statistics; the public has direct access only to what the Ministry puts on its website. Parents can access some data online, mainly regarding school infrastructure, staffing, facilities and national examinations on an individual level. Generally, education stakeholders need to actively look for education data, because it is not automatically presented to them by the Ministry. Often the data available is in aggregate terms. Sharing SIIIR data with other ministries or organizations takes place only with bilateral agreements. 4.2 Other sources of information, surveys and studies and their suitability for M&E the implementation of the strategies 31. Romania routinely conducts several national household surveys and participates in the OECD PISA student assessment survey. The household surveys provide a potential measure for the education strategies’ population indicators—those indicators that measure outcomes for individuals outside schools including potential students and graduates. The PISA study provides measurement of student learning outcomes benchmarked to the OECD. The team reviewed available surveys to check (i) whether the household surveys already do or could collect data relevant for measuring the education strategies’ population indicators, (ii) whether the PISA data provides measures of the education strategies’ indicators, and (iii) whether these surveys provide an accurate enough measure of whether targets are achieved or not, given their sampling methodology including their sample size. 32. Of the many monitoring indicators included in the four sector strategies, only 17 are population indicators; the rest refer to institutions (including schools) or their students and staff. The most relevant household surveys for providing data for the strategies’ population indicators are the Romania Household Budget Survey (HBS: Ancheta bugetelor de familie) and the Romania Labour Force Survey (LFS). Other surveys are also conducted in Romania; however, these are either not conducted regularly, owned by other organizations, or not relevant to the population indicators of the strategies. The review reveals that both surveys do—or with modifications to the questionnaires, could—provide measures of the strategies’ population indicators. However, the HBS would not provide accurate enough estimates to be a suitable data-source for most indicators. 33. Population indicators that measure participation rates can be estimated without household surveys by using enrolment data and population size projection. The resulting data gap for the strategies’ population indicators would be employment outcomes of graduates. This data gap could Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 20 be addressed by the LFS if the sample sizes are large enough—a similar analysis as conducted for the HBS would be required—and if not, then tracer studies would be required. 34. Three of the Early School Leaving indicators refer to learning achievement, particularly the percent of students achieving a certain standard of literacy and mathematics. The OECD PISA study provides data on the percentage of students achieving different levels of proficiency according to the OECD’s benchmarks. However, because the sample size is relatively small for this study, the targets would have to be achieved by a large margin in order for there to be a low probability of drawing a sample which would yield a false conclusion. As a result, the OECD PISA study, while providing an international benchmark for learning achievement, may not be suitable for monitoring the strategy’s progress. 35. The bottom line is that most of the data needed for the evaluation of the four strategies will have to be collected by the Ministry/Institute of Educational Sciences (IES) using tailored made instruments. Such instruments will be associated to the methodology for M&E the education strategies. 6. MANAGEMENT ISSUES AND TECHNICAL BOTTLENECKS 36. SIIIR is currently housed in a small organizational unit consisting of four staff members who are mostly technically-oriented IT operators. Based on the information received form the MNESR, two are IT teachers who were deployed from their schools and who are not familiar with SIIIR. The other two are more familiar with how to operate the SIIIR, how to store and process data in it and how to get pre-designed and pre-programmed output from it; apparently, none are high-level IT specialists and none has experience in education systems management. They do not have the capacity to service and maintain the system technically – a job that is contracted out to a private sector company – and their acquaintance with what needs to be done, and what can be done, from an education management point of view is limited. In this context, they are not in a position to serve the M&E function effectively. This presents a big challenge to the implementation and management of the M&E system. It is important to note, however, that in August 2016, a draft Government Decision was prepared to address this issue and introduce a separate Unit/Agency for IT within the MNESR and discussions on this proposal are ongoing. 37. Another big challenge to the M&E system in the Ministry is the staffing situation. This relates to the monitoring and evaluation of all the Ministry’s operations and not just the four strategies. Both the IT Office the Strategic Management and Public Policy Directorate lack staff to do all the regular work in their programs and the work added now to their work programs with the launching of the four strategies. The IT Office lacks staff with a higher level of data-base management and maintenance expertise to do more complex work as well as to judge whether the work done by contractors is of the needed quality. The Strategic Management and Public Policy Directorate has slightly increased in staff, but the increase is not in proportion to the increase in work that came with the launching of the four strategies (and other strategies that will be launched in the future). The only Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 21 M&E tasks for which there is enough staff are data collection and aggregation in the field, since these tasks are being carried out by the schools with the support of school inspectorates. But there are not enough staff in the Ministry to process data in the data base, to maintain the data base, to analyze the data and then disseminate it to potential users either on demand or proactively. 38. The issue is not just a matter of having the right number of positions in the IT Office or in the Strategic Management and Public Policy Directorate. It is also a matter of having the right types of positions for people with the right expertise. Both the IT Office and the Strategic Management and Public Policy Directorate lack staff to do, or to oversee more advanced work, such as, for example, the integration of data or the analysis of relationship between multiple indicators or the projection of future trends. There is in the Ministry no trained statistician with expertise in statistics generally, or in education statistics specifically. The IT Office is not staffed with quality assurance specialists, communications specialists, or data analysts, all of whom are crucial for a well-operating EMIS. 39. Two challenges remain even if the number of positions increases to match operational requirements: the first is the difficulty to get the skilled and highly-skilled people that are needed in a market where people with these skills can command in the private sector salaries that are much higher than what they can get in the civil or public service. The Ministry may have to recruit people with lower levels of skill and train them, but currently it does not have institutionalized professional development and training plans for its IT and M&E staff, and this should change. All the highly- skilled staff that have been working so far on EMIS/SIIIR were financed mostly under projects funded from European funds. As is stated in the SABER EMIS report, “The hiring and retention of highly qualified and skilled human resources remains a substantial challenge and a threat to the sustainability of the EMIS.” 40. The second challenge is to ensure an adequate operational budget for IT and M&E operations. With the addition of the four strategies, the monitoring as well as the evaluation operations that will need to be carried out have greatly increased in number and complexity. Particularly in evaluation, many indicators will require the collection of data that is not currently collected by SIIIR. To take but one example, in the four strategies there are outcome indicators that focus on socially disadvantaged groups, such as Roma, but SIIIR currently does not collect data on them. Also, many research staff in IES will be involved in the evaluation, but even if they devote most of their time to it, they will not be able to carry out all the necessary data collection for representative samples in the field. They will need to hire data collectors and will require larger operating budgets than they currently have. Budget requirements may increase also due to the need to replace paper-based data collection instruments with digital instruments. It is highly likely that much data for the monitoring of the four strategies, which is not yet digitally collected, will have to start being collected digitally. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 22 7. KEY ORGANIZATIONAL ARRANGEMENTS FOR THE DEVELOPMENT AND OPERATION OF THE M&E SYSTEM 41. The organizational arrangements for the development of the M&E system are different to the organizational arrangements for its operation. An outline of the arrangements for the development of the system has already been formulated and formalized through a Minister’s approval (in November 2015) and is now being put into place. It is presented and elaborated on below and is then followed by a description of some proposed key arrangements for the operation of the system. 6.1. What key tasks are involved in the operation of an M&E system? 42. There are eight (8) key tasks that must be carried out to keep an M&E system running effectively, once the system is designed and developed. The tasks are as follows: (a) Data Collection and transmission – This task consists of the gathering of primary data at its origin. In education systems, individual children, parents, teachers and schools are all sources of primary data. Data collection refers to the gathering of the data from them, before the data is aggregated. Collection methods vary, depending on factors such as the type of data being collected, the frequency of its collection and the available collection technology. Typically, functionaries at school are the ones who collect the data and send it onwards to inspectorate offices either on paper or digitally. The most recent technologies that are being introduced, or are likely to be soon introduced, into the collection of data for monitoring in different education systems are computer-based wide area networks and hand-held digital transmission devices that send primary data from the individual persons or facilities on site directly into local, and/or regional and/or national data bases. (b) Data Aggregation – This task consists of adding up primary data collected from individuals and facilities on site, at origin, into larger groupings as needed. This includes, for example, the aggregation of data collected from individual students into school-based data, the aggregation of individual school data into a school district data base and the aggregation of data from individual districts into regional and/or national data bases. With digital transmission, data aggregation can become an automatic task once it is designed and built into an M&E system. M&E systems often also receive and use data that is coming from external agencies, such as a National Institute of Statistics, in which case the data has been collected, aggregated and often processed before it reaches the system. (c) Data Processing and transformation – This task consists of the turning of raw data into useful information – producing descriptive and analytic statistics and indicators that are intended for use in the policy and strategy-related work, as well as the operational work. M&E systems are designed to do at least some of this automatically, but much is done by them on demand. Various digital and non-digital tools are deployed for these purposes. All systems are designed to do this Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 23 on demand. Data received from external sources cannot be used automatically in M&E systems, because in most cases the different digital systems do not communicate. (d) Storage and maintenance of the basic data, the aggregated data, and statistics – One key characteristic of M&E systems is the continuous flow of new data into them that needs to be added to existing data, stored and used continually to produce and/or modify information. Non-digital items too, such as research papers and survey documents, need to be stored. It is important to ensure that the storage process is going on well, that stored data can be found quickly and easily, that access is controlled and that confidentiality is maintained. (e) Servicing and maintaining the IT infrastructure – This task is closely linked to the task of storing and maintaining data. All M&E systems today are dependent on the availability and proper functioning of IT hardware as well as general operational software (which is to be distinguished from specific M&E software). This IT infrastructure is spread over multiple sites throughout the country. It is found in the schools, the district and regional offices and at head office. The task consists of servicing and maintaining the hardware, as well as continually updating the operational and data storage software. It includes also the provision of user support which, in this case, relates not only those who enter information into the M&E system and those who use the information produced by it internally, but also users who are external to the system and have access to selected portions of it. (f) Encouraging and educating potential users to use the information – this task consists of taking deliberate action to develop a culture of information-based decision-making and thus increase the demand for information and analysis among potential users. Action can be undertaken to improve their awareness regarding the availability of information, to literally educate them on the potential uses of available information and encourage them to use it. It is particularly important to ensure that the utility of information and analysis for better decision-making is recognized among managers at all levels from the Ministry’s top managers all the way down to the schools’ directors. (g) Moving the information to intended users – this task consists of the dissemination of information to potential users. This can be proactive when the information is moved to users automatically without them asking for it, or reactive when the information – whether some raw data or aggregated data, or some specialized analysis – is provided on demand. The dissemination can be in the form of written, paper-based reports or digital reports; it can also be in the form of verbal communication when in response to individual verbal requests. SIIIR data is only disseminated to the public in the form of official reports and national statistics; parents are also able to access data about their schools online. Generally, education stakeholders need to actively look for education data, because it is not automatically presented to them by the government. Often the data available is in aggregate terms. Sharing SIIIR data with other ministries or organizations takes place only with bilateral agreements. Schools and parents do Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 24 have access to data regarding school infrastructure, staffing and facilities. While schools use the SIIIR to some extent, they are not using it to its full potential. (h) Managing the M&E system – the seven tasks described above are very different and, at the same time, closely linked and interdependent. Together, they require a vision, a set of clear objectives and priorities as well as considerable planning; they require also a budget with which to acquire human resources, technology and a variety of related services. In other words, they need to be well managed, under one organizational roof and one manager. 43. It is important to recognize that these tasks are key components of M&E systems. They need to be thought through during the design of the Ministry’s M&E system, because they can influence both the design of the system and the organizational arrangements within it. 6.2. Organizational arrangements for the development of the M&E system 44. As indicated in the introduction to this report, the M&E system covers four strategies. Each of the strategies was developed initially by a Ministry team consisting of persons with expertise in the domain of the strategy. Three of the strategies were developed with the help of the WB. The VET strategy was developed fully by the National Center for Development of Vocational Education and Training. 45. The Ministry appointed a general director to manage the implementation of each strategy and a technical group to help the general director in his/her management. The Ministry also set up a monitoring committee to oversee the implementation of all four strategies and a technical secretariat to help the monitoring committee oversee the implementation as presented in Figure 2 below. Here are the four general directors/directors in charge of the four strategies and the corresponding four technical groups:  General Director for Pre-University Education: in charge of the ESL strategy and ESL technical groups  General Director for Higher Education: in charge of the tertiary education strategy and the tertiary education technical group  Director of the National Authority for Qualifications (NAQ): in charge of the LLL strategy and the LLL technical group; and  Director of the National Center for Vocational Education and Training Development: in charge of the VET strategy and the VET technical group Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 25 Figure 2. Organizational arrangement within the MNESR for M&E the implementation of education strategies MONITORING COMMITTE TECHNICAL GROUPS (TGs) TG- ESL- GD for Pre-Unviersity TECHNICAL SECRETARY EVALUATION BODY Education (STRATEGIC (INSTITUTE OF TG-TED-GD for Higher Education MANAGEMENT AND EDUCATIONAL PUBLIC POLICIES SCIENCES) TG-LLL - National Authority for DIRECTORATE) Qualification TG-VET - National Centre for TVET 46. The Monitoring Committee is headed by a Secretary of State. It has a double role: (a) to help the Secretary of State monitor and control the implementation of all strategies; and (b) to coordinate action among them. In its monitoring and control role, the Monitoring Committee will first receive and review the initial evaluation plan of each of the strategies and, then, quarterly reports on the progress of each. Coordination among the four strategies is of importance because each of the four has some areas of interest that overlap with areas of interest in one or more of the other three, and because each has some activities and some outputs that may impact the activities and therefore the outputs of the other. In its coordinating role, the Committee will review and resolve all implementation issues that result from such overlaps and inter-dependencies. 47. The technical secretariat is housed in the Strategic Management and Public Policy Directorate and headed by the director of this unit. Its role is to provide administrative and technical support to the Secretary of State, the Monitoring Committee as well as the TGs. This includes the organization of training for members of the TGs, the calling of meetings, the preparation of agendas and, where necessary, materials for the meetings, and the production of minutes following them. Between meetings of the Monitoring Committee, the technical secretariat will also serve as a communications channel delivering messages back and forth among the Secretary of State, the Monitoring Committee, the general directors in charge of the four strategies and the TGs. 48. Each of the four general directors is the strategy’s ‘owner/manager’. The strategy is, in a sense, ‘his’ or ‘hers’, and he/she is expected to manage its implementation, as well as its monitoring, with the help of managers and experts under him/her. Each of the four strategies consists of many distinct Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 26 and diverse operations. His/her job is to allocate specific operations to specific managers and experts and charge them with managing these operations. He/she is responsible for the overall implementation and monitoring of the strategy, while they are responsible for the implementation and monitoring of their operations. As the owner of the strategy, he/she reports to the Secretary of State and is accountable to him/her. As the manager of the strategy, he/she coordinates action on all operations, holding the managers and implementation staff below, in turn, accountable to him/her. With the help of the technical group, he/she monitors the status and progress of each operation, helping the managers and staff who are in charge of each operation to resolve operational issues. Ultimately, he/she is expected to control the operations and ensure their effective implementation. 49. The managers and experts who are in charge of specific operations under the general directors may be regarded as ‘operations managers’ or ‘project managers’. Their job is to produce operational plans for their operations/projects and then ensure that these plans are being carried out on time, as per plan and as per technical, professional and administrative specifications. In other words, they are not only the managers, but also the monitors, of the operations/projects assigned to them. At the operational level, monitoring is a management function. 50. Each technical group consists of functionaries representing the MNESR’s relevant directorates and agencies that are expected to play a role in the implementation and monitoring of the strategies. Each is headed by the general director (or director, as in the case of NAQ), but it is important to note that the technical groups are not included in the chain of command. They are in the system as advisory bodies providing professional/technical support to the general director who is the only one who has management authority over the managers and experts who are managing specific operations (as pointed out above). 51. A summary of the above mentioned roles and responsibilities of main players is presented in Figure 3 below. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 27 Figure 3. Roles and responsibilities by level Coordinates action on all strategies, monitors + controls implementation of all strategies with the help of the Monitoring Committee; all decisions that bind all strategies SoS and all TGs are discussed and made at this level Coordinate action on all activities/operations in their strategies with the help of the relevant Technical Group; monitor + GDs control implementation, integrate/discuss data reported by directors on progress of all operations Produce activity plan/operational plan for each activity/operation. Manage, monitor & control Directors operational work. Ensure that planned activities (a) are being carried out (b) on time, as per plan, and (c) substantively, as per plan Staff Carry out all operational work; collects, aggregates processes, stores and reports operational data Note: Secretary of State = SoS; GDs= General Directors 52. Best practice M&E systems require that strategies will be evaluated by independent professionals who are not the designers, planers and implementers of the strategies. At the same time, this is also in line with the Guidance document issued in 2015 by the European Commission on Monitoring and Evaluation of European Cohesion Policy – European Social Fund for the Programming Period 2014-20202. Therefore, the Ministry has charged the Institute of Educational Sciences (IES) with the responsibility for evaluating all the strategies. It also set up a fifth technical group headed by the IES director to focus on the evaluation. Its members come from this Institute and cover each of the four strategic areas. 53. As is the case with the four strategies and the other technical groups, the evaluation TG is an advisory body providing professional/technical support to the IES director. It is not in the chain of command and is not responsible for the evaluation work. The overall responsibility for this work lies with the director of the IES who is at the top of the chain of command in the Institute. The director will formally assign one expert within the Institute to manage and oversee the evaluation of each individual strategy with the help of additional staff where necessary. The manager will report to the director and will be accountable to him. This manager will also be a member of the corresponding Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 28 strategy implementation TG. Together, the four evaluation managers and the staff helping them with the evaluation work will form a transversal coordinating and collaborating task team, but accountability for the evaluation of each strategy will remain with the relevant manager. 6.3. Organizational arrangements for the operation of the M&E system 54. M&E may be viewed not just as an operation, or an operational system, but also as a key organizational function. As noted earlier, it is a function that consists of two distinct but closely- linked and inter-dependent functional areas: monitoring and evaluation. Moreover, as a function it consists of eight distinct tasks, as noted earlier too: data collection, data aggregation, data processing, data storage, data dissemination, IT maintenance and servicing, user education/promotion, and user support. 55. From an organizational point of view, all major functions need to have what may be referred to as an ‘organizational home’ – a unit within the organization that is designated for, and typically dedicated to, the performance of all the tasks that make up the function. Usually, when a major function is housed in a dedicated organizational home, it is more likely that it will be better led and managed than when it has no dedicated home. It is more likely that it will have a leader/manager, high enough in the organization, and it is more likely that the leader/manager will take good care of it and fight for it when necessary: he/she will develop a vision for it, will formulate policies and strategies for it, will mobilize funds for it, will put in place organizational arrangements for each of its tasks, will develop norms and standards for each task and will guide, monitor and control it. 56. As a function, M&E relates not only to the Ministry’s four strategies, but to all its strategies and all its operations. The eight tasks listed above in section 6.1 apply to all the organization’s strategies and all its operations – not just to some. Therefore, when designing the organizational arrangement for the monitoring and evaluation of a specific strategy, or a specific set of strategies, it is advisable to do so within the overall organizational context of the M&E function. 57. The Ministry has recently established an organizational home for the M&E function: it is the Strategic Management and Public Policies Directorate. This is the right place for the M&E function, because a key role of M&E is to enhance policy-making and strategic management. For the performance of this role, the Directorate is, however, highly dependent on the availability of information, i.e., the collection, aggregation, storage, processing and maintenance of data by the IT Office in the Directorate for Assets, Investments and IT. As noted earlier, the capacity of this Office is very limited. The team believes that Ministry’s intention to establish a dedicated unit/agency for IT is, therefore, an important step in the right direction. However, it recommends that the Ministry will also assess the operational requirements of the unit/agency, establish its human resources’ needs, in terms of both the number of staff and the mix and level of skills needed, and then staff it accordingly. 58. A similar recommendation needs to be made regarding the Strategic Management and Public Policies Directorate. The Directorate’s current capacity to effectively carry out its work is limited, although it has recently received some additional staff. This situation has arisen due to the dramatic increase in its workload that has come with the launching of the four strategies and the need to monitor Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 29 and evaluate them. As is clear by now, the term ‘strategy’ has been used by the Ministry in a very general sense to describe what may be referred to as a ‘strategic area’. ESL, Tertiary Education, LLL and VET, are all strategic areas. But each of these four areas has several broad strategic objectives/thrusts that can be seen as (lower-level) strategies themselves: the first three have four broad strategic objectives/thrusts each, and the last has five – altogether 17 broad strategic objectives/thrusts In addition, a new strategy is going to be launched in soon for infrastructure. Even though the Directorate is not responsible for carrying out the operational monitoring of the four strategies encompassing 17 broad strategic objectives and the evaluation of their achievement, it is still responsible for helping the Secretary of State and the Monitoring Committee in their oversight of all these strategies and strategic objectives. In this context, it is already having significant administrative work and is already responsible for much of the coordination among the technical groups, as well as the communication between them and the monitoring committee. Thus, the probability that the Directorate will very soon require additional staff is high. 59. Finally, the comments below about arrangements for the operation of the M&E system apply to the whole M&E function and not just to the monitoring and evaluation of the four strategies. 60. For organizing the M&E function, the Ministry needs to distinguish among three types of operations: (i) its regular operations which are its continuing, on-going core business, (ii) its individual projects, which are one-time operations aimed at improving its core business and which have a beginning and an end, and (iii) its strategies which may all be regarded as multi-project operations. 61. Regular operations need to be both monitored and evaluated. The monitoring as well as the evaluation of these operations are a centralized functional responsibility. They are to be clearly distinguished from supervision, which is a decentralized function. Organizationally, they need to be, and are already, placed in the Ministry’s head office, i.e., the M&E in the Strategic Management and Public Policies Directorate and the IT in the Directorate for Assets, Investment and IT and, probably, in an IT unit/agency in the future. 62. Individual projects, too, need to be monitored and evaluated. However, their monitoring, which is clearly a project management activity, needs to be inside the organizational units that are implementing them, while their evaluation needs to be placed outside these units, because of the need for objectivity. Evaluations may also depend on the project specificity and the financier. 63. Strategies need to be evaluated, but as was stated earlier, they cannot be evaluated without monitoring. The monitoring of strategies is different, however, to the monitoring of operations and projects: in the case of individual projects and operations, the focus is on activities, while in the case of strategies the focus is on the results of these activities. The data collected for the monitoring of results is different to the data collected for the monitoring of activities. 64. To conclude, the monitoring and evaluation of strategies, also depend on sound M&E methodologies and associated instruments looking into both qualitative and quantitative aspects. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 30 65. The next output to be delivered by the Bank team under this RAS Agreement will be focused on the preparation of the methodology and instruments for the M&E of the implementation of the Strategies for Reducing ESL, TE, LLL, and VET. In addition, other tasks will cover training for the MESR staff on the proposed methodology and instruments, peer reviews of the draft M&E reports to be prepared by the MNESR, as well as a revision of the methodology and instruments, as needed, subsequent to their application. Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 31 Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European Competența face diferența!Proiect selectat în cadrul Programului Operațional Capacitate Administrativă, cofinanțat de Uniunea Europeană, din Fondul Social European 32