ROMANIA Advisory Services Agreement on Improving Monitoring and Evaluation Capacity in the Context of EU-funded Programs in Romania (2021–2027) (P174133) Inception Report August 2020 “Project cofinanced from European Structural and Investment Funds” Disclaimer This report is a product of the International Bank for Reconstruction and Development / the World Bank. The findings, interpretation, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. This report does not necessarily represent the position of the European Union or the Romanian Government. Copyright statement The material in this publication is copyrighted. Copying and/or transmitting portions of this work without permission may be a violation of applicable laws. For permission to photocopy or reprint any part of this work, please send a request with the complete information to either: (i) the Ministry of European Funds (1-1B, Bucharest – Ploiesti Street, Victoria Office Building, Sector 1, Bucharest, Romania); or (ii) the World Bank Group Romania (31, Vasile Lascăr Street, 6th Floor, Sector 2, Bucharest, Romania). ii Acknowledgements This inception report has been delivered in August 2020, under the provisions of the Advisory Services Agreement on Improving Monitoring and Evaluation Capacity in the Context of EU- funded Programs in Romania (2021–2027), signed between the Ministry of European Funds and the International Bank for Reconstruction and Development on June 30, 2020, and prepared under the guidance and supervision of Cem Mete (Practice Manager, Social Protection and Jobs, Europe and Central Asia) and Tatiana Proskuryakova (Country Manager, Hungary and Romania). This report was drafted under the coordination of Victor Sulla (Senior Economist) and Maddalena Honorati (Senior Economist), task team leaders, and Annette M. Richter (Senior Monitoring and Evaluation Specialist), Adina Iorganda (Social Protection and Labor Specialist), Elena Botezatu (Expert), Alina Bosoi (Expert), and Ludovica Cherchi (Social Protection and Labor Specialist). The team was supported by Andrei Zambor. The team would like to express its gratitude for the excellent cooperation, guidance, and feedback provided by the representatives of the Programs Evaluation Office in the Ministry of European Funds, in particular to Mrs. Claudia Magdalina, Alexandra Voiculescu, and Maria Malureanu. iii Abbreviations AIR Annual implementation report BRCT Regional Office for Territorial Cooperation CCMAP Committee for the Coordination of the Partnership Agreement CF Cohesion Fund CLLD Community-led local development CMU Country Management Unit CPR Common Provision Regulation DG Directorate General EAFRD European Agricultural Fund for Rural Development EC European Commission EMFF European Maritime and Fisheries Fund ENI European Neighborhood Instrument ERDF European Regional Development Fund ESF European Social Fund ESF+ European Social Fund Plus ESIF European Structural and Investment Funds ETC European Territorial Cooperation EU European Union GDP Gross domestic product GoR Government of Romania GNI Gross national income GP General Practice IB Intermediate Body ICT Information and communications technology IE Impact evaluation IP Investment priority IPA Instrument for Pre-Accession Assistance ITI Integrated Territorial Investment JS Joint Secretariat (for ETC Programs) LAG Local Action Group LFS Labor Force Survey LIOP Large Infrastructure Operational Program M&E monitoring and evaluation MA Managing Authority MARD Ministry of Agriculture and Rural Development MC Monitoring Committee MEF Ministry of European Funds MPWDA Ministry of Public Works, Development and Administration MT Medium-term outcome NGO Nongovernmental organization NPRD National Program of Rural Development OP Operational Program OPAC Operational Program Administrative Capacity OPC Competitiveness Operational Program OPTA Operational Program Technical Assistance PA Partnership Agreement PF Performance Framework POAD Support Disadvantaged Persons Operational Program POCU Human Capital Operational Program POPAM Marine and Fisheries Operational Program RAS Reimbursable Advisory Services RDA Regional Development Agency iv ROP Regional Operational Program SCO Simplified cost option SFC2014 System for Fund Management in the European Union SILC Statistics on Income and Living Conditions SIMPOP Information System for the Management of the Operational Program for Fisheries SMART Specific, measurable, achievable, relevant, timely ST Short-term outcome TCBP Training and capacity-building plan TNA Training needs assessment TO Thematic objective ToC Theory of change WB World Bank YEI Youth Employment Initiative v Glossary Audit Authority The Audit Authority is a national, regional, or local public authority or body designated for each operational program and responsible for verifying the effective functioning of the management and control system. It also monitors project compliance with national and European regulations. The audit authority's tasks include checking whether the management and control systems are working efficiently (system audits). It is also responsible for carrying out controls focusing specifically on declared expenditure (operational audits). Ensuring adequate separation of functions between the main authorities (managing/certifying authorities, intermediate bodies) is an important part of these controls. Beneficiary A public or private body, responsible for initiating or initiating and implementing an operation (project), or a body receiving state aid or a body implementing a financial instrument or a fund of funds, as appropriate. 1 Certifying A certifying authority is responsible for guaranteeing the accuracy and probity Authority of statements of expenditure and requests for payments before they are sent to the European Commission. Management of the European Regional Development Fund, European Social Fund and Cohesion Fund is shared with member countries, regions and other intermediary bodies. A certifying authority is nominated by one or more of the aforementioned groups for each operational program co-financed by these Funds. Specific responsibilities of certifying authorities include: (i) certifying compatibility of expenditure with national and EU rules and criteria; (ii) ensuring that sufficient information is received from the relevant Managing Authorities to support their claims; (iii) taking account of audit reports; (iv) maintaining computerized records; and (v) keeping account of unused/recovered funds to be returned to the Commission. Cohesion Policy Cohesion policy is the European Union’s strategy to promote and support the “overall harmonious development” of its Member States and regions. It aims to strengthen economic and social cohesion by reducing disparities in the level of development between regions. The policy focuses on key areas established at EU level, known as thematic objectives (for the current period) or policy objectives (for 2021–2027). Approximately 32.5 percent of the EU budget 2014–2020 (equivalent to about €351.8 billion over seven years, in 2014 prices) is allocated to financial instruments which support cohesion policy. These are managed and delivered in partnership between the European Commission, the Member States and stakeholders at the local and regional level. Common A common indicator is an indicator with agreed definition and measurement unit indicators to be used when relevant in Operational Programs, permitting aggregation to the national and EU level. 2014–2020 CF and ERDF regulations define common output indicators. 2 Decommitment Under the automatic decommitment principle, if a sum committed to a program has not been claimed by the end of the second year following the program's adoption, any unpaid money ceases to be available to that program. This mechanism aims to improve both their speed of development and the monitoring of flows of program funding. Direct results The direct results indicator matches the direct effects (immediate and short- indicator term effects) of the intervention for the direct addressees. Direct results are project results, i.e., the direct benefit and outcome of program interventions strictly related to (or derived from) the use of project (program) outputs. These results, for instance, refer to the performance of beneficiaries, investments triggered, increased access to services. Direct results indicators are aggregated at program level from the project level. There is a baseline value that may or may not be 0. 3 1 As per Art. 2 of EU Regulation no. 1303/2013. 2 https://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/indic_post2020/indic_post2020_p1_en.pdf 3 https://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/indic_post2020/indic_post2020_p1_en.pdf vi Immediate The immediate results indicator reflects the situation at the end of the support results indicator provided. This may mean the exit of the participants/entities, i.e., the last day in (ESF specific) which they are involved in a project activity. In some cases, the termination of support is similar to the end of the project. 4 Indicators The entire set of indicators used for monitoring the implementation of the ESIF. system Intermediate Any public or private body which acts under the responsibility of a Body managing or certifying authority, or which carries out duties on behalf of such an authority, in relation to beneficiaries implementing operations. 5 Investment In the 2014–2020 programming period, investment priorities set out detailed Priorities objectives and forms the basis for defining specific objectives within operational programs, based on the needs and characteristics of the program area, they are defined in the fund-specific regulations, for each of the 11 thematic objectives. Longer-term The longer-term results indicator reflects the situation of the results indicator participants/entities/communities supported at a certain period after the end of (ESF-specific) the support (usually six months), highlighting the change that occurred as a result of the intervention. 6 Managing A managing authority is responsible for the efficient management and Authority implementation of an operational program. A managing authority may be a national ministry, a regional authority, a local council, or another public or private body that has been nominated and approved by a Member State. Managing authorities are expected to conduct their work in line with the principles of sound financial management. Milestones The milestones are intermediate targets set for indicators to be achieved the middle of the programming period (2018, for the current period, 2024 for the next). Monitoring EU Structural and Investment (ESI) funding must be spent effectively and in accordance with its defined purpose. For this reason, extensive evaluation and monitoring procedures have been established through EU legislation to check that operational programs using ESI funding are performing properly and delivering results that can be checked against agreed criteria. The Commission has established a monitoring system that harnesses strict certification, control and correction mechanisms. Member States have to carry out checks and audits of projects that receive funding, and the Commission itself can make on-the-spot inspections. The financial management rules for ESI Funds require Member States to assess the implementation of operational programs through a certification body, an auditing body and a monitoring committee. Monitoring Monitoring Committees are appointed by the Member States to check that Committee operational programs (OPs) which use European Structural and Investment (ESI) funding are being correctly implemented. These committees are chaired by the relevant Member State (or managing authority) and comprise regional, economic and social partners. A monitoring committee's key tasks include: (i) assessing the effectiveness and quality of OPs; (ii) approving criteria for financing under each OP; (iii) making periodical reviews of OPs and their progress toward specific targets; (iv) examining the results of implementation to assess whether those targets have been met; and (v) where necessary, proposing revisions to OPs, including changes related to their financial management. Output indicator An output indicator relates to the specific deliverables of the intervention. It measures what is produced or bought by the program expenditure and investments through the projects. It is measured at project level and then aggregated at program level and has no baseline value. 7 4 POCU Indicators Guide http://mfe.gov.ro/pocu/fise/ghid-indicatori-pe-sectiuni-rev-am-pocu/#_Toc437793349 5 https://iate.europa.eu/entry/result/2146475/en 6 Idem 7 7 https://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/indic_post2020/indic_post2020_p1_en.pdf vii Participant The person who is part of the target group of the project and who actively participates in an activity within the project. 8 Partnership For each programming period, each Member State has to produce a Partnership Agreement (PA) Agreement (PA) in cooperation with the European Commission. This is a reference document for programming interventions from the Structural and Investment Funds and link them to the aims of the European strategy. It defines the strategy and investment priorities chosen by the relevant Member State and presents a list of national and regional operational programs (OPs) which it is seeking to implement, as well as an indicative annual financial allocation for each OP. Performance The performance framework was introduced in the 2014–2020 regulations to framework monitor the implementation and delivery of program outputs as planned. It is a specific mechanism linked to the performance reserve (6 percent of the program allocation) and the performance review (in 2019 and at closure). The 2019 performance review exercise is meant to incentivize timely delivery as achievement of milestones at the end of 2018 releases the performance reserve to performing priorities in the national and regional programme. Nonperforming priorities will lose the allocated reserve, which shall be reallocated. Policy In the 2021–2027 period, the priorities of the ESI Funds have been consolidated Objectives from 11 to 5, and they were renamed policy objectives: 1. A Smarter Europe, through innovation, digitization, economic transformation and support to small and medium-sized businesses 2. A Greener, carbon free Europe, implementing the Paris Agreement and investing in energy transition, renewables and the fight against climate change 3. A more Connected Europe, with strategic transport and digital networks 4. A more Social Europe, delivering on the European Pillar of Social Rights and supporting quality employment, education, skills, social inclusion and equal access to health care. 5. Europe closer to citizens, by supporting locally led development strategies and sustainable urban development across the EU. Project The process of appraisal, evaluation and selection of project proposals submitted evaluation for financing under ESI funds. Program An evaluation carried out at the level of the operational program. More generally, evaluation evaluation can be defined as the systematic and objective assessment of an ongoing or completed action or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. The Better Regulation package defines evaluation as the assessment of the effectiveness, efficiency, coherence, relevance and EU added value of one single EU intervention, thus adding coherence and EU value added. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision– making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an intervention. An assessment, as systematic and objective as possible, of a planned, ongoing, or completed intervention (OECD). 9 Regulations Regulations are the strongest form of EU legislation. They are of general application, binding in their entirety and directly enforceable in all Member States. Legislation relating to EU cohesion policy usually takes the form of regulations. Simplified Cost The use of flat rate financing, standard scales of unit costs and lump sums instead Options of reimbursing actual expenditure based on invoices. 10 Specific Specific indicators are established at the level of each OP and monitored at indicators national level. Specific For the 2014–2020 period, specific objectives further detail the investment objective priorities 8 http://mfe.gov.ro/pocu/1-1-glosar-de-termeni/ 9 European Commission, Guidelines on linking planning/programming, monitoring and evaluation, July 2016. 10 https://iate.europa.eu/entry/result/3541722/en viii For the 2021–2020 period, they detail the 5 policy objectives set under the Cohesion Policy and are more or less equivalent to the investment priorities of the 2014–2020 period. Sustainability The EU attaches great importance to the durability of the investments funded period through the ESI Funds. In means sense, project beneficiaries are required to maintain the results achieved after the official end of the project. Project partners must consider how the investments will be secured, developed, made use of, replicated or transferred during or after the end of the project, for a period of up to 5 years. Their activity is monitored by the MA, at least once during this period. Thematic In the 2014–2020 programming period, the ESI Funds will support 11 priorities, Objectives also known as thematic objectives: 1. Strengthening research, technological development and innovation 2. Enhancing access to, and use and quality of information and communication technologies (ICT) 3. Enhancing the competitiveness of small and medium-sized enterprises (SMEs) 4. Supporting the shift toward a low-carbon economy in all sectors 5. Promoting climate change adaptation, risk prevention and management 6. Preserving and protecting the environment and promoting resource efficiency 7. Promoting sustainable transport and removing bottlenecks in key network infrastructures 8. Promoting sustainable and quality employment and supporting labor mobility 9. Promoting social inclusion, combating poverty and any discrimination 10. Investing in education, training and vocational training for skills and lifelong learning 11. Enhancing institutional capacity of public authorities and stakeholders and efficient public administration Source: https://ec.europa.eu/regional_policy/en/policy/what/glossary. ix Table of Contents Acknowledgements ............................................................................................................................... iii Abbreviations ........................................................................................................................................ iv Glossary ................................................................................................................................................. vi Executive Summary .............................................................................................................................. 1 1. Introduction .................................................................................................................................. 4 1.1. Background ............................................................................................................................ 4 1.2. Objectives............................................................................................................................... 5 2. General framework of the ESIF M&E system in Romania ...................................................... 6 2.1. Overview of ESIF for the 2014–2020 programming period ........................................................ 6 2.2. ESIF for the 2021–2027 period.................................................................................................... 9 2.2.1 Monitoring component of the ESIF M&E system ............................................................... 10 2.2.2 Evaluation component of the ESIF M&E system ................................................................ 17 3. Methodological framework and scope of proposed work ....................................................... 19 3.1. Key elements of the work program............................................................................................. 19 3.2. An integrated M&E framework.................................................................................................. 22 3.3. Methodological approach: Assessment framework and data collection.................................... 23 4. Overview of RAS-funded activities ........................................................................................... 27 4.1. Activities at inception stage ....................................................................................................... 27 4.2. Activity 1: Assess the existing M&E system ............................................................................... 28 4.3. Activity 2: Options to improve the M&E system of EU co-funded investments in Romania ..... 37 4.4. Activity 3: Propose improvements in the M&E system .............................................................. 45 5. Workplan summary: Deliverables, milestones, and timeline ................................................. 51 6. Requested support and information gaps................................................................................. 52 7. Limitations of the analysis ......................................................................................................... 52 8. Main stakeholders....................................................................................................................... 53 9. Risks and risk mitigation ........................................................................................................... 57 10. Quality assurance mechanisms .................................................................................................. 58 11. Dissemination and completion ................................................................................................... 60 Annexes ................................................................................................................................................ 64 Annex 1: Structural and investment funds and OP structure ........................................................ 64 Annex 2: Literature summary on results-based M&E .................................................................. 67 Annex 3: M&E communication channels..................................................................................... 69 Annex 4: Quality control and quality assurance checklist for deliverables .................................. 70 Annex 5: Documents available for review ................................................................................... 71 Annex 6: Minute of the Inception Workshop ............................................................................... 73 References: ........................................................................................................................................... 76 Figures Figure 1: Monitoring mandate per implementation level ...................................................................... 12 Figure 2: Monitoring and reporting flow............................................................................................... 13 Figure 3: M&E function and process .................................................................................................... 16 Figure 4: RAS’s proposed results chain ................................................................................................ 20 Figure 5: Theory of change for the M&E RAS ..................................................................................... 21 Figure 6: An integrated system: measuring M&E performance at four levels ...................................... 22 Figure 7: Main project tasks .................................................................................................................. 24 x Figure 8: Ten steps to designing, building, and sustaining a results-based M&E system ..................... 26 Figure 9: Two-step sampling method, graphical representation ........................................................... 36 Figure 10: Methodological approach for IE sub-activity ...................................................................... 43 Figure 11: Process for identifying and addressing training needs ......................................................... 47 Figure 12: Outputs and milestones ........................................................................................................ 51 Tables Table 1: Operational Programs, 2014–2020 ............................................................................................ 8 Table 2: Tasks for monitoring interventions funded through ESIF....................................................... 16 Table 3: Dimensions of the M&E system assessment framework and core questions ......................... 25 Table 4: The SMART criteria checklist for quantitative analysis of indicators .................................... 30 Table 5: Identification of groups of interest .......................................................................................... 36 Table 6: Key M&E stakeholders ........................................................................................................... 54 Table 7: Stakeholders engagement ........................................................................................................ 56 Table 8: Proposed evaluation grid ......................................................................................................... 59 Boxes Box 1: Key changes to the Cohesion Policy M&E for the 2021–2027 period ........................................ 5 Box 2: European Structural and Investment Funds (ESIF) ..................................................................... 7 Box 3: EU Cohesion Policy Objectives for 2021–2027 ........................................................................ 10 Box 4: POCU-Form, the POCU instrument for collecting indicators ................................................... 11 Box 5: Performance Review.................................................................................................................. 12 Box 6: Systems for electronic data exchange ........................................................................................ 14 Box 7: Suspension of payments by the EC ........................................................................................... 15 Box 8: Participants vs. beneficiaries in ESIF interventions .................................................................. 17 Box 9: Overview of M&E assessment framework ................................................................................ 24 Box 10: Example of stratified two-step random sampling method for selecting participants for the online survey ......................................................................................................................................... 36 xi Executive Summary 1. Romania is expected to receive approximately €30.6 billion of funding from European Structural and Investment Funds (ESIF) in the 2021–2027 programming period. These funds will support Operational Programs (OPs) across all European Union (EU) Cohesion Policy objectives and cover key areas, including human development, sustainable development and infrastructure, and private sector development. Alongside this ambitious program, the Government of Romania (GoR) will need to implement changes in monitoring and evaluation (M&E) practices, based on the EU-level push toward harmonized and simplified M&E. 2. Thus, the GoR is initiating a thorough assessment of the system used to monitor and evaluate ESIF-funded programs in the 2014–2020 period. This assessment aims to both identify the previous program’s challenges and successes and determine how to adjust the system going forward, in light of the EU’s new requirements. Therefore, the GoR, and specifically the Ministry of European Funds (MEF), have engaged the World Bank (WB) through a Reimbursable Advisory Services (RAS) Agreement aimed at improving M&E capacity in the context of EU-funded programs in Romania. The objective of this RAS is to support the Government of Romania in building capacity for developing and operationalizing a monitoring and evaluation (M&E) system to track and monitor programs co-funded by European Union (EU) funds in the 2021-2027 programming period. Across three activities, the WB will collaborate with the GoR to learn what about the M&E system worked (and what did not) in the 2014–2020 period, determine options for system improvements, and support the GoR to implement these. 3. The RAS consists of three activities: • Activity 1: Assessment of the existing M&E system of EU-funded measures • Activity 2: Options to improve the M&E system of EU co-funded investments in Romania • Activity 3: Propose improvements in the M&E system 4. This inception report is a deliverable under the RAS. It serves as a roadmap for the RAS team’s activities, 11 and specifically: (i) describes the methodological aspects of each activity to be carried out by the WB under this Agreement; and (ii) provides a detailed description, indicative timetable, and Gantt chart of all activities to be carried out under the RAS. The document presents an initial discussion of the context by briefly describing the scope and implementation arrangements for ESIF-funded programs. It then summarizes the current M&E practices under the 2014–2020 period, including the institutional mandates for M&E functions, key tools, and EU requirements and regulations that shape M&E practices. Notably, the monitoring function is distributed across multiple layers, paralleling the multi-level implementation of ESIF-funded (operational) programs and projects. Monitoring and reporting responsibilities are established at each level, and an integrated reporting system links from the project, to the OP, to the Priority Axis, and finally up to the Partnership level. This complex system is vulnerable to coordination challenges, capacity gaps, and appears to be burdensome for stakeholders. The evaluation function seems more concentrated, with the MEF taking a key leadership role, and a growing number of evaluations are being carried out. However, it is an open question whether there is uptake of evaluation findings; the capacity for impact evaluation appears limited and the overall evaluation culture is nascent, at best. 5. Based on a snapshot of the current M&E system, the RAS team proposes adopting a methodological approach for all three RAS activities. This discussion focuses heavily on the M&E system assessment the team will undertake through Activity 1. The team has developed this initial approach based on several conceptual and practical M&E tools. The first is an integrated M&E framework, which has been a useful reference point for designing and implementing M&E systems that cover a broad scope of activities or sectors. Next, the team foresees employing an M&E system assessment tool that follows from WB experience with M&E system assessments. This tool aims to 11 The RAS team is an extended team of World Bank staff and external consultants who contributed to this report. 1 provide a holistic view of the M&E system, covering its scope and objective, policy and legal foundations, institutional architecture, functions, roles and responsibilities, tools, capacity, and incentives. 6. In addition to the M&E system assessment tool, the methodological approach also borrows from the widely used Ten Steps to a Results-Based Monitoring and Evaluation System. While Ten Steps sets out a sequence for developing a results framework and M&E systems, the ESIF M&E system is already well along in this sequence. Thus, the team will use the later steps (6–9) as a basis for an in- depth M&E review that focuses on the OP M&E plans, results frameworks, indicators, M&E guidelines, evaluation plans, and individual evaluations. 7. Moreover, the team foresees applying mixed methods to collect data. Data collection will include a carefully structured set of activities, such as document reviews, surveys, diagnostic and key informant interviews, focus groups, and case studies. The team will also exploit information that can be gleaned from the relevant information systems to undertake a quantitative analysis of indicators. 8. Next, the RAS team foresees designing options for M&E system improvements and thoroughly vetting these options. In close collaboration with the MEF, the RAS team will develop a vision and scope for the M&E system and various system options. Key stakeholders will have the opportunity to provide a reality check and develop ownership over these options through participatory approaches such as focus groups, consultations, and surveys. Based on these, the RAS team will draft a strategy that presents the selected option and all of its institutional and practical ramifications. 9. As part of the M&E system improvement options, the RAS team will also provide input on OP-level evaluation plans and guidance on rigorous impact evaluations. Through this support, the RAS team will help OPs meet new EU M&E requirements for the development of evaluation plans. Moreover, the RAS team also intends to identify opportunities for rigorous impact evaluations, based on the evaluation needs and topics included in the OPs’ evaluation plans. 12 10. The third RAS activity entails support to the MEF and other stakeholders to help implement the selected option for M&E system improvements. The team will design this support by conducting a training needs assessment, proposing a training and capacity development plan, and then by providing select training. 11. Given the complex nature of RAS activities, the team acknowledges the critical importance of risk assessment, and offers mitigation options in this report. The current ESIF M&E system is already complex and involves many stakeholders at different levels. It will be challenging, but important, to capture their honest input in order to assess the M&E system. Then, it will be necessary to foster their continued engagement in RAS activities and build ownership over system improvements. The risk assessment section describes the initial mitigation measures the RAS team plans to employ. 12. Similarly, strong stakeholder engagement is critical for implementing M&E system improvements. The RAS team recognizes that for the assessment to succeed, stakeholders must attend focus groups, respond to surveys, and participate in interviews. But each stakeholder has their own incentives and interests. Accordingly, the RAS team included a brief stakeholder analysis that shows who is involved in the current M&E system, what their interests are, and potential channels for their engagement. 13. Given that RAS activities are highly participatory and geared toward knowledge generation, dissemination will be critical to ensure activities achieve their full impact. Thus, this report includes a brief dissemination strategy that shows how lessons will be shared and identifies communication channels for informing stakeholders. Notably, information sharing will be important for 12 Following from the International Initiative for Impact Evaluation (3ie) impact evaluation is defined as “analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context”. http://www.3ieimpact.org/our-expertise/impact-evaluation. In addition, the rigorous methodology usually refers to the use of a counterfactual, often through experimental or quasi experimental methods. 2 encouraging stakeholder involvement, and knowledge sharing is already identified as an important component in this RAS, to build on the lessons identified through the M&E system assessment. In particular, the RAS foresees knowledge sharing both among stakeholders in Romania, and more broadly with stakeholders in EU member states on lessons derived from evaluation plans and rigorous impact evaluations. Across all dissemination activities, this report identifies the process for sharing information with a broad set of stakeholders. It also explains that through information sharing, more stakeholders benefit from project lessons and knowledge, and this generates a broader support base for improvements. Ultimately, knowledge sharing, and support building are critical to sustaining M&E system improvements, so they become institutionalized and continue beyond the RAS engagement. 14. Lastly, the RAS team will deliver this M&E support to the GoR through seven outputs. Output 1a is a methodological report and a policy brief for non-technical audiences that describes the team’s approach for assessing the ESIF M&E system. The Output 1b will report on institutional, procedural and legislative assessment. The report will cover the stocktaking of the M&E system’s performance, the assessment of the institutional, legislative, and procedural framework of M&E of EU co-funded investments, and the beneficiary survey. Output 2a is a technical report summarizing options for improvement of existing M&E system. Output 2b will include specific improvements on OP indicators (a draft indicator table and related metadata) along with OP-level evaluation plans. Output 3a is a technical report will include a proposed implementation plan, training and capacity building plan, and training needs assessment. This report will summarize (i) an implementation plan for an improved M&E system; (ii) a training needs assessment; and (iii) a training and capacity-building plan. In addition, the RAS team will deliver selected trainings and assess these in the Output 3b. Finally, the RAS team will summarize all RAS activities in Output 4. The summary report of all activities provided by the Bank under this Agreement to be delivered at the end of this engagement. 3 1. Introduction 1.1. Background 15. In the 2021–2027 programming period, Romania is expected to receive approximately €30.6 billion of funding from European Structural and Investment Funds (ESIF). In February 2020, the Government of Romania (GoR) approved a memorandum related to future Operational Programs (OPs) and the framework of implementing institutions. The new OPs cover a wide range of sectors pertaining to key areas of human development, sustainable development and infrastructure, and private sector development issues. The memorandum recognizes the need to strengthen the role of the Ministry of European Funds (MEF) in coordinating evaluations to be carried out at the OP level by each Managing Authority (MA). 16. The experience with implementing ESIF funds under both the 2007–2013 and 2014–2020 programming periods reveals critical challenges that need to be addressed. As noted below, these were highlighted in several assessments and beneficiary surveys, as well as in implementation experience. Monitoring and evaluation (M&E) are key in this undertaking. Going forward, Romanian authorities will need to consider emerging lessons on M&E activities from the 2014–2020 period. • In terms of monitoring, a recently completed assessment 13 of common indicators in the 2014– 2020 period identifies three main challenges for the post-2020 period: harmonization, simplification, and thematic coverage. • Regarding harmonization, a stocktaking exercise across EU Member States carried out by ESIF Directorate Generals (DGs) in 2017 identified considerable differences between ESIF monitoring practices and emphasized the lack of common direct results indicators for the European Regional Development Fund (ERDF) and the Cohesion Fund (CF). • On simplification, desk research and consultations with the MA indicate the need for simplification, which would facilitate the identification, definition, and reporting of appropriate common indicators. • Regarding thematic coverage, consultations with the MA highlight that OP-specific output indicators are introduced in 70 percent of cases due to the lack of appropriate common output indicators for the corresponding thematic objectives (TOs) and investment priorities (IPs). 17. The new European Commission (EC) regulations envisage key changes compared to the current period, especially regarding M&E arrangements. One of the key modifications is to standardize indicators, with common output and results indicators being proposed for all funds. To respond to these common challenges, the indicator set foreseen for the 2021–2027 period 14 is expected to be based on common concepts for output and results indicators across Cohesion Policy instruments (ERDF, CF, ESF+). It will include a more complete list of common output indicators (i.e., indicators measuring an intervention’s specific deliverables), and will also propose a new framework of common results indicators (i.e., measuring the short-term effects of the interventions supported, with particular regard for direct beneficiaries such as target populations or users of infrastructure). Box 1 lists other key M&E changes. 13 Development of a system of common indicators for ERDF and CF interventions after 2020. Sweco/t33 srl consortium, 2018. 14 Evaluation Network presentations from June 2018 and May 2019. 4 Box 1: Key changes to the Cohesion Policy M&E for the 2021–2027 period The main changes to the M&E policy for 2021–2027 are as follows: • A complete list of common output indicators was introduced that measure the intervention’s specific deliverables. • A new list of common results indicators was introduced, measuring the (short-term) effects of the interventions supported (with reference to the direct beneficiaries, population targeted, or users of infrastructure). • All financial inputs and indicator data will be reported by specific objective to ensure transparency. Financial data will be reported quarterly. • Reporting on indicators will be done every two months, instead of twice a year, feeding public open data. • The categorization system will be updated with climate-conscious new green tracking. • The performance framework will cover all output and results indicators, instead of a select number. • Performance reserve is replaced by the 5+2 rule: initial resource allocation for 5 years – the final 2 years to be planned based on interim success. • Evaluations performed by the Member States must assess a program/interventions’ relevance, effectiveness, efficiency, relevance, and EU added value. • Evaluations must include interim lessons learned, problems and opportunities to improve the actions, and their impacts. • Member States are required to publish all evaluations on the program website. • The ex-ante evaluation is no longer obligatory. Source: Draft CPR regulation. 18. The approach foreseen for 2021–2027 acknowledges that evaluations will continue to play a crucial role in assessing OPs’ relevance, effectiveness, efficiency, and EU added value. In the 2021–2027 period, Romanian authorities will be required to carry out and publish an OP-level impact evaluation by June 2029. Evaluations must be carried out by functionally independent evaluators and follow evaluation plans to be submitted to the EC within one year of adopting an OP. Against this background and given the WB’s long-standing engagement on M&E of government strategies and programs, the MEF requested WB support to strengthen the M&E capacity in the context of the 2021– 2027 ESF co-funded programs by establishing a new RAS Agreement. 1.2. Objectives 19. This document lays out the RAS team’s approach to supporting Romania’s MEF in strengthening M&E of EU-funded programs. It serves as the inception report for the RAS Agreement between the WB and the GoR and it is the first report to be delivered under this agreement. This report describes all activities to be conducted as part of the RAS, explains each activity’s methodological approach, and provides an indicative timetable and Gantt chart of all activities to be carried out. 20. The current exercise draws on the lessons of the 2014–2020 programming period and aims for improvements to be operational in the framework of the 2021–2027 period. As such, it begins with an assessment of the existing M&E system and legislative and institutional framework. It identifies the system’s strengths and weaknesses, as well as its main parameters (i.e., legal, operational, or technical). Beginning with the results of the comprehensive assessment, but also considering best practices at the international level, the project will define the basis for a more streamlined, effective, and efficient M&E system. Support for implementing the newly defined system is also planned, through the provision of detailed implementation plans and guidance, as well as staff trainings and selected hands- on initiatives. 21. This report describes the activities the RAS team will undertake to support the Ministry of European Funds in strengthening M&E, along with the anticipated methodology, timetable, 5 and working arrangements. The report is structured as follows: this section explains the scope of the engagement and the results targeted through the RAS activities. Section 2 provides background on the 2021–2027 programming period, overviews the current system for monitoring EU-funded projects in Romania, and outlines the specific M&E requirements at the EU level. This discussion also captures changes in the upcoming programming period that are motivating the MEF to pursue stronger M&E of EU co-funded programs. Section 3 lays out the methodology expected to best accomplish the deliverables under this RAS. Section 4 further explains how the team foresees implementing RAS activities. Section 5 summarizes deliverables, milestones, and timeline. Section 6 notes information gaps and support the RAS team requests from the MEF. Section 7 identifies the limitations of this analysis. Section 8 offers a brief stakeholder analysis to identify the key stakeholders who will be involved in implementing RAS activities. Section 9 details risks that might affect project implementation, as well as their possible solutions and mitigations. Section 10 briefly describes quality assurance mechanisms to be provided with respect to the data used, outputs, language and translation, and validation of deliverables. Section 11 highlights dissemination and completion activities. 2. General framework of the ESIF M&E system in Romania 2.1. Overview of ESIF for the 2014–2020 programming period 22. As an EU member, Romania benefited from a total allocation of up to €33 billion in ESIF funds during 2014–2020 (see Box 2). 15 Approximately €22.4 billion are allocated under the Cohesion Policy Funds 16 ERDF, European Social Fund (ESF), and the CF, with up to €400 million added from the Youth Employment Initiative (YEI). 17 Funds allocated to the development of the agricultural sector and rural areas are supplemented by €8 billion from the European Agricultural Fund for Rural Development (EAFRD) 18 and by approximately €168 million from the European Maritime and Fisheries Fund (EMFF). The available funding is completed by the allocations under the Territorial Cooperation Goal (Interreg). 19 15 https://www.fonduri-ue.ro/images/files/comunicate/2015/15.12/Alocari.programe.2014.2020.pdf 16 https://ec.europa.eu/regional_policy/en/policy/what/glossary/c/cohesion-policy 17 https://ec.europa.eu/social/main.jsp?catId=1176 18 https://ec.europa.eu/regional_policy/en/policy/what/glossary/e/european-agricultural-fund-for-rural- development 19 https://ec.europa.eu/regional_policy/en/policy/cooperation/european-territorial/ 6 Box 2: European Structural and Investment Funds (ESIF) Five European structural and investment funds are jointly managed by the EC and the EU. The ESIF funds mainly focus on the following areas: research and innovation; digital technologies; supporting the low-carbon economy; sustainable management of natural resources; and small businesses. The purpose of all these funds is to invest in job creation and a sustainable and healthy European economy and environment. • European Regional Development Fund (ERDF)—promotes balanced development in the different regions of the EU. • European Social Fund (ESF)—supports employment-related projects throughout Europe and invests in Europe’s human capital: its workers, its young people, and all those seeking a job. • Cohesion Fund (CF)—funds transport and environment projects in countries where the gross national income (GNI) per inhabitant is less than 90 percent of the EU average. In 2014–2020, these are Bulgaria, Croatia, Cyprus, the Czech Republic, Estonia, Greece, Hungary, Latvia, Lithuania, Malta, Poland, Portugal, Romania, Slovakia, and Slovenia. • European Agricultural Fund for Rural Development (EAFRD)—focuses on resolving the particular challenges facing the EU’s rural areas. • European Maritime and Fisheries Fund (EMFF)—helps fishermen to adopt sustainable fishing practices and coastal communities to diversify their economies, improving the quality of life along European coasts. Source: European Commission, https://ec.europa.eu/info/funding-tenders/funding-opportunities/funding- programmes/overview-funding-programmes/european-structural-and-investment-funds_en 23. Given the wide range of challenges facing Romania, all EU Cohesion Policy TOs 20 have been supported. The Partnership Agreement (PA) and the OPs detail the strategy, priorities, and arrangements for using ESIF funds during 2014–2020 (see Table 1). The funding is aimed at supporting strategic investments for tackling the country’s significant development challenges, income inequalities, and productivity gap, compared to more developed members of the EU. At the same time, EU funding is meant to help Romania unlock its economic potential, make use of opportunities, and avoid the pitfalls of the global economic environment. Approximately 51.2 percent of ERDF funds have been allocated to support research and development (TO1), information and communication technology (ICT) (TO2), and competitiveness (TO3). Significant resources are directed toward supporting the shift to a low-carbon economy (TO4)—30 percent of the ERDF allocation. The ESF’s share amounts to 30.8 percent, i.e., approximately €4.8 billion, out of which of 23 percent is targeted to social inclusion and the fight against poverty. 20 https://ec.europa.eu/regional_policy/en/policy/what/glossary/t/thematic-objectives 7 Table 1: Operational Programs, 2014–2020 Operational Program EU Fund EU Allocation Responsibl Relevant (million €) e Ministry Regulation 21 Investment for Growth and Jobs Goal 1303/2013 22 1 Human Capital (POCU) ESF+YEI 4,326.84 MEF 1304/2013 23 2 Competitiveness (OPC) ERDF 1,329.78 MEF 1301/2013 24 3 Large Infrastructure (LIOP) ERDF+CF 9,418.53 MEF 4 Technical Assistance (OPTA) ERDF 212.76 MEF 5 Regional (ROP) ERDF 6,700.00 MPWDA 6 Administrative Capacity (OPAC) ESF 553.19 MPWDA 1304/2013 7 Support Disadvantaged Persons (POAD) FEAD 440.0 MEF Territorial Cooperation Goal MPWDA 1303/2013 8 Interreg V-A Romania-Bulgaria 25 ERDF 258.5 MPWDA 1299/2013 9 Interreg V-A Romania-Hungary 26 ERDF 231.8 MPWDA 10 Danube Transnational Program 27 ERDF 262.9 MPWDA 11 INTERREG EUROPE 28 ERDF 432.3 MPWDA 12 URBACT III 29 ERDF 96.3 MPWDA 13 INTERACT III 30 ERDF 39.3 MPWDA 14 ESPON 2020 31 ERDF 48.6 MPWDA 15 INTERREG IPA Romania- IPA II 88.1 MPWDA 231/2014 & Serbia 32 447/2014 16 Romania-Ukraine 33 ENI 66.0 MPWDA 232/2014 & 17 Romania-Moldova 34 ENI 89.1 MPWDA 897/2014 18 Back Sea Basin 35 ENI 53.9 MPWDA 19 Hungary-Slovakia-Romania- ENI 81.3 MPWDA Ukraine 36 Source: MEF: http://mfe.gov.ro/programe/ and https://www.fonduri-ue.ro/po. 24. In line with the EU’s Better Regulation Agenda, 37 performance orientation has been a key priority for the implementation of ESIF funds during the 2014–2020 period. As such, the focus on results was streamlined in the relevant legislation and transposed at the level of the OP. The following key elements were introduced to improve soundness, traceability, and accountability across ESIF interventions: 21 This refers to the main regulation establishing the fund. Other implementing regulations may also apply. 22 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32013R1303 23 https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX%3A32013R1304 24 https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX%3A32013R1301 25 https://www.interregrobg.eu/ro/, 26 https://interreg-rohu.eu/ro/home/ 27 http://www.interreg-danube.eu/, 28 https://www.interregeurope.eu/ 29 https://urbact.eu/ 30 http://www.interact-eu.net/#home 31 https://www.espon.eu/ 32 http://www.romania-serbia.net/ 33 https://ro-ua.net/ro/ 34 https://www.ro-md.net/ro/ 35 https://blacksea-cbc.net/ 36 https://huskroua-cbc.eu/ 37 EC Communication Better Regulation: Delivering better results for a stronger Union, 2016 https://eur- lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52016DC0615&from=EN 8 • More streamlined programming and focus on the intervention logic, 38 which required all investments to be linked to the 11 EU thematic objectives, that Member States support their needs assessment with evidence, and that planned interventions be linked to a set of quantified results. • Ex ante conditionalities, 39 which required Member States to fulfill certain conditions for the effective use of ESIF funds by the end of 2016. These conditions are linked to policy and strategic frameworks, regulatory frameworks, and sufficient administrative and institutional capacity. The general ex ante conditionality no. 7 requires the existence of “a statistical basis necessary to undertake evaluations to assess the effectiveness and impact of the programs, as well as the existence of a system of results indicators necessary to select actions, which most effectively contribute to desired results, to monitor progress toward results and to undertake impact evaluation.” • A Performance Framework (PF), 40 which required the establishment of measurable milestones and targets for a number of indicators or key implementation steps. Under the Investment for Growth and Jobs goal, the PF is linked to a Performance Reserve, of 6 percent of OP funding, which is made available if the 2019 Performance Review shows the milestones set in the PF were met. • A common set of indicators to measure progress toward achieving results. 41 Indicators and corresponding targets must be set for each priority, with targets expressed in qualitative or quantitative terms. These indicators are the basis for monitoring, evaluating, and reviewing performance and are used to assess progress in program implementation with a view toward achieving objectives. 2.2. ESIF for the 2021–2027 period 25. In the 2021–2027 programming period, Romania is expected to receive approximately €30.6 billion of funding from ESIF: €17.3 billion through the ERDF, €8.4 billion through the ESF, €4.5 billion through the CF, and €392 million through the European Territorial Cooperation (ETC) Program. 26. In line with the EC’s proposals for the 2021–2027 period, the Partnership Agreement sets out the key development priorities and the structure for implementing OPs. These are then the key instruments for putting the objectives of the Cohesion Policy into practice at the national level. As such, Romania foresees interventions under all five policy objectives, which the EC defined as a more intelligent, greener, connected, and social Europe that is closer to its citizens. GoR has carried out six rounds of negotiations with the EC to date, covering all five policy objectives described in Box 3, and addressing the detailed intervention logic for each objective. 38 Art. 14, 15, 26 and 27, CPR. 39 Art. 19 and Annex XI, CPR. 40 Art. 20-22 and 96 and Annex II, CPR. 41 Art. 26, CPR and Fund-specific regulations. 9 Box 3: EU Cohesion Policy Objectives for 2021–2027 Five main objectives will drive EU investments in 2021–2027:  A Smarter Europe, through innovation, digitization, economic transformation, and support to small and medium-sized businesses.  A Greener, carbon-free Europe, implementing the Paris Agreement and investing in energy transition, renewables, and the fight against climate change.  A more Connected Europe, with strategic transport and digital networks.  A more Social Europe, delivering on the European Pillar of Social Rights and supporting quality employment, education, skills, social inclusion, and equal access to health care. • A Europe closer to citizens, by supporting locally led development strategies and sustainable urban development across the EU. Source: Art. 4, CPR. 27. In February 2020, the GoR approved a memorandum related to future OPs and the framework of implementing institutions. While the planned institutional set up builds on the existing (2014–2020) programming period’s management structure, it will be expanded, both in the number of OPs and institutions involved. 28. The future institutional structure includes 6 ministries, 19 MAs (8 of which are the Regional MAs) and at least 9 Intermediary Bodies (IBs), with some new OPs, such as the Health OP, the Financial Instruments OP, 42 and the Just Transition OP. The MEF will manage 7 OPs, financed from ERDF, ESF+, and the Just Transition Mechanism. At the same time, the MEF’s coordination role is expected to be enhanced compared to the 2014–2020 period, to increase the added value of EU investments and optimize the overall use of funds. The Certifying and Paying Authority in the Ministry of Public Finance and the Audit Authority will carry out their functions, similar to the current period. As such, the planned institutional structure is extensive, showing strong interrelations both among public authorities involved, as well as with regard to financing. 29. The new programing and institutional set-up pose important challenges. Along with the new requirements of the Cohesion Policy, selected OPs are also being supported through multiple funds. In addition, there is an increased territorial focus—as per the eight Regional OPs and enhanced arrangements for use of Integrated Territorial Investments (ITI), integrated urban development, and community-led local development (CLLD). These arrangements will raise important challenges, both with regard to the institutional framework for funds management and the actual implementation of interventions. 30. Although the new EU regulations provide for fewer, clearer, and shorter rules, as well as a flexible framework focused on simplification and performance, the M&E function will be more prominent than in the previous programming period. As such, the M&E provisions proposed EU Common Provisions Regulation (CPR) 43 include transmitting financial data quarterly and physical indicator data to the EC every two months, increased harmonization through the use of a common set of indicators linked to specific objectives, performance linked to both output and results indicators, extended use of simplified cost options (SCOs), and a midterm review of all funds in 2024. 2.2.1 Monitoring component of the ESIF M&E system 31. This section details the set up and requirements related to the ESIF monitoring function. The next section details the evaluation function. 42 The Intelligent Growth and Digitalization OP was renamed Intelligent Growth, Digitalization and Financial Instruments OP. 43 Proposal for a Regulation of the European Parliament and of the Council laying down common provisions on the European Regional Development Fund, the European Social Fund Plus, the Cohesion Fund, and the European Maritime and Fisheries Fund and financial rules for those and for the Asylum and Migration Fund, the Internal Security Fund and the Border Management and Visa Instrument https://eur-lex.europa.eu/legal- content/EN/TXT/?uri=COM%3A2018%3A375%3AFIN 10 32. Monitoring can be defined as “the continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives.” 44 Thus, monitoring embodies the regular tracking of development activity inputs, activities, outputs, and outcomes at the project, program, sector, and national levels. Monitoring generates quantitative data and gives feedback on the implementation of instruments/measures, facilitating corrections of deviations from operational objectives. Monitoring helps make public spending accountable and provides valuable information to be used in the evaluation of instruments/measures. 33. Within the framework of ESIF funds implementation, monitoring is an integral component of the results-oriented approach and is structured around the OPs. It builds on the program’s logic of intervention and supports observing progress and partial results as implementation progresses. It also provides early detection of any deviations from how activities were planned initially. In addition, monitoring activities intend to support evidence-based decision making, facilitate communication, and inform evaluation at the OP level (see Section 4). 34. EU regulations (CPR and fund-specific regulations) establish most requirements for the M&E function. National authorities transpose these regulations at the PA and OP level, and the EC and the Audit Authority check and approve this transposition during the comprehensive accreditation process. However, the actual instruments for ensuring its implementation are program-specific, such as POCU-Form (see Box 4). Box 4: POCU-Form, the POCU instrument for collecting indicators The POCU-Form.B package is an Excel-based instrument that covers the following stages of data collection at the beneficiary level, and their reporting to the IBs and MA: • Registration of participant-related data in POCU operations (through the Individual Registration Form). • Registration of data of entities participating in POCU operations (through the Entity Registration Form). • Automatic validation of numerous information entered in the different sections of the Individual Registration Form and the Entity Registration Form. • Automatic centralization of the data regarding the target group of the project (within the registration form for the target group). • Automatic extraction of data on common and program-specific indicators, based on recorded data. The data are centralized for each project (in the Indicator Registration Form) and are transmitted to the IB in a format that will allow centralization at the desired level and size. The POCUForm.OI and POCUForm.AM packages allow the centralization of data on common and program-specific indicators, at the level of the IB/MA, in order to facilitate bi-annual/annual reporting to the EC. Source: MEF; Annex 17, http://mfe.gov.ro/pocu/7-lista-anexelor/. 35. Both the CPR and fund-specific regulations set out monitoring and reporting requirements that apply at the OP level. The monitoring systems set up by the MAs at the OP level must ensure proportionate reporting requirements, as well as the availability of comprehensive information on progress made at key review points. They must also reflect the informational needs in a given year and be timed alongside the Performance Review, as described in Box 5. 44 OECD 2002. 11 Box 5: Performance Review The Performance Review is carried out in line with the provisions of the CPR, based on the information and assessments presented in the annual implementation report submitted by Member States in 2019 (for 2014–2020) and in 2025 (for 2021–2027), through the SFC2014 system.a The EC determines which programs and priorities have achieved their milestones. Milestone achievement is assessed separately for each of the funds and each regional category,b taking into account the indicators, their milestones, and their achievement values. Depending on the outcome of the Performance Review, the Performance Reserve (for the 2014–2020 period) or the remaining allocation (for the 2021–2027 period) may be allocated definitively. Generally, the proposed reallocation of the Performance Reserve must be consistent with the thematic concentration requirements and minimum allocations, as set out in the regulations. The allocation of the Performance Reserve may lead to modifications in the targets set for indicators, as well as to other changes in the OP. Notes: a. SFC2014 is the common information system of the EC to manage jointly with Member States the implementation of several EU funds, including ESIF; b. Under the Cohesion Policy there are three regional categories: less developed regions (where GDP per inhabitant was less than 75 percent of the EU average); transition regions (where GDP per inhabitant was between 75 percent and 90 percent of the EU average); and more developed regions (where GDP per inhabitant was more than 90 percent of the EU average). Source: Regulation 1303/2013 (CPR) and New CPR (draft). 36. Taking a systems perspective, the monitoring of ESIF funds parallels its multi-level implementation structure. As depicted in Figure 1, specific stakeholders have a monitoring mandate at each level of implementation. Figure 1: Monitoring mandate per implementation level EC Fund EC, CCMAP, MEF, MAs, MCs Operational Program EC, AMs, MCs, IBs Priority Axis/ Priorities EC, AMs, line ministries, GoR Investment Priority/ Specific MAs Objective MAs, IBs, line ministries (as Projects beneficiaries Participants, Entities MAs, IBs, EC (for common indicators) 37. These mandates are operationalized through monitoring activities at each level. As shown in Figure 2, monitoring activities include project-level monitoring, IP monitoring, and Priority Axis- level, OP-level, and PA-level monitoring. Moreover, these monitoring activities cascade up into the monitoring activities at the next level, leading to an integrated monitoring function. This cascade occurs specifically through the reporting products each stakeholder is responsible for producing. 12 Figure 2: Monitoring and reporting flow • EC and CCMAP •Observe the implementation across OPs, for each EU fund • MAs • Product: annual implementation report • MAs •Obtain information on the use of financial resources & indicator achievement •Geared toward evidence-based decision making • Product: annual implementation report • Monitoring officers track multiple projects • Beneficiary: observe, document, and report on the progress of the project (also called monitoring data) • Product: project technical report 38. The monitoring process starts at the project level, where beneficiaries and monitoring officers track progress on objectives and results, realization of indicator targets, financial monitoring, and milestones. Once the project is approved and contracted, data from the financing application (objectives, results, targets, and financial values) becomes the baseline and reference point for project implementation and monitoring. Beneficiaries are responsible for observing, documenting, and reporting on the project’s progress (called monitoring data). Monitoring also tracks project management, whether activities are implemented on schedule, respect for equal opportunities and non- discrimination, state aid, and sustainable development. During the sustainability period, monitoring ensures that projects maintain results (and indicators), as well as respect the principles of equal opportunities and non-discrimination. To this end, monitoring officers use document analysis and verification (primarily), special (ad hoc) onsite visits, regular onsite visits, cross visits, ex post monitoring, and verification of data uploaded into the electronic management systems (see Box 6). If indicator targets are not met, the value of the contract is usually diminished, meaning that the beneficiary is penalized. 13 Box 6: Systems for electronic data exchange SMIS 2014+ is a web-based system used only by authorities for the eight OPs under the Cohesion Policy. It covers the entire project life cycle and includes additional dedicated modules for programming, evaluation, and audit. It contains the relevant information, rules, and controls for ERDF, CF, and ESF as specified in the general regulation and in the delegated and implementing acts. MySMIS is a web-based system that allows data exchange between beneficiaries or potential beneficiaries and authorities. MySMIS is integrated with SMIS and covers the entire project life cycle. As the monitoring component of MySMIS was not functional until recently, some MAs have set up additional data collection instruments, such as the POCU Indicators Collection Instrument. eMS is a is a web application developed by the Interact Program for the benefit of all Interreg programs. It is a monitoring system with a communication portal to support the submission, appraisal, approval, contracting, monitoring, and management of projects under Interreg programs (financed by the ERDF, IPAII and ENI). The system supports the collection of all information on submitted as well as approved projects, their implementation and achievements, the necessary changes in implementation as well as project completion. SFC2014 is the EC’s electronic information exchange that facilitates shared fund management between Member States and the EC, as mainly described in Article 74.4 of EU Regulation no. 1303/2013. For the 2014–2020 period , this common information system is a platform to facilitate the joint management of the DG REGIO, DG EMPL, DG AGRI, DG MARE and DG HOME funds. The set-up and functioning of the SFC2014 is established in the Commission Implementing Regulation 184/2014,a as per EU Regulation no. 1303/2013 regarding ESIF funds. Notes: a. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014R0184. Source: MEF, EC, Interreg RO-HU 39. Each project is assigned a monitoring officer and represents “the interface” between the MA and the beneficiary. Monitoring officers are based either in the IBs, Regional Development Agencies (RDAs), the JSs, or the MAs, and their responsibilities are specified both in the institution’s internal procedures and the individual job description. Thus, the monitoring officers play a major role in the overall M&E process and are directly responsible for observing the project. A monitoring officer may have several projects assigned. MAs set up the actual procedures for project monitoring, including the monitoring and reporting process during implementation, data to be collected, and the actions for the technical verification of reimbursement claims by beneficiaries. 40. Project monitoring cascades up to the OP and IP level, where MAs perform the monitoring. This monitoring focuses on obtaining and delivering quantifiable, accurate, and reliable information with respect to the use of financial resources and the achievement of indicators. 45 Through the data collected, MAs can meet reporting requirements and make evidence-based decisions, thereby contributing to improved program management. 41. Annual implementation reports (AIRs) are the key documents for monitoring OPs. As per the regulations, implementation reports are required to set out certain information46 on the OP’s financial and physical progress. AIRs also include information on any issues affecting program performance, including the achievement of target values. The reports submitted in 2019 also assess progress toward achieving the program objectives, and on the OP’s contribution to achieving the EU’s strategy for smart, sustainable, and inclusive growth. 47 In addition, the MAs draft other various reports and collect additional data. This is usually process-related and offers insight into the number of calls for proposals, submissions, and the application process, as well as overall demand for funding under specific operations, results of the evaluation, bottlenecks, quality of the project proposals, and contracting. Other 45 Physical indicators refer to all indicators except financial ones. 46 Starting with 2016. 47 https://eur-lex.europa.eu/legal-content/en/ALL/?uri=CELEX%3A52010DC2020 14 information is related to specific topics of interest, such as integrated territorial instruments 48 or other data at the sector level. 42. Finally, monitoring at the OP, IP, and PA levels feeds into the monitoring activities of the Monitoring Committee (MC) and the EC. Based on the information MAs provide, these stakeholders examine topics related to OP implementation, including progress on achieving objectives, indicators, and overall performance, as well as implementation problems. At the same time, these stakeholders provide feedback to the MAs in the form of observed progress on OPs, and then monitor how MAs follow up on these observations in the form of corrective actions. If systemic deficiencies (as defined by the regulations) are identified in OP implementation or management, the EC may suspend payments (see Box 7). Box 7: Suspension of payments by the EC The EC may suspend all or part of the interim payments at the level of IP or OP if one or more of the following conditions are met:  there is a serious deficiency in the effective functioning of the OP’s management and control system, which has put the EU’s contribution to the OP at risk and for which corrective measures have not been taken;  a statement of expenditure is linked to an irregularity that has serious financial consequences and has not been corrected;  the Member State has failed to take the necessary action to remedy the situation, leading to an interruption of the payment deadline by the EC;  there is a serious deficiency in the quality and reliability of the monitoring system or of the data on common and specific indicators;  there is a failure to complete actions to fulfill an ex ante conditionality; and  the Performance Review shows evidence that there has been a serious failure in achieving a priority’s milestones relating to financial and output indicators and key implementation steps set out in the Performance Framework. Source: Art. 142, CPR. 43. The EC and the Committee for the Coordination of the Partnership Agreement (CCMAP) monitor implementation across OPs, at the level of the PA and each EU fund. Data from program- level implementation is transmitted by the MAs via the SFC2014 electronic system and through the implementation reports (especially the AIRs). The EC also has a representative in the MC. 44. In sum, these monitoring activities are linked across implementation level to allow for an integrated monitoring function. In turn, monitoring activities involve multiple stakeholders and a variety of data collection, analysis, and reporting tools and mechanisms. The main stakeholders involved in monitoring ESIF-funded interventions are: (i) beneficiaries; (ii) IBs, Regional IBs, including RDAs or BRECO; (iii) MAs and national authorities (for cross-border programs); (iv) MCs and CCMAP; (v) the MEF; and (vii) the EC (see Figure 3). 48 These include Integrated Territorial Investments (ITI), community-led local development (CLLD), and other place-based instruments. 15 Figure 3: M&E function and process 45. Table 2 details specific data collection, validation, reporting, and performance management tasks for each stakeholder, as well as at which implementation/monitoring level they perform these tasks. Table 2: Tasks for monitoring interventions funded through ESIF Stakeholders Data Data Data Performance collection validation reporting management C P O P P O P P O P P OP PA P A P A P A Beneficiaries X IBs, including Regional IBs, RDAs, X X X X X X and BRECOs MAs X X X X X Monitoring Committees X MEF X X X X CCMAP X X Functional Working Group for X Performance Assessment Statistical Office, research X institutions, public institutions Note: C = context; P = project level; OP = Operational Program, Priority Axis/Investment Priority/regional level; PA = Partnership Agreement level/thematic objective. Source: Authors’ analysis, based on OPs and available procedures. 46. Looking specifically at indicators, the CPR and fund-specific regulations establish the indicator system and system of common indicators used in ESIF monitoring. The indicator system consists of output, as well as results indicators. ESF results indicators are immediate and longer-term. Indicators are of two types: common (established and observed at the EU level) and specific (established and observed at the national level). While common indicators are applicable at the fund level, the 16 majority of indicators are specifically designed for each program. 49 Indicators also rely on ESIF-specific definitions of participants and beneficiaries as described in Box 8. Next, ESF immediate results indicators reflect the change in situation at the end of support, while longer-term results indicators reflect the situation after the end of support, usually over a period of six months. The indicator system also includes financial indicators. Box 8: Participants vs. beneficiaries in ESIF interventions Participants are persons who: (i) receive direct support from the ESIF-funded intervention; (ii) can be identified; (iii) who can provide information on their characteristics; and (iv) for whom specific expenditure is earmarked. All four conditions apply cumulatively. Entities are legally constituted organizations. They can either implement—fully or in partnership— or be supported by projects. In the former case, they have signed a financing contract with the MA and are considered beneficiaries. In the latter case, in the same way as for participants, entities are counted when they benefit directly from support that incurs expenditure, for monitoring purposes. Beneficiaries are not usually considered as entities benefitting from support, except for state aid or de minimis schemes. Source: ESF Regulation 1304/2013, available at: https://eur-lex.europa.eu/legal- content/EN/TXT/PDF/?uri=CELEX:32013R1304&from=en. 47. Finally, for the monitoring function to work well, its underlying components and tools must be effective and well-aligned. Such underlying components include (i) how responsibilities are assigned across institutions implementing ESIF funds; (ii) the capacity of institutions and beneficiaries; (iii) overall endorsement of an M&E culture at the government level, among implementing agencies and bodies; and (iv) data availability and quality. Given the large number of stakeholders involved in monitoring, the institutional set-up is particularly important for ensuring flows of information. Yet as the majority of implementation data comes from projects, it is equally important to have quality guidance and data collection instruments. Not least, M&E quality is very much demand-driven: if decision makers are using and demanding high quality M&E information, then stakeholders will be incentivized to generate such information. Both the supply and demand sides must be engaged effectively in order to achieve a well-functioning M&E system. 2.2.2 Evaluation component of the ESIF M&E system 48. Evaluation can be defined as “the process of determining the worth or significance of a development activity, policy or program.” 50 As per CPR 1303/2013, “evaluations assess the effectiveness, efficiency and impact of the program,” 51 contributing to the improvement of its implementation and providing solid information with respect to what works (and what does not) and why. One of the primary tasks of an evaluation is to identify the effects that can be directly attributed to relevant interventions. Evaluation can also address the effectiveness of the mechanisms used, so as to enable solutions for their improvement. While often regarded as cumbersome or having limited use, the independent assessment of progress can result in more diligent and strategic operational-level activities. 49. Article 54 of EU Regulation no. 1303/2013 establishes the general framework for evaluating ESIF funds for the 2014–2020 programming period. Ex ante evaluations are primarily meant to improve the design quality of each program. In turn, evaluations conducted during implementation and ex post assess effectiveness, efficiency, and impact. These evaluations examine the program’s contribution to the EU’s strategy for smart, sustainable, and inclusive growth, taking into account targets established in that strategy and in accordance with specific requirements established in 49 Definitions of output, immediate results, and longer-term results indicators are presented according to European Commission (EC) Monitoring and Evaluation of European Cohesion Policy, European Social Fund Guidance document, available at: https://ec.europa.eu/sfc/en/system/files/ged/ESF%20monitoring%20and%20evaluation%20guidance.pdf. 50 OECD 2002. 51 Art. 54.1, CPR. 17 the fund-specific rules. At least once during the programming period, an evaluation should assess how support from ESIF funds has contributed to the objectives for each investment priority. Evaluations across all programs under the Partnership Agreement are also encouraged. 50. Evaluations during the programming period are tailored to program needs. Each OP establishes an evaluation plan that identifies the evaluation needs and specific evaluations foreseen. Member States are responsible for evaluations carried out ex ante evaluation and during implementation, while the EC, or Member States cooperating closely with the EC, carry out ex post evaluations. Moreover, evaluations can cover programs, priorities, and themes across programs. A December 2022 summary report for each OP will summarize main evaluation findings, assess main outputs and results, and comment on the information reported (Art. 114.2, CPR). One of the main purposes would be to feed into the ex post evaluation under the lead responsibility of the EC. 51. The EC encourages Member States to include evaluations of interventions implemented in the previous programming period in the current or subsequent period. The logic is that it can take years for the effects of an intervention to be fully achieved or to materialize (e.g., for large-scale infrastructures, research and technological development projects, etc.). Accordingly, the MEF Evaluation Plan comprised evaluations of several interventions from the 2007–2013 programming period. 52. MCs must examine evaluations and the recommendations that follow, and then pass these to the EC, preferably in electronic format. The MCs may issue recommendations to the MAs regarding evaluation of the program and should monitor the actions that the MAs have taken as a result of its recommendations. 52 53. Evaluators must be functionally independent of authorities responsible for the program’s preparation and implementation. 53 The EC provides extensive guidance to Member States on how to set up and conduct the evaluation, so as to ensure the appropriate level of independence and impartiality. This is necessary to obtain unbiased expert opinions with respect to the subject being assessed, while also ensuring that evaluators are adequately acquainted with the interventions they assess. The EC services consider it best practice to assign the evaluation to external experts or to an organization that is not responsible for implementing the program, nor to any intermediate bodies reporting to it. 54. Based on these requirements, the Romanian authorities have set up an evaluation system with two types of evaluation plans—at the level of the Partnership Agreement and for each OP. At the PA level, the evaluation plan addresses themes such as Europe 2020, gross domestic product (GDP), unemployment, country-specific recommendations, 54 and EU thematic objectives. 55 It also considers the implementation mechanisms and the horizontal aspects of the system in which the funds are implemented. At the OP level, evaluation plans take a specific meso (OP level) and micro (project) level focus and address issues related to the programs’ specific objectives. 55. The evaluation plans state the evaluation strategy, main evaluation themes, and recommended methods. Evaluation plans aim to support evidence-based decisions and are designed to support the preparation of progress reports to be submitted to the EC, including evaluation conclusions and findings. These evaluation plans also support improvements in the overall the quality of evaluations, through appropriate planning, identifying the evaluation’s data needs, and by ensuring that adequate funding and management resources are made available for evaluations. Based on the OP, the plan will include a number of evaluation themes. 56. The evaluation function is based on three pillars: the Programs Evaluation Office (PEO), the Evaluation Steering Committee, and the Scientific Committee. Together, they are responsible 52 Art. 49.4, CPR. 53 Art. 54, CPR. 54 https://ec.europa.eu/info/publications/2020-european-semester-country-specific-recommendations- commission-recommendations_en 55 https://ec.europa.eu/regional_policy/en/policy/what/glossary/t/thematic-objectives 18 for coordinating and implementing the Evaluation Plan, as well as monitoring and promoting the quality of evaluation activities throughout the entire evaluation cycle. 57. The PEO established within the MEF is part of General Directorate of Analysis, Programming, and Evaluation. The PEO’s mission is twofold: • to ensure a coordinated national evaluation system and to develop OPs’ evaluation capacity; and • to plan and manage the evaluations of the Partnership Agreement and those of the programs for which MEF acts as the Managing Authority. 58. Following up on the EC’s recommendation, each MA outside the MEF has set up a separate evaluation unit responsible for designing and implementing their evaluation plan, with the support of independent evaluators. A common Evaluation Unit supports the evaluation function at the OP level under the Territorial Cooperation Goal. 59. The Evaluation Steering Committees includes the main actors—either public institutions or economic and social partners—that are interested in the evaluation results or are essential for implementing the evaluation, such as data providers. Five Steering Committees are established at the PA level, one for each evaluation topic 56, and separate committees are set-up for each OP. 60. The Scientific Committees supports the evaluation, by ensuring the quality of evaluation at key times: this includes completing terms of reference, accepting the methodology proposed by the evaluator, and approving various versions of evaluation reports. 61. The Functional Working Group for Performance Assessment, at the level of the PA, is also responsible with the coordination, complementarity and synergies between the operational programs and their evaluation activities, including their evaluation plans. 62. Essential to evaluation is getting uptake of the evaluation findings and recommendations. Specifically, the results and recommendations should speak to authorities responsible for designing and implementing the program that was evaluated. Thus, it is crucial that results are effectively communicated to relevant stakeholders. To this end, the evaluation plans include measures to communicate results and monitor actions that follow from the evaluation results. Such strategies envisage three main target groups: i. those responsible for managing and conducting evaluations (evaluators, managers of evaluation, evaluation and scientific coordination committees, data providers); ii. users of evaluation (policymakers and social partners who use evaluation results, supporting the process of drafting public policies); and iii. the general public, which has an interest in good governance. 63. The functionality and performance of the evaluation function are contingent on a wide array of factors. These include the design and preparation of the evaluation, evaluators’ independence and expertise, overall endorsement of an M&E culture, and data availability and quality. Not least, communicating and using the results of the evaluations to inform decision making is an essential part of a well-performing monitoring system. 3. Methodological framework and scope of proposed work 3.1. Key elements of the work program 56 The evaluation themes at PA level are the following: (i) evaluation of the contribution to the smart, inclusive and sustainable growth; (ii) evaluation of the relevance of the development needs and of the contribution to meeting the EU thematic objectives and the Country Specific Recommendations; (iii) evaluation of applying the horizontal principles; (iv) evaluation of the capacity and the coordination mechanisms for implementing ESIF; and (v) evaluating progress in meeting indicators targets in the Performance Framework. 19 64. The objective of this engagement is to support the GoR as it builds capacity to develop and operationalize an M&E system to track and monitor programs co-funded by EU funds in the 2021–2027 programming period. The core activities of this RAS are geared toward learning what has and has not worked with the M&E system under the 2014–2020 period, determine options for system improvements, and support the GoR in implementing improvements. This strengthened M&E system will contribute to: (i) measuring results toward the achievement of the objectives of EU co-funded programs, as set out by the Partnership Agreement and OP documents; (ii) informing programming decisions and broader policies based on data and regular reports from the system; (iii) strengthening transparency and accountability related to the allocation of European funds and thus increasing the effectiveness of EU co-funded measures to support Romania’s development trajectory; and (iv) enhancing the capacity within the MEF and other agencies to manage data, as well as to monitor and evaluate EU co-funded programs. 65. To achieve the expected results, the RAS will focus on three activities: • Activity 1: Assessment of the existing M&E system of EU-funded measures. • Activity 2: Options to improve the M&E system of EU co-funded investments. • Activity 3: Propose improvements in the M&E system. 66. Figure 4 illustrates a results chain for the RAS, including the inputs to be applied in activities, outputs generated, short- and medium-term outcomes, and anticipated final impacts. 67. The RAS is expected to be implemented through June 2023 in close working partnership with the MEF technical team. Through this engagement, the WB aims to enhance the GoR’s capacity to better leverage EU co-funded investments in Romania by improving M&E systems and maximizing the impact of investments. In drafting this report, the RAS team builds on initial documentary review and input from the MEF technical team. The Results Framework for the M&E RAS is detailed in Figure 4. Figure 5 provides more detailed illustration of the theory of change (ToC) for the RAS activities. Figure 4: RAS’s proposed results chain Inputs Activities Outputs ST Outcomes MT Outcomes Impact Assessment of the • Improved capacity of Increased knowledge capacity to existing M&E * Methodological report and on the overall design, measure results systems to track policy brief for non-technical strengths and toward the and measure the audiences weaknesses of the achievement of results of EU co- * Full assessment report existing M&E system the objectives of funded Staff EU co-funded investments Fully programs Shared vision and operationalized Knowledge • Informed Developing * Technical report summarizing improved definition M&E system of program and options for findings and recommendations of operational and to track and International policy decision- improving the * Draft indicator table and institutional measure M&E best making process M&E system evaluation plans arrangements for the results of FSIF practices • Increased M&E system Funds in transparency * Technical report summarizing Improved Romania Budget and implementation plan and TNA and competences of the accountability training and capacity-building staff Implementing key • Enhanced plan improvements in capacity of * Technical trainings Clearly defined plans the M&E system MEF and other * Selected capacity-building and set-up for agencies in the support implementing field of M&E * Training assessment report recommendations 20 Figure 5: Theory of change for the M&E RAS Fully operationalized M&E PDO system to track and measure results of ESI funds in place Enhanced capacity within Improved capacity to measure MEF and other agencies to results toward achieving the Strengthened Informed decision and policy manage data, and to monitor R objectives of EU co-funded transparency and and evaluate EU co-funded programs making process in place accountability programs Data collection and The relevant staff Data is structured as to IR 1 The system can produce M&E data can OP Evaluation Plans reporting are carried (MEF/other agencies) has allow adequate accurate and consistent understanding of progress contribute to improving and proposals for out in line with the the necessary competences day-to-day activity and data sets, in support of achieved and support rigorous IE are procedures defined for understanding, performance (A2, A3) M&E activities (A2) decision making (A2) elaborated (A2) (A2) interpreting and using the IR 2 Effective M&E procedures Key indicators for Timely availability of are in place at the level of measuring performance quality and relevant the institutions involved are in place (A1, A2) data (A2) (A1, A2) Assess the capacity of existing Main assumptions/hypotheses: M&E systems to track and Develop options for improving Implementing improvements to H1. Data is available as to allow for adequate assessment of the capacity of the the M&E system (A2) the M&E system (A3) existing M&E system. measure the results of EU co- H2. Relevant actors (MEF, line ministries) are engaged in the project facilitating funded investments (A1) assessments and implementation of recommendations. A2.1. Define an overall vision for the H3. The recommendations for improving the M&E system will be A1.1. Stocktaking of the performance of applied/operationalized across the entire EU funds system. M&E system A3.1. Develop an implementation plan, M&E systems in 2014-2020 H4. MEF will be able to actively assume the coordination role and take the including a TNA A2.2. Define institutional and operational necessary measures for improvement and correction, where relevant. A1.2. Assess institutional, legislative and aspects H5. The development of IT systems and procedures will support the procedural framework of M&E of EU co- implementation of recommendations formulated (e.g., data collection and A2.3. Support ongoing work on A3.2. Develop a TCBP in line with the funded investments developing indicators for each OP validation, reporting, data sharing, etc.). TNA A1.3. Assess beneficiary feedback on H6. A functional collaboration system is in place at the system level, allowing A2.4. Identify opportunities for reporting requirements, procedures and implementing rigorous impact evaluations effective implementation of M&E recommendations, as to achieve the planned instruments based on an online survey of (IE) at the level of projects (interventions) A3.3. Trainings and capacity results. a sample of beneficiaries of programs in OPs 21 building activities H7. Relevant staff participates in trainings and capacity building activities and the 2014-2020 period, across all OPs PA2.5. Support producing a applies newly acquired competences in day-to-day work. involved proposed evaluation plan for each H8. Changes in the legal and regulatory framework do not affect project implementation and achievement of expected results. 68. The WB supports a highly participatory approach. To this end, the RAS team will work in close partnership with the MEF technical team, relying on them to access relevant administrative and survey data needed to carry out the activities, validate outputs, and facilitate communications with managerial and technical staff within the MEF, with other government agencies, as well as relevant stakeholders. Selected outputs will be further discussed with external stakeholders (representatives of municipalities, civil society organizations, private service providers, and academia) at consultative events jointly organized by the MEF and RAS teams. 69. An M&E Working Group will be formed playing a consultative role in RAS implementation. This group will be comprised of representatives from relevant line ministries, subordinate agencies, and the National Statistical Institute. 3.2. An integrated M&E framework 70. The MEF has set out to strengthen its M&E of EU co-funded investments by identifying challenges and successes in the previous programming period, determining opportunities for improvement, and seeking advice on how to best implement these. Accordingly, the RAS team designed three activities to support the MEF. First, the RAS team will undertake a comprehensive stocktaking of M&E performance in the past programming period. This exercise will inform the second and third activities: options for M&E system improvements (Activity 2) and support to the MEF in the implementation of the selected option (Activity 3). Thus, this section proposes the methodology for each activity and sub-activity. As the methodology for Activities 2 and 3 will depend heavily on the stocktaking outcomes, the discussion focuses on the methodology for Activity 1. Figure 6: An integrated system: measuring M&E performance at four levels Na ti o n a l /EU Go a l s an d Ob j e cti ve s I (p e rfo rm a n c e me a su re s) II ry Pl Se c to r/M i n i s t ans an d Ob j e ve c ti s (p e rfo rm a n c e me a su res) OP Stra te g y fo r Im p inggoa l e me n t ls III IV Pro g ra m m e s , Su b -p ro gra mm e s a n d Ac ti v i ti e s (pe rfo rma nc em ea su re s ) 71. In assessing the current M&E system and proposing potential improvements, the RAS team will use an integrated M&E framework as a reference point. Figure 6 is a stylized framework 22 showing a hierarchy of implementation levels, broadly applied to ESIF’s M&E system. 57 Such a framework offers a coherent logic of how results from programs, sub-programs, and OP activities contribute to sector or ministry plans and objectives, and finally cascades up to national/EU goals and objectives. This logic is measured through specific indicators leading to an integrated framework consisting of a nested hierarchy of results. Such a framework can be developed starting from stated goals. A broad strategy or directional statement would presumably provide a roadmap for converting those goals into strategies and action plans. In other words, there would be a logical link with the expected results at the national level, and a clear path for how to get there. This alignment of expectations underscores the importance of measuring performance at these four levels. 72. Such a framework should identify expected results and appropriate performance indicators at each of the three levels: national/MEF, OP, and program/project. Additionally, it should be noted that these represent different levels of aggregation in terms of indicators and data needed to populate them. They also represent different levels of performance information serving the needs of different “users” of M&E information. 73. Both the assessment of the current framework and the development of the next one will be aligned with the ESIF multi-level implementation structure (Figure 1). ESIF supports the delivery of the EU Cohesion Policy goals, set out in 11 thematic objectives (for the 2014–2020 period) and 5 policy objectives for the 2021–2027 period. These are implemented through OPs, comprising a series of investment priorities or specific objectives (for 2021–2027). The investment priorities are largely related to different policy sectors or subsectors (such as health infrastructure and services). OPs are implemented through projects (financing contracts), targeting participants, entities or issues on several topics. As such, the assessment will have a comprehensive, top-down approach, covering both the strategic (fund, OP, sector) and the specific (project, participant) levels. To ensure that assessment on the first programming period is relevant for the coming period, the team will check how the investment priorities align across the programming periods. Then, the assessment will be performed keeping the 2021–2027 objectives in mind. 74. This integrated M&E framework can facilitate effective performance management and improved programming, as well as exploit potential planning and implementation synergies. It depends on strong coordination among different OPs and corresponding interventions. With an integrated M&E framework, users can identify complementarities and potential synergies at the strategic, programming, and project level. As such, the M&E can support decision making at all relevant levels. 3.3. Methodological approach: Assessment framework and data collection 75. The proposed methodological approach is geared toward designing and implementing an improved ESIF M&E system that is better adapted to the needs of stakeholders. The M&E system will aim to facilitate increased efficiency and effectiveness in the management of EU funds. Figure 7 illustrates the RAS team’s approach, which takes the experience and lessons from the 2014–2020 period, identifies M&E needs and requirements of the 2021–2027 period and applies both in designing M&E system improvements. 57 Figure 6 is a reference point for communicating an integrated system of performance levels. The team acknowledges that the ESIF M&E system does not correspond exactly, as it does not measure performance at ministry level, but only for Cohesion Funds. Moreover, the lower layers (programs, subprograms) are also relevant to OPs as these are also considered programs. 23 Figure 7: Main project tasks 2. Support the 3. Support the 1. Assess the 2014– design and set- implementation 2020 system up of the 2021– of the 2021– 2027 system 2027 system 76. The RAS team anticipates employing a comprehensive yet well-defined stocktaking approach that builds on key takeaways on results-based M&E. 58 Results-based M&E is a powerful public management tool that can be used to help policymakers and decision makers track progress and demonstrate the impact of a given project, program, or policy. It differs from traditional implementation- focused M&E in that it moves beyond emphasis on inputs and outputs to a greater focus on outcomes and impacts. The RAS team selected this emphasis based on experience with M&E systems and familiarity with the M&E literature. Annex 2 notes some key results-based M&E takeaways from the literature, while Annex 1 describes the theory of change adopted to the Romanian context. 77. The RAS team foresees using a combination of an M&E assessment framework and Kusek and Rist’s Ten Steps to a Results-Based Monitoring and Evaluation System. The assessment framework will provide a holistic view of the M&E system, covering its scope and objective, policy and legal foundations, institutional architecture, functions, roles and responsibilities, tools, capacity, and incentives. Box 9 explains each of these dimensions. In addition, the assessment approach will take into account whether the current system includes an integrated M&E framework as described above. Box 9: Overview of M&E assessment framework The assessment framework arrives at a holistic picture of the M&E system by reviewing each of the following dimensions and gathering input from M&E system stakeholders. • Scope & objective: What is the overall vision for the M&E system, its purpose and the coverage? • Policy, legal, and planning foundation for M&E: What are key policies, practices, and frameworks needed to support an effective M&E system? • Institutional architecture: Who are the M&E stakeholders and what are their respective mandates within the M&E system? • Functions: What are the core functions of the system? These usually refer to monitoring and evaluation, though sometimes reporting is identified as a separate function. • Roles & responsibilities: Who is responsible for carrying out what M&E activities (and tasks, when we get to the more operational level)? • Tools: What do the relevant stakeholders use to carry out their M&E responsibilities (for example: M&E plans, data collection tools, etc.). • Capacity & incentives: What is the capacity of the system in terms of human resources, staff skills to collect data, analyze, report, manage evaluations, and disseminate results? Source: Authors; Jespersen 2017. 78. Then, the team will borrow from the Ten Steps to undertake more in-depth analysis on M&E plans, results frameworks, indicators, and reporting. This combined approach can also help identify and recommend options for improving the M&E system design and implementing the selected option for improvements. Beginning with the M&E assessment framework, this tool facilitates analysis of the different M&E system dimensions and was based on the World Bank’s experience with assessing 58 Ten Steps to a Results-Based Monitoring and Evaluation System (Kusek and Rist, 2004). How to build M&E systems to support better government (Mackay, 2007). M&E system assessment framework and Sector M&E assessment framework (unpublished). 24 M&E systems in other countries. The tool itself consists of detailed questions for each dimension and a questionnaire to use in interviews and desk reviews. Though the framework has one questionnaire adapted to the national M&E system, and another for sectoral systems, the team foresees tailoring the framework to the Romanian context. The team has started adapting the framework accordingly: Table 3 lists indicative questions under each dimension of the framework that will be addressed initially through the stocktaking. The forthcoming methodological report will include the full framework and questionnaire. The assessment framework arrives at a holistic picture of the M&E system by reviewing each of the following dimensions and gathering input from M&E system stakeholders. Table 3: Dimensions of the M&E system assessment framework and core questions Framework dimension Indicative questions Scope & objective • What is the overall vision for the M&E system, its purpose, and coverage? What is the scope (coverage) for the current program period and what scope is expected for the next program period? • What are key stakeholders trying to achieve through the M&E system (for example, increased transparency, improved accountability, better value for money)? Legal & policy basis • What are key policies, practices, and frameworks needed to support an effective M&E system? • Is there a national M&E policy in place? How does this impact ESIF’s M&E system? • What laws, acts, or regulations does the MEF rely on for its M&E system? • What are relevant EU legislative requirements? • To what extent is the current M&E system compliant with legislative requirements? • Are there legal requirements/policies supporting both monitoring and evaluation? Institutional • Who is considered the owner of the M&E system (would most stakeholders architecture identify the reference system as pertaining to the MEF?) • Who are the M&E stakeholders? How are they influencing the quality of the M&E system? • Is the system managed centrally or decentralized? • Is the system linked to other public sector management such as planning and budgeting? • Are there any entities (such as inter-ministerial councils) that coordinate or set priorities for the system? Functions • Are M&E functions closely linked or quite distinct functions? • Does the system have performance management attributes? Is it intended to? • Are monitoring activities and evaluations conducted regularly? At which level? What are the specific purposes of M&E (for example, is evaluation done for purposes of accountability, learning, etc.) • Are any of the functions conducted independently (often the case for evaluation)? Roles & • Who are the key players, what are their role and responsibilities? responsibilities • What are the M&E institutional arrangements? • Are they aligned to the objectives/requirements of an effective M&E system? • What is the procedural framework? Which actors are involved in the procedural framework? Tools • Is there a results framework specified at the level of the Partnership Agreement, for OPs, for projects? Is it being used? • What type of monitoring reports and evaluations were produced in the past programming period? • How regular is reporting? • Are M&E reports delivered on time? • Is data for reports available on schedule? • What do stakeholders do with M&E products (reports, evaluations)? • Are M&E guidelines in place (and who produced these guidelines)? • Do M&E staff and relevant stakeholders apply the guidelines? 25 Capacity & incentives • Is M&E staff in place to conduct relevant activities? What type of staff, and to what extent do they have specialized M&E skills? • Does the human resource system include specific position descriptions for M&E specialists? • What is the capacity of the system in terms of human resources, staff skills to collect data, analyze, report, manage evaluations, and disseminate results? • What is the talent pool for evaluation and what does the M&E skill pipeline look like? • What are the behaviors and attitudes of the M&E staff? • How is the evaluation culture perceived by main actors? How can it be implemented more effectively? • How should the institutional culture be aligned to the shared vision in the M&E system? 79. Next, the team will conduct an in-depth analysis of M&E functions and tools in the 2014– 2020 program based on Kusek and Rist’s Ten Steps .Figure 8 illustrates the ten steps an entity should follow to develop and implement an M&E system. The starting point is a readiness assessment at the level of the administration (step 1), planning (steps 2–5) of M&E through the development of a logical framework, outcomes, indicators to measure those outcomes; definition of baselines, targets, and data sources, and finally the functioning, upkeep (steps 6, 7, and 10) and use of M&E (steps 8 and 9). Given the maturity of ESIF’s M&E system, the team expects to focus on the later steps (6–9) to conduct in- depth analysis of the results frameworks, indicators, data collection, and reporting in the previous programming period. Figure 8: Ten steps to designing, building, and sustaining a results-based M&E system Source: Jody Zall Kusek, Ray C. Rist, 2004 80. Finally, the RAS team will apply a mixed methods approach to collect data both for the M&E assessment framework and the relevant components of Ten Steps. The team anticipates using the following data collection techniques, with instruments to be included in the forthcoming methodology report: • background analysis and literature review • desktop review of relevant documents and data analysis • diagnostic interviews with the MAs to assess the current state of M&E and capacity gaps • key informant interviews • (virtual) focus groups • beneficiary, MA, and IB surveys • quantitative analysis of available OP performance data (for selected OPs) 81. Notably, several techniques will be employed simultaneously, and data from one approach is likely to inform another data collection instrument. Consequently, the data collection will be an iterative process that will be continuously updated based on the team’s findings. The team foresees 26 collaborating closely with the MEF technical team to design and implement data collection, including on requirements such as sample design for surveys. 82. Building on the M&E system assessment, the team will consider options for M&E system improvement and articulate an overall vison and scope for the system based on key requirements and extensive consultation. Under Activity 2, the team will develop a detailed strategy for implementing M&E system improvements, define the system’s objective and the key requirements, explain the institutional set-up of the system, and identify key actors and relationships between different institutions. Both the vision and system requirements will be subject to an extensive consultation and analysis and bring in key experts in the field. 83. In developing and implementing M&E system improvements, the team will reference international best practices and gather input from focus groups and feedback surveys. International best practices and EU experience will help inform M&E system design, where the RAS team will determine what experience applies to Romania. Then, to reality check the M&E system design options, focus groups will provide insight into legal, institutional, and operational issues to be tackled with each option. Finally, surveys will help identify the M&E capacity gaps to be addressed and inform training curricula and material under Activity 3. Section 4 details all activities foreseen under the RAS. 4. Overview of RAS-funded activities 4.1. Activities at inception stage 84. As this RAS became effective in June 30th 2020, activities thus far have focused on mobilizing a team, engaging with the client and key stakeholders, and planning initial deliverables. The team mobilization brought together expertise in M&E international best practices, EU M&E requirements, and in-depth understanding and hands-on experience with M&E in Romania. The team engaged with the MEF and plans to start meeting with MEF counterparts to build a better shared understanding of the RAS scope and first activity. This engagement has also helped establish practical coordination such as the transmission of relevant documentation and legislation and planning for the inception workshop. 85. The RAS team and the MEF engaged a broader set of stakeholders in RAS activities through an inception workshop. Annex 6 includes the minutes from this workshop. Next, the RAS team foresees consulting with a significant number of actors in ESIF’s M&E system for the stocktaking. Moreover, broad stakeholder participation and buy-in will be critical for the RAS team to support MEF in developing optimal M&E system design options and to support with implementation. It is therefore critical to have strong stakeholder engagement from the beginning, and so the inception workshop was an important opportunity to inform stakeholders about RAS activities and ensure a shared understanding of objectives. Along these lines, Section 8 and Annex 3 of this report further discuss key messages to transmit to different stakeholders. 86. The team has started background analysis based on documentation transmitted by the MEF and foresees several initial interviews with key stakeholders. The background analysis allowed the team to describe what is known about the MEF’s current M&E set up and to begin developing instruments for the stocktaking, as discussed in Section 4.2. 87. Finally, the team drafted this inception report along with the methodological approach detailed throughout this section. The forthcoming report will present an updated methodology based on further background analysis; it will seek to identify quantitative data sources that are currently or could be used to track the effectiveness of M&E activities throughout ESIF’s M&E system. Finally, the report will also include the updated data collection instrument to be applied in the M&E stocktaking. 27 4.2. Activity 1: Assess the existing M&E system 88. The objective of this activity is to assess the capacity of existing M&E systems—in the MEF as well as the line ministries—to track and measure the results of EU co-funded investments based on the experience of the 2014–2020 financing period. The team has developed this initial approach based on several conceptual and practical M&E tools. The intervention logic is measured through specific indicators, leading to an integrated framework consisting of a nested hierarchy of results. The team foresees employing an M&E system assessment tool that follows from WB experience with M&E system assessments. This tool aims to provide a holistic view of the M&E system, covering its scope and objective, policy and legal foundations, institutional architecture, functions, roles and responsibilities, tools, capacity, and incentives. In addition to the M&E system assessment tool, the methodological approach also borrows from the widely used Ten Steps to a Results-Based Monitoring and Evaluation System. Moreover, the team foresees applying mixed methods to collect data as required for this methodological approach. Data collection will include a carefully structured set of activities such as documentary review, surveys, diagnostic and key informant interviews, focus groups, and case studies. The team will also exploit information that can be gleaned from the relevant information systems to undertake a quantitative analysis of indicators. Key evaluation questions:  To what extent were the M&E activities performed during the 2014–2020 programming period relevant, useful, and applied in decision and policy making, and why?  To what extent was the governance of the M&E system adequate in terms of participation, being responsive to new developments and identified needs, and facilitating transparency and accountability?  To what extent does the current M&E system comply with legislative requirements?  What are some key positive elements of the M&E system to be maintained in the 2021– 2027 programming period? 89. Three core sub-activities are envisioned in this section. The M&E system assessment framework and relevant Ten Steps (i.e., in-depth M&E analysis) will form the basis of the methodological approach foreseen for the three sub-activities noted under Activity 1: 1.1 Take stock of ESIF’s M&E system’s performance and M&E in relevant line ministries and subordinate agencies. This section will include a comprehensive stocktaking of the performance of M&E systems of MEF and relevant line ministries and subordinate agencies, through a quantitative analysis of selected key performance indicators, relevant sectoral indicators, administrative data, as well as existing survey data; 1.2 Assess the relevant institutional, legislative, and procedural M&E framework. The WB team will support the MEF in an assessment of the relevant institutional, legislative and procedural M&E framework (i.e., data collection instruments, data management and reporting systems) including the current relevant Romanian and EU legislation; 1.3 Assess beneficiary feedback on reporting requirements, procedures, and instruments. The team will support the MEF in an assessment of: (i) feedback from surveys, interviews and focus group discussions from a sample of beneficiaries of programs and relevant other Government stakeholders on reporting requirements, procedures and instruments used during the 2014-2020 financing period; and (ii) results of relevant studies previously carried out by MEF. 28 90. Beginning with sub-activity 1.1, the next subsections discuss how the M&E system assessment framework and in-depth M&E analysis will be applied, including motivating questions, key steps, and data collection techniques. Notably, sub-activities 1.1 and 1.2 will rely on the same data collection channels, which are described in depth under sub-activity 1.2. Key evaluation questions:  What is the overall functionality of M&E at the OP level and at the level of the Partnership Agreement, and to what extent is there compliance with relevant regulations?  At the level of the Partnership Agreement and the OPs, is the institutional and legislative M&E framework sufficient to ensure that M&E functions perform in accordance with EU requirements? What are the gaps, challenges, and possible solutions to the used in the next programming period?  Are the existing M&E roles, responsibilities, and tools sufficient to ensure that M&E activities and outputs are in line with EU requirements? What are the gaps, challenges, and possible solutions to the used in the next programming period?  To what extent is the organizational culture at each level (MA, ministries, etc.) supportive of M&E?  Across the system, is there sufficient skill and human resources available to perform the M&E activities according to the requirements and procedures in place? 4.2.1 Sub-activity 1.1: Take stock of ESIF’s M&E systems’ performance and M&E in relevant line ministries and subordinate agencies 59 91. The RAS team will undertake a comprehensive stocktaking of the M&E system’s performance. The team will assess key OP objectives as well as relevant sectoral indicators for the 2014–2020 period. This exercise will feature quantitative analysis of a sample of key performance indicators, as well as relevant sectoral indicators based on performance monitoring data from EU-funded projects (MySMIS). The analysis will also use administrative data from the MEF as well as line ministries and subordinate agencies, along with survey data from existing surveys (EU SILC, LFS). 59 According to legal agreement: a comprehensive stocktaking of the performance of M&E systems of MEF and relevant line ministries and subordinate agencies, through a quantitative analysis of selected key performance indicators, relevant sectoral indicators, administrative data, as well as existing survey. 29 The overarching questions motivating this analysis include:  At the level of the OP, to what extent did the results framework for 2014–2020 suffice for monitoring progress and capturing results? Did this framework allow for the results orientation envisioned by the EU Cohesion Policy?  Were indicators sufficient to capture the intended results? What lessons and good practices could inform M&E design for the next programming period?  Is data collection and coverage sufficient for reporting on indicators?  Is M&E information being generated and communicated in a way that facilitates use for policy making? Does this actually happen? What are the main challenges, factors of influence, lessons learned, and possible options to improve the delivery, communication, and use of M&E results? 92. The team will rely on the same data collection channels used for the M&E system assessment described below, but focus in greater depth on OP M&E tools, such as: • The overall results framework, including the intervention logic and objectives (at the OP level, but also at the level of the Partnership Agreement if available) • Any M&E guidelines or M&E plans (at the level of OPs, key line ministries, and subordinate agencies) • A sample of indicators and indicator fiches from the OP results framework • Key OP M&E/performance reports (along with relevant inputs from other performance reports and subordinate levels) • Evaluation plans and a sample of individual evaluations • Use of M&E information 93. The in-depth analysis will build on the data collection approaches described under sub- activity 1.2. For example, the background/document review will look at the OP results frameworks, M&E plans, M&E guidelines, and relevant reports. Through this review, the team will have a better understanding of how M&E is conducted in each OP, and specifically the design of the OP’s results frameworks. The RAS team will want to identify whether the OP used a nested hierarchy or results in its framework, as this design would influence the team’s approach for the quantitative analysis of indicators. The background review will also require further information on MySMIS, to understand how indicators are organized within the system, whether they can be identified by level of intervention and OP, and whether it is feasible to identify the relationship between indicators (i.e., outputs contributing to outcomes). 94. The quantitative analysis of indicators will assess how relevant key performance and sectoral indicators were used to monitor performance on EU-funded projects in the 2014–2020 period. This analysis will feature an in-depth assessment of a sample of indicators drawn from MySMIS, along with analysis of administrative data from the MEF as well as line ministries and subordinate agencies, along with survey data from existing surveys (EU SILC, LFS). The RAS team foresees first discussing the sampling approach with the MEF’s M&E team, to determine whether the sampling needs to include clustering of indicators, such as the outputs and outcomes related to one intervention. The RAS team would also need input on sample stratification, to ensure appropriate representation by OP (or other strata). Table 4 lists indicative questions and checklist items to be applied in the analysis, which largely draws from Ten Steps and applies SMART criteria (that which is specific, measurable, achievable, relevant, and timely). Table 4: The SMART criteria checklist for quantitative analysis of indicators M&E tool Indicative questions/checklist Potential scoring - Yes/No Results framework - Does this framework reflect the program being - 5-point scale (to implemented? what extent) 30 - How are results aggregated across implementation level? - Is the results framework aligned to decision making? - Are there key performance indicators at the three levels and by indicator type (input, activity/process, output, outcome)? - Is there a clear line of sight (nested hierarchy of results between intervention levels)? - Does the results framework include a log frame, indicator matrix, and a data collection and reporting plan? - 5-point scale (to Indicators - Are there an appropriate number of indicators to capture if some capture everything needed in the results framework? SMART criteria are - Do indicators meet SMART criteria? met) - Are definitions unambiguous? - Yes/No - Are appropriate formulas specified (method for - Score across all calculating indicators provided, such as definitions indicators in OP of numerators and denominators where relevant)? - Can generate sub - Do indicators align with the program logic? scores if interested - Do indicators have baselines and targets? - Are indicators measured in the appropriate units? - Yes/No (or scale) Data collection - Are data sources specified? - Generate percentage (with potential in - Are data sources appropriate and credible? score across all depth assessment of - What is the integrity of data collection? indicators in OP data quality) - Is data availed for the indicators (include any reasons - Can generate sub why missing or not provided)? scores if interested - Which are the main data sources for monitoring and evaluating each OP? - Has the quality of these sources been assessed? - To what extent is the data exchange between various actors in the M&E system functional? - Is data exchange accessibility and connectivity considered adequate? - Yes/No Reporting - Are the required reports being produced? - 5-point scale (to - Are they produced on time? what extent) - Are reports complete? Are all indicators actually included as per the scheduled targets? - Is performance calculated (i.e., deviation from targets)? Is underperformance flagged? - Do reports include remedial actions foreseen to correct underperformance? - Yes/No Use of M&E - To what extent do reports meet decision makers’ - 5-point scale (to information needs? what extent) - Are decision makers applying any recommended remedial actions that follow from M&E reports (and where those recommendations are considered useful?) - What M&E information is available online? 95. Interviews and focus groups will be key components for the quantitative analysis of indicators. For example, the quantitative analysis will identify gaps in indicators, and the document review might show which indicators were missing data or not reported. The RAS team will need to triangulate data and seek further explanation of these gaps by interviewing relevant M&E actors. The focus groups may also provide insights into systematic problems that arise with various indicators, by bringing together various actors along the indicator “value chain”. The focus groups may also be a useful forum for discussing report requirements and identifying any redundant reporting requirements that could be streamlined. Finally, the interviews and focus groups will provide a critical reality check: are 31 M&E plans, guidelines, and other regulations applied as foreseen? If not, why the discrepancy, and can these issues be addressed through improved design? Based on this in-depth analysis, the RAS team will provide an aggregate assessment of M&E tools in each OP (based on Table 4) and gather input for M&E design in the next programming period. 4.2.2 Sub-activity 1.2: Assess the relevant institutional, legislative, and procedural M&E framework 60 96. This second sub-activity will assess the institutional, legislative, and procedural framework of M&E of EU co-funded investments (including data collection instruments as well as data management and reporting systems) in the 2014–2020 period. The institutional and procedural assessment will cover the MEF, all line ministries, and subordinate agencies that were involved in implementing ESIF in the 2014–2020 period, including RDAs. The assessment of the legislative framework will cover current Romanian and EU legislation underlying M&E of ESIF co- funded investments. Thus, for this sub-activity, the team will rely on the M&E assessment framework dimensions on (i) scope and objectives; (ii) legal and policy basis; and (iii) institutional architecture. Moreover, the team will also develop a holistic picture of the M&E system by undertaking a system mapping. a. MEF interviews 97. The first stage of M&E system mapping will feature in-depth interviews with MEF staff. Mapping the actors in the M&E system is important for knowing who to query when filling out the assessment framework. Initially, the team expects to include entities within the MEF, other relevant line ministries, and subordinate agencies, but the initial discussion with the MEF’s M&E team will also identify who else to include in this assessment. b. Documentary review 98. The RAS team will carry out an in-depth document review. Data collection has already begun to inform the M&E system assessment, such as the legal and policy basis for M&E. The review will cover the regulatory framework for M&E of ESIF funds, key programming documents, as well as any M&E guidelines either at the OP level, developed by the MEF, or produced for the GoR. This review will also identify possible data sources and stakeholders. c. Diagnostic interviews 99. Based on MEF guidance, the RAS team will conduct diagnostic interviews with other key stakeholders. 61 Then the team foresees continuing with key informant interviews geared specifically toward completing the M&E system assessment framework. MEF’s input will also be critical to determine who to interview, including what category of respondents to target. First, the RAS team will require confirmation that the appropriate categorization of respondents is being used. For example, are relevant categories managerial and technical? Often, identifying respondents by position title (such as monitoring officers) can help. Second, respondents would need to represent all relevant implementation levels and all OPs. Third, the RAS team would work with the MEF technical team to determine the appropriate number of respondents. Here, the objective is to ensure the team gathers reliable information to complete the framework. An initial respondent might lack information and refer the team to a different respondent to fill in the gaps. Thus, an exact number of respondents is not likely to be determined upfront. But data collection will aim to be manageable: respondents may also participate in Activities 1 and 3, so the team would not want to fatigue respondents with excessive data collection. Finally, the MEF would establish a point of contact in each MA to organize interviews with the appropriate respondents. 60 According to legal agreement: an assessment of the relevant institutional, legislative and procedural M&E framework (i.e., data collection instruments, data management and reporting systems) including the current relevant Romanian and EU legislation. 61 Diagnostic interviews are few and geared towards ensuring the RAS team is developing the right instruments and targeting the correct respondents and information sources. 32 The tentative list of interviews is as follows: • Representatives of the MEF Programming and System Coordination General Directorate and Directorate for Coordination of SMIS and IT; • Representatives of each MA (21 MAs in total, approximately 60 persons interviewed) and selected national authorities, IBs, RDAs, and BRECOs (at least one per OP) • Representatives of the Functional Working Group for Performance Assessment 100. For the entities where coverage is not extensive—national authorities, IBs, RDAs, and BRECOs—the selection of respondents will be based on preliminary findings from the review of documents and data on indicators, consultations with the MA, and the number of projects implemented by the respective body. 101. The interviews will be conducted remotely (online or by phone) and feature the questionnaire referenced in Section 3.3 (M&E system assessment framework tailored to Romanian context). The RAS team foresees applying a semi-structured approach and using translation services, if requested. d. Online surveys 102. As it may not be feasible to collect all information on the assessment framework through individual interviews, targeted online surveys can also help fill the gap. The survey topics will include information on the characteristics of the M&E systems, the key difficulties encountered in meeting the requirements, the strengths and weaknesses of the systems, and possible ways in which the M&E requirements might be improved in the future. The surveys will largely include items from the M&E system assessment framework, with the same response scale. Separate surveys are envisaged for MAs and MCs. Reaching participants will require support from the MEF technical team and from MA management. • Survey A will target all staff with M&E responsibilities both at the managerial and technical capacity, at program and project level. It will mainly contain closed questions and focus on the functioning of the M&E system components. • Survey B will target all the members of the OP, MCs, and CCMAP. It will mainly contain closed questions and will focus particularly on the quality, availability, and use of M&E results. e. Focus group discussions 103. While the interviews will help complete much of the framework, focus groups help address questions that specifically rely on interrelations between actors and components of the M&E system. These focus groups can also validate previously acquired information and test M&E system improvement options being considered for the next programming period. They will be conducted online or face-to-face, depending on physical distancing rules in place, based on a facilitation guide prepared in advance, and will be facilitated by the RAS team. The expected outcome is to develop a consensus view among participants on challenges and key issues encountered in the data collection so far (getting at more of the why and how questions). The focus groups can also facilitate the development and discussion of potential improvements in M&E system design, with a view toward the next programming period. Translation services will be ensured. Four focus groups are envisaged, targeting the following groups: • Representatives and relevant stakeholders for ESF-funded programs; • Representatives and relevant stakeholders for ERFD and CF-funded programs, except cross- border; • Representatives and relevant stakeholders for programs under the Territorial Cooperation Goal; • Representatives and relevant stakeholders for EARDF and EMFF-funded programs. 104. The focus group format will encourage participants to share views on the performance of the M&E system and its components, identify challenges or influencing factors, and suggest options for improvement. The debate will be based on a presentation delivered by the facilitator, 33 grounded in the M&E system assessment’s preliminary findings. 62 The goal is to identify common issues across OPs, and the expected outcome is a list of main challenges and possible solutions for improving the M&E system’s design and performance. f. Case studies and stakeholder analyses 105. The RAS team anticipates producing several case studies based on data collected through interviews, surveys, and focus groups. The case studies would paint a holistic picture of how several OPs undertake M&E, along with the challenges and weaknesses of the current set up. Ideally, the interviews will help identify several potential cases that can provide interesting lessons to all stakeholders. Currently, four case studies are envisaged, one for each of the following categories of OPs: • ESF-funded programs • ERFD and CF-funded programs, except cross-border • Territorial Cooperation Goal programs • EARDF and EMFF-funded programs 106. Finally, the RAS team will conduct a stakeholder analysis to identify the influence that different institutional actors have on the M&E system, at the OP and PA levels. The stakeholder analysis will also treat beneficiaries as a group. In this analysis, stakeholders will be identified, characterized, and grouped based on strength, predictability, level of interest, involvement, speed of reaction, ability to generate structures, credibility, as well as their possible reactions and attitudes with respect to the functioning of the M&E system. This analysis can also capture their incentives and disincentives in promoting effective M&E and whether they are likely to support system improvements. This stakeholder analysis will also inform the implementation plan in Activity 1 and the training and capacity-building plan in Activity 3. 4.2.3 Sub-activity 1.3: Assess beneficiary feedback on reporting requirements, procedures, and instruments 63 107. This activity assesses beneficiary feedback on reporting requirements, procedures, and instruments based on an online survey of a sample of beneficiaries from the 2014–2020 programming period, across all OPs involved. As part of a holistic assessment of the M&E system, the RAS team will design and implement a survey that solicits feedback from beneficiaries on reporting requirements, procedures, and M&E instruments. In cooperation with the MEF and relevant line ministries, the team will carry out in-depth interviews and focus group discussions with selected beneficiaries as well as government stakeholders. This survey builds on key findings from sub-activities 1.1 and 1.2 and will deepen the knowledge on challenges and constraints beneficiaries faced in their data collection and reporting activities. 62 For the Growth and Jobs Goal, at least two participants will be selected from each fund, one for the monitoring and one for the evaluation function. For the Territorial Cooperation Goal, at least one participant for each OP is expected. Representatives of institutions coordinating multiple OPs under the same fund (in MEF and MPARD) are also expected to attend. 63 According to legal agreement: an assessment of: (i) feedback from surveys, interviews and focus group discussions from a sample of beneficiaries of programs and relevant other Government stakeholders on reporting requirements, procedures and instruments used during the 2014-2020 financing period; and (ii) results of relevant studies previously carried out by MEF. 34 The survey will inform key questions, such as:  Are beneficiaries aware of and do they understand their reporting duties? Does this awareness vary by OP and/or fund?  Have beneficiaries received any guidelines or training on their reporting duties? If so, has this support helped?  Do beneficiaries have access to information systems to automatize data gathering and reporting?  Are beneficiaries able to comply with the reporting duties?  How much of their time and what human resources were required to respond to the administrative burden?  Is there misalignment of the beneficiaries’ reporting duties? Are they reporting the same information more than once?  Have beneficiaries identified any suggestions for how to improve the reporting procedures, and have their suggestions been addressed?  How do beneficiaries perceive the benefit of the M&E system, and can they access monitoring data? 108. Through effective data triangulation, the RAS team foresees using relevant input from the documentary review and quantitative analysis of indicators for the beneficiary survey. From the documentary review, the team will catalog any information regarding beneficiaries’ perception and feedback on the reporting procedures and the M&E system. The document review will inform the survey design, definitions, and formulation of questions. It will also suggest what topics to explore in greater depth. Through the quantitative analysis of indicators and review of reporting, the RAS team can also identify patterns in the timing of the reporting and the completeness of the data. In turn, this information can be used to identify how many and what type of beneficiaries might face challenges in complying with their reporting duties. 109. For data and evidence collection, the team will rely on direct feedback from beneficiaries gathered through focus groups and an online survey. The survey design will be informed by the analysis of existing documentation on the roles and responsibilities of beneficiaries, the reporting procedures, and the beneficiaries’ feedback. The RAS team will collaborate with the MEF in designing the online survey, which will not be anonymous. However, the RAS team will guarantee that the summary of the findings will not allow for individual beneficiary respondents to be identified. Moreover, the survey questions will be validated through a pilot, with a small number of beneficiaries in different OPs, and the survey results will be triangulated through focus groups or interviews with beneficiaries, which will both verify the results and analyze in-depth specific topics selected for the assessment. 110. The beneficiary survey is expected to capture the perspective of beneficiaries across all OPs. The representativeness of the survey will be assessed further, but it is envisioned to be representative at the fund level. A sample frame will be needed, including contact information for the online survey. The RAS team proposes using a probabilistic method with stratified random sampling to select beneficiaries to contact for the online survey (presented in Box 10 and Figure 9). This method allows the team to first identify which specific groups of beneficiaries should be included in the survey, according to their characteristics, and then to select a statistically significant number of beneficiaries for 35 each group of interest. Moreover, the random selection of the final sample reduces the probability of selection bias. 64 Box 10: Example of stratified two-step random sampling method for selecting participants for the online survey Step 1: For each fund and OP, identify the groups of interest according to beneficiaries’ characteristics. Table 5 presents examples of such characteristics and will be defined in collaboration with the MEF. Table 5: Identification of groups of interest High quality reporting Low quality reporting All beneficiaries belonging to a specific OP Beneficiary type: private Group 1 Group 2 sector firm Beneficiary type: public Group 3 Group 4 sector institution Beneficiary type: NGO Group 5 Group 6 Step 2: For each identified group, randomly select the beneficiaries for the online survey. Figure 9: Two-step sampling method, graphical representation All beneficiaries Beneficiaries belonging to a specific OP Beneficiaries belonging to each group identified in Step 1 Beneficiaries selected for the online survey 111. The selection of beneficiaries included in the survey will be made to ensure a representative sample for key areas of the analysis. Potential characteristics to be used for selection include: the type of the beneficiary (NGO, private sector firm, etc.); the amount of funding received by each beneficiary; and quality of the reporting (beneficiaries that comply with their reporting duties on 64 However, participation in the online survey will not be mandatory. Thus, there is likely to be selection bias in the beneficiaries who actually complete the survey and the final nonresponse rate could bias the results. To reduce the impact of nonresponse, the team will oversample participants for the online survey. Another source of bias could occur if the sampling frame from which the survey participants are selected does not correspond to the most updated list of beneficiaries at the time of the survey; to minimize this potential source of error the M&E RAS team will coordinate with the MEF to receive the most updated list of beneficiaries. 36 time). In cases where there is no data on beneficiary compliance with reporting, the team could rely on a proxy. The indicators will allow the team to identify how many and what types of beneficiaries’ struggle to comply with their reporting duties and to classify beneficiaries into two or more groups according to the quality of their reporting. Finally, the number of characteristics and interest groups identified will influence the final sample size and number of beneficiaries included. 4.2.4 Key deliverables of Activity 1 Key Deliverables of Activity 1:  Output 1a— Methodological report and a policy brief for non-technical audiences - summarizing the work approach used for sub-activities 1.1–1.3. The report is due 2 months after RAS initiation.  Output 1b— Report on institutional, procedural and legislative assessment - An assessment report summarizing the findings of sub-activities 1.1–1.3. The report is due 6 months after RAS initiation. 4.3. Activity 2: Options to improve the M&E system of EU co-funded investments in Romania 112. The objective of this activity is to develop options for improving the M&E system of EU co-funded investments. The RAS team foresees designing options for M&E system improvements and thoroughly vetting these options. In close collaboration with the MEF, the RAS team will develop a vision and scope for the M&E system and various system options. Key stakeholders will provide a reality check on these options and develop ownership through participatory approaches such as focus groups, consultations, and surveys. Based on these, the RAS team will draft a strategy that presents the selected option and all its institutional and practical ramifications. The activity will seek answers to the following questions:  What is the objective and scope of M&E in the next programming period?  What are the main gaps based on the assessment, and how should these be addressed?  How can the M&E system support decision making and maximize use of EU funds?  What specific roles and responsibilities does the new M&E framework imply for MEF and other relevant agencies? 113. To achieve this objective, the activity will involve the following sub-activities: 2.1 Support the MEF in defining the M&E system’s overall vison and scope for EU co-funded programs. Providing guidance to MEF in its definition of an overall vision and scope of an effective M&E system, aligned with the conceived institutional framework for the implementation of EU co-funded programs and projects by MEF, relevant regional development agencies and relevant line ministries during the 2021- 2027 financing period; 2.2. Support the MEF in defining improvements in the institutional and operational arrangements. In line with activity 2.1 described above, making recommendations to MEF to improve the institutional and operational arrangements within the MEF, including as regards: (a) defining roles and responsibilities relevant for the M&E system; (b) the governance, supervision and accountability benchmarks; 37 (c) the responsible unit(s) for carrying out relevant activities; (d) the proposed required competences/skills of staff; (e) principles, good practices and procedural aspects of coordination and cooperation across the entities relevant for the M&E system; and (f) connectivity and accessibility of data between various selected M&E systems operated by the Romanian Government. 2.3. Support ongoing work to develop indicators for each OP to monitor ESIF performance and outcomes. Providing support to MEF in its development of proposed indicators for each OP to be used to monitor the performance and outcomes of the European Structural and Investment Funds (ESIF) investments by developing metadata on specific indicators and identifying potential proposed additional indicators to the ones agreed upon as part of the OPs; 2.4. Identify opportunities for implementing rigorous impact evaluations (IEs) at the level of projects (interventions) in OPs. Providing support to MEF in its identification of opportunities to implement rigorous impact evaluations (IEs) with a view to improve or reorient project interventions in OPs, as well as inform decisions about whether to continue, discontinue, replicate or scale up such interventions. The Bank will carry out this activity in close coordination with activities 2.1 - 2.3 described above; and 2.5 Support producing a proposed evaluation plan for each OP. Propose a draft evaluation plan for each OP, which will include: (i) proposed key evaluation criteria including IEs; (ii) suggested evaluation questions; (iii) proposed experimental and/or quasi-experimental design options; and (iv) proposed key methodological considerations. 114. The team will identify concrete design options for improving the existing system and addressing new M&E requirements, looking to international best practices and EU experience applicable to the Romanian context. 4.3.1 Sub-activity 2.1: Support the MEF in defining the M&E system’s overall vison and scope for EU co-funded programs 115. The activity will define an overall vison and scope of the M&E system for EU co-funded programs in Romania. The vision and scope will be aligned with the institutional framework for implementing EU co-funded programs and projects in the 2021–2027 period, which covers the MEF, RDAs, as well as relevant line ministries. While Activity 1 will show where the current system is, Activity 2 would indicate where the MEF (Romania) wants to go with M&E, how it will be integrated with other activities, and how it will support decision making so as to maximize use of EU funds. Key questions addressed in this activity:  What is the objective and scope of M&E in the next programming period?  How do the relevant institutional actors envisage an effective M&E system for ESIF funds in Romania for the next programming period, and how can it be achieved?  What measures can be deployed to achieve a shared vision, scope, and value for an effective M&E system? How should the institutional culture be aligned to the shared vision in the M&E system? 116. The activity will support MEF in defining the system’s purpose and attributes is key in this undertaking. While these will be informed by the stocktaking and based on international best practices and EU requirements, an improved system is likely to include: • increased focus on performance 38 • M&E information that can feed into policy and decision making • a faster, streamlined, and more nimble system, where performance information can be generated with greater frequency and shorter turn around • a system tailored to the peculiarities of each OP • an increased focus on evaluation and fostering an evaluation culture 117. With initial ideas regarding vision, scope, and functionality, the RAS team plans to undertake broader consultation using a participatory process. The planned approach includes documentary analysis (including identifying best practices at the international level), as well as extensive consultation (interviews and focus groups). Expert panels may be also used to validate best options in terms of system attributes and implementation strategy. Focus groups could serve as vehicles for gathering input and buy-in on the M&E system options proposed. The RAS team would work with the MEF to determine who to include in the focus group, which would ideally represent a broad swath of stakeholders. These focus groups would also address, sequentially, the vision, scope, and system’s purpose (sub-activity 2.1), and then the system’s attributes and institutional arrangements (sub-activity 2.2). As such, the discussion is unlikely to err on the side of being too abstract and conceptual, without a practical and operational grounding related to sub-activity 2.2. 118. Then, the assessment would drill down on what realistic system options look like, considering: • existing legal, institutional, or regulatory constraints • current technical capabilities, and those that can be feasibly built during the next program period • the overall culture of doing business among responsible authorities and beneficiaries • the need for strict compliance with the EU’s specific requirements for M&E • ownership of the existing M&E system, whether there is a need for broader ownership of an improved system, and the feasibility of achieving such ownership 119. The team is likely to recommend M&E system improvements that encourage progress and performance, while also accommodating existing constraints and limitations. A more active role for M&E may also be targeted, to support decision making and enhance complementarities among funds and interventions. Key actions for implementing the improved M&E system, as well as key actors and their role in the overall process, will also be identified. The role of each actor in the overall administration system, its relationship with other authorities (based on existing capabilities), and the current institutional and legal framework will also be considered. 4.3.2 Sub-activity 2.2: Support the MEF in defining improvements in the institutional and operational arrangements 120. With this sub-activity, the RAS team aims to generate recommendations to the MEF on institutional and operational arrangements that translate vision and scope into functional reality. These arrangements would be geared toward an M&E system that supports: • Improved performance. The RAS team would conduct further analysis to identify key drivers of performance and how these relate to the current institutional set up, roles, and responsibilities. • Supporting decision making. How can we ensure that decision makers have the right information when they need it, and can understand the data and use it for decision making? Recommendations will touch on how to make data accessible and easy to interpret, ensure timely availability, and facilitate comparisons over time to assess progress. The RAS team will also further detail how to deliver data and reports tailored to the needs of different key users. • A nimble, efficient system. A tentative goal is for data collection, analysis, and reporting to support data transmission to the EC every four months. This transmission would ideally cover both financial and physical performance, be recent, and cover all reporting levels (e.g., interventions, specific objective, TOs, ITI, or CLLD, by category of region, etc.) concerned. To 39 ensure data quality, validation and consistency also need to be built into the operational arrangements. • Increased support to and focus on evaluation. Here recommendations would identify the operational and institutional changes required by the evaluation process. These recommendations may include defining and implementing evaluation guidelines, designing evaluation plans, and identifying the institutional coordination needed to ensure that data is available for specific evaluations. To afford evaluation greater importance, recommendations following from the evaluations need to be put into practice. In turn, this process depends on appropriate institutional arrangements, where evaluation results are effectively channeled and communicated to managers and thereby integrated into the program management. For example, specific instruments could support tracking the recommendations that follow from the evaluation, specifically by priority/scope/sector/responsible actor, and the status of those recommendations. 121. Ensuring the practicality of M&E system design is key to achieving a functional and efficient system. As carried out under sub-activity 2.1, this assessment will most likely highlight challenges and further information gaps. The RAS team foresees addressing such gaps through additional desk review, surveys, and key informant interviews as part of sub-activity 2.2.1. The analysis will cover the following topics: • institutional arrangements within the MEF (organizational structure, roles and responsibilities, relationships between different entities/directions/departments, coordination, etc.) • applicable policies and roles within the larger framework • governance and accountability • operational arrangements (procedures, instruments and tools, IT systems, data collection and reporting routines) • staffing (existing skills, competences, and needs for development, performance assessment, etc.) • linking M&E to decision making (i.e., using data from M&E to support evidence- based decision making, availability of consistent data sets and timely availability of data, use of data visualization methods, etc.) • connectivity with other information systems, interoperability and accessibility 122. The assessment will also look at how M&E products feed into the OP’s management and implementation system as whole. M&E should inform project call design, project selection, and contracting. Financial control and risk management can also be improved with the support of M&E. A detailed analysis will be carried out for each individual function, but also in an integrated manner, by looking at the relevant interconnections and how the existing system is operating as a whole. 123. The RAS team foresees summarizing these recommendations supporting the MEF strategy for an improved M&E system. This strategy will: • Consider principles, good practices, and procedural aspects of coordination and cooperation across entities relevant to the M&E system. • Consider the current functionality of the data exchange system working between various selected M&E systems. • Identify the main institutional and operational gaps and how these can be addressed. • Indicate who would need to address these gaps (roles and responsibilities) and key steps to be taken. • Capture the views of key stakeholders regarding the proposed improvements (based on the focus group discussion). • Identify instruments needed to implement changes and the relevant timeframe. • Suggest how to build the capacities of key stakeholders to translate the proposals into practice. 124. The key actors will also be identified, together with their role implementing the system improvements. Moreover, the strategy will spell out specific steps to be taken at the level of the 40 institutions involved, along with supporting actions by other institutions involved only indirectly. This strategy will be included in the technical report for sub-activities 2.1 and 2.2. 125. The methodological approach will start with an in-depth assessment of existing arrangements and identification of gaps. Instruments to be used include documentary research, interviews, focus groups, and surveys addressed to MEF personnel involved in M&E and staff with coordination responsibilities. 4.3.3 Sub-activity 2.3: Support ongoing work to develop indicators for each OP to monitor ESIF performance and outcomes 126. This sub-activity focuses on the results frameworks and indicators at the OP level, ensuring they adequately capture the intervention logic, meet common indicator requirements, and include well-defined indicators. In carrying out this work, the RAS team will need to balance two requirements. One is the new approach of the 2021–2027 EU regulations that promotes simplification and harmonization. This approach will be applied to how interventions are defined all the way to M&E arrangements, and at a practical level entails the adoption of common indicators. Second, OPs require results frameworks and indicators that are tailored to their interventions. As common indicators tend to be one size fits all, OPs are likely to require additional indicators that reflect their expected results and outcomes. Moreover, OPs would also need to meet these two requirements while keeping the total number of indicators manageable, considering the existing technical M&E constraints. 127. With a view toward balancing simplification, tailoring indicators to OP needs, and keeping the number of indicators manageable, the RAS team will first review each OP’s intervention logic and results framework. The team will borrow steps 2 and 3 from Ten Steps: • Agreeing which outcomes to monitor and evaluate. • Selecting key indicators to monitor outcomes. 128. Starting with step 2, the team will work with OPs to plot the program logic, objectives, activities, outputs, and expected results. The team will assess the overall coherence of this intervention logic and revert to relevant OP actors to clarify any gaps identified. This assessment will need to be tailored to each OP, as most will have already had their program logic approved by the EC. Then, the team will follow step 3, which involves considering how well the current indicators will monitor outcomes. In this step, the team will also bring in common indicators defined at the EU level, integrating them into the results framework when possible. The general approach is that new indicators would only be added as a last resort: only if the common set of indicators defined at the EU level do not suffice for tracking progress in implementing the OP. 129. Based on the step 2 and 3 analysis, the RAS team will summarize the specific results framework gaps and indicator recommendations for each OP. Before recommending any new indicators, the team would need to follow a clear set of criteria that determine the need for new indicators and the actual indicators to be defined. For example, one criterion could refer to the value of adding a new indicator, versus the option of adjusting/redefining the activity, to allow results to be quantified through the existing set of common indicators. 130. Next, the RAS team will work with OPs to develop and refine indicators per the analysis above. Such efforts would ideally address a selection of the gaps in the results framework, rather than addressing every gap. Here, the team will apply steps 4, 5, and 6 from Ten Steps and foresees that support may include: • Adding new indicators to complete the results framework, when these additions are absolutely necessary. • Ensuring data collection responsibilities are defined and planned for common indicators. • Improving existing indicators to ensure they meet SMART criteria. • Drafting indicator fiches where needed for new, common, and refined indicators. • Identifying data sources and data collection arrangements. 41 • Defining baselines and targets that will help establish the scope of change anticipated. 131. Notably, this work can also build on the quantitative analysis of indicators (Activity 1), to the extent that indicators from the previous programming period recur in the new results framework. As the quantitative analysis will have already identified indicators that do not meet SMART criteria, the areas for improvement will already be highlighted. 132. Ultimately, these indicator improvements are geared toward establishing a results framework that fosters performance. Appropriate indicators and data collection support timely and accurate implementation monitoring and can thereby flag underperforming areas. Thus, the indicators can be used to interpret physical progress together with financial data. Then, the well-specified results framework allows for in-depth analysis to understand where the challenges lie and what adjustments and management decisions are needed. 4.3.4 Sub-activity 2.4: Provide support to MEF in identifying opportunities to implement rigorous impact evaluations 133. To increase the use of rigorous IEs, the RAS team will describe key challenges in the current programming period and opportunities to implement IEs in the future. This sub-activity will inform decisions about whether to continue, discontinue, replicate, or scale up impact evaluation. The RAS team will also build on the findings of sub-activities 2.1–2.3 and include recommendations from this sub-activity in the OP’s evaluation plans for sub-activity 2.5. 134. Per CPR regulations, MAs will need to meet new requirements for IEs in the coming programming period. Impact evaluations are a specific type of evaluation that assesses changes in individuals’ well-being that can be attributed to a specific project, program, or policy. 65 IEs have been implemented in the structural funds’ context since the 2007 programming period, according to the EU requirements and in line with the OP’s evaluations plans. According to the proposed CPR regulation, each MA will be obliged to evaluate each program by mid-2029 to assess its impact. In implementing this sub-activity, the RAS team foresees addressing the following questions:  How and when should IEs be implemented?  What criteria should be used to select IE to reorient/scale up calls for proposals?  How and when will the support be provided? 135. The team will work with stakeholders to learn what challenges they have faced in defining potential IEs and implementing IEs. The team intends to use qualitative and participatory research methods to discuss challenges to IE implementation and design. This approach aims to facilitate ownership of IEs and making institutions more receptive to implementing them. 136. The RAS team foresees a two-phased approach to developing the IE output under sub- activity 2.4: (i) a research phase; and (ii) a co-design phase. Figure 10 illustrates this approach. The starting point is to establish the status quo: what are the current practices and challenges in defining IEs, how are they used, and how can they be implemented by the Romanian public administration? The team will also share lessons on how other countries have approached these topics. In generating these findings and recommendations, the team will build on in-depth research on Romania’s existing program evaluation functions, including formal arrangements and informal practices, as well as review international experiences. Subsequently, the team will engage stakeholders in collaboration to developing specific recommendations for criteria for selecting IEs for implementation. 65 Impact Evaluation in Practice - Second Edition, Gertler, Paul J. Martinez, Sebastian Premand, Patrick Rawlings, Laura B. Vermeersch, Christel M. J., World Bank, 2011. 42 137. Three criteria will be used to filter the data collected in the research phase and to select approaches for the co-design phase. These are: • Uptake of the IE recommendations in decision-making processes (type, level, and field of influence exercised; stakeholders affected). • Coverage of the IEs within MAs (in how many institutions IEs were implemented). • Number of IEs that have been implemented. 138. The research phase draws on desk research, a review of international experience, and stakeholder engagement. Desk research for this component expands on the relevant initial findings included in Activity 1 and centers on IE challenges and current practices, along with administrative roles and responsibilities of decision makers and implementers. This research will also review relevant laws, regulations, plans, and procedures that define roles, responsibilities, criteria, and attributes for designing and implementing IEs. The regulatory review will be complemented with an analysis of approximately 20 ESIF IE reports compiled by the MEF, the Ministry of Regional Development, and other ministries and public institutions. 139. The RAS team will facilitate two rounds of focus groups to clarify, compare, and validate the desk research findings (first round), and to collect qualitative data to inform the proposed IEs (second rounds). The focus groups will be tentatively comprised of evaluation specialists from Evaluation Units within the MEF, the Ministry of Regional Development, and the Ministry of Transport, senior experts from RDAs, and selected MAs. The focus groups will be complemented by semi- structured interviews (conducted face-to-face or by phone, as conditions allow) with high-level and mid- level managers or senior experts from the Evaluation Units and MAs. Finally, the review of international experience will examine the IE uptake and challenges in various European and non-European countries, with targeted case studies on three countries that provide valuable learning opportunities for the specific Romanian context. Figure 10: Methodological approach for IE sub-activity IE Challenges IE Opportunities Desk review and legal/institutional arrangements Stakeholders, focus groups, and interviews Ongoing Review of international best practices Inventory of IE Identification of fields of intervention design Co- Identification of current IE Recommendations on opportunities for challenges IEs implementation 140. The team will develop final recommendations for improving IE uptake based on the review of international experience and an analysis of a sample IE. These recommendations will follow from the results of the desk research and the qualitative data collected during the focus groups and interviews (conducted face-to-face or by phone, as conditions allow), while considering the existing procedures within the MEF and relevant entities. As such, the recommendations will include a general framework for impact evaluations, along with specific IEs to consider on different topics as informed by the research phase. 4.3.5 Sub-activity 2.5: Support producing a proposed evaluation plan for each OP. 141. This activity will support MEF in proposing an evaluation plan for each OP, covering key evaluation criteria (including impact evaluations), possible evaluation questions, as well as design options (e.g., experimental vs quasi-experimental) and key methodological considerations. 43 142. With the increased focus on evaluation in the 2021–2027 programming period, MAs will benefit from having evaluation plans that signal proactive, institutional commitment to evaluations. As noted under sub-activity 2.4, the proposed CPR for 2021–2027 requires the MA or responsible structure to evaluate the programs to assess their effectiveness, efficiency, relevance, coherence, and added value in order to improve the overall design and implementation arrangements of ESIF funds. Evaluations will include lessons learned, problems, and opportunities. These evaluations will be planned ahead by the MAs based on specific evaluation plans (that can cover more OPs) 66 that need to be submitted to the MCs no later than one year after the program’s approval. 143. Against this background, the RAS team will support the MEF and corresponding MAs in drafting an evaluation plan for each OP. This sub-activity will benefit from the outputs from sub- activities 2.1, 2.2, 2.3, and 2.4. Moreover, the evaluation plans drafted for each OP would include relevant IEs proposed under sub-activity 2.4. In implementing this sub-activity, the RAS team will target to the following set of questions:  What are key problematic fields of activities that should be subject to an evaluation plan?  What are the EU requirements in terms of evaluations?  What are potential experimental and/or quasi-experimental design options?  Are there any good practices in terms of drafting and implementing evaluation plans in Romania and in the EU?  What actors will be involved in drafting and endorsing the evaluation plans? 144. As with sub-activity 2.4, the team will use the two-phased approached in drafting evaluation plans for the OPs: a research phase and a co-design phase. In drafting evaluation plans for each OP, the RAS team will need to ensure compliance with EU regulations and EC guidance, and incorporate lessons learned and best practices from national, European, and international experiences. Therefore, the desk research phase will entail: • collecting information on European and national legislative requirements and national practices in developing evaluations for each OP/field of activity; • establishing whether previous evaluation plans have been implemented, the challenges in their implementation, and outcomes of these plans; and • reviewing EU and international experience in evaluation plan design and implementation, to identify innovative design options that could be easily transferred to the Romanian context. 145. In the co-design phase, the RAS team will engage institutional counterparts in a thorough discussion of evaluation plans’ concept and content. The team will target counterparts who will later be engaged in the plan’s approval and implementation, in order to encourage buy-in and to increase the plan’s sustainability. The team will also cross-reference outcomes from sub-activities 2.1.–2.4 in selecting these counterparts. The selection will need to be in line with the proposed vision for an improved M&E system and take into account the implementation arrangements anticipated for evaluations, specifically which stakeholders have which responsibilities in developing evaluation plans and implementing evaluations. 146. Thus, the team foresees applying the following steps and methods: i. Conduct a desk research of relevant documents to better understand EU regulations and EC guidance, current challenges, the implementation status of former evaluation plans and important outcomes, main actors involved, evaluation reports, and other Technical Assistance evaluation activities. 66 As per Art. 39.5, draft CPR regulation. 44 ii. Review best practices and experiences at the European and international level regarding the design and implementation of evaluation plans. This review can inform the development of a common approach for designing and elaborating evaluation plans for each OP. iii. Organize a consultative workshop on the proposed approach for designing evaluation plans with relevant technical staff from each MA and the MEF Evaluation Unit. iv. Establish thematic teams for designing the evaluation plans for each OP. The team will be comprised of evaluation specialists from each MA, as well as WB experts. The scope of this activity will be to continually discuss the findings and the proposed OP evaluation plan. 4.3.6 Key deliverables of Activity 2 Key Deliverables of Activity 2:  Output 2a— Technical report summarizing options for improvement of existing M&E system - technical report summarizing sub-activities 2.1 and 2.2. The report is due 12 months after RAS initiation.  Output 2b— Draft indicators table, related metadata and evaluation plan - draft indicator table and related metadata (sub-activity 2.3) and evaluation plans (sub- activities 2.4 and 2.5). The report is due 18 months after RAS initiation. 4.4. Activity 3: Propose improvements in the M&E system 147. The Bank will provide support to MEF in its development of a draft implementation plan to facilitate proposed improvements in its M&E system, as well as carry out training and capacity building activities for M&E teams in the MEF and relevant line ministries. 148. The RAS team will support the MEF and other stakeholders to implement the selected option for M&E system improvements. The team will design this support first by conducting a training needs assessment, then by proposing a training and capacity development plan, and lastly by providing select training. Given the complex nature of the RAS activities, the team acknowledges the critical importance of risk assessment and offers mitigation options in this report. The current ESIF M&E system is already complex and involves many stakeholders at different levels. The activity will seek to answer the following questions:  Who is responsible, and for what, in developing and rolling out a new M&E system for EU co-funded investments?  What would be the timeline and specific milestones for rolling out an M&E system?  What are the organizational capacity needs required for the effective operation of the system, and how and when should these be addressed? Against this background, the activity will comprise the following sub-activities: 4.1. Support developing an implementation plan, including a training needs assessment (TNA), for the MEF and other line ministries or subordinate agencies. The Bank will provide support to MEF in its development of a draft implementation plan with a draft roadmap to improve/upgrade the existing M&E system, including identifying roles and responsibilities across the organizations involved. The Bank will also assist the MEF in developing a training needs assessment (TNA) that will identify potential gaps related to 45 the technical and management skills required for the effective implementation of improvements in the M&E system for ESIF investments; 4.2 Provide support to the MEF in developing a training and capacity-building plan (TCBP) in line with the TNA developed in sub-activity 3.1. The Bank will provide support to MEF in its development of a draft training and capacity building plan (TCBP), including a detailed proposed training curriculum designed to address the technical and managerial skills gaps identified in the TNA developed under activity 3.1 above; 4.3 Carry out a series of trainings and capacity-building activities. The Bank will carry out a select number of training and capacity building activities in small group settings for key staff from MEF, relevant line ministries and relevant subordinate agencies, in line with the TCBP developed under activity 3.2. A final summary assessment report capturing lessons covered, relevant guidelines and feedback from participants, will be prepared by the Bank at the end of the training program 4.4.1 Sub-activity 3.1: Support developing an implementation plan, including a TNA, for the MEF and other line ministries or subordinate agencies 149. The implementation plan will outline a specific roadmap for establishing the new system, including roles and responsibilities across the various agencies involved. It will be accompanied by a TNA that will identify specific gaps in terms of technical and management skills required to implement changes in ESIF’s M&E system. The implementation plan and TNA will be informed by key deliverables developed in Activities 1 and 2, and will be based on surveys of all key personnel involved in M&E, as well as focus groups with technical and lead stakeholders. 150. Activity 3 focuses on what is needed for recommendations to be fully operational and how to properly prepare staff to implement improvements. Therefore, the first objective of this sub- activity is to draft a realistic implementation plan together with the MEF, with a roadmap and responsible bodies to facilitate adoption of proposed M&E system improvements. The second objective is to assist the MEF with a TNA that identifies what technical and management skills are required to effectively implement M&E system improvements, and where the skill gaps are. In this sub-activity, the RAS team will address the following questions:  How should be the recommendations for M&E system improvements be put into practice (taking into account institutional, procedural, legal, and staffing, etc.)?  What is the timeline for operationalization?  What institutions/structures (at what level) will be responsible for implementing the proposed plan? How will these institutions be capacitated to do so? 151. The draft implementation plan will be informed by the following parameters: • EU legislative requirements and guidance on M&E systems, as identified in the desk review of earlier RAS activities. • The challenges communicated by MEF, MA, and IB staff through data collection for the M&E system assessment—namely, the focus groups, technical interviews (conducted face- to-face or by phone, as conditions allow), and other consultative working groups. • A summary of key lessons and recommendations emerging from Activities 1 and 2, with the aim of translating these into good practices and institutional knowledge. Thus, each section will include practical recommendations, concrete actions, and corresponding roles and responsibilities for implementation. These will be framed as either (i) short- or medium- term measures that can be implemented with readily available resources; or (ii) in the case of issues that cannot be tackled effectively in the short term, follow-up actions on these measures for the remaining programing period. 46 • The focus will be on hands-on approaches that can generate positive change in a relatively short or medium timeframe, or, alternatively, can potentially create the organizational setting for longer-term improvements. • It will advance the design, development, or implementation of specific tools, methodologies, and other measures that are feasible and/or could be implemented with existing RAS resources or as future actions to serve the wider implementation of ESIF M&E priorities in Romania. 152. The RAS team together with MEF will draft the implementation plan using the following steps and instruments. First, the team will collaborate with key counterparts to draft an implementation plan outline. Second, the team will work with the MEF to organize a consultative workshop with key MEF and MA stakeholders. Here, the team would present the options for the implementation plan and the draft roadmap and receive constructive feedback, as well as gain a common understanding of the activities that will be envisaged for implementing the proposal. Third, the RAS team would draft the full document based on the parameters above and the outcomes of the consultative workshop. The final deliverables would be the proposed implementation plan in RAS Output 3a. 153. Per the RAS Agreement, a TNA will complement the implementation plan. It will also inform the TCBP (sub-activity 3.2) that is the basis for the trainings to be delivered under sub-activity 3.3. Figure 11 shows the sequential link between the implementation plan, the TNA, the TCBP, and training delivery. Specifically, the TNA will be used to design the content, method, and sequencing of the training activities. Figure 11: Process for identifying and addressing training needs Identify skill areas for Identify Implement the relevant M&E roles and Map the existing areas/skills gaps Map skill needs trainings responsibilities training needs in current training around a TCBP according to the (implementation plan) opportunities TCBP 154. Through a training needs assessment, the RAS team intends to identify the knowledge or skills that need improvement or are completely missing in the reference organization. Here, the reference organizations will include the MEF, MAs, line ministries involved in ESIF M&E, and relevant subordinate agencies. The training and skill objectives are based on the recommendations for M&E system improvements. Then, the needs analysis focuses on the training priorities and critical needs, and the potential training participants. Designing effective training requires getting to know the future trainees—what is their existing knowledge base, their expectations and motivation, and specific objectives to be met through training activities. The needs analysis can also suggest whether training will in fact resolve the issue targeted: what may appear to be a training issue could also be a procedural issue, a motivational issue, or a performance issue. Particular for M&E, trainees often acquire new knowledge and skill, but lack the enabling environment to apply these. 155. The training needs analysis will include: • an overview of the M&E institutional structure, culture, and climate; • information on M&E training and delivery methods currently available in Romania; and • M&E skills displayed by staff and what requires improvement. 156. The methodological approach for the TNA will start with a document analysis, building on research undertaken in Activities 1 and 2. The documents analysis will capture first who the M&E staff is, what their responsibilities are, any performance gaps that exist, and what trainings have already been delivered to the M&E staff. This review will also flag updates to the M&E system that could influence the skill and training needs of the staff. 157. Next, the RAS team will seek input from stakeholders through focus groups and TNA surveys. The focus groups will include MEF, MA and IB representatives and aim to identify M&E training topics. These topics will be categorized as general (applicable to all structures) or specific 47 (reflecting specific requests from certain departments), based on their duties and challenges. The TNA survey will target a broader group of stakeholders engaged in M&E activities and the sample will be discussed with the MEF. The survey instrument will cover existing M&E skill needs (both technical and managerial), future skill needs, and ascertain what topics to include in the training activities. This activity will be rolled out by (i) drafting the survey instrument; (ii) identifying the sample to target; (iii) administering the survey (online); (iv) validating and interpreting the data; and (v) preparing the training needs analysis report as part of Output 3a. 4.4.2 Sub-activity 3.2: Provide support to the MEF in its development of a draft TCBP 158. In this sub-activity, the RAS team will support the MEF in drafting a TCBP, with a proposed training curriculum corresponding to the technical and managerial skills gaps identified in the TNA. This TCBP will be motivated by the following questions:  What are the training needs of the responsible staff?  What is their professional profile?  What skills are missing?  What are the capacity needs of the M&E system as a whole? 159. The TCBP will be comprised of two key parts: the training plan and the capacity-building plan. In turn, these components will focus on activities that are key to successfully implement an improved M&E system. Thus, the proposed training and capacity-building activities will align with the implementation plan and the TNA designed in sub-activity 3.1. Moreover, the frequent interaction with counterparts in Activities 1 and 2 will give the RAS team an opportunity to directly observe skill gaps and capacity needs. The TCBP will also take those observations into account. As such, the plan will be structured around future activities, capturing the expected results, timeline, and entities responsible for delivering the training. The plan will also explicitly identify training to be delivered by the RAS team under sub-activity 3.3. 160. The training plan will include training curricula developed and deployed in two phases: a research phase and a co-design phase. The research phase will tap into previous document analysis and TNA consultation. Specifically, the RAS team plans to use the TNA findings as a starting point for drafting a list of competencies and skill profiles for the evaluation specialist and the monitoring specialist. By comparing the ideal profile and the current expertise of the staff, the team will determine the key gaps to be addressed through training and the corresponding topics. The RAS team would then validate the proposed topics through an online survey targeting a sample of relevant IB and MA staff. The IBs and MAs could inform the selection of respondents (sampling), and respondents would then rate the relevance of specific topics. Based on those ratings, the training plan would then recommend designing a portfolio of modules that comprehensively address the skill gaps and constraints identified in the TNA. 161. Moreover, the RAS team will consider the following parameters in designing the training plan: • Personnel targeted by training: having a balance between sessions dedicated to MEF and MA personnel and sessions organized jointly for MA and IB personnel. • Themes targeted: with separate trainings on program evaluation and monitoring and reporting. • Organizing joint trainings with MA and the IBs staff, keeping the same participants across various trainings: this approach can help develop shared M&E practice, enhance internal coherence, and improve the efficiency of MA-IB communication and their internal coordination. 48 • Length of training sessions: Delivery options can include intensive two- or four-day modules, complemented by practical work sessions. • Thoroughly track participants to determine who has participated in which training and avoid duplicate training. The TNA will also provide some insights on trainings already received, thereby reducing redundancy. • Location of training: 1–2-day sessions will be organized offsite so that participants can focus exclusively on the course and trainings are not interrupted by office activities. • Number of participants: Keeping the number between 12 and 20 participants can ensure good interaction, engagement, and sufficient time for teamwork, team-building activities, feedback, presentations, and sharing different perspectives. • Sustaining the training: By publishing training materials, guidelines, and webinars on the MEF/MAs/IBs internal e-platform, participants have an easily accessible reference to their new knowledge and can apply this, thereby making the training sessions more sustainable. 162. While the training plan conveys more immediate training delivery, the TCBP focuses on the longer-term success and sustainability of the newly designed M&E system. It captures the change process at an institutional level, and thereby complements the training activities described above. It builds on the capacity assessment, M&E vision, and implementation plan developed in Activities 1 and 2. Thus, through the TCBP, the RAS team will recommend how to foster broad support and ownership over the planning and implementation of M&E system improvements. The approach takes what needs to change as a starting point, but then considers the organizational culture in which those changes are to occur and the relevant incentives of each actor. The research from previous activities will inform this plan, and the capacity-building activities will be drafted following the co-design principle. Overall, the methodological approach is similar to that described in sub-activity 3.1, which consists of the following activities and methods: • designing an outline of the TCBP in collaboration with the key counterparts; • organizing a consultative workshop with key stakeholders from the MEF and relevant MAs aiming to present the draft TCBP and gain feedback; • drafting the TCBP based on the parameters mentioned above and outcomes of the consultative workshop; and • capturing the findings and the proposed TCBP in Output 3a. 4.4.3 Sub-activity 3.3: Carry out a select number of training and capacity-building activities 163. The TCBP will include detailed training curricula for both technical and managerial skills areas identified by the TNA. Based on the TNA and the TCBP, the RAS team will deliver a selection of training and capacity-building activities targeted at institutions and staff responsible for delivering the M&E system improvements. These trainings will have been explicitly identified in the TCBP, and the RAS team can then specifically determine: • who needs to be trained in the selected M&E topics (per the TCBP) • how will participants be selected to attend the training • what is the training delivery modality • what capacity-building activities are needed both as a complement to the targeted training, and for the overall implementation of M&E system improvements 164. Next, the RAS team will design specific trainings, taking into account the parameters addressed in the TCBP. The team foresees that the training will target key staff from the MEF, relevant 49 line ministries, and relevant subordinate agencies. The RAS team will initially communicate the skill needs highlighted in the TNA and the training plan. Based on this information, key counterparts in the targeted entities will select participants. One key condition for selection is that the participant must not have previously participated in training on the targeted topic. 165. The anticipated training topics will balance theory and practice, with a view toward progressive capacity building. Thus, the RAS team will build out training materials based on the rich information already captured in the previous activities: these can provide case studies and good practices from the Evaluation Unit, Monitoring Unit, and others. With trainings designed around progressive skill, sessions will aim to continuously improve technical skills and be customized by structure, role, and M&E responsibilities. These will likely involve topics such as monitoring data quality and impact evaluations. Training will also aim to make M&E more efficient: teamwork will help optimize indicator guides, develop internal guidelines for substantiating organizational changes, design evaluation guides, and generate homegrown solutions to common M&E challenges raised by participants. 166. Lastly, the RAS team will use training evaluations for quality assurance and to continuously improve training materials. Participants will be requested to evaluate training sessions through a post-training online survey. They will be asked to assess the overall training, its content, interaction with facilitators, and overall organization. Based on these evaluations, the RAS team will prepare a final summary assessment report capturing the training sessions delivered, topics covered, materials used, and feedback from participants. 167. Beyond the training delivered, the RAS team will also seek to support the MEF with select capacity enhancement activities specified in the TCBP. These activities are geared toward addressing other capacity factors related to the enabling environment for implementing changes, stakeholder engagement, and relevant incentives. Some support may relate to the MEF’s role in incentivizing implementation of changes, such as establishing good practices or celebrating the success of some MAs. Moreover, addressing chokepoints may entail more one-on-one support to specific counterparts. In this spirit, the RAS team will agree with the key counterparts on the type of support needed, the timeline for delivery, and working arrangements. The hands-on support might include: (i) support for the preparation of various methodologies/procedures; (ii) the development/update and implementation of monitoring instruments; (iii) support activities and tools targeted at M&E beneficiaries and stakeholders, specifically guidance, indicator fiches, and guides; and (iv) information sessions on evaluation for the members of the evaluation networks. 4.4.4 Key deliverables of Activity 3 Key Deliverables of Activity 3:  Output 3a— Technical report including a proposed implementation plan, training and capacity building plan, and training needs assessment - technical report summarizing an implementation plan together with a TNA and TCBP (sub-activities 3.1 and 3.2). The report is due 24 months after RAS initiation.  Output 3b— Summary assessment report of all training and capacity building activities delivered under this Agreement - implementing technical trainings (sub- activity 3.3). Ongoing activity from 24 months through 34 months after RAS initiation. 50 5. Workplan summary: Deliverables, milestones, and timeline 168. As discussed in the earlier sections, the RAS team will produce reports, provide support in developing implementation plans, and conduct training exercises. The final RAS output (Output 4) will be “Summary report of all activities provided by the Bank under this Agreement”. This report will serve the needs of the key stakeholders, while having an appeal to a broader audience. The report will present the main outputs of the RAS and illustrate the WB’s contribution to provide support to MEF for M&E system improvements, while presenting evidence of how RAS outputs have influenced MA and IB M&E practices. Final RAS output:  Output 4— Summary report of all activities provided by the Bank under this Agreement The report is due 38 months after RAS initiation. Figure 12 presents the main output milestones toward RAS completion. Figure 12: Outputs and milestones Results IR Out Out Out Out Out Out Out put put put put put put put 1a 1b 2a 2b 3a 3b 4 Mon Mon Mon Mon Mon Mon Mon Mon ths ths 3 ths ths ths ths ths ths 1–2 4–6 7–12 13– 19– 25– 38 18 24 36 Inception report Output 1a: Methodological report and a policy brief for non-technical audiences Output 1b: Report on institutional, procedural, and legislative assessment Output 2a: Technical report summarizing options for improvement existing M&E system Output 2b: Draft indicators table, related metadata, and evaluation plan Output 3a: Technical report including a proposed implementation plan, TCBP, and TNA Output 3b: Summary assessment report of all training and capacity-building activities delivered under this Agreement Output 4: Summary report of all activities provided by the bank under this Agreement 169. The outputs presented in the Gantt chart above will be complemented by a semi-annual progress report in English that shall be submitted within twenty-one (21) calendar days from the end of each period. The progress reports will include a description of activities completed or in progress in the reporting period, next steps planned for the following reporting period and the outputs. 170. The activities included in the RAS are expected to support the Ministry of European Funds to develop and implement a comprehensive M&E system to track and measure the results of ESIF funds in Romania. The implementation of the M&E system will contribute to: (i) measuring results toward the achievement of the objectives of EU co-funded programs, as set out by the Partnership 51 Agreement and OP documents; (ii) informing programming decisions and broader policies based on data and regular reports from the system; (iii) strengthening transparency and accountability related to the allocation of European funds and thus increasing the effectiveness of EU co-funded measures to support Romania’s development trajectory; and (iv) enhancing capacity within the MEF and other agencies to manage data and monitor and evaluate EU co-funded programs. 6. Requested support and information gaps 171. Based on the activities described in Section 4, this section now highlights key information and coordination to be requested from the MEF and other key counterparts. In this section, we provide an indicative list of support and information that the team has identified based on the initial methodology described. As the forthcoming methodology report will provide an updated, more detailed approach for each activity, the list below can be considered indicative and will be updated. As such, the RAS team kindly requests information and support as follows: • Background review, noting any key documents the RAS team has missed, particularly key legislation • Participating in an initial diagnostic interview to tackle fundamental questions on the current M&E system design and ambitions for the next programming period • Access to MySMIS for the quantitative analysis of indicators and preparing indicator refinements under Activity 2 • Access to other relevant databases or raw data where indicator baselines or targets need to be generated • Potential access to some of the registry databases that will be identified • Establishing the first point of contact for key counterparts • Collaborating as necessary on sample design to define stratification strategy • Encouraging respondents’ participation in focus groups and interviews • Encouraging beneficiary response to online surveys • Provide feedback on best practice examples that should be considered • Identifying training participants in the MEF and recommendations for engaging other entities in M&E trainings 7. Limitations of the analysis 172. The RAS team acknowledges that several key limitations exist across all methodological approaches anticipated for the three RAS activities. The first issue relates to scope and breadth. Given the need to analyze the full M&E system and its many actors, the scope of the analysis is designed to be quite broad. However, it will never be exhaustive, as some actors or components are not on the team’s research radar. And some system components or actors will simply be inaccessible, unavailable, or unwilling to participate. Thus, the team can only be reasonably certain that they have captured a full picture of the system and seek information to identify gaps. 173. Second, as the subject of this analysis is a highly complex M&E system, data collection and analysis will also be challenging. The M&E system consists of multiple reporting layers and accountability relationships. Some MAs are nested under the MEF, others under different ministries, and the system includes actors both within and outside government. This structure alone can make for unruly management of RAS activities, let alone data collection. But on top of this complexity, the analysis foreseen in all activities builds on findings of previous activities, and the team will need to triangulate information in order to get a complete picture of the system and its performance. It will be very important to maintain some consistent reference points so as to compare and combine data across activities. For example, the RAS team will want to use one actor or group of actors as the reference point and compare their views on reporting requirements and capacity gaps. Thus, the team will need to create an appropriate data cataloging approach to accommodate the heavily qualitative data coming from a complex system potentially relying on software such as NVivo. 174. Third, the qualitative methods and surveys may be subject to significant selection and response bias. As noted for Activity 1, any online survey has a significant risk of bias due to a high 52 nonresponse rate. Thus, information collected may not be fully representative of all beneficiaries’ experience. Therefore, all online surveys will require adequate pre-testing to ensure the instrument is appropriate (and does not cause further attrition). The team will also seek an effective sampling strategy and aim to track, in real time, whether certain beneficiaries appear more likely to respond than others. Finally, continuous follow up with beneficiaries can encourage them to complete the survey and lead to a higher response rate. 175. Finally, focus groups are also subject to selection bias or may simply be too dysfunctional to yield useful data. To begin with, the RAS team will need to work closely with the MEF to ensure the right participants—those with the relevant information and background to participate in the discussion—are mobilized for the focus groups. However, the team cannot guarantee who will attend the focus group and/or whether they will contribute to the discussion. In addition, as much as focus groups intend to collect divergent perspectives and arrive at some consensus, there is always the risk that certain participants lack trust or are openly hostile to each other. At best, the facilitator would need to be expertly trained to manage such situations. Next, the RAS team will need to contend with the subjective interpretation of qualitative information: different researchers might arrive at different conclusions, and there is not necessarily a way to resolve conflicting information. In addition, the Covid- 19 crisis implies that focus groups may be conducted virtually for the foreseeable future. Without face- to-face communication, both focus groups and interviews might not prove as effective in building trust among respondents or yield the same quality of information. Lastly, the RAS team will need to balance the respondent fatigue and institutional memory in selecting and targeting participants for all research activities. Over three years, it is likely that the team will engage repeatedly with the same respondent. The advantage is continuity in respondents (similar to a panel survey). The disadvantage is that overreliance on some respondents may lead to bias, and they will eventually grow weary of discussing M&E. 8. Main stakeholders 176. Stakeholder engagement is fundamental to achieving RAS outcomes. Thus, this section briefly summarizes the key actors to be engaged in RAS activities, what their interests are, and how they will be engaged. This section complements the discussion of RAS activities in Section 4, where various stakeholders are already referenced as target groups for focus groups and other consultative activities. This stakeholder analysis is also an important reference for the discussion on dissemination (see Section 11). 177. Given the scope of ESIF funds, RAS activities will reach a broad swath of beneficiaries. In particular, the RAS will offer recommendations and influence decisions for an M&E system design that will impact a sizable number of beneficiaries. Thus, many will have an interest or stake in, or will be impacted by, the decisions. These beneficiaries encompass every public or private entity that has accessed or will access ESIF funds. M&E staff in the MEF and other ministries or entities designated as MAs or IBs will also be impacted by RAS recommendations, as these will influence their daily M&E activities. There is likely to be especially strong interest in the M&E simplification measures that may arise from RAS recommendations. 178. To determine the best strategy for engagement, this initial analysis identifies stakeholders’ characteristics and roles, their interests, and to what extent they influence RAS outcomes. Broadly speaking, the main stakeholder groups and their interests are as follows: • The MEF and MAs—these stakeholders are critical for meeting RAS objectives. The team will need to rely on MAs and other key MEF departments to every extent possible, as they hold leverage in decision making critical for implementing recommendations that follow from RAS activities. • IBs—play a key role in delivering M&E procedures, advice, information, and regulation on the ground, specifically through the monitoring officers. The RAS will rely on the IBs to adopt changes in their M&E practices in order to realize efficiencies and streamline M&E. 53 • Beneficiaries—important stakeholders with a key interest in any policy measures that reduce the monitoring and reporting burden. RAS recommendations may target M&E system improvements that can translate into less monitoring and reporting by beneficiaries. • Participants—key for ESIF impact are a largely untapped end-user group that will indirectly benefit from any proposals to improve monitoring and reporting practices, especially in the ESI funds. Their participation in ESIF-funded projects has the capacity to either curb or accelerate beneficiaries’ project implementation and uptake of ESIF-funded projects and impacts. With improved M&E practices, beneficiaries are likely to redirect resources from M&E-related project implementation procedures to activity implementation, interaction with participants, and generating benefits for participants that lead to longer-term impact. • Central government institutions (ministries and agencies)—key actors for coordination, given the potential overlap between ESIF M&E and any ministry systems. They also may benefit from or show interest in applying RAS recommendations for M&E system improvements in their ministries, thereby reducing duplication, particularly in reporting. These institutions are gatekeepers for administrative data, and it will be important to coordinate with them to streamline data collection and reporting. • Researchers, academia, and NGOs—these are highly supportive of ESIF M&E streamlining and deeply engaged in building a culture of evaluation. They are willing to engage in exchanges of ideas, provide input for discussion, and are potential amplifiers in disseminating the RAS recommendations and lessons. This capacity can be built by involving this group throughout RAS implementation in activities such as focus groups and surveys. 179. Furthermore, RAS activities will involve several key players that are implicated in the current M&E system (see Table 6). This table also notes their interests and roles in implementation: Table 6: Key M&E stakeholders Entity M&E/Implementation Relevance to RAS activities responsibilities MEF – Programs • leads ESIF program evaluation • key counterpart for RAS sub- Evaluation Office • conducts evaluations for MEF- activities on evaluation based MAs • designs and implements multiannual evaluation plans for MEF-based MAs • produces the National Multiannual Evaluation Plan • leads the Functional Working Group for Performance Assessment MEF – Program • leads program monitoring and • responsible for implementing the Monitoring Unit reporting activities RAS recommendations in the • methodologically coordinates and monitoring function provides guidance to other MAs MEF – Analysis • leads OP negotiations for the next and Programming programming period Directorate MEF MAs and IBs • responsible for implementing the • offer a sector perspective on the following OPs:, Competitiveness challenges of the M&E system in OP, Large Infrastructure OP, the current programming period Human Capital OP & OP for • will also be responsible for Assisting the Disadvantaged implementing RAS Persons (POAD), Technical recommendations for the next Assistance OP, and their future programming period, in successors collaboration with the Program 54 • coordinate the monitoring activities Evaluation Unit and the Program at the project and program level for Monitoring Unit those OPs • monitor some specific projects • participate in evaluation exercises conducted by the Program Evaluation Unit MPWDA’s MAs • responsible for implementing the • offer a sector perspective on the and IBs, including following OPs: Regional OP, challenges of the M&E system in RDAs and JSs Administrative Capacity OPs, the current programming period Territorial Cooperation Goal OPs • responsible for implementing • coordinate the monitoring activities RAS recommendations for the at the project and program level next programming period, in M&E for those OPs collaboration with the Program • monitor some specific projects Evaluation Unit and the Program Monitoring Unit Line ministries • responsible for implementation of with IB function – the current programming period Ministry of delegated tasks Transport, Ministry of Education and Research, Ministry of Communications and Information Society Monitoring • oversee the overall implementation Committees of the OPs The evaluator’s • plays a major role in delivering • platform for discussion and network solid and innovative ESIF analysis of RAS recommendations evaluations • builds an evaluation culture • provides inputs to future socioeconomic evaluations Data providers – • provides certain data for ESI M&E • critical actors for discussing data National Statistics needs data provision access, harmonization, and data Institute, line • establishes and implements data sharing protocols ministries and quality practices (statistical data) agencies, research • supports development of data institutes, other systems stakeholders with responsibilities for data provision Beneficiaries – • apply project monitoring and • provide feedback on current M&E Public and private reporting requirements practices is important, what needs entities, NGOs, to change and how in the future LAG, other legal programming period persons • directly be impacted by the proposed recommendations • their feedback will be a test to determine whether RAS recommendations lead to M&E improvements on the ground 55 180. The principle of co-design will guide stakeholders’ engagement. RAS activities will engage key stakeholders from the inception workshop and throughout in order to generate buy-in, smoothly implement RAS activities, and lessen the resistance to changes that would follow from RAS recommendations. Practically speaking, co-design means that key stakeholders will collaborate closely with WB experts and MEF from the onset, shaping the evaluation questions, research design, training design, final recommendations, and outputs In addition, the RAS team realizes that the stakeholders need to be involved, and that their interests are fluid and change over time. 181. Stakeholders will be targeted through specific channels that take into account their interests and potential influence on implementation of RAS activities. Throughout the RAS, workshops (including virtual ones), interviews, and trainings will help engage and keep an open line of communication with stakeholders. Table 7 illustrates where different stakeholders taking part into RAS implementation, calibrates their capacity for generating or supporting changes, and identifies actions to target them. Table 7: Stakeholders engagement  Interests Capacity and motivation to Category of  How will they be influenced by the generate/supp Possible actions to address their interests stakeholders RAS? ort change  How will they influence the RAS? (Wish/Can) MEF – - Main counterpart, key stakeholder +W Will be involved in and consulted on all Program - Main beneficiary with high interest in +C activities of the RAS through regular Evaluation RAS outcomes bilateral meetings, workshops, seminars, Unit - Very high influence on RAS progress interviews, and trainings and sustainability MEF – - Key stakeholder +W Will be involved in and consulted on all Program - Main beneficiary of RAS monitoring +C activities of the RAS regarding monitoring Monitoring outcomes activities and indicator design, through Unit - Very high influence on RAS monitoring workshops, seminars, interviews, and recommendations trainings MEF – - Important stakeholder that will provide +W Will be involved in and consulted on some Analysis and information regarding next Program -C activities of the RAS regarding future Programmin Period’s OPs Program Period’s framework, through g Directorate - Medium influence on RAS progress interviews and workshops or seminars MEF’s MAs - Key stakeholder +W Will be involved in and consulted on all - Highly influenced by RAS outcomes, as +C activities of the RAS regarding monitoring they will have to change some M&E activities and indicator design, through practices and institutional arrangements workshops, seminars, interviews, and - Will highly influence the RAS as they trainings will co-design some of the activities and recommendations MPWDA’s - Key stakeholder +W Will be involved and consulted in all MAs - Highly influenced by RAS outcomes, as +C activities of the RAS regarding monitoring they will have to change some M&E activities and indicator design, through practices and institutional arrangements workshops, seminars, interviews, and - Will highly influence the RAS as they trainings will co-design some of the activities and recommendations IBs - Important stakeholder for the +W Will be involved and consulted in all sustainability of RAS outcomes -C activities of the RAS regarding monitoring - Highly affected by RAS outcomes, as activities and indicator design, through changing monitoring practices at the workshops, seminars, interviews, and monitoring officer level from the IBs trainings will be important for RAS success - Medium influence on RAS progress The - High interest in RAS outcomes +W Will be consulted regularly regarding evaluator’s - Less affected by RAS outcomes -C flagship RAS activities and will be mainly network involved in RAS dissemination activities 56  Interests Capacity and motivation to Category of  How will they be influenced by the generate/supp Possible actions to address their interests stakeholders RAS? ort change  How will they influence the RAS? (Wish/Can) - Low capacity to influence RAS progress Beneficiaries - Key stakeholder for sub-activity 1.3 of +W Will be consulted mainly during sub- the RAS -C activity 1.3 of the RAS and any other - Highly affected by the RAS outcomes, testing/piloting of RAS recommendations as they are applying the monitoring and Will be mainly involved in RAS reporting changes envisaged by the dissemination activities RAS - Low capacity to influence the RAS progress 182. Information is a valuable commodity for all stakeholders, though each has their own topics of interest or outputs from the RAS. Annex 5 details what specific information and messages will be targeted to which stakeholders, so as to appropriately respond to their need for information. 9. Risks and risk mitigation 183. The successful completion of the RAS relies on strong cooperation between the WB, the MEF, and the relevant line ministries, as well as on the ability to anticipate and mitigate potential implementation challenges. This section presents these challenges and potential mitigation measures. 184. The first operational risk concerns Activity 1 and meeting the data needs for an M&E system assessment through the methods and along the timeline described. The successful completion of Activity 1 relies on the timely delivery of data and documents needed to complete the M&E system assessment. It also requires stakeholders to be ready and able to participate in interviews and surveys proposed by the team. As both the MEF and line ministries may be providing required data, all teams implicated in RAS activities will need to be informed in advance of RAS objectives and data needs. Moreover, while the RAS team can anticipate most data needs in advance, further needs are likely to arise when discussing the M&E assessment methodology for Activity 1 with the MEF. Consequently, the RAS team might require additional data and information to be prepared and delivered on short notice. 185. To address operational risks emanating from possible delays in the availability of technical counterparts or data, the RAS team will: (i) organize a project inception workshop in partnership with the MEF and include all counterpart agencies, to clarify data and information needs upfront and strengthen the demand for and interest in RAS activities; (ii) establish and regularly update the M&E Working Group comprised of relevant line ministries, subordinate agencies, and the National Statistical Institute—established jointly with the MEF—on the status of the work, and address any delays or availability issues there; and (iii) prepare a methodology for the M&E system assessment that relies on multiple data and information sources, so that potential delays in data delivery do not compromise the successful output completion. 186. The second operational implementation risk relates to the Covid-19 pandemic. While the current pandemic does not present a challenge for RAS implementation or preparation of the outputs, the RAS team will commit to organizing virtual meetings to maximize participation among counterparts. The team will also set up a secured system for data sharing and storage, to guarantee remote access to the data required to complete the RAS outputs. 187. The third operational risk is related to Activities 2 and 3: delays in guidance and support can affect the delivery timeframe for RAS outputs. The M&E system needs to be developed around the institutional framework for the 2021–2027 programming period. If this is delayed, the RAS team will be missing a crucial element for defining in detail an overall vision and scope of the M&E system for EU co-funded programs in Romania. Moreover, the tasks completed under Activity 1 might highlight the uneven implementation capacity and needs of the different line ministries and OPs. These potential 57 differences might delay the implementation of some of the recommendations prepared under Activity 2, which might affect the overall timeline for Activity 3. 188. To mitigate the potential risks related to Activities 2 and 3, the RAS team will rely on the strong collaboration built through the M&E Working Group and on a detailed timeline for both activities. First, the RAS team will present the timeline and the information needs to the MEF and M&E Working Group in advance. Second, to strengthen the collaboration between the RAS team, the MEF, and the line ministries, the RAS team might decide with the MEF to invite the specific technical and implementation teams to the M&E Working Group, to get a more detailed understanding of the functioning of the OPs’ M&E systems and potential implementation challenges. Third, the implementation work plan and timeline will present a multi-track approach to address the potential support needed by counterparts and their different implementation starting points. 189. The final risk is related to potential delays due to the autumn 2020 political elections, which might affect both the priorities of the MEF and line ministries and their overall structure. The elections will take place during the current M&E system assessment (Activity 1), but the overall timeline of the project and the implementation of the proposed changes should not be affected, so no mitigation activities are required at this stage. 10. Quality assurance mechanisms 190. Assuring the quality of our support is integral to serving the MEF’s needs and increasing their capacity. If the team’s analysis does not meet minimum quality standards, it may lead to erroneous or irrelevant results, along with conclusions and recommendations that may lead decision makers astray. A quality process leads to a quality product or deliverable, and quality must be guaranteed through appropriate mechanisms, both for the reports and for the process. 191. The team is fully aware that data will be a crucial input for quality analysis and that data collection may be challenging. Unavailable or poor-quality data, along with suboptimal planning or design of RAS activities, can lead to poor performance, as reflected in the quality of deliverables. If the minimum quality standards are not met during the RAS, it not only undermines the assessment of the M&E system and improvements recommended. It also erodes confidence in the M&E activity in general and its overall importance in managing the OPs. 192. Quality assurance and timely output delivery are key in RAS implementation. The WB applies the same standards of quality and due diligence to RAS engagements as it does for other operations. Therefore, the same quality assurance principles, policies, and procedures are applied to all outputs delivered by the WB. Key steps in the quality assurance process within the WB include the following: • regularly informing the counterpart, the WB Country Management Unit (CMU) and General Practice (GP) of progress, including submission of progress reports and back-to- office reports when missions end; • collaborate internally to ensure accurate budget allocation, accruals calculation, expense projections, and billing; • regularly update data in the WB operations portal; • control the quality of key outputs through appropriate mechanisms, including through the peer review process that involves at least two peer reviewers appointed by the RAS task team leaders; and • consult with and disseminate the outputs to the MEF as applicable. 193. The quality assurance process requires an internal review of outputs to be delivered before being submitted to the MEF. The internal review process includes three layers of review before final approval: • the first layer of revision is performed by the RAS core team, comprised of a task team leader and other staff/consultants appointed to lead the task; 58 • the second layer of revision is performed by the SPJ GP general director and appointed staff to review the document; and • the third layer is performed by the CMU staff and country director. 194. If considered necessary, the RAS team can organize timely decision meetings to review the output quality. The outputs will be drafted in English and be subject to professional editing and Romanian translation. The RAS team will review the final translated documented before it is sent to the counterpart. 195. Given the WB’s broader mandate to provide global knowledge, the quality assurance process considers how the knowledge gained from the RAS exercise may feed into broader discussions at the regional or global level. This may take place even if the individual RAS outputs are confidential, as broad lessons may be synthesized as appropriate. 196. The overall quality of the assessment depends on ensuring adequate quality during each of its stages: during the preparation, while performing the assessment, and through its deliverables. The following aspects will be observed: • compliance with the deadlines set in the timeline; • effective communication with MEF; • informing MEF representatives about the difficulties encountered at any stage of the evaluation process; • ensuring the necessary resources to smoothly run each stage and achieve the expected results; • monitoring and quality control of the experts involved in the project; • effective monitoring during project implementation and early detection of problems; and • ensuring appropriate standards regarding the content, expression, and formatting of reports. 197. More generally, the assessment of the outputs’ quality 67 is based on the following criteria: • quality of evidence • quality of analysis • extent to which lessons are based on evidence and analysis • results orientation • internal consistency • consistency with guidelines • conciseness 198. Table 8 shows the proposed evaluation grid. Table 8: Proposed evaluation grid Criteria Interpretation Extent to which the The activity/deliverable respects the terms of reference and activity/deliverable responds to the provides the necessary information. requirements and needs of the client Extent to which the activity/output The activity/deliverable covers all OPs in the terms of reference covers the subject and all aspects highlighted in the methodology. Extent to which the process is open Stakeholders were involved in the process and in discussing its and participatory outcomes and proposed solutions, in order to take into account their views. 67 Based on the WB IEG assessment criteria of the ICR. 59 Design of activities is appropriate The design of the activities has led to the achievement of the proposed results. Data used for the assessment is The data is of adequate quality to inform the analyses. reliable Information sources are clearly Information sources are referenced accordingly. identified Analysis is rigorous Limitations are clearly stated Results of the assessment are credible The results are logical and substantiated by data analysis and by appropriate interpretations and assumptions; the findings are derived from data and analysis. Conclusions are impartial and realistic Recommendations correspond to the conclusions and are useful to decision makers and other stakeholders Project implementation took place as described in the initial report 199. Based on this criteria, and to ensure the highest quality before being submitted to the counterpart, the output must show that: (i) it is tightly written and contains a complete independent critique of the situation; (ii) there is a clear link between the narrative, the findings, and the evidence; (iii) it provides a candid, accurate, and substantiated set of observations that are aligned to the output objective; (iv) it is concise, follows the guidelines, seeks to triangulate data to reach conclusions, and is focused on results; (v) the quality of evidence and analysis is substantial and informs all aspects of the output; and (vi) there is a well-articulated theory of change and the lessons are specific, useful, and based on evidence of what was observed. 200. Quality of evidence comprises data quality. Only data provided officially by the Romanian and international authorities, or that which is publicly available, will be analyzed and referenced accordingly. Raw databases will be cleaned, officials’ opinions expressed during interviews and focus groups will be referenced as such, and other reviewed documents and analysis will be used diligently in the best interest of the assessment. 201. Co-design, teamwork, and rigorous research methods are drivers of the quality assurance mechanism. Co-design involves collaboration between the team of experts and counterparts from the onset, in question framing, research design, as well as in shaping recommendations to allow fine-tuning to local and organizational specificities. Teamwork involves close collaboration between teams to have both a dynamic and adaptable work framework and a participatory framework that facilitates consultations to build a shared understanding of objectives, methods, limitations of work, and solutions. Rigorous research methods allow for all factors to be considered, and relevant stakeholders are consulted, fine-tuning both the qualitative and quantitative research methods. 11. Dissemination and completion 202. This RAS holds great promise for generating knowledge to be disseminated at the national and international level. As the RAS will end in the second quarter of 2023, this section focuses on the dissemination activities developed by the RAS team and the Evaluation Unit (MEF) to be implemented over three years of strong cooperation. More specifically, dissemination activities aim to share the knowledge gathered through the WB’s cooperation with the MEF, while engaging with as many stakeholders as possible to build continuous support for the activities developed through the RAS. Moreover, considering that this RAS is one of few EU-funded investments tackling the improvement of ESF, CF, and ERDF-financed interventions, there is substantial value in sharing knowledge and lessons learned during the implementation. Thus, the team will work with the MEF to identify areas of high 60 interest to external audiences. Moreover, disseminating lessons from this type of advisory support aligns perfectly with the WB’s role as an international learning and knowledge-sharing organization. 203. The RAS team foresees strong interest in knowledge generated on the implementation of impact evaluation and evaluation plans. As evaluation is of growing prominence in EU regulations on M&E, multiple Member States are likely to be facing similar challenges in developing impact evaluations and evaluation plans. Thus, there may be high demand for these lessons. Moreover, evaluation plans are also an underexplored topic at the international level. Though several countries use them systematically, to date there has been little cross-country analysis of this experience. As Activity 2 will review EU and international experience in evaluation plan design and implementation, this analysis will fill a critical knowledge gap. 204. The Covid-19 crisis will partially influence the timeline for the dissemination activities. Digital and virtual communication instruments will be employed until at least the end of the first quarter of 2021, depending to a large extent on how the pandemic evolves. In-person engagements can be introduced after the crisis ends and Covid-19 health and safety measures are no longer needed. Due to these limitations, the team and the MEF might decide to limit virtual dissemination activities that involve large numbers of participants, and favor smaller virtual meetings, which allow for more audience engagement. At the same time, virtual events might facilitate participation from multiple counterparts and stakeholders that otherwise might have been more difficult to involve in in-person events. 205. The dissemination activities target two broad audiences: Romanian and EU-related audiences and WB internal audiences, each with their specific dissemination activities. • Romanian and EU-related audiences:  Romanian ministries, agencies, the National Statistical Institute, and other government bodies at the central and local level involved in RAS implementation  Other ministries, agencies, and government entities at the central and local level that could be interested in developing an M&E culture  NGOs and other relevant nongovernmental institutions  Romanian evaluators  Romanian and international universities (students and academics/researchers)  Romanian and international practitioners  Romanian and international journalists  Relevant Directorates within the EC • World Bank internal  Europe and Central Asia Social Protection and Jobs GP, as well as other GPs within the same region that might be involved in OPs that are part of the RAS, in partnership with the WB CMU in Romania  Social Protection and Jobs GPs and other relevant GPs from other regions, such as Asia, Latin America, and Africa  Others within the WB such as staff and consultants 206. The M&E Working Group will be an important platform for dissemination. The group will convene regularly to receive updates on the status of RAS implementation and discuss key findings and recommendations from the outputs. 207. Each communication instrument will benefit from a consistent message that will be delivered through all dissemination activities, with varying degrees of detail and jargon adapted to the specific communication needs of the target audience. Specifically, the RAS team and the MEF will select which output will be disseminated and discussed with the audiences listed above. A unified narrative and storyline for the RAS will be designed and deployed along all knowledge-sharing and dissemination activities with the selected audiences. The storyline will be consistent with the terms of 61 reference and the results of the RAS, with emphasis on the changes achieved by the collaboration between the RAS team and the MEF. 68 208. A variety of communication products will accommodate specific audiences’ needs. Knowledge-based communication platforms will be progressively deployed to effectively inform different audiences. For example, citizens could be reached with an infographic, an NGO with information published on a webpage, whereas the EC, evaluators, or researchers may need a case study, detailed output, or event. The advantage of this approach is both efficiency and effectiveness, as the more information and knowledge reach broader audiences, the more the system’s resistance to change diminishes. 209. The RAS will employ dissemination activities based on the following products. Each activity will be organized in collaboration with the MEF, including by providing support to specific MEF initiatives. • Events  Synchronous virtual events will be used for direct audience engagement. Such events could target internal WB audiences as well as specific MEF and line ministry audiences: informal WB staff knowledge-sharing sessions (called brown bag lunches); small online workshops with MEF officials and/or NGOs and other beneficiaries, evaluators’ community, etc.  Live events could take place at the WB office in Bucharest, relevant ministries, universities, and other public or private centers across the country in partnership with the MEF  Knowledge-sharing events for relevant MEF staff in might be organized when regular working conditions are resumed • Reports and presentations  All online events will include PowerPoint presentations that will be part of the M&E- RAS knowledge collection  Publicly disclosed outputs will be modified (edited for length, formatted, or branded); the MEF Evaluation Unit will decide which outputs will be made available to the public  Publicly disclosed outputs and summary of reports will be made available in physical and digital format  physical: at information kiosks of government institutions  digital: on relevant websites of the MEF, WB, other ministries and national or local public institutions, universities, etc.  Non-publicly disclosed outputs will be made available to MEF officials through the MEF’s and other MA/IBs’ intranet (according to internal document management policies and operational procedures) • Web  Publicly disclosed outputs will be published and cross-linked on as many websites as possible, including relevant websites of the MEF and the Evaluation Unit, GoR, MEF, WB, other stakeholders’ web pages, NGOs, universities, etc. • Social media  Videos that capture short testimonials, interviews, and subject-specific presentations in documentaries or lectures (vlogs)  Blogs that provide greater depth to specific topics may be developed and distributed through social media  Public events announcements or press release  Short text and links to relevant websites where detailed documents and reports can be found  Digital flyers (based on physical ones) • Flyers  One-page infographics 68 Annex 3 presents results communication channels. 62  Multi-subject folded on location of the events, flyers at receptions at government and non-government institutions, including university research centers. 63 Annexes Annex 1: Structural and investment funds and OP structure The contribution of activities toward achieving the expected results—i.e., outcomes and projected development objective—is detailed in the RAS theory of change (ToC). 69 Along with the main activities, planned outputs and expected outcomes, the ToC presents the main contextual aspects, relevant for the RAS implementation, as well as the influencing factors (support factors and barriers) that may affect implementation and the results achieved. These refer, but are not limited to:  Support factors:  increased focus on M&E at EU level;  GoR Memorandum defining OPs and institutional architecture for the 2021–2027 programming period;  Client’s openness and decision to successfully implement the project  (Potential) barriers:  Frequent legal and institutional changes;  High level of bureaucracy at the level of public administration70 (high level of reporting, duplicating reports, uncorrelated data sources and national/regional registries that are not interconnected etc.)  Insufficient coordination among relevant authorities in implementing the EU funds;  High complexity of the OPs (multi-fund OPs, enhanced use of territorial instruments such as ITIs and CLLD, regional OPs etc.) and institutional set-up for the next programming period. The context section in the ToC is based on the current background set-up (see also Section 1.1). Any changes in the context, that may affect implementation, as well as changes with regard to factors of influence or the strategy planned should be reflected in a revised version of the ToC, considering that they could potentially influence the achievement of the set objectives. 69 Based on Ray Rist mode for ToC (Linda Morra Imas, Ray Rist: The Road to Results, 2009) and models presented on University of Wisconsin-Madison website 70 Assessments of the M&E institutional .framework carried out as part of the RAS Supporting the Implementation of Romania’s Human Capital Operational Programme (POCU) 2014-2020 (P162775). 64 Context Sub-Activities Development Activities Outputs Outcomes Objective Clear set of rules defined at EU level, with common output and results indicators, Stocktaking of the performance of M&E clear reporting requirements etc. systems in 2014-2020 Methodological report for the assessment Assess institutional, legislative and Measuring results towards Memorandum approved in Feb. 2020, Assess the capacity of procedural framework of M&E of EU co- the achievement of the defining OPs and institutional set-up for existing M&E systems Policy brief for non-technical 2021-2027 PP funded investments objectives of EU co-funded to track and measure audiences programs the results of EU co- Assess beneficiary feedback on reporting Complex OP structure (21 OPs, 19 MAs, at funded investments requirements, procedures and least 9 IBs) including multi-fund OPs. Full assessment report on the instruments based on an online survey of capacity of the existing M&E system Informing programming a sample of beneficiaries of programs the to track and measure results of EU decisions and broader Increased territorial focus 2014-2020 period, across all OPs involved co-funded investments policies based on data and regular reports from the Define an overall vision for the M&E system Development and Institutional structure building on existing system one for 2014-2020, looking at valorizing Technical report summarizing implementation of a lessons learned findings and recommendations for comprehensive M&E Develop options for Define institutional and operational system to track and aspects improving the M&E system improving the M&E measure results of system Strengthen transparency and Need for harmonization of approach, European Structural Support ongoing work for developing accountability indicators and procedures and Investment Funds indicators for each OP Draft indicator table for each OP in Romania Need for simplification of rules and Identify opportunities for implementing procedures (more extensive use of SCOs) rigorous impact evaluations (IE) at the OP Evaluation Plans level of projects (interventions) in OPs Enhance capacity within MEF IT system (SMIS 2014) only partly Technical report summarizing and other agencies to functional Support producing a proposed evaluation implementation plan and TNA and manage data, and to monitor plan for each OP TCBP and evaluate EU co-funded Need to increase the role of evaluation in Implementing Develop an implementation plan, programs support of decision making improvements to the including a TNA Technical trainings M&E system Main assumptions/Hypotheses: 1. Data is available as to allow for adequate assessment of the capacity of the existing Develop a TCBP in line with the TNA Training assessment report M&E system. 2. Relevant actors (MEF, line ministries) are engaged in the project facilitating Training and capacity-building activities assessments and implementation of recommendations. 3. The recommendations for improving the M&E system will.be 65 applied/operationalized across the entire EU funds system. Extensive experience of Legal basis (GoR Memorandum) Ongoing projects for Increased focus on Client’s openness and RO authorities in defining institutional architecture development of M&E Support 4. MEF will be able to actively assume the coordination role in the field of M&E and monitoring and decision to implement take the necessary measures for improvement and correction, where relevant. managing and and providing for a strengthened capacity at the level of evaluation at EU level the project successfully 5. The development of IT systems and procedures will support the implementation of implementing EU funds coordination role of MEF different institutions recommendations formulated (e.g. data collection and validation, reporting, data sharing etc.). 6. A functional collaboration system is in place at system level, allowing adequate Different High inertia High complexity of the Line Sub-optimal Frequent priorities at at the level system defined for the ministries/authori collaboration at Barriers implementation of M&E recommendations, as to achieve the planned results. legal and High level of 7. Relevant staff participates in trainings and capacity-building activities and applies the level of of public funds management (large ties’ willingness to system level institutional bureaucracy newly acquired competences in day to day work. actors administrati no. of actors, multi-fund assert their (including data changes exchange) 8. Changes in the legal and regulatory framework do not affect project implementation involved on approach, territorial focus independence and achievement of expected results. Institutional Structure for the management of EU Funds for the 2021–2027 programming period Source: MEF. 66 Annex 2: Literature summary on results-based M&E Literature review was a key step in designing the overall methodological approach, as well as the instruments used for implementing the project activities. The following topics were identified as generally agreed upon in the international practice and as of particular relevance for the RAS: • M&E are integral yet distinct functions of policy and program preparation and implementation. They are critical tools of the effective design, implementation and delivery public policies and services, by providing the necessary evidence for decision making 71 . The use of M&E throughout the policy cycle contributes to an improved results-oriented management, by linking interventions to their expected outcomes and enhances transparency and accountability 72. Not least, it promotes learning and knowledge sharing 73) • There is no single agreed design of the M&E system. Set-ups depends on donor, policy or country-specific factors, such as need and purpose of the information to be gathered, availability and quality of existing data, internal and external capacity to conduct analysis and evaluations, the amount of resources dedicated to the M&E functions, as well as the overall readiness toward M&E (Leiter, 2016). • A number of common components define effective M&E systems internationally. M&E depend on sound systems with reliable, timely, high-quality input and usable and available information output (Edmunds, R., Marchant, T. 2008). In order for this to exist, well-functioning M&E systems generally bring together a number of components on three layers, starting with enabling conditions and ending with the central purpose of the information produced—its use in decision making. These components include aspects such as: purpose and coverage, policy, legal and planning foundation, institutional architecture, functions, roles and responsibilities, tools, capacity & incentives (Jespersen, 2017 74, UNAIDS, 2008). • A functioning M&E system entails a process of multiple steps toward reaching maturity. While not compulsory, it is recommended that the set-up of the M&E system begin with an overall readiness assessment, (Kusek and Rist, 2004). The next steps involve establishing a good theory of change (ToC), including setting up indicators, identifying roles and functions, ensuring the necessary tools and clarifying the purposes for M&E (Kusek and Rist, 2004, Pasanen, T., and Shaxson, L.,2016). Using the results of the M&E for decision making, developing capacity and enhancing a supportive environment for M&E are also parts of the process (Evalsed, 2013). • M&E are embedded within the framework of the ESIF implementation, supporting the results-oriented approach of the EU Cohesion Policy. Both functions are built on the program logic of intervention and they support observing progress, as well as evidence-based decision making (EC, 2014). Key requirements are set out in the CPR and the Fund-specific regulations and their implementation is constantly observed and assessed (EC, 2018). • Previous assessments of M&E systems and capacity improvement processed used a mix of methods. These include data analysis, key informant interviews, focus groups and surveys of different types of stakeholders, as well as case studies. The focal point of most assessments 71 OECD, 2019. 72 EC 2019. 73 World Bank, 2016. 74 Unpublished 67 entails the benchmarking of the system against a predefined set of criteria and expected level of performance, which is either set out explicitly by program rules or agreed as part of the methodology, by expert opinion (EC, 2018; Gorgens and Kusek, 2009). 68 Annex 3: M&E communication channels Target Targeted key messages Communication channels Proposed timeline* audiences MEF staff Highlight the results of the M&E RAS The M&E Working Group MA’s and IBs’ (especially effects) and sustainability of those experts as well as results Online events for small groups. other GoR During these events, other ministries and The results and lessons of the M&E RAS will products will be introduced, and TBD agencies shape the implementation of the 2021–2027 the audience will be encouraged programming period. to review/contribute Small keep-in-touch events on the WB premises with key stakeholders High level visits/missions Cross-country knowledge exchange events General public WB worked with the MEF to streamline and Web: WB RO/country webpage TBD (Citizen, improve the M&E delivery system of the & MEF/Evaluation Unit academic, 2021–2027 programming period. This support webpage (http://www.evaluare- practitioner, resulted in: structurale.ro/) & others NGO/civil a) Improved strategy for the ESIF M&E Social media society) system, thus leading to an improved Events, infographics, blogs, evaluation culture; vlogs, and videos. b) Enhanced evaluation planning. Publicly disclosed reports. c) Better coordination between ESF+, ERDF and CF interventions. For the Evaluation specialists, we aim to encourage academics and their institutions to consider the M&E RAS as a rich field for further research and publication. World Bank Staff The M&E RAS is a successful, WB intranet webpage/shared TBD (ECA) implementation-focused modality of folder collaboration on cross-cutting issues, which Infographics, blogs, vlogs and World Bank Staff could be replicated in other EU MS or other online learning events (Asia, Latin accession/neighborhood countries. Virtual Brown Bag Lunches America, Africa (WB internal dissemination teams) events) on subjects regarding: M&E system enhancement, building an evaluation culture in socioeconomic development field and M&E practices in the EU structural funds environment Other Best practice on how to build an M&E system Technical level meeting for TBD International in socioeconomic development investment disseminating results with Organizations funds. relevant IFIs/international organizations – OECD, EC Country Desk/DG EMPL *TBD with the counterpart 69 Annex 4: Quality control and quality assurance checklist for deliverables Criteria Checked: Observations: 1. The extent to which the activity/deliverable responds to the requirements and needs of the beneficiary 2. The extent to which the activity/output covers the subject 3. The extent to which the process is open and participatory 4. The design of the activities is appropriate 5. The data used for the assessment is reliable 6. The information sources are clearly identified 7. The analysis is rigorous 8. Limitations are clearly stated 9. The results of the assessment are credible 10. The conclusions are impartial and realistic 11. The recommendations correspond to the conclusions and are useful to the decision makers and other stakeholders 12. The implementation of the project took place as described in the initial report 70 Annex 5: Documents available for review Topic Possible questions to Documents explore M&E system What are the objectives of CPR and Fund – specific regulations (see Table objective and the M&E systems of ESI 1), as well as subsequent implementing legal basis Funds in Romania and how regulations are they grounded in the Partnership Agreement and OPs legislation? Evaluation Plans at OP and PA level 75 Does the current system of indicators support results- orientation? What are the gaps and challenges related to the system of indicators? Is there an M&E Plan? How is it designed and implemented? M&E system What is the available OPs functions capacity for M&E at the Government decisions regarding the set-up and level of each OP? functioning of the MEF, MPARD, MARD, and How is the routine program other institutions with responsibilities related to monitoring performed? the M&E function of the ESI Funds OPs. How is data collected, Organization and functioning regulations of MCs validated, aggregated and for each OP and the PA and other bodies with reported? responsibilities related to the M&E function How is evaluation Project and Program Monitoring Operational performed? Procedures of entities with responsibilities related to the M&E function How are M&E Results disseminated and used for AIRs of OPs for 2017, 2018 and 2019. performance management? Other reports available on the progress of the OPs Institutional What are the organizational Government decisions regarding the set-up and set-up, levels structures with M&E functioning of the MEF, MAs/IBs and other decision, functions? institutions with responsibilities related to the mandates M&E function of the ESI Funds OPs. What institutional arrangements are in place to Organization and functioning rules of MCs and plan, coordinate and manage other bodies with responsibilities related to the the M&E System M&E function Project and Program Monitoring Operational Procedures of entities with responsibilities related to the M&E function Existing partnerships/protocols with 3rd parties for data collection (is any), such as with the National Statistics Institute or research institutes 75 https://www.fonduri-ue.ro/images/files/documente-relevante/2016/RO.Plan.evaluare.AP.pdf 71 Roles and Who are the stakeholders in Government decisions regarding the set-up and responsibilities the M&E systems of ESI functioning of the MEF, MAs/IBs and other Funds? institutions with responsibilities related to the M&E function of the ESI Funds OPs. What are the roles and responsibilities of each Organization and functioning rules of MCs and stakeholder? other bodies with responsibilities related to the M&E function Who are the “champions” supporting M&E? Project and Program Monitoring Operational Procedures of entities with responsibilities Who are the stakeholders related to the M&E function ensuring data quality? Who are the stakeholders ensuring data dissemination and use? Tools What guidance is available EC guiding documents, such as: for the staff responsible for - ESF M&E guidance 76 M&E? - ESF Annex D, – practical guidance on data collection - and validation 77 - ERDF M&E guidance 78 - Guidance Document on Indicators of Public Administration Capacity Building - Guidance Document on Evaluation Plans Terms of Reference for Impact Evaluations Guidance on Quality Management of External Evaluations 79 - Guidelines establishing and implementing the evaluation plan of 2014–2021 Rural Development 80 - WORKING PAPER Indicators for integrated territorial and urban development 81 National Guiding Documents - POCU Indicators Guidelines and Fiches 82 - ROP Indicators Guide 83 76 https://ec.europa.eu/sfc/en/2014/document/esf-monitoring-and-evaluation-guidance 77 https://ec.europa.eu/sfc/en/system/files/ged/Annex%20D%20- %20Practical%20guidance%20on%20data%20collection%20and%20validation.pdf 78 https://ec.europa.eu/regional_policy/sources/docoffic/2014/working/wd_2014_en.pdf 79 https://ec.europa.eu/regional_policy/sources/docoffic/2014/working/evaluation_plan_guidance_en.pdf 80 https://www.madr.ro/docs/dezvoltare- rurala/evaluare_/Ghid_pentru_stabilirea_si_implementarea_planului_de_evaluare_pentru_programele_de_dezvo ltare_rurala_2014-2020.pdf 81 https://www.espon.eu/sites/default/files/attachments/Working%20Paper%20Indicators%20for%20integrated%20 development.pdf 82 http://mfe.gov.ro/pocu/fise/ 83 www.inforegio.ro › Ghidul_de_indicatori_POR__RO_Final 72 - OPTA Indicators Guide 84 - Methodology for monitoring physical and financial indicators of OPC Capacity & What is the human capacity AIRs of OPs incentives for M&E? Other reports available describing the capacity of How is the M&E culture the M&E system, such as the EC Study on the promoted/supported? M&E systems of the ESF (2017), reports developed by the WB on POCU, 85 evaluation reports of OPs, coordinated by the MEF or by MAs in other ministries. Good practices What practices/instruments Relevant documents describing M&E systems of and other for other Member States or relevant programs implemented internationally, experiences used internationally could be with particular focus on OPs implementing ESI from EU applied for improving the Funds in other EU member states (so looking for Management of the ESI best practices), such as EC Study on the M&E Funds in Romania? systems of the ESF (2017) Annex 6: Minute of the Inception Workshop INCEPTION WORKSHOP MINUTES OF THE MEETING Date: August 20, 2020 Location: Webex PARTICIPANTS World bank (WORLD BANK) Team: Victor Sulla, Annette Richter, Adina Iorganda, Elena Botezatu, Alina Bosoi, Anastasia Gadja, Andreea Scarlat, Madalina Rudareanu Ministry of European Funds (MEF): Ionut Micu MEF Evaluation Unit: Claudia Magdalina, Alexandra Voiculescu, Maria Mălureanu Workshop guests: Alexandra Ionita, Alina Maria Iuga ( N-W RDA, expert), Anca Ginghina, Andreea Scarlat, Angelica Vladescu (MEF DGPCS – BEP, Head of Unit), Dragos Vlad (MEF SMIS and IT Directorate, Director), Aurel Moise, (MEF, COP, counselor), Catalin Lacatusu, Cerasela Doniceanu, Claudia Coman, Claudia Vasilca (AC MA, Public manager), Cristina Badescu (MEF DCSM, Director), Cristina Ciobotaru (CTE, counselor), Cristina Patrascoiu (MEF, TA OP MA, Director), Diana Iliescu (MEF, DCSM, counselor), Dumitru Dan Nicula (RDA BI, director), Gina Paun (RDA BI, Head of Unit), Izabella Gradinescu (RDA N-E, Head of Unit), Luiza Radu (ROP MA, general director), Madalina Chera (MEF, TA OP MA, Head of Unit), Magda Lungu (RDA S-W OLTENIA, head of Unit), Maria IStrate (Transport IB, Director, Marilena Alecu (RDA S-W OLTENIA, director), Mihaela Raducan (MEF DCSM, Head of Unit), Ovidia Caba (RDA Center, Director), Radac Georgiana (RDA W, Director), Roxana Chitu (AC OP MA, counselor), Simona Vasile (CTE- Evaluation unit, counselor), Sorin Grigorescu (RDA NE, Head of Unit), Tudor Udrea (RDA center, head of Unit), Vasile Asandei (RDA NE, General Director), Vladimir Rovintescu (HC OP MA, General Director, Gabriela Macoveiu, Cristina Radu, Raluca Dumitru, Mihaela Raducan (MEF DCSM, head of Unit). 84 https://www.fonduri-ue.ro/images/files/programe/AT/POAT_2014/Ghid_indicatori_POAT_2014- 2020_august_2016.pdf 85 Advisory Services Agreement on Supporting the Implementation of Romania's Human Capital Operational Programme (POCU) 2014-2020, signed between the MEF and the International Bank for Reconstruction and Development on October 12, 2017 (POCU RAS). 73 Background Romania is expected to receive approximately EUR 30.6 billion of funding from European Structural and Investment Funds. These funds will support operational programs across all 11 EU thematic objectives. The Government of Romania will also need to implement changes in monitoring and evaluation (M&E) practices. Therefore, the Ministry of European Funds have engaged the World Bank through a reimbursable advisory service (RAS) agreement aimed at Improving Monitoring and Evaluation Capacity in the Context of EU-funded Programs in Romania. The World Bank will collaborate with the Government of Romania in evaluating the current M&E system in the 2014-2020 period, determining options for system improvements, and supporting implementation of the recommendations. The World Bank’s RAS team jointly with the MEF organized a workshop introducing the new RAS engagement. The main objective of the workshop was to present the main RAS activities to the government counterparts, clarify and answer any questions they may have had, and establishing a working relationship with the stakeholders. WORKSHOP AGENDA: • Introductory points by the General Director and head of the MEF evaluation unit; • World Bank’s Presentation “Improving Monitoring and Evaluation Capacity in the Context of EU-funded Programs in Romania (2021-2027) objectives and timeline”; • Questions & Answers MAIN POINTS: MEF introductory points The GD introduced the project and highlighted the importance of the RAS-funded activities in light of the proposed changes in the upcoming 2021-2027 programming period. The GD focused on the newest elements in the key new M&E system being under advanced stage of negotiations with the EC. Data quality for measuring performance framework indicators continues to be a priority for system audits. The successful implementation of the OPs will depend on the coordinated efforts of all the stakeholders and will ultimately contribute to the improvement of the quality of life under all aspects and levels. The key areas of the change will be: • Increased frequency of transmission of financial data to the Commission from 3 to 4 times a year. The reached values of the indicators will be transmitted twice a year. • The evaluation plans will need to be improved, while annual implementation planning process will no longer be necessary; • Performance framework will include all program indicators (not only those that represent 60% of the priority axis allocation), and the final allocation for the last two years will depend on their fulfillment. The World Bank presentations The task Team leader of the RAS presented the rationale, objectives and scope of the project. The RAS will be implemented on a three-year period starting August 2020, aiming to support the MEF using an evidence– based approach in developing and operationalizing an M&E system to track and monitor EU funded programs for 2021-2027. The RAS consists of the following three main activities: (i) Assessing of the existing M&E system of EU-funded measures; (ii) Developing options to improve the M&E system of EU co-funded investments; and (iii) Supporting in implementing improvements in the M&E system. Subsequently, the World Bank team presented the RAS project’s objectives and main activities: (1) the inception report and the key methodological aspects governing RAS’s implementation, (2) description of all envisioned activities, (3) the timeline, and (4) the collaborative arrangements and the next steps of the engagement. The methodological approach for delivering the RAS activities entails strong conceptual foundation and evidence- 74 based analysis. The building blocks of the RAS activities will be: (i) an integrated M&E framework, (ii) a M&E system assessment tool from international best experience with M&E system assessments, and (iii) the World Bank ten steps approach to a results-based monitoring and evaluation system. Qualitative and quantitative data collection methods will be utilized including: focus groups, consultations & surveys with broad set of beneficiaries and stakeholders. A collaborative approach will be adopted in developing M&E options that will be discussed with different stakeholders. Q&A session The presentation was followed by a substantive discussion. The key points were: • An effective cooperation should ensue between the RAS team and all the stakeholders. MEF has already requested contact information from key stakeholders to facilitate this dialogue going forward. • Establish communication and information sharing between RAS and other M&E TA activities. It is important to consider the timeline of the M&E process at the level of different OPs. Some are more advanced, while newly established MAs (e.g. RDAs) are at the beginning stages. In this regard, MEF emphasized importance of the coordination across different OPs. The RAS should play a key role in establishing an integrated and fully coordinated M&E system. The RAS will focus on building an integrated M&E system; thus, it will have to correlate with other TA projects with M&E components. • The MEF team also pointed that the RAS aims to analyze and provide recommendations for improving the MySMIS IT system from the perspective of the data collection for M&E systems. This will not include evaluation and selection of projects, financial control, audit etc. • The work will be focused on developing M&E system centered on performance and performance assessment, and less focused on regulations. The codesign principle will be used when deciding what needs to be changed or improved. • Recommendations for improvement should be reached in a participatory manner. The World Bank team pointed to the next steps in the implementation of the RAS, such as the evaluation of the current M&E system and the importance of establishing connections with all the OPs, potentially through a working group with representatives from all institutions involved. The group could also facilitate data exchange and sharing. The importance of engagement of the relevant actors in the process was underlined. List of abbreviations CTE – European Territorial Cooperation DCSM –System Coordination and Monitoring Directorate IBs -Intermediate Bodies M&E – Monitoring and Evaluation MA – Management Authority MEF – Ministry or European Funds RAS – Reimbursable Service Agreement RDA – Regional Development Agency ROP – Regional Operational Program TA – Technical assistance 75 References: 1. 2014–2020 Administrative Capacity OP; 2. 2014–2020 Competitiveness OP; 3. 2014–2020 Fisheries and Marine OP; 4. 2014–2020 Human Capital OP; 5. 2014–2020 Hungary-Slovakia-Romania-Ukraine ENI CBC Programme 2014–2020; 6. 2014–2020 Joint OP Black Sea Basin 2014–2020; 7. 2014–2020 Joint OP Romania-Ukraine; 8. 2014–2020 Large Infrastructure OP; 9. 2014–2020 OP for assisting the Disadvantaged Persons; 10. 2014–2020 Regional OP; 11. 2014–2020 Technical Assistance OP; 12. 2021–2027 Draft Health OP; 13. 2021–2027 Draft Human capital OP; 14. 2021–2027 Draft Just Transition OP; 15. 2021–2027 Draft Regional OPs (next programming period); 16. 2021–2027 Draft Smart growth and digitization OP; 17. 2021–2027 Draft Social inclusion and integrated territorial development OP; 18. 2021–2027 Draft Sustainable development OP; 19. 2021–2027 Draft Technical Assistance OP; 20. 2021–2027 Draft Transport OP; 21. 2021–2027 Indicators Guidelines, MEF, 2020; 22. Assessing progress in meeting performance framework indicators, Initial Report, MEF, July 2020; 23. Cohesion policy OPs general architecture 2021–2027 available at http://mfe.gov.ro/minister/perioade-de-programare/perioada-2021-2027/; 24. EC Communication Better Regulation: Delivering better results for a stronger Union, 2016 https://eur-lex.europa.eu/legal- content/EN/TXT/PDF/?uri=CELEX:52016DC0615&from=EN; 25. Edmunds, R., Marchant, T. (2008) Official Statistics and Monitoring and Evaluation Systems in Developing Countries: Friends or Foes? https://paris21.org/sites/default/files/3638.pdf 26. ESPON 2020 Programme; 27. EU 2021–2027 Simplification Handbook, https://ec.europa.eu/regional_policy/sources/docgener/factsheet/new_cp/simplification _handbook_en.pdf; 28. EU Budget for the Future Brochure, EC website, https://ec.europa.eu/commission/sites/beta-political/files/budget-may2018-new- framework-glance_en.pdf; 29. European Commission (EC) Monitoring and Evaluation of European Cohesion Policy, European Social Fund Guidance document, available at: https://ec.europa.eu/sfc/en/system/files/ged/ESF%20monitoring%20and%20evaluatio n%20guidance.pdf; 30. European Semester - Recommendation for a Council Recommendation on the 2020 National Reform Programme of Romania and delivering a Council opinion on the 2020 76 Convergence Programme of Romania, EC, 2020, https://eur-lex.europa.eu/legal- content/EN/TXT/PDF/?uri=CELEX:52020DC0523&from=EN; 31. EVALSED - Evaluation guide: The resource for the evaluation of Socio-Economic Development; 32. Evaluating Administrative Burdens of Structural and Investment Funds Beneficiaries, April 2015; 33. Evaluating Plan for Partnership Agreement, available at https://www.fonduri- ue.ro/images/files/documente-relevante/2016/RO.Plan.evaluare.AP.pdf; 34. Evaluation culture measurement report in Romania in the context of the EU cohesion policy, first measurement cycle, January 2013; 35. Evaluation culture measurement report in Romania in the context of the EU cohesion policy, second measurement cycle, December 2013; 36. Evaluation culture measurement report in Romania in the context of the EU cohesion policy, third measurement cycle, September 2014; 37. Evaluation report – ILMT, MEF, 2019; 38. FEAD mid-term evaluation report 2018 http://publications.europa.eu/resource/cellar/51421b36-54f8-11e9-a8ed- 01aa75ed71a1.0001.01/DOC_1; 39. FHI 360 (2013), Participatory Monitoring and Evaluation System Assessment Tool, Framework and Operational Guide for implementation https://www.fhi360.org/sites/default/files/media/documents/fhi360-sat-oct2013.pdf; 40. Gertler, Paul J.; Martinez, Sebastian; Premand, Patrick; Rawlings, Laura B.; Vermeersch, Christel M. J. 2016. Impact Evaluation in Practice, Second Edition. Washington, DC: Inter-American Development Bank and World Bank. https://openknowledge.worldbank.org/handle/10986/25030Other 41. Guidance document “Annex D—Practical Guidance on Data Collection and Validation.”, https://ec.europa.eu/sfc/en/system/files/ged/Annex%20D%20- %20Practical%20guidance%20on%20data%20collection%20and%20validation.pdf 42. Guidance document “Programming Period 2014–2020: Monitoring and Evaluation of European Cohesion Policy, European Social Fund”, https://ec.europa.eu/sfc/en/system/files/ged/ESF%20monitoring%20and%20evaluatio n%20guidance.pdf 43. http://www.oecd.org/gov/policy-monitoring-evaluation.htm; 44. Impact assessment accompanying ERDF and CF Regulation, EC, 2018 https://eur- lex.europa.eu/legal-content/EN/TXT/?uri=SWD%3A2018%3A283%3AFIN; 45. Impact assessment accompanying ESF+ Regulation & European Globalization Adjustment Fund (EGF), EC, 2018 https://ec.europa.eu/commission/sites/beta- political/files/budget-may2018-esf-egf-swd_en.pdf; 46. Indicators’ Guide and indicators’ fishes for POAT 2014–2020; 47. Indicators’ Guide and indicators’ fishes for POC 2014–2020; 48. Indicators’ Guide and indicators’ fishes for POCA 2014–2020; 49. Indicators’ Guide and indicators’ fishes for POCU 2014–2020; 50. Indicators’ Guide and indicators’ fishes for POIM 2014–2020; 51. Indicators’ Guide and indicators’ fishes for POR 2014–2020; 52. INTERACT III OP 2014–2020; 53. Interreg – IPA CBC Romania-Serbia 2014–2020 Programme; 54. INTERREG EUROPE 2014–2020 Cooperation Programme 2014–2020; 77 55. Interreg V-A Romania – Bulgaria Programme; 56. Interreg V-A Romania-Hungary OP 2014–2020; 57. Joint OP Romania – Republic of Moldova 2014–2020; 58. Kusek, Rist (2004) Ten Steps to a Results-Based Monitoring and Evaluation System https://www.oecd.org/dac/peer- reviews/World%20bank%202004%2010_Steps_to_a_Results_Based_ME_System.pdf 59. Leiter, T. (2016), Key considerations for monitoring and evaluation of community- based adaptation to climate change: lessons from experience http://www.adaptationcommunity.net/?wpfb_dl=381 60. Linda Morra Imas, Ray Rist: The Road to Results, 2009 and models presented on University of Wisconsin-Madison website; 61. LIOP 2014–2020 Evaluation Plan, 2017; 62. Mackay, K. (2007), How to build M&E systems to support better government (English). Washington, D.C.: World Bank Group http://documents.worldbank.org/curated/en/689011468763508573/How-to-build-M-E- systems-to-support-better-government; 63. Mackay, K. (2007), How to build M&E systems to support better government (English). 2007. Washington, D.C.: World Bank Group; 64. McMaster, Irene and Kah, Stefan (2017) The Performance Framework in Cohesion Policy: Expectations and Reality. https://core.ac.uk/reader/146504550; 65. MEF – Operational Procedure PO.DGPSMISCSCEI.09 for evaluating the Partnership Agreement and operational procedures, General Directorate for Programming, SMIS, System Coordination and European and international cooperation, 14.05.2018; 66. Administrative Capacity OP 2014–2020 Evaluation Plan, 2018 67. Pasanen, T., and Shaxson, L. (2016) How to design a monitoring and evaluation framework for a policy research project. A Methods Lab publication, London: Overseas Development Institute.https://www.odi.org/sites/odi.org.uk/files/odi- assets/publications-opinion-files/10259.pdf 68. Performance Framework Guide, MEF, November 2014; 69. POCU 2014–2020 Evaluation Plan, 2017; 70. Polverari, Laura (2016) 2014–2020 EU cohesion policy : results-orientation through better monitoring. European Structural and Investment Funds Journal, 4 (1). pp. 26–34. ISSN 2196–8268 https://strathprints.strath.ac.uk/57614/; 71. POR 2014–2020 Multiannual Evaluation Plan, 2019; 72. Praefcke, Dieter Muller (2010), The use of monitoring and evaluation in agriculture and rural development projects Findings from a review of implementation completion reports http://www.fao.org/3/am292e/am292e00.pdf; 73. Proposal for a Regulation of the European Parliament and of the Council laying down common provisions on the European Regional Development Fund, the European Social Fund Plus, the Cohesion Fund, and the European Maritime and Fisheries Fund and financial rules for those and for the Asylum and Migration Fund, the Internal Security Fund and the Border Management and Visa Instrument https://eur-lex.europa.eu/legal- content/EN/TXT/?uri=COM%3A2018%3A375%3AFIN; 74. Proposal for a Regulation of the European Parliament and of the Council on the European Regional Development Fund and on the Cohesion Fund https://eur- lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2018%3A372%3AFIN; 78 75. Proposal for a Regulation of the European Parliament and of the Council on the European Social Fund Plus (ESF+) https://eur-lex.europa.eu/legal- content/EN/TXT/?qid=1530189516904&uri=CELEX:52018PC0382; 76. Regulation for the organization and functioning of the MEF, November 2019; 77. Retrospective evaluation of POS DRU - employment field (2020) http://www.evaluare- structurale.ro/ro/list-viewreports-menu/details/8/189/rapoarte-evaluare-anul- public%C4%83rii-2016,-2017,-2018,-2019,-2020-implementarea-planului-de- evaluare-a-programului-opera%C8%9Bional-capital-uman-2014-2020-evaluarea- interven%C8%9Biilor-%C3%AEn-domeniul-ocup%C4%83rii-for%C8%9Bei-de- munc%C4%83; 78. Simplification Manual for the next programming period (Romanian version) available at http://mfe.gov.ro/wp- content/uploads/2019/11/f43d9206cfae98c16c5d7aafdcbd29a3.pdf; 79. Special Report No 15/2017 “Ex ante conditionalities and performance reserve in Cohesion: innovative but not yet effective instruments”; 80. Steffen, Soulejman, Janus (2016), Becoming a Knowledge-Sharing Organization: A Handbook for Scaling Up Solutions through Knowledge Capturing and Sharing; 81. Study on a system of common indicators for ERDF and CF post 2020 https://ec.europa.eu/regional_policy/en/information/publications/studies/2018/develop ment-of-a-system-of-common-indicators-for-european-regional-development-fund- and-cohesion-fund-interventions-after-2020-part-i-thematic-objective-1-3-4-5-6; 82. The Danube Transnational Programme 2014–2020; 83. UNAIDS, (2009), 12 Components Monitoring and Evaluation System Assessment Tool https://www.measureevaluation.org/resources/training/capacity-building-resources/m- e-of-hiv-aids-programs-in-india-english/session-3-routine-data-and-systems- assessment/systems- assessment/MERG%2012%20Components%20ME%20System%20Strengthening%20 Tool.pdf/view; 84. URBACT III OP 2014–2020; 85. USAID, 2014, Technical Brief: Measurement of M&E System Strengthening Application, Lessons, and Recommendations from a Retrospective Case Study Approach in Côte d’Ivoire and Nigeria https://www.measureevaluation.org/resources/publications/fs-14-130; 86. World Health Organization (2009), Monitoring and evaluation of health systems strengthening. An operational framework https://www.who.int/healthinfo/HSS_MandE_framework_Nov_2009.pdf; 79 ROMANIA Project title "Implementation of the Evaluation Plan of the Human Capital Operational Program 2014-2020" Beneficiary of project: Ministry of European Funds Publishing date: October 2020 Project cofinanced from the European Social Fund through the Human Capital Operational Program 2014-2020, Priority Axis 7: Technical Assistance „The content of this material does not necessarily reflect the official position of the European Union or of the Government of Romania”. 80