Design for Impact INTERNATIONAL DE VELOPMENT IN FOCUS A State Aid Evaluation for Romania Georgiana Pop, Mariana Iootty, Miriam Bruhn, and Claudia Ruiz Ortega I N T E R N AT I O N A L D E V E L O P M E N T I N F O C U S Design for Impact A State Aid Evaluation in Romania GEORGIANA POP, MARIANA IOOTTY, MIRIAM BRUHN, AND CL AUDIA RUIZ ORTEGA © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http:// creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: Pop, Georgiana Mariana Iootty, Miriam Bruhn, and Claudia Ruiz Ortega. 2021. Design for Impact: A State Aid Evaluation in Romania. Washington, DC: World Bank. doi:10.1596/978-1-4648-1703-8. License: Creative Commons Attribution CC BY 3.0 IGO Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third- party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; e-mail: pubrights@worldbank.org. ISBN: 978-1-4648-1703-8 DOI: 10.1596/978-1-4648-1703-8 Cover image: World Bank / Veronica Elena Gadea Cover design: Debra Naylor / Naylor Design Inc. Contents Acknowledgments  vii Executive Summary  ix Abbreviations  xxiii CHAPTER 1 Introduction  1 Notes  7 References  8 CHAPTER 2 Description of State Aid Schemes (Step 1)   9 De minimis scheme to incentivize access to finance for MSMEs (the RCG scheme)  9 State aid scheme to support regional development and job creation (the MPF scheme)  13 State aid scheme to support the upgrade and modernization of the R&D&I (the MEF scheme)  14 Notes  17 References  17 CHAPTER 3 Definition of Evaluation Questions and Result Indicators (Step 2)  19 Dataset  19 Evaluation Questions and Result Indicators   21 Notes  25 References  27 CHAPTER 4 Empirical Assessment (Step 3)   29 The Romanian Counter-Guarantee Fund Scheme   29 The Ministry of Public Finance Scheme   40 The Ministry of European Funds Scheme   50 Notes  60 References  62 CHAPTER 5 Conclusions: Implications for State Aid Design and Implementation, and Lessons Learned (Step 4)   63 Results  63 Key lessons for future impact evaluations   67 Notes  69 References  70  iii iv | Design for Impact Appendix A Selection Criteria for the MEF Scheme (Scoring Phase)   71 State Aid Intensity for the Ministry of European Funds Appendix B  Scheme  75 Appendix C List of Variables in the Business Registry Dataset   77 List of Variables in the Structural Business Survey Appendix D  dataset  79 Methodology Applied to Estimate Markups and Appendix E  Revenue-Based Total Factor Productivity   81 ­ Romanian Counter Guarantee Scheme: Firms Included in Appendix F  the Analysis (Auxiliary Tables)   85 Identification Strategy of the Romanian Counter Guarantee Appendix G  Scheme Evaluation: Robustness Checks   89 Impact of the Romanian Counter-Guarantee Fund Scheme: Appendix H  Robustness Checks (Auxiliary Tables)   95 Appendix I Ministry of Public Finance Scheme: Firms Included in the Analysis (Auxiliary Tables)   99 Appendix J Identification Strategy of the Ministry of Public Finance Scheme Evaluation: Robustness Checks   101 Ministry of European Funds Scheme: Firms Included in the Appendix K  Analysis (Auxiliary Tables)   105 Identification Strategy of the Ministry of European Funds Appendix L  Scheme Evaluation: Robustness Checks   107 Boxes ES.1 State aid framework and implementation in Romania   x ES.2 The European Commission’s approach to state aid impact evaluation   xiii 1.1 EU definition of state aid   2 3.1 Running an ex post evaluation of a state aid scheme: Key data features   20 Figures B.ES1.1 Share of total state aid expenditure by policy objective, Romania vs. EU-28 average  xi B.ESB1.2 Share of total state aid expenditure by aid instrument, Romania vs. EU-28 average  xi 1.1 Share of total state aid expenditure by policy objective: Romania vs. EU countries  4 1.2 Share of total state aid expenditure by aid instrument: Romania vs. EU countries  5 1.3 The state aid evaluation plan: Analytical steps   6 2.1 Micro, small, and medium enterprises’ access to loans index: 2017 and 2018  11 2.2 Micro, small, and medium enterprises’ access to equity index: 2017 and 2018  12 2.3 Gross domestic product per capita and unemployment rate across counties in Romania, 2018   13 2.4 Research and development expenditure in EU countries, 2018   15 4.1 Effects of the Romanian Counter-Guarantee Fund scheme on beneficiaries over time, 2012–17   33 Contents | v 4.2 Direct effects of the Romanian Counter-Guarantee Fund scheme on beneficiary firms across regions   34 4.3 Direct effects of the Romanian Counter-Guarantee Fund scheme on beneficiary firms across sectors   35 4.4 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiaries over time   37 4.5 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiaries across regions   38 4.6 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiaries across sectors   39 4.7 Direct effects of the Ministry of Public Finance scheme on beneficiaries over time (rejected applicants control group)   45 4.8 Direct effects of the Ministry of Public Finance scheme on beneficiaries over time (matched control group)   46 4.9 Employment trends for Ministry of Public Finance beneficiaries and rejected applicants  48 4.10 Spillover effects of the Ministry of Public Finance scheme on nonbeneficiaries in beneficiary sectors over time   50 4.11 Direct effects of the Ministry of European Funds scheme on beneficiaries over time (rejected applicants control group)   56 4.12 Direct effects of the Ministry of European Funds scheme on beneficiaries over time (matched control group)   57 4.13 Spillover effects of the Ministry of European Funds scheme on nonbeneficiaries in beneficiary sectors over time   59 G.1 Entry rates of treatment and control siruta territorial units (level 2) over time, 2008–17  93 Tables B.ES.1.1 Recent RCC opinions on state aid   x ES.1 Summary of results and main areas for improving state aid design and implementation   xix ES.2 Areas for improving data collection and analysis   xx 2.1 Description of the three state aid schemes under analysis   10 3.1 Grantor dataset: Available information by aid scheme   21 3.2 Master dataset by aid scheme   22 3.3 Evaluation questions and result indicators by aid scheme   23 4.1 Romanian Counter Guarantee Fund scheme beneficiaries by sector classification  30 4.2 Romanian Counter Guarantee Fund scheme beneficiaries by region   30 4.3 Direct effects of the Romanian Counter Guarantee Fund scheme on beneficiary firms  32 4.4 Spillover effects of the Romanian Counter Guarantee Fund scheme on nonbeneficiary firms    36 4.5 Competition effects of the Romanian Counter Guarantee Fund scheme  40 4.6 Ministry of Public Finance scheme beneficiaries and rejected applicants by cohort  41 4.7 Ministry of Public Finance scheme beneficiaries and nonbeneficiaries by sector  41 4.8 Ministry of Public Finance scheme beneficiaries and nonbeneficiaries by region  42 4.9 Direct effects of the Ministry of Public Finance scheme on beneficiaries  44 4.10 Spillover effects of the Ministry of Public Finance scheme on nonbeneficiary firms in beneficiary sectors   49 4.11 Competition effects of the Ministry of Public Finance scheme   51 4.12 Ministry of European Funds scheme beneficiaries and rejected applicants by cohort  52 vi | Design for Impact 4.13 Ministry of European Funds scheme beneficiaries and nonbeneficiaries by sector  53 4.14 Ministry of European Funds scheme beneficiaries and nonbeneficiaries by region  54 4.15 Direct effects of the Ministry of European Funds scheme on beneficiaries  56 4.16 Spillover effects of the Ministry of European Funds scheme on nonbeneficiaries in ­ beneficiary sectors   58 4.17 Competition effects of the Ministry of European Funds scheme   60 5.1 Data requirements for impact evaluations   69 A.1 Selection criteria for Ministry of European Funds scheme—Operation 2.1.1. “Research projects in partnership between universities/research and development institutions and enterprises”   71 A.2 Selection criteria for Ministry of European Funds scheme—Operation 2.1.2. “High-level research and development projects involving foreign specialists”  72 A.3 Selection criteria for Ministry of European Funds scheme—Operation 2.3.3. “promoting innovation in enterprises”   73 ­ B.1 State aid intensity (under Ministry of European Funds scheme) by R&D expenditure categories and firm size   75 C.1 List of variables in the Business Registry database   77 D.1 List of variables in the Structural Business Survey dataset”   79 F.1 Distribution of the Romanian Counter-Guarantee Fund scheme beneficiary firms by their 2-digit nomenclature of economic activities sector   85 F.2 First and last year’s Romanian Counter-Guarantee Fund scheme support received  87 G.1 Pre-scheme summary statistics for Romanian Counter-Guarantee Fund scheme ­ beneficiaries and control group   90 G.2 Pre-scheme summary statistics for noneligible firms in siruta territorial units (level 2) with and without Romanian Counter-Guarantee Fund beneficiaries  91 G.3 Test of parallel trends in the pre-scheme period   92 H.1 Impact of the Romanian Counter-Guarantee Fund scheme on beneficiary firms over time   95 H.2 Impact of the Romanian Counter-Guarantee Fund scheme on beneficiary firms  96 H.3 Impact of the Romanian Counter-Guarantee Fund scheme on beneficiary firms (using beneficiaries opened any year)   96 H.4 Pre-scheme trends in outcomes of beneficiary firms (that joined the Romanian Counter-Guarantee Fund scheme in 2013 or later) and their control group  96 H.5 Impact of the Romanian Counter-Guarantee Fund (RCG) scheme on beneficiary firms that joined the RCG scheme in 2013 or later  97 H.6 Spillover effects of the Romanian Counter-Guarantee Fund scheme over time  97 H.7 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiary firms  98 H.8 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiary firms by employment rate of region  98 H.9 Impact of the Romanian Counter-Guarantee Fund scheme on beneficiaries’ market share using an alternative measure of market share  98 I.1 Pre-scheme summary statistics for Ministry of Public Finance scheme beneficiaries and ­ nonbeneficiaries   100 J.1 Pre-scheme summary statistics for Ministry of Public Finance scheme beneficiaries and ­ control groups  102 J.2 Pre-scheme summary statistics for nonbeneficiaries   103 K.1 Pre-scheme summary statistics for Ministry of European Funds scheme beneficiaries and nonbeneficiaries   106 L.1 Pre-scheme summary statistics for Ministry of European Funds scheme ­ beneficiaries and control groups   108 L.2 Pre-scheme summary statistics for nonbeneficiaries   108 Acknowledgments This state aid evaluation builds on the Technical Assistance on Assessing the Effects of State Aid on Market Outcomes in Romania (2018–2020). The technical assistance was funded by the European Union via the Structural Reform Support Programme (SRSP) and implemented by the World Bank in cooperation with the European Commission’s Directorate-General for Reform Support (DG REFORM). The advisory services for the Romanian Competition Council were performed by a World Bank team led by Georgiana Pop, World Bank Global Lead for Competition Policy. The evaluation was prepared by a team led by Georgiana Pop and including Mariana Iootty (Senior Economist, Global Competition Policy Team), Miriam Bruhn (Senior Economist, Development Research Group), Claudia Ruiz Ortega (Economist, Development Research Group), Jorge Pena (Expert, Global Competition Policy Team), and Andrei Silviu Dospinescu (Local Project Coordinator). Gonçalo Coelho Banha (Expert, Global Competition Policy Team) provided selected inputs. Richard Record (Lead Economist), Catalin Pauna (Senior Economist), and Hernan Winkler (Senior Economist), all with the World Bank, provided excel- lent peer review comments. Christine Qiang (Practice Manager) ensured overall implementation oversight. The World Bank team is grateful to the Romanian Competition Council, the Ministry of Public Finance, the Ministry of European Funds, and the Romanian Counter-Guarantee Fund for fruitful discussions. Comments on earlier versions of this evaluation were provided by a team from the Romanian Competition Council that included Alis Barbulescu, Daniel Diaconescu, Sorina Palan, Stefan Popescu, Mihai Dragoi (State Aid Department), Oana Neg (External Relations and Communications Department), Paul Prisecaru, Oana Romano, Vlad Nerau, Ionut Busega, and Mihai Busu (Chief Economist’s Office). Irina Avram, Nicoleta Georgescu, Georgiana Ionit , ă (Ministry of Public Finance), Ana Maria Bușoniu, Cornelia Băiculescu, Adelina Pafitis, Corina Costea (Ministry of European Funds), Flavia Anghel, and Ramona Naidin (Romanian Counter-Guarantee Fund) provided excellent inputs. The team is equally grateful for comments from the European Commission provided by Iulia Mirela Serban (DG REFORM), Enrico Pesaresi (DG REFORM), and Daniele Vidoni (DG COMP).  vii viii | Design for Impact The advisory services benefited from strategic guidance from the Romanian Competition Council’s President Bogdan M. Chiritoiu, Vice-President Elena Kleininger, and Competition Councillor Dragos Vasile. Executive Summary The provision of state aid is often justified by the goals of alleviating market failures or addressing equity concerns. State aid is generally prohibited, but the European Union legislation leaves room to declare such aid compatible with a competitive environment when it is designed to promote policy objectives driven by either efficiency or equity objectives.1 State aid can help correct market failures that may occur when the market is not able to provide the optimal level of a good or service. For example, information asymmetries may exist between lenders and borrowers, which might lead to a suboptimal supply of credit or externalities associated with knowledge spillovers that characterize research, development, and innovation. Given its selective nature, state aid may cause inefficiencies and distort market functioning. The provision of state aid influences the recipient’s cost structure (fixed and variable) while creating a wedge between the prices per- ceived by the beneficiary firm and the real underlying costs. It affects firms’ decisions on what to produce, how to produce it, and how much to charge for it. By providing an advantage to beneficiaries, state aid affects the way these firms interact with their competitors and can generate undesirable side effects on competition and economic efficiency in several ways. First, when resources are channeled to inefficient firms, state aid may prolong their life span and help them gain market share while more productive peers shrink and eventually cease to exist. Second, it might distort firms’ incentives to invest in productivity-enhancing activities, given the expectation that failing firms might receive state aid transfers in the future (soft budget constraints).2 Third, state aid might help a single firm (or group of firms) to consolidate their market power, or even to foreclose (existing and potential) competitors. Through all these channels, state aid can therefore distort the market selec- tion process, which in turn may hamper productivity growth,3 while generat- ing productive, allocative, and dynamic inefficiencies.4 Romania grants state aid for a variety of objectives, ranging from regional development, research, development, and innovation, to sectoral develop- ment, using several instruments such as grants, equity interventions, and  ix x | Design for Impact counter-guarantees. Romania spends around 0.54 percent of gross domestic product (GDP) on state aid—below the European Union (EU)–28 average of 0.76 percent of GDP, according to the 2019 State Aid Scoreboard, the European Commission’s benchmarking instrument for state aid. The aid dedicated to regional development is 2.5 times higher in Romania than the EU average, reflecting existing regional disparities. By contrast, the share of aid for research and development (R&D) and sectoral development is around half of the EU aver- age. Direct grants represent over 40 percent of total state aid expenditures in Romania, in line with the European Union’s state aid rules, which are applicable in Romania. National state aid legislation was developed with respect to those rules. The Romanian Competition Council ensures coordination on state aid matters with the European Commission (see box ES.1). BOX ES.1 State aid framework and implementation in Romania Government Emergency Ordinance (GEO) application of the European state aid rules. The no. 77/2014 lays out the national state aid framework RCC (a) elaborates Romania’s official position on the in Romania to ensure compliance with the EU state application of the European state aid rules; (b) facili- aid rules. A series of regulations on state aid monitor- tates the exchange of ideas and information with rep- ing, state aid inventory, intensity of state aid for regional resentatives of other member states and the EC; and development, and financial transparency, among oth- (c) coordinates the Romanian authorities’ position on ers, are also in force and are available (in Romanian) at the EC proposals regarding the extension and revi- http://www.ajutordestat.ro​/­legislatie-ajutor-de-stat​/­le sion of specific European regulations. The number of gislatie-nationala-in-vigoare/. RCC opinions on state aid is on an upward trend, The Romanian Competition Council (RCC) plays ­having increased almost threefold since 2015. In 2019, a key role in ensuring the application of the state aid 115 state aid measures were developed with the assis- legal framework and is the national contact author- tance of the RCC, of which (a) 20 measures are at the ity for state aid matters. The RCC represents Romania national level and are financed from national funds, before the European Commission (EC), and, in keeping (b) 10 measures are at the national level and are with procedures on state aid, submits the notifications, financed from European funds, and (c) 85 measures information, and evaluations on state aid, to the EC.a are from local authorities (table B.ES.1.1). The RCC has intensified its state aid activity in Figures B.ES.1.1 and B.ES.1.2 present the state recent years, and has also been playing a key role in aid  distribution by policy objective and by aid the context of the 2019 EC assessment on the instrument. TABLE B.ES.1.1  Recent RCC opinions on state aid 2015 2016 2017 2018 2019 Number of RCC opinions on state aid (including de minimis aid) 43 62 101 77 115 regarding compliance with the EU regulations Source: Romanian Competition Council 2019. Note: RCC = Romanian Competition Council. Continued Executive Summary | xi Box ES.1, continued FIGURE B.ES1.1 Share of total state aid expenditure by policy objective, Romania vs. EU-28 average EU28 RO 0 20 40 60 80 100 Percent Regional development Environmental protection including energy savings Research and developement including innovation Training Sectoral development SMEs including risk capital Culture Employment Social support for individual consumers Heritage conservation Rescue and restructuring Compensation for damages caused by natural disasters Promotion of exports and internationalization Closure aid Other Source: World Bank elaboration based on 2019 State Aid Scoreboard, which uses 2018 historical data. Note: The “Other” category includes measures that vary in terms of content and allocated funds across EU countries. The measures are aggregated in the category Other. RO = Romania; SMEs = small and medium enterprises; EU = European Union. FIGURE B.ESB1.2 Share of total state aid expenditure by aid instrument, Romania vs. EU-28 average EU28 RO 0 20 40 60 80 100 Percent Direct grant Tax advantage Other Loan/soft loan/repayable advances Risk capital instrument Guarantee Subsidized services Equity intervention Debt write-off Source: World Bank elaboration based on 2019 State Aid Scoreboard, which uses 2018 historical data. Note: The “Other” category includes measures that vary in terms of content and allocated funds across EU countries. The measures are aggregated in the category Other. RO = Romania; EU = European Union. a. Except for agricultural state aid, where the national contact authority is the Ministry of Agriculture and Rural Development. xii | Design for Impact THE GOAL OF STATE AID IMPACT EVALUATION The ex post evaluation of three state aid schemes in Romania assesses whether and to what extent the aid objectives have been fulfilled and mea- sures their spillover effects as well as the effects on competition outcomes. The evaluation focuses on three selected state aid schemes: (a) the Romanian Counter-Guarantee Fund (RCG) scheme, a de minimis aid scheme, implemented by the Romanian Counter-Guarantee Fund and designed to incentivize access to finance for micro, small, and medium enterprises (MSMEs); (b) the Ministry of Public Finance (MPF) scheme, implemented by the Ministry of Public Finance to support regional development and job creation; and (c) the Ministry of European Funds (MEF) scheme, implemented by the Ministry of European Funds to support the upgrade and modernization of research, development, and innovation (R&D&I). The selection of schemes for evaluation reflects a combi- nation of factors, notably the importance of the aid scheme in supporting key policy objectives, the scheme design and complexity, and the instruments used. Given the novelty of state aid impact evaluation in Romania, this evalua- tion seeks to inform the Romanian Competition Council and the line minis- tries on the effectiveness of state aid, on whether it distorts competition, and on the implications for state aid design and implementation. These aspects are fundamental to improving the efficiency of public spending and minimizing market distortions. The evaluation provides a first application of ex post meth- odologies to the evaluation of state aid, presenting the key tools required for sim- ilar evaluations going forward.5 THE APPROACH TO STATE AID IMPACT EVALUATION The ex post evaluation follows a four-step approach to determine the effec- tiveness of the state aid schemes at beneficiary and economywide levels, in line with the European Commission approach to state aid impact evaluation (European Commission 2014). First, it establishes the objectives of the state aid scheme. Second, it formulates a set of questions to be answered through an empirical analysis to determine if the objectives have been met. Third, it carries out an empirical analysis to assess the incentive’s effects on the beneficiaries and its indirect effects on nonbeneficiaries and competition. Fourth, it takes stock of overall state aid efficiency and effectiveness and proposes areas for improving future design and implementation (see box ES.2). An ex post evaluation allows policy makers to measure the direct and indi- rect effects of state aid. The direct effects of the state aid scheme concern the extent to which the scheme may have triggered an “incentive effect” at the benefi- ciary level (that is, whether, and to what extent, it caused the beneficiary to follow a different course of action). The indirect effects, on the other hand, refer to the scheme’s positive or negative spillover effects on nonbeneficiaries, and the scheme’s impact in terms of competitive market outcomes at the aggregate level. The latter reveals the extent to which eligible firms are able to expand their oper- ations at their competitors’ expense (Rotemberg 2019). The negative effects on competition can take a variety of forms, as follows: (a) crowd-out effects, which imply that channeling scarce resources to inefficient firms may hinder the growth or lead to the exit of more efficient firms from the market; (b) soft budget con- straints, which imply that firms’ incentives to invest in productivity-enhancing activities might be distorted because market operators anticipate that failing firms Executive Summary | xiii BOX ES.2 The European Commission’s approach to state aid impact evaluation The European Commission’s state aid impact balance direct and indirect effects to conclude evaluation follows a structured approach to identify whether the policy has fulfilled its objectives. key evaluation questions, the most appropriate  5. Proportionality and appropriateness consid- methodologies, key milestones of the impact erations: imply an assessment of whether the evaluation process, and the implementing body. It amount and chosen aid instrument were the least includes the following: distortive in light of the concrete circumstances of the case and whether the instrument was the  1. Description of the objectives and underlying most appropriate to achieve the objective. rationale of the aid scheme. This includes the  6. Assessment of the evaluation questions needs and problems the state aid scheme intends through result indicators. Result indicators to address, the target beneficiaries, and the allow the evaluation questions to be answered in expected impact of the aid, including external an objective and effective manner, showing the factors that can affect the scheme. Sector-spe- direct and indirect effects of the aid scheme. The cific state aid may contain additional or specific choice of concrete result indicators will depend requirements (for example, the broadband state on the characteristics of the aid scheme and its aid guidelines, which require, inter alia, a focus objectives. on whether the assumptions made by the Com-  7. Identification of the methodology that would mission for the compatibility assessment of the describe the effects of the aid scheme. The aid have been achieved in practice). evaluation plan should set out the main methods  2. Definition of the scope of the evaluation or tools and discuss why these methods are likely through evaluation questions. The evaluation to be appropriate. To estimate the effect of the aid questions need to be precise to be answered on aid beneficiaries, a counterfactual analysis will quantitatively and with the necessary support- be developed comparing the outcome of these ing evidence. The questions should address firms as a result of the aid with the hypothetical both the direct and indirect effects of the aid scenario in which they have not received it. scheme, as well as aid proportionality and  8. Description of the data required and avail- effectiveness. ability of the information. When planning  3. Direct impact of the aid on beneficiaries: an impact evaluation, it is important to review requires an assessment of the beneficiaries’ the existing data sources, decide whether they change of behavior (for example, the aid changed provide enough information for the evaluation, the beneficiaries’ actions, their market position, and guarantee access to data to evaluators. Data or financial stability) and the extent of the effects should be analyzed at the most granular level and created by the aid (for example, the magnitude of should have the same source for aid beneficiaries the effects created by the aid and heterogeneity and the control group. of the results among different beneficiaries). The  9. Development of a timeline and main mile- absence of any direct effect on the beneficiaries’ stones of the evaluation process. The timeline conduct (that is, incentive effect) is a strong indi- will vary according to the concrete aid scheme cator of policy failure. and should be agreed on with the Commission.  4. Indirect impact of the aid: requires an analysis 10. Designation of the body conducting the evalua- of the positive and negative externalities as a tion and/or its selection criteria. The evaluation result of the aid (for example, spillover effects plan should describe the entity conducting the on other firms or regions, crowding-out effects evaluation or its detailed selection criteria. The on investment, or aggregated effects on competi- selected entity should be at least functionally inde- tion and trade). The Commission will generally pendent from the authority granting the aid. Source: European Commission 2014, “Common Methodology for State Aid Evaluation,” May 28, page 12. xiv | Design for Impact might receive state aid transfers in the future; and (c) market power effects, which imply that state aid might reinforce or increase the market power of one or more firms by giving them an incentive to foreclose existent and potential competitors. Further, an ex post evaluation is paramount for informing future state aid design and implementation. If state aid is not effective in fulfilling its original objectives, this can be an indication of a waste of public resources. Even when a state aid scheme is adequately designed to meet its objective and is appropriately implemented, an ex post evaluation may reveal its ineffectiveness should its costs outweigh the benefits (Friederiszick, Röller, and Verouden 2006). Therefore, the evaluation results can provide useful information on how to improve state aid design and the implementation of similar schemes in the future. OBJECTIVES OF THE THREE SELECTED STATE AID SCHEMES The RCG de minimis scheme aimed to boost access to finance for micro, small, and medium enterprises. In Romania, the lack of own guarantees, in combination with a shallow financial sector, has been one of the main reasons behind the rejection of credit submissions by new MSMEs. By providing a count- er-guarantee for guarantees related to the cofinancing of investments and micro- finance, the RCG scheme’s goal was to ease access to financing for MSMEs, thus enabling their long-term development and ability to create jobs. The RCG scheme also aimed to reduce regional disparities by promoting entrepreneurship in the eight development regions of Romania. The total amount allocated for this scheme was €36.49 million, with a maximum exposure for each beneficiary MSME of €1.2 million and an eligible state aid amount of maximum €200,000 per firm over a three-year period. The disbursement period covered 2012–18. The MPF scheme sought to support regional development through invest- ment and job creation. Even if Romania has been spending four times more in regional development than the EU average, regional disparities have persisted in terms of county-level per capita GDP as well as employment opportunities.6 To tackle this issue, the MPF scheme included investment and job creation thresh- olds, which had to be met for firms to qualify for aid. In addition, the aid intensity and structure of eligible cost defined under the MPF scheme varied as a way to address the country’s regional disparities, particularly the gap between Bucuresti-Ilfov (the capital city and surrounding area) and the other regions. The total amount allocated for the scheme was the highest among the three ana- lyzed schemes, at €1 billion. The disbursement period covered 2009–18. The MEF scheme aimed to support R&D&I upgrade and modernization. Romania currently lags its EU peers in terms of R&D&I aid spending. To counter Romania’s lackluster performance on R&D&I, the MEF scheme focused on strengthening the links between universities and businesses in Romania, and on transferring R&D&I spending toward new products, services, and technologies. The total amount allocated for the scheme was €300 million. The disbursement period covered 2008–20. KEY FEATURES OF THE STATE AID EVALUATION Measuring the state aid effects requires building a counterfactual situation to assess what would have happened in the absence of support. This requires formulating a set of questions to be answered through an empirical analysis to Executive Summary | xv determine if the objectives have been met. The empirical analysis builds on var- ious sources of consistent micro-level information for both beneficiaries and nonbeneficiaries. In particular, three main sources of information were used to run the ex post evaluation of the three state aid schemes: (a) grantor datasets on the aid beneficiaries, (b) the Business Registry (2011–17), and (c) the Structural Business Survey (2008–17). The evaluation questions refer to both direct and indirect effects of the three state aid schemes. Questions related to direct effects aim at assessing the extent to which the state aid scheme had any effect on the course of action taken by the beneficiaries (the incentive effect), and the extent to which the effects of the aid were homogeneous or heterogeneous across regions and sectors, and over time. On the other hand, questions related to indirect effects are formulated to assess whether the scheme had spillover effects on the activity of other firms (nonbeneficiaries) and whether the scheme had any impact on competition out- comes (at market level). Several result indicators were assigned to each question so the effects could be quantified. To capture the direct effects on beneficiaries or spillover effects on nonbeneficiaries, the following result indicators were defined: (a) prob- ability of firms closing, (b) employment levels, (c) turnover levels, (d) investment level, (e) R&D expenditures, and (f ) a revenue-based measure of total factor pro- ductivity (TFPR) in aided firms. Regarding indirect effects on competition, the indicators capture structural parameters and dynamic aspects pertaining to mar- ket share reallocation and pricing power. These included (a) market share of aided firms (defined at Nomenclature of Economic Activities [NACE] 4-digit sector and county level or national level, depending on the scheme); (b) markup of aided firms; and (c) firm-level measure of allocative efficiency (defined at 4-digit NACE sector and county level or national level, depending on the scheme). RESULTS The RCG scheme met its objective of supporting entrepreneurship in Romania. The results show the scheme increased employment and turnover of beneficiary firms and reduced the probability of aided firms closing. The following describe specific results: • Employment and turnover have increased as a result of the scheme . Employment increased by 43 percent and turnover increased by 148 percent, while the probability of aided firms closing has decreased by 23 percent. However, direct effects in terms of productivity were negative, with TFPR shrinking by 1.6 percent among beneficiary firms. These estimates were statistically significant and validated by robustness checks. • The positive effects of the scheme were heterogeneous across regions and sectors. Beneficiaries in the North-West and Bucuresti regions, which together account for 46 percent of all beneficiaries, experienced large increases in terms of turn- over and employment. The wholesale and retail trade sector, which accounts for the largest proportion of beneficiaries (30.6 percent), experienced the smallest increase in turnover and employment and the smallest reduction in the probability of closing. On the other hand, the other services sector7 experi- enced the largest increases in turnover and employment, while beneficiaries in administrative activities8 and construction and others9 experienced the largest drop in the probability of closing. xvi | Design for Impact • Among nonbeneficiary firms, the effects of the scheme were negative in terms of employment and were heterogeneous across regions. The scheme appears to have led to job displacement effects, with the average number of workers employed by nonbeneficiary firms reduced by 2.7 percent. However, the drop in the number of workers (as well as the drop in firm creation) among nonbeneficiaries was significantly higher in regions with high employment rates (Bucuresti, South-East, and West regions), suggesting worker replacement is harder in regions with tighter labor markets. • The analysis did not show any evidence that the RCG scheme distorted competition. This is expected, given the small size of the eligibile state aid (that is, a maximum of €200,000 per firm over any three fiscal years, irrespective of the firm size or location). The market share of aided firms increased only when considering a very granular market definition; but even in that case, the increase was only 2.7 percent.10 The MPF scheme fulfilled its main objectives, with robust evidence of a positive direct effect on employment and, to some extent, on investment. ­ In particular, the ex post evaluation showed the following: • Employment increased among beneficiaries; the magnitude of the effect depends on the control group used as reference. Employment increased by 33.7 percent and 53.1 percent, respectively, for the rejected applicants control group and the matched control group. This amounts to between 420 and 662 employees per firm, which is well above the targeted employment growth as defined by the scheme of at least 50 additional employees per firm. • Evidence is mixed whether the MPF scheme increased investment among benefi- ciaries. Results showed that the MPF scheme increased investment in beneficia- ries by 66.4 percent, when using the rejected applicants control group, but this effect was not statistically significant at conventional levels (only at the 13.8 per- cent level). When using the matched control group, the estimated effect was larger, by 198 percent, and was statistically significant at conventional levels.11 • The effects on productivity for the beneficiary firms were not statistically significant. • Regarding indirect effects, the MPF scheme increased employment in nonbenefi- ciaries by 5.9 percent and productivity (measured by TFPR) by 1.2 percent. The estimated impact on employment corresponds to 12 employees per firm, on average. Although these indirect effects are smaller when compared with direct effects, they accrue to more firms because the scheme had 690 nonben- eficiaries compared with 17 beneficiaries. That is, for every job generated in a beneficiary, the MPF scheme generated another job in in a nonbeneficiary firm. • Finally, the analysis showed no evidence that the MPF scheme had distortive effects on competition at the market level. The analysis showed that effects on the market share and firm-level measure of allocative efficiency are small and not statistically significant. This means there was no evidence that the scheme triggered the expansion of beneficiaries’ market share at the expense of nonbeneficiaries, or that beneficiaries have increased their pricing power. ­ Similarly, there was no evidence that the aid has hindered the allocation of resources to the most productive businesses. Regarding the MEF scheme, the analysis found evidence that the scheme ­ ulfilled its objective of promoting R&D efforts. Results indicate the following: f • R&D expenses of beneficiaries increased as a result of the program; the magnitude of the effect depends on the control group used as a reference. Executive Summary | xvii The analysis showed that the size of input additionality (defined as the change in private R&D expenditure that can be attributed to public funding) is large, representing an increase of 172 percent in R&D expenses when using the control group of rejected applicants, and a 342 percent increase when using the matched control group.12 No significant evidence indicated that the scheme affected beneficiaries’ performance measured in terms of employment, turnover, or productivity (TFPR). • The analysis presented no evidence of spillover effects from the MEF scheme on nonbeneficiaries that belong to beneficiaries’ sectors. • Similarly, there was no evidence indicating that the MEF scheme had distortive effects on competition at the market level. Overall, the results from the MEF scheme evaluation should be taken with a grain of salt for several reasons. First, it was not possible to capture any effect the scheme might have had on universities and research institutions. Second, the assessment does not capture the impact of the scheme on innovation output measures, such as patents or publications. Third, given time span constraints in the SBS, the dataset used for the analysis, the analysis does not have enough time to detect an effect on employment, sales, and TFPR.13 LESSONS LEARNED AND POLICY IMPLICATIONS RCG scheme Despite the positive results achieved by the RCG scheme, it is important to review the scheme’s overall implementation to maximize its efficiency and effectiveness to inform future aid design. When taking the impact evaluation results together, it is not clear whether the positive direct effects of the RCG scheme are outweighed by potential displacement effects related to negative spillovers on nonbeneficiaries. Whereas widening the direct effects of the RCG scheme will depend, in principle, on factors associated with access to finance in general, the following factors could be reviewed: • State aid design, especially in terms of operating rules and key terms of the counter-guarantees offered. • State aid implementation, in terms of adopting a different sectoral focus. For instance, the wholesale and retail trade sector concentrates the largest proportion of beneficiary firms but has the smallest positive direct effects in terms of investments, turnover, and job creation, as well as large negative spillover effects in terms of job displacement. In this context, it might be worth considering the possibility of giving more emphasis to technology- and knowledge-intensive service activities, which are well known drivers of productivity growth and regional transformation. MPF scheme For the MPF scheme, the ex post evaluation results suggest that opportuni- ties exist to streamline the scheme design and implementation to improve its efficiency. With a total cost of €1 billion, the scheme fulfilled its objectives of generating jobs and promoting investment, although the impact lacks statis- tical significance for the latter. Furthermore, the lack of direct effect on pro- ductivity performance might suggest that the additional jobs created are not xviii | Design for Impact necessarily more productive. In light of the evaluation results, the following lessons are highlighted: • State aid design factors to consider might include (a) expanding the minimum targeted level of job creation; (b) restricting the eligible expenditures to fixed costs that are likely to translate into new technologies, new processes, and eventually new products; and (c) revising the aid intensity parameters (particularly relevant since the aid is provided on a rolling basis). • Regarding state aid implementation, it is important to monitor and collect more data on investment for years for post-implementation as they become available, and to have more detailed information about investment at the firm level—on tangible and intangible assets—to enable a more fine- tuned analysis. MEF scheme Regarding the MEF scheme, opportunities can be taken to streamline the design and implementation of the scheme. Despite the limitations of the MEF scheme evaluation presented in this evaluation, mostly resulting from data con- straints, it is still possible to identify some areas for improvement: • In state aid design, it would be important to focus the R&D support on applied research rather than to finance closer to the market activities.14 • Regarding state aid implementation, it is important to strengthen the capacity of the granting authority to document and file the information provided by beneficiaries and rejected applicants during the selection process. That information includes not only the scores obtained by firms during the evaluation process, but also basic information, such as the date when the application was rejected, and summarized information on the objective and scope of granted projects. CROSS-CUTTING ISSUES A key lesson from this ex post evaluation concerns the need to have access to sufficient, consistent, and accurate micro-level data. A critical step for a successful ex post impact evaluation is to build a master dataset containing information provided by the aid granting authority as well as data capturing the result indicators of both aid beneficiaries and nonbeneficiary firms. A thorough and robust state aid evaluation is crucial for policy purposes; it allows reviewers to confirm or reject whether the assumptions underlying the approval of the aid scheme are still valid and is useful to help improve the design of future aid schemes. Having sufficient, consistent, and accurate micro-level data at hand is necessary to ensure the development of a robust evaluation. In this context, the information to be used in future evaluations in Romania should be suitable to measure the outcomes of the aid schemes at the appropriate level of granularity, both across time and space. Table ES.1 summarizes the key impact evaluation results and main areas for improving state aid design and implementation, and table ES.2 highlights key areas for improving data collection and analysis. Executive Summary | xix TABLE ES.1  Summary of results and main areas for improving state aid design and implementation NUMBER OF FIRMS GRANTOR AND AID INCLUDED IN THE AREAS FOR IMPROVING STATE AID DESIGN AND SCHEME OBJECTIVE ANALYSIS KEY IMPACT EVALUATION RESULTS IMPLEMENTATION Romanian Sample of 784 Direct effects Review the design of the RCG scheme, Counter-Guarantee beneficiaries; no especially in terms of operating rules and key Increased the number of workers in Fund information available terms of the counter-guarantees offered. beneficiary firms by 43 percent Scheme “Increasing for rejected applicants Increased turnover of the average Review implementation of the RCG scheme, in access to finance for terms of adopting a different sectoral focus; beneficiary firm by 148 percent MSMEs to stimulate consider the possibility of giving more entrepreneurship Reduced the likelihood of benefi- emphasis to technology- and knowledge-inten- and job creation” ciary firms closing by 23 percent sive service activities, which are well-known Shrank TFPR by 1.6 percent among drivers of productivity growth and regional beneficiary firms transformation (for example, financing acquisition of intangible assets). Spillover effects • Reduced the average number of workers employed by nonbenefi- ciary firms by 2.7 percent • Showed drops in employment and in firm creation among nonbeneficiary firms in regions with high employment rates • Indirectly benefited some sectors, such as health and education • Possible negative spillovers on the performance of nonbenefi- ciaries in other sectors, such as the wholesale and retail trade sector and manufacturing Competition effects • Showed no robust evidence that the RCG scheme had distortive effects on competition • Increased beneficiary firms’ local market share by 2.7 percent (only when market is defined at 4-digit NACE and siruta territorial unit at level 2a) Ministry of Public Sample of 17 Direct effects • Consider expanding the minimum targeted Finance scheme beneficiaries and 54 • Increased employment in level of job creation. rejected applicants beneficiaries by between 33.7 • Consider restricting the eligible expenditures “Promoting regional (firms that appear in percent and 53.1 percent, to fixed costs that are likely to translate into development by all 10 years, 2008–17). depending on the control group new technologies, new processes, and stimulating used eventually new products. investment and job • Showed a statistically significant • Consider revising the aid intensity parame- creation” increase in investment at ters (particularly relevant since the aid is conventional levels, only when provided on a rolling basis). using the matched control group • Consider adding more years of investment (increase of 198 percent) data as they become available and having more detailed information about investment Spillover effects at the firm level—on tangible and intangible • Increased employment in assets—to enable a more fine-tuned analysis. nonbeneficiaries by 5.9 percent • Increased productivity in nonbeneficiaries (measured by TFPR) by 1.2 percent Competition effects • No evidence that the MPF scheme had distortive effects on competition at the market level Continued xx | Design for Impact TABLE ES.1, continued NUMBER OF FIRMS GRANTOR AND AID INCLUDED IN THE AREAS FOR IMPROVING STATE AID DESIGN AND SCHEME OBJECTIVE ANALYSIS KEY IMPACT EVALUATION RESULTS IMPLEMENTATION Ministry of Sample of 39 Direct effects • Consider focusing the R&D support on European Funds beneficiaries and 26 • Increased R&D expenses of applied research rather than on finance scheme rejected applicants beneficiary firms by between closer to the market activities. (firms that appear in 172 percent and 342 percent, • Strengthen the capacity of the granting “Supporting the all 10 years, 2008–17). depending on the control group authority to document and file the informa- upgrade and used tion provided by beneficiaries and rejected modernization of • No effect detected on turnover, applicants during the selections process; R&D&I” employment, and TFPR include not only the scores obtained by firms during the evaluation process, but also basic Spillover effects information, such as the date when the • No evidence of spillover effects application was rejected, and summarized from the MEF scheme on information on the objective and scope of nonbeneficiaries granted projects). Competition effects • No evidence that the MEF scheme had distortive effects on competition at the market level Note: MSME = micro, small, and medium enterprise; TFPR = revenue-based measure of total factor productivity; RCG = Romanian Counter-Guarantee Fund; NACE = Nomenclature of Economic Activities; MPF = Ministry of Public Finance; MEF = Ministry of European Funds; R&D&I = research, development, and innovation. a. Siruta is a classification system used in Romania by the National Institute of Statistics to register administrative-territorial units. Siruta level 2 corresponds to municipalities, towns, and communes. TABLE ES.2  Areas for improving data collection and analysis DATA SOURCE AREAS FOR IMPROVING DATA COLLECTION AND ANALYSIS Administrative data • Cover the longest time frame possible, covering before-and-after scheme disbursement and implementation. • Include unique firm identifiers to allow the complete mapping of beneficiaries and rejected firms. • Collect information on geographic location, not only for firms’ headquarters but also for firms’ production and operation plants. Data provided by granting authority • Collect and file information not only on beneficiaries, but also on rejected firms that applied to the scheme. • Include the most detailed level of information on how beneficiaries spent the funding. Source: World Bank elaboration. NOTES 1. The core provisions on state aid under the Treaty on the Functioning of the European Union are reflected in Articles 107–109; see https://ec.europa.eu/competition/state_aid​ /­legislation/provisions.html. 2. See Kornai, Maskin, and Roland (2003) for further discussion about soft budget constraint. 3. Economic literature identifies the market sorting effect—measured by entry and exit of firms—as a key driver of productivity growth. See, for instance, Syverson (2011); Foster, Haltiwanger, and Krizan (2001); Bartelsman, Haltiwanger, and Scarpetta (2013); and Arnold, Nicoletti, and Scarpetta (2011). 4. Productive inefficiency means that inputs are not combined in a cost-minimizing way to produce a given level of output. Allocative inefficiency arises when scarce productive resources are not combined in a way to benefit the whole society to the greatest extent possible. Dynamic inefficiency is associated with inadequate incentives to invest in productivity-enhancing activities. Executive Summary | xxi  5. Evaluation is part of the state aid modernization initiative in the European Union, whereby member states are authorized to grant more aid within the framework of the new General Block Exemption Regulation (GBER), without prior clearance by the European Commission, in exchange for more ex post controls for significant measures, transparency, and evaluation. Under the GBER, evaluation is required for schemes with large budgets (average annual budget exceeding €150 million) in some aid categories: regional aid, except regional operating aid, aid for small and medium enterprises and access to finance, aid for R&D&I, energy and environmental aid, except aid in the form of reductions in environmental taxes under Directive 2003/96/EC, and aid for broadband infrastructures. See https://ec.europa.eu/competition/state_aid​ /­modernisation/evaluation_faq_en.pdf.  6. Significant disparities are seen between the 42 counties in Romania (41 plus the municipality of Bucuresti). For example, GDP per capita in Bucuresti was more than double the national average of €10,472 in 2018, with counties such as Vaslui and Botosani having a GDP per capita of around 50 percent of the national average. The unemployment rate in Vaslui was 8.4 percent in 2018, much higher than the 1.3 percent unemployment rate in Bucuresti and the national average of 3.3 percent.  7. The “other services” sector includes the following 2-digit NACE sectors: accommodation; food and beverage service activities; creative, arts, and entertainment activities; libraries, archives, museums, and other cultural activities; sports activities and amusement and recreation activities; activities of membership organizations; and repair of computers and personal and household goods.  8. The administrative services sector includes the following 2-digit NACE sectors: publishing activities; motion picture, video, and television program production; activities auxiliary to financial services and insurance activities; legal and accounting activities; activities of head offices; management consultancy activities; architectural and engineering activities; technical testing and analysis; scientific research and development; advertising and market research; other professional, scientific and technical activities; veterinary activities; rental and leasing activities; employment activities; security and investigation activities; services to buildings and landscape activities; office administration; office support and other business support activities.  9. The construction and others sector includes the following 2-digit NACE sectors: electricity, gas, steam, and air conditioning supply; sewerage, waste collection, treatment, and disposal activities; materials recovery; construction of buildings; civil engineering; specialized construction activities; land transportation and transportation via pipelines; warehousing and support activities for transportation; postal and courier activities; telecommunications; computer programming, consultancy and related activities; and information service activities. 10. The effect was statistically significant only when firm market share is defined at the 4-digit NACE sector and at the most disaggregated local unit level (as indicated by the Computer System of the Register of Territorial Administrative Units—siruta is a numerical symbol specific to a certain locality). 11. In analysis of the MPF scheme, the rejected applicants control group includes rejected applicants that applied in the year before beneficiaries in each cohort first received funds. The matched control group is found using a matching method separately for each cohort of beneficiaries based on 18 variables. 12. The rejected applicants control group consists of the firms that applied to the program and were rejected. The matched control group is found using a matching method separately for each cohort of beneficiaries based on 12 variables. 13. Given that the analysis uses SBS data spanning 2008–17—and that the MEF beneficiary firms analyzed received funds for the first time in 2012, 2013, and 2014—the analysis has a maximum of 6 years postimplementation of the MEF scheme. This period might not be enough to detect effects on employment, sales, and TFPR. 14. A strong stylized fact emerging from the literature is that subsidies that are used to finance closer-to-market activities tend to be less effective and distort competition more when compared with subsidies that tackle basic and applied research; see, for instance, Czarnitzki, Hottenrott, and Thorwarth (2011). xxii | Design for Impact REFERENCES Arnold, J. M., G. Nicoletti, and S. Scarpetta. 2011. “Regulation, Resource Reallocation, and Productivity Growth.” European Investment Bank Papers 16 (1): 90–115. Bartelsman, E., J. Haltiwanger, and S. Scarpetta. 2013. “Cross-Country Differences in Productivity: The Role of Allocation and Selection.” American Economic Review 103 (1): 305–34. Czarnitzki, D., H. Hottenrott, and S. Thorwarth. 2011. “Industrial Research versus Development Investment: The Implications of Financial Constraints.” Cambridge Journal of Economics 35: 527–44. European Commission. 2014. “Common Methodology for State Aid Evaluation.” Commission competition​ Staff Working Document, SWD (2014) 179 final, Brussels. https://ec.europa.eu/­ /state_aid/modernisation/state_aid_evaluation_methodology_en.pdf. Foster, L., J. C. Haltiwanger, and C. J. Krizan. 2001. “Aggregate Productivity Growth: Lessons from Microeconomic Evidence.” In New Developments in Productivity Analysis, edited by E. Dean, M. Harper, and C. Hulten. Chicago: University of Chicago Press. Friederiszick, H. W., L-H. Röller, and V. Verouden. 2006. “European State Aid Control: An Economic Framework.” Handbook of Antitrust Economics, edited by P. Buccirossi, 625–69. Cambridge, MA: MIT Press. Kornai, J., E. Maskin, and G. Roland. 2003. “Understanding the Soft Budget Constraint.” Journal of Economic Literature 41: 1095–236. Rotemberg, M. 2019. “Equilibrium Effects of Firm Subsidies.” American Economic Review 109 (10): 3475–513. Syverson, C. 2011. “What Determines Productivity?” Journal of Economic Literature 49 (2): 326–65. Abbreviations BR Business Registry CUI Unique Registration Code (Tax Identification Number) EC European Commission EU European Union GBER General Block Exemption Regulation GDP gross domestic product GEO Government Emergency Ordinance IHS inverse hyperbolic sine MEF Ministry of European Funds MPF Ministry of Public Finance MSMEs micro, small, and medium enterprises NACE Nomenclature of Economic Activities NUTS Nomenclature of Territorial Statistics Units R&D research and development R&D&I research, development, and innovation RCC Romanian Competition Council RCG Romanian Counter-Guarantee Fund SBS Structural Business Survey SIRUTA Information System of the Register of Administrative-Territorial Units SMEs small and medium enterprises TFPR revenue-based measure of total factor productivity WTO World Trade Organization  xxiii 1 Introduction State aid is defined as an advantage in any form whatsoever conferred on a selec- tive basis to undertakings by public authorities.1 According to Article 107 (1) of the Treaty on the Functioning of the European Union, a measure is deemed to constitute state aid if the following four conditions are fulfilled.2 First, state aid is granted through state resources. The aid can be delivered in a variety of ways, such as through the allocation of grant subsidies, the provision of interest and tax relief, or the purchasing of goods and services on preferential terms. Second, it confers a selective economic advantage to certain undertakings or to the produc- tion of certain goods. In this regard, it is worth stressing that not all measures that favor economic operators fall under the notion of aid, only those that grant an advantage in a selective way to certain undertakings or categories of under- takings or to certain economic sectors. Third, it distorts or threatens to distort competition. Fourth, it has cross-border effects as it affects trade between the European Union’s (EU) member states. Provision of state aid is often justified by the goals of alleviating market ­failures or addressing equity concerns. State aid is generally prohibited, but the EU legislation leaves room to declare it compatible when it is designed to pro- mote policy objectives driven by either efficiency or equity o ­ bjectives (see box 1.1). As for the efficiency objective, state aid can help correct market failures, which are defined as those situations where the market is not able to provide the optimal level of a good or service. An example of market ­ failure that would justify the provision of aid is the presence of asymmetric informa- tion between lender and borrowers that might lead to a suboptimal s ­ upply of credit, particularly for micro, small, and medium enterprises (MSMEs). Another example is the externalities (caused by knowledge spillover) that characterize research, development, and innovation (R&D&I) activities, so state aid would be justified to avoid underfunding by filling the gap between social value attributed to these activities and returns that private agents extract from them.3 As for equity considerations, state aid can help promote wealth redistribution in situations where the outcome of the market-clearing process is not socially acceptable or is inequitable. Because state aid entails a transfer of public resources, it implies opportunity costs and has consequences for the efficiency of public spending. State aid is costly and should be understood as a state transfer of funds that flow from  1 2 | Design for Impact BOX 1.1 EU definition of state aid The general prohibition of state aid is stated in Article • Regional aid. Government transfers aimed at 107(1) of the Treaty on the Functioning of the supporting regions in which the income is ­ European Union: all state aid that meets the four cri- ­ unusually low, or serious underemployment can teria mentioned in the text is incompatible with the be witnessed, are excluded from the ban on state common market. However, Article 107(2) and (3) of aid (Article 107(3)(a) and 107(3)(c)). the same treaty highlight the potential positive effects • Horizontal aid. Horizontal aid is not limited to of state aid and provide possible derogation from this a specific economic sector but rather refers to general prohibition. Specifically, these paragraphs general state aid that may be awarded to any ­ provide well-defined exception categories of state aid company in any sector, if the necessary criteria are that are considered compatible with the common met. This aid encompasses the following state aid market: state aid having a social character (Article categories: state aid for small and medium enter- 107(2)(a)); state aid to alleviate the damages caused by prises, state aid for research and development, natural disasters (Article 107(2)(b)); state aid for the state aid for environmental protection, state aid for promotion of projects of European interest (Article rescuing and restructuring companies, state aid for 107(3)(b)); and state aid for the preservation of cul- employment, and state aid for further education. tural heritage (Article 107(3)(d)). In addition, the • Sectoral aid. Sectoral aid targets certain economic European Commission developed three different sectors, such as coal mining, fishing, agriculture, exception categories that nullify the general ban on audiovisual production, broadband, ­ broadcasting, state aid and thereby partially subsume the excep- electricity (stranded costs), postal services, tions of Article 107(1)—regional, horizontal, and sec- transportation, steel, and various other branches ­ toral aid. of industry that are considered sensitive. taxpayers to companies. Collecting taxpayers’ money entails both direct admin- istrative costs as well as indirect costs associated with distortive effects of taxa- tion (the shadow cost of taxation4). In addition to these costs, there is also opportunity cost, because granting of state aid always comes at the expense of other alternative public expenditures that could eventually generate higher returns. The opportunity cost of state aid adds to the challenge of improving public spending efficiency, especially in a context where public demand for and cost of key government expenditures—such as health care, education, and pen- sion systems—are all increasing. This is particularly true in developing countries, where fiscal resources are constrained and returns to public investment in areas such as education and health are typically higher than in developed countries.5 In this context, it is fundamental to assess the effectiveness of state aid against its original objectives. Unless state aid transfers are proved to contribute to legiti- mate and distributive policy objectives, they can represent a waste of resources. Moreover, even in situations in which state aid is designed to address an existing market failure and is implemented appropriately, it may still not be worth it, either because its impact is smaller than was anticipated or because costs are extremely high (Friederiszick, Röller, and Verouden 2006). Given its selective nature, state aid may cause inefficiencies and generate undesirable side effects on market functioning while distorting the Darwinian engine that competition is expected to trigger in an economy. The provision of Introduction | 3 state aid affects the recipient’s cost structure (fixed or variable) while creating a wedge between the prices perceived by the beneficiary firm and the real under- lying costs. It then affects how these firms make key production decisions—on how to produce, what to produce, and how much to charge for it—while nonre- cipient firms make their choices under distinct parameters. Therefore, by pro- viding an advantage to beneficiaries, state aid affects the way these firms interact with their competitors and can generate undesirable side effects on competition and economic efficiency in several ways. First, when resources are channeled to inefficient firms, they prolong the firms’ life span and help them to gain market share while more productive peers shrink and eventually cease to exist. Second, it might distort firms’ incentives to invest in productivity-enhancing activities, given the expectation that failing firms might receive state aid transfers in the future (soft budget constraints).6 Third, state aid measures might be used by a single firm (or group of firms) to cement or increase their market power, or even to foreclose (existent and potential) competitors. Through all these channels, state aid can therefore distort the Darwinian engine that competition is expected to trigger in the economy—the market selection process. This can hamper pro- ductivity growth 7 while generating productive, allocative, and dynamic inefficiencies.8 In this context, any ex post evaluation of state aid should factor in two key aspects. First is the extent to which the transfer of public resources has triggered any incentive effect at the beneficiary level. Second are the indirect effects, which include spillovers to nonbeneficiaries as well as impacts on competition outcomes at market level. To the extent that state aid programs might finance activities that firms would have undertaken regardless of the incentive, the key question to be answered at the beneficiary level is whether the aid has caused the beneficiary to take a different course of action, and how significant the impact of the aid has been—in other words, whether public financial support induces addi- tional activity compared with what the beneficiary would have carried out with- out such support (the so-called incentive effect). It is also important for the evaluation to measure the indirect effects at the market level. Given that effects of support programs on aggregate market outcomes depend on the extent to which eligible firms expand at their competitors’ expense (Rotemberg 2019), it is crucial to assess not only the spillover effects on nonbeneficiaries but also the effects on competition outcomes at the market level. Romania grants state aid for a variety of objectives, using several instruments such as grants, equity interventions, and counter-guarantees, among others. In terms of total state aid expenditure, Romania spends around 0.54 percent of GDP on state aid, below the EU-28 average of 0.76 percent of gross domestic product (GDP), according to the 2019 State Aid Scoreboard. As a comparison, the regional peers’ spending is above the EU-28 average, except for the Slovak Republic at 0.48 percent of GDP.9 Out of the four main objectives at the EU level, two of those—regional development and environmental protection and energy savings—represent over 85 percent of the total state aid spending in Romania. The share of regional development spending is around 2.5 times higher in Romania than the EU average, reflecting existing regional disparities. On the other hand, the share of aid for research and development, including innovation and sectoral development, is around half of the EU average (figure 1.1). In terms of aid instruments, direct grants (including interest rate subsidies) are most pop- ular at the EU level. In Romania, they represent over 40 percent of total state aid expenditure (figure 1.2). 4 | Design for Impact FIGURE 1.1 Share of total state aid expenditure by policy objective: Romania vs. EU countries EU28 UK SE FI SK SI PT PL AT NL MT HU LU LT LV CY IT HR FR ES EL IE EE DE DK CZ BG BE RO 0 20 40 60 80 100 Percent Regional development Environmental protection including energy savings Research and developement including innovation Training Sectoral development SMEs including risk capital Culture Employment Social support for individual consumers Heritage conservation Rescue and restructuring Compensation for damages caused by natural disasters Promotion of exports and internationalization Closure aid Other Source: World Bank elaboration based on 2019 State Aid Scoreboard, which uses 2018 historical data. Note: The “Other” category includes measures that vary in terms of content and allocated funds across EU countries. Because of a lack of homogeneity across EU countries, the measures are aggregated in a category with others. SMEs = small and medium enterprises. Introduction | 5 FIGURE 1.2 Share of total state aid expenditure by aid instrument: Romania vs. EU countries EU28 UK SE FI SK SI PT PL AT NL MT HU LU LT LV CY IT HR FR ES EL IE EE DE DK CZ BG BE RO 0 20 40 60 80 100 Percent Direct grant Tax advantage Other Loan/soft loan/repayable advances Risk capital instrument Guarantee Subsidized services Equity intervention Debt write-off Source: World Bank elaboration based on 2019 State Aid Scoreboard, which uses 2018 historical data. Note: The “Other” category includes measures that vary in terms of content and allocated funds across EU countries. Because of a lack of homogeneity across EU countries, the measures are aggregated in a category with others. For writeout of country names, see https://abbreviations.yourdictionary.com/articles/ abbreviations-european-union.html. 6 | Design for Impact Against this backdrop, this evaluation assesses the direct and indirect impacts of selected state aid schemes in Romania. It aims to go beyond the effect at the beneficiary level, in terms of scheme objectives, and is expanded to include second-order effects in terms of spillovers on nonbeneficiary firms as well as on market outcomes to assess whether schemes have been distortive from a competition standpoint. The evaluation follows a common framework for all the schemes based on an evaluation plan that includes a four-step approach. The four analytical steps combine qualitative and quantitative analysis (figure 1.3), which is in line with the European Commission (2014) approach to state aid impact evaluation: • In step 1, the analysis presents an overall description of objective of the state aid scheme, which comprises the needs and problems the scheme intends to address as well as key characteristics of the scheme, such as targeted benefi- ciaries and investments, method of allocating the aid, and aid intensity, among others. • Step 2 describes the datasets used to run the quantitative assessment to mea- sure the scheme effects; formulate the specific questions to be answered by the empirical exercise; and select the corresponding result indicators that are assessed accordingly. The selection of questions and outcome variables is driven by two measurement angles: (a) direct effects at the beneficiary level, to capture the incentive effect, and (b) indirect effects at the level of nonben- eficiaries, to capture spillover effects on other firms, as well as at the market level, to capture effects in terms of competition outcomes. • Step 3 specifically covers the quantitative assessment of the aid scheme. The empirical assessment follows two angles—the direct effect and the indirect effect—which are complementary. The direct effect (summarized as the incentive effect) requires understanding of the counterfactual (whether the aid has caused the beneficiary to take a different course of action) and cap- tures how significant the impact of the aid has been in order to answer whether the aid scheme achieved its policy objective. The indirect effect cap- tures both the degree to which the aid scheme has generated spillover effects at nonbeneficiaries, as well as the extent to which it has caused distortions in competition outcomes. If possible, the analysis uses the empirical results of FIGURE 1.3 The state aid evaluation plan: Analytical steps Step 1. Step 2. Definition of Step 3. Step 4. Description state evaluation questions Empirical Implications aid scheme and result indicators assessment for state aid design and • Program • Data description • Empirical analysis implementation objective • Definition of of direct and and key testable questions indirect effects characteristics • Selection of results indicators to be assessed Source: World Bank elaboration. Introduction | 7 direct and indirect effects to provide insights into the proportionality and appropriateness of the aid scheme. • Finally, step 4 discusses the implications of the empirical analysis (on direct and indirect effects) for the design and implementation of the aid scheme. To streamline the efficiency and efficacy of the scheme, the discussion focuses on potential adjustments to scheme design—such as redefining the scheme objective, changing target beneficiaries, and altering selection criteria—and scheme implementation. NOTES 1. The current analysis is restricted to the EU context, which is why the term state aid is used. The term subsidies is referred to by the World Trade Organization (WTO) in the context of the agreement on subsidies and countervailing measures. The specific rules that regulate each of them are different; see Rubini (2009) and Ehlermann and Goyette (2006) for detailed discussion and comparative analysis between WTO and EU rules. However, they carry a common (and key) characteristic, which is the degree of selectivity in the use of public resources. Therefore, the terms state aid and subsidies are often used interchange- ably. Both refer to government provision of financial advantages to undertakings by trans- ferring economic resources to them. This can be done in several ways, encompassing both spending instruments (such as grants, capital injections, loans, guarantees, and purchase of goods and services) as well as taxation, when governments provide special or differential taxation treatment. 2. Articles 107–109 are the main provisions dealing with state aid in the Treaty on the Functioning of the European Union and provide the legal framework to the European Commission to exercise its control in this area. 3. The degree of market failure severity tends to differ according to type of R&D&I activity. Fundamental and applied research often lead to stronger knowledge spillovers or external- ities, which make it more difficult for firms to extract the full returns from their activities. In this case, market failure associated with knowledge spillover is severe, and firms are likely to suffer from heavier financial constraints. On the other hand, research activities related to product development, which are closer to markets, are less likely to lead to knowledge spillovers, so market failure is less of a concern, and firms are subject to softer financial constraints. See Czarnitzki, Hottenrott, and Thorwarth (2011). 4. Shadow costs of taxation correspond to welfare losses generated by tax collection, since levying taxes usually leads economic agents to make different decisions than if these taxes were not levied. See Schwalbe (2006) for a theoretical discussion about this topic in the context of state aid. 5. For instance, see Psacharopoulos and Patrinos (2018) for recent evidence on returns to education. 6. See Kornai, Maskin, and Roland (2003) for further discussion about soft budget constraints. 7. Economic literature identifies the market-sorting effect—measured by entry and exit of firms—as a key driver of productivity growth. See, for instance, Syverson (2011); Foster, Haltiwanger, and Krizan (2001); Bartelsman, Haltiwanger, and Scarpetta (2013); and Arnold, Nicoletti, and Scarpetta (2011). 8. Productive inefficiency means that inputs are not combined in a cost-minimizing way to produce a given level of output. Allocative inefficiency arises when scarce productive resources are not combined in a way to benefit the whole society to the greatest extent possible. Dynamic inefficiency is associated with inadequate incentives to invest in pro- ductive enhancing activities. 9. Regional peers have higher state aid expenditures as a percentage of GDP when compared with Romania: for instance, 0.88 percent in Slovenia, 0.99 percent in Bulgaria, 1.03 percent in Poland, 1.54 percent in Croatia, 1.6 percent in the Czech Republic, and 1.83 percent in Hungary. 8 | Design for Impact REFERENCES Arnold, J. M., G. Nicoletti, and S. Scarpetta. 2011. “Regulation, Resource Reallocation, and Productivity Growth.” European Investment Bank Papers 16 (1): 90–115. Bartelsman, E., J. Haltiwanger, and S. Scarpetta. 2013. “Cross-Country Differences in Productivity: The Role of Allocation and Selection.” American Economic Review 103 (1): 305–34. Czarnitzki, D., H. Hottenrott, and S. Thorwarth. 2011. “Industrial Research versus Development Investment: The Implications of Financial Constraints.” Cambridge Journal of Economics 35: 527–44. Ehlermann, C-D., and M. Goyette. 2006. “The Interface between EU State Aid Control and the WTO Disciplines on Subsidies.” European State Aid Law Quarterly 5 (4): 24. doi: 10.21552​ /ESTAL/2006/4/88. European Commission. 2014. “Common Methodology for State Aid Evaluation.” Commission Staff Working Document, SWD (2014) 179 final, Brussels. https://ec.europa.eu​ ­/competition/state_aid/modernisation/state_aid_evaluation_methodology_en.pdf. Foster, L., J. C. Haltiwanger, and C. J. Krizan. 2001. “Aggregate Productivity Growth: Lessons from Microeconomic Evidence.” In New Developments in Productivity Analysis, edited by E. Dean, M. Harper, and C. Hulten. Chicago: University of Chicago Press. Friederiszick, H. W., L-H. Röller, and V. Verouden. 2006. “European State Aid Control: An Economic Framework.” Handbook of Antitrust Economics, edited by P. Buccirossi, 625–69. Cambridge, MA: MIT Press. Kornai, J., E. Maskin, and G. Roland. 2003. “Understanding the Soft Budget Constraint.” Journal of Economic Literature 41: 1095–236. Psacharopoulos, G., and H. A. Patrinos. 2018. “Returns to Investment in Education: A Decennial Review of the Global Literature.” Education Economics 26 (5): 445–58. Schwalbe, U. 2006. “Welfare Effects of Financing State Aid, New Developments in European State Aid Law.” Unpublished paper. Syverson, C. 2011. “What Determines Productivity?” Journal of Economic Literature 49 (2): 326–65. Rotemberg, M. 2019. “Equilibrium Effects of Firm Subsidies.” American Economic Review 109 (10): 3475–513. Rubini, L. 2009. The Definition of Subsidy and State Aid: WTO and EC Law in Comparative Perspective. Oxford: Oxford University Press. 2 Description of State Aid Schemes (Step 1) The selection of three state aid schemes for the ex post state aid impact evalua- tion aimed to provide relevance and broad coverage in terms of instruments and the aid scheme objective. The selected aid schemes involve some of the largest grantors in Romania, notably the Ministry of Public Finance, the Ministry of European Funds, and the Romanian Counter-Guarantee Fund. The next subsec- tions present a brief description encompassing key aspects of each scheme, including scheme objective, target beneficiaries and investment, as well as other design and implementation details (see table 2.1, at the end of the section, for a summary). DE MINIMIS SCHEME TO INCENTIVIZE ACCESS TO FINANCE FOR MSMEs (THE RCG SCHEME) Access to finance in Romania is limited, and micro, small, and medium enter- prises (MSMEs) are particularly affected. A shallow financial sector in Romania limits the availability of long-term finance. The banking sector is the leading financial intermediary, but bank loans to private enterprises amounted to 11.2 percent of gross domestic product (GDP) in 2019. As such, access to finance is limited for MSMEs, particularly in terms of loans and equity, according to the 2019 European Investment Fund’s SME Access to Finance Index (figures 2.1 and 2.2). The de minimis state aid scheme implemented by the Romanian Counter- Guarantee Fund (RCG scheme) was designed to incentivize access to finance for MSMEs while supporting entrepreneurship.1 The lack of own guarantees is one of the main problems faced by MSMEs and one of the main reasons for the rejec- tion of their credit applications. The RCG scheme was implemented to address this constraint. By providing counter guarantees, the scheme aims to increase access of newly established MSMEs to funding and to stimulate job creation through support to the recipient MSMEs. In addition, it hopes to address regional disparities in Romania in terms of MSME density at the regional level, with Bucuresti-Ilfov and the North-West regions ahead of other regions.2 The scheme is addressing this issue by promoting entrepreneurship in the eight  9 10 | Design for Impact TABLE 2.1  Description of the three state aid schemes under analysis DE MINIMIS SCHEME TO STATE AID SCHEME TO SUPPORT STATE AID SCHEME TO SUPPORT THE INCENTIVIZE ACCESS TO FINANCE REGIONAL DEVELOPMENT AND UPGRADE AND MODERNIZATION OF SCHEME ASPECT FOR MSMEs (RCG SCHEME) JOB CREATION (MPF SCHEME) R&D&I (MEF SCHEME) Scheme objective Promoting entrepreneurship. Promoting regional develop- Boosting research and scientific ment by stimulating investment knowledge transfers to the Increasing access to finance for and job creation. economy. MSMEs. Stimulating R&D-based activities Simulating job creation. for young firms. Stimulating new products, services, and technologies development. Aid instrument Counter guarantee Grant Grant Eligibility criteria and MSMEs up to 250 employees All enterprises, except those All enterprises, except those target population and up to 3 years old. considered to be in difficulty. considered to be in difficulty. (size, age, and sector) Some sectors were not eligible Some sectors were not eligible No sector restriction. (such as, agriculture, aquaculture, (such as agriculture and forestry, and fish farming; siderurgy; manufacture of textiles, shipbuilding; synthetic fibers; transportation, and storage). manufacturing of beverages; All regions were eligible. tobacco; gambling). Method of allocating First-come, first-served. Call for proposals. Call for expression. aid Selection procedure Cumulatively met all the Cumulatively met all the Cumulatively met all the conditions conditions / eligibility criteria. conditions / eligibility criteria. / eligibility criteria. All firms that applied and met All firms that applied and met Scoring based on three (or four, the criteria were accepted. the criteria were accepted. depending on the operation) pillars for a maximum of 30 points; firms that reached at least 21 points were selected. Aid intensityb and Aid intensity varied with type of Aid intensity varied on the Aid intensity varied with the size of eligible expenditures investment and working capital. amount invested and on the firm and eligible types of R&D development region and was activities/expendituresc between All type of investments covered capped at 40 percent for 25 percent and 70 percent. by the bank that granted the Bucuresti-Ilfov and at 50 guarantee, including investments Multiple R&D expenditure types percent for the other regions. made from European funds and were eligible.c For investment above €100 working capital necessary for the million, the intensity cap was commissioning of these 34 percent. investments, were eligible. Investments in tangible and Microfinance for microenterpris- intangible assets and costs es was also eligible. related to the creation of new jobs (such as gross salary costs and mandatory social insurance costs) were eligible. Total allocated budgetd €36.49 million €1 billion €300 million (US$46.88 million) (US$1.46 billion) (US$438.6 million) Application period July 1, 2012–December 31, 2013 2009–13 2008–13 Disbursement period 2012–18 2009–18 2008–20 and aid frequencye Yearly disbursements Yearly disbursements Yearly disbursements Source: World Bank elaboration based on official documents establishing each of the schemes. a. Development regions of Romania: C = Center; I = Bucuresti-Ilfov; NE = North-East; NV = North-West; SE = South-East; SM = South-Muntenia, SVO = South-West Oltenia; V = West. b. Aid intensity is defined as the ratio (expressed as a percentage) between the amount of aid and the eligible costs. c. The following types of R&D activities/expenditures are included: industrial research; experimental research; feasibility studies (for experimental research); acquiring and verifying industrial property rights (for industrial research); acquiring and verifying industrial property rights (for experimental research); consulting services for innovation; employing high-skilled specialists; and specific activities. d. Total allocated budget reflects the amounts indicated in the underlying decrees establishing the schemes. The conversion used the annual average exchange rate, from the National Bank of Romania, for the year when the scheme was published, namely, 2008 for the MPF and the MEF schemes, and 2012 for the RCG scheme. e. Disbursement period refers to the period in which the state aid is paid to the beneficiaries, while application period refers to the period in which the calls under the scheme can be organized and the financing contracts can be signed. Note: MSMEs = micro, small, and medium enterprises; RCG = Romanian Counter-Guarantee Fund; MEF = Ministry of European Funds; R&D&I = research, development, and innovation. Description of State Aid Schemes (Step 1) | 11 FIGURE 2.1 Micro, small, and medium enterprises’ access to loans index: 2017 and 2018 France Slovenia Austria Poland Italy Belgium Luxembourg Spain Malta Czech Republic Sweden Germany Portugal Lithuania Croatia United Kingdom Slovak Republic Estonia Finland Cyprus Romania Bulgaria Hungary Ireland Greece Latvia Denmark Netherlands 0 0.5 1 Loans, 2017 Loans, 2018 Source: The European Investment Fund’s SME Access to Finance Index, June 2019 Update (EIF 2019). Note: MSME access to finance index is a composite indicator with values ranging from 0 to 1. Values should not be interpreted as percentages or absolute values; for example, a value of 0.5 does not mean that a country’s performance is average in terms of access to finance. Instead, it implies that a country’s performance is average with regard to the best- and worst- performing countries. development regions of Romania while ensuring the stability and long-term development of the MSME sector. The target of the RCG scheme was young MSMEs. The selection process stipulated that eligible firms were represented by MSMEs with up to 250 employees, per EC Regulation no. 361/2003, and fewer than 3 years in opera- tion. In terms of eligible sectors, the selection reflects the incompatibility with the common market of state aid granted to certain sectors, as reflected in European Commission (EC) Regulation no. 800/2008. The excluded sectors 12 | Design for Impact FIGURE 2.2 Micro, small, and medium enterprises’ access to equity index: 2017 and 2018 Sweden Finland Denmark United Kingdom Estonia Germany Netherlands Malta Greece Belgium France Ireland Cyprus Slovenia Spain Latvia Lithuania Czech Republic Luxembourg Hungary Italy Bulgaria Croatia Austria Romania Poland Portugal Slovak Republic 0 0.5 1 Equity, 2017 Equity, 2018 Source: The European Investment Fund’s SME Access to Finance Index, June 2019 Update (EIF 2019). Note: MSME access to finance index is a composite indicator with values ranging from 0 to 1. Values should not be interpreted as percentages or absolute values; for example, a value of 0.5 does not mean that a country’s performance is average in terms of access to finance. Instead, it implies that a country’s performance is average with regard to the best- and worst- performing countries. were agriculture, aquaculture and fish farming; siderurgy; shipbuilding; syn- thetic fibers; mining of coal and lignite; manufacturing of beverages; tobacco; gambling; and the armament industry. In the case of road transportation, the value of the counter guarantee exposure was limited to half of the value avail- able for the other eligible sectors. The counter guarantee was given to all MSMEs that cumulatively met the conditions and criteria provided by the scheme and were within the budget of the scheme. Description of State Aid Schemes (Step 1) | 13 The RCG scheme provided counter guarantees with a maximum exposure of €1.2 million (US$1.54 million) for guarantees related to the cofinancing of invest- ments and microfinance.3 The maximum exposure was smaller for MSMEs in the road transportation sector and was set at €0.6 million (US$0.77 million). The maximum counter guarantee could not exceed 80 percent of the guarantee related to (a) cofinancing of the investments made from European funds, includ- ing the capital necessary for the commissioning of these investments; (b) financ- ing of investments, other than those already stipulated, including the circulating capital necessary for the commissioning of such investments; and (c) microfi- nance for microenterprises. For guarantees related to the financing of the work- ing capital, the counter guarantee could not exceed 60 percent of the guarantee provided by the banks. The RCG scheme set the effective counter guarantee pre- mium to zero to incentivize MSMEs’ access to finance. STATE AID SCHEME TO SUPPORT REGIONAL DEVELOPMENT AND JOB CREATION (THE MPF SCHEME) Economic potential and employment opportunities vary significantly across Romania’s regions. Figure 2.3 shows how county-level disparities in GDP per capita and employment opportunities play out in Romania. GDP per capita in Bucuresti-Ilfov was more than double the national average of €10,472 in 2018, with counties like Vaslui and Botosani having a GDP per capita of around percent of the national average.4 The unemployment rate in Vaslui was 50  ­ 8.4 percent in 2018 and 4.8 percent for the whole North-East region, much higher than the 1.2 percent unemployment rate in Bucuresti-Ilfov.5 These dis- parities have been driven by several factors, such as urbanization, productive specialization, demographics, and migration. Further, it is important to FIGURE 2.3 Gross domestic product per capita and unemployment rate across counties in Romania, 2018 Vaslui 8 Mehedinti Teleorman Dolj Buzau 7 Unemployment rate (2018) Olt Galati 6 Bacau Ialomita Suceava 5 Harghita Dambovita Neamt Salaj Gorj Calarasi 4 Vrancea Braila Maramures Covasna Tulcea Alba Valcea 3 Botosani Iasi Arges Bistrita-Nasaud Mures Constanta Giurgiu Prahova Satu Mare Brasov 2 Sibiu Arad Cluj Bucharest Bihor 1 Timis Ilfov 0 5,000 10,000 15,000 20,000 25,000 GDP per capita (current euro, 2018) Source: World Bank elaboration based on National Institute of Statistics data. 14 | Design for Impact maximize regional potential while ensuring equal opportunities for firms and individuals to achieve their potential (World Bank 2018). To support regional development, the Ministry of Public Finance designed a state aid scheme (MPF scheme) that included minimum investment and job cre- ation thresholds as eligibility criteria. Enterprises located in all the development regions were eligible. The scheme does not exclude firms from more developed regions from receiving aid.6 It addresses the development gaps between Bucuresti- Ilfov and the other regions and stipulates a lower aid intensity for Bucuresti-Ilfov. Enterprises considered to be in difficulty, as defined by EC Regulation no. 800/2008, are not eligible to receive aid. In terms of eligible sectors, the selection reflects the incompatibility with the common market of state aid granted to cer- tain sectors. There are several excluded sectors: agriculture and forestry; mining of coal and lignite; manufacture of textiles; electricity, gas, steam, and air condi- tioning supply; and transportation and storage. All the applications for the financ- ing agreement were accepted if they cumulatively met all the conditions and criteria defined by the scheme and were within the budget of the scheme. The aid intensity and structure of eligible cost defined under the MPF scheme varied as a way to address the country’s regional disparities. The MPF scheme includes investment and job creation thresholds that must be met for firms to qualify for aid. The target value of investment varies between €5 million and €30 million (US$7.3–43.8 million), and the number of jobs to be created varies between 50 and 300. The aid intensity defined under the MPF scheme varied, reflecting the development gap between Bucuresti-Ilfov and the other regions.7 This variation was in line with the European Union (EU) guidelines on regional aid, as follows: • For investments whose eligible costs were less than €50 million (US$73 mil- lion), the gross aid intensity may not exceed 50 percent of the costs related to the initial investment or the wage costs with the newly hired staff for 2 years. • For investments or jobs created in Bucharesti-Ilfov, the maximum aid inten- sity was 40 percent. The aid intensity also varied depending on the size of the investment, reflecting the decision of the Commission to reduce the maximum aid intensities for large investment projects to limit competition distortions.8 In terms of eligible expenditures, investments in tangible and intangible assets and costs related to the creation of new jobs were eligible. Eligible costs for investments in tangible assets included costs related to building industrial, tourism, and medical facilities; purchasing industrial machinery and equipment; and purchasing other industrial and medical furniture and equipment. Eligible costs for investments in intangible assets included technology transfer costs through the acquisition of intellectual property rights, such as patents, licenses, know-how, or nonpatented technical knowledge. Eligible costs for the creation of new jobs included the wage costs with the new staff hired as a result of the investment project for 2 years, costs with gross wages, and costs with compul- sory social insurance. STATE AID SCHEME TO SUPPORT THE UPGRADE AND MODERNIZATION OF THE R&D&I (THE MEF SCHEME) Romania lags the EU average in terms of R&D expenditures. Gross domestic expenditures on R&D were just 0.5 percent of total gross domestic Description of State Aid Schemes (Step 1) | 15 expenditures in Romania, the lowest in the EU in 2018. The R&D intensity in the business enterprise sector was also very low, at around 20 percent of the EU average. The weak performance is reflected by Romania’s share of R&D expenditures as a proportion of GDP (0.5 percent in 2018), which is well below the EU average (2.12 percent of GDP in 2018), placing the coun- try at the bottom of the ranking in the category of modest innovators figure 2.4). The country’s poor innovation performance is also reflected in (­ the low number of patent applications, which stands at 5.1 applications per million inhabitants, compared with the EU-28 average of 106.8 applications per million as of 2017.9 In this context, the Ministry of European Funds implemented a state aid scheme (MEF scheme) to support the upgrade and modernization of R&D&I and to address key causes of Romania’s modest R&D performance. The MEF state aid scheme focused on stimulating the transfer of R&D results to new prod- ucts, services, and technologies, and on boosting the implementation of research results and scientific knowledge. The focus of the state aid scheme reflects the need to strengthen the links between science and business in Romania, which are undeveloped, as reflected in the high level of technology imports not substi- tuted by home grown innovation. The underfinancing of research led to a signif- icant brain drain of researchers. Romania ranks last in the EU in terms of R&D personnel as a percentage of the active population. In terms of human resource development, the scheme aimed to attract specialists from abroad and to encour- age private initiative by stimulating young businesses to develop innovative, R&D-based activities.10 The MEF scheme encompasses three operations with different and comple- mentary objectives. The groups of targeted beneficiaries under each operation did not overlap. The MEF scheme was implemented under the Sectoral Operational Program for Increasing Competitiveness and covered the following operations: • Operation 2.1.1 (Research projects in partnership between universities / research and development institutions and enterprises) focused on the FIGURE 2.4 Research and development expenditure in EU countries, 2018 3.5 R&D expenditures (% of GDP, 2018) 3.0 2.5 2.12 2.0 1.5 1.0 0.50 0.5 0.0 RO CY T LV BG SK LT R IE EL LU PL ES PT IT EE U UK CZ EU SI NL FR FI BE K E AT SE 28 D M H D H Source: World Bank elaboration based on Eurostat data. Note: R&D = Research and Development; GDP = gross domestic product. 16 | Design for Impact critical actors in the R&D&I process—namely, universities, research insti- tutes, and industry—and on the synergies between them. It aimed at support- ing the research partnership between universities / research institutions and industry by financing the research carried out by universities / research insti- tutions for and on behalf of enterprises. • Operation 2.1.2 (High-level R&D projects involving foreign specialists) aimed at creating core teams with high-level scientific and technological competen- cies within a research institution / university or a host enterprise, by attract- ing specialists from abroad. • Operation 2.3.3 (Promoting innovation in enterprises) aimed at stimulating enterprise innovation by financing projects that developed new or substan- tially improved products, as a way to capitalize on research-and-development results and patented ideas. The selection criteria defined under the MEF scheme varied by operation. The process to select beneficiaries under the MEF scheme followed a two- pronged approach: • Firms that first cumulatively met all the conditions and eligibility criteria would qualify for the second stage. • Second, projects presented by firms were rated by points (out of 30), and firms that got at least 21 points had been granted aid, within the limit of the budget, while the others got no aid. In this second stage, the scoring of firms’ project applications was structured based on the following criteria: (a) project relevance, (b) quality and maturity of the project, and (c) sus- tainability and operating capacity. For operation 2.1.2, an extra category was added to reflect the specificity of the operation and focused on the capacity to financially support the research team and the ability to capital- ize on project results. However, in terms of individual criteria, the opera- tions were nonhomogeneous. An example, in this sense, is the category of relevancy. Operation 2.1.1 focused on obtaining research results applicable in the market and on the capacity of the enterprise to apply the research results. Operation 2.1.2 focused on the scientific novelty of the results and explicitly considered the number of newly created jobs as criteria. Operation 2.3.3 focused on the ability of the firm to differentiate its inno- vative products and processes from competing products and processes on the market. See appendix A for an extensive description of the selection criteria applied under each operation during the scoring stage. The structure of eligible costs to be financed also varied according to the targeted beneficiaries and objectives of each operation. The intensity of the state aid varied depending on the size of the firms, with large firms being associated with lower aid intensities. This reflects the decision of the European Commission to reduce the maximum aid intensities for large proj- ects to limit competition distortions. The state aid intensity also varied depending on the type of activity to be financed. The frequency and budget of these activities varied between operations, resulting in a different state aid intensity profile for different operations. Appendix B presents a detailed description of state aid intensities by eligible type of R&D activities to be financed and by firm size. Description of State Aid Schemes (Step 1) | 17 NOTES  1. The Romanian Counter-Guarantee Fund was created as a specialized financial institution and established as a joint stock company, having as main shareholders the Romanian state through the Ministry of Economy, Trade, and Business Environment (68 percent) and the Romanian Post-Privatization Fund (32 percent). The purpose of this fund is to counter guarantee all guarantees granted by guarantee funds—Romanian legal persons—for credits and other financing instruments obtained by small and medium enterprises from commer- cial banks and other sources. The fund’s aim is to improve the access of SMEs to financing, by taking over part of the risk undertaken by guarantee funds. For more details, see http:// www.contragarantare.ro/webincident/ro/fisiere/publicatii/Romanian%20Financial%20 Directory.pdf.  2. The density range was 5.6 MSMEs per 100 inhabitants for Bucuresti-Ilfov, 3.0 for the North-West region, 1.9 for the South-West Oltenia region, and just 1.6 for the North-East region. The values reflect available National Statistical Office data for 2018.  3. The conversion used the annual average exchange rate, from the National Bank of Romania, for 2012, representing the year the scheme was published.  4. Romania’s development regions are Bucuresti-Ilfov region, Center region, North-East region, North-West region, South-East region, South-Muntenia region, South-West Oltenia region, and West region. In terms of 2018 GDP per capita (expressed in current euros), the regions ranked in ascending order as follows: North-East region (6,426); South-West Oltenia region (7,797); South-Muntenia region (8,496); South-East region (8,838); North- West region (9,337); Center region (10,015); West region (10,926); and Bucuresti-Ilfov region (23,408).  5. In terms of the unemployment rate in 2018, the regions ranked in descending order as fol- lows: South-West Oltenia region (5.9 percent); North-East region (4.8 percent); South-East region (4.6 percent); South-Muntenia region (4.0 percent); Center region (2.9 percent); North-West region (2.3 percent); West region (1.8 percent); and Bucuresti-Ilfov region (1.2 percent).  6. The European Commission guidelines on national regional aid for 2007–13 differentiate between regional aid granted to regions with GDP per capita in purchasing power stan- dards below 75 percent of the EU average and above. In the case of Romania, only Bucuresti- Ilfov is above the threshold.  7. Aid intensity is defined as the ratio (expressed as a percentage) between the amount of aid and the eligible costs.  8. In accordance with the EC guidelines on national regional aid for 2007–13, https://eur-lex. europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52006XC0304(02)&from=EN. For large investment projects whose eligible costs exceed the leu equivalent of €50 million (US$73 million), the intensity was calculated by adjusting the regional ceiling, according to the value of the eligible costs, as follows: (a) for eligible-cost installments with a value between the equivalent in leu of €50 million (US$73 million) and the equivalent in leu of €100 million euros (US$146 million), the state aid was 50 percent of the regional ceiling; (b) for eligible-cost installments whose value exceeds the equivalent in leu of €100 million (US$146 million), the state aid was 34 percent of the regional ceiling.  9. Data are from the Eurostat database (indicator defined as patent applications to the EPO by priority year). See https://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=pat​ _ep_ntot&lang=en. 10. Data are from Eurostat database (indicator defined as R&D personnel and researchers). See https://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=rd_p_perslf&lang=en. REFERENCES EIF (European Investment Fund). 2019. “The EIF SME Access to Finance Index June 2019 Update.” Working Paper 2019/58, EIF Research & Market Analysis. World Bank. 2018. Rethinking Lagging Regions: Using Cohesion Policy to Deliver on the Potential of Europe’s Regions. Washington, DC: World Bank. 3 Definition of Evaluation Questions and Result Indicators (Step 2) DATASET Having consistent micro-level information for both beneficiary and nonbenefi- ciary firms is necessary to implement an ex post impact evaluation of a state aid scheme. Measuring the incentive effect from any scheme requires building a counterfactual situation to assess what would have happened in the absence of support. To do that, it is important to obtain identical information both for aid beneficiaries and for the control group, except for data on aid receipts (and appli- cations, when available). In practice, this involves combining several sources of information, including not only data on the aid receipts managed by the scheme grantors but also data from firm-level administrative sources (see box 3.1). The current analysis uses three major sources of information to run the ex post evaluation of the three state aid schemes. These include grantor datasets, the national Business Registry, and the Structural Business Survey (SBS). The first source is the grantor datasets. Table 3.1 summarizes the information provided by each aid-granting authority. For the Romanian Counter-Guarantee Fund (RCG) scheme, the grantor dataset provided by the granting authority con- tained the following information: • The list of beneficiaries identified by CUI (the unique tax identification or registry number) • The amount of aid, transfer per beneficiary per year • Information on territorial unit (corresponding to siruta1 level 2) where the beneficiary’s headquarters is located, as reflected in the trade registry • The beneficiary’s bank’s name • There was no information on rejected applicants.2 For the Ministry of Public Finance (MPF) scheme, the grantor dataset pro- vided information on the following: • The list of beneficiaries identified by CUI • The amount of aid transfer per beneficiary per year • Information on territorial unit (corresponding to siruta level 2) where the beneficiary made the investment financed by the aid • The list of rejected applicants identified by CUI  19 20 | Design for Impact BOX 3.1 Running an ex post evaluation of a state aid scheme: Key data features To run an ex post evaluation of a state aid scheme, it is as financial balance sheet data, industrial census, necessary to assemble a master dataset that combines and business surveys): at least two sets of information at the firm level: •  Information on outcome measures at the firm (a)  information provided by the aid-granting author- level (for example, number of employees, ity, and (b) data capturing the result indicators of both turnover, labor cost, capital stock, and material aid beneficiaries and firms that were not covered by cost). This information is crucial to capture the aid scheme. Ideally, this master dataset should the result indicators of the evaluation and to contain the following features: account for factors that can potentially influ- • Information provided by aid-granting authorities. ence the firm’s outcomes and decision to apply This should include the following: for the aid scheme. • List of beneficiaries. •  Time at which the firm’s outcomes are mea- • List of rejected applicants. sured (usually expressed in years). Time cov- • Period under which the aid was received (usually erage should be as long as possible and extend expressed in years). before and after the aid scheme implementa- • Amount of aid received (by year) by beneficiary. tion period. • Information about the aid-granting process, that •  Information about the geographic location of is, how beneficiary firms were selected, and how the firm’s activity (ideally, on where the firm other applicants were rejected. If the granting operates at the plant level, not the firm’s head- of the aid is made using a scoring mechanism, it quarters location). is crucial to obtain data on score results for both •  Information about the sector classification rejected applicants and beneficiaries. assigned to the firm, at the most granular level. • Information on how each beneficiary spent the aid For multiproduct firms, the sector classifi- received (for example, operation costs, labor costs, cation of the most relevant product should and fixed capital expenditures). suffice. • Information on the geographic location of the •  The master dataset can be complemented with investment made by the beneficiary, when the aid additional information on specific topics from is used to finance investment in new production other sources to enrich the analysis. Regardless facilities. of the number of sources used to assemble the • Information capturing firm-level result indicators master dataset, it is crucial to have a unique (usually using data of administrative origin, such firm identifier to allow for consistent matching between all sources. Source: World Bank elaboration. Finally, for the MEF scheme, the dataset provided by the granting authority contained the following information: • The list of beneficiaries with CUI • The amount of the yearly aid transfer • The territorial unit (corresponding to siruta level 2) where the beneficiary’s headquarters is located • The list of rejected applicants with respective CUI The second source is the Business Registry (BR). The BR contains balance sheets collected by the Romanian National Agency for Fiscal Administration. This dataset comprises a census of formal firms in Romania—with no size thresh- old restriction—and covers 712,445 unique firms in the 2011–17 period. Firms are Definition of Evaluation Questions and Result Indicators (Step 2) | 21 TABLE 3.1  Grantor dataset: Available information by aid scheme ORIGINAL NUMBER OF AID SCHEME/ BENEFICIARIES AND REJECTED GRANTOR AVAILABLE INFORMATION FIRMS REJECTED BENEFICIARIES APPLICANTS REJECTED LIST LIST AID APPLICANTS (IDENTIFIED (IDENTIFIED BY TRANSFER BENEFICIARIES (UNIQUE BY CUI) CUI) PER YEAR OTHER (UNIQUE CUIs) CUIs) Siruta territorial unit (level 2) where RCG scheme ✓ — ✓ beneficiary’s headquarters is located 863 — and beneficiary’s bank name Siruta territorial unit (level 2) where MPF scheme ✓ ✓ ✓ the beneficiary made the investment 37 129 financed by the aid Siruta territorial unit (level 2) where MEF scheme ✓ ✓ ✓ 133 149 beneficiary’s headquarters is located Source: World Bank elaboration. Note: Siruta corresponds to a classification system used by the National Institute of Statistics to register administrative-territorial units. Siruta territorial unit at level 2 corresponds to municipalities, towns, and communes. — = not available; CUI = Unique Registration Code; RCG = Romanian Counter-Guarantee Fund; MPF = Ministry of Public Finance; MEF = Ministry of European Funds. classified by 4-digit NACE level of aggregation. The information available in the BR on the firm’s territorial unit (corresponding to siruta level 13) refers to the firm’s headquarters. The full list of variables available in the version of the BR shared with the World Bank team is described in appendix C. The third source is the Structural Business Survey (SBS). The SBS is a survey conducted by the National Institute of Statistics of Romania. This survey is exhaustive for firms with at least 20 employees and provides a representative sample for firms with less than 20 employees. As such, the dataset has two strata: (a) one that is completely enumerated, covering a census of all enterprises with at least 20 employees; and (b) a random stratum, covering some smaller firms. The dataset encompasses a wider set of variables (compared with the BR) avail- able at the headquarters level for the 2008–17 period for 58,086 unique surveyed firms. The firms are classified at the 4-digit NACE level. The information on a firm’s territorial unit (corresponding to siruta level 2) available in the SBS refers to the firm’s headquarters. Appendix D describes the full list of variables avail- able in the version of the SBS dataset shared with the World Bank team. Drawing from these three sources, a master dataset was assembled for each scheme. For the RCG scheme, the master dataset resulted from the combination of the grantor dataset with the BR. In addition, the information on firms’ territo- rial unit (at siruta level 2) for all firms listed in the BR, as well as information on the beneficiary’s bank branch, were appended. For the MPF scheme, the master dataset combined the grantor information with the SBS dataset, while the BR was used as an intermediary step. A similar two-step procedure was employed for the MEF scheme. Table 3.2 summarizes the key information sources used to assemble the master files and shows the final number of firms included in each master dataset. EVALUATION QUESTIONS AND RESULT INDICATORS The ex post evaluation aims at answering questions pertaining to the direct and indirect effects of the state aid scheme: 22 | Design for Impact TABLE 3.2  Master dataset by aid scheme NUMBER OF BENEFICIARIES NUMBER OF REJECTED APPLICANTS IN THE ORIGINAL FINAL NUMBER IN THE ORIGINAL FINAL NUMBER AID SCHEME/ INFORMATION TIME GRANTOR IN THE MASTER GRANTOR IN THE MASTER GRANTOR SOURCES COVERAGE DATASET DATASET DATASET DATASET RCG grantor dataset + 2011–17 863 784 — — BR + beneficiary’s RCG scheme bank branch locality + siruta territorial unit code (level 2) MPF grantor dataset + 2008–17 37 34 129 95 BR + SBS + siruta MPF scheme territorial unit code (level 2) MEF grantor dataset + 2008–17 133 80 149 54 BR + SBS + siruta MEF scheme territorial unit code (level 2) Source: World Bank elaboration. Note: Time coverage corresponds to overlap between scheme implementation and disbursement period and the SBS/BR time coverage. Siruta corresponds to a classification system used by the National Institute of Statistics to register administrative-territorial units. Siruta territorial unit at level 2 corresponds to municipalities, towns, and communes. — = not available. The final number of firms included in the master dataset results from a cleaning (identification) process that uses information provided by each grantor as well as BR and SBS datasets. For confidentiality reasons, the details of this cleaning process cannot be disclosed. RCG = Romanian Counter-Guarantee Fund; MEF = Ministry of European Funds; MPF = Ministry of Public Finance. • Questions related to direct effects aim at assessing the extent to which the scheme had any effect on the course of action taken by the beneficiaries (the incentive effect). • Questions related to indirect effects are formulated to assess whether the scheme had spillover effects on the activity of other firms (nonbeneficiaries) and whether the scheme had any impact on competition outcomes (at the market level). Result indicators are assigned to each question so direct and indirect effects can be quantified. The information accessible in the firm-level datasets (containing information on both aid beneficiaries and nonbeneficiaries) is the ­ binding constraint to select the result indicators to be used in the analysis. Table 3.3 summarizes the evaluation questions as well as the result indicators that will be used to assess the impacts of each aid scheme. Two key aspects are noteworthy. First, the evaluation questions—capturing direct and indirect effects—vary across aid schemes. The schemes’ objectives are different, and result indicators adopted for the empirical analysis differ accordingly. Second, all result indicators are measured at the firm level. Some indicators correspond to (or had to be slightly adapted from) variables contained in the firm’s original administrative source (BR or SBS). For the RCG scheme, that is the case for the indicators employment level in aided firms (or in nonbeneficia- ries) and turnover in aided firms (or in nonbeneficiaries), as defined in the BR dataset.4 For the MPF scheme, this applies to employment level in aided firms (or in nonbeneficiaries), turnover in aided firms (or in nonbeneficiaries), and net investment in aided firms (or in nonbeneficiaries), as defined in the SBS. For the MEF scheme, whose master dataset draws from the SBS, the following indicators were already included in the SBS: employment level in aided firms ­ Definition of Evaluation Questions and Result Indicators (Step 2) | 23 TABLE 3.3  Evaluation questions and result indicators by aid scheme GRANTOR AND AID IMPACT SCHEME OBJECTIVE LEVEL EVALUATION QUESTIONS RESULT INDICATORS The incentive effect Probability of firm closing, among beneficiary firms at siruta territorial unit (level 2) Was there any increase in the level of employment as a result of the aid? Employment level in aided firms Was there any increase in turnover as a result Turnover in aided firms of the aid? Revenue-based measure of total factor productiv- Was there any effect on the probability of ity (TFPR) in aided firms Direct closing firms (among beneficiary firms) as a effects result of the aid? What was the impact of the scheme over time? Heterogeneity of the incentive effect RCG Have beneficiaries been affected differently Increasing access to by the aid (according to region or sector)? finance for SMEs to stimulate entrepre- Probability of firm opening, among nonbeneficia- neurship and job ry firms at siruta territorial unit (level 2) creation Spillover effects Probability of firm closing, among nonbeneficiary How have the nonbeneficiaries behaved as a firms at siruta territorial unit (level 2) result of the aid? Employment level in nonbeneficiary firms How heterogeneous were the spillover Turnover in nonbeneficiary firms Indirect effects at region or sector level? effects Revenue-based measure of total factor productiv- ity in nonbeneficiary firms Competition effects Market share of aided firms (defined at 4-digit NACE sector and county levels) Has the scheme led to the increase of market share / markup of beneficiaries? Markup of aided firms Has the scheme had any effect on allocative Firm level measure of allocative efficiency efficiency? (defined at 4-digit NACE sector and county levels) The incentive effect Employment level in aided firms Was there any increase in the level of Turnover in aided firms employment as a result of the aid? Net investment in aided firms Was there any increase in investment as a Direct Revenue-based measure of total factor productiv- result of the aid? effects ity in aided firms Was there any increase in turnover as a result of the aid? MPF Was there any effect on productivity as a result of the aid? Promoting regional development by Spillover effects Employment level in nonbeneficiary firms stimulating invest- How have the nonbeneficiaries behaved as a Turnover in nonbeneficiary firms ment and job result of the aid at the 4-digit NACE sector creation Net investment in nonbeneficiary firms level? Revenue-based measure of total factor Indirect productivity in nonbeneficiary firms ­ effects Competition effects Market share of aided firms (defined at 4-digit NACE sector and national level) Has the scheme led to the increase of market share or markup of beneficiaries? Markup of aided firms Has the scheme had any effect on allocative Firm-level measure of allocative efficiency efficiency? (defined at 4-digit NACE sector and national level) (continued) 24 | Design for Impact TABLE 3.3, continued GRANTOR AND AID IMPACT SCHEME OBJECTIVE LEVEL EVALUATION QUESTIONS RESULT INDICATORS MEF Direct The incentive effect Employment level in aided firms effects Supporting the Was there any increase in the level of R&D expenditures in aided firms upgrade and employment as a result of the aid? Turnover in aided firms modernization of the Was there any increase in R&D expenditures R&D&I Revenue-based measure of total factor productiv- as a result of the aid? ity in aided firms Was there any increase in turnover as a result of the aid? Was there any effect on productivity as a result of the aid? Indirect Spillover effects Employment level in nonbeneficiary firms effects How have the nonbeneficiaries behaved as a R&D expenditure in aided firms Turnover in result of the aid at 4-digit NACE sector level? nonbeneficiary firms Revenue-based measure of total factor ­ productivity in nonbeneficiary firms Competition effects Market share of aided firms (defined at 4-digit NACE sector and national level) Has the scheme led to an increase of market share or markup of beneficiaries? Markup of aided firms Has the scheme had any effect on allocative Firm-level measure of allocative efficiency efficiency? (defined at 4-digit NACE sector and national level) Source: World Bank elaboration. Note: RCG = Romanian Counter-Guarantee Fund; SMEs = small and medium enterprise; NACE = Nomenclature of Economic Activities; MPF = Ministry of Public Finance; MEF = Ministry of European Funds; R&D&I = research, development, and innovation. (or in nonbeneficiaries), turnover in aided firms (or in nonbeneficiaries), and R&D expenditures of aided firms (or of nonbeneficiaries). In addition, other indicators were computed as follows: • For the RCG scheme (drawing from the BR dataset): o Probability of firm closing, among beneficiary firms at siruta territorial unit (level 2), is equal to 1 if the firm stops reporting information from year t onward. o Probability of a firm opening, among nonbeneficiary firms at siruta terri- torial unit (level 2), is equal to 1 if the firm was created at year t and 0 otherwise. o Firm revenue-based measure of total factor productivity (TFPR) is esti- mated assuming a translog production function, while the estimation pro- cedure follows Ackerberg, Caves, and Frazer (2015).5 o Firm market share of aided firms is computed as the market share—­ measured in turnover6—of the firm at the 4-digit NACE sector and county levels.7 o Firm markup, a proxy of a firm’s pricing power, is computed following the methodology presented by De Loecker and Warzynski (2012), where markup is defined as the ratio of the production elasticity of the flexible input to the expenditure shares in that input.8 o Firm-level measure of allocative efficiency is computed as a cross product between two terms defined at the 4-digit NACE sector and county levels: (a) the percentage deviation of the firm’s market share from the average market share, and (b) the deviation of the firm’s productivity from the average firm-level productivity in the sector.9 Definition of Evaluation Questions and Result Indicators (Step 2) | 25 • For the MPF scheme (drawing from the SBS dataset): o Firm revenue-based measure of total factor productivity (TFPR) is com- puted following the same methodology as described above (for the RCG scheme). o Firm market share is computed as the market share measured in turnover of the firm at the 4-digit NACE sector nationwide.10 o Firm markup, a proxy of pricing power, is computed following the same methodology as described above. o Firm-level measure of allocative efficiency is computed following the same methodology as described above, except for the fact that market share and productivity average are measured at the 4-digit NACE sector at the national level.11 • For the MEF scheme (drawing from the SBS dataset): o Firm revenue-based measure of total factor productivity (TFPR) is com- puted as described earlier (for the RCG scheme). o Firm market share is computed as the market share—measured in turn- over—of the firm at the 4-digit NACE sector nationwide. o Firm markup, as a proxy of pricing power, is computed following the same methodology as described earlier (for the RCG scheme) o Firm-level measure of allocative efficiency is computed following the same methodology as described for the MPF scheme (measured at the 4-digit NACE sector at the national level). Further, the competition effects will be proxied by firm-level measures that capture competition dynamics at the market level: firm-level market share, firm- level markup, and firm-level measure of allocative efficiency. This approach will allow the assessment of whether aid schemes have affected not only structural parameters as the firm’s market share but also parameters capturing the firm’s pricing power and dynamic aspects related to the market share reallocation pro- cess. For the purpose of this evaluation, markets where firms operate are defined by the sector classification (at the most granular level available) assigned to the firm and by the geographic location where firm operates (if this information is available). In this regard, it is worth acknowledging that the market definition applied in the current analysis does not coincide with the classical definition of “the relevant market” in antitrust analysis. The latter typically considers the degree of product substitution and the geographic location of both producers and consumers, among others.12 Against this backdrop, the following are considered: • In the case of the RCG scheme, the competition measures are defined by the combination of geographic and product markets, based on data at the county (siruta level 1) and 4-digit NACE levels, respectively. • In the case of the MPF scheme, the competition measures are based on prod- uct markets (at 4-digit NACE) at the national level. • As for the MEF scheme, the competition measures are based on product mar- kets (at 4-digit NACE) at the national level. NOTES 1. Siruta is a classification system used in Romania by the National Institute of Statistics to regis- ter administrative-territorial units. The Territorial-Administrative Register is correlated with NUTS (Nomenclature of Territorial Statistics Units) level 5, and it is structured on three levels: 26 | Design for Impact (1) counties and Bucharest municipality; (2) municipalities, towns, communes; and (3) locali- ties, villages, and Bucharesti sectors. According to the latest 2019 data, there are a total of 16,510 administrative-territorial units, out of which 42 are at level 1, 3,181 are at level 2, and 13,287 are at level 3. For more details, see http://colectaredate.insse.ro/senin/classifications. htm?selectedClassification=SIRUTA_AN_2013&action=methodology&gobacktosearch=​ true&locale=ro.  2. The banks selected which firms got the guarantee, and RCG accepted all requests for a counter guarantee from the firms that had already received the guarantee.  3. As per the Siruta classification system, siruta territorial unit at level 1 corresponds to coun- ties and Bucharest municipality.  4. For the RCG scheme, the turnover and employment indicators are obtained from the BR, but a cleaning process was applied, so the result indicators are not identical to the original variables. For instance, if data on employment of a firm were missing in 2015 but there were data on it in 2014 and 2016, the average value was imputed to recover the information in 2015. Another example is if employment was missing and the firm reported zero turnover and zero expenses from workers, then zero value was imputed.  5. See appendix E for further details on how TFPR is estimated.  6. Measuring market share with turnover data provides a more accurate measure than using volume data, as both price increase and volume expansion could be a manifestation of mar- ket power. In addition, volume (quantum) data were not available in the BR or SBS datasets.  7. As per the siruta classification system, county corresponds to siruta territorial unit level 1.  8. This methodology requires estimating a production function in order to recover the output elasticity corresponding to the flexible input. The estimation assumes a translog produc- tion function (at 2-digit NACE Rev. 2 level of aggregation)—with material, labor, and capital as inputs—where material is selected as the flexible input. The estimation procedure fol- lows Ackerberg, Caves, and Frazer (2015). This method corrects for both collinearity and endogeneity of input choices. Appendix E presents the methodology applied to estimate productivity measures (TFPR) and to recover markup.  9. That is,  ∆s    ( it , j , c )  x ∆TFPRit , j , (3.1)  t , j ,c  s  where Sit,j,c is the firm i market share at time t measured at 4-digit NACE sector j and county c, and DTFPRit,j,c is the deviation of the firm i productivity (measured as TFPR) from the average firm-level productivity in the 4-digit NACE sector j and county c, at time t. This firm-level measure of allocative efficiency is interpreted as the contribution of a firm i to aggregate allocative efficiency. The measure is positive when the firm has above-average productivity and above-average market share, which means the firm contributes positively to aggregate allocative efficiency. On the other hand, if a firm has below-average productiv- ity but above-average market share, this measure is negative and implies some imperfec- tions in the economy that allow the less productive firm to keep a higher market share than would correspond to its lower than average productivity. 10. The reason why firm market share is computed nationwide for the MPF scheme analysis is because the geographic location information available for beneficiaries, drawing from the MPF grantors’ dataset, refers to location of the investment funded by the aid, while the geographic location information available for rejected applicants (and remaining nonben- eficiaries), drawing from the SBS, refers to headquarters location. With this inconsistency, it is not possible to run an unbiased comparison of firm market share between treated (ben- eficiaries) and control groups (rejected applicants or nonbeneficiaries). In addition, since most of the MPF beneficiaries are firms manufacturing tradable goods (to be discussed in the second section of chapter 4), it is reasonable to assume that their market is not local. 11. Likewise, the firm-level measure of allocative efficiency applied for the MPF scheme is computed nationwide. 12. The European Commission defines a relevant market and its product and geographic components as follows: (a) a relevant product market comprises all those products and/or ­ services that are regarded as interchangeable or substitutable by the consumer by reason of the products’ characteristics, their prices, and their intended use; and (b) a relevant Definition of Evaluation Questions and Result Indicators (Step 2) | 27 geographic market comprises the area in which the firms concerned are involved in supply of products or services and in which the conditions of competition are suffi- the ­ ciently homogeneous. See also https://eur-lex.europa.eu/legal-content/EN/TXT​ /?uri=LEGISSUM:l26073. REFERENCES Ackerberg, D. A., K. Caves, and G. Frazer. 2015. “Identification Properties of Recent Production Function Estimators.” Econometrica 83: 2411–51. De Loecker, J., and F. Warzynski. 2012. “Mark-Ups and Firm-Level Export Status.” American Economic Review 102 (6): 2437–71. 4 Empirical Assessment (Step 3) THE ROMANIAN COUNTER-GUARANTEE FUND SCHEME Firms included in the analysis The master dataset for the Romanian Counter-Guarantee Fund (RCG) scheme contained 784 beneficiary firms. All these firms are used in the analysis. They are concentrated in North-West, Bucuresti-Ilfov, and South-East and operate mainly in the wholesale and retail trade sector. As summarized in “Definition of Evaluation Questions and Result Indicators (Step 2)” chapter, the master dataset for the RCG scheme resulted from the combination of the RCG grantor dataset and the Business Registry (BR). There was no information for rejected appli- cants (see table 3.2). The 784 beneficiaries are concentrated in 275 siruta territo- rial units (at level 2), out of the 3,170 sirutas covered in the BR data. Across these 275 sirutas, beneficiary firms represented only a small fraction of all firms that were eligible according to the scheme design (see table 2.1, in “Description of State Aid Schemes (Step 1)” chapter, for eligibility criteria and target popula- tion). For instance, the average share of beneficiary firms (relative to all eligible firms in the same siruta) was 5.5 percent, with the median share being 2.7 per- cent. In terms of sector, the wholesale and retail trade sector concentrated most beneficiaries, with about 30 percent of beneficiaries (table 4.1).1 In terms of geographic location, beneficiaries were located across the eight regions of the country, with a larger presence in the North-West, Bucuresti-Ilfov, and South- East regions (table 4.2). In appendix F, table F.2 reports the years in which ben- eficiary firms received support for the first time and when the support ended, with 312 beneficiaries first receiving the RCG scheme aid in 2012. By 2014, almost all beneficiaries had started receiving the RCG support. A large share of benefi- ciaries kept receiving support over consecutive years, with 260 beneficiaries receiving their last support in 2017. Identification strategy To evaluate the direct, spillover, and competition impacts of the RCG scheme, a difference-in-differences framework is applied where firms located in treated and control siruta territorial units are compared.2 Treated sirutas are defined as  29 30 | Design for Impact TABLE 4.1  Romanian Counter Guarantee Fund scheme beneficiaries by sector classification SECTOR CLASSIFICATION NUMBER OF BENEFICIARIES % CUMULATIVE % Wholesale and retail trade 240 30.61 30.61 Other services 134 17.09 47.7 Professional, technical, admin, activities 129 16.45 64.16 Construction, transportation, IT, electricity 119 15.18 79.34 Manufacturing 119 15.18 94.52 Health, education 43 5.48 100.00 Total 784 100.00 Source: World Bank elaboration. Note: RCG = Romanian Counter-Guarantee Fund. TABLE 4.2  Romanian Counter Guarantee Fund scheme beneficiaries by region REGION NUMBER OF BENEFICIARIES % CUMULATIVE % North-West 230 29.34 29.34 Bucuresti-Ilfov 131 16.71 46.05 South-East 120 15.31 61.35 Center 76 9.69 71.05 South-West 64 8.16 79.21 North-East 63 8.04 87.24 West 52 6.63 93.88 South-Muntenia 48 6.12 100.00 Total 784 100.00 Source: World Bank elaboration. Note: RCG = Romanian Counter-Guarantee Fund. the 275 siruta territorial units (at level 2) that had RCG beneficiary firms at any point in time, while control sirutas consist of the remaining 2,895 siruta territo- rial units (at level 2) that had no beneficiaries of the RCG scheme.3 The baseline specification is given by equation 4.1, as follows: yit = αi + βt + γTreatmenti * Postt + εit , (4.1) where yit is the outcome variable for firm i and year t, αi is a firm fixed effect, and βt is a year fixed effect. The variable Treatmenti is equal to 1 if the siruta territorial unit (level 2) of firm i had beneficiaries of the RCG scheme and 0 otherwise. The term εit is an error term, clustered at the siruta location level. The coefficient γ represents the treatment effect of the RCG scheme on outcome yit. The sample of firms used to evaluate these impacts varies. To measure the direct impact of the RCG scheme, as well as the competition impacts at the mar- ket level, the treatment and control group were defined as follows: • The beneficiary firms in treated siruta territorial units (at level 2) are restricted to those firms that were created prior to the first disbursement of the scheme. That is, the treatment group comprises only the beneficiary firms that started operations between 2009 and 2011. Empirical Assessment (Step 3) | 31 • The control group includes firms that operate in sirutas territorial units (at level 2) that had no scheme beneficiaries. The control group consists of firms operating in siruta territorial units with no beneficiaries of the RCG scheme, further refined based on three criteria to guarantee compa- rability with beneficiary firms. First, the control group is restricted to firms created between 2009 and 2011. Second, the control group is limited to firms operating in the same 217 4-digit Nomenclature of Economic Activities (NACE) sectors as the beneficiary firms in treated siruta terri- torial units (at level 2).4 Third, only firms in the control group that remained opened by 2012 are considered, as all beneficiary firms were in operation when the scheme began. To study the indirect impact of the RCG scheme on nonbeneficiaries (that is, the spillover effects), treated and control siruta territorial units are restricted to firms that, given their sector and age, were not eligible for the scheme. This exer- cise allows comparison of the evolution of firms that did not receive state aid (and were in close proximity to state aid beneficiary firms) with similar firms that did not participate in the scheme and were located in siruta territorial units (level 2) where the RCG scheme was not rolled out. The spillover effect of the RCG scheme is then measured as the difference in outcomes between these two groups in the years before and after the scheme. To test if the control groups are valid comparison groups for the analysis of the direct, competition, and indirect effects of the program, prescheme outcome measures for firms in the treatment and control groups were contrasted. Results discussed in appendix G show that the control groups are indeed valid for the exercise. Results Direct effects of the RCG scheme on beneficiaries: The incentive effect Was there any increase in the level of employment as a result of the aid? What about turnover and productivity? Was there any effect on the probability of firms closing (among beneficiary firms) as a result of the aid? There is evidence of pos- itive direct effects from the RCG scheme on entrepreneurship. The scheme increased employment and turnover and reduced the likelihood of firm exit among beneficiary firms. The direct impact of the RCG scheme on beneficiaries is summarized in table 4.3.5 The estimates in column 1 indicate that the scheme increased the number of workers in beneficiary firms by 43 percent. There is also evidence of a large and significant effect of the scheme on the turnover of beneficiary firms: as a result of the scheme, turnover of the average beneficiary firm increased by 148 percent (column 2). The scheme also reduced the likeli- hood of firm exit by 23 percent among beneficiary firms (column 3). On the other hand, direct effects in terms of productivity were negative: Revenue-based total factor productivity (TFPR) shrank by 1.6 percent among beneficiary firms (column 4), which, as reported below, may be driven by firms in specific sectors (other services) and certain regions (Bucuresti-Ilfov, North-West, and South-Muntenia). A battery of robustness checks was conducted to validate these findings, with all results shown to be robust. First, the evaluation assessed whether the effect of the RCG scheme on employment and turnover is driven by the composition of 32 | Design for Impact TABLE 4.3  Direct effects of the Romanian Counter Guarantee Fund scheme on beneficiary firms (1) (2) (3) (4) NUMBER OF WORKERS PROBABILITY OF CHARACTERISTIC (IHS) TURNOVER (IHS) CLOSING TFPR (LN) Average treatment effect 0.430*** 1.478*** −0.229*** −0.016*** p-value 0.000 0.000 0.000 0.006 Number of observations 96,102 128,933 169,708 51,251 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme on beneficiary firms for the outcomes displayed in each column. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit level 2. TFPR = revenue-based total factor productivity; RCG = Romanian Counter-Guarantee Fund. ***, **, and * indicate 1, 5, and 10 percent significance levels. firms that remained opened after the scheme started. Given that the panel data- set is balanced, information on firms that close is set to missing after the year they closed. In an alternative specification, information on employment and turnover is set to zero in the years after a firm is closed. The results, pre- sented in table H.2 of appendix H, show even larger and statistically significant effects on employment and turnover of beneficiary firms as a result of the scheme. Second, the evaluation tested whether the direct effects of the scheme are driven by the sample of beneficiaries that opened prior to 2012. To do this, the direct effects of the scheme were evaluated using all the beneficiary firms in the sample, including those that opened in the years after the scheme started. The results, shown in table H.3 of appendix H, validate that the impact estimates are not driven by the sample of beneficiaries used in the baseline specification. Third, the evaluation assessed whether results are robust to one specific sub sample of firms: the 91 beneficiaries that, although they opened between 2010 and 2011, received their first RCG scheme aid in 2013 or later. The prescheme trends for the years 2011 and 2012 and impact estimates after 2013 were exam- ined using this subsample of beneficiaries. In this exercise, the control group is further limited to firms in control siruta territorial units (level 2) that are eligible for the aid scheme and opened between 2010 and 2011. The parallel pretrend tests on employment, turnover, and TFPR (table H.4 of appendix H) show no significant differences in trends for the years 2011 and 2012 across these out- comes for the subsample of beneficiaries and their control group. The impact estimates of the RCG scheme, presented in table H.5 of appendix H, show posi- tive and statistically significant effects of the scheme for this subsample of beneficiaries. What is the impact of the scheme over time? Have beneficiaries been affected differently by the aid (according to region or sector)? The impact of the scheme varied over time, with beneficiaries experiencing greater changes in the first 3 years of the scheme. This trend was slightly reversed thereafter. Figure 4.1 plots the yearly differences in outcome variables across beneficiaries and their control firms relative to 2011 (the prescheme year). The figure shows that across all result indicators, beneficiaries experienced greater changes in the first 3 years of the scheme and reverted slightly thereafter. This period coincides with the pat- tern of disbursements to beneficiaries over time, which was concentrated in the first years. There is also evidence that beneficiaries have been affected differently by the aid scheme—first, across regions, with the North-West and Bucuresti-Ilfov Empirical Assessment (Step 3) | 33 FIGURE 4.1 Effects of the Romanian Counter-Guarantee Fund scheme on beneficiaries over time, 2012–17 a. Number of workers (IHS) b. Sales (IHS) 0.8 2.5 0.6 2.0 Impact estimate Impact estimate 1.5 0.4 1.0 0.2 0.5 0 0 2012 2013 2014 2015 2016 2017 2012 2013 2014 2015 2016 2017 c. Probability of closing d. TFPR (ln) 0.01 –0.10 0 Impact estimate Impact estimate –0.15 –0.01 –0.20 –0.02 –0.25 –0.03 –0.30 –0.04 2012 2013 2014 2015 2016 2017 2012 2013 2014 2015 2016 2017 Source: World Bank elaboration. Note: Each point represents the difference in the average outcome between beneficiaries and their control firms, relative to the difference in 201 1 (1 year before the first disbursement of aid of the RCG scheme). Vertical lines are 95 percent confidence intervals. Impact estimates are reported in the y-axis, with a value of 1 corresponding to a 100 percent increase. Estimates include firm and year fixed effects, with standard errors clustered at the siruta territorial unit level 2. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Table H.1 in appendix H presents the regression coefficients used in this figure. IHS = inverse hyperbolic sine; TFPR = revenue-based measure of total factor productivity. regions experiencing larger effects in terms of employment, turnover, probabil- ity of firms closing, and productivity. Figure 4.2 plots the contribution of benefi- ciaries in each region to the average impact estimate reported in table 4.3. The estimates of the scheme effects on the number of workers and turnover are smaller in the South-West region than in other regions. This region, however, only concentrates 6 percent of all beneficiaries. In contrast, estimates of the scheme effects on all outcomes are large in regions such as North-West and Bucuresti-Ilfov, which together concentrate 46 percent of all beneficiaries. Second, across sectors, the wholesale and retail trade sector experienced the smallest increase in turnover and employment and the smallest reduction in the probability of firm closing. Figure 4.3 plots the contribution of beneficiaries in the different economic sectors to the average impact estimate reported in table 4.3. Compared with other sectors, the effect of the scheme on the number of workers of beneficiary firms in health and education tend to be smaller; but this result is not statistically different from zero. Beneficiary firms in all other sectors experienced a large increase in employment of 40 percent or more. Beneficiaries in sectors with greater increases in employment also experienced a larger boost in their turnover. In this regard, it is worth highlighting that the 34 | Design for Impact FIGURE 4.2 Direct effects of the Romanian Counter-Guarantee Fund scheme on beneficiary firms across regions a. Number of workers (IHS) b. Sales (IHS) Bucuresti-Ilfov Bucuresti-Ilfov Center Center North-East North-East North-West North-West South-East South-East South-Muntenia South-Muntenia South-West South-West West West –0.5 0 0.5 1.0 –2 0 2 4 Impact estimate Impact estimate c. Probability of closing d. TFPR (ln) Bucuresti-Ilfov Bucuresti-Ilfov Center Center North-East North-East North-West North-West South-East South-East South-Muntenia South-Muntenia South-West South-West West West –0.4 –0.3 –0.2 –0.1 0 –0.10 –0.05 0 0.05 Impact estimate Impact estimate Source: World Bank elaboration. Note: Each point represents the difference in the average outcome between beneficiaries and their control firms in a given region. Horizontal lines are 95 percent confidence intervals. Impact estimates are reported in the x-axis, with a value of 1 corresponding to a 100 percent increase. Estimates include firm and year fixed effects, with standard errors clustered at the siruta territorial unit level 2. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. sector that concentrates the largest proportion of beneficiaries—wholesale and retail trade—experienced the smallest increase in both employment and turnover and the second smallest reduction in the probability of closing.6 While beneficia- ries in health and education had a negative drop in productivity performance, the impact estimate for beneficiaries in other sectors is not statistically different from zero. Indirect (spillover) effects of the RCG scheme on nonbeneficiaries How have the nonbeneficiaries behaved as a result of the aid? The spillover effects of the scheme on the activity of nonbeneficiary firms were estimated to be neg- ative in terms of employment. To evaluate the existence of spillover effects from the RCG scheme, the treatment group (consisting of noneligible firms located in the same siruta territorial units—level 2—of beneficiary firms) and the control Empirical Assessment (Step 3) | 35 FIGURE 4.3 Direct effects of the Romanian Counter-Guarantee Fund scheme on beneficiary firms across sectors a. Number of workers (IHS) b. Sales (IHS) Trade Trade Other services Other services Construction and others Construction and others Health, education Health, education Manufacturing Manufacturing Admin activities Admin activities 0 0.5 1.0 0 2 4 6 Impact estimate Impact estimate c. Probability of closing d. TFPR (ln) Trade Trade Other services Other services Construction and others Construction and others Health, education Health, education Manufacturing Manufacturing Admin activities Admin activities –0.35 –0.30 –0.25 –0.20 –0.15 –0.10 –0.15 –0.10 –0.05 0 0.50 Impact estimate Impact estimate Source: World Bank elaboration. Note: Each point represents the difference in the average outcome between beneficiaries and their control firms in a given sector. Horizontal lines are 95 percent confidence intervals. Impact estimates are reported in the x-axis, with a value of 1 corresponding to a 100 percent increase. Estimates include firm and year fixed effects, with standard errors clustered at the siruta territorial unit level 2. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. group (consisting of noneligible firms located in siruta territorial units—level 2— where the RCG was not rolled out) are contrasted, and the difference-in-differ- ences results of impact estimates thus reflect the indirect impact that the RCG scheme had on firms that did not receive the scheme but were in close proximity to beneficiary firms. The results given in table 4.4, column 1, indicate that the likelihood of firm creation among nonbeneficiary firms was negative but not sta- tistically significant. The results from column 2 suggest that the scheme reduced the average number of workers employed by nonbeneficiary firms by 2.7 percent. The impact on employment can be taken as potential evidence of displacement effects. This negative spillover effect in terms of employment is driven by firms that remained opened and shrank in size. As a robustness check, a similar regression was estimated using an alternative measure of employment that is set to zero in the years before a firm opens and after a firm closes. The results given, pre- sented in table H.7 of appendix H, show that the economic magnitude of the drop in employment among nonbeneficiary firms remains, but it is no longer significant because of larger standard errors. 36 | Design for Impact TABLE 4.4  Spillover effects of the Romanian Counter Guarantee Fund scheme on nonbeneficiary firms (1) (2) (3) (4) (5) PROBABILITY OF NUMBER OF PROBABILITY OF OPENING WORKERS (IHS) TURNOVER (IHS) CLOSING TFPR (LN) Average treatment effect −0.009 −0.027*** −0.025 0.005 0.000 p-value 0.174 0.008 0.551 0.753 0.662 Number of observations 7,772,500 2,449,694 3,482,749 5,440,750 1,435,461 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme on firms not eligible for the scheme and located in siruta territorial units (level 2) with beneficiaries. The sample period examined in column 1 spans 2008–17, as the probability of opening can be traced back in time using the year of opening of firms. The sample period analyzed in columns 2–6 spans 2011–17, which corresponds to the years where data are available. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). RCG = Romanian Counter-Guarantee Fund; TFPR = revenue-based measure of total factor productivity. Significance level: * = 10 percent, ** = 5 percent, *** = 1 percent. Results also showed that the negative spillover effects of the scheme in terms of employment among nonbeneficiaries are concentrated in the years after 2014. The impact of the scheme over time on the probability of opening, turnover, and number of workers of nonbeneficiary firms is illustrated in figure 4.4. Except for the probability of a firm opening, all figures plot the yearly differences in out- come variables across nonbeneficiaries in treated and control siruta territorial units (at level 2) relative to 2011 (the pre-scheme year). Since data on the proba- bility of opening are available from 2008 onward, the yearly differences in prob- ability of opening are relative to this year. The figure for employment illustrates that the spillover effects of the scheme among nonbeneficiaries are concentrated in the years after 2014, when most beneficiaries of the RCG scheme already began receiving support. The figures also show a negative change in the proba- bility of opening and turnover after 2014, although the effects are not statistically significant. How heterogeneous were the spillover effects at region or sector level? Spillover effects to nonbeneficiaries varied across regions, with a larger decrease in the rate of firm creation in Bucuresti-Ilfov. The drop in the number of workers was greater in the Bucuresti-Ilfov, South-East, and West regions. A further examina- tion of the heterogeneity of the impact estimates for nonbeneficiary firms across regions is illustrated in figure 4.5. The figure plots the contribution of each region to the average impact estimate of the probability of opening and the number of workers (conditional and unconditional on firms being opened) reported in table 4.4 and table H.7 (in appendix H). As figure 4.5 shows, the rate of firm cre- ation among firms not eligible for the scheme slows down in the postscheme years in siruta territorial units (level 2) with beneficiaries. While this result is not significant, the drop in the rate of firm creation in Bucuresti-Ilfov is much larger and statistically significant than in other regions, with an estimated impact of about 3 percentage points. Similarly, the drop in the number of workers (condi- tional on firms that are opened) for the average nonbeneficiary firm is greater in the Bucuresti-Ilfov, South-East, and West regions. The impact estimate on employment for all nonbeneficiaries is uneven; it is negative and statistically sig- nificant for nonbeneficiary firms in the Bucuresti-Ilfov region. Results also showed that the drop of firm creation and drop in number of workers among nonbeneficiaries were significantly higher in regions with high Empirical Assessment (Step 3) | 37 FIGURE 4.4 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiaries over time a. Probability of opening b. Sales (IHS) 0.02 0.1 0 0 Impact estimate Impact estimate –0.1 –0.02 –0.2 –0.04 –0.3 –0.06 2009 2010 2011 2012 2013 2014 2015 2016 2017 2012 2013 2014 2015 2016 2017 c. Employment for firms opened (IHS) d. Employment for all firms (IHS) 0.05 0.1 Impact estimate Impact estimate 0 0 –0.1 –0.05 –0.2 –0.10 2012 2013 2014 2015 2016 2017 2012 2013 2014 2015 2016 2017 Source: World Bank elaboration. Note: Each point represents the difference in the average outcome between nonbeneficiaries in treated siruta territorial units (level 2) and their control firms, relative to the difference 1 year prior to the first year in the graph. Vertical lines are 95 percent confidence intervals. Impact estimates are reported in the y-axis, with a value of 1 corresponding to a 100 percent increase. Estimates include firm and year fixed effects, with standard errors clustered at the siruta territorial unit level 2. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Table H.7 in appendix H presents the regression coefficients used in this figure. employment rates. To further investigate factors that may be driving the regional differences in the probability of a firm opening and employment as a result of the scheme, an exercise was conducted to relate the regional employment rate to the impact estimates across regions. While regional employment in Romania is on average 58.6 percent over the sample period, the employment rate in the Bucuresti-Ilfov, North-West, and West regions is consistently higher, with rates of 75.3 percent, 66.7 percent, and 60.8 percent, respectively. Concretely, the dif- ference-in-differences impact estimate was interacted with a dummy variable that equals 1 for regions with employment rates above the 75th percentile to test if regions with higher employment rates are those where displacement effects were noticeable, because nonbeneficiary firms may find it more difficult to replace workers. The results of this exercise are presented in table H.8 (in appen- dix H). The findings provide evidence consistent with this hypothesis, as they show that the indirect impact of the scheme on the probability of a firm opening 38 | Design for Impact FIGURE 4.5 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiaries across regions a. Probability of opening b. Employment for firms opened (IHS) Bucuresti-Ilfov Bucuresti-Ilfov Center Center North-East North-East North-West North-West South-East South-East South-Muntenia South-Muntenia South-West South-West West West –0.04 –0.03 –0.02 –0.01 0 0.01 –0.10 –0.05 0 0.05 Impact estimate Impact estimate c. Employment for all firms (IHS) Bucuresti-Ilfov Center North-East North-West South-East South-Muntenia South-West West –0.15 –0.10 –0.05 0 0.05 Impact estimate Source: World Bank elaboration. Note: Each point represents the difference in the average outcome between nonbeneficiaries in treated siruta territorial units (level 2) and their control firms in a given region. Horizontal lines are 95 percent confidence intervals. Impact estimates are reported in the y-axis, with a value of 1 corresponding to a 100 percent increase. Estimates include firm and year fixed effects, with standard errors clustered at the siruta territorial unit level 2. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. and employment (conditional and unconditional to firms being opened) was larger in regions with high employment rates. Spillover effects also varied substantially across sectors. Some sectors— such as health and education—benefited indirectly from the RCG scheme, while for other sectors—such as wholesale and retail trade, and manufactur- ing—the scheme may have had negative spillovers on the performance of nonbeneficiaries. In this case, the variation of spillover effects happened in magnitude and sign. Figure 4.6 illustrates the heterogeneity of the impact estimates for nonbeneficiary firms across sectors. The figure shows great heterogeneity of spillovers across sectors. Although the likelihood of opening a new firm in treated siruta (level 2) territorial units (that is, siruta territorial units that had beneficiary firms at any point in time) is lower in sectors such as trade and construction, this likelihood is higher (albeit not significant) in sectors such as other services. Similarly, though the number of workers among firms not eligible for the scheme drops in treated siruta territorial units (level 2) in sectors such as wholesale and retail trade and Empirical Assessment (Step 3) | 39 FIGURE 4.6 Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiaries across sectors a. Probability of opening b. Employment for firms opened (IHS) Trade Trade Other services Other services Construction and others Construction and others Health, education Health, education Manufacturing Manufacturing Admin activities Admin activities –0.04 –0.02 0 0.02 0.04 –0.10 –0.05 0 0.05 0.10 Impact estimate Impact estimate c. Employment for all firms (IHS) Trade Other services Construction and others Health, education Manufacturing Admin activities –0.2 –0.1 0 0.1 0.2 Impact estimate Source: World Bank elaboration. Note: Each point represents the difference in the average outcome between nonbeneficiaries in treated siruta territorial units (level 2) and their control firms in a given sector. Horizontal lines are 95 percent confidence intervals. Impact estimates are reported in the x-axis, with a value of 1 corresponding to a 100 percent increase. Estimates include firm and year fixed effects, with standard errors clustered at the siruta territorial unit level 2. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. manufacturing, nonbeneficiaries in the health and education sector seem to have increased their average number of workers in treated siruta territorial units (level 2). The same patterns, though uneven, are observed for the alter- native employment measure that includes firms that are not opened. These results may imply that across sectors, the degree of complementarity between beneficiary and nonbeneficiary firms varies. Competition effects Has the scheme led to the increase of market share and markup of beneficiaries? What about effects on allocative efficiency? Across the main market-level out- comes, there is no robust evidence that the RCG scheme had distortive effects on competition. Table 4.5 shows the effects of the RCG scheme on firm-level measures that capture competition dynamics at the market level: firm-level market share, firm-level markup, and firm-level measure of allocative effi- ciency. In this case, the market is defined by the interaction between product and geographic dimensions (4-digit NACE sector and county levels, respec- tively). All estimates of average treatment effects are found to be statistically insignificant. As a robustness check, a more granular measure of market share, defined at the 4-digit NACE sector and siruta territorial unit level 2, was used to examine changes in competition induced by the RCG scheme. The results presented in table H.9 (in appendix H) show evidence that 40 | Design for Impact TABLE 4.5  Competition effects of the Romanian Counter Guarantee Fund scheme (1) (2) (3) FIRM MARKUP FIRM MARKET FIRM-LEVEL MEASURE OF (LN) SHARE ALLOCATIVE EFFICIENCY Average treatment effect 0.028 0.001 0.005 p-value 0.698 0.515 0.638 Number of observations 49,374 99,283 49,492 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG Scheme on market-level outcomes. All regressions include firm and year fixed effects. Market share and allocative efficiency are measured at the county and 4-digit NACE code levels. Standard errors are clustered at the siruta territorial unit level 2. ***, **, and * indicate 1, 5, and 10 percent significance levels. RCG = Romanian Counter-Guarantee Fund. beneficiary firms increased their local market share by 2.7 percent, and the results are statistically significant. THE MINISTRY OF PUBLIC FINANCE SCHEME Firms included in the analysis The master dataset for the Ministry of Public Finance (MPF) scheme contained 34 beneficiaries and 95 rejected applicants. As summarized in the “Definition of Evaluation Questions and Result Indicators (Step 2)” chapter, the master dataset for the MPF scheme combined the MPF grantor information with the Structural Business Survey (SBS) dataset, while the BR was used as an intermediary step. This master dataset has 34 beneficiaries and 95 rejected applicants (table 3.2). Table 4.6 displays the beneficiaries and rejected applicants found in the SBS data by cohort. For beneficiaries, the cohort is based on the year when the beneficiary first received funds from the MPF scheme (some firms received funds in multi- ple years). For rejected applicants, the same table shows the year when the appli- cation was received. Most beneficiaries in the SBS data received funds for the first time in 2012, 2013, 2014, or 2015. Most rejected applicants applied for the scheme in either 2011, 2012, 2013, or 2014. Out of these 34 beneficiaries and 95 rejected applicants, the impact evalua- tion focused on the firms that appear in all 10 years (2008 thorough 2017), result- ing in a final sample of 17 beneficiaries and 54 rejected applicants. Table 4.6’s balanced panel column also shows that not all firms appear in the data each year.7 The impact evaluation analysis focuses on firms that appear in all years from 2008 to 2017. Using this balanced panel allows better comparison of beneficia- ries against the control group in the pre-scheme period and therefore to assess whether the control group looks valid (described further in the “Identification Strategy” subsection). It also helps to reduce noise stemming from a changing sample composition. The impact evaluation analysis thus uses the 17 beneficia- ries in the 2012, 2013, 2014, and 2015 cohorts, along with the 54 applicants that were rejected in years 2011, 2012, 2013, and 2014. The final sample of beneficiaries operate mainly in the manufacturing of motor vehicles, rubber and plastics, electrical equipment, and machinery and Empirical Assessment (Step 3) | 41 TABLE 4.6  Ministry of Public Finance scheme beneficiaries and rejected applicants by cohort BENEFICIARIES IN COHORT REJECTED APPLICANTS YEAR SBS BALANCED PANEL SBS BALANCED PANEL 2009 1 1 3 0 2010 1 0 1 0 2011 1 0 17 9 2012 5 4 32 19 2013 6 4 18 12 2014 9 5 24 14 2015 9 4 0 0 2016 1 1 0 0 2017 0 0 0 0 2018 1 0 0 0 Total 34 19 95 54 Source: World Bank elaboration. Note: For beneficiaries, the cohort refers to the year when the firm first received funds from the MPF scheme. For rejected applicants, the table shows the year when the application was received. The balanced panel includes only firms that appear in all 10 years of the dataset (2008 through 2017). MPF = Ministry of Public Finance; SBS = Structural Business Survey. TABLE 4.7  Ministry of Public Finance scheme beneficiaries and nonbeneficiaries by sector SHARE OF 17 SHARE OF 17 SHARE OF BENEFICIARIES IN BENEFICIARIES NOT IN NONBENEFICIARIES 2-DIGIT NACE SECTOR THE ANALYSIS (%) THE ANALYSIS (%) (%) Manuf. motor vehicles 23.5 17.7 0.5 Manuf. rubber and plastic 11.8 17.7 1.2 Manuf. electrical equipment 11.8 11.8 0.6 Manuf. machinery and equipment 11.8 11.8 1.1 Wholesale trade 11.8 0.0 18.7 Manuf. food 5.9 5.9 4.1 Manuf. other non-metallic mineral prod. 5.9 0.0 1.2 Manuf. fabricated metal products 5.9 5.9 2.6 Wholesale and retail trade of motor vehicles 5.9 0.0 3.6 Human health activities 5.9 5.9 1.1 Manuf. textiles 0.0 5.9 0.7 Manuf. basic metals 5.9 0.5 Manuf. other transport equipment 5.9 0.3 Sports activities 5.9 0.3 Other sectors 63.4 Total 100 100 100 Source: World Bank elaboration. Note: For beneficiaries, data are for the year before they first received funds from the MPF scheme. Some of the beneficiaries not in the analysis do not have SBS data in the year before they started receiving funds from the MPF scheme. In these cases, either the year when they first received funds or the year after are used, depending on availability. Data for other nonbeneficiaries are for 2012. NACE = Nomenclature of Economic Activities. 42 | Design for Impact TABLE 4.8  Ministry of Public Finance scheme beneficiaries and nonbeneficiaries by region SHARE OF THE 17 SHARE OF THE 17 BENEFICIARIES IN THE BENEFICIARIES NOT IN REGION ANALYSIS (%) THE ANALYSIS (%) SHARE OF NONBENEFICIARIES (%) North-West 29.4 11.8 13.6 Center 23.5 23.5 12.1 North-East 0.0 0.0 9.7 South-East 5.9 5.9 10.7 South-Muntenia 17.7 35.3 10.6 Bucuresti-Ilfov 5.9 5.9 28.2 South-West Oltenia 5.9 5.9 6.2 West 11.8 11.8 8.9 Total 100 100 100 Source: World Bank elaboration. Note: For beneficiaries, data are for the year before they first received funds from the MPF scheme. Some of the beneficiaries not in the analysis do not have SBS data in the year before they started receiving funds from the MPF scheme. In these cases, either the year when they first received funds or the year after are used, depending on availability. Data for other nonbeneficiaries are for 2012. equipment sectors. Table 4.7 shows that the 17 beneficiaries in the analysis come from 10 different 2-digit NACE sectors. The largest group (corresponding to four firms) is in the motor vehicle manufacturing sector. Table 4.8 also shows that the 17 beneficiaries not used in the analysis come from similar sectors as the 17  beneficiaries used in the analysis. The distribution of nonbeneficiaries across sectors looks quite different from the distribution of beneficiaries. The sectors that contain the largest share of beneficiaries (manufacturing of motor vehicles, rubber and plastics, electrical equipment, and machinery and equip- ment) account for less than 4 percent of nonbeneficiaries. In fact, 63.4 percent of nonbeneficiaries are in sectors that did not have any beneficiaries. Most of the firms in this final sample of beneficiaries are in the North-West or Center regions. Table 4.8 shows that the 17 beneficiaries come from all regions of Romania, except for the North-East region. Most beneficiaries are in the North-West or Center regions. Of the remaining 17 beneficiaries not included in the analysis, the regional distribution looks similar, except that fewer firms come from the North-West region and more come from the South- Muntenia region. The regional distribution of beneficiaries is different from that of nonbeneficiaries. Most notably, while none of the beneficiaries are in the North-East region, this region contains 9.7 percent of nonbeneficiaries. The largest share of nonbeneficiaries (28.2 percent) is in Bucuresti-Ilfov, but this region contains only 5.9 percent of beneficiaries. Identification strategy A difference-in-difference analysis is applied to assess the direct effects of the MPF scheme on beneficiaries. The difference-in-differences approach com- pares changes in the outcomes of beneficiary and control firms over time. The effect for each cohort is estimated in a pooled regression of the form in ­equation 4.2: yict = α i + β ct + ∑ s = 2012 γ s MPFist + ε ict , 2015 (4.2) Empirical Assessment (Step 3) | 43 where yict is an outcome of firm i in cohort c in year t, such as employment or turnover; ai is a firm fixed effect; bct is a cohort-year fixed effect; and MPFist is a set of treatment dummy variables, one for each cohort. Each of these treatment dummy variables is equal to 1 for beneficiaries in the years when the beneficiary first received funds through the MPF scheme and for the years thereafter. The dummy variables are equal to 0 otherwise, that is, for beneficiaries in the years before they received funds and for control firms in all years. The error term εict is clustered at the firm level. The coefficients γs represent the effect of the MPF scheme on beneficiaries in cohort s . The overall effect of the scheme is calculated as the linear combination of the effects for each cohort, weighted by the share of beneficiaries in each cohort (Sun and Abraham 2020). To assess the indirect effects of the MPF scheme on nonbeneficiaries in beneficiary sectors (that is, the spillover effects), the analysis also uses a difference-in-differences approach with pooled regressions. In this analy- sis, the treatment dummy variables are based on the year when the first beneficiary in the same 4-digit NACE sector received funds from the MPF scheme. The sample of firms used to evaluate these impacts varies. To measure the direct impact of the MPF scheme as well as its competition impacts, two differ- ent control groups were defined: rejected applicants and matched control group. Those are defined as follows: • The first control group encompasses rejected applicants that applied in the year before beneficiaries in each cohort first received funds. For example, for the 2012 cohort of beneficiaries, the group of rejected applicants from 2011 is used as controls. • To find the second control group, the Euclidean distance matching method is applied separately for each cohort, based on 18 variables (see King and Nielsen 2019). These variables include the year of incorporation; the average inverse hyperbolic sine (IHS)8 of investment 4 years before the cohort first received funds; and yearly data on number of employees and turnover (both using IHS), log TFPR, and log markup, respectively, for the 4 years prior to receiv- ing funds. The matching procedure uses yearly data for employees, turnover, TFPR, and the markup to construct a control group that follows similar trends in these variables. For these variables, the matching is implemented on 4 years of pre-scheme data because this is the largest number of years available for all cohorts.9 A greedy matching algorithm without replacement was applied. This mechanism first calculates the Euclidean distance between each possi- ble pair of beneficiaries and nonbeneficiaries (including rejected firms). It then keeps the pair with the smallest distance and drops all other pairs that contain one of the firms in this pair.10 It then again keeps the pair with the smallest distance and repeats the process until only 17 pairs remain (one for each beneficiary). To measure the spillover effects of the scheme on nonbeneficiaries, a matched control group was constructed. To assess the indirect effects of the scheme on nonbeneficiaries, the analysis takes the nonbeneficiary firms that operated in the same 4-digit NACE sector as a beneficiary, giving 690 firms.11 A matched control group of firms in sectors without a beneficiary is constructed following the same Euclidean distance algorithm as described above. 44 | Design for Impact To test if the control groups are valid comparison groups for the analysis, pre-scheme outcome measures for firms in the treatment and control groups were contrasted. For the control groups used in the assessment of direct and competition impacts, the results presented in appendix J show that the matched control group looks more like the beneficiaries than the rejected applicants control group does. For the control groups used in the assessment of spillover effects over nonbeneficiaries, the results presented in the same appendix show that the matched control group had similar pre-scheme char- acteristics as nonbeneficiaries. Results Direct effects of the MPF scheme on beneficiaries: The incentive effect Was there any increase in the level of employment as a result of the aid? What about an increase in terms of investment? Was there any effect on turnover or productiv- ity as a result of the aid? Results indicate that the MPF scheme increased employ- ment in beneficiaries by between 33.7 percent and 53.1 percent, depending on the control group used. Table 4.9 summarizes the effect of the MPF scheme on bene- ficiaries.12 It shows two sets of estimates: (a) those obtained using the rejected applicants control group and (b) those obtained with the matched control group. Both sets of estimates indicate that the scheme increased employment in benefi- ciaries. The size of this effect represents a 33.7 percent increase in employment when using the rejected applicants control group and a 53.1 percent increase when using the matched control group (column 1), corresponding to between 420 and 662 employees per firm (relative to a pre-scheme mean of 1,247 employ- ees; see table J.1 in appendix J). There is also evidence that the MPF scheme increased investment and turn- over in beneficiaries, but only when using the matched control group and not when using rejected applicants as a control group. The estimates from the rejected TABLE 4.9  Direct effects of the Ministry of Public Finance scheme on beneficiaries (1) (2) (3) (4) CONTROL GROUPS EMPLOYMENT (IHS) INVESTMENT (IHS) TURNOVER (IHS) TFPR (LN) Rejected applicants control group Average treatment effect 0.337** 0.664 0.050 0.053 p-value 0.038 0.138 0.744 0.399 Number of observations 710 710 710 706 Matched control group Average treatment effect 0.531*** 1.984*** 0.397** 0.065 p-value 0.007 0.000 0.010 0.334 Number of observations 340 340 340 340 Source: World Bank elaboration. Note: Based on regressions that pool all cohorts and their control group, with standard errors clustered at the firm level. Regressions include firm fixed effects, as well as cohort-time fixed effects. Each average effect is the linear combination of the cohort effects, weighted by the fraction of beneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate 1, 5, and 10 percent significance levels. Empirical Assessment (Step 3) | 45 applicants control group do not show statistically significant effects on the other outcomes of beneficiaries (investment, turnover, and TFPR), as indicated by the p-value, which is greater than 0.1 for these three variables. The effect on investment is large and positive (a 66.4 percent increase) when using the rejected applicants control group, but it is only statistically significant at the 13.8 percent level.13 The estimated effect on TFPR is also not statistically significant when using the matched control group. However, the results from the matched control group suggest that the MPF scheme is associated with an increase in beneficiaries’ investment of 198.4 percent (column 2) and an increase in turnover of 39.7 percent (column 3). Although the average effects are statistically significant, especially for employment, the yearly effects are not clear. This is explained in part by the small number of firms in the sample. Figures 4.7 and 4.8 examine the direct effects of the MPF scheme on beneficiaries, showing difference-in-differences estimates over time. Figure 4.7 shows estimates obtained with the rejected FIGURE 4.7 Direct effects of the Ministry of Public Finance scheme on beneficiaries over time (rejected applicants control group) a. IHS employees b. IHS investment Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 2 10 1 5 0 0 –1 –5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme C. IHS turnover d. TFPR (ln) Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 1 0.6 0 0.4 –1 0.2 –2 0 –3 –0.2 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary first received funds from the MPF scheme. Each point represents the difference in the average between beneficiaries and control firms, relative to this difference in the first year the firms were observed (for example, for the 2015 cohort, 7 years before a beneficiary first received funds from the MPF scheme). Vertical lines are 95 percent confidence intervals. Estimates are from pooled regressions for all cohorts, which include firm and cohort-year fixed effects, with standard errors clustered at the firm level. The point estimates are the linear combinations of cohort effects, weighted by the fraction of beneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. 46 | Design for Impact FIGURE 4.8 Direct effects of the Ministry of Public Finance scheme on beneficiaries over time (matched control group) a. IHS employees b. IHS investment Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 10 1.5 5 1.0 0 0.5 0 –5 –0.5 –10 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme c. IHS turnover d. TFPR (ln) Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 1.5 0.6 1.0 0.4 0.5 0.2 0 0 –0.5 –1.0 –0.2 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary first received funds from the MPF scheme. Each point represents the difference in the average between beneficiaries and control firms, relative to this difference in the first year the firms were observed (for example, for the 2015 cohort, 7 years before a beneficiary first received funds from the MPF scheme). Vertical lines are 95 percent confidence intervals. Estimates are from pooled regressions for all cohorts, which include firm and cohort-year fixed effects, with standard errors clustered at the firm level. The point estimates are the linear combinations of cohort effects, weighted by the fraction of beneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. applicants control group, and figure 4.8 shows estimates from the matched con- trol group. Both figures plot the differences in outcome variables across benefi- ciaries and control firms in years relative to time 0 (when beneficiaries first received funds from the MPF scheme), compared with this difference in the first year the firms were observed (for example, 7 years before first receiving funds for the 2015 cohort, and 6 years before first receiving funds for the 2014 cohort). Consistent with the results in table 4.12, both figures 4.7 and 4.8 show that the estimated effect of the MPF scheme on employment was close to zero until the scheme started, and it increased thereafter. After the scheme started, the effect of the MPF scheme on employment in any given year is not statistically signifi- cant since the confidence intervals include zero. This lack of significance is in part due to the small number of firms in the sample (71 in figure 4.7 and 34 in figure 4.8). The statistical power in evaluating the effect of the MPF scheme on beneficiaries comes mainly from the large number of observations over time (10   per firm). Unlike the year-by-year estimates in figures 4.7 and 4.8, the estimates in table 4.12 pool all years before and after the scheme, respectively, leading to greater power and giving statistically significant results. Empirical Assessment (Step 3) | 47 Providing a clear explanation of the time path of the MPF scheme effect is not straightforward. This could be explained by sample composition or even by the noisy data, especially in the case of investment. For example, as figures 4.7 and 4.8 illustrate, the effect for employment increases in years 1 and 2 after the scheme and then stabilizes or drops in years 3, 4, and 5. The change in the effect here could be driven by sample composition. Because of the staggered nature of the analysis—that is, different cohorts entering the scheme in different years— some point estimates in figures 4.7 and 4.8 are based on a subset of cohorts only. All four cohorts are contributing only to the estimates for years −4 to 2. Changes in the estimates before and after this time range could thus also be driven by changes in sample composition. Figure 4.8 also illustrates the positive effect on turnover found in table 4.9 in the sample with the matched control group. For investment, the estimates are noisy, as evidenced by the large confidence inter- vals and the fact that the y axis ranges from −10 to 10 (corresponding to 1,000 percent differences), as opposed to −1.0 to 1.5 for turnover (100 percent or 150 percent differences). The effect on TFPR is close to zero before the scheme and then increases slightly, consistent with the positive point estimates in table 4.9. However, the effect on TFPR is smaller compared with the effects on other vari- ables (with the axis showing 20–60 percent differences) and not statistically sig- nificant, neither in the figures nor in table 4.9. Overall, the analysis found that the MPF scheme increased employment in beneficiaries, regardless of the control group used, and that the effects in turn- over and investment were positive, but only when using a matched control group. The results need to be interpreted with caution since they are subject to cave- ats arising from the nature of the impact evaluation of a scheme with a nonran- dom selection of beneficiaries. The empirical approaches applied in the current analysis are designed to ensure as much as possible that the measured effects of the MPF scheme are indeed causal instead of driven by factors other than the MPF scheme. Figures 4.7 and 4.8 show that the outcomes of beneficiaries and their control firms followed parallel trends before the MPF scheme, suggesting that the control groups are valid. However, this conclusion holds only if pre- scheme years are a good approximation of what post-scheme years would have looked like in the absence of the scheme. A concern is that since beneficiaries likely applied to the scheme precisely in years when future growth opportunities materialized, they may have grown at a faster rate than control firms, which did not have growth opportunities in these years, even in the absence of the scheme. This selection bias could explain why positive effects on investment and turn- over were detected when using the matched control group. Since rejected appli- cants also applied to the MPF scheme when they had growth opportunities, the selection bias is mitigated when using rejected applicants as a control group, where the effects of the MPF scheme on investment and turnover are not statis- tically significant. A robustness check is conducted to shed further light on the scheme effects, and the analysis suggests that the results are indeed valid even though beneficia- ries are selected on a nonrandom basis. Figure 4.9 examines the employment trajectory of rejected applicants. It plots the average IHS number of employees in beneficiaries and rejected applicants relative to the year when beneficiaries received funds and the year after rejected firms applied to the MPF scheme, respectively. To reduce noise, the period includes only those years where data for all cohorts are available. The figure shows that employment in rejected 48 | Design for Impact FIGURE 4.9 Employment trends for Ministry of Public Finance beneficiaries and rejected applicants IHS employees 7.5 Average IHS employees 7.0 6.5 6.0 –4 –3 –2 –1 0 1 2 Years relative to MPF scheme Beneficiaries Rejected applicants Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary first received funds from the MPF scheme (for rejected applicants relative to the year after the firm applied for the MPF scheme). The figure only includes years where data for all cohorts are available. IHS = inverse hyperbolic sine; MPF = Ministry of Public Finance. applicants continued the same trend after these firms applied to the scheme and were rejected, as it did in the years before the application. This continuation of the employment trend suggests that rejected applicants did not end up taking advantage of the growth opportunities they hoped to pursue based on their MPF application. That is, the absence of funding may have indeed prevented rejected applicants from pursuing planned projects. In contrast, employment in beneficiaries increased at a faster rate after these firms received funds from the MPF scheme. Overall, the patterns in figure 4.9 mitigate concerns that firms that applied for the MPF scheme would have grown faster in the future even in the absence of the scheme. Indirect (spillover) effects of the MPF scheme on nonbeneficiaries in beneficiary sectors How have the nonbeneficiaries (operating in the same 4-digit sector) behaved as a result of the aid? Results from the analysis indicate positive spillovers from the MPF scheme—in terms of employment and productivity—on nonbenefi- ciaries in beneficiary sectors. In principle, the scheme may generate either positive or negative spillovers for nonbeneficiaries. Positive spillovers may arise if the investments made by beneficiary firms build knowledge and skills, which may diffuse to other firms, for example, as employees switch firms. In addition, some nonbeneficiaries in the same sector may be suppliers to bene- ficiaries. Negative spillovers could occur when firms that receive the grant expand at the expense of those that do not. The results given in table 4.10 suggest that the positive spillovers from the MPF scheme on nonbeneficiaries in beneficiary sectors outweigh the negative spillovers. The MPF scheme increased employment in nonbeneficiaries by 5.9 percent (column 1) and Empirical Assessment (Step 3) | 49 TABLE 4.10  Spillover effects of the Ministry of Public Finance scheme on nonbeneficiary firms in beneficiary sectors (1) (2) (3) (4) EMPLOYMENT (IHS) INVESTMENT (IHS) TURNOVER (IHS) TFPR (LN) Average treatment 0.059* 0.319 0.057 0.012*** effect p-value 0.075 0.160 0.320 0.009 Number of observations 13,800 13,800 13,800 13,739 Source: World Bank elaboration. Note: Based on regressions that pool all cohorts and their control group, with standard errors clustered at the sector level. Regressions include firm fixed effects, as well as cohort-time fixed effects. Each average effect is the linear combination of the cohort effects, weighted by the fraction of nonbeneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate means are different at the 1, 5, and 10 percent significance levels. TFPR by 1.2 percent (column 4). The increase in employment corresponds to 12 employees per firm on average (relative to the pre-scheme average of 212 employees; see table J.2 in appendix J). Overall, these spillover effects on nonbeneficiaries are smaller in magnitude than the direct effects of the MPF scheme. For example, the scheme increased employment in beneficiaries by 33.7–53.1 percent, and by 5.9 percent in nonben- eficiaries. Although the firm-level effects on nonbeneficiaries are smaller (as seems reasonable for indirect compared with direct effects), they also accrue to more firms because there are 690 nonbeneficiaries compared with 17 benefi- ciaries. The estimated effects thus correspond to 690 × 12 = 8,280 jobs created in nonbeneficiaries and to between 17 × 420 = 7,140 and 17 × 662 = 11,254 jobs created in beneficiaries. That is, for every job generated in a beneficiary, the MPF scheme generated another job in a nonbeneficiary. Figure 4.10 shows that the difference-in-differences in outcomes across nonbeneficiaries in beneficiary sectors and their matched controls (nonbeneficiaries in nonbeneficiary sectors) were close to zero before the MPF scheme, suggesting the control group is valid. However, similar to the direct effects on beneficiaries, these estimated effects on nonbeneficiaries are subject to the caveat that beneficiary sectors may have experienced growth opportunities that coincided with the timing of the MPF scheme, so that firms in these sectors may have grown faster than firms in con- trol sectors even in the absence of the scheme. To mitigate this issue, a robust- ness check was performed with a matched control group of nonbeneficiaries from sectors where firms had applied to the MPF scheme and were rejected. These sectors may have had more growth opportunities than the broader set of all nonbeneficiary sectors. The results with this control group are very similar to the ones in table 4.10. Competition effects Has the scheme led to the increase of market share and markup of beneficiaries? How about effects on firm-level measure of allocative efficiency? Overall, there is no robust evidence shows that the MPF scheme had distortive effects on compe- tition at the market level. Table 4.11 shows the effects of the MPF scheme on firm beneficiaries’ market share, markup, and measure of allocative efficiency. In this case, the market (under which measures of a firm’s market share and measures 50 | Design for Impact FIGURE 4.10 Spillover effects of the Ministry of Public Finance scheme on nonbeneficiaries in beneficiary sectors over time a. IHS employees b. IHS investment Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 0.4 3 0.3 2 0.2 1 0.1 0 0 –1 –0.1 –2 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme c. IHS turnover d. TFPR (ln) Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 0.8 0.10 0.6 0.05 0.4 0.2 0 0 –0.2 –0.05 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary in the same 4-digit NACE sector first received funds from the MPF scheme. Each point represents the difference in the average between nonbeneficiaries and control firms, relative to this difference in the first year the firms were observed (for example, for the 2015 cohort, 7 years before a beneficiary in the sector first received funds from the MPF scheme). Vertical lines are 95 percent confidence intervals. Estimates are from pooled regressions for all cohorts, which include firm and cohort-year fixed effects, with standard errors clustered at the firm level. The point estimates are the linear combinations of cohort effects, weighted by the fraction of nonbeneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. of allocative efficiency are computed) is defined as the 4-digit NACE sectors of beneficiaries. The estimates from both control groups suggest that the MPF scheme decreased the markup by about 30 percent, but these estimates are not statistically significant at conventional levels (only at around the 15 percent level). The effects on the market share and firm-level measure of allocative effi- ciency are small and not statistically significant. A possible reason for this find- ing is that before the scheme started, on average, beneficiaries had a relatively small market share of 2.6 percent (see table J.1 in appendix J). THE MINISTRY OF EUROPEAN FUNDS SCHEME Firms included in the analysis The master dataset of the Ministry of European Funds (MEF) scheme consisted of 80 beneficiaries and 54 firms that applied but were rejected. As the “Definition Empirical Assessment (Step 3) | 51 TABLE 4.11  Competition effects of the Ministry of Public Finance scheme (1) (2) (3) FIRM-LEVEL MEASURE OF CONTROL GROUPS FIRM MARKUP (LN) FIRM MARKET SHARE ALLOCATIVE EFFICIENCY Rejected applicants control group Average treatment effect −0.275 −0.002 −0.093 p-value 0.166 0.690 0.311 Number of observations 705 709 705 Matched control group Average treatment effect −0.313 −0.001 −0.057 p-value 0.143 0.826 0.504 Number of observations 340 340 340 Source: World Bank elaboration. Note: Based on regressions that pool all cohorts and their control group, with standard errors clustered at the firm level. Regressions include firm fixed effects, as well as cohort-time fixed effects. Each average effect is the linear combination of the cohort effects, weighted by the fraction of beneficiaries in each cohort. Firm market share and firm-level measure of allocative efficiency are calculated at 4-digit NACE sector level. of Evaluation Questions and Result Indicators (Step 2)” chapter summarizes, this master dataset was obtained by combining the MEF grantor information with information from the SBS dataset, using the BR as an intermediary step to match both datasets. The master dataset includes 80 beneficiaries and 54 rejected applicants. Table 4.12 shows the number of beneficiaries and rejected applicants that were matched to the SBS data by cohort. For beneficiaries, the cohort corresponds to the year when the firm received funds from the MEF scheme for the first time. For firms that applied but were rejected, the applica- tion dates are unknown. Thus, rejected applicants are not assigned to a specific cohort. The MEF scheme beneficiaries that were found in the SBS data received funds for the first time in 2012, 2013, and 2014. Of the firms that were found in the SBS data, the impact evaluation focused on those that appear in all 10 years (2008 through 2017), which narrowed the final sample to 39 beneficiaries and 26 rejected applicants. As table 4.12 shows, of all the firms found in the SBS, only some appear in the data every year. For the impact evaluation analysis, the sample of firms is restricted to firms that reported information each year from 2008 to 2017. With this balanced panel, the analysis can compare beneficiaries to firms in the control group in the years prior to the implementation of the scheme to examine the validity of the control group, which is described in more detail in the next section. Focusing on the balanced panel also guarantees that any changes found as a result of the program are not stemming from changes in the sample composition. Thus, 39 beneficiaries in the 2012, 2013, and 2014 cohorts, along with 26 rejected applicants, are used in the impact evaluation. The beneficiaries in the final sample operate mainly in the sectors of com- puter programming, manufacturing of machinery and equipment, and manufac- turing of electrical equipment. As shown in table 4.13, the 39 beneficiaries in the analysis belong to 20 2-digit NACE sectors. The sector with the most beneficia- ries is the computer programming sector, with seven beneficiary firms mapped to it. Table 4.13 also shows that there is some overlap between the sectors of the 39 beneficiaries used in the analysis and the 41 beneficiaries not used in 52 | Design for Impact TABLE 4.12  Ministry of European Funds scheme beneficiaries and rejected applicants by cohort BENEFICIARIES IN COHORT REJECTED APPLICANTS YEAR SBS BALANCED PANEL SBS BALANCED PANEL 2012 38 25 n.a. n.a. 2013 19 6 Application dates not known 2014 23 8 Total 80 39 54 26 Source: World Bank elaboration. Note: For beneficiaries, the cohort refers to the year when the firm first received funds from the MEF scheme. For rejected applicants, application dates are not known. The balanced panel includes only firms that appear in all 10 years of the dataset (2008 through 2017). SBS = Structural Business Survey; n.a. = Not applicable. the analysis. However, some sectors, most notably human health services, are represented only among the beneficiaries not used in the analysis. When exam- ining nonbeneficiary firms, their distribution across sectors is quite different from that of beneficiaries. About 53 percent of nonbeneficiaries are in sectors that did not have any beneficiaries. The region that concentrates most beneficiaries in the final sample is Bucuresti-Ilfov. The 39 beneficiaries covered in the analysis are in all regions of Romania except for the West region (table 4.14). Bucuresti-Ilfov concentrates 43.6 percent of beneficiaries included in the analysis. An even higher fraction of the 41 beneficiaries not in the analysis are from Bucuresti-Ilfov (58.5 percent). None of these 41 firms are in the South-East region, but three of them are in the West region. Compared with beneficiaries, nonbeneficiaries are less likely to come from Bucuresti-Ilfov, with only 28.2 percent of nonbeneficiaries located in this region. Identification strategy The approach used to evaluate the direct effects of the MEF scheme on benefi- ciaries consists of a difference-in-differences methodology. The differ- ence-in-differences approach compares changes in the outcomes of beneficiaries and their control group over time. The effect for each cohort of beneficiaries is estimated in the pooled regression shown in equation 4.3: yict = α i + β ct + ∑ s = 2012 γ s MEFist + ε ict , 2014 (4.3) Here, yict corresponds to an outcome of firm i in cohort c in year t, such as R&D expenses; ai is a firm fixed effect; bct is a fixed effect at the cohort-year level; and MEFist consists of treatment dummy variables for each cohort. These treatment dummy variables are equal to 1 for beneficiaries in the years when beneficiaries first received funds through the MEF scheme and for the years thereafter. The dummy variables are equal to 0 otherwise, that is, in years before beneficiaries received funds as well as for control firms in all years. The error term εict is clus- tered at the firm level. The coefficients γs represent the effect of the MEF scheme on the beneficiaries in cohort s. As in the MPF scheme, the average effect of the MEF scheme is calculated as the linear combination of the effects for each cohort, weighted by the share of beneficiaries in each cohort. Empirical Assessment (Step 3) | 53 TABLE 4.13  Ministry of European Funds scheme beneficiaries and nonbeneficiaries by sector SHARE OF THE 39 SHARE OF THE 41 SHARE OF BENEFICIARIES IN THE BENEFICIARIES NOT IN NONBENEFICIARIES 2-DIGIT NACE SECTOR ANALYSIS (%) THE ANALYSIS (%) (%) Manuf. food 2.6 4.1 Manuf. chemicals 2.6 4.9 0.6 Manuf. pharmaceutical 2.6 2.4 0.2 Manuf. rubber and plastic 5.1 7.3 1.2 Manuf. other non-metallic mineral products 2.6 0.0 1.2 Manuf. basic metals 5.1 0.0 0.5 Manuf. fabricated metal products 5.1 2.4 2.6 Manuf. computer, electronic, and optical 2.6 4.9 0.5 Manuf. electrical equipment 10.3 2.4 0.5 Manuf. machinery and equipment 15.4 4.9 1.1 Other manufacturing 2.6 2.4 0.5 Sewerage 2.6 0.0 0.0 Waste collection 2.6 0.0 1.2 Specialized construction activities 2.6 0.0 3.5 Wholesale trade 7.7 9.8 18.7 Publishing activities 2.6 0.0 0.7 Telecommunications 2.6 4.9 0.6 Computer programming 18.0 19.5 1.2 Information service activities 2.6 2.4 0.3 Architectural and engineering activities 2.6 4.9 1.4 Printing 0.0 2.4 0.6 Motion picture, video, and television 0.0 production 2.4 0.3 Real estate activities 0.0 2.4 1.6 Activities of head offices 0.0 4.9 1.4 Advertising and market research 0.0 2.4 1.0 Education 0.0 4.9 0.5 Human health activities 0.0 7.3 1.1 Other sectors 0.0 0.0 53.1 Total 100 100 100 Source: World Bank elaboration. Note: For beneficiaries, data are for the year before firms first received funds from the MEF scheme. Some of the beneficiaries not in the analysis do not have SBS data in the year before they started receiving funds from the MEF scheme. In those cases, the closest available year is used. Data for nonbeneficiaries are for 2012. MEF = Ministry of European Funds; NACE = Nomenclature of Economic Activities. The same difference-in-differences approach is used to assess the spillover effects of the program. The spillover effects are defined as the indirect effects that the MEF scheme had on nonbeneficiaries that belong to sectors where ben- eficiaries operate. In this analysis, the treatment dummy variables MEFist from equation (4.3) are based on the year when the first beneficiary in the same 4-digit NACE sector received funds from the MEF scheme. 54 | Design for Impact TABLE 4.14  Ministry of European Funds scheme beneficiaries and nonbeneficiaries by region SHARE OF 39 BENEFICIARIES IN SHARE OF 41 BENEFICIARIES NOT SHARE OF REGION THE ANALYSIS (%) IN THE ANALYSIS (%) NONBENEFICIARIES (%) North-West 7.7 12.2 13.6 Center 15.4 9.8 12.1 North-East 10.3 2.4 9.7 South-East 5.1 0.0 10.7 South-Muntenia 7.7 7.3 10.6 Bucuresti-Ilfov 43.6 58.5 28.2 South-West Oltenia 10.3 2.4 6.2 West 0.0 7.3 8.9 Total 100 100 100 Source: World Bank elaboration. Note: For beneficiaries, data are for the year before they first received funds from the MEF scheme. Some of the beneficiaries not in the analysis do not have SBS data in the year before they started receiving funds from the MEF scheme. In these cases, the closest available year is used. Data for nonbeneficiaries are for 2012. The sample of firms used to evaluate these impacts varies. To measure the direct impact of the MEF scheme as well as its competition impacts, two differ- ent control groups were defined—rejected applicants and a matched control group: • The first control group, rejected applicants, consists of the 26 firms that applied to the program and were rejected. These 26 firms are used as the control group for each cohort of beneficiaries; that is, for the 2012, 2013, and 2014 cohorts of beneficiaries. • A second control group is found using the Euclidean distance matching method, performed separately for each cohort of beneficiaries and based on 12 variables. These variables include the year of incorporation, the average inverse hyperbolic sine (IHS)14 of R&D expenses, TFPR and markup 4 years before the cohort first received funds, and yearly data on IHS turnover and IHS number of employees for the 4 years prior to receiving funds. The match- ing procedure uses annual data for number of workers and turnover to find nonbeneficiary firms that follow similar trends as beneficiaries in these vari- ables. For these variables, the matching is implemented on 4 years of pre- scheme data because this is the largest number of years available for all cohorts.15 The matching was based on a greedy matching algorithm without replacement, which first calculates the Euclidean distance between each pos- sible pair of beneficiaries and nonbeneficiaries (including rejected firms). Then, the algorithm keeps the pair with the smallest distance and drops the remaining pairs that contain one of the firms in this pair. The algorithm then again keeps the pair with the smallest distance and repeats the process until only 39 pairs remain, that is, once each beneficiary has been paired with its most similar nonbeneficiary. The algorithm is then run again on the remain- ing firms to find the second-best match for each beneficiary. Thus, the match- ing procedures yield 78 control firms, two for each beneficiary.16 The spillover effects that the MEF scheme had on nonbeneficiaries are mea- sured using a matched control group. The indirect effects of the scheme on nonbeneficiaries are measured as follows. First, the firms that operated in the same 4-digit NACE sector as a beneficiary are selected, totaling 1,840 firms.17 Empirical Assessment (Step 3) | 55 A matched control group of firms in sectors without a beneficiary is found using the same Euclidean distance algorithm described above. Outcome measures of firms in the treatment and control groups in the pre-scheme years were compared to test the validity of the control groups. Appendix L shows the results of the validity of the control group used in the evaluation of the direct and competition effects of the MEF scheme (table L.1 in appendix L). No statistically significant differences are seen across bene- ficiaries and either the rejected applicants or the matched control group. One caveat of using the rejected applicants as a control group is that these firms were rejected because they obtained worse scores in the MEF scheme evalu- ation, which may imply that these firms have lower expected growth paths (and presumably lower expected R&D expenses) than beneficiaries. Given that the scores are not available for most rejected applicants, the analysis cannot distinguish how far from the cutoff the sample of rejected firms was. However, this concern is reduced by evidence from McKenzie and Sansone (2019), who examined a business plan competition and found that scores do not necessarily predict future firm growth. Table L.2 in appendix L also dis- plays the tests conducted to compare the treatment and control groups for the evaluation of the spillover effects of the MEF scheme. The results again suggest that the matched control group is a valid comparison group based on its pre-scheme characteristics. Results Direct effects of the MEF scheme on beneficiaries: The incentive effect Was there any increase in R&D expenditures as a result of the aid? What about an increase in terms of employment or turnover? Was there any effect on productivity as a result of the aid? Results indicate that the MEF scheme increased R&D expenses of beneficiaries by between 172 percent and 342 percent, depend- ing on the control group used. The direct effect of the MEF scheme on bene- ficiaries is summarized in table 4.15. The top panel of the table shows the estimates obtained using the rejected applicants as control group. The bot- tom panel shows estimates obtained using the matched control group. Both sets of estimates indicate that the scheme increased R&D expenses of bene- ficiaries. As column 2 shows, the size of this effect is large, representing an increase of 172 percent in R&D expenses when using the control group of rejected applicants and a 342 percent increase when using the matched con- trol group. No effect of the scheme is detected on the employment, turnover, or TFPR (columns 1, 3, and 4). The yearly effects of the MEF scheme show that the only outcome affected by the program over time is R&D expenses, which increase a year after the aid is received. Figures 4.11 and 4.12 illustrate the difference-in-differences estimates of the MEF scheme on beneficiaries over time. Figure 4.11 shows estimates obtained using as control group the rejected applicants, and figure 4.12 shows the estimates when the control group is obtained by the matching algorithm. The two figures plot the differences in outcome variables across beneficiaries and control firms in years relative to time 0 (when beneficiaries first received aid from the MEF scheme), compared with this difference in the first year that the firms were observed (for example, 6 years before receiv- ing funds for the 2014 cohort). The figures show evidence consistent with the 56 | Design for Impact TABLE 4.15  Direct effects of the ministry of european funds scheme on beneficiaries (1) (2) (3) (4) CONTROL GROUPS EMPLOYMENT (IHS) R&D EXPENSES (IHS) TURNOVER (IHS) TFPR (LN) Rejected applicants control group Average treatment effect 0.086 1.723* 0.068 0.074 p-value 0.566 0.097 0.679 0.628 Number of observations 1,170 1,170 1,170 1,143 Matched control group Average treatment effect 0.133 3.421*** 0.052 −0.063 p-value 0.126 0.000 0.589 0.222 Number of observations 1,170 1,170 1,170 1,157 Source: World Bank elaboration. Note: Based on regressions that pool all cohorts and their control group, with standard errors clustered at the firm level. Regressions include firm fixed effects, as well as cohort-time fixed effects. Each average effect is the linear combination of the cohort effects, weighted by the fraction of beneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate 1, 5, and 10 percent significance levels. FIGURE 4.11 Direct effects of the Ministry of European Funds scheme on beneficiaries over time (rejected applicants control group) a. IHS employees b. IHS R&D expenses Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 0.5 6 4 0 2 –0.5 0 –2 –1.0 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme d. IHS turnover c. TFPR (ln) Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 1.0 0.6 0.4 0.5 0.2 0 0 –0.5 –0.2 –1.0 –0.4 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary first received funds from the MEF scheme. Each point represents the difference in the average between beneficiaries and control firms, relative to this difference in the first year the firms were observed (for example, for the 2014 cohort, 6 years before a beneficiary first received funds from the MEF scheme). Vertical lines are 95 percent confidence intervals. Estimates are from pooled regressions for all cohorts, that include firm and cohort-year fixed effects, with standard errors clustered at the firm level. The point estimates are the linear combinations of cohort effects, weighted by the fraction of beneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity; R&D = research and development. Empirical Assessment (Step 3) | 57 FIGURE 4.12 Direct effects of the Ministry of European Funds scheme on beneficiaries over time (matched control group) a. IHS employees b. IHS R&D expenses Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 0.6 10 0.4 5 0.2 0 0 –0.2 –0.4 –5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme c. IHS turnover d. TFPR (ln) Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 1.0 0.4 0.2 0.5 0 0 –0.2 –0.5 –0.4 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary first received funds from the MEF scheme. Each point represents the difference in the average between beneficiaries and control firms, relative to this difference in the first year the firms were observed (for example, for the 2014 cohort, 6 years before a beneficiary first received funds from the MEF scheme). Vertical lines are 95 percent confidence intervals. Estimates are from pooled regressions for all cohorts, which include firm and cohort-year fixed effects, with standard errors clustered at the firm level. The point estimates are the linear combinations of cohort effects, weighted by the fraction of beneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity; R&D = research and development. results in table 4.15. Beneficiaries do not experience any change in their employment, turnover, and TFPR as a result of the MEF scheme. However, relative to the control group, they increase their R&D expenses a year after receiving the first MEF grant. The yearly effect on R&D expenses calculated with the matched control group (figure 4.12) is statistically significant for all postscheme years. The lack of significance of the yearly effects on R&D expenses in figure 4.11 is because these effects are smaller than those in figure 4.12 (in line with table 4.15). Unlike the year-by-year estimates in these figures, the estimates in table 4.15 pool all years before and after the scheme, respectively, leading to greater power and giving statistically significant results for the rejected applicants control group. It is difficult to draw conclusions about the time path of the effect on R&D expenses. This could be explained by changes in sample composition. The time paths of the effect of the MEF scheme on R&D expenses are shown in 58 | Design for Impact ­ igures 4.11 and 4.12. R&D expenses of beneficiaries show a large increase rela- f tive to the control group 1 year after receiving MEF funds for the first time. This effect remains relatively constant in years 2 and 3. It then increases in year 4. The change in the effect here could be driven by sample composition. Given the staggered nature of the analysis—that is, different cohorts entering the scheme in different years—some point estimates in figures 4.11 and 4.12 are based only on a subset of cohorts. All four cohorts are contributing only to the estimates for years −4 to 3. Changes in the estimates before and after this time range could thus also be driven by changes in sample composition. The overall results indicate that the MEF scheme increased R&D expenses of beneficiary firms, regardless of the control group used, with no effect detected on turnover, employment, and TFPR. As with the evaluations of the previous schemes, the results need to be interpreted with caution since they are subject to caveats arising from the ex post impact evaluation of the scheme, which had no random selection of beneficiaries. Evidence from figures 4.11 and 4.12 show that prior to the MEF scheme, beneficiaries and their control firms followed parallel trends in their outcomes. Indirect (spillover) effects of the MEF scheme on nonbeneficiaries in beneficiary sectors How have the nonbeneficiaries (operating in the same 4-digit sector) behaved as a result of the aid? The analysis detected no spillovers from the MEF scheme on nonbeneficiaries that belong to sectors with beneficiaries. As table 4.16 shows, nonbeneficiaries from the same 4-digit NACE sectors as beneficiaries did not experience any statistically significant change in their employment, R&D expenses, turnover, and TFPR as a result of the MEF scheme. Figure 4.13 illus- trates the difference-in-difference estimates in outcomes across nonbeneficia- ries that operate in beneficiary sectors and their matched control group, which corresponds to nonbeneficiaries in sectors with no beneficiaries. The figure shows no statistically significant difference between the treatment and control groups in the various outcomes during the pre-scheme years, which suggests that the matched control group used to evaluate the spillover effects on nonben- eficiaries is valid. The figure also shows no evidence of statistically significant spillover effects of the MEF scheme. TABLE 4.16  Spillover effects of the Ministry of European Funds scheme on nonbeneficiaries in ­beneficiary sectors (1) (2) (3) (4) EMPLOYMENT (IHS) R&D EXPENSES (IHS) TURNOVER (IHS) TFPR (LN) Average treatment effect 0.030 0.035 0.020 −0.009 p-value 0.248 0.678 0.637 0.620 Number of observations 36,800 36,800 36,800 35,446 Source: World Bank elaboration. Note: Based on regressions that pool all cohorts and their control group, with standard errors clustered at the sector level. Regressions include firm fixed effects as well as cohort-time fixed effects. Each average effect is the linear combination of the cohort effects, weighted by the fraction of nonbeneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity. Empirical Assessment (Step 3) | 59 FIGURE 4.13 Spillover effects of the Ministry of European Funds scheme on nonbeneficiaries in beneficiary sectors over time a. IHS employees b. IHS R&D expenses Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 0.15 0.5 0.10 0.05 0 0 –0.5 –0.05 –0.10 –1.0 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme c. IHS turnover d. TFPR (ln) Percent (1 corresponds to 100%) Percent (1 corresponds to 100%) 0.3 0.05 0.2 0 0.1 0 –0.05 –0.1 –0.2 –0.10 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 –6 –5 –4 –3 –2 –1 0 1 2 3 4 5 Years relative to MPF scheme Years relative to MPF scheme Source: World Bank elaboration. Note: The x axis shows years relative to the year when a beneficiary in the same 4-digit NACE sector first received funds from the MEF scheme. Each point represents the difference in the average between nonbeneficiaries and control firms, relative to this difference in the first year the firms were observed (for example, for the 2014 cohort, 6 years before a beneficiary in the sector first received funds from the MEF scheme). Vertical lines are 95 percent confidence intervals. Estimates are from pooled regressions for all cohorts, which include firm and cohort-year fixed effects, with standard errors clustered at the firm level. The point estimates are the linear combinations of cohort effects, weighted by the fraction of nonbeneficiaries in each cohort. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. TFPR = revenue-based measure of total factor productivity; R&D = research and development. Competition effects Has the scheme led to the increase of market share or markup of beneficiaries? What about effects on firm-level measure of allocative efficiency? The analysis found no evidence indicating that the MEF scheme had distortive effects on competition at the market level. Table 4.17 summarizes the impact of the MEF scheme on firm beneficiaries’ markup, market share, and measure of allocative efficiency. The market is defined as the 4-digit NACE sectors of beneficiaries (at the national level). The estimates from both control groups find no statistically significant impact of the MEF scheme on these outcomes. 60 | Design for Impact TABLE 4.17  Competition effects of the Ministry of European Funds scheme (1) (2) (3) FIRM-LEVEL MEASURE OF CONTROL GROUPS FIRM MARKUP (LN) FIRM MARKET SHARE ALLOCATIVE EFFICIENCY Rejected applicants control group Average treatment effect 0.061 −0.001 11.933 p-value 0.725 0.692 0.256 Number of observations 1,143 1,170 1,143 Matched control group Average treatment effect 0.092 0.000 2.399 p-value 0.480 0.935 0.824 Number of observations 1,157 1,170 1,157 Source: World Bank elaboration. Note: Based on regressions that pool all cohorts and their control group, with standard errors clustered at the firm level. Regressions include firm fixed effects, as well as cohort-time fixed effects. Each average effect is the linear combination of the cohort effects, weighted by the fraction of beneficiaries in each cohort. Market share and allocative efficiency are calculated relative to other firms in the same 4-digit NACE sector at the national level. NOTES 1. The 2-digit NACE code sectors included in each group category are listed in table F.1 of appendix F. 2. The difference-in-differences method compares the changes in outcomes over time between the treatment group (comprising the beneficiaries of the aid scheme) and the con- trol group (comprising the firms that are not enrolled in the aid scheme). This method eliminates any differences between these groups that are constant over time. This method works if, over time, both the beneficiaries and the control group are affected in the same way by the other factors that also affect performance. It can then be concluded that the aid is the only relevant factor that explains the observed change in performance of beneficia- ries relative to the control group. 3. As per the Siruta classification system, the siruta territorial unit at level 2 corresponds to municipalities, towns, and communes. 4. Firms in the BR data operate in 614 4-digit NACE sectors. Beneficiary firms are concen- trated in 217 sectors. 5. The average equivalent nominal value of the counter-guarantee provided to beneficiaries was €27,900. This value results from the division of the total amount provided under the RCG scheme throughout the implementation period (€24.08 million) by the total number of beneficiaries (863). The current analysis considers a sample of 784 beneficiaries as explained in the “Firms Included in the Analysis” section. It is worth highlighting that this average equivalent nominal value differs from the gross subsidy equivalent of the count- er-guarantee. The latter is calculated as the difference between the counter-­ guarantee premium that a beneficiary should have paid under market conditions and the count- er-guarantee premium actually paid by the beneficiary, taking into account the value of the counter-guarantee and its duration. 6. Overall, the scheme reduced the exit rate of beneficiaries across all sectors, with the prob- ability of firms closing dropping from 18 percent in the wholesale and retail trade sector to about 28 percent in sectors such as administrative activities and construction and others. 7. Not all firms appear in the balanced panel because some firms opened after 2009. Some firms may also fall below the 20-employee inclusion threshold in certain years. All 34 ben- eficiaries are in the dataset in 2017, implying that none of them closed. Of the 19 beneficia- ries, 2 of them (from the 2009 and 2016 cohorts) were not used in the analysis because these cohorts included only 1 firm, and there were no rejected firms that applied in years close to these cohorts that could serve as a control group. Empirical Assessment (Step 3) | 61  8. The IHS is used to reduce influence of outliers. It has the advantage over the log function in that it allows for zeroes. As for the log function, changes in the IHS can be interpreted as percentage changes.  9. The matching is not implemented on yearly data for investment since these data vary widely from year to year, reflecting the lumpy nature of investment. 10. Power calculation is the name given to the process for determining the sample size for a research study. The technical definition of power is that it is the probability of detecting a true effect when it exists. In the current exercise, power calculations were used to deter- mine the size of the control group, that is, how many matched control firms to select for each treated firm. These calculations applied the following parameters: α equals 0.05 and a power of 0.9. Effect sizes were those from the results with the rejected applicant control group. To detect a 33.9 percent increase in R&D expenses, one matched control firm is enough, since the estimation uses 10 years of data and employment is highly correlated across years (correlation = 0.97). In fact, power calculations show that only four firms are needed in each the treatment and control group to detect a 33.9 percent increase in employ- ment. Investment has a much lower correlation over time (0.42), so that having 10 years of data does not increase the power for investment in the same way it does for employment. Even if 10 control firms were selected for each beneficiary, the analysis would need 80 beneficiaries to detect a 66.4 percent increase in investment. The algorithm was thus set to select only one matched firm per beneficiary—that is, the best match—which should pro- vide enough power to detect the effect on employment. More matched control firms were not added to the control group because the second, third, and following best matches become increasingly less like beneficiaries, and in any case, there are not enough beneficia- ries to increase the power for detecting effects on investment. 11. For nonbeneficiaries that switch sectors, the sector is defined in the year when the benefi- ciary in the same sector first received funds. As was done for beneficiaries, to construct a balanced panel, only nonbeneficiaries that appear in all 10 years were kept in the sample. 12. The average equivalent nominal value of the aid per beneficiary amounted to €9.14 million. This value results from the division of the total amount disbursed under the MPF scheme throughout the implementation period (€338.16 million) by the total number of beneficia- ries (37). The analysis considers a sample of 34 beneficiaries, as explained in “Firms Included in the Analysis” section. 13. The fact that the effect on investment is not statistically significant may be due to the rela- tively small sample size and the noise in the investment variable, which has a low correla- tion over time (0.42). Power calculations with α equal to 0.1 suggest that the power for detecting a 66.4 percent increase in investment with 10 years of data and 17 firms in the treatment group (54 in the control group) is only 0.38. 14. The IHS is used to reduce the influence of outliers. It has the advantage over the log func- tion in that it allows for zeroes. As for the log function, changes in the IHS can be inter- preted as percentage changes. 15. The matching is not implemented on yearly data for R&D expenses because there is large variation in these data from year to year. For TFPR and markup, averages across years are used because these variables are missing in some years. 16. In the current exercise, power calculations were used to determine how many matched control firms should be selected for each treated firm. These calculations applied the fol- lowing parameters: alpha equals to 0.05 and a power of 0.9. Effect sizes were those from the results with the rejected applicant control group. To detect a 172.3 percent increase in R&D expenses, one matched control firm is enough, since the effect is large. In fact, power cal- culations show that 22 firms are needed in each treatment and control group, respectively, to detect a 172.3 percent increase in R&D expenses. Still, the algorithm was set to select two matched firms per beneficiary—that is, the two best matches—to provide more power to detect a possible effect on employment. Although the analysis with the rejected applicants control group shows no statistically significant effect on employment, the coefficient is positive, corresponding to an 8.6 percent increase. Because the analysis uses 10 years of data and the correlation of employment is very high over time (0.97 compared to 0.53 for R&D expenses), 39 treatment firms and 78 control firms would be enough to detect an 8.6 percent increase in employment with the parameters given above. 17. Some nonbeneficiaries switch sectors over time. In these cases, the sector is defined as the year when the beneficiary in the same sector first received funds. To construct a balanced panel, only nonbeneficiaries that appear in all 10 years were kept in the sample. 62 | Design for Impact REFERENCES King, G., and R. Nielsen. 2019. “Why Propensity Scores Should Not Be Used for Matching.” Political Analysis 27 (4): 435–54. Sun, L., and S. Abraham. 2020. “Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects.” Background Paper, Massachusetts Institute of Technology, Cambridge, MA. http://economics.mit.edu/files/14964. 5 Conclusions: Implications for State Aid Design and Implementation, and Lessons Learned (Step 4) The objective of the ex post evaluation of selected state aid schemes in Romania is to assess whether and to what extent the objectives of each aid scheme have been fulfilled and to measure their spillover effects as well as the effects on com- petition outcomes. The analysis applied rigorous microeconometric techniques to assess the relative positive and negative direct and indirect effects stemming from the schemes. RESULTS The de minimis scheme to incentivize access to finance for MSMEs (the RCG scheme) Regarding the Romanian Counter-Guarantee Fund (RCG) scheme, the analysis found evidence that the scheme fulfilled its objective of supporting entrepre- neurship in Romania. Results showed that the scheme increased the employ- ment and turnover of beneficiary firms and reduced the probability of aided firms closing. Except for the revenue-based measure of total factor productivity (TFPR), all the other result indicators applied to quantify the direct effects pointed to the economic additionality of the scheme. Specifically, employment and turnover have increased (by 43 percent and 148 percent, respectively) as a result of the scheme, while the probability of aided firms closing has decreased by 23 percent. All these estimates were statistically significant, and all the robust- ness checks validated these findings. The analysis showed that the positive direct effects of the RCG scheme varied across regions and sectors. The scheme’s impacts varied across regions, with scheme beneficiaries in the North-West and Bucuresti-Ilfov regions—which together account for 46 percent of all beneficiaries—experiencing large increases in terms of turnover and employment. There was also heterogeneity of scheme effects across sectors. For example, the wholesale and retail trade sector, which accounts for the largest proportion of beneficiaries (30.6 percent), experienced the smallest increase in turnover and employment and the smallest reduction in the probability of closing. On the other hand, the other services sector  63 64 | Design for Impact experienced the largest increases in turnover and employment, while beneficia- ries in administrative activities and construction and others experienced the largest drop in the probability of closing. Results also showed that the spillover effects of the RCG scheme on nonben- eficiaries were negative in terms of job creation, suggesting potential job dis- placement.1 However, the magnitude of the spillover effect on nonbeneficiaries in terms of job creation is smaller compared to the positive direct effect on ben- eficiaries. The evaluation showed the following: • The scheme reduced the average number of workers employed by nonbene- ficiary firms by 2.7 percent. This can be interpreted as a potential sign of a job displacement effect. • The degree of complementarity between beneficiary and nonbeneficiary firms varies across sectors. Some sectors, such as health and education, ben- efited indirectly from the RCG scheme. For other sectors, such as wholesale and retail trade and manufacturing, the scheme may have had negative spill- over effects on the performance of nonbeneficiaries. • The negative spillover effects on nonbeneficiaries varied across regions. In particular, the drop in the number of workers among nonbeneficiaries (as well as the drop in firm creation) was significantly higher in regions with high employment rates. The latter suggests that nonbeneficiary firms may find it more difficult to replace workers in regions where labor markets are tighter. In particular, the drop in the number of workers (conditional on firms that are open) for the average nonbeneficiary firm is greater in the Bucuresti-Ilfov, South-East, and West regions, and the drop in the rate of firm creation in Bucuresti-Ilfov is much larger and statistically significant than in other regions (with an estimated impact of about 3 percentage points). Not surprisingly, the analysis did not show any evidence that the RCG scheme distorted competition. All estimates of average treatment effects of the scheme on beneficiaries’ market share, markup, and measure of allocative efficiency were found to be statistically insignificant. The analysis found evidence that the market share of aided firms increased only when a very granular market definition was considered (at 4-digit Nomenclature of Economic Activities and siruta territorial unit level 2), but even in this case the increase was only 2.7 percent. These results were somehow expected because the RCG scheme was a de minimis scheme. Typically, de minimis aid implies that small amounts of state aid are given to a firm, which cannot exceed €200,000 over any three fiscal years irrespective of the firm’s size or location. Hence, given the small size of aid, the ex-ante assumption that schemes that qualify under the de minimis regulation are unlikely to distort com- petition is confirmed by the ex post analysis. In fact, the scale of the scheme is small when compared with the market size: the total amount of aid provided by the RCG scheme is negligible when compared with the total value of turnover by micro, small, and medium enterprises (MSMEs) in Romania in the same period, as per Eurostat figures.2 The state aid scheme to support regional development and job creation (the MPF scheme) Regarding the Ministry of Public Finance (MPF) scheme, there is robust evi- dence of a positive direct effect on employment and, to some extent, Conclusions: Implications for State Aid Design and Implementation, and Lessons Learned (Step 4) | 65 on investment. This suggests that the state aid objective has been fulfilled. The analysis found the following statistically significant effects of the MPF scheme: • Beneficiaries’ employment increased by between 33.7 percent and 53.1 per- cent, depending on the control group used as reference. This increase cor- responds to between 420 and 662 employees per firm (relative to a pre-scheme mean of 1,247 employees), which is well above targeted employ- ment growth as defined by the scheme of at least 50 additional employees per firm. • Evidence on investment is mixed. Overall, the results suggest that the MPF scheme had a positive effect on investment, but this impact lacks statistical significance, at least partially. There is evidence that the MPF scheme increased investment in beneficiaries by 66.4 percent, when using the rejected applicants control group, but this effect was not statistically significant at conventional levels (only at the 13.8 percent level). When using the matched control group, the estimated effect was larger, by 198 percent, and was statistically significant at conventional levels. The par- tial lack of statistical significance can be explained by the relatively small sample size, noisy investment data, and the fact that the investment effects may take longer to materialize. • The MPF scheme did not render any significant effects on productivity for the beneficiary firms. Evidence showed positive spillovers—in terms of employment and pro- ductivity—of the MPF scheme on nonbeneficiaries in beneficiary sectors. For every job generated in a beneficiary, the MPF scheme generated another job in a nonbeneficiary. Employment in nonbeneficiaries increased by 5.9 per- cent, and productivity (measured by TFPR) increased by 1.2 percent. The estimated impact on employment corresponds to 12 employees per firm on average (relative to the pre-scheme average of 212 employees). Although these effects are smaller when compared with direct effects, they accrue to more firms—690 nonbeneficiaries compared to 17 beneficiaries. The esti- mated effects thus correspond to 690 × 12 = 8,280 jobs created in nonbenefi- ciaries and to between 17 × 420=7,140 and 17 × 662=11,254 jobs created in beneficiaries. That is, for every job generated in a beneficiary, the MPF scheme generated another job in a nonbeneficiary. However, because the MPF grantors’ dataset does not provide information on how the beneficiaries spent the grant—specifically, the proportion used to cover investments in intangible and tangible assets (fixed costs) or wage bill and social insurance costs (marginal costs)—it is not possible to infer positive spillovers. For example, eventual diffusion of knowledge and skills that could be associated with further investment in new technology (for example, as employees switch firms within the same sector) cannot be implied. In addition, there is no evidence that the MPF scheme rendered any ­distortive effects on competition at the market level. The analysis showed that effects of the MPF scheme on beneficiary firms’ market share, markup, and measure of allocative efficiency are not statistically significant. The scheme has not trig- gered the expansion of beneficiaries’ market share at the expense of nonbenefi- ciaries or increased beneficiaries’ pricing power. By the same token, there was no evidence that the aid has hindered the allocation of resources to the most productive businesses. 66 | Design for Impact The state aid scheme to support the upgrade and modernization of the R&D&I (the MEF scheme) As for the MEF scheme, the analysis found evidence that the scheme fulfilled its objective of promoting R&D efforts. Results indicate that the MEF scheme increased the R&D expenses of beneficiaries, depending on the control group used. The analysis showed that the size of input additionality (defined as the change in private R&D expenditures that can be attributed to public funding) is large, representing a 172 percent increase in R&D expenses when using the con- trol group of rejected applicants and a 342 percent increase when using the matched control group. On the other hand, there was no significant evidence that the scheme affected beneficiaries’ performance when measured in terms of employment, turnover, or TFPR. The assessment also found no evidence of spillover effects from the MEF scheme on nonbeneficiaries that belong to beneficiary sectors, nor did evidence indicate that the MEF scheme had distortive effects on competition at the mar- ket level. The conclusions drawn from the estimated impacts of MEF scheme need to be interpreted with caution, for several reasons: • First, the grants provided through the MEF scheme were structured across different operations under the Sectoral Operational Program for Increasing Competitiveness. The operations had different objectives: promoting research collaboration between universities/research institutions and firms and financing the development (or upgrading) of new or existing products by existing firms. They also had different targeted actors operating in the R&D&I ecosystem, that is, universities or research institutions and firms.3 Because the evaluation used the firm as the key unit of analysis, it was not possible to capture any effect the scheme might have had on universities and research institutions. • Second, regarding the effects at firm level, the analysis employed result indi- cators that capture a partial picture of the innovation process—namely, the R&D expenditure angle—and more general performance measures such as productivity, turnover, and employment. In this regard, it is worth highlight- ing that innovation output measures, such as patents or publications, could not be captured by the analysis. • Third, given that Structural Business Survey (SBS) data span 2008–17 and that beneficiary firms under analysis received funds for the first time in 2012, 2013, and 2014, a maximum of 6 years of MEF scheme data are available. Therefore, this may not be enough time to detect effects on employment, sales, and TFPR.4 • Fourth, because the SBS has two strata (one completely enumerated cov- ering a census of all enterprises with at least 20 employees, and a ran- dom stratum covering some smaller firms), it was not possible to draw a complete picture of beneficiaries and rejected applicants that are below 20 employees, that is, the small firms. In other words, given the SBS size-coverage restriction, the analysis could not explore the size dimen- sion, and more specifically, the heterogeneity of the scheme impacts across firms of different sizes (and with distinct levels of financial constraints).5 Conclusions: Implications for State Aid Design and Implementation, and Lessons Learned (Step 4) | 67 KEY LESSONS FOR FUTURE IMPACT EVALUATIONS State aid design and implementation Despite the positive results achieved by the RCG scheme, it is important to review the scheme’s overall implementation to maximize its efficiency and effectiveness for informing future aid design. The impact evaluation results did not clearly show whether the direct positive effects of the RCG scheme are out- weighed by potential negative spillover effects for nonbeneficiaries. Broadening the direct positive effects of the RCG scheme will depend on factors associated with access to finance in general.6 Other factors that may need to be reviewed are related to (a) the design of the RCG scheme, particularly its operating rules and key terms of the counter-guarantees offered;7 and (b) the implementation of the scheme. It was out of the scope of the current evaluation to run a simulation (or a sensitivity analysis) to identify the best combination of counter-guarantee parameters that would be able to maximize the performance results for the ­beneficiary firms. On the other hand, findings on impacts of the RCG scheme suggest opportu- nities to streamline the implementation of state aid schemes that have similar characteristics. The estimates of average treatment effect varied across sectors; for example, wholesale and retail trade, which concentrates the largest propor- tion of beneficiary firms, experienced the smallest positive direct effects in terms of investments, turnover, and job creation and large negative spillover effects in terms of job displacement. Thus, it is reasonable to ask whether the aggregate effect of the scheme could be optimized if the scheme were implemented with a different sectoral focus. In this regard, giving more emphasis to technology- and knowledge-intensive service activities in the future might be worth considering. For example, attention could be given to financing for the acquisition of intangi- ble assets. Although the risks associated with these types of investments and sec- tors are perceived to be higher, their returns also tend to be high because they are well-known drivers of productivity growth and regional transformation.8 The results of the MPF scheme impact evaluation suggest opportunities to rationalize the scheme design to improve efficiency. With a total cost of €1 bil- lion, the scheme fulfilled its objectives of supporting regional development by generating jobs and promoting investment, though the results lack statistical sig- nificance for the latter.9 In this context, the discussion should involve ways to streamline the aid design to improve its efficiency, for example, by (a) expanding the minimum targeted level of job creation; (b) restricting the eligible expendi- tures to fixed costs that are likely to translate into new technologies, new pro- cesses, and eventually new products; and (c) revising the aid intensity parameters. This is particularly relevant because under the MPF scheme, the aid is provided on a rolling basis. In addition, the impact evaluation showed no direct effect on productivity performance, which might suggest that the additional jobs created are not necessarily more productive. Regarding the MEF scheme, the evaluation showed potential opportunities to restructure the design and implementation of the scheme to meet the objec- tives of upgrading and modernizing R&D&I. The MEF scheme encompassed operations with different objectives and different targeted actors of the R&D&I ecosystem. Although the evaluation presented in this report had its limitations, 68 | Design for Impact mostly caused by data constraints, and did not allow the detection of which of these operations was more effective, it is still possible to indicate some areas for design improvement. The most relevant one is to focus the R&D support on applied research rather than to finance activities that are closer to the market.10 As for implementation, an important aim is to strengthen the capacity of the granting authority to document and file (in an electronic format) the informa- tion provided by beneficiaries and rejected applicants during the selection pro- cess. Such information includes not only the scores obtained by firms during the evaluation process but also basic information, such as the date when the appli- cation was rejected, and summary information on the objective and scope of granted projects. Data This evaluation showcased the complexity of working with multiple sources of data and information, as well as the importance of having access to appropriate microeconomic data (at the most disaggregated level possible). A critical step for a successful ex post impact evaluation is to build a master dataset containing information provided by the aid-granting authority as well as data capturing the result indicators of both aided beneficiaries and nonbenefi- ciary firms. As for the data-capturing result indicators, the current analysis drew from two administrative data sources: the Business Registry (BR) and Structural Business Survey (SBS). In this regard, it is worth highlighting that the versions of these datasets shared with the World Bank team had some restrictions: • First, in terms of time coverage, this is particularly relevant for the BR dataset, since its time span (2011–17) was only 1 year prior to RCG scheme implemen- tation (2012). • Second, the SBS version shared with the World Bank team did not contain the firms’ unique identifier (CUI) information; this was addressed through the data-cleaning process described in the “Definition of Evaluation Questions and Result Indicators (Step 2)” chapter.11 • Third, the SBS dataset is headquarters based, not plant-level, data. This has somehow restricted the scope of the analysis of the MPF scheme and pre- cluded the analysis from making inferences about the MPF scheme effects at the local level. As for the data provided by the aid-granting authorities, there were additional limitations: • For the RCG scheme, there was no information on rejected firms that applied to the scheme, which has precluded the analysis from refining the control group of the evaluation. This was also partly related to the instrument used and the availability of data on the rejected applicant at the grantor level. It would have been useful to know if, in the siruta territorial units (at level 2) where the RCG scheme was not rolled out, there were firms that applied for the scheme but were not selected. • Regarding the MPF scheme, it would have been useful to obtain detailed information on how beneficiaries spent the grant received under the scheme— specifically, to split amounts received into expenses that targeted investment (tangible and intangible) versus those for wage support. Conclusions: Implications for State Aid Design and Implementation, and Lessons Learned (Step 4) | 69 TABLE 5.1  Data requirements for impact evaluations DATA SOURCE FEATURES Administrative data Cover the longest time frame possible, covering before and after scheme disbursement and implementation Include a unique identifier for firms, to allow the complete mapping of beneficiaries and rejected applicants Contain information of geographic location not only for firms’ headquar- ters but also for their production and operation plants Data provided by granting Collect and file information not only on beneficiaries but also on rejected authority applicants that applied to the scheme Include the most detailed level of information on how beneficiaries spent the funding Source: World Bank elaboration. • For the MEF scheme, it would have been useful to obtain the complete set of scores received by beneficiaries and rejected applicants in the selection pro- cess. This would allow a different methodology (regression discontinuity) to be applied to assess the scheme’s impact. In addition, it would have been use- ful to get information on rejected applicants, notably about the date of rejec- tion, which would allow the refinement of the control group definition used in the analysis presented in the report. The need to have access to sufficient, consistent, and accurate micro-level data is the most important lesson. A thorough and robust state aid impact evalu- ation is crucial for policy purposes. It allows researchers to confirm or reject whether the assumptions underlying the approval of the aid scheme are still valid and is helpful for improving the design of future aid schemes. Having suffi- cient, consistent, and accurate micro-level data at hand is fundamental to ensure the development of a robust evaluation. In this context, the information to be used in future evaluations in Romania should be suitable to measure the out- comes of the aid schemes at the appropriate level of granularity, both across time and space. Table 5.1 summarizes the key data features for future evaluations. NOTES 1. As explained in the “Indirect (spillover) effects of the RCG scheme on nonbeneficiaries” section, the negative spillover effect over nonbeneficiaries in terms of employment is driven by (nonbeneficiary) firms that remained open and shrank in size. This negative effect was significantly higher in regions with high employment rates, which suggests that job displacement effects created by the RCG scheme might be associated with labor market tightness at the regional level. Overall, the negative effect over non-beneficiaries is, how- ever, much smaller than the direct effect of employment creation in the beneficiary firms. 2. As per Eurostat official figures; see https://appsso.eurostat.ec.europa.eu/nui/submitView- TableAction.do. 3. As discussed in the “Description of State Aid Schemes (Step 1)” chapter of this report, oper- ation 2.1.1 aimed at supporting the research partnership between universities and research institutions and firms; operation 2.1.2 targeted universities and research institutions and aimed at attracting foreign specialists to conduct high-level R&D projects; and operation 2.3.3 aimed at stimulating enterprise innovation by financing the development of new or upgraded products. 70 | Design for Impact  4. In fact, in a study of R&D grants in Poland, Bruhn and McKenzie (2019) found that 3–4 years after receiving grants, 53 percent of funded projects had a product ready for sale. However, that product accounted for only about 1 percent of total sales at that time.  5. One stylized fact from the empirical literature on R&D evaluation is that R&D support granted to small firms tends to be more effective than that granted to large firms; see, for instance, Bronzini and Piselli (2014) and Hall, Mairesse, and Mohnen (2010).  6. For instance, such factors could include availability of bank agencies at the local level, basic interest rates stemming from macroeconomic and fiscal conditions, credit information sys- tems, and insolvency law and the way it deals with loan default.  7. For instance, the coverage ratio, guarantee assignment process, pricing, and loss-sharing mechanisms in case of the borrowing SME’s default.  8. For further discussion on regional transformation, see World Bank (2018).  9. With regard to employment impact, assuming that the scheme exacerbated only costs asso- ciated with job creation, the estimated cost per job created by the scheme ranges from €51,192 to €64,850 (for the implementation period considered in the empirical analysis, 2012–15). These numbers result from the division of total disbursement value of the scheme (€1 million) by the number of jobs created in two scenarios: (a) 15,420 generated jobs (8,280 in beneficiaries and 7,140 in nonbeneficiaries); and (b) 19,534 generated jobs (8,280 in ben- eficiaries and 11,254 in nonbeneficiaries). Despite some caveats, these numbers, even if considered as upper-bound figures, probably carry a high welfare cost when compared with the counterfactual where the employees enter unemployment, because the average wage in Romania for a 4-year period amounts to €34,144. Average annual wage per employee in Romania is €7,536 (in current values), according to the SBS dataset. For a 4-year period, the total wage cost would amount to €34,144. 10. A strong stylized fact emerging from the literature is that subsidies used to finance clos- er-to-market activities tend to be less effective and distort competition more when com- pared with subsidies tackling basic and applied research; see, for instance, Czarnitzki, Hottenrott, and Thorwarth (2011). 11. For confidentiality reasons, the details of the data-cleaning process cannot be disclosed. REFERENCES Bronzini, R., and P. Piselli. 2014. “The Impact of R&D Subsidies on Firm Innovation.” Economic Working Papers 960, Bank of Italy, Rome. Bruhn, M., and D. McKenzie. 2019. “Can Grants to Consortia Spur Innovation and Science- Industry Collaboration? Regression-Discontinuity Evidence from Poland.” World Bank Economic Review 33 (3): 690–716. doi: 10.1093/wber/lhx014. Czarnitzki, D., H. Hottenrott, and S. Thorwarth. 2011. “Industrial Research versus Development Investment: The Implications of Financial Constraints.” Cambridge Journal of Economics 35: 527–44. Hall, B., J. Mairesse, and P. Mohnen. 2010. “Measuring the Returns to R&D.” Handbook of the Economics of Innovation, edited by B. H. Hall and N. Rosenberg (Vol. 2), 1033–82. Amsterdam: Elsevier. World Bank. 2018. Rethinking Lagging Regions: Using Cohesion Policy to Deliver on the Potential of Europe’s Regions. Washington, DC: World Bank. APPENDIX A Selection Criteria for the Ministry of European Funds Scheme (Scoring Phase) TABLE A.1  Selection criteria for Ministry of European Funds scheme—Operation 2.1.1. “Research projects in partnership between universities/research and development institutions and enterprises” MAXIMUM SCORE 1. Project relevance 10 1.1 The project’s contribution to the overall development of the economic sector 2 1.2 The extent to which the proposed project will contribute to results directly 2 applicable on the market and to the increase of the company’s competitiveness 1.3 The need for research activities/services 2 1.4 Develop the company’s skills to use R&D results and activities 2 1.5 The possibility of creating new jobs within the company; contribute to the 2 promotion of sustainable development and equal opportunities Maximum score 2. Quality and maturity of the project 10 2.1. Correlation between the proposed activities, the necessary resources, and the 2 project results 2.2 Degree of preparation/maturity of the project 2 2.3 Project budget structure 2 2.4 Project implementation methodology; risk analysis 2 2.5 The capacity of the team involved in the implementation of the project 2 Maximum score 3. Sustainability and operating capacity 10 3.1 The ability of the enterprise to apply the results of the project after its completion 2 3.2 Implementing the results of the project in production 2 3.3 Existence of an experienced team to implement the project results 2 3.4 Sustainability of demand 2 3.5 Providing technical and administrative support 2 Source: Guide for Applicants for Operation 2.1.1, available at http://www.poscce.research.gov.ro/ro/node/node/nid/2354. Note: R&D = research and development.  71 72 | Design for Impact TABLE A.2  Selection criteria for Ministry of European Funds scheme—Operation 2.1.2. “High-level research and development projects involving foreign specialists” MAXIMUM SCORE 1. Project relevance 10 1.1 Scientific relevance 5 • Degree of novelty of the proposed scientific result/solution; • The contribution of the project to the global development of the field in question, in terms of research, development, and innovation; • Highlighting the stimulating effect of R&D activities for large enterprises. 1.2 The impact of the project 5 • The extent to which the proposed project will contribute to directly marketable results; • The contribution of the project to the development of the scientific competitiveness of the requesting institution; • Number of newly created jobs; • The contribution to the promotion of sustainable development and equal opportunities. Maximum score 2. Quality and maturity of the project 10 2.1. Coherence and feasibility of the project 5 • Correlation between the proposed activities, the necessary resources, and the purpose of the project; • Project budget structure. 2.2. Ability to implement the project 5 • Project implementation methodology; • The experience and managerial qualities of the specialist from abroad; • The capacity of the proposed team to implement the project; • Risk analysis. Maximum score 3. Sustainability 5 • Ability to financially support the research team created under the leadership of the international high-level specialist, after the grant will cease; • The applicant’s ability to capitalize on the results of the project after the cessation of the grant; • Providing technical and administrative support. Maximum score 4. Exchange of experience between enterprises and universities/institutions 2 • The specialist from abroad comes from a research environment different from the one specific to the host institution, respectively: - If the applicant is an enterprise, the specialist has worked for at least 1 year in a university or research institute abroad, and - If the applicant is a research organization, the specialist worked for at least 1 year in a foreign company. Source: Guide for Applicants for Operation 2.1.2, available at http://www.poscce.research.gov.ro/ro/node/node/nid/1692. Note: R&D = research and development. Appendix A | 73 TABLE A.3  Selection criteria for Ministry of European Funds scheme—Operation 2.3.3. “Promoting innovation in enterprises” ­ MAXIMUM SCORE 1. Project relevance 10 1.1 The innovative character of the project 5 • Clear and positive differentiation from competing products/processes on the market; • Ability of the applicant to innovate (to implement an innovation); • Contribution to the promotion of sustainable development and equal opportunities. 1.2. Increasing the technological level of the enterprise 5 • Improving the technological level of the enterprise in terms of product, process technology, and/or environmental technology; • Highlighting the stimulating effect of R&D activities for the enterprise; • Number of jobs created or maintained. Maximum score 2. Quality and maturity of the project 10 2.1 Economic and market feasibility 5 • Viability of project financing; • Increasing the financial stability of the enterprise. 2.2. Coherence and maturity of the project 5 • Correlation between objectives, activities, deadlines, and proposed costs; • Identifying the risks of project implementation and measures to eliminate them; • The degree of maturity of the project. Maximum score 3. Sustainability and operating capacity 10 3.1 Sustainability 5 • The capacity of the enterprise to continue the proposed activities after the cessation of the non-reimbursable financing; • The level of cooperation with customers, suppliers, and other enterprises in the respective field; • Level of market expansion. 3.2 The capacity of the enterprise to implement the project 5 • The existence of an experienced team; • The existence of an endowment that would allow the start of the project; • The existence of a modern management. Source: Guide for Applicants for Operation 2.3.3, available at http://www.poscce.research.gov.ro/ro/node/node/nid/2393. Note: R&D = research and development. APPENDIX B State Aid Intensity for the Ministry of European Funds Scheme TABLE B.1  State aid intensity (under Ministry of European Funds scheme) by R&D expenditure categories and firm size SHARE OF R&D INVESTMENT IN AID (%), BY FIRM SIZEa ELIGIBLE TYPES OF R&D ACTIVITIES LARGE MEDIUM SMALL Industrial research 50 60 70 Experimental research 25 35 45 Feasibility studies (for experimental 40 50 50 research) Acquiring and verifying industrial property — 60 70 rights (for industrial research) Acquiring and verifying industrial property — 35 45 rights (for experimental research) Consulting services for innovation — A maximum of €0.2 million for 3 yearsb Employing high-skilled specialists — 50%c Specific activities — — 100d Source: Order no.3388/2008 establishing the MEF scheme. Note: — = not applicable; R&D = research and development. a. Small enterprises are defined as those with 10–49 persons employed, medium-size enterprises with 50–249 persons employed, large enterprises with 250 or more persons employed. b. If the service provider does not have a national or European certification, the aid will not cover more than 75 percent of the eligible costs. c. For a period of maximum 3 years. d. For a maximum of €1 million.  75 APPENDIX C List of Variables in the Business Registry Dataset TABLE C.1  List of variables in the Business Registry database VARIABLE DESCRIPTION nr_crt Criterion number—indexing variable cui Registration number of the company (CUI) denumire Name of the company anul_infiintarii Founding year caen NACE code denumire_caen Sector description judet County (siruta code level 1) imob_necorp Intangible assets (RON) imob_corp Tangible assets (RON) imob_fin Financial assets act_imob Fixed assets—Total inv_ts Short-term investments (RON) act_circ Current assets—Total capital Assets (RON) ca_neta Net turnover (RON) ca_pv Turnover from sold production (RON) ca_vm Turnover from goods purchased for resale (RON) ca_rca Turnover from granted trade discounts (RON) Turnover from interest income of entities removed from the general ca_cleasing register and that are still running leasing contracts (RON) ca_se Operating subsidies (RON) (Continued)  77 78 | Design for Impact TABLE C.1, continued VARIABLE DESCRIPTION ven_pimncc Firm production for its own and capitalized purposes (RON)a ven_expl Operating revenues—Total (RON) ch_mpmc Raw materials and materials cost alte_chmat Other material costs alte_chext Other operating costs ch_marf Costs with finished goods ch_pers Cost of labor ch_expl Operating cost—Total (RON) prof_expl Operating profit (RON) pierd_expl Operating loss (RON) nr_med_sal Average number of employees nr_ef_sal Actual number of employees at the end of the period December 31 prod_spcap Firm production for its own and capitalized purposes (RON)a Source: Ministry of Public Finance. a. For firm production for its own and capitalized purposes (return on net worth = RON), two labels are used: prod_spcap for 2011–14 and ven_pimncc for 2015–17. NACE = Nomenclature of Economic Activities; RON = Romanian Lei. APPENDIX D List of List of variables in the Structural Business Survey dataset TABLE D.1  List of variables in the Structural Business Survey dataset” VARIABLE DESCRIPTION an_refer Year of the statistical data cod Criterion number—indexing variable incep Year of incorporation of the firm judet County (siruta code level 1) localitate Locality siruta Siruta code level 2 fp Form of ownership of the firm caen_rev2 Main activity—sector (NACE) of the firm nms Average number of employees sala Salaries, allowances, and vouchers ca Turnover to be distributed vpf Income from the sale of finished products vls Income from the sale of the works expd Export value sits Total stocks at the beginning of the year sfts Total stocks at the end of the year invr42 Net investments impinv Net investments from imports invhard Net investments in hard products (hardware) invsoft Net investment in software products val_added Value added chmfa Cost with raw materials achma Other material costs (Continued)  79 80 | Design for Impact TABLE D.1, continued VARIABLE DESCRIPTION enapa Other external costs insi Intangible fixed assets, initial balance intsf Intangible fixed assets, final balance ccdsi Set-up and research expenses, initial balance ccssf Set-up and research expenses, final balance Source: National Institute of Statistics. Note: NACE = Nomenclature of Economic Activities. APPENDIX E Methodology Applied to Estimate Markups and Revenue-Based Total Factor Productivity The production approach will be applied to estimate firm-level markups as a proxy for pricing power in Romania. The implementation of this method follows two steps. The first is to define and estimate a production function to reflect the technology firms have access to. For estimation purposes, a firm i that belongs to sector j (defined at the 2-digit Nomenclature of Economic Activities [NACE] level) production function at year t is represented by a flexible translog technology as follows: yjit = a  1,jmjit + a2,jljit + a3,jkjit + a4,jm2jit + a5,jl2jit + a6,jk2jit + a7,j mjitljit + a8,jmjitkjit + a9,jljitkjit + a0,j + ωjit + ejit, (E.1) where all the variables are in natural logarithms and yjit is the firm’s annual total revenue from operations, mjit is the total expenditure in intermediate materials, kjit is the capital stock (sum of tangible and intangible assets), and ljit is the total number of hours worked per year (the natural logarithm of the product of the number of employees times the number of months worked times the number of hours worked per month). The revenue-based measure of total factor productivity (TFPR) is identified as any change in firms’ revenue not caused by variations in the inputs of production. More specifically, in (E.1), TFPR (in natural logarithm) is identified by the following: lnTFPRjit = a0,j + wjit + ejit . (E.2) The term ωjit is known by the firm, but not by the econometrician, which generates a problem of simultaneity between the inputs and productivity. The last term (εjit) is assumed to be a completely random productivity shock not antic- ipated by the firms when making their input choices. The Ackerberg, Caves, and Frazer (2015) approach is followed to solve the endogeneity problem and the anticipated productivity shock is divided in two terms: wjit = g (wjit-1 ) + xjit, (E.3) where the first term, g(w jit−1), is the inertia of the productivity process, that is, highly productive firms at t−1 will also have a high productivity level at t (present).1 This is the term that firms know when making their input choices. The second term, ξjit, is the innovation of the productivity process and is the key  81 82 | Design for Impact to the estimation process. It can be interpreted as any change in the level of productivity that is completely new at time t and uncorrelated with past productivity. Since the innovation is uncorrelated with past productivity, it is also uncor- related with the optimal choices of the variable inputs—that is, materials and labor—made at moment t−1. Moreover, since capital is a fixed input and it takes time to change its level, we can also assume that the innovation of productivity is uncorrelated with the capital input choices made at moment t. This set of assumptions allows us to form a set of moment conditions, as follows: E(xitlit-1 ) = E(xitmit-1 )=E(xitKit ) = 0, (E.4) which states that we can use the lack of correlation between lagged flexible inputs and contemporaneous capital, on the one hand, and productivity innova- tion on the other to identify (estimate) the parameters of the production function in equation (E.1).2 To account for differences in production technologies across sectors, the TFPR estimation, as discussed above, allows for heterogeneous sector-specific (NACE 2-digit) production functions.3 Using the estimations of the production function, it is possible to compute the output elasticity of the variable input (here assumed as materials) in sector j where firm i operates, as follows:  = 1 β M, j Nj ∑ (α  1, j  4, j m + α + 2α jit  7, jl + α jit  8, j m ) jit (E.5) ∀i ∈j In this regard, it should be noted that all firms in the same sector j are assigned a common elasticity. The interpretation is that firms in the same sector have access to the same technology, but they differ both in terms of TFPRit and the level of variable input (material) optimally chosen. Once the output elasticity of the variable input is estimated, the second operational step is to use information on expenditures on variable inputs to compute a firm’s markup. Specifically, a firm’s markup can be directly computed as follows:  β M, j µ jit = Mshare jit (E.6) where β was estimated in the previous step and Mshare is the ratio of cost of M, j jit variable input (materials) to corrected revenue is defined as P M M jit (E.7) Mshare jit = ,  Y jit    exp( ε jit )  where PMMjit is total cost of raw materials, Yjit is total revenue, and εjit is the unanticipated productivity shock, obtained as a residual in the estimation of the production function (see equation E.2). Appendix E | 83 NOTES 1. Following the terminology used in the literature, the current analysis assumes that produc- tivity follows a first-order Markov process in the sense that it depends only on its own past values. The functional form is approximated by a polynomial of degree 3 on the first lag of TFPR. 2. A moment condition is a statistical equality that holds in the population under study. 3. Observations with revenue, employment, capital (tangible assets and intangibles), and cost of raw materials with value zero or value lower than zero are excluded from the TFPR estimation. REFERENCE Ackerberg, D. A., K. Caves, and G. Frazer. 2015. “Identification Properties of Recent Production Function Estimators.” Econometrica 83: 2411–51. APPENDIX F Romanian Counter Guarantee Scheme: Firms Included in the Analysis (Auxiliary Tables) TABLE F.1  Distribution of the Romanian Counter-Guarantee Fund scheme beneficiary firms by their 2-digit nomenclature of economic activities sector SECTOR # CLASSIFICATION 2-DIGIT NACE CODE FIRMS A. Wholesale and 45 Wholesale and retail trade and repair of motor 50 retail trade vehicles and motorcycles 46 Wholesale trade, except motor vehicles and 100 motorcycles 47 Retail trade, except motor vehicles and motorcy- 90 cles B. Other services 55 Accommodation 17 56 Food and beverage service activities 55 90 Creative, arts, and entertainment activities 1 91 Libraries, archives, museums, and other cultural 2 activities 93 Sports activities and amusement and recreation 16 activities 94 Activities of membership organizations 1 95 Repair of computers and personal and household 9 goods 96 Other personal service activities 33 C. Professional, 58 Publishing activities 5 technical, and 59 Motion picture, video, and television program 6 administrative production, … activities 66 Activities auxiliary to financial services and 3 insurance activities 69 Legal and accounting activities 4 70 Activities of head offices; management consul- 23 tancy activities 71 Architectural and engineering activities; technical 22 testing and analysis (continued)  85 86 | Design for Impact TABLE F.1, continued SECTOR # CLASSIFICATION 2-DIGIT NACE CODE FIRMS 72 Scientific research and development 4 73 Advertising and market research 17 74 Other professional, scientific, and technical 8 activities 75 Veterinary activities 2 77 Rental and leasing activities 6 78 Employment activities 3 80 Security and investigation activities 3 81 Services to buildings and landscape activities 14 82 Office administrative, office support, and other 9 business support activities D. Construction, 35 Electricity, gas, steam, and air conditioning 3 transport, IT, supply electricity 37 Sewerage 1 38 Waste collection, treatment, and disposal 6 activities; materials recovery 41 Construction of buildings 22 42 Civil engineering 2 43 Specialized construction activities 30 49 Land transport and transport via pipelines 34 52 Warehousing and support activities for transpor- 2 tation 53 Postal and courier activities 1 61 Telecommunications 3 62 Computer programming, consultancy, and 11 related activities 63 Information service activities 4 E. Manufacturing 10 Manufacture of food products 27 13 Manufacture of textiles 2 14 Manufacture of wearing apparel 13 15 Manufacture of leather and related products 2 16 Manufacture of wood and of products of wood 13 and cork, except furniture; … 17 Manufacture of paper and paper products 4 18 Printing and reproduction of recorded media 10 20 Manufacture of chemicals and chemical products 2 21 Manufacture of basic pharmaceutical products 1 and pharmaceutical preparations 22 Manufacture of rubber and plastic products 12 23 Manufacture of other non-metallic mineral 8 products 25 Manufacture of fabricated metal products, except 6 machinery and equipment (continued) Appendix F | 87 TABLE F.1, continued SECTOR # CLASSIFICATION 2-DIGIT NACE CODE FIRMS 26 Manufacture of computer, electronic, and optical 1 products 27 Manufacture of electrical equipment 3 28 Manufacture of machinery and equipment n.e.c. 1 31 Manufacture of furniture 8 32 Other manufacturing 6 F. Health, education 85 Education 12 86 Human health activities 28 87 Residential care activities 1 88 Social work activities without accommodation 2 784 Source information: Codul CAEN- Lista domeniilor de activitate, accessed from https://www.listafirme​ .ro/specificatii/caen.asp. Note: NACE = Nomenclature of Economic Activities. TABLE F.2  First and last year’s Romanian Counter-Guarantee Fund scheme support received YEAR WHEN SUPPORT BEGAN 2012 2013 2014 2015 2016 2017 TOTAL 2012 155 56 27 32 16 26 312 2013 0 63 44 28 52 195 382 2014 0 0 46 1 3 38 88 2015 0 0 0 0 1 0 1 2016 0 0 0 0 0 1 1 Total 155 119 117 61 72 260 784 Source: World Bank elaboration. APPENDIX G Identification Strategy of the Romanian Counter Guarantee Scheme Evaluation: Robustness Checks To test if the control groups are valid comparison groups, pre-scheme outcome measures for the firms in the treatment and control groups were contrasted. The extent to which the control groups are valid comparison groups for each treatment group depends on how the selection into the RCG scheme took place. In the case of the direct effect, equation 4.1 (presented in the main text, in the “Identification Strategy” subsection of chapter 4) would overestimate the impact of the RCG scheme if beneficiary firms expected to grow and received the scheme right before this growth was to happen. While the determinants of firm application to the RCG scheme are unknown, it is possible to compare pre- scheme outcomes for the firms in the treatment and control groups. One data limitation is that the BR starts in 2011, which leaves only 1 year before the RCG scheme began to compare pre-scheme outcomes. This limitation was overcome in two ways. First, data from 2011 were used to compare the pre- scheme average outcomes of beneficiary firms and their control group. Table G.1 displays the summary statistics of the sample of firms used to measure the direct effects of the scheme (that is, beneficiary firms and their control group). Columns 1 and 2 present the number of observations and the average and standard errors of each outcome for all RCG beneficiaries. Columns 3 and 4 present the same summary statistics for the baseline treatment group of the direct effects of the scheme: beneficiaries that opened between 2009 and 2011, and thus were already in operation a year prior to the scheme’s first disbursement. Columns 5 and 6 display the summary statistics for firms in the control group: firms created between 2009 and 2011, that are from siruta territorial units (at level 2) where the RCG scheme was not rolled out and that, based on their age and sector, would have been eligible under the scheme. Column 2 shows that among all beneficia- ries, 23.8 percent were firms that opened in 2011, 17.8 percent were created in 2010, and only 6.7 percent started operations in 2009. The remaining 51.7 ­percent of beneficiaries were firms that opened in 2012 or 2013. As column 4 shows, of the sample of beneficiaries already opened by 2011, 49.3 percent of them were created in 2011, whereas 36.8 percent and 13.9 percent of them started in 2010 and 2009, respectively. While entry rates of firms in the control group show a similar pattern, a higher fraction of firms in the control group was created in 2009. In terms of employment, beneficiary firms in the treatment group employed on average four more workers than the average firm in the control  89 90 | Design for Impact TABLE G.1  Pre-scheme summary statistics for Romanian Counter-Guarantee Fund scheme beneficiaries and control group ­ (1) (2) (3) (4) (5) (6) BENEFICIARIES IN NONBENEFICIARIES IN THE TREATMENT GROUP THE CONTROL GROUP ALL BENEFICIARIES (CREATED BEFORE 2012) (CREATED BEFORE 2012) VARIABLE/RESULT INDICATORS N MEAN [SE] N MEAN [SE] N MEAN [SE] Probability of opening in 2011 747 0.238 361 0.493 23,883 0.447 [0.016] [0.026] [0.003]* Probability of opening in 2010 747 0.178 361 0.368 23,883 0.276 [0.014] [0.025] [0.003]*** Probability of opening in 2009 747 0.067 361 0.139 23,883 0.277 [0.009] [0.018] [0.003]*** Number of workers 269 8.628 269 8.628 15,240 4.578 [1.371] [1.371] [0.565] Number of workers (IHS) 269 1.972 269 1.972 15,240 1.453 [0.070] [0.070] [0.008]* Turnover 314 1,057,445 314 1,057,445 21,589 236,522 [94,839] [94,839] [4,986]*** Turnover (IHS) 314 11.784 332 11.784 21,589 9.773 [0.270] [0.270] [0.032] TFPR (ln) 215 7.051 215 7.051 8,357 7.319 [0.055] [0.055] [0.009]* Markup (ln) 209 1.017 209 1.017 7,909 1.243 [0.084] [0.084] [0.015]* Market share (county and 4-digit NACE 290 0.008 290 0.008 17,836 0.004 level) [0.002] [0.002] [0.000]*** Firm-level measure of allocative efficiency 209 −0.014 209 −0.014 7,918 −0.003 (county and 4-digit NACE level) [0.011] [0.011] [0.003]** Source: World Bank elaboration. Note: Columns 1 and 2 display summary statistics of all beneficiaries of the RCG scheme. Columns 3 and 4 report summary statistics of beneficiaries of the RCG scheme that began operations before 2012. Columns 5 and 6 display summary statistics of firms eligible for the RCG scheme based on their sector and date of creation, which are located in siruta territorial units (level 2) with no beneficiaries of the RCG scheme. Data are for 201 1. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. SE = standard error; TFPR = revenue-based measure of total factor productivity; NACE = Nomenclature of Economic Activities. ***, **, and * indicate mean differences between columns 4 and 6 at the 1, 5, and 10 percent significance levels. group; but due to large standard errors, this difference is not statistically signifi- cant. The difference in employment becomes significant once the inverse hyper- bolic sine (IHS) is applied,1 confirming that beneficiaries were larger in size than eligible firms in control sirutas—that is, siruta territorial units (at level 2) where the scheme was not rolled out. The average turnover of beneficiary and control group firms is statistically similar; but, as indicated by the TFPR, beneficiaries were on average less productive than their control firms before the scheme started. In terms of market-levels outcomes, beneficiary firms had, on average, lower markups compared with firms in their control group before the scheme started. The average measures that reflect competition dynamics at the county and 4-digit NACE sector levels show very low market shares for beneficiaries as well as for firms in the control group, suggesting these firms were small players in the market.2 Appendix G | 91 Second, the analysis compares the pre-scheme average outcomes of nonben- eficiary firms in siruta territorial units (at level 2), with and without beneficia- ries, that were ineligible under the scheme. Table G.2 shows that noneligible firms in siruta territorial units (at level 2) with and without beneficiaries under the scheme were, on average, similar across all outcomes. Even though these differences are statistically significant, their economic size is small.3 A common check for the validity of difference-in-differences estimates is to test if the outcomes of the treated and control groups followed a parallel trend in the pre-scheme period. Since the BR data begin in 2011 and the scheme began being implemented in 2012, this test is not feasible for most outcomes. One exception is the probability of a firm opening, which is updated to years prior to 2011 using BR information on the year of opening. Thus, a balanced panel from 2008 to 2011 is constructed; this panel updates the probability of firms opening over time. Doing this allows testing if the rate of firm creation between treatment TABLE G.2  Pre-scheme summary statistics for noneligible firms in siruta territorial units (level 2) with and without Romanian Counter-Guarantee Fund beneficiaries (1) (2) (3) (4) NONELIGIBLE FIRMS IN SIRUTA NONELIGIBLE FIRMS IN TERRITORIAL UNITS (LEVEL 2) WITH TERRITORIAL UNITS (LEVEL 2) RCG SCHEME WITHOUT RCG SCHEME VARIABLE/RESULT INDICATORS N MEAN [SE] N MEAN [SE] Probability of opening 548,900 0.015 228,945 0.015 [0.000] [0.000]*** Number of workers 254,108 14.479 96,389 11.565 [3.565] [3.227] Number of workers (IHS) 254,108 1.663 96,389 1.717 [0.002] [0.004]*** Turnover 383,735 384,212 145,590 375,073 [1,741] [2,660]** Turnover (IHS) 383,735 8.304 145,590 8.637 [0.010] [0.016]** Probability of closing 548,900 0.214 228,945 0.187 [0.001] [0.001]*** TFPR (ln) 162,079 7.126 64,621 7.136 [0.002] [0.003]*** Markup (ln) 158,221 1.218 63,019 1.075 [0.003] [0.005]*** Market share (4-digit NACE sector and county levels) 256,243 0.006 99,990 0.011 [0.000] [0.000]*** Firm-level measure of allocative efficiency (4-digit 158,507 −0.007 63,099 −0.007 NACE sector and county levels) [0.001] [0.001]*** Source: World Bank elaboration. Note: The sample is restricted to firms not eligible for the RCG scheme based on their sector and date of creation. The first two columns display summary statistics of firms in siruta territorial units (level 2) with beneficiaries of the RCG scheme. The last two columns display summary statistics of firms in siruta territorial units (level 2) with no beneficiaries of the RCG scheme. Data are for 2011. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Firm market share and firm-level measure of allocative efficiency are defined at the 4-digit NACE and county levels. SE = standard error. ***, **, and * indicate mean is different from beneficiaries at the 1, 5, and 10 percent significance levels. TFPR = revenue-based measure of total factor productivity; NACE = Nomenclature of Economic Activities. 92 | Design for Impact and control siruta territorial units (level 2) followed a parallel trend during the pre-scheme period. Equation G.1 displays the regression used to test for the par- allel pre-scheme trends assumption. entry ratest = αs + βt + g0 Trendt +g1 Treatments * Trendt + est, (G.1) where entry ratest corresponds to the fraction of new firms in siruta territo- rial units (level 2) s that are created in year t, and αs is a fixed effect at the siruta territorial units (level 2). Trendt is a linear time trend, and Treatments is equal to 1 for treated siruta geographic areas and 0 otherwise. εst is an error term, clustered at the siruta territorial units (level 2) level. The ­coefficient g1 tests for the parallel trend assumption in rate of creation of firms in the treatment and control siruta territorial units (level 2) in the pre-scheme period. Results of this test show that entry rates between these groups followed a similar and parallel path for years 2008–12. Table G.3 displays the result of the test summarized in equation G.1. While the tests find a positive and statistically significant difference in trends between treatment and control siruta territorial units (level 2) in the pre-scheme period, the economic magnitude of this difference is close to zero. Further evidence on the evo- lution of trends in entry rates between treatment and control siruta ­territorial units (level 2) prior to the scheme is illustrated in figure G.1. The figure corroborates that entry rates between these groups followed a ­similar and parallel path for years 2008–12, then increased their distance. TABLE G.3  Test of parallel trends in the pre-scheme period (1) (2) (3) ENTRY RATES IN TREATED VERSUS CONTROL SIRUTA TERRITORIAL UNITS (LEVEL 2) (2008–11) Trend −0.004*** −0.004*** −0.004*** p-value 0.000 0.000 0.000 Treatment −0.006** −0.006** p-value 0.010 0.010 Treatment*Trend 0.002** 0.002** 0.002** p-value 0.018 0.018 0.040 Constant 0.060*** 0.068*** 0.068*** p-value 0.000 0.000 0.000 Number of observations 12,680 12,680 12,680 R-squared 0.009 0.046 0.275 Siruta FE No No Yes Year FE No Yes Yes Source: World Bank elaboration. Note: The table reports the g2 coefficient of the regression probability of openingst = a + g0*Trendt + g1*Treatments+ g2*Treatments*Trendt + est, where a corresponds to a constant term; entry ratest is the fraction of new firms in siruta territorial unit (level 2) created at year t. Trendt is a linear time trend for the pre-scheme years (2008–1 1). Treatments is equal to 1 for treated siruta territorial units (level 2) where the scheme was rolled out and 0 otherwise. est, is an error term clustered at the siruta territorial unit (level 2). FE = fixed effects. ***, **, and * indicate 1, 5, and 10 percent significance levels. Appendix G | 93 FIGURE G.1 Entry rates of treatment and control siruta territorial units (level 2) over time, 2008–17 0.08 0.06 Firm entry rate 0.04 0.02 0 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Sirutas with RCG scheme Sirutas without RCG scheme Source: World Bank elaboration. Note: Treatment sirutas are defined as the siruta territorial units (level 2) that had beneficiary firms at any point in time, while control sirutas consist of the remaining siruta territorial units (level 2) that had no beneficiaries of the RCG scheme. RCG = Romanian Counter-Guarantee Fund. NOTES 1. The use of IHS is common in the empirical literature; its main advantage is to reduce the influence of outliers (similar to log-function) and allow changes to be interpreted as per- centage changes. 2. A more granular measure of market share, defined at the 4-digit NACE sector and siruta territorial units (at level 2), shows that firms in the control group had higher market shares than treated firms. 3. Results change if market share is measured at a more granular level: at the 4-digit NACE sector and siruta territorial unit (at level 2). In this case, the average local market share of treated firms in 2011 was 6.1 percent, compared with 35.1 percent of firms in the control group. The difference in local market shares stems from differences in the size of the mar- kets of control and treated firms and not from differences in their size and productivity. APPENDIX H Impact of the Romanian Counter-Guarantee Fund Scheme: Robustness Checks (Auxiliary Tables) TABLE H.1  Impact of the Romanian Counter-Guarantee Fund scheme on beneficiary firms over time (1) (2) (3) (5) NUMBER OF WORKERS (IHS) TURNOVER (IHS) PROBABILITY OF CLOSING TFPR (LN) Treatment*2012 0.190** 0.410** −0.111*** −0.003 p-value 0.001 0.041 0.000 0.530 Treatment*2013 0.472*** 1.776*** −0.222*** −0.020** p-value 0.000 0.000 0.000 0.048 Treatment*2014 0.530*** 2.015*** −0.259*** −0.026*** p-value 0.000 0.000 0.000 0.004 Treatment*2015 0.537*** 2.029*** −0.263*** −0.014 p-value 0.000 0.000 0.000 0.135 Treatment*2016 0.505*** 1.791*** −0.254*** −0.018* p-value 0.000 0.000 0.000 0.057 Treatment*2017 0.466*** 1.216*** −0.263*** −0.021* p-value 0.000 0.003 0.000 0.057 Number of observations 96,102 128,933 169,708 51,251 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme from 2012 to 2017 on beneficiary firms for the outcomes displayed in each column. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate 1, 5, and 10 percent significance levels.  95 96 | Design for Impact TABLE H.2  Impact of the Romanian Counter-Guarantee Fund scheme on beneficiary firms (1) (2) (3) (4) NUMBER OF WORKERS (IHS) TURNOVER (IHS) FOR FIRMS OPENED FOR ALL FIRMS FOR FIRMS OPENED FOR ALL FIRMS Average treatment effect 0.430*** 0.557*** 1.478*** 2.196*** p-value 0.000 0.000 0.000 0.000 Number of observations 96,102 143,207 128,933 163,971 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme from 2012 to 2017 on beneficiary firms for the outcomes displayed in each column. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). ***, **, and * indicate 1, 5, and 10 percent significance levels. TABLE H.3  Impact of the Romanian Counter-Guarantee Fund scheme on beneficiary firms (using beneficiaries opened any year) (1) (2) (3) (4) (5) (6) NUMBER OF WORKERS (IHS) TURNOVER (IHS) FIRMS OPENED ALL FIRMS FIRMS OPENED ALL FIRMS PROBABILITY OF CLOSING TFPR (LN) Average treatment effect 0.425*** 0.610*** 1.467*** 2.469*** −0.201*** −0.016*** p-value 0.000 0.000 0.000 0.000 0.000 0.007 Number of observations 158,378 262,327 214,771 296,756 312,144 79,210 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme from 2012 to 2017 on all beneficiary firms that opened from 2009 to 2013 for the outcomes displayed in each column. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate 1, 5, and 10 percent significance levels. TABLE H.4  Pre-scheme trends in outcomes of beneficiary firms (that joined the Romanian Counter-Guarantee Fund scheme in 2013 or later) and their control group (1) (2) (3) (4) (5) (6) NUMBER OF WORKERS (IHS) TURNOVER (IHS) TFPR (LN) Trend −0.016*** 0.354*** 0.001 p-value 0.006 0.000 0.792 Benef −0.736 −2.420 0.247 p-value 0.201 0.347 0.667 Trend*Benef 0.218 0.228 0.315 0.831 −0.095 0.013 p-value 0.116 0.213 0.593 0.258 0.441 0.311 Constant 1.553*** 1.267*** 8.555*** 8.397*** 7.311*** 7.312*** p-value 0.000 0.000 0.000 0.000 0.000 0.000 Number of observations 24,261 24,261 32,114 32,114 13,845 13,845 R-squared 0.000 0.963 0.002 0.926 0.000 0.999 Firm FE No Yes No Yes No Yes Year FE No Yes No Yes No Yes Source: World Bank elaboration. Note: The table reports the g2 coefficient of the regression yit = a + Symbol0*Trendt + g1*Treatmenti+ g2*Treatmenti*Trendt + eit, where a corresponds to a constant term and yit corresponds to the outcome displayed in each column. Trendt is a linear time trend for the pre-scheme years (2011–12). Treatmenti is equal to 1 for beneficiary firms opened by 201 1 that received their first aid in 2013 or afterward and is equal to zero for eligible firms opened by 201 1 located in siruta territorial units (level 2) where the RCG scheme was not rolled out. eit is an error term clustered at the siruta territorial unit (level 2). IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. FE = fixed effects. ***, **, and * indicate 1, 5, and 10 percent significance levels. TFPR = revenue-based measure of total factor productivity. Appendix H | 97 TABLE H.5  Impact of the Romanian Counter-Guarantee Fund (RCG) scheme on beneficiary firms that joined the RCG scheme in 2013 or later (1) (2) (3) (4) (5) (6) NUMBER OF WORKERS (IHS) TURNOVER (IHS) FOR FIRMS FOR ALL FOR FIRMS PROBABILITY OF OPENED FIRMS OPENED FOR ALL FIRMS CLOSING TFPR (LN) Average treatment effect 0.755*** 0.831*** 3.183*** 3.500*** −0.111*** −0.027 p-value 0.000 0.000 0.000 0.000 0.000 0.361 Number of observations 84,924 101,354 108,120 119,113 124,236 47,234 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme on beneficiary firms that received their first aid in 2013 or afterward for the outcomes displayed in each column. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate 1, 5, and 10 percent significance levels. TABLE H.6  Spillover effects of the Romanian Counter-Guarantee Fund scheme over time (1) (2) (3) (4) PROBABILITY OF OPENING TURNOVER (IHS) NUMBER OF WORKERS (IHS) FIRMS OPENED ALL FIRMS Treatment*2009 0.001 p-value 0.749 Treatment*2010 0.002 p-value 0.614 Treatment*2011 0.003 p-value 0.496 Treatment*2012 0.002 −0.011 −0.009 0.00901 p-value 0.572 0.824 0.581 0.589 Treatment*2013 0.003 0.026 −0.009 0.00121 p-value 0.459 0.251 0.177 0.945 Treatment*2014 0.002 0.015 −0.021** −0.01635 p-value 0.717 0.556 0.010 0.507 Treatment*2015 −0.014 −0.029 −0.044*** −0.03568 p-value 0.283 0.442 0.000 0.244 Treatment*2016 −0.021 −0.074 −0.055*** −0.06581 p-value 0.199 0.276 0.000 0.15 Treatment*2017 −0.022 −0.137 −0.064*** −0.09954 p-value 0.231 0.214 0.000 0.121 Number of observations 7,772,500 3,482,749 2,449,694 4,914,529 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates from 2009 to 2017 of the RCG scheme on firms not eligible for the scheme that are located in siruta territorial units (level 2) with beneficiaries. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). ***, **, and * indicate 1, 5, and 10 percent significance levels. 98 | Design for Impact TABLE H.7  Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiary firms (1) (2) (3) (4) NUMBER OF WORKERS (IHS) TURNOVER (IHS) FOR FIRMS OPENED FOR ALL FIRMS FOR FIRMS OPENED FOR ALL FIRMS Average treatment effect −0.027*** −0.033 −0.025 −0.206 p-value 0.008 0.303 0.551 0.424 Number of observations 2,449,694 4,914,529 3,482,749 5,309,964 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme on firms not eligible for the scheme that are located in siruta territorial units (level 2) with beneficiaries. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). ***, **, and * indicate 1, 5, and 10 percent significance levels. TABLE H.8  Spillover effects of the Romanian Counter-Guarantee Fund scheme on nonbeneficiary firms by employment rate of region (1) (2) (3) NUMBER OF WORKERS (IHS) PROBABILITY OF OPENING FOR FIRMS OPENED FOR ALL FIRMS Treatment*Post −0.001 −0.019*** 0.002 p-value 0.545 0.006 0.881 Treatment*Post*Employment rate −0.049*** −0.044*** −0.209*** p-value 0.000 0.004 0.000 Number of observations 7,772,500 2,449,694 4,914,529 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme on firms not eligible for the scheme that are located in siruta territorial units (level 2) with beneficiaries. Treatment*Post*Employment rate consists of the triple interaction of the treatment and post dummies with a dummy variable that equals 1 for regions with employment rates above the 75th percentile. All regressions include firm and year fixed effects. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Standard errors are clustered at the siruta territorial unit (level 2). ***, **, and * indicate 1, 5, and 10 percent significance levels. TABLE H.9  Impact of the Romanian Counter-Guarantee Fund scheme on beneficiaries’ market share using an alternative measure of market share (1) (2) FIRM MARKET SHARE AT 4-DIGIT NACE AND COUNTY FIRM MARKET SHARE AT 4-DIGIT NACE AND SIRUTA LEVELS TERRITORIAL UNIT (LEVEL 2) Average treatment effect 0.001 0.027*** p-value 0.515 0.008 Number of observations 99,283 117,502 Source: World Bank elaboration. Note: The table reports the difference-in-differences impact estimates of the RCG scheme on the market share of beneficiary firms. All regressions include firm and year fixed effects. Standard errors are clustered at the siruta territorial unit (level 2). ***, **, and * indicate 1, 5, and 10 percent significance levels. NACE = Nomenclature of Economic Activities. APPENDIX I Ministry of Public Finance Scheme: Firms Included in the Analysis (Auxiliary Tables) The comparison of pre-scheme summary statistics shows that beneficiaries are different from nonbeneficiaries in terms of sector, region, and firm characteris- tics. Table I.1 shows pre-scheme summary statistics for Ministry of Public Finance (MPF) scheme beneficiaries and nonbeneficiaries. The 17 beneficiaries not included in the analysis are on average younger than the 17 beneficiaries included in the analysis (which is one reason why 15 of them are not in the bal- anced panel used in the analysis). Not-included beneficiaries are also smaller than included beneficiaries in terms of employees, turnover, and market share. Not-included beneficiaries have, however, similar levels of investment, produc- tivity (TFPR), markup, and firm-level measures of allocative efficiency as included beneficiaries. Nonbeneficiaries are different from beneficiaries in almost all the variables given in table I.1: on average, nonbeneficiaries are much smaller, invest less, and are also less productive than beneficiaries. Overall—in terms of sector, region, and firm characteristics—beneficiaries are different from nonbeneficiaries. MPF scheme beneficiaries are thus not a representative sam- ple of the firms in Romania, as is to be expected from a scheme that targets firms that plan to undertake large investments and/or to generate many jobs. The 17 beneficiaries included in the analysis look like the 17 beneficiaries not included in the analysis, with the main difference being that not-included beneficiaries are younger and slightly smaller. It is thus not clear to what extent the results analyzed here can be extrapolated to the 17 beneficiaries not included in the analysis.  99 100 | Design for Impact TABLE I.1  Pre-scheme summary statistics for Ministry of Public Finance scheme beneficiaries and ­nonbeneficiaries 17 BENEFICIARIES IN 17 BENEFICIARIES NOT IN THE ANALYSIS THE ANALYSIS NONBENEFICIARIES VARIABLE/RESULT INDICATORS N MEAN/SE N MEAN/SE N MEAN/SE Year of incorporation 17 1999 17 2010*** 46,031 2001 [1.170] [1.196] [0.029] Number of workers 17 1,247 17 992 46,031 59*** [367.914] [770.646] [1.453] Number of workers (IHS) 17 7.072 17 5.647** 46,031 3.466*** [0.338] [0.429] [0.008] Investment (IHS) 17 17.403 17 17.684 46,031 7.733*** [0.639] [0.551] [0.029] Turnover (IHS) 17 19.819 17 16.772** 46,031 15.487*** [0.381] [1.242] [0.010] TFPR (ln) 17 9.051 17 8.801 38,627 7.751*** [0.249] [0.318] [0.008] Markup (ln) 17 0.321 16 0.200 38,601 1.181** [0.155] [0.131] [0.007] Market share 17 0.026 16 0.007* 45,995 0.001*** (4-digit NACE) [0.009] [0.003] [0.000] Firm-level measure of allocative efficiency 17 −0.142 16 −0.061 38,607 0.012 (4-digit NACE) [0.059] [0.046] [0.003] Source: World Bank elaboration. Note: For beneficiaries, data are for the year before they first received funds from the MPF scheme. Some of the beneficiaries not in the analysis do not have SBS data in the year before they started receiving funds from the MPF scheme. In these cases, either the year when they first received funds or the year after are used, depending on availability. Data for other nonbeneficiaries are for 2012. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Firm market share and firm-level measure of allocative efficiency are defined at the 4-digit NACE level (nationwide). SE = standard error; MPF = Ministry of Public Finance; NACE = Nomenclature of Economic Activities; TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate mean is different from beneficiaries included in the analysis at the 1, 5, and 10 percent significance levels. APPENDIX J Identification Strategy of the Ministry of Public Finance Scheme Evaluation: Robustness Checks To test whether the control groups are valid comparison groups for the analysis, pre-scheme outcome measures for firms in the treatment and con- trol groups were contrasted. For the control groups used in the assessment of direct and competition impacts, the comparison shows that the matched control group looks more like the beneficiaries than the rejected applicants do. Table J.1 shows the pre-scheme averages for the variables used in the analysis, separately for beneficiaries, the rejected applicant control group, and the matched control group. In the year before first receiving funds from the MPF scheme, beneficiaries had 1,247 employees on average (median 630 and average IHS employment 7.1). Firms in the rejected applicants con- trol group had only 581 employees on average. Beneficiaries and rejected applicants are also statistically significantly different in several other dimensions: investment, turnover, and market share. The average value of net investment was lei 60.1 million (median 3.35 million and average IHS investment 17.4), and the average value of turnover was lei 469 million (median 263 million and average IHS turnover 19.8). Firms in the matched control group had 753 employees on average, which is also less than bene- ficiaries, but this difference is not statistically significant. None of the aver- ages given in table J.1 are statistically significantly different across the beneficiary and matched control groups (as is to be expected since most of the variables were used in the matching exercise). Overall, in terms of the averages given in table J.1, the matched control group looks more like the beneficiaries than the rejected applicants do. For the control groups used in the assessment of spillover effects, the compar- ison shows that the control group had similar pre-scheme characteristics as non- beneficiaries. Table J.2 shows pre-scheme averages for these nonbeneficiaries and their matched controls. The average number of employees for nonbeneficia- ries was 212 (median 50 and average IHS employment 4.9), much smaller than for beneficiaries (1,247, as shown in table J.1). Average investment among non- beneficiaries was lei 3 million (median lei 268,964 and average IHS investment 12.1). Similarly, average IHS turnover among nonbeneficiaries was lei 73 million  101 102 | Design for Impact TABLE J.1  Pre-scheme summary statistics for Ministry of Public Finance scheme beneficiaries and ­control groups REJECTED APPLICANTS CONTROL MATCHED CONTROL BENEFICIARIES GROUP GROUP VARIABLE/RESULT INDICATORS N MEAN/SE N MEAN/SE N MEAN/SE Year of incorporation 17 1999 54 1998 17 1999 [1.170] [0.685] [1.266] Number of workers 17 1,247 54 581*** 17 753 [367.914] [89.671] [179.360] Number of workers (IHS) 17 7.072 54 6.509* 17 6.872 [0.338] [0.149] [0.251] Investment (IHS) 17 17.403 54 15.478** 17 16.649 [0.639] [0.465] [0.467] Turnover (IHS) 17 19.819 54 19.193* 17 19.755 [0.381] [0.159] [0.332] TFPR (ln) 17 9.051 53 8.560 17 8.838 [0.249] [0.192] [0.254] Markup (ln) 17 0.321 53 0.180 17 0.304 [0.155] [0.099] [0.151] Market share (4-digit NACE) 17 0.026 54 0.012** 17 0.022 [0.009] [0.003] [0.009] Firm-level measure of allocative 17 −0.142 53 −0.053 17 -0.095 efficiency (4-digit NACE) [0.059] [0.046] [0.034] Source: World Bank elaboration. Note: Data are for the year before each cohort first received funds through the MPF scheme. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Firm market share and firm-level measure of allocative efficiency are defined at the 4-digit NACE level (nationwide). SE = standard error; NACE = Nomenclature of Economic Activities; TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate mean is different from beneficiaries at the 1, 5, and 10 percent significance levels. (median lei 18 million and average IHS turnover 17.6). Thanks to the matching exercise, the control group for nonbeneficiaries had similar pre-scheme charac- teristics as nonbeneficiaries. In addition, for the difference-in-differences estimates to be causal, it is assumed that the outcomes of beneficiaries and the control group firms would have followed parallel trends in the absence of the MPF scheme. Indeed, data suggest that the outcomes of beneficiaries and both control groups followed parallel trends before the MPF scheme was implemented. Given that the scheme happened, this assumption is fundamentally untestable. However, if the trends in outcomes variables were parallel during the years before each cohort first received funds through the scheme, it is plausible that these trends would have continued to be parallel in the absence of the scheme. Figures 4.7 and 4.8 (in the main text, the section “Direct Effects of the MPF Scheme on Beneficiaries: The Incentive Effect” of chapter 4) show that the outcomes of beneficiaries and their control firms followed parallel trends before the MPF scheme, suggesting the control groups are valid. It is worth highlighting that difference-in-differences analysis may erroneously attribute increases in the ­ the outcome variables to the effects of the MPF scheme if beneficiaries received Appendix J | 103 TABLE J.2  Pre-scheme summary statistics for nonbeneficiaries NONBENEFICIARIES IN BENEFICIARY SECTORS MATCHED CONTROL FROM OTHER SECTORS VARIABLE/RESULT INDICATORS N MEAN/SE N MEAN/SE 690 1998 690 1998 Year of incorporation [0.186] [0.177] 690 212 690 178 Number of workers [22.321] [21.176] 690 4.928 690 4.919 Number of workers (IHS) [0.049] [0.045] 690 12.109 690 11.928 Investment (IHS) [0.177] [0.184] 690 17.561 690 17.506 Turnover (IHS) [0.053] [0.052] 690 8.608 690 8.609 TFPR (ln) [0.039] [0.039] Source: World Bank elaboration. Note: Data are for the year before the first firm in the sector received funds through the MPF scheme. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. SE = standard error; TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate means are different at the 1, 5, and 10 percent significance levels. aid from other schemes at the same time. Information on other aid received is not available. However, because the selection mechanisms were ad hoc, there is no reason to believe that other aid schemes used the same targeting mecha- nism as the examined scheme. That is, firms in both the treatment and control groups may receive aid from other schemes, which should not bias the effects measured in the current evaluation. APPENDIX K Ministry of European Funds Scheme: Firms Included in the Analysis (Auxiliary Tables) The comparison of pre-scheme summary statistics shows that beneficiaries are different from nonbeneficiaries in terms of sector, region, and firm characteris- tics. Table K.1 shows pre-scheme summary statistics for Ministry of European Funds (MEF) scheme beneficiaries and nonbeneficiaries. The 41 beneficiaries not included in the analysis are on average younger than the 39 beneficiaries included in the analysis (which is one reason why they are not in the balanced panel used in the analysis). Not-included beneficiaries are also smaller than included beneficiaries in terms of employees, turnover, and market share. Not- included beneficiaries have, however, similar levels of R&D expenses, TFPR, and markup as included beneficiaries. Nonbeneficiaries are more similar to the 41 beneficiaries not included in the analysis than to the 39 beneficiaries in the analysis. They are also relatively young and small. A notable difference is that nonbeneficiaries have lower R&D expenses than beneficiaries. Overall—in terms of sector, region, and firm characteristics—beneficiaries are different from non- beneficiaries. MEF scheme beneficiaries are thus not a representative sample of firms in Romania, as is to be expected from a scheme that targets firms that plan to undertake R&D. Since the 39 beneficiaries included in the analysis are older and larger than the 41 beneficiaries not included in the analysis, it is not clear to what extent the results here can be extrapolated to the 41 beneficiaries not included in the analysis.  105 106 | Design for Impact TABLE K.1  Pre-scheme summary statistics for Ministry of European Funds scheme beneficiaries and nonbeneficiaries 39 BENEFICIARIES IN THE 41 BENEFICIARIES NOT ANALYSIS IN THE ANALYSIS NONBENEFICIARIES VARIABLE/ RESULT INDICATORS N MEAN/SE N MEAN/SE N MEAN/SE Year of 1996 2003*** 45,99 2001*** incorporation 39 41 [0.686] [0.970] 9 [0.029] Number of 251.179 25* 45,99 59*** workers 39 41 [75.219] [7.042] 9 [1.488] Number of 5.377 3.362*** 45,99 3.466*** workers (IHS) 39 41 [0.190] [0.156] 9 [0.008] R&D 1.735 2.458 45,99 0.857** expenses 39 41 (IHS) [0.745] [0.696] 9 [0.012] Turnover 17.691 15.025*** 45,99 15.487*** (IHS) 39 41 [0.221] [0.375] 9 [0.010] TFPR (ln) 7.018 7.087 38,59 7.753*** 39 36 [0.401] [0.352] 8 [0.008] 0.862 38,57 1.180 Markup (ln) 39 36 0.961 [0.162] 2 [0.007] Market share 0.006 0.001*** 45,96 0.001*** (4-digit 39 41 NACE) [0.001] [0.000] 3 [0.000] Firm-level 39 −1.018 36 −21.729 38,57 −1.495 measure of [12.580] [22.263] 8 [0.425] allocative efficiency (4-digit NACE) Source: World Bank elaboration. Note: For beneficiaries, data are for the year before they first received funds from the MEF scheme. Some of the beneficiaries not in the analysis do not have SBS data in the year before they started receiving funds from the MEF scheme. In these cases, the closest available year is used. Data for other nonbeneficiaries are for 2012. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Firm market share and firm-level measure of allocative efficiency are at the 4-digit NACE level (nationwide). SE = standard error. NACE = Nomenclature of Economic Activities; R&D = research and development; TFPR = revenue-based measure of total factor productivity. ***, **, and * indicate mean is different from beneficiaries included in the analysis at the 1, 5, and 10 percent significance levels. APPENDIX L Identification Strategy of the Ministry of European Funds Scheme Evaluation: Robustness Checks To test whether the control groups are valid comparison groups for the analysis, pre-scheme outcome measures for firms in the treatment and control groups were contrasted. For the control groups used in the assessment of direct and competition impacts, the comparison shows that both the rejected applicants and the matched control group look similar to beneficiaries. Table L.1 shows the pre-scheme averages for the variables used in the analysis, separately for benefi- ciaries, the rejected applicant control group, and the matched control group. In the year before first receiving funds from the Ministry of European Funds (MEF) scheme, beneficiaries had 251 employees on average (median 88 and average inverse hyperbolic sine [IHS] employment 5.4). Firms in the rejected applicants control group had 916 employees on average, but this high average is driven by outliers (median 71.5 and average IHS employment 5.6). Beneficiaries and rejected applicants are not statistically significantly different in any of the variables given in table L.1. The average value of R&D expenses in beneficiaries was lei 278,345 (median lei 0, meaning the median beneficiary did not have R&D expenses in the year before receiving MEF funds, and average IHS R&D expenses 17.4), and the average value of turnover was lei 61.1 million (median 23.1 million and average IHS turnover 17.7). Firms in the matched control group had 235 employees on average. None of the averages in table L.1 are statistically significantly different across the beneficiary and matched control groups (as is to be expected since most of the variables were used in the matching exercise). For the control groups used in the assessment of spillover effects, the comparison shows that the control group had similar pre-scheme characteris- tics as nonbeneficiaries. Table L.2 shows pre-scheme averages for these nonben- eficiaries and their matched controls. The average number of employees for nonbeneficiaries was 115 (median 45 and average IHS employment 4.68), smaller than for beneficiaries (251 as shown in table L.1). Average R&D expenses among nonbeneficiaries were lei 22,666 (median lei 0 and average IHS R&D expenses 0.65). Similarly, average IHS turnover among nonbeneficiaries was lei 45.8 million (median lei 12.4 million and average IHS turnover 17.2). Thanks to the matching exercise, the control group for nonbeneficiaries had similar pre-scheme charac- teristics to nonbeneficiaries.  107 108 | Design for Impact TABLE L.1  Pre-scheme summary statistics for Ministry of European Funds scheme beneficiaries and control groups ­ REJECTED APPLICANTS’ MATCHED CONTROL BENEFICIARIES CONTROL GROUP GROUP VARIABLE/RESULT INDICATORS N MEAN/SE N MEAN/SE N MEAN/SE Year of incorporation 39 1996 78 1996 78 1995 [0.686] [0.620] [0.466] Number of workers 39 251 78 916 78 235 [75.219] [312.813] [49.254] Number of workers 39 5.377 78 5.600 78 5.352 (IHS) [0.190] [0.190] [0.130] R&D expenses (IHS) 39 1.735 78 1.430 78 0.773 [0.745] [0.429] [0.345] Turnover (IHS) 39 17.691 78 17.623 78 17.614 [0.221] [0.245] [0.152] TFPR (ln) 39 7.018 77 6.929 77 7.368 [0.401] [0.381] [0.189] Markup (ln) 39 0.862 77 1.120 77 0.724 [0.162] [0.146] [0.104] Market share 39 0.006 78 0.007 78 0.004 (4-digit NACE) [0.001] [0.002] [0.001] Firm-level measure 39 −1.018 77 −14.036 77 −31.429 of allocative [12.580] [6.949] [12.246] efficiency (4-digit NACE) Source: World Bank elaboration. Note: Data are for the year before each cohort first received funds through the MEF scheme. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. Firm market share and firm-level measure of allocative efficiency are defined at the 4-digit NACE level (nationwide). SE = standard error; TFPR = revenue-based measure of total factor productivity; NACE = Nomenclature of Economic Activities; MEF = Ministry of European Funds; R&D = research and development. ***, **, and * indicate mean is different from beneficiaries at the 1, 5, and 10 percent significance levels. TABLE L.2  Pre-scheme summary statistics for nonbeneficiaries NONBENEFICIARIES IN MATCHED CONTROL FROM BENEFICIARY SECTORS NONBENEFICIARY SECTORS VARIABLE/RESULT INDICATORS N MEAN/SE N MEAN/SE Year of incorporation 1,840 1998 1,840 1998 [0.110] [0.108] Number of workers 1,840 115 1,840 113 [7.965] [7.217] Number of workers 1,840 4.682 1,840 4.686 (IHS) [0.024] [0.024] R&D expenses (IHS) 1,840 0.654 1,840 0.546 [0.061] [0.055] (continued)  | 109  TABLE L.2, continued NONBENEFICIARIES IN BENEFICIARY MATCHED CONTROL FROM SECTORS NONBENEFICIARY SECTORS VARIABLE/RESULT INDICATORS N MEAN/SE N MEAN/SE Turnover (IHS) 1,840 17.160 1,840 17.141 [0.031] [0.030] TFPR (ln) 1,768 7.666 1,799 7.716 [0.037] [0.033] Source: World Bank elaboration. Note: Data are for the year before the first firm in the sector received funds through the MEF scheme. IHS stands for inverse hyperbolic sine, which is used instead of the log to allow for zeroes. SE = standard error; TFPR = revenue-based measure of total factor productivity; R&D = research and development. ***, **, and * indicate means are different at the 1, 5, and 10 percent significance levels. One assumption behind the difference-in-differences approach is that the outcomes of beneficiaries and firms in the control group would have followed parallel trends in the absence of the MEF scheme. A common test of this assump- tion consists of examining if the outcomes of beneficiaries and their control group followed similar trends prior to the implementation of the program. Overall, figures 4.11 and 4.12 (in the main text, in the section “Direct Effects of the MEF Scheme on Beneficiaries: The Incentive Effect” of chapter 4) show that the outcomes of beneficiaries and both control groups followed parallel trends before the introduction of the MEF scheme, which suggests the control groups are valid. One caveat of this approach is that the difference-in-differences anal- ysis may incorrectly attribute increases in the outcome variables to the effects of the MEF scheme if beneficiaries (and not their control group) received aid from other schemes at the same time. Although information on other aid received is not available, it would be unlikely that only beneficiary firms had access to other aid. If both treatment and control groups received aid from other schemes, this should not bias the effects measured by the difference-in-differences methodology. ECO-AUDIT Environmental Benefits Statement The World Bank Group is committed to reducing its environmental footprint. In support of this commitment, we leverage electronic publishing options and print- on-demand technology, which is located in regional hubs worldwide. Together, these initiatives enable print runs to be lowered and shipping distances decreased, resulting in reduced paper consumption, chemical use, greenhouse gas emissions, and waste. We follow the recommended standards for paper use set by the Green Press Initiative. The majority of our books are printed on Forest Stewardship Council (FSC)–certified paper, with nearly all containing 50–100 percent recycled content. The recycled fiber in our book paper is either unbleached or bleached using totally chlorine-free (TCF), processed chlorine–free (PCF), or enhanced elemental chlorine–free (EECF) processes. ­ More information about the Bank’s environmental philosophy can be found at http://www.worldbank.org/corporateresponsibility. S tate aid impact evaluation is new in Romania. Given its novelty, the ex post evaluation seeks to provide evidence on how effective state aid has been, on whether state aid distorted competition, and on the implications for state aid design and implementation. These aspects are fundamental to improving the efficiency of public spending and minimizing market distortions. The ex post evaluation focuses on three state aid schemes to assess whether and to what extent the aid objectives have been fulfilled, and it measures their spillover effects as well as the effects on competition outcomes. Selected based on the their importance in supporting key policy objectives, their design and complexity, and the instruments used, the three schemes include de minimis aid implemented by the Romanian Counter-Guarantee Fund and designed to incentivize access to finance for micro, small, and medium enterprises (MSMEs); state aid granted by the Ministry of Public Finance to support regional development and job creation; and state aid provided by the Ministry of European Funds to support the upgrade and modernization of research, development, and innovation. The analysis finds evidence that the state aid schemes met their objectives without distorting competition significantly. The results for the de minimis scheme to incentivize access to finance for micro, small, and medium enterprises showed that the scheme increased employment and turnover of beneficiary firms and reduced the probability of aided firms closing. The state aid scheme to support regional development and job creation fulfilled its main objectives, with robust evidence of a positive direct effect on employment creation and, to some extent, on investment. Regarding the state aid scheme to support the upgrade and modernization of research, development, and innovation, the analysis found evidence that the scheme helped promote research and development efforts. This state aid evaluation builds on the Technical Assistance on Assessing the Effects of State Aid on Market Outcomes in Romania (2018–2020). It was funded by the European Union via the Structural Reform Support Programme (SRSP) and implemented by the World Bank in cooperation with the European Commission’s Directorate General for Reform Support (DG REFORM). ISBN 978-1-4648-1703-8 SKU 211703