Strengthening Monitoring and Evaluation of Innovation Programs in the EU Short-Term Impacts of the First RCTs with Cohesion Policies SUMMARY REPORT June 18, 2025 0.1 SUMMARY Funded by the European Union The Commission’s Directorate-General for Regional and Urban Policy This version of the report is from June 18, 2025. This report summarizes the results from the first experimental evaluations within Cohe- sion Policy. The summary introduces and motivates a series of impact studies initiated under the “Strengthening Monitoring and Evaluation of Innovation Programs in the Euro- pean Union” collaboration between Directorate-General for Regional and Urban Pol- icy (DG REGIO) and the World Bank. The report includes policy briefs that summarize studies identifying short-term impacts from policy experiments in Europe. Project leads: Leonardo Iacovone, Francisco Campos, and Alexandra Avdeenko Project evaluation team members: • Czechia: Stephen J. Anderson, Francisco Campos, Leonardo Iacovone, Andreas Menzel, Florian Anselm Muench, Mariana De La Paz Pereira Lopez, Chiara Spina, Rahul Suhag • Spain: Ana Paula Cusolito, Patricio Dalton, Leonardo Iacovone, Santiago Reyes Ortega, Juan Rogers • Romania: Stephen J. Anderson, Alexandra Avdeenko, Juan Alberto Espinosa Bal- buena, Rajesh Chandy, Leonardo Iacovone, Łukasz Marek Marć • Croatia: Alexandra Avdeenko, Francisco Moraes Leitao Campos, Rajesh Chandy, Leonardo Iacovone, Namrata Kala Acknowledgements. At DG REGIO, we thank for the trust and support of Peter Berkowitz, Nicola De Miche- lis, Wolfgang Munch, Silvia Santos Alvarez, Vincent Leiner, Simona Androni, Enrico Pe- saresi, and Ana Pires. At the World Bank, the team is grateful for the support and guidance provided by Anna Akhalkatsi, Marina Wes, Cecile Thioro Niang, Ilias Skamnelos, Reena Badiani-Magnusson, Giovanni Bo, Daria Gulei, Karim Omar Lara Ayub, Martha Sofia Mora Alvarez, and Djam- ilya Salieva. Moreover, the team would like to thank for the critical input and continuous support by Natasha Kapil, Arianna Legovini, Todor Milchevski, Ana Budimir, and David McKenzie. Moreover, the ongoing work would not have been possible without the crucial input of several team members such as Jakob Gaertner, Isoke Nelson, Juan Pascual Torres, Andrei Coca, Carmen Fediuc, Barbora Dolezalova, Katerina Ovciarikova, Sara Gironi, Damian Iwanowski, and Anna Nellar Otahalova. Partners: We would like to thank Lucila Castro, Ana Varjacic, Patricia da Costa, Deni Nurkic, Vasile Asandei, Radana Kratochvilova, Nikolina Topic, Matija Srbic, Ante Magzan, Gabriela Macoveiu, Ionel Popa, Florin Ghimpu, Mirta Anjos, Lucija Ropret, Nikolina Relic Sic, Lidia Betoaea, and Monica Zlavog. This work is being conducted in collaboration with the following implementing partners: This work is being conducted in collaboration with the following research partners: CONTENTS 0.2 Contents 1 Romania: Financial Support to Digitalize 1.1 2 Spain: Financial Support to Innovate 2.1 3 Czechia: Identifying the Right Training and Additional Support for Firms 3.1 3.1 Scientific Decision-Making for Entrepreneurial Growth in Ostrava . . . . 3.2 3.2 AI Adoption and Innovation in Post-Coal Regions . . . . . . . . . . . . . 3.5 4 Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.1 AI Artificial Intelligence FUNDECYT-PCTEx Foundation for Mbps Megabits per second the Development of Science and Technology and the Ex- CRM Customer Relationship Manage- tremadura Science and Tech- MSIC Moravian Silesian Innovation ment nology Park Foundation Centre DESI Digital Economy and Society In- ITT Intention-to-treat RCT Randomized Control Trial dex IE impact evaluation DG REGIO Directorate-General for Re- RDA Regional Development Agency gional and Urban Policy JTP Just Transition Programme RDI research, development, and inno- ERP Enterprise Resource Planning vation JUNTA La Junta de Extremadura, the Regional Government of Ex- EU European Union tremadura SME Small and Medium Enterprise CONTENTS 0.1 Executive Summary Investments into More Private Sector Innovation and Competitiveness Innovation in policymaking to maximize returns on European Cohesion Policy investments is crucial for improving Europe’s competitiveness and addressing global and local chal- lenges. As global challenges grow, innovation must accelerate. Innovation is particu- larly vital for middle-income countries to avoid the middle-income trap and transition to high-income status by integrating advanced technologies, fostering new ideas, en- hancing productivity, and ensuring global competitiveness (World Bank 2024). More- over, innovation drives efficient use of capital, labor, and energy, which is essential for sustainable development, especially in light of rising debt, aging populations, and en- vironmental challenges (World Bank 2024). Significant funds are allocated to support the objectives of Cohesion Policies, making it one of the largest areas of EU investment. The Multi-annual Financial Frame- work for 2021-2027 allocated A C361 billion - dedi- C1,211 billion, with over a quarter - A cated to Cohesion Policy through the European Regional Development Fund, the Co- hesion Fund, and the European Social Fund Plus (European Court of Auditors 2023). Approximately one-tenth of this amount is directed toward Research and Innovation. Additionally, during this period, the Recovery and Resilience Facility (RRF) was estab- lished as a temporary instrument to aid recovery from the COVID-19 economic crisis, promoting sustainable growth and enhancing resilience against future shocks. While European Union is widely acknowledged as a major investor in innovation, significant regional disparities in innovation outcomes persist and limit its overall competitiveness. To remain on par with global counterparts, each Euro invested in in- novation, research, and development must yield greater returns and have a more sub- stantial impact on regional economic growth. Smart Specialisation Strategies, which focus research and innovation policy interventions on areas of competitive advantage, are a central component of the Cohesion Policy framework. While innovation is a pri- mary goal, the process to achieve it often lacks mechanisms for continuous learning and improvement. Successful policy development, particularly in innovation, must be accompanied by ongoing adaptation to evolving circumstances. Policymakers need to identify challenges, test innovative solutions, and refine approaches to address them. The evaluation community can assist by providing the necessary knowledge and ex- pertise to support a trial-and-error policymaking approach, helping determine which policies are most effective. A New Toolkit for Innovation in Policy-Making: Experimentation Despite this large funding pool, knowledge about the causal impacts of Cohesion Policy programs and their implementation approaches is limited. For the European Union (EU) to compete globally, it must ensure that the funds Member States invest yield substantial returns, which requires continuous policy improvement. For the 2021-2027 period, learning how to invest this funding (cost-)effectively is essential to strengthen policies that aim to create jobs, promote just transitions, enhance competitiveness, foster sustainable development, and ultimately improve citizens’ quality of life. Enhancing the effectiveness and efficiency of European Cohesion Policy invest- ments requires rethinking how beneficiaries most in need of public support are iden- tified. How can we know whether funded beneficiaries would have performed just as well without public funding? Or would other recipients have benefited more and gen- erated even greater returns on public investments? Good targeting and well-tailored programs are complex and require robust, reliable evidence on what works, for whom, under which conditions. Improving the effectiveness and efficiency of European Cohesion Policy invest- ments demands innovation in policymaking and experimentation to discover which investments work best for whom, under what conditions, and at what cost. To en- sure continued relevance in today’s rapidly evolving landscape, programs must avoid static implementation approaches by ensuring access to and analysis of quality data as well as conducting rigorous analyses.1 A commitment to continuous learning and adaptation is essential for maintaining a competitive edge and making informed public investment decisions. Experimental evaluations and policy experimentation provide valuable tools, through trial and error, to assess the effectiveness of implementation strategies, guide decision-making, and enhance accountability. Enhancing the effectiveness and efficiency of European Cohesion Policy invest- ments requires rethinking how evidence on what works is generated. Systematic re- views of Cohesion Policy evaluations emphasize the need to improve evaluation design and methods for more reliable findings, initiate data collection early, and invest in build- ing and maintaining evaluation capacity. Despite extensive evaluations of Cohesion Policy instruments, the evaluation toolkit is still dominated by qualitative evaluations (incl. case-studies) or ex-post evaluations conducted long after investments are made (Pellegrin et al. 2020), limiting opportunities to test, learn, and course-correct when it matters most (e.g., at mid-term).2 In their article “How to make evaluations of EU Co- hesion Policy more credible”, Asatryan et al. [2024a] conducted an AI-powered textual analysis of approximately 2,500 Member State evaluations. Their findings, alongside the work of Heinemann et al. [2024], highlight critical gaps in the current evaluation system, indicating that it lacks key elements necessary for producing fully credible pol- icy insights on the effectiveness of public investments.3 1 Policy experimentation allows for the rigorous measurement of the changes resulting from policies or in- terventions, for instance, focusing on changes in private sector outcomes such as business performance, managerial skills, and innovation practices. 2 David Alba, for instance, reports that Member States produced 1,500 evaluations of Cohesion Policies between 2021 and 2027, covering both current and previous programming periods, mostly drawing on the 2014-2020 experience. Of these, over 400 were impact evaluations, but the majority were mon- itoring and process evaluations. Key lessons learned include: improving evaluation design (with more targeted evaluation questions and scope aligned to budget); selecting appropriate evaluation methods (to ensure more reliable findings); broadening the use of data (anticipating and organizing data collec- tion at the intervention’s start); and building and maintaining evaluation capacity. Source: Link, accessed August 28, 2024. 3 For a summary of their work, see Asatryan et al. [2024b]. Similarly, Bravo-Biosca [2020] highlights a significant gap in robust evidence guiding innovation policy decisions. Many areas of policy would lack reliable evidence, and even where evidence exists, it would often show small or negligible effects. For reference, the What Works Centre for Local Economic Growth reviewed nearly 15,000 evaluations, find- A new strategic partnership supports the use of experimental evaluations − Randomized Control Trials (RCTs) − within Cohesion Policies, employing a method that can help identify what works and what does not, thus facilitating continuous improvement and adaptation to changing conditions. To foster creativity and pro- mote the exploration of new policy ideas and implementation methods, the World Bank’s Finance, Competitiveness, and Innovation Global Practice (FCI GP), in collab- oration with the European Commission’s DG REGIO, launched a new initiative which aims to strengthen the monitoring and evaluation of innovation programs in the EU. Since 2022, the partnership has sought to generate rigorous evidence in areas such as firm performance, productivity, entrepreneurship, innovation, and Small and Medium Enterprise (SME) development. By supporting the piloting of novel evaluation ap- proaches, this partnership enables scalable innovations across the EU and beyond. Short-term Causal Impacts of Innovative Investments into Competitiveness Policies Policy experiments in Cohesion Policy are emerging as a means to both learn from and inform future policy cycles. Similar to how companies invest in R&D to identify the most successful products for scaling up, policy experimentation and rigorous eval- uation can uncover novel interventions and the conditions under which returns on cur- rent investments are maximized. Moreover, random selection of beneficiaries among equally eligible candidates fosters fairness and transparency in the selection process. By ensuring all eligible firms have an equal chance of receiving funding, RCTs demon- strates that the selection process remains unbiased and free from favoritism. This, in turn, can enhance the credibility and legitimacy of the funding program. The first results from embedding experimentation within the Cohesion Funds are now available - they show that experimentation is not only feasible, but the results are also timely and insightful. This strategic engagement, titled Strengthening of Moni- ing that only a small fraction provided strong causal evidence. For similar findings, please refer to Dalziel [2018]. Overall, their key recommendations include the need for improving evaluation standards, for adopting an “evaluate first” approach (requiring ex-ante evaluations before scaling up important pro- grams), for using advanced counterfactual methods, for strengthening the link between evaluations and funding decisions, for fostering a European evaluation market, and for establishing an independent third- party advisory panel for Cohesion Policy evaluations. toring and Evaluation of EU Innovation Programs (‘Smart M&E project’), followed an open call for Managing Authorities interested in participating. An initial workshop provided basic technical assistance for designing rigorous experiments, leading to the establish- ment of five innovative policy experiments in four Member States under Cohesion Pol- icy.4 This progress was made possible by substantial investments in building local and institutional capacities, alongside a strategic agenda that helped Managing Authorities navigate organizational challenges. Over the past three years, the World Bank initiated the evaluations of the impact of instruments valued at over A C235 million (≈ US $254 million). These instruments were initially expected to launch in 2021/2022, with the new 2021-2027 programming cycle, but are still being rolled out in 2024. Insights from the experimental impact evaluations focus on SMEs - key drivers of growth - testing funding aimed at boosting competitiveness, innovation, and re- ducing disparities across the EU. SMEs as the primary target group of the new impact evaluations. SMEs are critical drivers of employment, economic growth, and innova- tion. In 2022, there were an estimated 23.1 million SMEs in the EU (Statista). Invest- ments in less developed regions through the EU Cohesion Policy aim to promote bal- anced growth, reduce disparities, and enhance economic stability and competitiveness across the EU. These funding schemes focus on digitalization, innovation, and com- petitiveness to address economic, social, and territorial disparities, helping less devel- oped areas catch up with more prosperous regions. A significant portion of the funding is dedicated to enhancing SME competitiveness, fostering innovation, and supporting R&D. Short-term impacts of these policy evaluations, a few months after the invest- ments were made, indicate that firms respond positively to the new policies and ap- proaches. The primary questions addressed by the impact evaluations (IEs) seek to better understand the impact of specific interventions on regional development, examining 4 Capacity-building events have helped cultivate a culture of evidence-based policymaking. Following an impact evaluation workshop in Lisbon in June 2022, which brought together 12 country teams from various Member States, six studies across five countries (Croatia, Czechia, Italy, Romania, and Spain) were selected for funding. For these projects, academic researchers, practitioners, and World Bank staff collaborated to develop rigorous evaluation designs. These experiments were tailored to the specific policies, knowledge gaps, and contexts, while also ensuring compliance with scientific standards. how various approaches to policy implementation influence the results. North-East Romania’s digitalization grants aim to boost SME productivity through targeted grants. The Cohesion Policy investment provides non-refundable grants to SMEs to support digital transformation. Using a randomized encouragement design, 519 out of 1,038 (most likely) eligible firms were selected to apply, with 333 eventually contracted. Early results show a 101% increase in digital investments, with notable rises in software, business administration, and production tools. While short-term impacts on profits and sales were not statistically significant, the program shows promise for long-term digital growth. Spain’s R&D grant program supports innovation in SMEs through a novel random- ized evaluation. In Extremadura, a matching grant scheme offers up to A C250,000 per firm to stimulate R&D, particularly in agriculture and ecological transition. A bin-ranking RCT design was used, where medium-quality applicants were randomly assigned to treatment and control groups. Treated firms projected a 48% increase in private R&D investment. The evaluation incorporated Bayesian methods and expert priors. The first short-term results also reveal that while public subsidies can significantly boost invest- ment, they may also displace other funding sources. Czechia’s innovative training programs enhance entrepreneurial and digital capa- bilities in post-coal regions. Two RCTs evaluated the impact of business and AI training under the Just Transition Programme. The Smart Start program in Ostrava improved business changes and monthly sales by 64%, emphasizing scientific decision-making. The AI Klub in Usti and Karlovy Vary led to a 117% increase in monthly sales and a 66% rise in profits, along with significant gains in AI adoption and innovation. These results highlight the effectiveness of structured training and peer learning in revitalizing disad- vantaged regions. Croatia’s Scinergy pilot fosters science-business collaboration through structured matchmaking. The program connects SMEs with scientists to support innovation, par- ticularly in AI and energy efficiency. Using simple randomization, 100 of 210 eligible firms were selected to participate in workshops and receive ongoing support. The ini- tiative led to an increase in collaboration attempts and successful partnerships. AI- focused firms saw substantial gains in product development, commercialization, and R&D employment. With the first results, the pilot already demonstrates high returns and supports scaling similar interventions. Overall, these reliable, impartial, and credible evidence on the impacts of the evaluated programs shall provide policymakers with the information needed to make decisions on resource allocation and improve outcomes for businesses and the broader economy. The collaboration has been designed to assist policymakers in identifying what works, for whom, and under which conditions. By learning from short-term im- pacts of their investments, Member States can improve their understanding of regional markets and their challenges, enhance policy implementation processes, and ultimately design more impactful programs. Note 1: Romania: Financial Support to Digitalize 1.1 NOTE 1 Romania: Financial Support to Digitalize What is the impact of digitalization on Romanian firms’ performance? Investment Romania’s program provides non-refundable grants to SMEs for digi- talization, aiming to boost productivity in the North-East region, with a focus on key digital performance indicators. Digitalization has the potential to transform industries, particularly SMEs, which in Romania lag behind the EU average in digital performance. Key barriers to digital transformation include a lack of financing, awareness, and ca- pacity to capitalize on digital investments. In North-East Romania, digitalization grants are awarded to eligible SMEs under the de minimis aid scheme, with a total alloca- C31,000,000 in the first call (A tion of A C26,350,000 ERDF and A C4,650,000 from the 1 state budget). Firms can receive grants ranging from A C15,000 to A C100,000, cover- ing 90% of eligible project costs. The World Bank studies the impact of digitalization on SME performance. The hypothesis is that financial support will boost productivity by enabling the acquisition of knowledge, services, and soft/hardware. As part of the study, the World Bank conducted a pre-assessment using Digital Economy and Soci- ety Index (DESI) indicators, as defined by the Regional Development Agency (RDA): The most common technology was “20+ percent of employees using connected mo- bile devices” (681 businesses, 66% of the sample), followed by “Internet connection of 1 Call guidelines section 3.3, p.11. Note 1: Romania: Financial Support to Digitalize 1.2 at least 30 Megabits per second (Mbps)” (620 businesses, 60%) and “50+ percent of employees with Internet access” (604 businesses, 58%).2 Table 1.1: DESI Indicators in North-East Romania DESI Indicators Total Percent 20+ percent of employed people use connected mobile devices 681 65.61 Internet connection of at least 30 Mbps 620 59.73 50+ percent of employed people w/ Internet access 604 58.19 Use a social media network 546 52.60 Own a website or a web page 516 49.71 Issue electronic invoices that allow automatic processing 395 38.05 Smartphones 373 35.93 Pay for internet advertising 279 26.88 It has a website with complex functions 130 12.52 IT uses Cloud Computing, high or medium complexity 108 10.40 Sales from e-Commerce activity exceed 1 percent of total turnover 105 10.12 Business-to-consumer web sales exceed 10 percent of total online sales 101 9.73 None of the above 43 4.14 Experiment In an experimental encouragement design, a randomly selected subset of participants is incentivized to join a program. This allows evaluation by comparing outcomes between encouraged and non-encouraged groups, adjusting for actual par- ticipation rates.3 Designing a compelling encouragement is crucial for the success of this approach. In collaboration with the North-East RDA, the World Bank established a process to ensure compliance with legal regulations for evaluating and implementing the digitalization program. The following steps were designed to integrate the RCT and derive key insights on improving digitalization in the region: 2 Table 1.1 lists the technologies included in the indicator and the percentage of businesses using them. 3 For example, among 2,000 pre-selected startups eligible for a training program, 1,000 are randomly encouraged (via financial incentives or expert support) to participate. At the same time, the other 1,000 receive no encouragement to enroll in the program (despite being eligible). After one year, business suc- cess indicators (e.g., business survival, revenue growth) are compared between both groups, accounting for actual participation. In this evaluation approach, the effectiveness of the encouragement to generate significant take-up (in the so-called “first stage”) is key. However, if few or none take up the program, it suggests ineffective encouragement, making evaluation difficult and potentially underestimating the program’s impact. Low take-up of the program after encouragement prevents any meaningful review of the program’s impact since the treatment and comparison groups would effectively be the same in terms of program participation. Note 1: Romania: Financial Support to Digitalize 1.3 First, the RDA and World Bank formed a learning partnership, publicly announced at the program’s launch. The first call for proposals was defined as non-competitive and framed as a pilot to generate learning. Beyond implementing the RCT, the goal was to adapt future digitalization programs better to meet the needs of SMEs. To this end, SMEs participated in a pre-assessment survey to (a) assess digital maturity, (b) ex- press interest in applying for the program, and (c) receive an automated, simplified preliminary eligibility assessment.4 Developed in Spring 2023 and piloted in May, the pre-assessment concluded in September 2023, identifying 1,038 likely eligible SMEs from over 1,500 applicants. Second, in October 2023, 519 firms were randomly selected from the eligible pool of 1,038 for funding. This random selection ensured comparability between selected and non-selected groups across various characteristics. The selection, conducted in front of a notary (Figure 1.1), ensured transparency, with the remaining 519 firms serving as a comparison group. Third, the selected 519 firms were invited to submit detailed digitalization invest- ment proposals via the national MySMIS IT platform. Of those, 372 firms (71.7%) sub- mitted final documents, including the required digital feasibility study designed by RDA by the end of Winter 2024. After verifying eligibility, RDA is finalizing contracts. 4 The survey checked eligibility using objective data, cross-verified with administrative information to streamline RDA’s review process. Note 1: Romania: Financial Support to Digitalize 1.4 Figure 1.1: Randomization Process with Notary’s Presence in Romania (b) Running the Code to Se- (a) Drawing the Random lect the Firms after Inserting Number Seed the Seed (c) Reviewing the Balance Statistics Source: Randomization event at the Regional Development Agency, October 2023. Note 1: Romania: Financial Support to Digitalize 1.5 Figure 1.2: Experimental Design in Romania Quick Pre-Assessment 1,038 Eligible Firms Randomization 519 Selected 519 Not Selected Encouraged to Formally Apply 393 (75.7%) Firms Submit a Digital Feasibility Study (Full Proposal) RDA Reviews Applications 333 (64.2%) Firms Contracted Note: Breakdown of the application process stages and experimental design. Only 76% of selected firms chose to formally apply for funding, and only 64% of selected firms were contracted as of 31 March 2025. Source: Implementation data from RDA. Note 1: Romania: Financial Support to Digitalize 1.6 Figure 1.3: Contract Signatures, by Calendar Month 140 130 120 110 Number of contracts signed 100 90 80 70 60 50 40 30 20 10 0 3 4 5 6 7 8 9 10 11 12 1 2 3 m m m m m m m m m m m m m 24 24 24 24 24 24 24 25 25 25 24 24 24 20 20 20 20 20 20 20 20 20 20 20 20 20 Date Note: Number of contracts signed between March 2024 and March 2025. Terminated contracts are not included. Source: Implementation data from RDA. Short-Term Impacts Digital investments increased by 101%. The primary evalua- tion metric on Intention-to-treat (ITT) effect, capturing the average impact of being of- fered the opportunity to apply for funding (as opposed to actually receiving the funds).5 Treated firms invested an average of RON 71,199 in digital tools, compared to a control group mean of RON 70,529, representing a 101% increase relative to the control mean. Software expenditures rose by 167%, digital tools for business administration saw a 291% increase, and production-related digital investments increased by 352%. Re- sults are reported for all firms, independent of whether they eventually signed the grant contract with RDA or not and for those who did so, they are reported only a few months after contract signature. In other words, the reported estimates are very conservative estimates of the treatment effects based on self-reported survey data. Firms in the treatment group spent RON 8,696 on software, compared to RON 5,207 in the control 5 The impacts of receiving the funds are reported in the full report. Note 1: Romania: Financial Support to Digitalize 1.7 group, a 167% increase. Moreover, expenditures in business administration tools rose to RON 37,082 among treated firms, compared to RON 12,752 in the control group. Treated firms spent RON 40,380 on production tools, compared to RON 11,474 in the control group. Furthermore, marketing tool investments rose by 118% and sales tool investments increased by 241%. Marketing-related digital expenditures increased to RON 209 among treated firms, compared to RON 177 in the control group. Treated firms also spent RON 3,312 on sales tools, compared to RON 1,374 in the control group. Customer Relationship Management (CRM) usage increased by 26% and Enterprise Resource Planning (ERP) usage increased by 22%. The share of firms using CRM soft- ware rose by 3.1 percentage points from a control mean of 11.7%. ERP adoption rose by 5.6 percentage points from a control mean of 26%. On average, no significant short-term impact on profits or sales yet. While treated firms showed slightly higher sales and labor productivity, the differences were not sta- tistically significant, likely due to the early stage of implementation. Treated firms were 10.7 percentage points more likely to attribute future revenue growth to new customers, compared to a control mean of 44%. Overall, while performance outcomes are not yet evident, the early results suggest strong potential for long-term impact. Tailored sup- port and continued monitoring will be essential to ensure inclusive and sustained digital transformation. Note 2: Spain: Financial Support to Innovate 2.1 NOTE 2 Spain: Financial Support to Innovate What is the impact of R&D grants on SMEs business performance in Western Spain? Investment Spain’s matching grant program awards up to A C250k per firm to boost R&D investment in small and medium companies, primarily in agriculture and eco- logical transition. To address market failures that keep regional R&D investment at sub- optimal levels, the La Junta de Extremadura, the Regional Government of Extremadura (JUNTA) and Foundation for the Development of Science and Technology and the Ex- tremadura Science and Technology Park Foundation (FUNDECYT-PCTEx) have devel- oped a matching grant program funded by the EU, offering large R&D grants to small and medium firms in Extremadura. Each cycle plans to award A C3.5 million in grants (up to AC250k per company), financing 35% to 80% of proposed R&D investments. Applicants are primarily small and medium companies, mainly in the agricultural or ecological transition sectors, with most grants supporting industrial research and ex- perimental development. The study evaluates the impact of large public R&D grants in Extremadura, Spain, using a RCT−−the first of its kind for such high-value subsidies (average appr. AC 138,000). The subsidy program, funded by the EU and implemented by the Junta de Extremadura and FUNDECYT-PCTEx, supports R&D in agro-industry, ecological transition, health, digital transformation, and tourism. Note 2: Spain: Financial Support to Innovate 2.2 Experiment In this RCT design, potential beneficiaries (e.g. firms) are ranked based on their application scores during the evaluation process, which considers their perfor- mance and eligibility.1 In Spain, the World Bank is collaborating with JUNTA to offer technical support aimed at generating valuable insights from this investment. A rig- orous impact evaluation requires a sufficient number of high-quality applications. To achieve this, the World Bank collaborated with the JUNTA and FUNDECYT-PCTEx to coordinate an advertising campaign aimed at attracting (at least) 100 beneficiaries. All applications were evaluated by external reviewers following pre-established guidelines and classified into three groups based on quality: group 1 (poor quality proposals) that do not meet minimum standards, group 2 (intermediate quality proposals) in the “in- termediate” range, and group 3 (highest quality projects) that are directly selected to receive grants. In more detail, proposals were scored on 11 objective criteria and cate- gorized into high, medium, and low potential. High-potential proposals were automat- ically funded; low-potential were rejected. Medium-potential proposals (56 total) were randomly assigned to treatment (22 funded) and control (34 not funded) groups.2 Ran- domization successfully balanced key firm characteristics, and there was no evidence of selective attrition in follow-up survey responses. Short-term Impacts Firms receiving the subsidy projected A C 65,000 more in pri- vate R&D investment for 2025−a 48% increase over the control group average. Data collection included baseline survey (pre-randomization), follow-up survey (3 months post-randomization), expert priors, and risk, ambiguity, and time preference surveys for 1 In more detgail, applicants are sorted into bins according to their scores, with the highest and lowest performers excluded from the evaluation study. Thereby, high scorers are automatically accepted into the program, while low scorers are rejected. The middle bin of applicants, equally eligible for the program, is randomly assigned either to the treatment (program/ funding) or comparison group. This design is effective when there are clear eligibility criteria and resources are limited. It ensures that top performers are prioritized while still providing a randomized comparison group to assess the program’s impact among participants with ex-ante unclear or disputed gains. For an example of this design in practice, see Karlan and Zinman [2011]. 2 Randomization was based on budgeted investment size to ensure balance across firm sizes and project scales. Note 2: Spain: Financial Support to Innovate 2.3 Figure 2.1: Experimental Design in Spain Applications Evaluation Low Quality Medium Quality High Quality Randomization (56 firms) Selected (22 firms) Not Selected (34 firms) evaluators. Additionally, the evaluation incorporated expert priors to improve precision and account for high variance and small sample size. The team analyses new Bayesian methods as a new, promising approach for evaluating complex, high-variance innova- tion programs. Before program implementation, expert priors revealed, for instance, that without funding, the average probability of successful implementation was 16%, which increased to 67% with funding. Expected sales growth (conditional on success) doubled from 8% to 16%. Treated firms exhibit greater growth in their annual R&D investments relative to the baseline period. Figure 2.2 demonstrates a sharp rightwards shift in the distribution of R&D growth among treated firms, measured in percentage terms. At the left tail are firms in control that were previously investing in R&D but now have no ongoing projects, leading to a growth of -100%. At the right tail are treated firms that previously invested minimal amounts but are now commencing large-scale R&D projects. Overall, public R&D subsidies can significantly boost private investment, but care- ful targeting is needed to avoid displacing other funding sources. Over half of the Note 2: Spain: Financial Support to Innovate 2.4 Figure 2.2: Density plot of projected R&D investment growth by treatment status control group firms planned to continue their R&D projects using alternative funding or internal resources, suggesting public subsidies may crowd out other funding sources. Note 3: Czechia: Identifying the Right Training and Additional Support for Firms 3.1 NOTE 3 Czechia: Identifying the Right Training and Additional Support for Firms Does participation in business training and network meetings improve busi- ness outcomes and enhance the effectiveness of innovation funding? Investment In Czechia, the World Bank evaluates whether enhancing entrepreneurial skills improves the effectiveness of a voucher scheme supporting business develop- ment, innovation, and digital transformation in regions affected by the coal industry’s decline. Several Czech regions are focused on economic diversification, prioritizing pri- vate sector growth, but entrepreneurial activity remains low. A new program, approved by the European Commission in September 2022, targets regions affected by the de- cline of the coal mining industry. It emphasizes business R&D, post-mining area re- generation, skills development, clean energy, and the circular economy, with a total C1.641 billion. A key component of the Just Transition Programme (JTP) is allocation of A the voucher scheme for entrepreneurs in Czech coal regions, offering different types ˇ of financial support.1 These vouchers are available in the Moravian-Silesian, Ustí, and 1 The first is the Voucher for Business Development, which provides 50,000 CZK (A C2,100) for new en- trepreneurs and up to 0.5 million CZK (A C21,100) for SMEs to start or expand businesses, improve pro- cesses, or receive coaching and education. The second is the Innovation Voucher , offering SMEs up to Note 3: Czechia: Identifying the Right Training and Additional Support for Firms 3.2 Karlovy Vary regions, with a focus on diversifying local economies reliant on carbon- intensive industries. As part of two independent impact evaluations, additional activi- ties were tested to improve business outcomes in these regions, partially also targeting innovation voucher recipients. For this, the World Bank has been working with local partners and chambers of commerce to extend the additional support to new poten- tial beneficiaries. 3.1 Scientific Decision-Making for Entrepreneurial Growth in Ostrava Training In the Moravian-Silesian region, the pilot aimed to foster entrepreneurial growth by addressing two key constraints: limited ability to validate business mod- els and low ambition to scale. In particular, the program emphasizes mindset training for innovation, offering ‘smart training’ that promotes a scientific approach to decision- making under uncertainty. This training, aimed at firms selected for entrepreneurial vouchers (and others), teaches participants to make informed decisions in uncertain environments, covering topics such as understanding innovation, risk vs. uncertainty, hypothesis testing, and A/B testing, culminating in participant presentations (Figure 3.1). Experiment The RCT involved 246 firms, with 163 assigned to the treatment group and 83 to the control group. The Smart Start program, implemented in partnership with the Moravian Silesian Innovation Centre (MSIC), delivered seven workshops fo- cused on scientific decision-making, customer validation, and structured business de- velopment. The evaluation measured business changes, motivations for change, and firm performance. Short-Term Impacts The short-term results indicate that Smart Start program sig- nificantly improved externally-oriented business changes and revenue growth among C42,200) to develop new products, optimize services, enhance processes, and create 1 million CZK (A C21,100) to as- innovation-related jobs. The third is the Digital Voucher , granting up to 0.5 million CZK (A sist SMEs in preparing for digital transformation by acquiring hardware and software. Note 3: Czechia: Identifying the Right Training and Additional Support for Firms 3.3 Figure 3.1: Smart Start Training in Czechia early-stage entrepreneurs in Ostrava. The scientific decision-making framework helped participants adopt a more structured, proactive approach to customer targeting and sales. Figure 3.2: Experimental Design for Smart Start Training in Czechia 246 Interested Firms Random selection 163 Invited to Smart Start Events ≈ 2 3 83 with No Support ≈ 1 3 Note 3: Czechia: Identifying the Right Training and Additional Support for Firms 3.4 In more detail, the Go-to-Market (GTM) change intensity increased by 47%. Treated firms scored 0.65 points higher on a 1-7 scale than the control mean of 1.39. 42% of treated firms made moderate or greater GTM changes, compared to 19% of control firms. Moreover, treated firms implemented 12% more high-intensity GTM changes than the control mean of 0.25. For Production and Operations Management (POM) changes showed, the differ- ences were not significant.Treated firms scored only 0.17 points higher than the control mean of 1.58 (11% increase, not significant). Program-related motivations for change increased by 98%. Treated firms cited 2.10 program-related reasons on average, compared to 1.06 in the control group. Treated firms were also 24 percentage points more likely to be in the top half of program- related motivations (29% control mean). Monthly sales increased by 64%. Treated firms reported CZK 103,325 higher monthly sales than the control mean of CZK 160,271. Overall, the sales index improved by 0.32 standard deviations. Note 3: Czechia: Identifying the Right Training and Additional Support for Firms 3.5 3.2 AI Adoption and Innovation in Post-Coal Regions Training The AI Klub program offered five in-person workshops covering AI appli- cations in business efficiency, marketing, sales, product development, and financial management. The impact evaluation includes complementary activities to strengthen the JTP outcomes. One such initiative is the ‘Smart Start’ project, launched with World ˇ and Karlovy Vary regions, this program focuses Bank’s support since 2022. In the Ustí on innovation vouchers, incorporating a ‘networking’ approach, where firms receive ex- pert advisory support and are encouraged to share knowledge and network to enhance innovation. The workshops emphasized hands-on learning, peer-to-peer exchange, and practical application of AI tools. Experiment To form the eligible pool, all voucher recipients were invited to express interest in the support program, as well as firms outside of this pool. Approximately 2 3 of interested firms were invited to participate in the events. The unequal allocation (with more firms in the treatment group) allows for the testing of further add-on activi- ties in future. The RCT involved 173 firms, with 117 assigned to the treatment group and 56 to the control group. The evaluation measured AI adoption, perceived effectiveness, innovation, business networking, and firm performance. Figure 3.3: Experimental Design for AI Training in Czechia 173 Interested Firms Random selection 117 Invited to AI-Klub Events ≈ 2 3 56 with No Support ≈ 1 3 Short-Term Impacts The AI Klub program significantly boosted AI adoption, inno- vation, and firm performance in structurally disadvantaged Czech regions. The group- Note 3: Czechia: Identifying the Right Training and Additional Support for Firms 3.6 based, hands-on workshop model proved effective in overcoming barriers to technol- ogy adoption and fostering peer learning. These findings support scaling similar in- terventions to accelerate digital transformation and economic revitalization in post- industrial regions. In more detail, AI adoption intensity increased by 15% and AI use in customer management rose by 118%. Treated firms scored 0.38 points higher on a 0-5 AI adop- tion scale, compared to a control mean of 2.62. Treatment led to a 0.72-point increase in AI use for customer management over a control mean of 0.61. AI use in product development increased by 48% and AI use in marketing rose by 31%. Treated firms scored 0.48 points higher than the control mean of 1.00. Treatment firms scored 0.58 points higher than the control mean of 1.87. Perceived AI effectiveness increased by 43%. Treated firms reported a 0.68-point increase on a 0-5 scale, relative to a control mean of 1.59. Perceived AI effectiveness also rose in marketing and customer management by 71%. Both functions saw 0.52 and 0.47-point increases, respectively, over control means of 0.73 and 0.66. Individual-level innovation increased by 67% and firm-internal innovation rose by 55%. Treated firms scored 1.04 points higher than the control mean of 1.56. Treat- ment led to a 0.62-point increase over a control mean of 1.13. New customer acquisition increased by 60%, monthly sales increased by 117%, and monthly profits increased by 66%. Treated firms scored 0.65 points higher than the control mean of 1.09. Treated firms saw a CZK 1,420,659 increase over a control mean of CZK 1,218,750. Treatment also led to a CZK 72,648 increase over a control mean of CZK 110,484. Overall, a composite measure reflects consistent gains across sales and profits. Firm growth index improved by 0.78 standard deviations. New business connections increased by 78%. Treated firms reported 3.82 more connections than the control mean of 4.88. Also, business discussions increased by 37%. The mean index rose by 0.45 points over a control mean of 1.23. Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.1 NOTE 4 Croatia: Supporting Private Sector Inno- vation through Matchmaking between Busi- nesses and Science Does connecting firms with scientists and supporting their collaboration fos- ter research partnerships and innovation? Investment Firms and scientists face collaboration barriers, and stronger support is needed to boost partnerships and innovation. Firms and scientists often struggle to collaborate due to information frictions, behavioral biases, uncertain transaction costs, and financial constraints. Firms rely on scientific expertise for product development or problem-solving, while scientists may need firms to commercialize their research. However, finding the right match is challenging, particularly for SMEs with limited re- sources for outreach. In Croatia, for instance, collaborations typically occur within uni- versity networks or through personal contacts, but emerging fields like Artificial Intelli- gence (AI) and Energy Efficiency might need external support. In general, despite re- cent improvements in Croatia’s Research, Development, and Innovation system, chal- lenges remain in applied research, science-business collaboration, and commercializa- tion of academic research. Only 8% of Croatian companies collaborate with higher Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.2 education institutions, far below the EU average. To address this, the Croatian gov- ernment has introduced reforms through the National Resilience and Recovery Plan and the Smart Specialisation Strategy, including increased funding for collaborative re- search, development, and technology transfer. However, funding alone is insufficient to foster effective science-business partnerships. For the research sector to generate innovative knowledge with economic spillover effects, and for businesses to enhance their R&D capacity, stronger collaboration between the two is vital. Matchmaking sup- port could potentially bridge this gap by connecting firms with relevant scientists, facil- itating faster identification of expertise, and coordinating meetings to overcome com- munication barriers and build trust. The Scinergy pilot in Croatia connects businesses and scientists to foster collabo- rations, support innovation, and drive R&D through matchmaking events and ongoing guidance.The pilot program was launched to strengthen business-science collabora- tion particularly in high-potential fields like AI and energy efficiency. A World Bank sur- vey revealed that 47% of companies lacked information about relevant scientists, high- lighting the need for improved collaboration to drive innovation and technology trans- fer. To strengthen science-business collaboration, Croatia is piloting a business-science matchmaking support program. For this purpose, the World Bank, in partnership with the Innovation Center Nikola Tesla and the Ministry of Science, Education, and Youth, designed the “Scinergy” pilot to facilitate science-business collaboration. Scinergy aims to help connect businesses and scientists based on their needs and expertise, sup- porting product development, joint funding applications, and public-private partner- ships. Participating firms are offered targeted connections with scientists across fields like AI, energy, and biotechnology, and are offered legal and communication support. Matched researchers and businesses first participated in a two-day matchmaking event in February 2024. The first day involved “speed-dating” sessions between firms and sci- entists, while the second day featured workshops for pairs interested in collaboration. These sessions encouraged ideation, with exercises like explaining potential solutions in simple terms and visualizing them through drawings (Figure 4.1).1 In March and April 2024, these pairs explored concrete R&D ideas, leading to detailed discussions and 1 A video of the event is available online: link. Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.3 joint project planning. Pilot monitoring revealed that newly established interactions between firms and scientists continued in this period, with discussions on collaboration details, intellectual property, legal frameworks, and project financing. Firms attended specialized lectures on these topics, which deepened their understanding and aimed to facilitate cooperation. Figure 4.1: Business-Science Matching in Croatia Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.4 Experiment Simple randomization, often implemented via a lottery, is a fundamen- tal method for evaluating project effectiveness. It ensures that each participant has an equal chance of being assigned to either the treatment or comparison group, minimiz- ing selection bias.2 In Croatia, applications, submitted by firms between November 16, 2023, and January 31, 2024, captured information on scientific collaboration interests, past experiences, and future partnership preferences. Open-ended questions allowed firms to detail their R&D needs, barriers, and goals. Over 250 firms applied, with 210 qualifying for the Scinergy initiative. Eligible businesses, regardless of size or sector, were those seeking collaboration with the scientific community.3 Finally, the compa- nies confirmed their interest prior to the final selection and the launch of the event. In February 2024, 100 firms were randomly selected from the 210 eligible applicants for the business-science matchmaking pilot. These firms were matched with scientists to address their R&D needs. In other words, using a lottery 100 out of 210 were assigned to receive matchmaking support, workshops, and post-event guidance. Figure 4.2: Experimental Design in Croatia Applications (260) Eligible Firms (210) Randomization Invited to Scinergy (100) Not Invited (110) 2 The process can involve several steps: defining the eligible population, assigning unique identifiers to participants, and using a random selection mechanism, such as a digital random number generator or physical lottery, to create the treatment and comparison groups. The treatment group receives the inter- vention, while the comparison group does not, allowing for a direct comparison of outcomes after the intervention to assess its impact. For example, in an entrepreneurship training program for 200 star- tups, each is assigned a number from 1 to 200. A random number generator selects 100 startups for the treatment group, which receives training on business management, marketing, and finance, while the comparison group does not. After one year, business survival rates and revenue growth are compared to evaluate the program’s impact. 3 The application data was used to match firms with scientists for the Scinergy event. If needed, the im- plementation team contacted the companies for further clarification. Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.5 Short-Term Impacts Collaboration attempts increased by 38% overall and ongoing or successful collaborations increased by 38%. The evaluation focused on collabora- tion activity, innovation outcomes, and downstream business performance. Treated firms were 18 percentage points more likely to have ongoing or successful collabora- tions, compared to a control mean of 47%. Across all firms, the treatment group was 22 percentage points more likely to attempt collaborations, relative to a control mean of 58%, representing a 38% increase. Firms mostly embarked on collaborative projects with scientists to develop new technologies in the AI sector. Figure 4.3a displays what type of collaboration activities have been started.Firms in the treatment group were more likely to receive expert ad- vice provided by academics for commercial innovation (ITT=0.15, C-mean=0.08, p<0.01), and to incorporate knowledge through the spreading of academic research and ex- pertise through informal interactions, training or publications (ITT=0.15, C-mean=0.193, p<0.05). These results show that the increase in collaborations for treated firms hap- pened almost equally in a formal and informal manner, explaining the rather weak ev- idence of Scinergy on the likelihood of signing a formal collaboration agreement with a Croatian Scientist. Figure 4.3b compares, between firms in the treatment and com- parison group, the R&D fields in which the collaborations started since the beginning of 2024. Treated firms are more likely to have started collaborations in electrical engi- neering and electronics (ITT=0.15, C-mean=0.083, p<0.01), computer science and soft- ware engineering (ITT=0.123, C-mean=0.182, p<0.05), and AI (ITT=0.12, C-mean=0.227, p<0.1), which is in line with the original goal of the pilot to target those sectors. Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.6 Figure 4.3: Collaboration Details (a) Type of Collaboration Activity Note: Figure 4.3a shows treatment effects (ITT estimates) on collaboration activities, derived from the survey question “Since the beginning of 2024, has your firm been starting any of the following collaboration activities with Croatian researchers?”. ITT estimates are obtained from OLS regressions controlling for randomization strata, lasso- selected firm baseline characteristics, and, when available, baseline value of the outcome variable (robust standard errors). Significance stars: *** p<0.01, ** p<0.05, and * p<0.10. Source: Survey data (follow-up 2025 Q1) and authors’ computations. (b) R&D Field of Collaboration Note: Figure 4.3b shows treatment effects (ITT estimates) on RD! (RD!) fields of collaboration, derived from the survey question “In which field(s) of RD! have the collaborations activities with Croatian researchers been happening?”. ITT estimates are obtained from OLS regressions controlling for randomization strata, lasso-selected firm baseline characteristics, and, when available, baseline value of the outcome variable (robust standard errors). Significance stars: *** p<0.01, ** p<0.05, and * p<0.10. Source: Survey data (follow-up 2025 Q1) and authors’ computations. Note 4: Croatia: Supporting Private Sector Innovation through Matchmaking between Businesses and Science 4.7 Firms originally requesting support in AI benefited the most: they experienced a 53% increase in the completion of activities on product/process improvement, com- mercialization of their new or improved products rose by 50%, and they increased research, development, and innovation (RDI) employment by over 180%. The pilot provides robust evidence that structured matchmaking support can significantly en- hance collaboration and innovation, especially for firms interested in receiving support in the AI field. Treated AI-interested firms were 20.7 percentage points more likely to complete product or process improvements compared to a control group mean of 39%, representing a 53% increase. AI-interested firms in the treatment group were 25.7 percentage points more likely to commercialize new or improved products, rela- tive to a control mean of 51%, indicating a 50% increase. Finally, treated AI-interested firms hired two more full-time RDI employees on average, compared to a control mean of 1.1, representing a 181% increase. The economic return on investment was substantial By reducing search costs and facilitating trust-building, the program helped unlock the potential of Croatia’s scien- tific and entrepreneurial ecosystems. The benefit-cost ratio is positive, especially for AI-interested firms, suggesting that every Euro invested generated more than a Euro benefits. These findings support scaling similar interventions across other innovation- intensive sectors. BIBLIOGRAPHY 4.8 Bibliography World Bank. World Development Report 2024: The Middle-Income Trap. World Bank, Washington, DC, 2024. doi: 10.1596/978-1-4648-2078-6. URL https://openknowledge.worldbank.org/server/api/core/bitstreams/ 8f49fae8-ba60-45ba-b4d9-82bc22a964d9/content. License: Creative Com- mons Attribution CC BY 3.0 IGO. Julie Pellegrin, Laurent Colnot, and Matteo Pedralli. The role of evaluation in cohesion policy, 2020. Study requested by the REGI Committee. Zareh Asatryan, Carlo Birkholz, and Friedrich Heinemann. Evidence-based policy or beauty contest? an llm-based meta-analysis of eu cohesion policy evaluations. An LLM-Based Meta-Analysis of EU Cohesion Policy Evaluations (April 01, 2024). ZEW- Centre for European Economic Research Discussion Paper , (24-037), 2024a. Friedrich Heinemann, Zareh Asatryan, Julia Bachtrögler-Unger, Carlo Birkholz, Francesco Corti, Maximilian von Ehrlich, Ugo Fratesi, Clemens Fuest, Valentin Lang, and Martin Weber. Enhancing objectivity and decision relevance: A better frame- work for evaluating cohesion policies. ZEW-Centre for European Economic Research Discussion Paper , (24-034), 2024. Zareh Asatryan, Carlo Birkholz, and Friedrich Heinemann. How to make evaluations of eu cohesion policy more credible. ZEW policy brief , 2024, 2024b. Albert Bravo-Biosca. Experimental innovation policy. Innovation Policy and the Econ- omy, 20(1):191–232, 2020. BIBLIOGRAPHY 4.9 M. Dalziel. Why are there (almost) no randomised controlled trial-based evaluations of business support programmes? Palgrave Communications, 4(12), 2018. Dean Karlan and Jonathan Zinman. Microcredit in theory and practice: Using random- ized credit scoring for impact evaluation. Science, 332(6035):1278–1284, 2011.