Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Paulo Correa Technology, and Innovation Reviews in Science, Public Expenditure Guidance Note 93076 Public Expenditure Reviews in Science, Technology, and Innovation A Guidance Note Paulo Correa © 2014 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. CONTENTS Foreword vii Acknowledgments ix Acronyms and Abbreviations xi Executive Summary xv 1. Introduction 1 2. Why Public Expenditures on STI 3 2.1. Case Study 1: Increasing Public Expenditures on R&D 3 2.2. Case Study 2: Promoting Collaboration between Research Institutions and Industry 4 2.3. What Can Go Wrong? 4 2.4. What is the Basic Objective? 5 2.5. How this Guidance Note Helps 6 3. Framework 7 3.1. A Results-Orientated Approach 8 3.2. Structuring Questions 12 3.3. The Type of Recommendations 14 3.4. Implementation Issues: A Summary 16 4. Inception Report 19 4.1. Country Paper 20 4.2. Data Assessment 25 4.3. Conclusion 28 Annex A: Comparison of Innovation Frameworks According to Main Content Categories 30 5. Functional Review 37 5.1. STI Budget Structure 39 5.2. A Practical Example 42 5.3. Other Existing Indicators 47 5.4. Conclusion 51 6. Operational Efficiency Assessment 52 6.1. Overview 53 6.2. Output Assessment 56 6.3. Assessment Questions 57 6.4. Methodological Issues 59 6.5. Conclusion 63 Annex A. Five Impact Evaluation Techniques 65 Contents iii 7. Effectiveness Assessment 66 7.1. Overview 67 7.2. Linking Inputs with Outcomes 68 7.3. Research Excellence 69 7.4. Science-Industry Collaboration and Technology Transfer 71 7.5. Business R&D, Startup Creation, and Technology Adoption 75 7.6. Conclusion 81 8. Final Report 83 8.1. Policy Relevance, Coherence, and Consistency 84 8.2. Composition and Level of R&D Spending 88 8.3. Governance Analysis 93 8.4. Institutional Reforms, Policy Recommendations, and Strategic Investments 98 8.5. Conclusion 98 Annex A: Action Plan Illustration from the Western Balkans Regional R&D Strategy for Innovation 99 9. Conclusions 101 Appendixes 103 Appendix A. Definitions 103 Appendix B. Data and Data Sources on Science, Technology and Innovation 105 Appendix C. Information on Country Performance and Innovation Benchmarks 113 References 118 Boxes 1.1: Definition of R&D Activities 1 3.1: Productivity as the Default Development Goal 9 3.2: Three Default Intermediate Outcomes 10 3.3: The “Policy Mix” Concept 13 4.1: Thailand’s Cassava Exports 22 4.2: Structure of the Research and Innovation System in Turkey 24 4.3: The World Bank Science, Technology, and Innovation Database 26 4.4: The Inception Report—Possible Structure and Useful Readings 29 5.1: Government STI Spending in Turkey According to Implementing Agency and Programs, 2005–08 38 5.2: GBAORD—Concept, Statistical Description, and Use 48 5.3: The Functional Review—Structure and Useful Readings 51 6.1: Challenges to Assessing the Efficiency of Programs 53 6.2: How the Call for Proposals and the Project Evaluation Stages May Affect Program Efficiency 56 6.3: Assessing the Outputs of STI Program with a Survey of Beneficiaries—Illustration from Poland Mid-term Evaluation 58 6.4: Assessing the Impact of Croatia’s RAZUM Program 62 6.5: The Operational Efficiency Review—Structure and Useful Readings 64 7.1: IP Regulation in the United States and Denmark 73 iv Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note 7.2: Knowledge-based Start-ups and the Valley of Death 77 7.3: International Good Practices for the Provision of Technology Extension Services 80 7.4: The Effectiveness Assessment—Structure and Useful Readings 82 8.1: Macro-Level Analysis on Social Returns to R&D, Infrastructure, and Human Capital 91 8.2: The Cost of Reaching the Two Percent of GDP Target for R&D in Turkey in 2009 93 8.3: Features of Well-Performing National Innovation Systems, as Outlined in the EU 2020 95 8.4: Addressing Horizontal Bottlenecks in Innovation Policy 96 8.5: The Final Report—Structure and Useful Readings 98 Figures 2.1: A Simplified View of Innovation Systems 5 3.1: The IOOI Approach to Assessment of Public Policies 7 3.2: Spending and Goals 8 3.3: Spending, Outputs, and Goals 8 3.4: The IOOI Model—Proposed Results Framework 10 3.5: The Structure of the Public Expenditure Review on STI 17 4.1: The Inception Report 19 4.2: Illustration of Different Measures of Countries’ NIS 23 B4.2.1: Overview of Turkey’s Research and Innovation System Governance Structure 24 5.1: The Functional Review 37 5.2: An STI Policy Taxonomy 44 B5.2.1: GBAORD as a Share of Total General Government Expenditure, 2010 (percent) 48 6.1: The Operational Efficiency Assessment 52 6.2: Challenges in Program Evaluation 54 B6.3.1: Other Benefits 58 B6.4.1: Opinions of Consequences of Not Receiving RAZUM Grant: Beneficiaries (n=20) 63 B6.4.2: Opinions of Consequences of Not Receiving RAZUM Grant: Non-beneficiaries (n=14) 63 7.1: The Effectiveness Assessment 66 7.2: Research Commercialization 72 B7.2.1: Valley of Death 77 8.1: The Final Report 83 8.2: Distribution of Public Support between Public and Private Sectors (2010 or latest year) 89 8.3: Direct and Indirect Support to Business R&D (2008-09 OECD Countries) 89 8.4: Share of Country’s R&D Expenditure Based on Research Type (2008) 90 8.5a: Operating Costs and Salaries in Croatia, 2006–14 (€ million) 90 8.5b: Capital Investments and Project Financing in Croatia, 2006–14 (€ million) 90 8.6: Horizontal and Vertical Coordination Challenges 95 9.1: The Proposed PER Exercise 101 C.1: The Global Innovation Index—Summary Structure 114 C.2: The Knowledge Assessment Methodology 115 C.3: The Innovation Union Index—Summary Structure 117 Contents v Tables 3.1: Expenditure Review—Summary Table 14 3.2: Expenditure Review by Type of Measure 15 3.3: Expenditure Review—Prioritization 16 3.4: PER Data Requirements—Potential Challenge and Proposed Instrument 17 4.1: Industrial Specialization, Knowledge Source, and Technological Capability 21 4.2: Most Recent Innovation Surveys (as of 2013) 27 4A.1: Comparison of Innovation Frameworks According to Main Content Categories 30 B5.1.1: Public Expenditures on Innovation and Technology Programs 38 5.1: Consolidated STI Sector Budget—Simplified Structure 39 5.2: R&D Expenditures by COFOG Classification 40 5.3: COGOF and Economic Classification—Illustration (in US$ 100,000) 41 5.4a: Preparing the STI Budget: Challenges and Proposed Solutions 43 5.4b: Appropriation of Expenditures: Challenges and Proposed Solutions 43 5.5: Example of Categorization of Programs Based on a Standard Taxonomy 45 5.6: Categories of Innovation Programs Based on Intended Outcomes 46 5.7: Illustration—Country Alfa Classification of R&D Expenditures in 2009 (US$ ‘000s) 47 5.8: Some Cases at the Borderline between R&D and Other Industrial Activities 49 5.9: Summary of Recommendations from Frascati Manual on How to Estimate GBAORD 49 5.10: GBAORD—National Data Collection Schemes (as of 2009) 50 6.1a: Input-Output Metrics—Illustration 57 6.1b: Example of Input-Output Indicator 57 B6.3.1: Outputs Generated by Beneficiaries 58 B6.3.2: Mean of Produced Outputs 58 6.2: Summary of Program Evaluation Methodologies 60 6A.1: Empirical Approaches to Impact Evaluation and Other Statistical Techniques 65 7.1a: Intermediate Outcomes—Illustrative Metrics 68 7.1b: Example of Intermediate Outcomes Metrics 69 7.2: Technical Regulations, Standards, Metrology, and Quality 82 8.1: Policy Input, Outputs, and Intermediate Outcomes 85 8.2: Country Alfa Consolidated STI Sector Budget—Illustration (US$ ‘000s) 86 8.3: Policy Relevance, Consistency, and Coherence 87 B8.1.1: Elasticity, Rate of Return, and Optimal Amount of R&D Investment 91 B8.2.1: Scenarios on Public R&D Expenditures 94 8A.1: Example: Excerpts from the Action Plan for the Western Balkans Regional R&D Strategy for Innovation 99 8A.2: Example: Action Plan for Regional Cooperation—Summary 100 9.1: Final PER Report: Possible Structure 102 A.1: Key Definitions 103 B.1: UNESCO STI Studies Series 109 B.2: Indicators Related to Science, Technology, and Innovation Performance 110 vi Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note FOREWORD Developing countries have not fully tapped into their potential, developing countries will need to address all innovative and entrepreneurial potential. If properly mo- those questions and better understand the returns of bilized, this potential can accelerate economic growth, public investments in STI. diversify exports, and create job opportunities. In this way, innovation can play a major role in promoting The purpose of this Guidance Note is to help countries the World Bank’s twin goals of shared prosperity and to assess the quality of public spending on STI. It adopts eradication of poverty. a results-oriented framework, combining the consolida- tion of STI expenditures with the analysis of their main Policy makers in developing countries are increasingly outputs, intermediate outcomes, and developmental aware of this untapped potential, as well as its power impact. The framework proposes the analysis of three to mitigate potential risks imposed by several global main sources of deficiencies: (i) program design/imple- challenges, including climate change and food scar- mentation; (ii) institutional conditions; and the (iii) city. For instance, attention is increasingly been paid to composition and level of public expenditure. The main the positive effect of public investment in agricultural product of this exercise is an integrated set of actionable research and development (R&D) and extension on measures combining institutional reforms with changes agricultural productivity and the income of low-paid in the policy mix (the composition and level of public workers. In order to mobilize untapped innovation potential and address those challenges, developing spending) and strategic investments. countries are spending more on science, technology, and innovation (STI). This note is one of a larger set of products—includ- ing policy notes, firm-level surveys, and a joint global Despite this growing effort, few governments can an- platform with the OECD (the Innovation Policy Plat- swer with confidence basic questions such as how much form)— developed by the World Bank Group to meet is spent on STI, by whom, and to what end. Verifying the the demands of our client countries in this field of in- results of those investments is a major challenge, as is novation policy. We hope you find them useful. assessing the effects of the design and implementation of programs, the existing framework conditions, or the Esperanza Lasagabaster overall policy mix. To be able to tap into their innovation Practice Manager, GTCDR Foreword vii ACKNOWLEDGMENTS This guidance note on the assessment of public ex- The team would like to acknowledge the valuable com- penditures on science, technology, and innovation was ments provided by William F. Maloney (Lead Economist, prepared by a World Bank team led by Paulo Correa World Bank), Carlos E. Piñerúa (Country Manager, (Lead Economist, World Bank). The team comprised World Bank), Jose Guilherme Reis (Program Leader, Carlos Gabriel Hinojosa, Andrew Myburgh, Qursum World Bank), Sylvia Schwaag Serger (Executive Direc- Qasin and Hari Subhash (Consultants), with inputs tor, Vinnova—Sweden), Mark Roland Thomas (Sector from Iwona Borovic (Consultant), Ana Paula Cusolito Manager, World Bank), and peer reviewers. (Economist, World Bank); Marjo Koivisto, Juan Rogers, Ron Myers, Danica Ramljak, Sebastian Penn, and Maria The team would also like to thank Fernando A. Blanco Pluvia Zuniga (Consultants). The team worked under the (Lead Economist, World Bank), Xavier Cirera (Econo- guidance of Esperanza Lasagabaster (Practice Manager, mist, World Bank), Justin Hill (Senior Private Sector World Bank) and Gerardo Corrochano (World Bank, Specialist, World Bank), Leonardo Iacovone (Senior Country Director) in his previous competence as Direc- Economist, World Bank), Giuseppe Iarossi (Lead Evalu- tor of the Innovation, Technology, and Entrepreneur- ation Officer, IEG), and Kiwan Kim (Lead Economist, ship (ITE) Global Practice, Finance and Private Sector World Bank) for their comments on initial versions Development (FPD). of this document. Acknowledgments ix ACRONYMS AND ABBREVIATIONS ASTI Agricultural Science and Technology Indicators BERD Business Enterprise R&D BNDES Brazilian Economic and Social Development Bank BoP Balance of payments BTYK Supreme Council of Science and Technology CEM Country Economic Memorandums CGIAR Consultative Group on International Agricultural Research CIS Community Innovation Survey COFOG Classification of Functions of Government EA Efficiency Assessment EC European Commission EFA Effectiveness Assessment EFTA European Free Trade Association EU European Union FDI Foreign direct investment FINAME Financing of Machinery and Equipment FR Functional Review GBAORD Government Budget Appropriation and Outlays for R&D GCI Global Competitiveness Index GERD Government-financed gross domestic expenditure on R&D GDP Gross domestic product GFS Government Finance Statistics GII Global Innovation Index GMP Good Manufacturing Practice GUF General university funds HACCP Hazard Analysis Critical Control Point ICA Investment Climate Assessments ICTSD International Centre for Trade and Sustainable Development IDB Inter-American Development Bank Acronyms and Abbreviations xi IMF International Monetary Fund INSEAD Institut privé d’enseignement supérieur IOOI Input-output-outcome-impact IP Intellectual property IPR Intellectual property rights IUS Innovation Union Scoreboard KAM Knowledge Assessment Methodology KE Knowledge Economy KEI Knowledge Economy Index KI Knowledge Index KOSGEB Small and Medium Enterprises Development Organization LEA Leader LCU Local currency unit M&E Monitoring and evaluation MoD Ministry of Development MoE Ministry of Economy MoIT Ministry of Industry and Trade MoNE Ministry of National Education MoSIT Ministry of Science, Industry, and Technology MRA Mutual Recognition Agreement MSES Ministry of Science, Education, and Sports MSTI Main Science and Technology Indicators MSTQ Metrology, standards, testing, and quality NABS Nomenclature for the Analysis and Comparison of Scientific Programs and Budgets NCBIR National Center for Research and Development NESTI National Experts on Science and Technology Indicators NIS National innovation system NSI National Survey of Innovation NSO National statistical office OECD Organisation for Economic Co-operation and Development PAR Partner PBS Applied Research Program PER Public expenditure review PRO Public research organization RAS Reimbursable Advisory Service RAZUM Development of the Knowledge-Based Companies R&D Research and development STI Research, development, and innovation SAR Special Autonomous Region SI International System of Units SME Small and medium-sized enterprises S&T Science and technology STEM Science, technology, science, engineering, and mathematics STI Science, technology, and innovation STIDATA Science and Technology Indicators database xii Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note STIP Science, technology, and innovation programs TFP Total factor productivity TPE Turkish Patent Institute TTGV Technology Development Foundation of Turkey TTO Technology transfer offices TUBA Turkish Academy of Sciences TUBITAK Scientific and Technological Research Council of Turkey TURKAK Turkish Accreditation Agency UNCTAD United Nations Conference on Trade and Development UNESCO United Nations Educational, Scientific, and Cultural Organization UNIDO United Nations Industrial Development Organization VAT Value-added tax WEF World Economic Forum WDI World Development Indicators WIPO World Intellectual Property Organization WTO World Trade Organization YOK Council of Higher Education YPK High Planning Council Acronyms and Abbreviations xiii EXECUTIVE SUMMARY This Guidance Note lays out a framework for the as- WHY CONDUCT A PUBLIC sessment of science, technology, and innovation (STI) EXPENDITURE REVIEW ON STI? expenditures in developing countries, with an emphasis on their contribution to economic development. Governments can improve a country’s STI performance by correcting for the externalities and uncertainty in- herent to the process of innovation. For this reason, INNOVATION MATTERS there has been a renewed emphasis on STI policies in developing countries in recent decades. However, few The importance of innovation for economic develop- governments can answer with confidence basic ques- ment is uncontested. It contributes to the twin goals tions of how much is being spent, by whom, for what of shared prosperity and poverty reduction by generat- purpose, and with what results. ing productivity gains that increase employment, raise wages, and improve access of the poor to products and For starters, governments tend to intervene on the as- services. Investing in innovation increases firm capabili- sumption that providing policy inputs (that is, money) ties and facilitates the adoption of new technologies to will automatically lead to the production of the results they desire. They therefore ignore the fact that the improve labor productivity. high-level developmental impacts that they aim to achieve, such as economic growth and job creation, are Innovation is commonly seen as the work of highly often “third order effects”: the indirect consequences educated labor in research and development (R&D) of outputs and outcomes that will only occur under departments, laboratories, or research institutes—and specific circumstances. therefore a “first world” activity. However, innovation is better characterized as the attempt to try out new Governments also underestimate the complexity of or improved products, processes, or ways to do things. the interactions and decision-making dynamics that For this reason, innovation is intrinsically linked to the underpin the development of policy. Many factors— ‘catching up’ of firms and countries, the engine of both internal and external—influence the capacity of economic development. policy to lead to desired results and impacts. After all, Executive Summary xv innovation is a systemic process in that it depends on The definition of those intermediate outcomes enables a variety of interactions among organizations, markets, the analysis to focus on the specific conditions neces- and individuals. sary for the achievement of identified outcomes. Inter- mediate outcomes also work as intermediate “links” between the public spending and their direct output, MAIN ISSUE the ultimate development goal. This note addresses how to assess whether resources This approach could also be applied to a different system reallocated from the market towards STI improve soci- of development goals and intermediate outcomes. In ety’s economic welfare. In other words, are taxpayers this case, adjustments to this framework will be nec- better off because money was spent on STI? essary to reflect the new issues at hand, starting with the corresponding adjustments to the inputs, outputs, While “economic efficiency” is the ultimate test for outcomes, and development goals to be considered. welfare-enhancing public policies, this Guidance Note addresses a more modest objective—namely, how to im- prove the impact of public spending in STI on economic ASSESSMENT QUESTIONS and social development. In other words, it focuses on the quality of public spending in STI. The assessment of public expenditures on STI is based on four main sets of questions: HOW TO ADDRESS IT 1. How much is spent by the government on STI, by whom, with what objectives? This note proposes a results-based framework to logi- cally link inputs, outputs, outcomes, and impacts. The • How much is spent in each of the intermediate development and application of such an instrument is outcomes? In particular, what is the consolidated the essence of the proposed public expenditure review STI budget? This includes the expenditure that is (PER). A PER helps focus analysis on (i) the fact that outside of pure R&D spending. development goals are second- and third- order effects 2. Are STI expenditures generating the expected out- of public spending, and (ii) that the impact of public puts? Are they doing it efficiently? Do programs and spending depends on a number of conditions that are funded activities generate the expected output with not affected by public spending per se. a reasonably level of inputs? • What design and implementation issues are af- DEFINING OUTPUTS, fecting the performance of programs and other OUTCOMES, AND IMPACT STI expenditures? The note proposes that “increasing productivity” (in- 3. Are public expenditures effective? Are outputs cluding labor productivity and total factor productivity or translating into intermediate outcomes? TFP) is the ultimate developmental goal to be achieved. • Which factors beyond the reach of the existing Three corresponding default intermediate outcomes interventions are affecting the emergence of the are identified based on the evidence provided by the expected outcomes? academic literature: (i) research excellence; (ii) collabo- ration of science and industry, including research com- 4. How does the composition and level of public ex- mercialization; and (iii) business innovation, including penditures in STI (the policy mix) affect its impact? STI and technology adoption and diffusion. Is the composition of public expenditures relevant xvi Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note to the country’s development stage, consistent with The main output of the PER is an actionable plan that existing higher-level goals, and coherent in terms of combines institutional (policy and program) reforms, the funded measures? changes in the policy mix (composition and level in • How does the governance of the national inno- public spending), and other strategic investments. vation system impact this allocation of resources? COMPARISON WITH THE NATURE OF PROPOSED SIMILAR EXERCISES SOLUTIONS The proposed PER builds on a large body of PERs By improving the quality of public expenditures in implemented by the World Bank Group and other STI, policy makers could increase economic efficiency. organizations, as well as several other exercises to To achieve that result, recommendations in the PER analyze national innovation systems and national inno- combine program and policy reforms—that aim to vation policies. However, the exercise proposed by this increase the operational efficiency and effectiveness of Guidance Note has two main differences with existing public spending on STI—with budgetary adjustments exercises. First, it seeks to go beyond R&D expenditures that reflect changes in the policy mix to increase its to encompass public investments in innovation—which relevance, consistency, and coherence. In sum, the is especially relevant for developing countries. Second, PER exercise provides recommendations related to the it aims to go beyond the description of the composition following actions: and level of public spending to shed some light on its impact (or how to improve its impact). • Improvement of the design and implementation of selected programs—based on the Efficiency As- HOW TO USE THE GUIDANCE NOTE sessment • Adoption of policy reforms and investments in new This guidance note is composed of nine chapters, start- programs to improve the systemic, institutional, or ing with an introduction. The second chapter provides market conditions for effectiveness—based on the two practical examples of public interventions in STI that Effectiveness Assessment were motivated by good intentions but ended up gener- ating bad outcomes. Chapter 3 describes the proposed • Changes in the policy mix, including recommenda- analytical framework and the remainder of the note is tions about changes in the composition and level dedicated to the implementation of that framework (the of public investments—based on the Policy Mix “how to”). Chapter 4 describes the Inception Report, Assessment and chapter 5 provides for the analysis of a STI budget. • Enhancement of organizations and processes (the The core of the analytical work is described in chapter 6 governance structure), through which research and on the operational efficiency analysis, chapter 7 on the innovation policies are managed—based on the effectiveness analysis, and chapter 8 on the final report Governance Analysis and policy mix analysis. Chapter 9 concludes. Executive Summary xvii CHAPTER 1 INTRODUCTION This Guidance Note lays out a framework for the as- “innovation” also encompasses off-the-frontier innova- sessment of public expenditures in science, technology, tions, that is, the adoption by firms of knowledge and and innovation (STI) in developing countries. Developing its adaptation for local contexts or new uses. Similarly, economies have been paying more attention to the con- in this note the term “research and development” is tribution of STI policies to their development strategies. employed in a broad sense, comprising creative work Consequently, investments in research and innovation undertaken on a systematic basis in order to increase by developing nations have increased substantially in the stock of knowledge and the use of this stock of the past decade. However, governments often lack the knowledge to devise new applications. Box 1.1 provides tools to properly allocate resources, ensure adequate the definitions adopted by the OECD’s 2002 Frascati returns on the spending, or even account for its use. Manual (OECD 2002). Innovation, when seen as the work of highly educated • Off-the-frontier innovations include incremental labor in research and development (R&D) laboratories improvements and innovations in process, product, of large companies or world-class academic institutions, organization, and marketing that may or may not is inevitably seen as a “first world” activity. However, be technology driven. as described in the World Bank’s 2010 report on “In- • At the firm level, R&D activities may or may not novation Policy for Developing Countries,” the term generate innovations but are often the way firms Box 1.1: Definition of R&D Activities Basic Research Experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundation of phenomena and observable facts, without any particular application or use in view. Applied Research Original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily toward a specific practical aim or objective. Experimental Development Systematic work, drawing on existing knowledge gained from research and/or practical experience, which is directed to producing new materials, products or devices, to installing new processes, systems and services, or to improving substantially those already produced or installed. Source: OECD 2002. Introduction 1 turn innovation into a routine in their business model, twin priorities of promoting shared prosperity and rather than a random event. eradicating extreme poverty. Innovation has long been • R&D does not need to be the source of the new recognized as a critical source of economic growth and idea or new knowledge: knowledge will often be an important activity for addressing major development embedded in internationally available capital and challenges, such as food scarcity; access to services, and intermediate goods. climate change (IEG 2013). The framework proposed in this note aims to help The note is organized in nine chapters including this countries to improve the quality of public expenditures introduction. The next two chapters are mainly con- in STI. Conceptually, the problem is to assess whether ceptual, aimed at explaining to the reader the analytical government’s reallocation of resources from the market framework proposed for the assessment of the quality toward STI, through taxation and public expenditures, of public expenditures on STI. Chapter 2 presents the improves economic welfare as compared to the market motivation for the work. Chapter 3 follows with a brief allocation. The focus of the framework is, therefore, on methodological discussion and the reasoning behind maximizing the social and economic returns of public the structure of a PER on STI. Readers interested only expenditures on STI. in the implementation of the PER can perhaps skip most of that discussion and focus on the remaining The approach in this note represents a middle ground five chapters. between two kinds of innovation policy assessments: (i) program-based evaluations, which by design do not Chapters 4–8 address in more detail how to implement take into account the systemic nature of the innovation the proposed framework. Each chapter corresponds to process, and (ii) assessments of innovation systems, one of the five modules constitutive of the PER for STI. which often struggle to establish priorities and condi- Chapter 4 describes the preparation of the Inception tions for impact. Our approach builds on lessons learned Report, chapter 5 address the challenges of building from a variety of World Bank studies, policy dialogues, and analyzing an STI budget, chapters 6 and 7 describe and projects in the area of innovation policy and public assessments of the operational efficiency and efficacy of expenditure reviews (PERs). those expenditures, respectively. Chapter 8 focuses the relevance, coherence, and consistency of the Helping countries improve the quality of public ex- policy mix and, building on the previous sections, penditures in STI is closely linked to the World Bank’s assesses the likely impact of public spending. Chapter 9 concludes the note. 2 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note CHAPTER 2 WHY PUBLIC EXPENDITURES ON STI Imperfect appropriability, information asymmetries, risk, on evidence. For example, one study had estimated that and uncertainty inhibit private investments in science, reaching the 3 percent target as defined by the Lisbon technology, and innovation (STI). Addressing the con- Agenda in the early 2000s1 would increase exports by ditions leading to this underinvestment by the private 13 percent and income by 12 percent above projected sector would likely raise long-term growth. This in turn levels by 2025. In five years, government expenditures provides an overarching rationale for public invest- on R&D doubled, raising the country’s total R&D levels ments. Nevertheless, investment remains at low levels. from 0.5 to more than 1 percent of GDP. For example, the average Organisation for Economic Co-operation and Development (OECD) country’s bud- Would those investments generate the intended out- getary allocation for research and development (R&D) comes in terms of competitiveness and growth? Hardly corresponded to about 0.7 percent of average gross so. In the same period, basic research in country Alfa domestic product (GDP) in 2011. increased from 22 to 44 percent as a share of total investments in R&D. For comparison, basic research Developing countries have been investing more in STI corresponded to 17 percent of total investments in R&D in recent years. However, few governments can answer in the United States and less than 15 percent in Japan with confidence the basic questions of how much is and Israel. While relevant, investments in basic research being spent, by whom, for what purpose, and with are less likely to generate innovation in the near term. what results. In this context, government spending on STI often translates into poor results and modest Moreover, the impact of public investments in R&D on impact, if any, on economic development. The point is economic development depends on how efficiently illustrated briefly with two case studies (the cases are technology is transferred from public research organiza- real—country names are omitted). tions to the market. This transfer is not an automatic process; it depends on a number of institutional and market factors that may or may not be in place. 2.1. CASE STUDY 1: INCREASING PUBLIC EXPENDITURES ON R&D Given the objective of raising export competitiveness, country Alfa would probably be better off by allocating Like other European Union (EU) member states, country a larger share of public resources to subsidize business “Alfa” committed to significantly increase its expendi- R&D, as opposed to basic research. tures on R&D to boost competitiveness and growth. Those expectations were, in principle, well-grounded 1. http://www.euractiv.com/future-eu/lisbon-agenda/article-117510. Why Public Expenditures on STI 3 2.2. CASE STUDY 2: PROMOTING outcomes which are very different from those that policy COLLABORATION BETWEEN makers are aiming for. How do good intentions, often RESEARCH INSTITUTIONS based on solid evidence and implemented through AND INDUSTRY well-known measures, generate poor, unintended consequences? In the early 2000s, “country Beta” aimed to increase collaboration between research institutions and indus- One general reason is that decision making about public try by favoring the location of companies on university policies often assumes that outputs and results will be campuses. To encourage that type of decision, the automatically achieved once the policy input (public government conditioned access to tax breaks on R&D funds) is made available. High-level developmental expenditures to the firm location in those technology impacts (such as competitiveness) are, however, “third- development zones. As physical space became scarce order effects”: they are the indirect consequence of and rent value increased, the government decided to outputs (first-order effects) and outcomes generated subsidize the construction and expansion of such zones. by the intervention under specific circumstances. Assuming that those first-order effects will generate A survey of tenants implemented years after the start second-order effects and that those will be trans- of the program showed, however, that only 4 percent formed into the desired high-level impact is a com- of tenants started collaborating with researchers after mon mistake in policy making. Similarly, omitting the locating in the technology development zones. More- conditions under which public expenditures will reach over, the combination of those measures resulted in a a desired outcome is a major cause of the misuse of supply of science parks the country (when normalized by public funds. the number of researchers or R&D investments) about 6 times larger than the United States—indicating that the Another reason why public spending in STI may fail supply of technology zones in the country was probably to reach the desired impact relates to the fact that excessive. These results are hardly close to the intended innovation is a systemic process. Innovation’s success goal of policy makers. What went wrong? depends on a variety of interactions among organiza- tions, markets, and individuals, comprising a “system” Probably the conditions leading to collaboration be- (see figure 2.1) (Patel and Pavitt 1994). For example: tween universities and industry were misinterpreted. The implicit understanding was that firms’ physical prox- • High-quality educational and research institutions imity to universities would increase research collabora- are less effective in generating economic growth if tion. Studies have shown, however, that firms prefer firms are not capable of making use of either the to be located close to university when collaboration research outputs or the graduates they generate. In already exists, and when that collaboration demands such a context, investing in public research—even if physical proximity. Policy makers possibly got the di- it succeeds in generating academic excellence—will rection of the causality wrong and thus ignored the not contribute to the competitiveness of firms or to factors affecting universities’ decisions to collaborate economic growth. with firms. • Similarly, entrepreneurs may be unable to put good ideas into practice due to the lack of venture 2.3. WHAT CAN GO WRONG? capital or angels investors willing to risk financing and nurturing the endeavor. Again, under these cir- The examples above illustrate how public spending on cumstances, public spending on pre-seed financing STI, even when based on a sound economic rationale, is unlikely to generate innovation and the desired may fail to generate the expected outputs or may lead to economic impacts. 4 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Figure 2.1: A Simplified View of Innovation Systems DEMAND CULTURAL FRAMEWORK Consumers (final demand) Entrepreneurial attitudes, propensity to innovate Producers (intermediate demand) and to take risk, mobility EDUCATIONAL POLITICAL AND RESEARCH SYSTEMS COMPANY SYSTEMS SYSTEM INTER- Government MEDIARIES Professional Governance Large companies education and Science, Mature SMEs Research training technology and Start -ups and Brokers Higher education innovation spin-offs and research policies Public sector Other policies research FINANCIAL SYSTEM NATURAL CAPITAL LEGAL, REGULATORY, FISCAL Consumers (final SYSTEM Natural resources (biotic and demand) abiotic), Producers (intermediate Regulations and standards IPR quality of environment demand) Source: Technopolis Group & MIOIR 2012. Policy makers also struggle to establish effective coordi- 2.4. WHAT IS THE BASIC OBJECTIVE? nation of STI policies and public expenditures. Policy de- sign and public spending involve different governmental The objective is to assess whether government’s real- organizations—ministries of science, economy, energy, location of resources from the market toward STI, defense, and so forth—often at the same hierarchical through taxation and public expenditures, improves levels. The frequent outcome is some sort of “negative” economic welfare as compared to the steady-state situ- coordination between organizations, whereby each respects the others’ commitments but does nothing to ation. In other words, are taxpayers are better off after integrate its actions. This result is hardly consistent with the intervention, as compared to their welfare without good principles of policy making. the intervention. Why Public Expenditures on STI 5 In practice, the objective is to help countries improve country (demand, cultural framework, political system) the quality of their public expenditures. This includes as well as groups of stakeholders (company system, building governments’ capacity to identify how much educational and research system) and certain types of is spent, by whom, and for what objectives. Another organizations (intermediaries). The figure outlines the practical goal is to improve the government’s capacity factors constitutive of a national innovation system to assess the likely contribution of public expenditures but does not provide any guidance on how to identify to the country’s economic development. the impact of existing policies or the conditions that are missing.2 2.5. HOW THIS GUIDANCE Let’s see an example of how the lack of an assessment NOTE HELPS framework affects policy planning. An adequate supply of human capital (scientists, engineers, technicians) is None of the existing methodologies for assessing a requisite for well-functioning innovation systems. In innovation policies focus on the impact of public ex- fact, it turns out that the country has fewer engineers penditures on STI. Program-based evaluations have a than expected given its development level. But this does narrower approach by design and are not expected to not necessarily imply that access to human capital is a capture the issues related to the systemic nature of in- constraint for the impact of public expenditures on STI. novation. On the other hand, most country-level analy- Rather if the demand for engineers has been systemati- ses of innovation systems often struggle in establishing cally low (as evidenced for instance by the evolution of priorities and conditions for impact, due to the mostly wages in that segment of the labor market), then the descriptive nature of the approach used. effectiveness of public expenditures supporting innova- tion would not be affected by the supply of engineers. An approach that combines both program and country elements is far from straightforward, for a number of A framework for the assessment of public expenditures reasons. The expansion of the range of objectives of on STI should guide the analyst like a map through the innovation policy and of the bundles of instruments landscape of the innovation system. Landmarks in that deployed has made for an increasingly complex policy system include stakeholders, organizations, programs, landscape. This widening of the “frame” of innovation and policies; economic, institutional, and political policy has led to new rationales for policy intervention environments; and the complex interactions between and has opened up a larger toolbox of policy instru- these entities. Developing this assessment framework ments. Beyond core innovation policies, such as those and providing guidance on how to implement it is the targeting science and technology (S&T) and education, main goal of this note. the impacts of other policies must be taken into ac- count. Taxation policy, competition laws and regula- 2. An important exception to the lack of impact evaluation is the tions, and so forth constitute the framework conditions recent OECD studies focusing on the mapping of countries’ policy for innovation (OECD 2012). mixes. OECD is developing a database with quantified information about selected policy instruments. The information is obtained by means of a policy questionnaire and the study has been implemented Referring to figure 2.1, the boxes represent areas of to a few developing countries. See http://stats.oecd.org/Index. the economic, institutional, and political conditions of a aspx?DataSetCode=REG_INNO_TL. 6 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note CHAPTER 3 FRAMEWORK The framework proposed in this note will help correct effects), consistent with the defined development the usual misconception that the impacts of policy impact are identified. This establishes the causal links inputs are the direct, immediate result of the interven- between the intervention and the overall impact—the tion. This approach is illustrated in figure 3.1, which logic of the intervention. depicts the structure of a generic logical framework for the assessment of public policies in general, using the The model can be decomposed into additional steps if input-output-outcome-impact (IOOI) model. useful. For instance, the “Inputs” box could be sepa- rated into three separate issues: what are the interven- Figure 3.1 shows that the process starts with identifi- tions, why they are chosen (economic rationale), and cation of the desired development impact. The arrows what is the expected benefit. Or one could include an indicate the backward-induction process adopted to “Activities” box between the “Inputs” and “Outputs” help identify the causal links between the interven- boxes if there is an interest in identifying the actions tion and the high-level goals. Intermediate outcomes to be taken through which the inputs are mobilized to (events that are immediate prerequisites for impact— generate specific outputs. As a rule, however, there is second-order effects) and outputs (results derived no benefit in describing all possible points. A map that directly from the intervention that may or may not exhaustively describes the environment is not always contribute to the intermediate outcome—first-order useful for navigation. Figure 3.1: The IOOI Approach to Assessment of Public Policies Inputs Outputs Outcomes Impact What are What are the How can What are the interventions? immediate results development goals developmental, high- of the be decomposed interventions? level goals of the How do you expect into measurable goals? interventions them to contribute Do they contribute (expenditures)? to the defined to the defined outputs? outcomes? Framework 7 The main advantage of this approach, as will be dis- one specific logical framework with different inputs and cussed later, is to provide an analytical framework that outputs to be considered, setting up the scope as well is results oriented—a perspective that is missing in most as the structure of the assessment to be undertaken. assessments of innovation systems. How can the IOOI Figure 3.2 and 3.3 describe a step-by-step application approach be applied to the case of public spending in of the IOOI methodology. science, technology, and innovation (STI)? The develop- ment and application of these instruments to assess the Figure 3.2 depicts the initial situation in which a given impact of a given set of public expenditures in STI is the amount of public expenditure on STI is expected to essence of the proposed framework. generate broad, often generic developmental goals. Figure 3.3 depicts the initial decompositions of the problem. First, public expenditures are divided according 3.1. A RESULTS-ORIENTATED to the beneficiary, namely, expenditures benefiting the APPROACH public sector and expenditures benefiting the private sector. In addition, the development objective is turned Development Impact into innovation (new, better, less expensive goods and As a general methodology, the IOOI model can be ap- services) and productivity growth. plied to any developmental impact or high-level goal, from export diversification to shared prosperity or Productivity growth is proposed as the “default” eradication of poverty. Each high-level goal will generate developmental goal. Box 3.1 discusses the rationale Figure 3.2: Spending and Goals Competitiveness, Public Spending Exports, in R&D and Impact Growth, Jobs, Innovation Servicing the Poor Note: ES = enterprise sector; PROs = public research organizations Figure 3.3: Spending, Outputs, and Goals Spending in the Outputs from Enterprise programs in the Public Sector Enterprise Sector Innovation: Spending New better Productivity in R&D products and growth and services Innovation Outputs from Spending in programs and PROs other spending in PROs Inputs Outputs Impact 8 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 3.1: Productivity as the Default Development Goal As Nobel laureate Paul Krugman said once, “productivity isn’t everything, but in the long run it is almost everything.” In simple terms, productivity is efficiency in production: how much output is obtained from a set of inputs. Generally speak- ing, the higher the productivity of a country—the more a country produces for a given set of inputs—the higher the living standards that it can afford and the more options it has to choose from to improve well-being. Labor productivity is the most common measure of productivity. Economists have found that labor productivity growth depends on two major factors: the accumulation of capital (human, physical, and so forth) and the growth of an unex- plained (or residual) portion that arguably reflects advances in production technologies and processes, referred to as total factor productivity (TFP) growth (for a review see Shackleton 2013). In fact, because capital is known to have decreasing returns, sustained growth in the long run depends primarily on TFP growth. In this sense, productivity growth is closely associated with innovation—with the invention of new products, tools, and technical processes that not only reduce the cost of extracting or producing raw materials and energy but also reduce the cost of transforming those inputs into finished products. In this sense, by promoting innovation, policies may also contribute to TFP and labor productivity growth. • Private-sector nonfarm TFP in the United States, which could be considered the technological frontier, has grown at an average annual rate of 1.6–1.8 percent. • The link between R&D, innovation, and productivity for developed economies has been established in a number of studies as described in Hall and Rosenberg (2010). • In Latin America, it has been shown that product innovation has a positive impact on employment growth, compen- sating for the neutral or negative effects of process innovation (Crespi and Tacsir 2010). STI policies often involve multiple development goals such as improving international competitiveness, increasing exports, raising per capita income, and increasing productivity. Public investment in STI may also have noneconomic objectives, such as those related to the environment and social sectors (health, education, basic services, and so forth). As the ultimate factor driving economic growth and rising living standards, productivity growth (labor productivity or TFP growth) seems a good candidate for the default impact or high-level goal of inputs—against which public expendi- tures on STI can be evaluated. While a relatively straightforward concept, a host of measurement issues emerge when constructing productivity indicators from actual production data. For a review of the concept, metrics and determinants, see Syverson (2011). Using micro-level data from manufacturing industries, Saliola and Seker (2011) estimate TFP levels for 80 developing countries from Eastern Europe, Central Asia, Latin America, Africa, and Asia. The study also estimates separate TFP values obtained at the industry level. These industry-level estimates are the most useful for policy makers in that they reveal comparative advantages of specific industries within countries. Source: OECD 2002. of choosing productivity as the default development be taken into account are (i) research excellence, (ii) goal. Other goals may be preferred depending on the science-industry collaboration and technology transfer, circumstances of this exercise. and (iii) business innovation. The analyst can further decompose intermediate outcomes if useful. Business Intermediate Outcomes innovation is decomposed into R&D and non-R&D in- Figure 3.4 completes the logical framework linking pub- novation (but other intermediate outcomes may prob- lic expenditures to immediate outputs and intermediate ably be collapsed to one of these three intermediate outcomes. Taking productivity as the highest develop- outcomes). Box 3.2 discusses the rationale for choosing ment objective, the three intermediate outcomes to these three intermediate outcomes. Framework 9 Figure 3.4: The IOOI Model—Proposed Results Framework Non-R&D-based Innovation, Technology Adoption Spending in and Diffusion Outputs from the Programs in the Enterprise Business R&D and Enterprise Sector Sector R&D-based Innovation Innovation: Public Spending in R&D and New, Better Productivity Innovation Products and Growth Expenditures for Services Technology Transfer and Science-Industry Collaboration Outputs from Spending in Programs and other PROs Spending in PROs Research Excellence Inputs Outputs Outcomes Impact Box 3.2: Three Default Intermediate Outcomes Research Excellence. In developing countries, innovation depends to a large extent on the ability to recognize, assimilate, and apply the value of new, external information to commercial ends (absorptive capacity).a Given that “knowledge” is a public good, public research can play an important role in building the country’s learning and innovative capacity by rais- ing the level of extra-industry knowledge. In this sense, public research can have a direct influence on firm productivity. • Consider the impact of public agricultural research, especially when combined with effective agricultural extension services. Recent simulations illustrate substantial gains in agricultural productivity to be achieved by higher R&D investments over the next 20 years in East Africa (Nin-Pratt 2011). • Agricultural research is particularly needed to help developing countries to address the challenges of climate change Research plays an important role creating the necessary technologies, and enabling developing countries to adapt them to their agricultural systems (Lybbett and Sumner 2010). • In addition, government investments in research provide training to graduates and scientists, some of whom join the private sector, raising its capacity to use new tools and knowledge to solve complex problems.b • Evidence suggests that university research has a significant effect on corporate patents, innovation, and productivity (the latter with a 20-year delay) (Adams 1990; Jaffe 1989). Science-Industry Collaboration and Technology Transfer. A second intermediate output through which public research can contribute to firm productivity is through efficient commercialization of research outputs and collaboration with the business sector—that is, efficient technology transfer from public research organizations (PROs). Sustainable impact of public R&D expenditures on economic development depends on the way the research results of public investment are transferred to the market through patents, licenses, joint ventures, or spin-off companies. The problem is not so much the existence or nonexistence of commercialization activity but whether the conditions for a massive and systemic (as oppose to rare and occasional) process of research commercialization are in place (Audretsch et al. 2010). Science-industry collaboration, from joint and contract research to training and technical consultancy, is a way to commer- cialize research capacity or, more broadly, knowledge that is available in PROs. For example, science collaboration in R&D activities can leverage technological spillovers through the stimulation of additional private R&D investment (Rosenberg and Nelson 1994). Small firms also use research alliances to gain access to research inputs, which would have been otherwise unavailable (Audretsch and Feldman 1996). (continued next page) 10 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 3.2 (continued) • Scott (2003) finds that research alliances with universities are a way through which firms improve their absorptive capacity. This is particularly true for firms that have downsized their R&D facilities. • Mueller (2006) uses a cross-sectional time series to find that regions with high levels of entrepreneurship and university-industry relationships experience a greater level of economic productivity (and growth). Business Innovation: R&D, Startup Creation and Technology Adoption. Most productivity gains emerge from day- to-day introduction of technological solutions and incremental improvements to products and processes of production (as opposed to new discoveries and patenting) (Trajtenberg 2006). Those improvements can occur routinely, in a structured way, or randomly. Business investments in R&D are a way to turn innovation into a routine activity for firms. Naturally, firms can innovative without R&D activities. • Studies have also shown that business investments in research and innovation have a positive and non-negligible impact on labor productivity growth in Latin America (Crespi and Zuniga (2010) and Crépon et al. (1998)) Investments in R&D are fundamental to enhancing a country’s “absorptive capacity.” Prior knowledge gives one the ability to acquire new information. But firms that have their own R&D are better able to use external information. Moreover, a firm’s absorptive capacity may be a byproduct of R&D investments; that includes the ability to adapt and adopt foreign technology, to benefit from spillover effects from foreign direct investment (FDI), and to gain from other sources of knowledge transfer. Start-ups, which very often do not result from R&D, are at the forefront of innovation. They introduce breakthrough technologies, have taken risks in nascent sectors such as the Internet and biotechnology in the past, and are active in areas such as nanotechnology today. They also play an important role in job creation. Kane (2010) finds that for the period 1977 to 2005, start-ups on average created 3 million jobs annually while existing firms lost around 1 million (Kane 2010). The adoption of modern technology is a way of introducing new products or services to the market. Increasing the amount of capital per worker (capital accumulation) is known to be a primary source of labor productivity growth. Differences in technology adoption have been shown to be an important determinant of the gaps in growth and per capita income across countries (Hall and Jones, 1999; Prescott, 1998). In fact, recent studies have shown that: • Differences in the rate of technology diffusion over the past two centuries can account for at least a quarter of the differences in per capita income across countries (Comin and Hobijn 2010). • About 45 percent of cross-country variation in income per capita can be explained by technology adoption at the intensive margin.c. • Countries that have caught up with the United States have been those that saw an acceleration in adoption of new technologies (Comin and Hobijn 2010). • Technology adoption is also an easy way for firms in a developing country to absorb technology that is being devel- oped and used in economically advanced countries. This has become particularly true during the era of globalization as there is broader availability of technology and new machinery (Eaton and Kortum 2001; Keller 2004). Notes: a. The term refers to “the ability to recognize the value of new information, assimilate it, and apply it to commercial ends” (Cohen and Levinthal 1990). See also Griffith, Redding, and Van Reenen (2003). b. For instance, the biotechnology industry has seen a sharp increase in the number of firms from the 1970s. Zucker et. al. (1998) show that that this was due to the diffusion of trained human capital made available through basic research investments in this field. c. Comin and Mestieri (2010) describe the intensive margin as follows: “Once a technology has been introduced, the intensive margin of adoption captures how many units of the good embodying it are demanded relative to aggregate demand. The intensive margin is determined by the productivity and price of goods that embody the technology and the cost that individual producers face in learning how to use it. Other things equal, these variables produce vertical shifts in the evolution of observable measures of technology adoption.” Framework 11 While some marginal overlap is inevitable, these are which generate outputs efficiently (programs and fairly distinctive intermediate outcomes. Their respective funded activities are operationally efficient) contributions to productivity growth are well docu- (ii) Effectively generate the expected intermediate mented in the academic literature, which makes one outcomes from these outputs (as conditions for confident about the causal links. For example: effectiveness beyond the public spending itself are presented) • Public research organizations (PROs) that are sup- ported by public spending need first to focus on In addition, STI policies should be relevant, coherent, research excellence. Then, knowledge accumulated and consistent. As discussed before, the performance in PROs needs to be transferred to the private sec- of a national innovation system (NIS) is recognizably tor (through technology transfer or science-industry related to its systemic nature. Consequently, the impact collaboration). Without the intermediate outcomes of a given policy instrument frequently depends upon of research excellence, PROs are unlikely to reach its interaction with other instruments that may or may higher development impacts. not exist. Moreover, policy measures, designed at dif- ferent occasions and somewhat different goals, are With this framework in mind, it is straightforward to introduced into settings that already contain an array recognize that the transition from output to outcomes of instruments, often with the same or overlapping depends on a number of conditions outside the reach of targets (OECD 2012: 156). the public spending (the original intervention). This may refer to framework conditions (such as a legal framework In such a context, policy design and implementation and regulation of intellectual property), market function- may or may not result in a coherent body of measures ing (for example, for early-stage financing), or institutions consistent with an intended public goal (common good). (such as management of public research organizations). Rather, policy measures may contradict each other, be re- Furthermore, these conditions are often specific to the dundant, too numerous, and operating at an excessively different intermediate outcomes. The point echoes the small scale. Also, there is no reason to assume, a priori, lessons described in chapter 2. For example: that policies will target economic and social goals that are relevant given the country development context. Con- • Investments in technology transfer offices may flicting interests and visions about the role of research not lead to more efficient technology if the rules and innovation, the compartmentalization of policies, for development of researcher’s career favor pure and the piling up of instruments over time are some of academic achievements (such as publications) to the factors hindering STI policies from being relevant. the detriment of collaboration with companies or commercialization of research. More recently, universities have started to explore ways to include 3.2. STRUCTURING QUESTIONS achievements in science-industry collaboration (such This PER on STI is primarily interested in understanding as patenting) as part of the promotion criteria. how governments can spend better in STI—in other words, how governments could improve the “impact” In Economic Terms (contribution) of STI expenditures on economic develop- The underlying hypothesis of this Guidance Note is that ment. From the IOOI model, the proposed core ques- to be welfare enhancing (that is, able to improve eco- tions (in bold) and sub-questions to be asked in assess- nomic efficiency), public spending in STI should achieve ing ways to improve the quality of public spending in STI the following objectives: become straightforward. They are summarized below: (i) Fund research and innovation activities in public • How much is spent by the government in STI, by research organizations or in the business sector whom, and for which expected objectives? 12 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note –– How much is spent in each of the intermedi- measures. Each of the four questions is addressed in ate outcomes? In particular, what is the con- chapters 5–8 of this note. solidated STI budget (going beyond pure R&D spending)? The Governance Structure • Are STI expenditures generating the expected out- The governance structure of STI policies—the orga- puts? Are they doing it efficiently, with a reasonably nizations, institutionalized rules, and procedures for level of inputs? designing STI policies—is an underlying determinant of –– What design and implementation issues are af- the allocation of public spending and its corresponding fecting the performance of programs and other results. Governance matters because policies are not STI expenditures? the result of rational choices from a single policy maker • Are public expenditures effective? Are the outputs (government) acting to maximize the common good. translating into intermediate outcomes? Public policies in general, and economic policy in partic- –– Which factors beyond the reach of the existing ular, are very often the outcome of bargaining processes interventions are affecting the emergence of the involving multiple stakeholders who possess different expected outcomes? access to resources and power, and whose actions re- • How is the composition and level of public ex- flect their private interests in a context of asymmetric penditures (policy mix) affecting its impact? Is the information.1 The issue is particularly relevant given composition of public expenditures relevant to the the multiplicity of actors designing and implementing country’s development stage, consistent with the research and innovation policies. By affecting the be- existing higher-level goals, and coherent in terms havior of stakeholders, different governance structures of the funded measures? induce the development of different “policy mixes” –– How does the governance of the NIS impact this and therefore the quality of public expenditures in allocation of resources? STI (box 3.3). The last question summarizes the core objective of the 1. A broadened concept, encompassing the way that the government PER exercise. It addresses the fact that public spending manages public research organizations (that is, “exercises control”), in the STI sector needs to be analyzed from a systemic arguably would link governance issues to the effectiveness of public point of view, reflecting the interdependence of policy spending on STI (this point is returned to later in the Guidance Note). Box 3.3: The “Policy Mix” Concept The “policy mix” concept, borrowed from other economic policy discussions, is defined as the combination of policy instruments that interact to influence the quantity and quality of STI investments in the public and private sectors. The term implies a focus on interactions and interdependencies between different policies as they affect the extent to which intended outcomes are realized. The assumption is that policy makers are underutilizing the full portfolio of instruments theoretically available to them. The policy mix concept, as intuitive as it may be, lacks clear normative implications. Often the term is associated with no- tions such as “balanced,” “appropriate,” or “effective,” qualities that are hard to define. An abstract “optimum” policy mix toward which the composition of public expenditures could be benchmarked probably does not exist. Rather, the optimum policy—the one that maximizes the impact of public expenditure—must be specific to each country context. This understanding is captured with the notion of “relevance.” “Consistency” between public expenditures and the high-level goals and “coherence” (for example, avoiding redundancy of programs) complete the attributes that this Guidance Note suggests to be used to characterize a balanced police mix. Source: Based on Flanagan, Uyarra, and Laranja (2010). Framework 13 3.3. THE TYPE OF (iv) The enhancement of the organizations and pro- RECOMMENDATIONS cesses (governance structure)—through which research and innovation policies are managed— By improving the quality of public expenditures in STI, based on the Governance Analysis. policy makers could increase economic efficiency. To achieve that result, recommendations on the PER will Table 3.1 provides the suggested structure for the sum- combine program and policy reforms—aiming at increas- mary matrix. It combines the four proposed intermedi- ing the operational efficiency and the effectiveness of ate outcomes (rows) and the four analytical dimensions public spending in STI—with budgetary adjustments that (columns). The matrix is supposed to be filled based on reflect changes in the policy mix to increase its relevance, the assessment carried out through each of the three consistency, and coherence. In sum, the PER exercise modules—efficiency, effectiveness, and policy mix as- provides recommendations for the following actions: sessment (the latter including the governance analysis). Note that the last column is not a necessary element (i) The improvement of the design and implemen- of the regular matrix. Rather, it simply indicates the tation of selected programs—based on the Ef- possibility of consolidating the results of the analysis ficiency Assessment by “intermediate outcome.” Table 3.1 also describes (ii) The adoption of policy reforms and investments some of the expected inputs for its cells: in new programs to improve the systemic, insti- tutional, or market conditions for effectiveness— • Cell (A) brings the broad recommendations from the based on the Effectiveness Assessment policy mix assessment. An example is the need to (iii) Changes in the policy mix, that is, recommenda- rebalance the policy mix toward more investments tions about changes in the composition and level in innovation, particularly non-R&D innovation, in of public investments—based on the Policy Mix order to improve the relevance of the policy mix for Assessment; the country’s development needs. Table 3.1: Expenditure Review—Summary Table Program Intermediate operational Effectiveness Governance outcomes efficiency conditions Policy mix structure (Sector analysis) Research Analysis of the (C) excellence research sector (G) Science-technology collaboration (F) transfer Business R&D and knowledge-based (B) startups Non-R&D business innovation and (E) technology adoption Overall Overall policy mix Overall governance analysis (A) analysis (D) 14 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note • Cells (B) and (C) address other policy mix issues how researchers’ career development is regulated (which happen at the level of intermediate out- to incentivize them to develop joint projects with comes). Examples are the balance between direct the business sector. and indirect subsidy for the business sector (B) and the balance between basic and nonbasic research (C). Tables 3.2 and 3.3 are possible developments of table • Cell (D) presents the recommendations related to 3.1. Table 3.2 helps organize the PER recommendations the overall procedures and organizations involved according to the nature of the measure to be under- taken, namely, program and policy reforms and strategic in policy design and implementation in the NIS. An investments. One advantage is to be able to identify an example is the revitalization of the NSI council and overall cost of the strategic investments by intervention the inclusion of a larger number of participants from (program, policy reform—represented by the vertical the private sector with voting power. arrow) or by intermediate outcome (horizontal arrow). • Cell (E) brings the recommendations for the im- Strategic investments involve, for instance, programs provement of operational efficiency of programs to support managerial training by small firms, invest- for non-R&D based innovation (E). An example is ments in the country’s metrology system, or investments the need to adjust technology support programs to in the modernization of the agricultural research and emphasize labor training and informational issues extension services. (in addition to access to finance). • Cell (F) shows the recommendations for the improve- Table 3.3 helps with the prioritization and ranking of ment of conditions for effectiveness for science- measures. It provides a simple example of a “dash- industry collaboration. An example is to reform board” that could be used in order to identify priority Table 3.2: Expenditure Review by Type of Measure Program operational Conditions for Policy efficiency effectiveness mix Governance structure (Sector analysis) Intermediate outcomes Strategic Strategic Strategic Reforms Program investments Policy investments Policy investments (Program and Strategic reform ($) reform ($) reform ($) policy) investments ($) Research Total costs, excellence Research sector Science- technology collaboration transfer Business R&D and knowledge-based startups Non-R&D business innovation and technology adoption Overall Total costs Overall costs Framework 15 Table 3.3: Expenditure Review—Prioritization Program Intermediate operational Effectiveness Governance outcomes efficiency conditions Policy mix structure Overall prority Research excellence + ++ +++ ++ Science-technology collaboration ++ +++ + transfer Business R&D and knowledge-based ++ + +++ startups Non-R&D business innovation and technology +++ ++ + adoption Overall + ++ +++ +++ priority areas of intervention and/or adjustment. Based on the covered during the PER exercises for data, funding, or assessment carried out through each of the three mod- related limitations). ules, each individual box is to be filled out through the use of a simple rating system. For example, if the analysis 3.4. IMPLEMENTATION led to the conclusion that the overall governance issues ISSUES: A SUMMARY are a main problem, this could be indicated by using a rating system (i.e. +++ = very high, ++ = average, + = Five Stages of Implementation below average). This note proposes that the PER on STI is implemented The final deliverable consists of an Action Plan of policy in five stages. The proposed stages are: (1) Inception and institutional reforms as well strategic investments Report; (2) Functional Review; (3) (Operational) Effi- (for instance, resulting from shifting resources from ciency Assessment; (4) Effectiveness Assessment; and one STI component to another). The Action Plan aims (5) Final Report (see figure 3.5). The Functional Review to enhance the impact of public spending on STI on and the Operational Efficiency and Effectiveness Assess- economic development. ments correspond to three out of the four structuring questions discussed in section 3.2. The Final Report In discussing the main results of the PER exercise and stage addresses the issues of coherence, consistency, considering its effective impact, the team may consider and relevance of policies and consolidates the results including two additional deliverables: (1) an action from the previous sections. Each stage builds on the plan for the development of STI statistics, and (2) a information/analysis obtained in the previous stage. monitoring and evaluation (M&E) system for the STI spending (which enables the country to continue col- The Inception Report stage corresponds broadly to the lecting program data and advance issues that were not preparation of a concept note. In essence it should 16 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Figure 3.5: The Structure of the Public Expenditure Review on STI Inception Functional Operational Effectiveness Final Report Report Review Efficiency Assessment Assessment Policy Mix and Assessment of the Governance Country Context, conditions outside logical framework, Program the reach of Policy Reforms and and data Budget Analysis Evaluation existing Strategic assessment expenditures Investments contain a clear definition of the objectives and scope of challenges are related to the availability and quality of the work, a thorough assessment of data requirements budgetary data and data related to the performance of and availability, and an implementation agreement. the programs. Statistics on STI for developing countries, at aggregate or firm level, are only partially available— Data Requirements very often with a significant time lag, and with quality Access to general STI statistics and data statistics and and comparability issues. Potentially available data on on public spending—is the central challenge for the public spending on R&D are limited by design (catego- implementation of the PER on STI. Table 3.4 summa- rized at 4-digit levels as a government function by the rizes the main data issues by stage of PER. Most of the IMF’s 2001 Government Finance Statistics Manual) (IMF Table 3.4: PER Data Requirements—Potential Challenge and Proposed Instrument Data collection Instrument Data required Potential challenge Proposed instrument Inception Report Data on country’s economic The World Bank WDI database has broad No instrument proposed development and aggregate coverage. Aggregate indicators on NIS also indicators on NIS have a global coverage Functional Review Data on government STI spending Government Budget Appropriation and Data to be generated (public sector budget) Outlays for R&D (GBAORD), the standard through a policy indicator, is available for a limited number questionnaire of countries. GBAORD does not cover non- R&D expenditures related to research or innovation expenditures Operational Efficiency Data on program results from Data may not be available or quality may be Data to be generated Assessment program management poor through a survey of beneficiaries Effectiveness Data on outcomes (scientific Science, technology, and innovation (STI) Data can be partially Assessment excellence; science-industry statistics and innovation surveys (IS)—as generated by a survey of collaboration and technology defined by UNESCO, OECD, and Eurostat— PROs transfer; business investments and national statistical office data will suffice. World Bank Enterprise in R&D; non-R&D innovation; R&D statistics are sometimes unavailable or Surveys (innovation module) technology adoption of poor quality are broadly compatible to Data on science performance may be very the Innovation Surveys expensive Framework 17 2001). Most expenditure data for non-R&D innovation- and technology are natural candidates, one may related programs are buried in the expenditures of other consider the involvement of ministries of economy agencies.2 or finance. • The team may also want to consider forming a • A thorough review of what data is available and consultative group or steering committee with key accessible is therefore essential. Accessibility mat- stakeholders. This not only could facilitate access ters because different parts of the administration to information but also serve as a sounding board not necessarily engaged in the exercise may have for the exercise. In addition, the mechanism could more or less willingness to generate (or gather) and facilitate consensus-building around the proposed provide the required information. measures and facilitate the future implementation • Three main instruments for data collection are: (i) of its recommendations. the policy questionnaire, (ii) a PRO questionnaire, and (iii) a survey of beneficiaries. The extension of the PER is also adjustable to data availability and access. In countries where STI statistics Implementation Arrangements are well advanced and public expenditures are cat- In terms of implementation arrangements, beyond egorized according to international best practices, it standard planning issues, adequate distribution of may be convenient to avoid the burdensome process responsibilities between the team and the govern- of information collection and limit the analysis to the ment counterpart is essential. Ideally, the government available information. Countries covered by Eurostat, counterpart should be responsible for providing ERAWACHT, and OECD—where part of the relevant information available in the public administration information is generated (see chapter 5 or appendix but not publicly available (such as information about B on data sources)—are primary candidates to adopt program budgets, beneficiaries, and outputs), or, at a this approach. minimum, facilitating access to it. While data collect- ing and processing may be time consuming, access The PER can be implemented in a gradual way without to the original registry of information is commonly necessarily aiming to reach the final phase of the full the biggest bottleneck. report. For example, governments can decide to carry out the first stages of analysis (Inception Report and • In this sense, the choice of the counterpart is critical. Functional Review) and draw enough conclusions While ministries in the fields of science, innovation, for policy making. It is evident, however, that the strength of the proposed approach comes from the 2. Indeed, very few, if any, of the public expenditure reviews imple- mented in the past decades reached a level of disaggregation in which implementation of the five stages of analysis of public R&D activities are covered. expenditures in STI. 18 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note CHAPTER 4 INCEPTION REPORT The Inception Report (IR) is the first stage in the imple- The chapter starts with a description of the Country mentation of the Public Expenditure Review (PER) and Paper in section 4.1. This includes a discussion of the provides important background for the rest of the strategic context and benchmarking. This is followed exercise. The information collected for the Inception by a discussion of the data assessment in section 4.2, Report facilitates planning, and provides a foundation including data sources, data sets, and reports. The for reaching an agreement with the counterpart(s) on chapter concludes with a proposed structure for the the approach for implementing the PER (see figure 4.1). Inception Report as well as useful readings. Figure 4.1: The Inception Report Operational Functional Effectiveness Inception Report Efficiency Final Report Review Assessment Assessment Summary • The scope of analysis, including, to the extent possible, the expenditures to be addressed, the programs to be reviewed, the The Inception Report includes: outcomes to be covered, and corresponding indicators—which 1. A Country Paper that may be expressed as an agreement on the logical framework to be adopted by the PER. The scope of analysis will be jointly (a) Describes the country’s economic performance and its main decided by the counterpart and the team based on the discus- challenges sion of the logical framework to be adopted, information (data) (b) Benchmarks the country’s national innovation system (NIS) constraints, the budget and time available for the exercise, as (c) Describes the main organizations, policies, and programs well as the counterpart’s level of engagement. 2. A Review of Data Availability and Accessibility • A conclusion on how to meet the corresponding informational 3. A draft Analytical Framework requirements, which may be expressed in a Data Gathering Plan. 4. An Implementation Plan • The implementation arrangements to be followed by the imple- menting team and the government counterparts. The Inception Report provides the factual basis that can be used to plan the rest of the PER. This includes coming to an agreement The main data sources for the Inception Report are publicly avail- with counterpart(s) on: able data sets and reports. These may be complemented with field interviews and focus groups. Inception Report 19 4.1. COUNTRY PAPER the developing countries which have been more suc- cessful in terms of growth of both exports and output The Country Paper aims to (i) generate analysis about have tended to increase the diversity and sophistica- the country development priorities; (ii) benchmark the tion of the products they produce and export (UNIDO country’s NIS; and (iii) describe the key organizations, 2009). In most cases, when diversification and sophis- policies, programs, and regulations of the country’s NIS tication are coupled, they are an outcome of “moving that are relevant to the subsequent stages of the exer- up the production ladder” from relatively simple mass cise. The strategic context section is useful background manufacturing activities, such as textiles or footwear, information for the discussion of the logical framework. to increasingly complex production processes, such as The overview of key institutions, programs, and policies metal-mechanical, chemical, or electronics industries will serve as the basis for the governance and budget (World Bank 2013). Hence the interest of policy makers analysis in the second stage of the preparation of the on addressing competitiveness as a high-level objective. PER (the functional review). The technological and innovative capabilities that a The Strategic Context country needs to move up the production ladder2 change significantly according to trade specialization In this section of the Country Paper, the analyst is and the level of country development. Some researchers advised to report on the main challenges faced by the argue that a country’s requirements for technological country in its current stage of social and economic de- capabilities become more stringent, particularly with velopment in order to have a preliminary understanding respect to innovation capabilities, as countries climb up of the country’s demand for innovation and technology. the development ladder (Fagerberg et al. 2010). On the other hand, less-developed countries would need more For example, the productive structure that emerged basic capabilities such as managerial training. in some Western Balkan countries after transition toward a market economy—with lower participation Following, World Bank (2013a) and Fagerberg et al. of R&D-intensive sectors such as the pharmaceutical (2010), table 4.1 provides a broad taxonomy articulating industry—reduced the demand for science-industry the type of industrial specialization, the corresponding collaboration and for research commercialization. On knowledge sources, and the associated technologi- the social side, one emerging challenge in some coun- cal capability. For instance, traditional manufacturing tries is the increasing costs in health care associated technological capabilities are more linked to access with an aging population. Such a context is rare in to trained labor and modern machinery at affordable low-income countries in Sub-Saharan Africa, where prices rather than to direct support for business R&D. the population is young and agricultural productivity Agricultural-intensive industries still rely upon access to and the income of impoverished agricultural workers machinery and intermediate inputs as the main source can be increased by adequate agriculture research of innovation. However, they may also benefit from and extension services. Demands for research and public support for agricultural research and extension innovation policy emerging from those two contexts services as well as better enforcement of phytosanitary are likely to be substantively different, as will be the measures. trade-offs faced by each country in addressing social and economic goals. The taxonomy is expected to help the analyst start identifying the country’s needs in terms of innovation Another important issue to be taken into account is and technology at a given moment in time. But during export competitiveness.1 Recent research suggests that 2. That is, “the ability to make effective use of technological knowl- 1. The remaining of the section is based on the World Bank (2013a): edge in efforts to assimilate, use, adapt, and change existing tech- “Trade and Competitiveness Toolkit.” nologies” (Kim 1997). 20 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 4.1: Industrial Specialization, Knowledge Source, and Technological Capability Industrial specialization Modes of innovation Main source of knowledge Technological capabilities Traditional Product innovation Most new techniques originate from Firm access to internationally competitive manufacturing (including design, machinery and chemical industries machinery, equipment, and intermediate logistics, distribution, and goods. Access to global value chains (Textiles and apparel, Most technology is transferred marketing) footwear, furniture, internationally, embodied in capital Managerial and labor skills compatible tiles, etc.) Incremental process goods with adoption of modern technologies and innovation (cost reduction) business practices Product characteristics and quality New designs and branding consistent with international Access to information for product compliance (product differentiation) standards (standards/technical regulations, including packaging and labeling) Certification capacity and internationally recognized certifiers Trademarks regime to enable firms’ appropriate innovation efforts Natural resource- Process innovation (cost Most new techniques originate Firm access to internationally competitive based reduction) from machinery, chemical, and machinery, equipment, and intermediate biotechnology industries goods. Access to global value chains (Sugar, tobacco, Main emphasis on wine, fruit, milk, health (food safety) and Knowledge is transferred Testing laboratory and internationally mining industry) environmental issues internationally, embodied in capital recognized accreditation goods and intermediate goods (such Certification of origin or Patent regulation and an efficient system of as fertilizers, pesticides, seeds, etc.) production technique intellectual property rights (organic products) Knowledge specific to an industry/ Public research system and public investments region/country may need to be in R&D generated Complex products Incremental product and Technological accumulation is Firm access to internationally competitive process innovation generated by the design, building, machinery, equipment, and intermediate (Automobile and and operation of complex goods. Access to global value chains auto components, Radical innovations based production systems or products consumer electronics, on scientific discoveries Labor skill (specialized workers, technicians, pharmaceuticals, Important user-producer interactions. engineers, and researchers) Provision of customized machinery, Learning from advanced users goods and services (e.g. Metrology laboratory upgrading toward equipment, and software, precision High in-house R&D for development internationally recognized accreditation, inter- precision instruments) equipment) of cutting edge technologies calibration schemes Public support to business R&D Public research system and public investments in R&D. Emphasis on research commercialization and science-industry collaboration Sources: World Bank 2013; Fagerberg et al. 2010. a catching-up process, the appropriate level of techno- order to move up the production ladder. Three main logical capability is a moving target in constant need of sources of trade opportunities may be considered: (i) improvement (Bell and Pavitt 1993). Assessing the coun- diversifying geographic markets, (ii) improving product try’s trade opportunities would help one understand sophistication, or (iii) exporting newly created products the capabilities needed by the country in the future in (innovation). Each type of opportunity will generate a Inception Report 21 Box 4.1: Thailand’s Cassava Exports Until 2012, Thailand’s exports of dried cassava to China were not subject to any quality or food safety regulation. Only a minimum level of starch content was required. By contrast, Thailand’s exports of cassava pellets to the EU are required to meet two demanding sets of standards: the Good Manufacturing Practice (GMP) code, covering sanitary and processing procedures, and the Hazard Analysis Critical Control Point (HACCP), as cassava pellets are an input into animal feeds. Thai- land’s successful entry into EC cassava markets required its domestic exporters to develop greater technological capabilities than needed for exporting to China. Source: World Bank 2013. specific set of technological capabilities, as in the case Assessments (ICAs), Trade Outcome Notes, and others. of geographic diversification of Thailand’s cassava ex- They provide a first glance at the competitiveness chal- ports (box 4.1). lenged faced by the country. Data on trade performance and industrial structure Benchmarking the Country’s NIS available from sources such as the World Bank’s World Development Indicators are a preliminary source of in- This section aims to give the PER team a sense of the formation. In addition, the “Trade and Competitiveness strengths and weaknesses of the country’s NIS. There Toolkit” (World Bank 2013) provides a comprehensive is no standard methodology to do such work. Different list of possible indicators, as well as qualitative and organizations present different but largely interchange- quantitative methodologies on how to develop a broad- able methodologies, often based on aggregate indica- er competitiveness assessment. As a starting point, we tors. The indicators try to cover the different aspects of suggest that the analyst focus on the following issues: a national innovation system (including science-industry collaboration, overall investments in R&D, innovation • What is the country’s current trade specializa- performance, and so forth). Four indicators are illus- tion? What are the innovation and technological trated in figure 4.2 for countries in Central and Eastern capabilities required for sustaining existing export Europe. Indicators shown are the Knowledge Economy performance? Index from the World Bank’s Knowledge Assessment • What are the country’s trade opportunities in terms Methodology, the Global Innovation Index, the Global of diversifying geographic markets, increasing prod- Competitiveness Index, and European Union’s Innova- uct sophistication, and introducing new products tion Union Scorecard (UIS). The methodologies for these (non-R&D or R&D-based?) indices are described in appendix C. • What are the innovation and technological capabili- ties needed by the country to explore each of the Overview of Key Institutions, opportunities identified? Programs, and Policies In developing the Country Strategic Context, a number Identify the key research and innovation stakeholders, of World Bank reports on competitiveness, productivity, including policy makers, implementing bodies, and trade, and innovation are potentially a useful start- beneficiaries (including public research institutes, higher ing point. Those topics are often covered by Country education institutions, and the largest beneficiaries in Economic Memorandums (CEMs), Investment Climate private sector). 22 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Figure 4.2: Illustration of Different Measures of Countries’ NIS (a) Knowledge Economy Index (b) GCI’s Global Competitiveness Index 1–7 (best) 9 9 8.1 4.6 4.6 9 8.1 9 8.0 8.0 4.6 4.5 4.5 8 8.1 8 8.1 8.0 8.0 7.4 7.6 7.6 4.6 4.4 4.4 7.4 7.3 Index 7.3 4.5 4.5 Index 7.6 7.6 4.4 4.4 4.4 Index 8 8 6.8 4.4 Index 7.4 7.4 6.8 7.3 7.3 Index Index 7 7 4.4 4.4 4.2 4.2 Index Index 6.8 6.8 6.0 6.0 7 7 5.7 5.7 4.2 4.2 4.1 6 6 6.0 6.0 4.2 4.2 4.1 4.1 4.1 4.1 4.1 4.1 Economy 4.1 Economy 6 5.7 5.7 4.1 4.1 Competitiveness 6 4.2 4.2 4.1 4.1 4.1 4.1 Competitiveness 4.1 Economy 4.1 Economy 5 5 Competitiveness Competitiveness 5 5 4 4 4 4 4 4 4 4 3.8 3.8 3 3 3.8 3.8 3.8 3.8 Knowledge Knowledge 3 3 3.8 3.8 2 Knowledge Knowledge 2 2 2 3.6 3.6 1 1 3.6 3.6 1 1 3.4 3.4 0 0 Global Global 3.4 3.4 Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR 0 0 Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Global Global Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Macedonia, Macedonia, Macedonia, Macedonia, Slovak Slovak Slovak Slovak Macedonia, Macedonia, Macedonia, Macedonia, Slovak Slovak Slovak Slovak Czech Czech Czech Czech Czech Czech Czech Czech Source: Knowledge Assessment Methodology 2012 (www.worldbank.org Source: Global Competitiveness Index (http://www.weforum.org/issues /KAM) /competitiveness-0/gci2012-data-platform/) (c) Global Innovation Index (d) Innovation Union Scorecard 60 60 0.5 0.5 0.4 0.4 60 60 0.5 0.5 0.4 0.4 50.2 50.2 0.4 0.4 Scorecard Scorecard 50 50 0.4 0.4 Index 0.4 0.4 0.4 Index 50.2 44.6 50.2 44.6 0.4 Scorecard Scorecard 50 50 0.4 0.4 0.3 0.3 0.4 0.4 Index Index 41.9 40.7 0.4 0.4 44.6 40.6 41.9 40.7 44.6 40.6 0.4 0.4 0.3 0.3 0.3 0.3 40 40 40.6 38.1 38.1 40.7 40.6 41.9 41.9 40.7 36.9 0.3 0.3 0.3 0.3 0.3 35.9 35.9 36.9 0.3 Innovation Innovation 40 40 38.1 38.1 35.9 36.9 0.3 0.3 0.3 0.3 35.9 36.9 0.2 0.2 0.2 0.2 Innovation Innovation 0.3 0.3 0.2 0.2 Union Union 30 30 0.3 0.3 0.2 0.2 Union Union 30 30 0.2 0.2 20 0.2 0.2 20 0.2 0.2 Innovation Innovation 20 20 0.2 0.2 Global Global Innovation Innovation 10 0.1 0.1 Global Global 10 0.1 0.1 10 10 0.1 0.1 0 0 0.1 0.1 0.0 0.0 0 0 Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR 0.0 0.0 Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Republic Hungary Poland Romania Rep. Croatia Serbia FYR Macedonia, Macedonia, Slovak Slovak Macedonia, Macedonia, Slovak Slovak Macedonia, Macedonia, Slovak Slovak Macedonia, Macedonia, Slovak Slovak Czech Czech Czech Czech Czech Czech Czech Czech Source: The Global Innovation Index (http://www.globalinnovationindex.org Source: Innovation Union Scorecard (http://ec.europa.eu/enterprise/policies /content.aspx?page=data-analysis) /innovation/policy/innovation-scoreboard/index_en.htm) Collect the key strategic documents (those establish- • What is the overall governance structure for manag- ing long-term vision, priorities, and so forth); the main ing research and innovation policies in the country? legal documents (related to key stakeholders, mandate • Who are the key innovation policy stakeholders? of policy making bodies, and so forth); the list of main Organize them using a hierarchical map that out- policies, programs, and their respective regulations; and lines the governance structure and provides detailed documentation of the composition of policy-making descriptions of roles and mandates (as illustrated for bodies, regulations, and mandates. the case of Turkey in box 4.2). Inception Report 23 Box 4.2: Structure of the Research and Innovation System in Turkey At the political level, the Turkish research system is led by the Supreme Council of Science and Technology (BTYK), a legally formalized body chaired by the prime minister. The BTYK determines, directs, and coordinates research and innova- tion policies, and is composed of relevant ministers, heads of public and private bodies, universities, and nongovernmental organizations. The Scientific and Technological Research Council of Turkey (TUBITAK) is affiliated to the Ministry of Science, Industry, and Technology (MoSIT) and acts as the secretariat of the BTYK. The Ministry of Development (MoD) and the High Planning Council (YPK) are two other important actors in the design and implementation of STI policies. The Ministry of National Education (MoNE) and the Council of Higher Education (YOK) design and implement education policies, and integrate them with research policies. The Turkish Academy of Sciences (TUBA) determines and recommends scientific priority areas and proposes legislation to the government on issues related to scientists and researchers. Figure B4.2.1: Overview of Turkey’s Research and Innovation System Governance Structure President of the Republic Prime Minister MoNE MoD MoF MoE MoSIT BTYK YOK TURKSTAT KOSGEB TUBITAK TURKAK TUBA TPE HM RDAs TTGV TOBB BTYK: Supreme Council of Science of TUBITAK: Scientific and Technological TURKSTAT: Turkish Statistical Institute Technology Research Council of Turkey RDAs: Regional Development MoF: Ministry of Finance KOSGEB: SME Development and Support Agencies Organization MoSIT: Ministry of Science, Industry TTGV: Technology Development and Technology HM: Undersecretariat of Treasury Foundation of Turkey MoNE: Ministry of National Education TURKAK: Turkish Accreditation Agency YOK: Higher Education Council MoD: Ministry of Development TPE: Turkish Patent Institute TUBA: Turkish Academy of Sciences TOBB: Union of Chambers and MoE: Ministry of Economy TSE: Turkish Standards Institute Commodity Exchange of Turkey At the operational level, the leading actor in the system is TUBITAK. It designs and implements programs to increase R&D activities of the public and private sectors and universities. The Small and Medium Enterprises Development Organi- zation (KOSGEB) and the Technology Development Foundation of Turkey (TTGV) are the other main bodies implementing industrial R&D support measures. The Turkish Patent Institute (TPE) carries out the procedures related to industrial and intellectual property rights. The Turkish Accreditation Agency (TURKAK) deals with the accreditation of organizations and laboratories. The primary research performer in the public sector is the Marmara Research Center of TUBITAK. It provides contractual research, testing, training, consultancy, analysis, and certification services in its research centers, and operates a technopark. TUBITAK’s institutes are the most active research organizations conducting research in their fields of special- ization. For nuclear research activities, the Turkish Atomic Energy Authority is the main body both for strategy preparation and for carrying out research activities. There are also the R&D centers operating under universities and various ministries, such as the ministries of Energy and Natural Resources, and Food, Agriculture and Livestock. Apart from public research agencies, the private sector established R&D centers in the context of Law No. 5746, which concerns the support of research and development activities. Moreover, the gains acquired from application of Technology Developing Zones Law No. 4691 create potential for private sector R&D. Source: Erdil and Çetin 2014. 24 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note 4.2. DATA ASSESSMENT among countries because, despite the widespread use of the Frascati Manual, significant usage gaps remain, Data quality, access, and the possibility of gathering especially in Africa, Central and South Asia, and Latin unavailable data affect the scope and depth of the appli- America and the Caribbean (OECD 2002). Data avail- cation of any given logical framework. Besides helping ability varies because the surveys of beneficiaries (at establish the final scope and depth of the overall PER, the point of use) are costly to implement. Countries the data assessment framework will help the team to with more fragile statistical systems, therefore, will decide if and when to start surveys and questionnaires. likely lack the resources to develop this information. When available, data provision will inevitably oc- STI data is normally generated by the national statistical cur with significant time lags (of two or more years offices through surveys of beneficiaries (at the “point sometimes, even for developed countries). Yet data of use”), following the methodology established by the coverage has improved recently: for instance, by the OECD (2002) Frascati Manual. mid-2000s, UNESCO estimated that some R&D data was available for more than half of African countries The Frascati Manual describes how to divide R&D ex- (Ellis 2008). penditure and employment into different categories. This includes the types of R&D, basic research. applied The feasibility of data collection (time, cost, and qual- research and experimental development, and the insti- ity) needs to be carefully considered. Relevant consid- tution conducting it namely business enterprises, gov- erations include the likelihood of local collaboration ernment (as is done with GBAORD), private non-profit, and willingness to bare the primary responsibility for higher education, and expenditure on institutions outside forging the collaboration with third parties within the the country or abroad. A number of definitions from the public administration. Frascati Manual are reproduced in appendix A. The main indicators from the Frascati Manual are as follows: For each of the following types of data—R&D statistics; innovation survey; budgetary information; program re- • Indicators for expenditure include: porting; beneficiary’s data; agreed outputs; and agreed –– Gross domestic expenditure on R&D (GERD) is outcomes—assess: the total intramural expenditure on R&D where intramural expenditures are all those performed • What data is publicly available and the quality of within the economy during a specific period the data (STI statistics) –– Gross national expenditure on R&D (GNERD) • Data availability and accessibility of data not directly comprises total expenditure on R&D financed by available (budgetary information) a country’s institutions and so includes expen- diture on R&D performed outside the country STI Statistics –– Government budget appropriations or outlays STI data is generated by the national statistical offices for R&D (GBAORD) this aggregates expenditure through surveys of beneficiaries (at the point of use), by government following the methodology established by the Frascati • Employment on R&D, including those workers Manual. The manual, originally written for the national employed directly on R&D as well as those providing experts in OECD member countries who collect and direct services such as R&D managers, administrators, issue national R&D data, became the standard for and clerical staff. conducting R&D surveys and data collection in other UN member states, for example through the science STI data in developing countries have limitations in and technology (S&T) surveys of the UNESCO Institute terms of quality and availability. Data quality varies for Statistics (UIS) (NESTI 2011). Inception Report 25 Box 4.3: The World Bank Science, Technology, and Innovation Database A useful source of data is the World Bank Science, Technology and Innovation (STI) Database—a “one-stop shop” for macro- and micro-level datasets on STI and entrepreneurship indicators. It aggregates 15 data sources, including some with world-wide coverage. The database comprises more than 500,000 records covering 180 countries and providing insight into almost 600 indicators. The indicators organized by source and categories and user-friendly devices allow the generation of country-level summaries. The website is: http://fpdweb.worldbank.org/units/fpdvp/fiedr/sti/Pages/Home.aspx.c. • UIS collects S&T data from more than 200 countries Table 4.2 presents a summary of the most recent na- around the work through biennial R&D surveys and tional innovation survey that was carried out by a list of through partnerships with other statistical organiza- non-OECD and non-Eurostat countries. The information tions. Data cover a number of variables related to was obtained through a metadata collection imple- STI, including those related to human resources. mented by UNESCO-UIS from September 2012 to April • OECD’s Main Science and Technology Indicators 2013. It shows that most of the surveys were conducted (MSTI) database compiles a similar range of data in 2012 and 2010. Although there is no harmony in the generated by country’s statistical offices with a focus years covered by these surveys, in 16 out of 24 countries on its member economies. the observation period had a length of three years. It is noteworthy that in 8 countries the national statistical Innovation Statistics office (NSO) was the agency in charge of the survey. In addition to the standard R&D statistics, a number of Information on Institutions, firm-level innovation surveys have been implemented in Policies, and Programs recent years. Firm-level data on innovation are available in most EU member countries in the format of Com- A useful source of information is the existing reports on munity Innovation Surveys (CIS), available online at the innovation policy. Several organizations perform, with Eurostat website.3 Innovation surveys are less frequent some regularity, analyses of national innovation systems, in developing countries: for instance, in the African benchmark exercises, reviews of policy trends, and so continent, only Morocco, South Africa, and Tunisia had forth. Some of those reports concentrate on gathering developed that instrument by 2008 (Ellis 2008). factual information about the recent developments at the organizational, policy, or program levels. • The 2013–14 round of the World Bank’s Enterprise Survey has a new module on innovation perfor- The appendix to this chapter summarizes the coverage mance, which is to a large extent compatible with of some of those reports—namely, the OECD Reviews the standard innovation surveys. Combined with of Innovation Policy, United Nations Conference on a variety of other data sources, firm-level data has Trade and Development (UNCTAD), STIP Reviews, the given a richer picture of innovative activity at the ERAWATCH Country Reports, and INNOTREND Mini- firm level and of the ways in which knowledge is Country reports—in terms of the following issues: generated and transmitted within and between governance and policy assessment, policy measures, firms (Hall and Mairesse 2006). innovation budget data, STI systems, economic per- formance and framework conditions, as well as main 3. http://epp.eurostat.ec.europa.eu/portal/page/portal/eurostat/ policies and programs. While frequency also varies, they home/. are useful sources of background information. 26 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 4.2: Most Recent Innovation Surveys (as of 2013) Country Survey name and year Observation period Institution in charge Azerbaijan On innovation activity of enterprises 2012 2011 (calendar year) The State Statistical Committee Belarus Innovation activity of organisation 2012 2011 (calendar year) National Statistical Committee of the Republic of Belarus China Industrial Enterprises Innovation Survey 2007 2004–2006 National Bureau of Statistics of China Hong Kong Survey of Innovation Activities 2010 2010 (calendar year) Census and Statistics Department SAR, China Colombia Quinta encuesta de desarrollo e innovación 2009–2010 (calendar year) Departamento Administrativo Nacional de tecnológica en la industria colombiana 2011 Estadísitica (DANE) Costa Rica Encuesta Nacional de Indicadores de Ciencia, 2010–2011 Ministerio de Ciencia y Tecnología Tecnología e Innovación 2012 Cuba Encuesta Nacional de Innovación 2006 2003–2005 (calendar year) Ministerio de Ciencia, Tecnología y Medio Ambiente (CITMA) Dominican Encuesta Nacional de Innovación 2010 2007–2009 (calendar year) Ministerio de Educación Superior, Ciencia y Republic Tecnología Ecuador Encuesta de Actividades de Innovación 2013 2009–2011 (calendar year) Secretaría Nacional de Educación Superior, Ciencia, Tecnología e Innovación (SENESCYT)/ Instituto Nacional de Estadística y Censos (INEC) Ethiopia Ethiopian National Innovation Survey 2011 2011 (fiscal year) Ministry of Science and Technology Indonesia Innovation survey in manufacturing industry 2011 2009–2010 Indonesian Institute of Sciences (LIPI) Lesotho Lesotho Innovation Survey 2012 2009/10–2011/12 Department of Science and Technology Malaysia National Survey of Innovation (NSI-6) 2012 2009–2011 Ministry of Science, Technology and Innovation Palestine Palestinian Community Innovation Survey 2010 2006–2008 Palestine Academy for Science and Technology (PALAST) Panama Encuesta de Investigación, desarrollo e innovación 2006–2008 (calendar year) Secretaria Nacional de Ciencia y Tecnología en el sector privado de Panamá 2008 Paraguay Encuesta para la determinación de la línea de base 2004–2006 Consejo Nacional de Ciencia y Tecnología de innovación tecnológica en empresas paraguayas (CONACYT) 2007 Peru Encuesta Nacional de Innovación el la Industria 2009–2011 Instituto Nacional de Estadística e Manufacturera 2012 Informática Philippines Survey of Innovation Activities by Establishments 2009–2010 Department of Science and Technology 2010 Serbia Community Innovation Survey 2010 2008–2010 Statistical Office of the Republic of Serbia Tunisia Enquête R&D et Innovation 2008 2005–2007 Bureau des Etudes et de la planification, Ministère de l’Enseignement supérieur et de la Recherche Scientifique Uganda National Innovation Survey 2012 2008–2010 (calendar year) Uganda National Council for Science and Technology (UNCST) Ukraine The innovative activity of enterprise survey 2010 2008–2010 (calendar year) State Statistics Service of Ukraine Uruguay IV Encuesta de Actividades de Innovación en 2007–2009 Agencia Nacional de Investigación e Industria/II Encuesta de Actividades de Innovación Innovación (ANII) en Servicios 2010 Zambia National Survey on Innovation 2012 2008–2010 Department of Planning and Development, Ministry of Science, Technology and Vocational Training Source: UNESCO-UIS 2013. Note: For Ecuador and Malaysia, the surveys were still ongoing when metadata were submitted. Inception Report 27 As discussed in table 4A.1 in annex A of this chapter, The 2011 INNO-Policy TrendChart mini country reports the reports on institutions, policies, and programs have comprise the most detailed information on innovation different levels of coverage, with some specializing in financing, containing STI budget by ministry/instru- some areas and not others: ment/financing source; broad composition of available national budgets by main categories of research and The OECD reports are the most comprehensive with innovation measures (with budget and programs under theoretical background, detailed analysis of a country’s each category); as well as description on future chal- economic performance, innovation framework condi- lenges for funding of innovation policy. INNO-Policy tions, STI system including SWOT analysis and broad TrendChart mini reports (2011) primarily focus on recent scope of data. In the majority of cases, each element changes in STI policy, existing innovation policy instru- is carefully introduced, assessed with suggestions for ments, and RD&I budgets. improvement built on international best practices suit- able to the examined country. Since these are not annual The UNESCO STI studies’ structure and content varies report, their structure differs among the country cases. among countries, and therefore, it is difficult to com- The OECD framework offers the most detailed analysis pare with other frameworks. The title of each of the of responsibilities of each institution in the STI manage- UNESCO’s study indicates the analysis area related either ment system with recommendations on improvements, to formulation of STI strategy or a review. including suggestions for creation of new agencies based on other countries’ experience. 4.3. CONCLUSION UNCTAD’s STIP reviews are similar to the OECD studies. The studies present careful analyses of selected sectoral This chapter described how to implement the Inception innovation systems. Report. The chapter began by describing the Country Paper. This included a discussion of the strategic context; ERAWATCH Annual Country Reports (started in 2009) benchmarking the country’s NIS; and an overview of characterize and assess the performance of national key institutions, programs, and policies. This provides a research systems and related policies. Since these are sense of the strengths and weaknesses in the NIS. This annual reports, each report builds on a previous one, was followed by a data assessment, which discussed therefore focusing on recent policy changes rather than STI statistics; innovation statistics; and information on repeating what has been already said. Thus every year’s institutions, policies, and programs. The availability, ease report focuses on specific barriers in reaching the Lisbon of accessing, and quality of data is a major constraint goal, and provides analysis on the country policy mix on the rest of the PER exercise and so this section pro- routes and instruments to address the barriers. Also vides important inputs into planning the rest of the PER the studies investigate contribution of national policy exercise. Box 4.4 provides a possible structure for the mixes to the realization of the European Research Area. Inception Report as well as useful readings. The next The reports offer policy assessment rather than recom- chapter describes the Functional Review, which provides mendations on the approaches in enhancing the policy. guidance on reviewing STI expenditures. 28 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 4.4: The Inception Report—Possible Structure and Useful Readings A proposed structure for the Inception Report as a standalone document Executive Summary 1. Introduction 2. Country Paper a. Economic performance and main challenges b. Organizations, policies, and programs in the national innovation system c. Benchmarking the national innovation system 3. Data Availability and Accessibility 4. Analytical Framework 5. Implementation Plan a. Scope of analysis b. Data collection requirements c. Implementation arrangements d. Timeline 6. Conclusion Useful readings Organisation for Economic Co-operation and Development (OECD). 2008a “OECD Reviews of Innovation Policy, Norway.” OECD, Paris. Available at: www.oecd.org/sti/innovation/reviews. United Nations Conference on Trade and Development (UNCTAD). 2012. “Science, Technology & Innovation Policy Review Dominican Republic.” UNCTAD/DTL/STICT/2012/1. United Nations, New York, NY. World Bank. 2009. “Turkey National Innovation and Technology System: Recent Progress and Ongoing Challenges.” Europe and Central Asia Region. Report No. 48755-TR. World Bank, Washington, DC. Inception Report 29 30 ANNEX A: COMPARISON OF INNOVATION FRAMEWORKS ACCORDING TO MAIN CONTENT CATEGORIES Table 4A.1: Comparison of Innovation Frameworks According to Main Content Categories EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports I. GOVERNANCE, INNOVATION POLICY ASSESSMENT Main current innovation policy priorities and √ √ √ √ challenges Innovation system governance—description √ √ √ √ (analyzed and assessed in great detail) + the regional level Assessment of each ministerial bodies in √ — — — innovation policy Innovation governance assessment √ √ √ √ (refers to national research system) (with SWOT analysis) +governance at the regional level Governance recommendations √ — — — (recommendation built on experiences from other countries) Summary of existing innovation evaluations √ √ — √ (if any) (mentioned) Benchmarking innovation performance √ √ — — Overall assessment of innovation/research √ √ √ √ Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note policy (with SWOT analysis) (with strengths and weaknesses) (continued next page) Table 4A.1 (continued) EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports II. POLICY MIX MEASURES Policy mix issues—theoretical background √ — — — Direct policy mix measures—description √ √ √ √ Examples of data used in direct policy mix S&T and innovation funds/programs— Types of instruments, some data Depends on a report, e,g. 2009 - measures their budget over time, expenditures on budget and distribution for reports incl. programs with criteria according to, loans, budget support, some programs. Contains good and overall budget listed along policy other resources; committed and practices in the use of STI policy mix routes executed budget. instruments. Indirect policy mix measures √ √ √ √ (detailed, with details and comparison (briefly described incl. Intellectual Depends on a report, e,g. in 2009 (recent changes in the innovation of practices in OECD countries; property, Quality system; no data) public funding trends for knowledge policy mix, no data) Estimated revenue losses due to R&D demand, including networks, cluster, tax incentives as a % of GBAORD; TTOs, etc. Generosity of fiscal support to R&D in (new initiatives with budgets) OECD countries) Assessment of policy mix issues/barriers √ √ √ — (with trends in financing + international (plus recommendation and case Main policy opportunities and risks, comparison; detailed budgets of studies for other instruments) Main barriers to R&D investments instrument ) and respective policy opportunities and risks Contribution of national policy mixes to the — — √ — realization of the European Research Area (briefly on international links) Very comprehensive (continued next page) Inception Report 31 32 Table 4A.1 (continued) EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports III. INNOVATION BUDGET DATA Level of detail Detailed estimates Basic Basic Very detailed and recent estimates (2010-2011) Data example • Estimates of total expenditures on GERD/BERD, selected programs GERD/BERD, selected programs • Innovation budgets of the main STI activities by source of funds budgets with results budgets with results government departments and (i.e. direct budgetary resources and agencies to which institution’ Multilateral • Broad composition of available financing institutions’ loans, Private national budgets by main universities, business sector, other categories of research and sources) innovation measures (i.e. • Estimated R&D appropriations by categories: Governance & horizontal ministry research and innovation policies; R&T; HR; Enterprise innovation; • Selected STI programs’ budget and market and innovation culture) with spending categories (i.e. financing budget and programs under each HR, innovation, basic/ applied R&D, category) scholarships, etc.) • Future challenges for funding of • Returns from the Norway’s research innovation policy fund • Research and innovation policy • Bottom-up funding of free basic measures with start and end date, research budget and commentary Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Data analysis Trends in R&D expenditures, their Deeper analyses regarding Data analysis from the perspective Description of trends in spending, with reasons, comparison with other biometric analysis; and patent of the reasons, main barriers to R&D major reasons without deeper data developing and the OECD countries analysis investments and respective policy analysis opportunities and risks (continued next page) Table 4A.1 (continued) EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports IV. ECONOMIC PERFORMANCE AND FRAMEWORK CONDITIONS FOR INNOVATION Macro-economic performance review √ √ — — Labor force issues—education, training, √ √ √ — mobility and flexibility Division among main R&D performers ICT infrastructure √ √ — — Innovation inputs indicators review √ √ √ — (detailed) (detailed) (brief) V. SCIENCE, TECHNOLOGY AND INNOVATION SYSTEM STI system and governance theory √ — — — Institutional and legislative framework of √ √ √ √ science, technology and innovation (STI) STI institutional structure √ √ √ — Governance at the regional level √ √ √ — (trends on RIS policy ) Innovation at sectoral level √ √ -— √ (brief) (detailed analysis of inn sys. in (brief sectoral specificities of recent selected sectors ) policy initiatives) (continued next page) Inception Report 33 34 Table 4A.1 (continued) EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports VI. STI MAIN ACTORS a. Innovation in business sector/ √ √ √ — entrepreneurship review (detailed) (brief) (brief) Financing through market (e.g. Venture √ √ — — capital, business angels ) (only mentioned) Assessment of the barriers for innovation in √ √ √ — the business sector (brief) b. Public research institutes (PRIs) — Description of current situation + legal √ √ √ — documents Details on PRIs √ √ √ — Details under which ministry, budget Information on some institutes 2009 reports: General information on with share of institutional funding, main and under which ministry they RIs number with names of the main focus areas, number of personnel. are, no budget data. ones, no data on budgets. 2010 : very basic inf. STIs challenges √ √ √ — (summary assessment of strengths and weaknesses of the national research system) Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Strategic vision with recommendation and √ — — — STIs case studies from other countries (continued next page) Table 4A.1 (continued) EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports PRIs Trends in policy reforms √ — √ — (based on the OECD countries) (assessment) Vision for PRI development strategy √ — — — (rec. based on the OECD experience) Governance of PRIs √ √ √ — PRIs financing issues √ √ — — Collaboration with the productive sector √ √ √ — (with rec. based on the OECD Knowledge transfer/knowledge experience) circulation (policy programs targeting this issue as policy mix—knowledge circulation) Examples of public-private partnerships for √ — — — science, techonology, and innovation for consideration Issue of monitoring and evaluation of √ √ √ — institutes’ performance (with performance evaluation practices in other countries) (continued next page) Inception Report 35 36 Table 4A.1 (continued) EU INNO Policy Trendchart ERAWATCH Country Reports OECD Reviews of Innovation Policy UNCTAD’s STIP Reviews (annual) (2011 mini reports) Characteristics and assessment of the performance of national research systems and related A comprehensive assessment of an Diagnostic analysis of STI policies/focus on recent policy Assessment of innovation policy Objective innovation system effectiveness changes trends Reports that the table is based on Norway and partly Peru El Salvador and Peru reports All reports All reports Policy framework for patenting by PRIs and √ — — — universities (with review of existing laws in other (general info on patent data) countries) Universities in research √ √ √ — c. The higher education sector—focus √ √ √ — on R&D cooperation and innovation (detailed) (brief, without indicators, trends, (recent changes in education policies plus assessment and recommendations comparisons) having impact on innovation) Education indicators √ √ √ — Challenges and bottlenecks √ √ √ — R&D activities, university funding and √ — — — programs (many indicators and data) HE governance and regulatory issues √ √ √ — (also assessment of the educational reforms) Research at universities √ √ √ — Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note CHAPTER 5 FUNCTIONAL REVIEW The Functional Review (FR) is the second stage in the The chapter starts with a discussion in section 5.1 of preparation of the Public Expenditure Review (PER). The how to prepare the STI budget, including an outline FR addresses questions related to how much is spent of its structure and an explanation of how to create it. by the government on science, technology, and inno- Section 5.2 continues with a practical example. Section vation (STI); by whom; and to achieve what objectives 5.3 concludes with a comparison between the proposed (see figure 5.1). STI budget and other similar indicators. Figure 5.1: The Functional Review Operational Functional Effectiveness Inception Report Efficiency Final Report Review Assessment Assessment Summary The objective of the Functional Review (FR) is to describe the flow of funds in the research and innovation sector. It de- scribes how much is being spent, by whom, and for what objectives. This includes identifying the intermediate outcomes that the spending aims to achieve. Four intermediate outcomes are proposed: (i) research excellence, (ii) science-industry collaboration and technology transfer, (iii) business R&D and startups, and (iv) technology adoption. Important challenges must be overcome to implement the FR. These include accessing budget and expenditure data, over- coming quality issues in the data, and identifying spending on innovation, when innovation is not among the categories typically used to categorize government spending. The Inception Report provides information on the programs and organizations operating as well as publicly available RDI spending data. Additional data comes from: • Government, including proposed, approved, and actual disbursements • Policy questionnaires (examples of which can be found on data collection on tax incentive support for R&D expen- ditures (OECD 2013b), and STI policies for nanotechnology (OECD 2008b) • Interviews with selected organizations, including ministries of finance, economic growth, and other sectors Functional Review 37 The chapter starts with a brief description of the The work can be complemented by a governance expected content of the mapping of STI spending in analysis—when, for example, the exercise is restricted section 5.1, followed by discussion of some assessment to this stage. The governance analysis focuses on how questions for the governance analysis in section 5.2. The the existing governance structure leads to the current remainder of the chapter is dedicated to the preparation allocation of resources. The information for this analysis of the STI Budget with a proposed STI budget structure is generated in the Inception Report discussed in chap- and how to implement it. Section 5.3 continues with ter 4 (main organizations, policies, and programs). The a practical example and section 5.4 concludes with a assessment questions for the governance analysis are comparison between the proposed STI budget and presented in chapter 8. other similar indicators. Box 5.1: Government STI Spending in Turkey According to Implementing Agency and Programs, 2005–08 The Turkish Government’s investments in public innovation and technology support programs have risen substantially in recent years and are projected to continue to increase. During the 2005–08 period, the government allocated a significant amount of additional resources (over US$1.5 billion) from its budget, primarily to the Scientific and Technological Research Council of Turkey (TUBITAK). Public resources allocated to innovation and technology support programs have more than tripled in the last 10 years with public R&D expenditures as a share of GDP rising from 0.67 percent in 2002 to 0.8 percent in 2006. Over the same period, the number of full time equivalent researchers grew from 23,995 to 28,000. The main public agencies in the Turkish NIS that implement STI support programs are TUBITAK, the Technology Development Foundation of Turkey (TTGV), the Small and Medium Enterprises Development Organization (KOSGEB), the Ministry of Industry and Trade (MoIT), and universities. A summary of the main allocation of funds within Turkey’s NIS from 2005 to 2008 is provided in table B5.2.1. Table B5.1.1: Public Expenditures on Innovation and Technology Programs Implementing Agency 2005 2006 2007 2008 Universities 274.2 278.7 256.3 253.5 TUBITAK (TUBITAK Research Centers) 108.8 155.0 141.8 183.3 TUBITAK (Turkey Research Area Programs)* 346.0 415.0 425.0 450.0 Academic Research Projects 90.0 80.0 85.0 105.0 Industrial Research Projects (of companies) 116.0 215.0 215.0 175.0 Research Projects of Public Institutions 50.0 50.0 50.0 65.0 Defense and Space Research Projects 50.0 60.0 65.0 80.0 Researcher Development 25.0 5.0 5.0 15.0 Science & Technology Awareness 15.0 5.0 5.0 10.0 Public Institutions (outside TUBITAK) 36.2 49.3 80.2 78.2 Nuclear Energy Council (TAEK) 6.3 13.1 20 18.9 Ministry of Industry and Trade** 11 16.9 17.6 Ministry of Agriculture and Rural Affairs 2.2 2.5 4 3.6 Ministry of Health 0.1 6.2 5.2 4.9 National Boron Research Institute*** 0.1 3 6 6.3 Ministry of Energy*** 1 KOSGEB 12.5 5.4 4.6 6.5 TTGV 8.9 35.6 35.4 35.5 State Planning Organization 1.1 10.0 18 18 Undersecretary of Foreign Trade 40.0 42.0 63.5 n.a. Notes: *TUBITAK funds the projects of other institutions’ R&D projects. **Includes SAN-TEZ program that supports PhD students’ theses that aim to solve company specific problems, and the support for the physical infrastructure of Technoparks. ***Includes programs in which the projects of other institutions are supported. n.a. = Not applicable Source: World Bank 2009. 38 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note 5.1. STI BUDGET STRUCTURE the four intermediate outcomes in figure 3.4. The last category, Other Expenditures (I), is introduced to The FR will provide a comprehensive review of the capture expenditures that implement other objectives spending of main organizations and programs of the or have unclear objectives but are nominally related to research and innovation sector. The objective is to be research or innovation. able to describe the basic flow of funds in the system: how much is spent, on what, by whom, and for what Table 5.1 can be further disaggregated as needed. For objective—that is, how the innovation system “func- example, Business R&D and R&D-Based Innovation can tions.” Box 5.1 provides a brief illustration of this ex- be further disaggregated into Tax Breaks (D) and Direct ercise for the case of Turkey from 2005 to 2009. The Subsidies (E) as indicated. It can also be adjusted to fit exercise is essentially descriptive but provides a first look different objectives established in the logical frame- at Turkey’s spending on STI. work. For example, it can include a specific category to capture investments in social innovation and the Table 5.1 presents a possible simplified structure for category’s corresponding intermediate outcomes and an STI budget. Note that the proposed categories development goal. correspond essentially to the logical framework es- tablished in chapter 3. In the table, budget category National Budget and Budgetary Records Research Excellence and Technology Transfer Budget Government budget documents (such as central gov- (C), and category Business Innovation Budget (G), ernment consolidated accounts, line ministry informa- reflect the two basic distributions of public funds tion, medium-term expenditure framework documents) described in figure 3.4 (see chapter 3). The four ad- are a primary source of information for the preparation ditional budget categories (Research Excellence (A), of the proposed consolidated STI budget. Public sector Expenditures for Technology Transfer and Science- budgets are presented according to (i) Classification of Industry Collaboration (B), Business R&D and R&D- Functions of Government (COFOG) or (ii) the Economic based Innovation (E), and Non-R&D-based Innovation, Classification. The majority of developing countries Technology Adoption and Diffusion (F) are exactly today follow the guidelines of the IMF’s Government Table 5.1: Consolidated STI Sector Budget—Simplified Structure Value (US$ 000’s Budget item current) Research Excellence (A) Expenditures for Technology Transfer and Science-Industry Collaboration (B) Research Excellence and Technology Transfer Budget (C)= (A+B) Business R&D and R&D based Innovation: Tax Breaks for Business R&D (D) and Direct Support to Business R&D and R&D-based Innovation (E) Non-R&D-based Innovation, Technology Adoption and Diffusion (F) Business Innovation Budget (G)= (D+E+F) R&D and Innovation Sector Budget (H) = (C+G) Other Expenditures (I) Consolidated R&D Budget (J)= H+I Functional Review 39 Finance Statistics Manual (IMF 2001), which includes well as the data quality, issues discussed in chapter 4, an initial version of the COFOG.1 The classifications are the following are some of the main challenges. summarized in table 5.2. One limitation refers to the identification of public Categories can be combined to provide, potentially, measures to promote innovation. R&D spending is a interesting analyses, as described in table 5.3. For ex- functional category under the COFOG classification ample, spending on “R&D Defense” can, in principle, be but “innovation” is not among those categories of combined with, among others, expenditures on “Com- government functions normally used. Therefore, most pensation to employees,” “Subsidies,” and “Grants” of the innovation-related expenditures may not be (IMF 2001: 78, para. 6.104). In addition, the categories directly identifiable in budgetary documents. This is shown in table 5.3 can be further disaggregated. For illustrated by Brazil’s FINAME (Financing of Machinery example, data on subsidies can be decomposed by type and Equipment)—a program implemented by the Bra- of beneficiary, including nonfinancial public corporations zilian Economic and Social Development Bank (BNDES) and nonfinancial private enterprises (which could be a that provides subsidized loans for the acquisition of first good indicator for the priority given to the public machinery and equipment by the private sector.2 As and private sectors in governments’ R&D policy). part of BNDES, the program as such does not appear in the national budget. Rather, it is “buried” in the BNDES Main Challenges budget line in the central budget. Budgetary data can help create an understanding of the Another limitation refers to coverage of expenditures patterns of public spending in STI for the country but under the R&D category of COFOG. This reports gov- its use involves important challenges worth keeping in ernment expenditures under this category follow the mind. Apart from access to budgetary information as definition of R&D of the Frascati Manual (OECD 2002): the category covers basic and applied research and ex- 1. The Classification of Functions of Government (COFOG) was de- veloped by the OECD and adopted by the United Nations in 1986. It 2. The program funds a major part of the investments of the manu- was adopted by IMF as the Government Finance Statistics Manual. facturing industry, about 20 percent of total aggregate gross fixed For more information see IMF (2011). capital formation in the country in the 2011–12 period. Table 5.2: R&D Expenditures by COFOG Classification 7 Total outlays 701 General public services 706 Housing and community amenities 7014 Basic research 7065 R&D Housing and community amenities 7015 R&D General public services 707 Health 702 Defense 7075 R&D Health 7024 R&D Defense 708 Recreation, culture, and religion 703 Public order and safety 7085 R&D Recreation, culture, and religion 7035 R&D Public order and safety 709 Education 704 Economic affairs 7097 R&D Education 7048 R&D Economic Affairs Environmental protection 710 Social protection 705 R&D Environmental protection 7108 R&D Social protection 7055 Source: Elaboration of data from IMF (2001). 40 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 5.3: COGOF and Economic Classification—Illustration (in US$ 100,000) Economic classification Compensation Use of goods Consumption to employees and services of fixed capital Subsidies Grants Social benefits Other expenses COGOF classification (21) (22) (23) (25) (26) (27) (28) 7014 Basic research 1 71 5 61 75 73 54 7015 R&D General public services 34 37 8 34 33 47 24 7024 R&D Defense 21 71 23 55 14 75 70 7035 R&D Public order and safety 70 1 75 74 48 19 28 7048 R&D Economic affairs 13 62 14 69 23 28 13 7055 R&D Environmental protection 28 58 27 43 68 69 18 7065 R&D Housing and community 27 3 21 42 12 56 51 7075 R&D Health 26 29 64 2 27 30 46 7085 R&D Recreation, culture, religion 34 65 33 13 41 53 11 7097 R&D Education 45 42 62 37 55 45 6 7108 R&D Social protection 32 4 46 68 25 5 24 Source: Elaboration of data from IMF (2001). Note: COGOF = Classification of Functions of Government. Functional Review 41 perimental development. But it excludes, for example, objective is not innovation and therefore they should other related scientific and technological activities (such not be included in the budget. as feasibility studies) and innovation activities other than R&D, including all those technical, commercial, and A fourth issue to be kept in mind refers to how expen- financial steps necessary for the implementation of the ditures are appropriated in the STI budget. Challenges results of R&D activity.3 This is particularly relevant for include: (i) how to address spending by public research the collection of data on public spending related to two organizations (PROs) that are not funded by taxation categories of the proposed STI budget: (i) technology (the gross versus net principle—see table 5.4b for more transfer and science-industry collaboration and (ii) non- details), (ii) how to appropriate indirect support and R&D innovation categories of the proposed STI budget. loans, (iii) how to appropriate multi-year projects, (iv) whether to include taxes involved in the expenditures or • For instance, prototypes are counted as R&D activi- not, and (v) estimating the share of specific expenditures ties as long as the primary objective is to make fur- not originally available. For example, the value of the sub- ther improvements. Pilot plans should be included as sidy incurred by the BNDES’ FINAME may not be readily long as the primary purpose is R&D. Trial production available from the source and may need to be calculated should be included if production implies full-scale (that is, monetized) as part of the overall exercise. testing and subsequent design and engendering. Patenting and licensing work should be excluded. A final challenge involves the classification of the mea- sure according to the different intermediate outcomes. A third issue refers to the scope of measures to be This step corresponds to the core of the work of the FR. included in the proposed STI budget. For example, some more developed countries may prefer to exclude Table 5.4a summarizes a proposal for addressing the a program like BNDES’ FINAME from the calculation of challenges discussed before. The problem of iden- their STI budget and classify the subsidy to the acquisi- tification and coverage of R&D spending may be tion of machinery and equipment among more general circumvented by close collaboration with the counter- “competitiveness” policies. Other public spending may parts, field interviews with key stakeholders, review of be innovation related or not depending on its effective policy documents, and review of third-party analyses application. of a country’s research and innovation policy. The two classification issues (boundaries of STI programs and • For example, the provision of labor training may be classification in terms of the intermediate outcomes) directly related to innovation if targeted to improve can be addressed by adopting the “primary purpose the technical skills of the workforce in the produc- of the intervention” concept based on a review of the tion process, improving quality, reducing re-work, corresponding policy and program documents (see the and improving firm productivity. In those cases, the Frascatti Manual (OECD 2002), Chapter 8, § 499–500). primary purpose of the public support seems to be Table 5.4b addresses appropriation issues. innovation (in the broad sense applied by this note). • On the other hand, several lifelong learning pro- 5.2. A PRACTICAL EXAMPLE grams may involve the provision of “general skills.” While those skills are vital for the employability of the In this section the exercise is illustrated with a practi- labor force and efficient adjustments of the economy cal example. The STI budget is prepared in four main to changes in global demand, the programs’ primary steps: identification of support measure; budgeting of support measures; classification of support measures 3. For detailed discussions about the boundaries of R&D activities see according to the intermediate outcomes; and budget OECD (2002), especially Chapter 2. consolidation. 42 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 5.4a: Preparing the STI Budget: Challenges and Proposed Solutions Challenge Proposed solution Comment Identification of STI measures and Field visits and close collaboration with counterpart Public spending on STI not always directly visible in coverage of R&D spending government budgets Agree with the counterpart on the list of programs to be Budgetary information on public R&D spending does not covered. A list of STI measures is presented in appendix B cover the full spectrum of policies necessary to transform Inclusion of measures in the STI R&D result into innovation budget Consult program-specific documents. Decide whether to include the expenditure in the STI budget or not based on Boundaries of STI measures not always clear from simple the primary purpose of the spending (if STI or not) examination of the budget items Appropriation of spending in STI See table 5.4b How much of the budgeted spending to include in the STI budget budget? Consult program-specific documents. Decide whether to How to classify the budgeted measures according to the Classification according to include the specific expenditure or not based on the primary intermediate outcomes intermediate outcomes purpose of the spending (if STI or not) Table 5.4b: Appropriation of Expenditures: Challenges and Proposed Solutions Challenge Proposed solution Comment Appropriations for which corresponding revenue is For example, if an R&D institute has a local gross budget of US$10 Gross/Net approach, net expected either from other government sources or other million including US$3 million income from the provision of contract principle sectors of the economy should be excluded according to research, then only the difference (US$7 million) should be counted as the net principle. (§488-489). net budgetary appropriations. Monetize the value of the subsidy based on the notion of The 2001 GFS Manual (IMF 2001) provides specific guidance on how “forgone revenue.” to appropriate those costs. Indirect funding (tax Loans that may be forgiven should be included, but loans Nevertheless, when such indirect support programs are undertaken breaks) and loans that are to be repaid and indirectly support industrial R&D as part of an integrated R&D policy (for example, when the sources via tax rebates, etc. should in principle be excluded. (§493) are documented and are included in inter-ministerial discussions of a science budget), they may be included. Multi-annual projects budgeted in only one year or over Multi-annual programs that are authorized at some stage but several years should be allocated to the STI budget of the budgeted over several years should be allocated to the years in which Multiannual projects year(s) in which they are budgeted, not in the years of they are budgeted, not the year of authorization. performance. (§495) Data on R&D expenditure should be reported at factor In Lithuania and in the Slovak Republic, VAT has been included in the Value-added tax (VAT) costs (i.e. VAT and other taxes should be excluded). (§371) calculation of government spending in R&D while in Bulgaria and the Czech Republic it was partially included (as of 2009). Estimates may be used to value the share of a given For example, coefficients are used to estimate the R&D share of Estimating the share of intervention in the overall budget of an organization. budget item (Austria), to separate R&D from non-R&D (Germany), to specific expenditures Make the approach transparent and simple. Use available calculate General University Funds (Sweden), etc. data to make inferences, when possible. Sources: Elaboration of data from of Eurostat (2012), IMF (2001), and OECD (2002). Note: The numbers following the symbol (§) refer to the paragraph of the Frascati Manual (OECD 2002). Identifying the Measures Figure 5.2 presents a program taxonomy that primar- ily distinguishes between supply-side measures and In order to identify the programs to be used, a checklist demand-side measures. Within supply-side measures, it of STI programs can be used. This illustration uses one of also makes a key distinction between financial measures the many different typologies of STI programs available.4 and support services. It may be used as an initial check list for the scope of STI spending. In principle, any other 4. Public support for STI can be delivered through multiple channels, taxonomy could be used. and through a variety of policy tools. An array of STI policy typologies have been developed for different purposes. Some of them are devel- oped on the basis of their target groups (for example, SME support Budgeting the Measures schemes, PhD grants, researcher grants), while others are developed on the basis of the policy challenge they are meant to address (for Following the taxonomy presented before (or any similar example, innovation financing schemes, skill development schemes). list of measures), a consolidation of the number of pro- Functional Review 43 44 Figure 5.2: An STI Policy Taxonomy Supply -side Demand side measures measures Finance Services Fiscal Support for public Support for Grants for Information and Public Support of Networking Systemic policies Equity support training and brokerage Regulation measures sector research industrial R&D measures Procurement private demand mobility support Public venture Corporation tax University funding. Tailored courses Grants for R&D. Contact Support for clubs. Cluster policies. Use of R&D Demand subsidies capital funds. reductions for Laboratory for firms. Collaborative databases. Foresight to build Supply chain regulations and procurement. and tax Mixed or volume or funding. Entrepreneurs- grants. Brokerage events. common visions. policies. standards to set Public incentives. subsidized private increment in R&D. Collaborative hip training. Reimbursable Advisory services. Co-location in innovation procurement of Articulation of venture funds. Reductions in grants. Strategic Subsidized loans. Prizes to International incubators, targets. innovative goods. private demand. Loss underwriting employers’ programs for secondments be spent on R&D. technology science parks etc. Technology Awareness and and guarantees. payroll tax and industry. Support Industrial watch. Patent platforms to training. Catalytic Tax incentives. social for contract research databases. coordinate procurement. contributions. research. studentships. Benchmarking. development. Personal tax Equipment Support for incentives for sharing. recruitment of R&D workers. scientists. Source: Edler and Georghiou 2007. Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 5.5: Example of Categorization of Programs Based on a Standard Taxonomy Description of Public resources Program/ the program or Public resources allocated disbursed organization expenditures (appropriations, US$ ‘000) (outlays) Appropriation issues Name the program and Present here the stated Present here the total Present here the total Describe the assumptions organization objective of the program allocation of public funds for amount of public related to appropriation and the nature of the the program funds disbursed at the exercise (table 5.4b), as instrument end of the period necessary Consider reporting different levels of budget preparation Ministry of Science/ Research grants: promote US$90,000 US$90,000 Obtained from budget National Science scientific research codes (COFOG, 71040) Foundation Ministry of Economy Center of Competence US$5,000 US$5,000 Estimated as 50% of MoE (MoE) Technological Platforms based on initiatives to promote centers of competence Ministry of Finance Subsidized loan for n.a. US$8,000 Estimate as 100% of acquisition of machinery forgone revenues in the and equipment past year Note: n.a. = Not applicable. grams and respective budgets can be carried out. This declared, while in other cases the objective will not be can be done by filling the information requested in table stated clearly. It is proposed that the team/analyst accept 5.5.5 The description of the program/expenditures will the program’s declared goal when it is sufficiently clear, generate the basic information for further classification or try to discern and follow the “principal purpose” according to the categories. At this stage, an estimation otherwise. The exercise is illustrated by table 5.6. Note of the budget of each measure is also undertaken, as that the proposed classification is not exhaustive, that is, illustrated with three examples in table 5.5. there may expenditures/programs that do not attempt to contribute to any of those four goals and should be Classifying Measures by classified under a fifth category (other goals/unclassi- Intermediate Outcomes fied). These four objectives work therefore as a first filter for examining the quality of the public expenditures. In this step, the current expenditures are classified Expenditures that cannot be classified by the team into according to their potential contribution to the four any of the four categories are not linked to the four intermediate outcomes defined previously. The effort intermediate outcomes. consists of identifying all public programs and cor- responding expenditures related to STI and classifying • Programs may be related to multiple goals (for ex- them according to each of the proposed four outcomes. ample, both provision of training and support for technology development zones). Detailed analysis The proposed exercise is to some extent arbitrary. of project documents outlining proposed outputs, Programs are not easily classified in any of those four impacts, and goals would offer some understanding categories. In many cases multiple objectives will be of the potential impact on different intermediate 5. The table only includes a limited number of program examples. outcomes outlined in the framework. Functional Review 45 46 Table 5.6: Categories of Innovation Programs Based on Intended Outcomes A B C D E F Technology transfer and science-industry Business innovation and Technology diffusion and Research excellence collaboration start-up creation adoption % of % of % of % of program program program program Link to resources Link to resources Link to resources Link to resources Program outcome linked to outcome linked to outcome linked to outcome linked to Program/output type budget (Y/N) outcome (Y/N) outcome (Y/N) outcome (Y/N) outcome Center of excellence Principal investigator grants International research cooperation grants Research grants Science programs Strengthening national research performers Cluster initiatives Technology consortia (feasibility and R&D stages) Support for patenting and IPR protections Implanting scientists and engineers in industry Support for technology transfer offices Innovation project subsidies Tax credits for R&D projects done with registered technical centers Credits to fund innovation projects and investments Support to implement completed innovation projects Subsidy for technical consulting (innovation vouchers) Subsidized loans for acquisition of machinery and equipment Tax breaks for acquisition of machinery and equipment Technology management scheme Overseas technology missions Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Managerial training Labor training Strengthening MSTQ system Source: Elaboration based on data from Technopolis (2009) Note: IPR = intellectual property rights; MSTQ = metrology, standards, testing, and quality; R&D = research and development. Estimating the Budget by (GBAORD) is worth taking into consideration. GBAORD Intermediate Outcomes is the standard concept used for consolidating public expenditures on R&D using budgetary information. Once programs have been classified according to the The characteristics and potential use of GBAORD are four proposed outcomes, an estimate can be established presented in box 5.2. Eurostat and the Organization of the volume of funding allocated to each intermediate for Economic Cooperation and Development (OECD) outcome. Table 5.7 illustrates a hypothetical STI budget reports GBAORD statistics for 59 countries, including EU for 2009. If budgets from a single measure can be allo- cated to more than one outcome, then the expenditure member states and candidate countries, European Free related to that measure should be further decomposed, Trade Association (EFTA) countries, Japan, the Republic and an estimate may need to be made. Otherwise, of Korea, the Russian Federation, and the United States. the previous estimate is fully appropriated in the STI The statistics can be accessed online and used to create budget. This is illustrated with the case of centers of international comparisons. excellence—note that the original US$5,000 is further divided into two different intermediate outcomes (col- The main limitation of GBAORD from the point of view laboration science industry and business R&D). of the exercise proposed by the PER is its coverage, which is focused on the R&D activity as defined by the Frascati Manual (OECD 2002). Thus, GBAORD essen- 5.3. OTHER EXISTING INDICATORS tially covers the research excellence component and the direct subsidy component (for example, matching grants Government Budget Appropriations to business enterprise R&D (BERD)). To complete the STI or Outlays on R&D budget, the PER exercise needs to extend its coverage The relationship between the STI budget and the Gov- to the “innovation-related” activities not covered by ernment Budget Appropriations or Outlays on R&D GBAORD. Those activities are mainly related to science- Table 5.7: Illustration—Country Alfa Classification of R&D Expenditures in 2009 (US$ ‘000s) Intermediate outcomes Science-industry collaboration Research and technology Business R&D Technology Other goals/ Government programs excellence transfer and startup adoption unclassified Research grants (general) 90,000 Young research grants 10,000 Matching grant for collaboration 1,000 between PROs and business sector Tax breaks for business R&D 12,000 Centers of competence 3,000a 2,000a Matching grants for proof of concept 1,000 Conditional loan for prototype 1,000 development Support to patenting by researchers 1,000 Subsidized loan for acquisition of 50,000 equipment Technology development zones program 80,000 Total STI budget 100,000 5,000 15,000 50,000 80,000 Note: a. US$2,000 of the resources for the development of centers of competence was used to fund a matching grant scheme to promote business investments in R&D. Functional Review 47 Box 5.2: GBAORD—Concept, Statistical Description, and Use Unlike other statistical surveys carried out in the field of research and development, Government Budget Appropriations or Outlays for R&D (GBAORD) data are based on the analysis and identification of all appropriations spent on research and development (R&D) from public budgets. This means that the approach is based on the funder of research and develop- ment activities (here the state represented by administration), unlike the performance-based approach, which is adopted for example in the R&D survey. Under definitions provided in the Frascati Manual (OECD 2002), GBAORD covers all appropriations or outlays allocated to R&D from public budgets to support research and development, including all contributions to international R&D programs or institutions abroad. The data are based on final budget appropriations (figures as voted by parliament for the coming year—provisional data) and actual outlays (money paid out during the year—final data). Public budgets cover the central government budget and provincial budgets when its contribution is significant. GBAORD also covers general university funds, which are narrowly defined in line with the Frascati Manual as a sum of money given to universities by the ministry of education in support of their overall research activities. GBAORD includes both current costs and capital expenditure. Figure B5.2.1 illustrates GBAORD’s share of total general government expenditures. This indicator is commonly used to indicate the government’s effective priorities. Data is also available in national currency, at 2000 prices, and per inhabitant, among others categories. GBAORD is also broken down in accordance to Nomenclature for the Analysis and Comparison of Scientific Programs and Budgets (NABS) at chapter and sub-chapter levels, corresponding to different socioeconomic objec- tives, as for example civil versus non-civil R&D. The goal is to help countries decide which R&D fields need more funding. Figure B5.2.1: GBAORD as a Share of Total General Government Expenditure, 2010 (percent) 2.0 1.77 1.8 1.56 1.48 expenditure, 2010 (percent) GBAORD as a share of total 1.6 1.36 general government 1.4 1.24 1.2 1.01 0.96 1.0 0.74 0.73 0.71 0.8 0.6 0.37 0.4 0.2 0 ria bl ic ia tia tv ia ia ar y ia ia p. on an an en 8 ga oa g Re -2 l u t La u v ep Es Cr un om ak EU Bu R ith H Slo L R ov ch Sl Cze Country Source: Elaboration of data from the Eurostat website and OECD (2002). industry collaboration, research commercialization, and 5.9 shows the main recommendations from Frascati non-R&D innovation, especially for technology adop- Manual on how to estimate GBAORD. Compilation of tion. Table 5.8 illustrates some cases at the borderline GBAORD statistics relies basically on administrative data between R&D and other industrial activities. and the accuracy of figures is considered appropriate. Most of the OECD and EU member states follow closely If GBAORD data is not available, it is recommended that those recommendations from the Frascati Manual. the team consider the possibility of calculating it to en- Table 5.10 summarizes how some client countries are able comparability of part of the STI expenditures. Table calculating the indicator. 48 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 5.8: Some Cases at the Borderline between R&D and Other Industrial Activities Cases Treatment Remarks Prototype Include in R&D As long as the primary objective is to make further improvements Pilot plant Include in R&D As long as the primary purpose is R&D Industrial design and drawing Divide Include design required during R&D. Exclude design for production purposes. Industrial engineering and tooling up Divide Include “feedback” R&D and tooling up industrial engineering associated with development of new products and new processes. Exclude for production processes. Trial production Divide Include if production implies full-scale testing and subsequent further design and engineering. Exclude all other associated activities. After-sales service and trouble-shooting Exclude Except “feedback” R&D. Patent and license work Exclude All administrative and legal work connected with patents and licenses (except patent work directly connected with R&D projects). Routine tests Exclude Even if undertaken by R&D staff. Public inspection control, enforcement of Exclude standards, regulations Source: Elaboration on UNESCO (2009) Table 5.9: Summary of Recommendations from Frascati Manual on How to Estimate GBAORD Category Principles Detailed considerations and examples Type of expenditures All outlays on government expenditure on Particular emphasis is given to distinguishing R&D from non-R&D activities. covered R&D-related activities. Covers both current costs and capital expenditures, including social security Follows the concept of R&D activities as funds (§494 and 491). defined in chapter 2 of the Frascati Manual Contributions to international R&D programs and institutions should be (§481-485). included (§496). For example, GBAORD excludes the share of prototype development and development contracts not related to R&D, such as the manufacturing of a prototype itself. For example, appropriations to the Consultative Group on International Agricultural Research (CGIAR) should be included. Level of government Central government should always be included. Includes public general university funds (GUFs). covered State level governments should be included when spending is significant. Local governments should always be excluded Data classification and Spending should be reported by purpose of For example, a research project to develop fuel cells to provide power in reporting socioeconomic objective, and on the basis of the remote areas financed by the Ministry of Agriculture should be classified as intended use of the funds at the time they are “agriculture, forestry, and fishing,” but the project content is “energy.” committed (§497-499). The actual reporting level chosen will depend on practical possibilities. Data collection GBAORD data should be based on the funder Data is basically obtained by means of text analysis, document reviews, and rather than the performer. subsequent data validation process. Data should be collected by national statistical institutes. Sources Budget proposals (figures presented to parliament). Forecast (estimates of funding before beginning of budget discussion). Initial budget appropriations (figures as voted by Final budget appropriations (initial budget plus changes introduced during the parliament). year). Data drawn from ministries, universities, and other administrative sources. Source: Elaboration of data from OECD (2002) and Eurostat (Reference Metadata Structure). Note: Numbers in parentheses are section numbers from the Frascati Manual (OECD 2002). Functional Review 49 Table 5.10: GBAORD—National Data Collection Schemes (as of 2009) Country Government coverage Sources Stages of data collectiona Bulgaria Central government GBAORD surveyb and budget Final data: vii Cyprus Central government National data Provisional data: iv Local government National budget data Final data: v Czech Republic Central government STI Information System (STI state budget) Provisional data: iv Provincial government Universities for Sector of Economic Activity Final data: vii Latvia Central government Ministry of Education and Science Four stages (no detail available) Lithuania Central government National budget account of Ministry of Finance All stages (i–vii) Hungary Central government Budget units (mainly ministries) Final data: vii Poland Central government Annual Report on the Execution of the Budget for Final data (no detail available) Science—Ministry of Science and Higher Education Romania Central government Ministry of Education and Research and the Romanian Final data: v Academy of Science and other government levels managing funds Slovenia Central government Ministries Provisional data: iv Final data: vii Slovak Republic Central government Ministry of Education Provisional data: iv Final data: vii Source: Eurostat (2012) Note: a. The stages of data collection include: (i) forecasts (estimates of funding before beginning of budget discussion); (ii) budget forecasts (preliminary figures as requested by ministries, especially for inter-ministerial discussions); (iii) budget proposal (figures presented to the parliament for the coming year); (iv) initial budget appropriations (figures as voted by the parliament for the coming year, including changes introduced in the parliamentary debate); (v) final budget appropriations (figures as voted by the parliament for the coming year, including additional votes during the year); (vi) obligations (money actually committed during the year); and (vii) actual outlays (money paid out during the year). b. Survey of expenditures. Indicators from the OECD following indicators are useful for the purpose of the Policy Mix Database PER exercise: In addition to the GBAORD indicators, the OECD Policy • GBAORD: Public spending to PROs versus the private Mix Database produces information on a number of sector. aspects of public spending in R&D based on the frame- work of the International Survey of Resources for R&D • Private sector: direct support versus indirect support. (OECD 2010b). Data availability varies by indicator but is • PROs: Competitive versus institutional block grants, mainly focused on OECD countries with some indicators HEI versus public research institutes, basic versus available for Czech Republic, Estonia, Hungary, Mexico, non-basic research; socioeconomic objectives, and Poland, the Slovak Republic, Slovenia, and Turkey. The civil society versus non-civil society research. 50 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Government-Financed Gross sciences, and humanities); and (iii) type of R&D activity Domestic Expenditures on R&D (basic, development, and experimental research). The sum of all the government expenses in R&D is com- monly denominated “government financed.” It is part 5.4. CONCLUSION of the R&D statistics generated by national statistical offices according to the “source of funds” for R&D This chapter discussed the analysis of budgetary data. It activities. The information reported refers to govern- described how to collect data on STI spending and pro- ment expenditures to finance R&D projects regardless vided a proposed budget structure for STI. The proposed the executing unit. Other sources of funding for which budget structure links expenditure to the intermediate data are provided are higher education institutions, outcomes described in chapter 3. The chapter then business, private nonprofit, and abroad. Overall, R&D provided a practical example, after which a number of a expenditures are also classified by (i) sector of perfor- number of indicators where reviewed that complement mance (government, higher education institutions, the budgetary indicators. Box 5.3 describes a possible business, private nonprofit, and unspecified); (ii) field of structure for the FR report, as well as a number of use- science (natural science, engineering, and technology; ful readings. The next chapter reviews the Operational health and medical science, agricultural sciences, social Efficiency Assessment. Box 5.3: The Functional Review—Structure and Useful Readings Possible structure of the Functional Review as a standalone document: 1. Introduction: objectives and scope, as agreed in the Inception Report 2. Basic description of STI spending: Who spends, how much, and to achieve what objectives 3. Governance structure and the existing expenditures 4. Consolidated STI budget 5. Conclusions Useful readings: Fowler, Martin, Patrick Abbott, Stephen Akroyd, John Channon, and Samantha Dodd. 2011. “Forest Sector Public Expen- diture Reviews: Review and Guidance Note.” Program on Forests (PROFOR). World Bank, Washington, DC. World Bank. 2007. “Spending for Development: Making the Most of Indonesia’s New Opportunities. Indonesia Public Expenditure Review 2007.” World Bank, Washington, DC. World Bank. 2011 “Romania Functional Review: Research, Development, and Innovation Sector.” World Bank, Washington, DC. Functional Review 51 CHAPTER 6 OPERATIONAL EFFICIENCY ASSESSMENT The Operational Efficiency Assessment (EA) is the third implementing an EA and approaches to overcome step in the implementation of the PER (see figure 6.1). them. Section 6.2 describes how to evaluate the The main objective of the EA is to provide an evalua- outputs produced by programs relative to their in- tion of the efficiency of the supported programs and puts. This is followed by section 6.3, which details therefore of the public expenditures. how to evaluate the design and implementation issues that affect operational efficiency. Section 6.4 The chapter begins with an overview of the EA in concludes by describing approaches to assess pro- section 6.1. This section describes the challenges grams’ efficiency. Figure 6.1: The Operational Efficiency Assessment Operational Functional Effectiveness Inception Report Efficiency Final Report Review Assessment Assessment Summary The objective of the EA is to establish whether programs and funded activities are efficient. In other words, do they lead to the expected outputs given a reasonable level of inputs? The EA also aims to determine what design and implementation issues are affecting the efficiency of programs and funded activities. The EA focuses on a selection of initiatives that are agreed in the Inception Report. These should represent a substantial proportion of public expenditures on each of the intermediate outcomes considered. There are a number of challenges in reviewing the selected programs, including measuring and valuing outputs. Another challenge is measuring the benefits from a program that spill over to institutions that did not participate in the programs. The EA uses results from the Functional Review to select programs for more detailed analysis. Sources of data for the EA include (i) survey(s) of beneficiaries, (ii) peer and panel reviews, and (iii) focus groups and case studies. Microeconomic modelling may also be used. 52 Operational Efficiency Assessment 6.1. OVERVIEW Challenges There are several challenges in assessing the operational Whereas the Functional Review (FR) is intended to efficiency of programs. They include the time horizon provide a comprehensive overview of expenditures, of the evaluation, measuring and valuing outputs, and implementation of the EA should focus on selected addressing spillover effects (see box 6.1). Note that the programs for feasibility reasons. An agreement achieved concrete nature of those challenges as well as their in the Inception Report stage should have defined these importance will be program specific. priority programs. For instance, research programs often aim to improve • To the greatest extent possible, however, these pro- the research capacity of young scientists. Research ca- grams should represent a sufficiently large share of pacity is, however, an intangible output for which indict- the public expenditure in each of the intermediate ors such as hours of training and number of researchers outcomes considered. The classification of public trained are poor metrics. Agreeing on what to measure spending according to the intermediate outcome, and which metrics to use is of crucial importance for performed in the FR phase, will help the analyst to program evaluation. check the balance of programs. • Data for this work will be generated by survey of Another important challenge in assessing the efficiency beneficiaries or collected from program managers of programs refers to the attribution problem, that is, (when available). The OECD has performed a num- connecting the intervention with the estimated effect. ber of these surveys and examples of the question- The issue is to estimate how much output would have naires used can be found in OECD (2003), OECD been generated in the absence of intervention, every- (2008b), OECD (2008c) and OECD (2013b). thing else held constant. Box 6.1: Challenges to Assessing the Efficiency of Programs • Nature of “success and failure” in research and innovation. The metric of success in some policy domains is fairly straightforward. For example, children vaccinated versus children not vaccinated is a clear metric for success and failure of immunization programs. In other cases, the failure of a project—scientific or innovation experiment—funded by a government is not necessarily a metric that indicates failure of the investment program that funds the project. • Time horizon. The time lag between the expenditure and the desired effect varies. For example, the impact of public support for additional firm R&D may be observable in the short term. Exports, on the other hand, are more likely ob- servable over the longer term. • Measurement. Tangible quantitative results, such as sales growth, are easier to measure than less tangible outputs, such as training received. • Valuing knowledge outputs. Even quantifiable knowledge products may be hard to quantify and compare. For instance, the value of intellectual property (and thus of a patent or spinoff company) depends on a number of assumptions and subject to debate. • Addressing spillover effects: Spillover effects are non-market effects on third parties (rather the direct beneficiary of intervention). For instance, a series of failed innovation attempts may generate enough information to enable the suc- cess of others. Accounting for such effects is not straightforward. For example, studies have argued that traditional output indicators—such as patents, sale of new products, and profit margins—fail to capture the full effects of R&D programs. Therefore, Buisseret et al. (1995) advocated that it was necessary to account for changes in the breadth of innovation activities and corporate business/technology strategies. These ideas became associated with the concept of “behavioral additionality.” Operational Efficiency Assessment 53 For example, simply comparing the government spend- what is to be measured (for example, spillovers); and ing of a matching grant to promote business R&D with the attribution problem. Note that the definition of the amount of R&D invested by the program beneficia- impact is essentially the difference between the ries may not give you a good estimate of the program program results achieved for targeted beneficiaries impact. Some of that investment could have occurred in discounted by program benefits obtained by non- the absence of the program, a situation in which public targeted beneficiaries. funds would be “crowding out” private investments. Also, there is the possibility that events different from A related topic is to learn whether beneficiaries adjust the measure itself (say for instance the location of a strategically to the program and simply reduce their research department of a foreign company) increased investments, replacing them with funding from the pro- the business sector’s propensity to invest. gram (a substitution effect). This issue of “additionality” is a central topic in the evaluation of programs. Around The challenges of evaluating STI programs are fur- half of the innovation policy evaluations in Europe (con- ther illustrated in figure 6.2. The figure depicts the ducted between 2002 and 2007) investigated the issue challenges in terms of timing (short-, mid-, or long- of behavioral additionality implicitly or explicitly (Gök term results); type of results (for example, higher R&D and Edler 2011). Three types of additionality impact are in the firm versus innovation and productivity gains); often considered in this literature: Figure 6.2: Challenges in Program Evaluation Short term Mid term Long term Inter- and intra- NEW: industry diffusion Businesses Market expansion Products Growth Performance Processes of awardees* Spillovers Economic effects Additionality Increased R&D Impact spending, expanded goals, acceleration of collaboration, technological advances Non- participants −1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Announce Announce competition award Project life Post-project period Prospectively designed Impact Evaluation *Note: As discussed in the main text the evaluation should consider the impact of spillovers—that is, benefits to firms that did not receive awards. 54 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note • Input additionality is the most widely used concept • The description above can help the analyst choose for measuring effectiveness of STI programs and the appropriate metric for the time of evaluation. deals with the extent to which a firm’s inputs to the For instance, project completion can be chosen as STI process (often firms’ spending) have changed the evaluation metric if it is considered too early to due to the intervention (Clarysse et al. 2009). publish results. Alternatively, one could argue that papers submitted would be a more robust evaluation • Output additionality deals with the extent to which measure. The evaluation could be done in stages for the firm’s output has changed as a result of having which the “paper submission” and “paper accep- received a subsidy. Apart from the standard output tance” performances could be compared.1 indicators, output additionality includes the follow- • The description above also helps localize the main ing: (1) increments to the firm’s stock of knowledge factors affecting the emergence of expected impacts capital resulting from the R&D project; (2) develop- within that chain of events. This in turn allows the ment of the firm’s capabilities, which might influ- EA to define the main bottlenecks for success. For ence subsequent R&D productivity; and (3) benefits example, the success of a well-defined and designed derived by the firm from commercial application of program may be hindered by poor implementation the R&D result (Roper et al. 2004). of calls (not enough coverage, or insufficient time • Behavioral additionality deals with changes in firm between announcement and application deadline) behavior that resulted from having received public or long delays in the disbursement of funds. support. Behavioral additionality includes the chang- es in the breadth of innovation activities, changes Key Assessment Issues in technological and business strategies of the firm, Once the program details are well understood, the and changes in the capacity of the firm to engage in evaluation can be planned. The core of the operational innovative processes (Buisseret et al. 1995). efficiency assessment consists of the following set of issues: (i) what is the rationale for the program, (ii) how Need for a Tailor-Made Approach the program is designed, and (iii) how the program is implemented (Stiglitz 2000). Consistency among those The wide range of research and innovation measures three elements of the intervention is essential for its implies the adoption of tailored evaluation approaches. success. A tailor-made evaluation requires detailed knowledge of the logic of the intervention. The rationale is similar In terms of the rationale for the program, it is gener- to the one described in Chapter 3 except for the level ally accepted that an intervention is more likely to be of details and for its focus (on one program rather than successful if it focuses on the market, institutional, or the whole of public spending). In this case, however, it systemic failure that it aims to correct. While a widely may be useful to further break down the problem into known principle, very often programs will depart from several additional steps. This further detailing may help it for different (mostly political economy) reasons. Fail- the analysis to address some of the challenges above. ures that are addressed by STI investment include the following: • For example, a research grant program, typical a two-stage chain of events (funding → publication), • Market failures are often associated with time- could be divided into several more steps: available inconsistent preferences, information asymmetries, funding → call for proposals → project applied → project selected → project implemented → project 1. This is an example of the importance of leaving, as the result of the work, a good monitoring mechanism. Without completed → paper submitted for application → such an instrument in place, the additional information and paper accepted for publication → paper published. analysis would be unfeasible. Operational Efficiency Assessment 55 Box 6.2: How the Call for Proposals and the Project Evaluation Stages May Affect Program Efficiency Calls for proposals may be open-ended or have a fixed deadline; while selection may be done only by national experts or include international experts. In both cases the options involve important trade-offs with different implications for the efficiency of the program. Open-ended calls impose, for example, less burden on the applicants (and perhaps on program management) but raise the risk of selecting projects that perhaps would not rank among the best options (and perhaps do not merit support). Close-ended calls improve the selection mechanism toward the best projects—at least among the applicants—and thus use public resources more efficiently. Yet, some argue that those administrative deadlines may not be consistent with scientific research. Not surprisingly, several top research organizations keep using open-call systems for some programs. Selection processes limited to national experts may compromise the independence of evaluations—especially when the local scientific community is small. Program managers, however, sometimes argue that access to international experts is impractical, costly, and runs the risk of excluding scientists that do not master a foreign language. non-competitive markets, principal-agent problems, whether the “failure” originally identified is remedied. externalities, or public goods.2 Two aspects are of particular interest: open-ended ver- • Institutional failures encompass traditional “gov- sus closed-ended call for projects, and national versus ernment failures,” as for example poorly defining international evaluation of projects (see box 6.2). property rights or enforcing contracts. They also include the failure to establish a system of rules that 6.2. OUTPUT ASSESSMENT encourages individual interactions according to the common interest (Hodgson 2006). The primary objectives of the EA are to address the two • Systemic failures refer to the risks imposed by inter- questions: (i) do programs and funded activities gener- linkages and interdependencies in a system, includ- ate the expected results with a reasonably amount of ing a national innovation systems or financial system. inputs, and (ii) what design and implementation issues are affecting those results? This section discusses how When designing a program, the eligibility criteria for selecting the target group need to be consistent with to address the first question. The second question is the rationale/objective of the program. While it is not addressed in section 6.3. possible to identify perfectly those truly deserving sup- port, adherence to the original objective of the program Tables 6.1a and 6.1b summarize the first task of the is a way to minimize the two typical selection errors: EA: combining inputs and outputs for the subsequent denying support to those who deserve and need it, and analysis. Note that the first two columns are generated helping those who do not deserve or need support. in the previous stage (FR)—as the STI budget has been completed. The third column (Outputs) is therefore the Among several implementation issues, the selection focus of attention. In order to fill the table, three sources process—the criteria by which beneficiaries of the in- of information are envisaged: surveys of beneficiaries, tervention are selected—is very important. These criteria monitoring reports, and focus groups. determine if the targeted group is actually helped and Box 6.3 illustrates the application of the survey of ben- 2. Those market failures are not mutually exclusive. Informa- eficiaries, combined with interviews with the program tion problems often provide part of the explanation of missing markets. In turn, externalities are often thought to raise from managers and focus groups with beneficiaries for an missing markets. See Stiglitz (2000). applied research program in Poland. 56 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 6.1a: Input-Output Metrics—Illustration Intermediate outcome Policy/program (budget/disbursement) Outputs National Science Foundation Research Grants Research projects funded Research projects completed (number and value) Note: Government executed GERD would provide an Research excellence aggregate figure Academy of Science’s Program for the Advancement of Research Research Institute for Marine Biology Research projects with the private sector (number and Institute for Agricultural Research value) Note: Distribution between basic and experimental research funded by the government is a first Science-industry approximation (from budget) collaboration Research Institute for Marine Biology Institute for Agricultural Research Innovation Vouchers Program (from Ministry of Value of disbursements and number of firms covered Economy) Cost of Tax-breaks for business R&D (forgone Value of business R&D Business investments in revenue) R&D Matching grants Program for or early stage Number and value of knowledge-based startups created funding from Ministry of Science that received funding from the program Technology Extension Services and Matching Number of firms assisted Technology adoption grants from Min. of Ind. Number of firms certified Number of individuals trained Agricultural Extension Services from Ministry of Agriculture Table 6.1b: Example of Input-Output Indicator Program/organization and budget Output type (example) Indicator(s) Estimate Doctoral and post-graduate training Hours of training 800 hours of classroom work delivered Courses created 30 new courses created Program to fund scientific MA or PhD programs created 4 new MA programs research/national science foundation 10 new PhD programs Budget: US$25 million Research projects funded Number of projects Papers published in top journals H-level of publications Modernization of infrastructure Value of infrastructure investment 6.3. ASSESSMENT QUESTIONS General • What is the stated objective of the program? Are Programs can deliver different levels of output with the sustainable, measureable, achievable goals defined? same level of inputs depending on the way they are Is the target group well identified? designed and implemented. The following questions • How do observable outputs compare with expected help assess the how effectively programs are translating results? Is the target group being adequately cov- inputs into outputs. ered? Is the funding provided adequate and timely? Operational Efficiency Assessment 57 Box 6.3: Assessing the Outputs of STI Program with a Survey of Beneficiaries—Illustration from Poland Mid- term Evaluation In October 2013, Poland’s National Center for Research and Development (NCBIR) engaged the World Bank to undertake a mid-term review of the Applied Research Program. The analysis included an evaluation of program design and implementation and success in tar- geting of intended beneficiaries. It relies on the information obtained from interviews with program managers, a survey of beneficiaries and applicant non-beneficiaries, and focus group discussions. Table B6.3.1 illustrates the results for the Applied Research Program (PBS) programs according to the respondent (leader [LEA] vs. partner [PAR]; sub-programs (Path A and Path B) and call for proposals (CF1 and CF2). PBS is a research grant program dedicated to promoting research collaboration. Publications are the main output of program beneficiaries, followed by master theses, and lastly new product prototypes. A small percentage of beneficiaries announce creation of a new product or upgrading an existing product. These results are in line with the fact that most beneficiaries are scientific units. They also demonstrate the program’s poor performance in generating outputs with high economic impact. • Publications appear as the most important output, followed by MA and PhD theses, and new product prototypes. • Leaders published more and provided more MA theses than partners: 57 percent versus 40 percent, and 19 percent versus 9 percent, respectively. Table B6.3.2 provides some quantitative indicators. It shows that publications are the most important output. The mean of publications is higher for leaders, 2.1, than for partners, 1.2. Results in Path A (2.0) are larger than in Path B (1.3). And since innovation takes time, the mean of publications in CFP1 is higher than that of CFP2 (2.1 versus 0.8). The mean for the rest of the outputs does not exceed 0.5. Table B6.3.1: Outputs Generated by Beneficiaries Table B6.3.2: Mean of Produced Outputs LEA PAR PA CFP1 CFP2 LEA PAR PA CFP1 CFP2 Patent 12% 9% 14% 12% 9% Patent 0.2 0.1 0.2 0.1 0.1 Industrial design 1% 2% 2% 1% 2% Industrial design 0 0 0 0 0 New product prototype 14% 15% 16% 16% 10% New product prototype 0.3 0.3 0.3 0.4 0.1 New product 8% 9% 11% 9% 7% New product 0.1 0.1 0.2 0.1 0.1 Upgraded product 10% 5% 8% 9% 5% Upgraded product 0.4 0.1 0.2 0.3 0.1 New processes 13% 14% 13% 15% 10% New processes 0.2 0.6 0.2 0.5 0.2 Upgraded processes 12% 8% 11% 12% 7% Upgraded processes 0.2 0.2 0.2 0.2 0.1 Publications 57% 40% 55% 61% 22% Publications 2.1 1.2 2 2.1 0.8 Master theses 19% 9% 18% 15% 10% Master theses 0.5 0.1 0.4 0.3 0.2 Ph.D theses 9% 5% 9% 7% 7% Ph.D theses 0.1 0.1 0.1 0.1 0.1 Other 8% 7% 9% 10% 3% Other 0.2 0 0.2 0.2 0 Other benefits. Yet, a large proportion of participants strongly agree that the program increased firms’ innovative capacity. This includes increasing the internal knowledge and capabilities of employees, a better understanding of issues and problems, and better use of existing know-how. However, only 10 percent of beneficiaries strongly agree that the program improved companies’ competitive position both nationally and internationally. A similar ranking is observed for expected benefits (see figure B6.3.1). Figure B6.3.1: Other Benefits Improve competitive position internationally 36% 28% 18% 9% New employment of profesionals 46% 20% 16% 10% Improve competitive position nationally 32% 24% 25% 10% 1 (no benefit) Increased effeciency 18% 26% 34% 16% Use of new equipment 26% 18% 28% 21% 2 Enhanced reputation and image 15% 25% 33% 22% 3 Additional innovation 18% 19% 31% 26% Increased capacity for conducting innovation 12% 18% 40% 27% 4 Networking and development colloboration 12% 22% 33% 27% Use of existing know-how 10% 20% 35% 32% 5 (high benefit) Better understanding of issues and problems 8% 17% 35% 37% Extended internal knowledge and capabilities 7% 11% 35% 44% 0% 50% 100% Source: World Bank 2014. 58 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note • Are there reasons to suppose that the beneficiary • How is the decision-making process conducted? would have adopted the expected change in Who takes the final decision—panel or experts, behavior without the intervention? If so, to what investment committee, director-general? extent? • Which criteria were employed to determine the number of beneficiaries in each call? Is there a rank- Input-Output ing of the application results? • Could funding amounts be reduced without com- prising the current performance? Could higher Implementation Issues output levels be achieved with the existing funding • Are applications managed on an open-ended basis level? Are there similar sources of funds to which or in the format of closed calls? How many calls for potential beneficiaries do or could apply for? proposals were conducted (if not open-ended call)? • Is it possible to reduce administrative costs without • Is the application process clear and transparent? affecting the quality of program management? Is Does it take too long or is it too expensive to apply it possible to reduce transaction (monetary and for the program? Is the timing of the call for propos- nonmonetary) costs for applicants? als appropriate? • Is the selection process transparent and fair? Who The following aspects of program design and imple- integrated the selection committee? Does it use mentation have a direct effect on program perfor- international peer reviewers? Are evaluators’ skills mance: economic rationale, eligibility criteria, selection consistent with the project goals? process, decision-making process, and management conditions. Implementation Conditions Economic Rationale • Management. How are funds disbursed? What are the reporting requirements to beneficiaries? What is • What specific market, institutional, or systemic the frequency of the field visits? What are the audit- failure justifies economically the intervention? How ing requirements? How are programs monitored? is the intervention supposed to correct the market failure (that is, what changes in the behavior of • Staffing Issues. Is the staff properly paid and well- economic agents are expected to be generated)? trained? Are other material conditions (physical and financial infrastructure) commensurate with • How is the program or expenditure expected to the workload? Are they sufficiently insulated from contribute to the defined outcomes? How does major political pressures? Are they incentivized to the program complement other existing programs? improve performance? Program Design 6.4. METHODOLOGICAL ISSUES • Do the eligibility criteria reach the right target group? Are there unnecessary criteria? Are there Different methodologies have been applied to evalu- missing criteria? ation of research and innovation programs. All meth- • Which criteria were employed to determine the odologies show important trade-offs in terms of the number of beneficiaries in each call? Is the selec- quality of analysis and the resources requirements (see tion too restrictive (excluding proposals of sufficient table 6.2). Moreover, the types of conclusions and quality) or too loose, thus supporting a proposal of analysis allowed by each of methodologies are often insufficient quality? Are proposals order? complementary to each other. For this reason, to the Operational Efficiency Assessment 59 Table 6.2: Summary of Program Evaluation Methodologies Methods Description Pros and cons Microeconomic From reduced-form modelsa to The most robust type of evaluation (with randomized control trials as modelling randomized control experiments using the gold standard). Depends on high-quality data (often panel data) firm or individual level data not always available. Scope of analysis may be narrow. Survey of beneficiaries Generate qualitative (soft) data from Simple to implement and relative low cost. Effective in generating program’s applicants (beneficiary and output information. Unable to credibly address the attribution non-beneficiary. problem. Subjectivity of responses. Peer/panel reviews Use of international experts to assess Simple to implement. Grants access to program specific expertise. the quality of the program, often Useful for the assessment of implementation and administrative benchmarking the program against an aspects. Risk of persons/country biases and limited use of data-based assumed good practice evidence. Focus groups/case Structured interviews of program clients Simple to implement. Good starting point for the understanding of studies for the understanding of the strengths the intervention logic. Does not allow generalized conclusions about and weaknesses of the program. the program. Source: Adapted from Technopolis (2009). Note: For an illustration see Ozcelik and Taymaz (2008). extent possible, a combination of the four approaches To obtain information about program outputs, three is recommended. sources of information are envisaged: interviews with managers (and monitoring reports); and focus groups • Microeconomic modelling, of which randomized through semi-structured interviews and surveys of control trials are the gold standard of impact evalu- beneficiaries. With the survey of beneficiaries it is also ation but are also often difficult to implement due to possible to obtain a first, tentative approximation to data requirements, time limitations, and (sometimes) the attribution problem. As a rule, however, the team is political sensitivities. advised to look exhaustively for opportunities to imple- • Focus groups and case studies may provide a simple, ment quantitative assessments, including exploring the cost-effective way to understand the logic of the different approach to more rigorous impact evaluations intervention (including the different outputs and as discussed below. their timing). • Peer/panel reviews may be the only effective option Semi-Structured Interviews to evaluate science parks/incubators in the short Semi-structured interviews are the instruments for ex- term (as quantitative analysis—such as comparison ploratory research that precede the implementation of of the survival rates of the incubated and nonincu- the survey of beneficiaries (or any deeper quantitative bated firms—may be only feasible at a later stage). analysis). Semi-structured interviews are conducted in • Surveys of beneficiaries have a number of advantages order to explore in more detail the interviewees experi- over other approaches, but surveys come with some ence regarding the program. Prior to interviews, guides drawbacks. The data generated is more representative for interviews are developed for each type of program than that provided by focus groups, and peer/panel beneficiary. In the course of interview, questions are reviews. This comes at a higher cost as surveys are directed towards topics related to benefits and attitudes typically more expensive to design and implement. on particular program. 60 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Researcher effects are one of the possible biases in quali- gain additional insight into program additionality, ap- tative research (Miles and Huberman 1994). Involve- plicants who were denied the benefit (non-beneficiary ment of several researchers in case study development applicants) should also be surveyed. is important to avoid the researcher bias. In addition, triangulation by researcher (Denzin 1978) positively Further insights are often possible when one focuses on affects research validity. For example, more than one incremental changes. For instance, a beneficiary company researcher is involved in the interview process; notes are can be asked “would the company abandon the project returned to the interviewee before used and researchers were the grant not awarded?” If the answer is “no,” then that did not participate in the interview are used to the the beneficiary can be asked the following questions:3 report the case studies. • Scale. Would the project be performed on a smaller The Survey of Beneficiaries budget? • Scope. Would the project be performed on a less- Surveys of beneficiaries belong to the category of highly innovative level (lower risk/premium project)? structured questionnaires. Such questionnaires have a large core of common questions, but are often adjusted • Acceleration. Would the project be performed over to the surveyed population (for example, beneficiary a longer time period? versus non-beneficiary, firm versus PRO, or leader ver- • Was there a change in the non-persistent behavior sus participant) according to emphasis of the analysis. related to STI activities as a result of the intervention? Responses are typically collected on five-point Likert • Was there a change in the persistent behavior related scale (1 = lowest and 5 = highest) and sometimes use to STI activities as a result of the intervention? “yes-no” multiple choice questions. Response rates of 50–70 percent for beneficiaries (lower for non-bene- Mirroring a standard procedure in impact evaluation ficiaries) and follow-up interviews are often necessary. analysis, “before and after” questions are frequently Questionnaires broadly follow the following sections: used. For instance, beneficiaries of an innovation sup- (1) general information, (2) performance prior to the port program can be asked to agree/disagree with each grants project, (3) information about the project, (4) of the following statements: results and outputs of the project, (5) estimated impact without project, and (6) attitudes about the program • “Prior to the support provided by the program, we had no formal process of new product development Survey of Beneficiaries and but now we have it.” the Additionality Issue • “Prior to the support provided by the program grant, Additionality is described in detail in section 6.1. Can we had a formal process of new product develop- one fully address the additionality issue without a stan- ment and have now improved this process.” dard impact evaluation? The simple answer is “no.” Surveys of beneficiaries can, however, provide a first Box 6.4 illustrates the use of a survey of beneficiaries to glance at the problem. assess the impact of Croatia’s RAZUM Program. To assess the additionality issue, questions are asked Impact Evaluation involving a hypothetical “counterfactual scenario,” While useful, a survey of beneficiaries is not the in- that is, a hypothetical situation where respondents strument to be adopted if causality (attribution) is the had to imagine what would had happened in the case of not being awarded the grant (Hsu et al. 2009). To 3. For a literature review see Hsu et al. (2009). Operational Efficiency Assessment 61 central issue for the evaluation. Teams are therefore • The guideline provides ideas and technical advice encouraged to explore the data availability and consider on how to measure the effectiveness of science, the different options in terms of econometric strategy. technology, and innovation programs (STIP). It ad- Five main techniques are available (three related to dresses the specific challenges of evaluating STIP, impact evaluation and two statistical techniques often from the assessment of the intervention logic to the used for empirical micro-level work): randomization, re- choice of the most appropriate method to solve the gression discontinuity, matching, instrumental variable, attribution problem. and differences-in-differences. Table 6A.1 in annex A to • Much attention is devoted to the topic of data, this chapter presents some of the properties of those discussing pros and cons of different data sources, techniques. For a detailed review of the issues involved data quality issues, and strategies for data collec- in the implementation of impact evaluation exercises in tion. The toolkit analyzes the potential application science and technology policies see Crespi et al. (2011). of experimental and quasi-experimental methods to Box 6.4: Assessing the Impact of Croatia’s RAZUM Program RAZUM (Development of the Knowledge-Based Companies) was a conditional loan implemented by Croatia’s Innovation Agency from 2007 to 2012 that supported investments in R&D by small and medium-sized enterprises (SMEs). Behavioral and output additionaility was measured by means of a survey of beneficiaries. According to the survey’s responses, the intervention enabled companies to increase their capacity for conducting innova- tion and R&D, and to extend knowledge and capability of the staff through hiring of highly educated professionals. In most cases these changes promise to be permanent. New product development process was positively affected in a large majority of the cases, promising better innovation capability. For most companies that received RAZUM support, work on the project generated additional ideas for innovations. When asked what would have happened had they not received the RAZUM grant, 6 companies (30 percent) reported that they would have abandoned the project entirely. The majority (86 percent) of the remaining firms would have relied on their own resources, while some of them would have tried banks and venture capital funds. Three firms would have tried to find money through strategic partnerships and some other R&D subsidies. However, the absence of RAZUM money would not be without consequences. Most companies would have proceeded on a smaller budget, which would have affected the duration of the project (would have been longer), scope of the project (smaller), R&D capacity through additional employ- ment of R&D staff (lower), and innovativeness level of the project (also lower) (figure B6.4.1). A similar pattern is found in the case of non-beneficiaries. Those firms were either in the evaluation process (passed the pre-selection phase) or were approved and waiting for financing. In the hypothetical situation of not receiving RAZUM funding, 2 companies out of 14 would have abandoned this project and started another one, whereas all other companies would have proceeded with their projects (interestingly, no firm declared that it would not continue with that or any other project). However, the absence of th RAZUM grant would have had consequences for the duration, scope, R&D capacity, and overall quality of the projects (figure B6.4.2). • In the absence of RAZUM grant, the vast majority of respondents said that they would have proceeded (i) with the project but over a longer timeframe (92.9 percent), (ii) on a smaller budget (85.7 percent), (iii) with a reduced scope (85.7 percent), and (iv) with inadequate equipment and/or machinery (71.4 percent). • In terms of outcomes, many companies would have not hired additional employees (71.4 percent), and the innova- tiveness level of the output would have been lower (42.9 percent). (continued next page) 62 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 6.4 (continued) Figure B6.4.1: Opinions of Consequences of Not Figure B6.4.2: Opinions of Consequences of Not Receiving RAZUM Grant: Beneficiaries (n=20) Receiving RAZUM Grant: Non-beneficiaries (n=14) Number Number of companies of companies Number Number of companies of companies 0 0 2 2 4 4 6 68 108 10 14 14 12 12 16 16 0 0 5 5 10 10 15 15 Our Our company company havehave would would proceeded proceeded withwith the project but aon a small 14 14 Our Our company company wouldwould go with go with the the the project but on small project project but over but over a longer a longer timeframe timeframe 13 13 budget budget Our Our company company would havehave would proceeded proceeded withwith the project the project but over but over a longer a longer 13 13 Our Our company company would would havehave proceeded proceeded timeframe timeframe withwith the project the project aon but but on a small small budget budget 12 12 Our Our company company would havehave would proceeded proceeded withwith the project the project but with but with reduced reduced 12 12 Our Our company company wouldwould the the go with go with scope scope project project but with but with reduced reduced scopescope 12 12 Our Our company company would havehave would proceeded proceeded withwith the project the project but without but without 11 11 Our Our company company would would the the go with go with additional additional employment employment project project but without but without additional additional 10 10 Our Our company company wouldwould havehave proceeded proceeded employment employment withwith the project the project but innovation but the the innovation 8 8 levellevel wouldwould be lower be lower Our Our company company would would go with go with the the project project withwith but but inadequate inadequate 10 10 Our Our company company would would havehave proceeded proceeded equipment and/or machinery equipment and/or machinery withwith the project the project but without but without 7 7 collaboration collaboration withwith universities/Ris universities/Ris Our Our company company wouldwould go with go with the the project project but innovation but the levellevel the innovation wouldwould 6 6 Our Our company company would havehave would proceeded proceeded be lower 6 be lower withwith the project the project at same at the way way the same we we 6 did (are doing) did (are doing) at RAZUMat RAZUM Our Our company company wouldwould the the go with go with Our Our company company wouldwould not have not have gonegone project project but without but without collaborating collaborating withwith 6 6 ahead ahead withwith this project this project but we we would butwould 5 5 firms firms have done another one have done another one instead instead Our Our company company would would the the go with go with Our Our company company would havehave would proceeded proceeded project project but without but without colloberation colloberation withwith 5 5 withwith the project the project but with but with inadequate inadequate 4 4 universities/Research universities/Research Institutes Institutes equipment and/or equipment and/or machinery machinery Our Our company company wouldwould havehave proceeded proceeded Our Our company company would would not not go go ahead withwith ahead withwith the project without collaborating 4 the project the project butwe butwe would would startstart another another 2 2 the project without collaborating 4 withwith firmsfirms one one instead instead Out Out company company wouldwould not have not have gone gone Our Our company company wouldwould not not go go ahead ahead withwith withwith ahead ahead the project the project and and we would we would 2 2 the project the project butand butand we would we would do do0 0 not have not have donedone another another one one instead instead another another one one instead instead Source: Elaboration of data from Radas et al. (2011). STIP. For each method, the paper highlights charac- ter described how to how to assess outputs. It reviewed teristics and assumptions, practical issues related to the need to evaluate whether the outputs generated the implementation, and strengths and weakness by programs were additional. This was followed by specifically related to the application to STIP. discussion of a number of questions that can be used in outputs assessments. The chapter then reviewed methodologies that can be used in implementing pro- 6.5. CONCLUSION gram evaluations. Box 6.5 outlines a possible structure for the EA as well as a number of useful readings. The This chapter discussed how the EA is used to review the next chapter describes how to evaluate the effective- efficiency of STI programs. The first section in this chap- ness of STI programs. Operational Efficiency Assessment 63 Box 6.5: The Operational Efficiency Review—Structure and Useful Readings Possible structure of the EA as a standalone document: 1. Introduction: Objectives and Scope, as agreed in the Inception Report 2. Overview of Programs and Funded Activities included in the Review 3. The Efficiency of Program and Funded Activities 3.1 Program or funded activity A–Z (a section for each one reviewed): 3.1.1 Description of the program 3.1.2 Review of the program’s outputs 3.1.3 Evaluation of the program’s efficiency 3.2 Summary of findings 4. Design and Implementation Issues 4.1 Review of governance issues affecting efficiency 4.2 Proposals for interventions to improve efficiency 5. Conclusions Useful readings Edler, Jakob, Paul Cunningham, Abdullah Gok, and Philip Shapira. 2013 “Impacts of Innovation Policy: Synthesis and Conclusions Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project.” Manchester Institute of Innovation Research, Manchester Business School, funded by NESTA OECD. 1997. “Policy Evaluation in Innovation and Technology: Towards Best Practices.” OECD, Paris. Ruegg, Rosalie, and Irwin Feller. 2003 “A Toolkit for Evaluating Public R&D Investment Models, Methods and Findings from ATP’s First Decade.” Prepared for the Economic Assessment Office Advanced Technology Program, National Institute of Standards and Technology, Gaithersburg, MD. 64 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note ANNEX A. FIVE IMPACT EVALUATION TECHNIQUES Table 6A.1: Empirical Approaches to Impact Evaluation and Other Statistical Techniques Impact evaluation technique Definition Main advantage Feasibility challenges Randomization Individuals/communities/firms are Often addressed to as the “gold Political constraints, especially for randomly assigned into participation standard” ongoing programs. Counterfactual: randomized-out group By design: selection bias is zero Internal and external validity issues on average and mean impact is revealed Difficult to extrapolate the results to a larger population Perceived as a fair process of allocation with limited resources Regression Exploit the rule generating assignment Identification built in the Threshold has to be applied in discontinuity into a program given to individuals only program design practice, and individuals should not above a given threshold be able manipulate the score to Delivers marginal gains from the become eligible Counterfactual: individuals just below program around the eligibility the cut-off who did not participate cut-off point Matching Match participants with non- Does not require randomization, Requires very good data: need to participants from a larger survey nor baseline (pre-intervention control for all factors that influence data) program placement Counterfactual: matched comparison group. Each program participant Requires significantly large sample is paired with one or more non- size to generate comparison group participants that are similar based on observable characteristics Instrumental Identify variables that affects Does not require the The estimated effect is local: IV variables (IV) participation in the program, but not heterogeneity assumption of identifies the effect of the program outcomes conditional on participation matching only for the sub-population of those (exclusion restriction) induced to take up the program by Easier to implement (once the instrument Counterfactual: The causal effect the IV is identified) and less is identified out of the exogenous demanding in terms of data Therefore different instruments variation of the instrument collection identify different parameters Difference-in- Observations over time: compare Can be in principle combined Requires at least two cross-sections difference observed changes in the outcomes for with matching to adjust for of data, pre-program and post- a sample of participants and non- pre-treatment differences that program on participants and non- participants affect the growth rate participants Counter-factual: changes over time for Need to think about the evaluation the non-participants ex-ante, before the program Source: Elaboration from Goldstein (2010). Operational Efficiency Assessment 65 CHAPTER 7 EFFECTIVENESS ASSESSMENT The Effectiveness Assessment (EFA) is the fourth com- The chapter begins with an overview of the EFA in ponent of the Public Expenditure Review (PER) (see section 7.1. This section describes the main questions figure 7.1). The EFA evaluates the extent to which policy that are answered and the factors that inhibit the outputs are being transformed into expected outcomes. transformation of outputs into outcomes. A number of This includes evaluating what factors beyond the reach indicators for intermediate outcomes are described in of existing interventions affect the achievement of ex- section 7.2. This is followed by three sections that out- pected outcomes (the “conditions for effectiveness”). line what questions need to be answered to understand Figure 7.1: The Effectiveness Assessment Operational Functional Effectiveness Inception Report Efficiency Final Report Review Assessment Assessment Summary The objective of the EFA is to determine whether the expenditure and related outputs are leading to intermediate outcomes, such as an increase in the volume and quality of scientific papers, increased licensing by research institutions, and increased adoption of technologies in the business sector. The EFA then evaluates what conditions are facilitating or inhibiting the intermediate outcomes from being generated. The EFA will typically analyze the national innovation system (NIS) by considering four sets of intermediate outcomes: (i) research excellence, (ii) science industry collaboration and technology transfer, (iii) business R&D and startups, and (iv) non-R&D innovation and technology adoption. These outcomes are measured by analyzing a number of indicators, which provides an indication of where the system is working effectively. These findings allow evaluation of the conditions for effectiveness, include issues such as the presence of appropriate research infrastructure or the presence of appropriate intellectual property regulation. The starting point for the EFA is analysis of outputs from the Operational Efficiency Assessment. Additional data for the EFA will come from a combination of publicly available sources, surveys, interviews of program managers, R&D statistics, and budgetary information. Information on the conditions for effectiveness will largely come from a policy survey, interviews, and survey data on the enterprise sector. 66 Effectiveness Assessment whether the conditions for effectiveness are present. policy outputs into one of the identified intermediate Section 7.3 addresses research excellence; section 7.4 outcomes. In broad terms, those are binding market, addresses science-industry collaboration and technol- institutional, or systemic failures, that hinder the impact ogy transfer; and section 7.5 addresses business R&D, of public investments in STI. What market, institutional startup creation, and technology adoption. or systemic failures commonly hinder the impact of public spending on STI? What factors are affecting the impact of programs and public spending? Without the 7.1. OVERVIEW ambition of being exhaustive this section summarizes some central issues for each of the four intermediate The Effectiveness Assessment (EFA) aims to assess the goals established above. extent to which each of the four outcomes defined in the intervention logic are being reached, as a result • Research excellence may be affected by (i) the gov- of the STI policy intervention. This implies moving up ernance regime of public research organizations; (ii) one level, from the analysis of first order effects—as described in the previous section—to the analysis of access to research infrastructure; (iii) availability of second order effects, or outcomes. well-trained researchers; and (iv) access to research funding. This is done with the objective of understanding if the • Science-industry collaboration and more efficient policy intervention is leading the emergence of changes, technology transfer may be affected by (i) the and whether these changes are in line with the original incentive regime under which researchers and objectives. Note that the production of outcomes is not public research organization (PROs) operate; (ii) in full control of those responsible for implementing the existence and quality of intermediaries, such as programs. This is unlike outputs, which are a direct result technology transfer organizations; (iii) mechanisms of program implementation (for example, the number of collaboration such as voucher schemes, joint of projects financed). research projects, and centers of competence; and (iv) the availability of hard infrastructure such as Assuming the default intermediate outcomes (see techno- and science-parks. chapter 3) are used, four intermediate outcomes are • Business R&D and firm startup may be inhibited considered (note that business innovation is separated by (i) business environment factors (such as entry into two separate intermediate outcomes in order to and exit regulations); (ii) support to business invest- facilitate the analysis): ments in R&D; (iii) access to mentorship, incubation services, and early stage finance; and (iv) public • Are science, technology, and innovation (STI) expen- procurement. ditures promoting research excellence? • Non-R&D innovation and technology adoption • Are STI expenditures stimulating better science- may be stalled by (i) import costs of machinery, industry collaboration and more efficient technology equipment, and intermediate goods; (ii) access and transfer? quality of manufacturing extension services; (iii) • Are STI expenditures enabling business research and standard regulation and access to metrology and development (R&D) and firm startups? quality services (metrology, standards, testing, and • Are STI expenditures stimulating non-R&D innova- quality [MSTQ] systems); and (iv) labor skills and tion and technology adoption? access to credit. As discussed in chapter 3, factors beyond the reach The next section presents detailed discussions for each of public spending may affect the transformation of issue listed above. It provides a checklist of the main Effectiveness Assessment 67 factors to be aware of, in order to be able to assess the 7.2. LINKING INPUTS extent to which each “condition for success” exists in WITH OUTCOMES a particular country. This checklist can be completed through the use of standard quantitative indicators Table 7.1a describes the types of outcomes that will be (some of these indicators are provided in the following quantified and measured in order to establish to what sections), or through qualitative assessment tools such extent the four proposed outcomes are being gener- as interviews. The issues suggested are by no means ated. Table 7.1b provides an example of how this table exhaustive, nor are they all relevant for all countries. can be structured, again using the “research excellence” They are meant to be used as a reference source for outcome as an example. the implementation of this exercise. The first step is to define a streamlined list of outcome To illustrate, an agency implementing a program can types that will be associated to each of the four pro- ensure that a research project by a PRO is selected and posed outcomes. Similar outcome types and indicators financed (output), the agency does not fully control can be used for multiple programs implementing activi- whether the research project leads to the publication ties of the same nature. Once this is done, indicators of a scientific article in an international peer-review and measurement units for each outcome type should journal (outcome). be defined. Outcome types and their respective indica- Table 7.1a: Intermediate Outcomes—Illustrative Metrics Intermediate outcomes Possible indicator Volume and quality of scientific Citations per capita, publications per capita in top 10% of outputs journals, and indicators such as the h-index that measure the quality of citations Outputs relative to expenditure GERD divided by triadic patents, and GERD divide by papers published in top journals Research excellence and Internationalization of researchers Collaborations with foreign researchers/research organizations productivity and development of new research Collaboration across PROs networks Availability of research skills Supported PhD students going to become researchers Improved use of research Occupation rate of new laboratories and research facilities infrastructure Revenues from services provided to the market as share of total Research commercialization, science-industry collaboration revenues IP licensed and spinoff companies from PROs (number and value) Imports of machinery and equipment; import of intermediate goods Technology adoption by manufacturing, agriculture, and service Quality certification (ISO 9,000; environmental standards) sectors Computer use by firms; Internet use; intensity in the use of tractor, fertilizers (per hectare) Number of firms introduction new products or processes; share of firm revenues coming from innovation Business innovation, business R&D, and startups IP rights registered (trademarks, patents) Survival and capitalization of knowledge-based startups (number and value of knowledge-based startups five years old or more) 68 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 7.1b: Example of Intermediate Outcomes Metrics Intermediate Outcome Outcome type Indicator Internationalization of researchers and Collaborations with foreign researchers/research development of new research networks organizations Collaboration across public research organizations Research excellence (Volume and quality of scientific Availability of research skills Supported PhD students going to become outputs) researchers Improved use of research infrastructure Occupation rate of new laboratories/research facilities tors are then organized according to the four proposed reviews. They focus on the effectiveness conditions for outcomes. This allows categorizing all STI outcome three intermediate outcomes: (i) research excellence, (ii) types into the four proposed outcomes, regardless of science industry collaboration and technology transfer, the program that is generating them. and (iii) business innovation. The sections are structured around questions to facilitate use and should be seen Some of the proposed indicators for intermediate as a non-exhaustive checklist. Also, the relevance of outcomes are available from standard data sources for the sections and questions within each section will vary STI data as reviewed in appendix B. An example would from country to country. be government-financed gross domestic expenditure on R&D (GERD). Indicators may also be generated 7.3. RESEARCH EXCELLENCE with simple manipulation of available indicators (for example, GERD per triadic patents). Information from Research excellence and productivity depend on the surveys—like those proposed for business innovation— adequate supply (quantity and quality) of human will be more difficult to find. Program outputs would resources, infrastructure, and funding, as well as the be collected directly from program managers. In some incentives under which the researchers and managers cases, R&D statistics or budgetary information may be in PROs operate. Researchers and managers of PROs used as a first approximation at the aggregate level. respond rationally to a system of rules and regulations that embed different payoffs to choices they face. Public Analysis of intermediate outcomes metrics provides an investment in STI will be less effective if that system is indication of where the system is working effectively not conducive to research quality and productivity. and where it is not. This provides a starting point for a more in-depth review of the conditions that are lead- Four sets of questions for assessing research excellence ing to the system working effectively or not. The next are presented below. three sections explain how to conduct these in-depth Effectiveness Assessment 69 i. The presence of and the ability to attract highly motivated and well-trained human resources Relevance to An increase in public investments in R&D may not lead to an increase in research output the outcome and quality without adequate access to quality human resources. This requires the presence of a higher education system that is capable of producing academic talent, and incentives that can attract and retain this talent. Assessment • Does the higher education system provide a steady supply of quality academic talent, checklist compatible with the country’s needs? • Are the PROs able to attract and maintain top academic talent? Is the share of local scientists abroad significant? Why? • Are the opportunities for young researchers fair, transparent, and effective? • Do employment regulations provide for the substantive reward of high performance and effective punishment of recurrent underperformers? • What is the degree of integration of local research with the international scientific com- munity? ii. Researchers have access to appropriate research infrastructure for research excellence Relevance to Quality human resources need access to modern infrastructure to conduct excellent the outcome research. This requires investment in research infrastructure that is in line with modern standards and research priorities of the country. In addition, collaboration with interna- tional labs and research facilities allow developing countries to leverage resources outside without costly investments. Assessment • Are research facilities and infrastructure (in major fields of the country) up to interna- checklist tional standards and compatible with the country’s needs? • Is there a roadmap/strategy for investments in infrastructure research? Are actual invest- ments in research infrastructure in line with the outlined research strategy? • Can researchers access research infrastructure anywhere in the country? How is re- search infrastructure regulated? • Can researchers access the relevant international research facilities? iii. Research funds are administered appropriately Relevance to Access to funding in a predictable and stable way is another important factor contributing the outcome to research excellence. The way those resources are distributed (competitively or not) is also crucial. Finally, researchers funded by third parties (specially the business sector) are more likely to engage in results-driven work. Assessment • Is the flow of public funding stable and predictable? Is there any source of earmarking for checklist public spending in R&D? If so of what type? • Are allocations of funds to different PROs made in a competitive manner? What is the composition between block funding and competitive funding for public research institutes and universities? • Are researchers properly incentivized to look for third-party funds? Do researchers control funds mobilized from third parties? 70 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note iv. The governance of PROs is effective and geared toward research excellence Relevance to The rules and procedures governing how PROs operate also generate incentives that may the outcome or may not be consistent with reaching research excellence and research productivity. Rules governing the relationship between the government and PROs include those related to the selection and performance of senior managers. Therefore, of interest is the delegation of powers from the government to the PRO, the incentives under which managers operate, and the accountability to the public. Assessment • Do PROs have clearly defined missions and research priorities? checklist • Are public resources made available through a performance-based method? • How much managerial power has been delegated to managers compared to other decision-making bodies of the PRO (for example, scientific committees or governing bodies)? To what extent are the incentives of the different board members aligned with those of the PRO? • Do regulations of PROs effectively empower managers to achieve results? How adjust- able are PROs’ budgets (between activities and years)? How much autonomy do PROs have to manage human resources? • Are top management positions for PROs filled by a competitive meritocratic process? • Are there monitoring systems to measure management targets/goals? How does the government (owner) exercise control? 7.4. SCIENCE-INDUSTRY technological and economically viable innovations. COLLABORATION AND Technology commercialization is a multistage process TECHNOLOGY TRANSFER1 involving different stakeholders—researchers, faculties, coordinating/managing organizations, private/public Universities and research institutes are large beneficia- technology transfer intermediaries, and the enterprise ries of public investments in R&D. The pace and effec- sector. These stakeholders’ objectives often differ from tiveness of the transformation of research outputs—or, research commercialization (see figure 7.2). more broadly, academic knowledge—into new or better products and processes has a substantial impact on the In the remainder of this section, some issues are iden- contribution of those public investments to economic tified that need to be addressed when assessing the development. By improving the process of knowledge conditions for effective technology transfer, including transfer from PROs, countries can increase innovation in an adequate incentive regime for researchers and PROs the economy, and raise productivity, and create better and the efficient provision of intermediation services. job opportunities. Factors facing the incentive regime here include the regulation of intellectual property (IP) rights and em- Research commercialization does not evolve naturally ployment regulation, among other rules. Within a large and linearly from research and the discovery of scientific set of intermediation services, the session concentrates solutions. Rather, the process normally faces unfavor- on technology transfer offices, science and technology able economic incentives and an inadequate supply parks, the development of a pipeline of potentially of complementary services to translate new ideas into investable projects, and financial support for science- 1. Based on Correa and Zuniga (2013). industry collaboration. Effectiveness Assessment 71 Figure 7.2: Research Commercialization The Process of Research Commercialization: Schematic View Identification of technologies with potential commercial interest Attracting private partners Assessment Protection Prototype/ Discovery/ Technical Strategy (IPR Commercial- Research Proof of Marketing/ Disclosure value/market or not and ization Concept Promotion potential how) Development Gaps Technology • Licensing to transfer assistance established firms TT is a multistage process involving different • Joint ventures actors; researchers, industry, institutional • Spinoffs (licensing coordinators (TTOs), public sector agencies and and/or ownership) Financial and non- market and financial intermediaries financial support i. Intellectual Property Regulation Relevance to PROs are not necessarily interested in managing and actively seeking to commercialize the outcome research, as it is a very complex activity with high sunk and transaction costs as well as uncertain returns. Similarly, scientists are rarely interested in commercializing their research results, as investments in strictly academic or administrative tasks tend to yield higher net returns.2 Therefore, without clearly defining rights and obligations of those key stakehold- ers, it is unlikely that an efficient commercialization process will emerge. How is the IP of publicly funded research performed by PROs regulated in the country? Assessment • Who owns the IP—government (funding agency), university (or faculty/department), or checklist researcher? If not the researcher, is there a minimum share of royalties assigned to research- ers? Are regulations for ownership and royalties unambiguous? • Is there an obligation for the PRO to manage its research base and actively pursue the development and commercialization of IP? Which organization is supposed to perform this task (faculty, department, university, PRO)? Are there penalties for nonenforcement? • Do researchers have the obligation to disclosure their research activity and results to the PRO? If so, to whom and under which confidentiality rule? Are researchers obliged to engage in commercialization efforts? If so for how long? What are the penalties for noncompliance? 2. There are several reasons for this. One is that most scientific results are far from a stage in which they can be commercialized. Additional research, sometimes for a few years, is needed until a decision can be made about the commercial potential of a discovery. Another difficulty refers to the matching process, that is, finding an investor interested in nurturing a spinoff or a firm interested in buying the license. Therefore, except in the cases of breakthroughs with clear commercial potential, technology transfer from PROs will not follow naturally from research. 72 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 7.1: IP Regulation in the United States and Denmark As discussed above, altering the incentives of key stakeholders requires changing the expected payoffs of their alternatives. One way to do that is by creating a cost of noncompliance with the established rule. In this regard, developed countries have sometimes designated a series of legal responsibilities for PROs and researchers that benefit from public funds for research. In the United States, the U.S. Bayh-Dole Act transferred to research universities the responsibility for managing IP rights related to publicly funded research (originally belonging to the funding agencies). In Denmark, the 1999 Law on Inventions on Public Research Institutions established that researchers must disclose their inventions and assist in the commercialization process when needed. These regulations have the advantage of clarifying roles among stakeholders and are easy to monitor. Source: Correa and Zuniga 2013. ii. Employment Regulation Relevance to Human resources policies have an impact on the performance or researchers not only in the outcome terms of research excellence and productivity but also its commercialization. Criteria for career development of scientists may or may not reward commercialization efforts and col- laboration with industry. Do employment regulations limit the participation of researchers in entrepreneurial activities or research activities with the private sector? More specifically: Assessment • Is there a provision for sabbatical years for the researcher employment in the spinoff checklist company through which her or his research will be commercialized? • Is there any equivalence between academic achievement (such as publications) and technology transfer achievements (such as patenting, licensing, and volume of contract research with industry) for career promotion or in terms of financial compensation? • Do regulations enable/encourage internships from and to companies? iii. Other Regulations Assessment Do PROs have the legal mandated and operational flexibility to efficiently manage IP rights checklist (for example, managing a portfolio of spinoff companies)? • Is the use of PRO resources (such as infrastructure facility, research material, and re- searchers’ time) in collaborations with the private sector properly regulated? Does this regulation impose excessive financial and nonfinancial costs (red tape) for the private sector? iv. Technology Transfer Offices Relevance to Technology transfer offices (TTOs) are a particular type of organizational arrangement that the outcome permits specialization and economies of scale. They emerged in the past three decades as one of the models used by developed countries to promote manage research and imple- ment post-research commercialization efforts.3 3. Experience shows that the most successful institutions are the ones that devote sufficient resources that such coordinating entities can fully deploy their missions. See Debackere and Veugelers (2005) and Siegel et al. (2007). Effectiveness Assessment 73 Assessment • Are there TTOs? Do they cover most of the country’s research base? checklist • Are there arrangements for long-term sustainability of the TTOs? Are they endowed with sufficient financial resources? Does the salary structure reward the performance of TTO staff? Are the right skills available in the market? Is the TTO staff appropriately prepared and connected to the PRO researchers and industry? Are PROs aware of the long-term and public nature of TTO activities (that is, that the TTO is not a short-term profit-making organization)? Are PROs committed to support their TTOs in the long term? v. Science and Technology Parks Relevance to The primary role of science and technology (S&T) parks is to enable collaboration between firms the outcome and research institutions, facilitating the emergence of spin-off and start-up companies. The im- plicit assumption is that knowledge spillovers are location-specific (Link, Scott and Siegel, 2003). Are there S&T in the country? If so, is there evidence that firms installed in the park collaborate more or better with the respective PRO? Assessment • Do the S&T parks cover most of the country’s research base? Are S&T parks physically checklist close to research institutions? What was the motivation for the creation of the park (discuss the feasibility study/demand assessment)? Is it privately or public managed? Are the parks involved in the commercialization effort? Do they provide incubation services? vi. Science-Industry Collaboration Relevance to Science-industry collaboration in R&D is a major channel of technology transfer. There is the outcome ample evidence of the positive impact of joint research on business innovation in devel- oped countries.3 The positive impact of science-industry collaboration on firm innovation for developing countries is increasingly pointing to this direction (Crespi and Zuniga 2012). There is limited understanding about market or institutional failures causing poor col- laboration between science and industry. One hypothesis points to reputational issues, asymmetric information, and transaction costs (Audretsch, Bönte, and Krabel 2010). Policy instruments to foster science-industry joint research include research grants, matching grants, and tax-incentives. Assessment • Are there voucher schemes to promote science-industry collaboration? Is there any checklist evidence of sustained behavioral change from SMEs? • Does the program target the private sector or PROs? Is there a focus on SMEs? What are the goals of the program, eligibility criteria, and selection process? Are there any other non-voucher programs? • Are there centers of competence and centers of excellence? 4. For the United States, a study showed that firms taking part in Cooperative Research and Development Agreements (CRADAs) with federal laboratories were significantly superior in terms of technological performance when compared with other firms. According to Hall (2002), these consortia agreements, backed by real budgets and cost-sharing among parties, allowed internalization of spillovers and maximization of innova- tion possibilities and patenting. 74 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note 7.5. BUSINESS R&D, are dominant, entrepreneurs engage in a process of STARTUP CREATION, AND “creative destruction” and free markets become the TECHNOLOGY ADOPTION “innovation machine” (Baumol 1990, 2002). Business Innovation: R&D, Startup Creation Frictions unlikely to support innovation and technology adoption include labor market rigidities, poor labor Why do firms choose different techniques even when skills, limited access to credit, inadequate access to the “best practice” is available? Why do some firms internationally available knowledge, weak rule of law, introduce better products or reduce their costs while and excessive red tape (Parente and Prescott 1994). others don’t? Unfortunately there is no simple explana- Consequently, countries reach higher levels of efficiency tion for these questions. As a general starting point, one at different rates not because they have access to differ- may consider that profit-seeking entrepreneurs choose ent stocks of knowledge, but rather because they differ between productive (innovation, technology adoption), in the amount of constraints placed on the technology un-productive (rent seeking), or destructive strategies choices of their citizenry (Parente and Prescott 2005). based on the different net pay-offs embedded in the governance/institutional regime of a given society. Five sets of questions for assessing business innovation When economic incentives for productive strategies are presented below. i. A business environment that encourages innovation Relevance to In assessing the impact of public expenditures in STI, it is important to take into account the outcome the business environment in which firms operate. The broad governance regime (including rule of law and contract enforcement), market entry and exit, labor regulation, red tape, and so forth play an important role in ensuring that research can be smoothly converted into innovative products. Assessment • Is it easy for new businesses to enter the economy? checklist • Can start-ups and businesses readily access financing for innovative products? • Do businesses have access to human resources with research capabilities? • Is the legal system effective in dealing with contract enforcement issues? • Does the government provide support for business R&D? • Are failing businesses allowed to exit easily? ii. Investment Readiness Relevance to Timely access to mentors and networks can be critical in helping entrepreneurs who are seeking the outcome to market new products or penetrate new markets. These resources help entrepreneurs gain ac- cess to advice on strategic planning and marketing, financial resources, technological resources, and so forth. Connection to those networks and mentors is important to avoid the creation of graveyards of ideas, proof of concepts and prototypes. Effectiveness Assessment 75 Assessment • Are there funding schemes for the development of proof of concepts and prototypes? checklist How much is the market test taken into account from the early stages of project devel- opment? • Are there mentoring services for the preparation of “investable” projects? Are there con- nections to potential users of the technology or possible investors? • Are seed financing schemes available (see the discussion of the Valley of Death in box 7.2)? Financing is often unavailable for the additional research that is needed to move beyond proof of concept, and prototypes. These activities are neither eligible for stan- dard research grants nor attractive options for venture capitalists and so entrepreneurs, and researchers engaged in entrepreneurial activities struggle to finance ideas that would succeed were they to find the funds. iii. The private sector has access to resources required to innovate Relevance to Innovation is not cheap. By its nature, it involves a higher degree of risk than business the outcome ventures that operate in established markets with tested products. Firms therefore require access to cheap capital, machinery, and associated assets to be encouraged to take the risks of innovation. Assessment • Do innovating firms have access to cheap debt? checklist • Do firms have access to equity financing? (see the discussion of the Valley of Death in box 7.2)? • Do firms have access to equipment and machinery as discuseed in the section of the “Affordability and availability of machines and equipment” in the Technology Adoption and Diffusion section? iv. Government support for business R&D increases innovation 5 Relevance to Fiscal support compensates firms for the externalities and risks involved in the R&D activ- the outcome ity. It is designed to reduce the cost of private investments and increase the volume of R&D invested by the firm. The critical issue is to understand whether the incentive was generated by additional investments—that is, investments that would not have been made without the incentive. The number of OECD countries that implement an R&D tax incentive scheme rose from 18 in 2004 to 26 in 2011. Assessment • Are government monetary support programs for business R&D appropriately targeted? checklist • Are support programs designed to suit the needs of the economy? • Are the levels of direct and indirect support to private sector adequate? • Are the administrative requirements to access and use government support compatible with international good practices? • See the fifth question below for a list of questions on Tax Incentives. 5. Based on Correa and Guceri (2013). 76 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 7.2: Knowledge-based Start-ups and the Valley of Death Knowledge-based start-up firms often need substantial, externally provided funds to successfully grow. Their need for funding is often greater than the funding that friends or family can offer. At the same time, these firms are often too young and under-developed to receive funding from more formal sources of funding such as banks, venture capital, or private equity funds. Studies have shown that in developing countries, friends and family usually contribute up to US$50,000, enough for an entrepreneur to get started but not enough to generate sufficient revenues to scale the business. To grow, entrepreneurs may need an injection of capital, but banks have little appetite to lend to high-risk businesses with insuf- ficient cash flows. Venture capital and private equity firms (which may not exist in some countries) usually do not invest less than US$1 million—too much for a seed-stage start-up. The resulting funding gap is called the “valley of death.” It can be difficult to traverse, which leads to failure of start-ups. Firms can get through the valley of death with the help of pre-seed funding to achieve positive cash flows and so build successful and prosperous businesses. Figure B7.2.1: Valley of Death Idea Stage Pre-seed and seed stage Early and Late Stage Cash flows $0 VALLEY OF DEATH Source: World Bank (2014). v. Tax Incentives for Research and Development6 Relevance to Governments are increasingly relying on tax breaks to support to R&D efforts by firms. A the outcome number of studies have found that R&D tax credits increase R&D expenditure, and lead to an increase in innovation.7 The economic rationale for tax breaks is that they increase R&D by lowering the cost of carrying out R&D in the private sector, and thus overcome the market failure that arises because the social returns from R&D are lower than the private returns. Tax breaks can take a number of forms. These include tax credits (and enhanced deduction schemes) that allow firms to deduct R&D expenditures (or more) from their tax liabilities, depreciation allowances, and unconditional cash refunds which provide benefits to compa- nies that are loss making and so don’t pay taxes. 6. Based on Correa and Guceri (2013). 7. As discussed in Correa and Guceri (2013) some studies have found that some tax break schemes have not increased R&D expenditure. Effectiveness Assessment 77 Assessment • Is there a tax break scheme for R&D in place? checklist • What type of tax breaks are available? In other words do they come in the form of tax credits, depreciation allowances, unconditional cash refunds, or other? • What types of firms are eligible for the R&D tax breaks? • What types of expenditures are eligible for tax breaks? • What is the red tape associated with access to tax incentives? • Is there large-scale take up of the tax break scheme? • Is the value of the tax incentives large enough to affect behavior? Technology Adoption and Diffusion As a result, the capacity of developing countries to in- novate depends, on the one hand, on foreign sources of Innovation in developing countries is based mostly on knowledge and technology and, on the other, the coun- adoption, recombination, and adaptation of existing try’s capacity to absorb, adapt, and diffuse innovation. technologies rather than on the development of new technology. Innovation is therefore more “new to the Five sets of questions for assessing technology adoption market” or “new to the firm” than “new to the world.” and diffusion are presented below. i. Existence of regulation framework Relevance to Building an enabling environment that is both attractive to foreign investment and lo- the outcome cally supportive of innovation, adaptation of technology, and dissemination of knowledge requires an adequate institutional framework. Government policies to support innovation should embark on reforms that update the regulatory and institutional framework for in- novation and remove bureaucratic, legislative, and regulatory obstacles to innovation—and particularly technology adoption and diffusion. These obstacles often affect competition laws, licenses to operate, government authorizations, technical norms and standards, and customs procedures. For instance, monopoly rights may represent a barrier to the adop- tion of technologies in the sense that industry insiders with monopoly rights to the current technology will resist the adoption of better production techniques. This suggests that more competitive economies are likely to be characterized by higher absorptive capacity. Assessment • Is trade regulation and legislation acting as a driver for the influx of new knowledge checklist and technology to the country (including embodied technology, contact with foreign suppliers, and FDI)? • Are anti-competitive regulations and firm conduct acting as a barrier to the introduction and diffusion of technologies in certain sectors? • Is the existing framework of technical regulations and standards acting as a driver for the influx and diffusion of technologies? • Do national technical regulations and standards build on international standards? • Does the domestic regulatory framework encourage attract foreign companies and FDI? 78 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note ii. Affordability and availability of machines and equipment Relevance to The acquisition of machinery and equipment represents a major source of knowledge for the outcome innovation in firms. The technological know-how embodied in machinery enables firms to employ more efficient production processes and thus raise the quality of their own prod- ucts and processes. Embodied technology diffusion is about the introduction into produc- tion processes of machinery, equipment, and components that incorporate new technology developed in other firms either domestically or abroad (Papaconstantinou et al. 1996). The extent to which firms can acquire machinery is defined by the availability and affordability of machinery and equipment. Assessment • Are firms able to acquire new technologies at competitive prices in local markets? checklist • Are there high tariffs or other restrictions on the importation of machinery, equipment, and intermediate goods? • Are there restrictions or high tariffs on importation of used equipment and machinery? • Is it difficult or costly to import machinery and equipment? • Can firms rent machines? • What is the depreciation policy embedded in the country’s tax policy? • Is there an adequate supply of financing instruments (at reasonable cost) for firms to purchase new machinery and equipment? • Are there shared facilities enabling firms to access high-cost and modern technologies? • Are there sufficient skills in the local labor market to operate cutting-edge machines and technologies? iii. Technology extension services8 Relevance to In many countries technology extension programs fall into the category of “innovation” the outcome policies and programs. However, the role of extension work within the spectrum of such programs is unique. Technology extension aims to improve the productivity and competitive- ness of existing businesses through the adoption of the most appropriate technologies for their fields of activity. It aims to promote learning that is articulate, thoughtful, and repeat- able by the companies so they are able to develop new skills for the future. Several countries have programs to support manufacturing SMEs, of which comparative studies have been conducted. Most developing countries face special challenges in terms of the skill level of their workforce. The introduction of new technologies and processes requires more carefully designed assistance processes than those used in developed coun- tries. The design and implementation of technology extension programs for these countries should keep this situation in mind. Supplying technology extension services is not synonymous with offering financing. Support or advice on access to funding sources from other agencies may be a component of im- provement projects, but they are not the main focus. Typical mechanisms by which tech- nology extension services address SMEs information gap are assessed using the questions below (see box 7.3 for a summary of international good practices): 8. Based on Rogers (2013). Effectiveness Assessment 79 Box 7.3: International Good Practices for the Provision of Technology Extension Services Demand driven/mission oriented. Technology extension programs should be geared to the needs of the industrial customers they serve and well informed about the nature of the demand for improvements as they feed a proactive vision for companies to solve their problems and make such improvements. Practice-oriented technology applications. The applications of technology promoted by extension service programs should be primarily practical and of proven value among industry leaders. Providers must avoid recommending highly abstract projects or innovative but untested concepts that are more appropriate in research laboratories than in SMEs with challenging business problems. Decentralization. Service centers must be distributed in regions where the demand for their assistance is documented and understood. Being near their customers helps providers understand regional variation in the needs of SMEs and makes their programs more visible and easily accessible to companies that are potential customers. Target SMEs. When the pressure to become self-financing becomes too high, service providers tend to migrate to more capable, generally larger companies that have more resources and do not need subsidized consulting. The opposite mis- take is focusing on micro-enterprises that do not have the capability of absorbing and leveraging the services received. Critical role of human resources. Staffing the program with competent personnel who are familiar with SMEs and the delivery of industrial extension services is absolutely critical. The desirable profile of these providers, then, has three dimensions: they must have knowledge of technology, knowledge of the business environment of companies, and the ability to communicate in interpersonal relationships that grow out of improvement projects. Source: Rogers 2013. Assessment • Is there provision of information on opportunities for improvement in existing technolo- checklist gies, best practices, international trends, relevant regulations, business networks, oppor- tunities to become government suppliers, and others? • Is technical assistance and consulting in the context of improvement projects designed individually for interested companies? • Is training of plant and administrative staff for the effective use of technologies more advanced than those previously used by the company? 80 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note iv. Standards and technical regulation9 Relevance to Standards and technical regulations can play an important role facilitating innovation. They the outcome consist of rules and guidelines for products, processes or production methods. Compliance with standards is voluntary and they can be set by the state, private sector, or civil soci- ety. In contrast, technical regulations are mandatory and are typically set by the state. By codifying the technical characteristics of products and processes, standards and technical regulations embody technological knowledge and best practices. Because information in them is nonproprietary, they create a pool of technical knowledge that can be transferred across companies and countries, freely accessed by entrepreneurs, scientists, and engineers, and used to generate new ideas and technologies. Moreover, standardization stimulates innovation by helping to build focus, cohesion and critical mass in the emerging stages of technologies and markets. They also play an important role facilitating trade by reducing risks and transaction costs. Governments play a leading role in designing and implementing certain standards and technical regulations. This includes ensuring that national standards and regulations are consistent with international ones, especially those in important export markets. Govern- ments can also play an important role facilitating the adoption of standards by firms which may find standards to complex and demanding to implement by themselves. The state can promote awareness about standards and design appropriate capacity-building programs to ensure that standards do not exclude local companies from domestic and export markets. A number of services that are needed by the private sector to comply with technical regula- tions, standards, metrology, and quality requirements are described in table 7.2. Assessment • Are local firms finding that their absence of compliance with local or international stan- checklist dards and regulations is restraining their ability to supply local or international markets? • Are metrology facilities and testing laboratories able to supply the services needed by local firms? • Do firms report a need for capacity building to satisfy the requirements of local or inter- national markets? • Are local regulations and standards consistent with those in major export markets? 7.6. CONCLUSION This chapter discussed how to evaluate the effectiveness business R&D, startup creation, and technology adop- of STI programs. It first described how to link inputs tion. Box 7.4 outlines a possible structure for the EFA as with outcomes. There followed sections that evalu- well as a number of useful readings that go into more ated conditions for effectiveness in research excellence, detail on the approaches described in this chapter, as science-industry collaboration and technology transfer, well as illustrating their application. The next chapter discusses the final report. 9. Based on Swann (2010), Kaplinsky (2010), and Guimón (2014). Effectiveness Assessment 81 Table 7.2: Technical Regulations, Standards, Metrology, and Quality Compliance area Business needs Services needed Product standards/technical Access to standards/technical regulations Reference center in standards body or other regulations, including packaging and labeling Product testing Conformity assessment recognized by the Testing laboratory upgrading toward (international) client internationally recognized accreditation, mutual recognition agreements (MRAs) Accuracy of measurement Internationally recognized equipment Metrology laboratory upgrading toward calibration, measurement traceability to the internationally recognized accreditation, inter- International System of Units (SI) standard calibration schemes Consistent product characteristics and Enterprise Quality Management System Certification capacity and internationally quality Certification (ISO 9000) recognized certifiers Management of environmental Enterprise Environmental Management Certification capacity and internationally impact System Certification (ISO 14000) recognized certifiers Box 7.4: The Effectiveness Assessment—Structure and Useful Readings Possible structure of the EFA as a standalone document: 1. Introduction: Objectives and Scope, as agreed in the Inception Report 2. Research Excellence 3. Science-Industry Collaboration and Technology Transfer 4. Business R&D, Startup Creation, and Technology Adoption 5. Policy Recommendations and Conclusions Useful reading: European Parliamentary Research Service (EPRS). 2014. “Measuring Scientific Performance for Improved Policy Making.” Science and Technology Options Assessment, EPRS, European Parliament, PE 527.383. European Commission. 2012a. “Evaluation of Innovation Activities: Guidance on Methods and Practices.” European Com- mission Directorate-General for Regional Policy. Fahrenkrog, Gustavo, Wolfgang Polt, Jaime Rojo, Alexander Tubke, and Klaus Zinocker. 2002. “RTD Evaluation Toolbox— Assessing the Socio-Economic Impact of RTD-Policies.” Strata Project HPV 1 CT 1999–00005. Indecon. 2008. “Value for Money Review of Science Foundation Ireland.” Department of Enterprise Trade and Employment by Indecon International Economic Consultants. 82 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note CHAPTER 8 FINAL REPORT The Final Report is the last stage of the Public Expenditure shape that policy mix? In addition, it is envisaged that the Review (PER) (see figure 8.1). The Final Report addresses Final Report will integrate the findings of the Efficiency the final analytical questions: (i) how is the policy mix af- Assessment (EA), the Effectiveness Assessment (EFA), and fecting the impact of public spending in research, develop- the Functional Review, as well governance analysis that ment, and innovation (RDI); and (ii) how is the national in- may have been conducted for the Functional Review (as novation system (NIS) governance structure contributing to described in chapter 3 and chapter 5). Figure 8.1: The Final Report Operational Functional Effectiveness Inception Report Efficiency Final Report Review Assessment Assessment Summary The ultimate objective of the Final Report is to provide a fact-based set of recommendations that describe how policy makers’ strategic goals can be achieved through policy reforms and strategic investments. These recommendations can take the form of a plan that links the achievement of the strategic goals to the required inputs, outputs, and outcomes. The Final Report recommendations arise from an analysis that complements and extends the work done in previous stages of the PER. The analysis begins with a review of the NIS policy mix, which considers whether policies maximize returns to public investment by considering their relevance, coherence, and consistency. There follows a review of the composition and level of spending, including issues such as the mix of direct versus indirect support to business R&D, operating versus capital investments, and basic versus experimental and applied research. Then, a governance analysis evaluates how poli- cies are made and implemented, and how to improve this process. The Final Report is largely based on data collected for the previous stages of the PER. This data may be complemented with additional material. In particular, interviews may be used to collect additional information for the governance section from policy makers and implementation agencies. Final Report 83 The chapter first presents in section 8.1 an analytical In order to address such questions, this section sug- approach that can be used to assess the relevance, gests the approach in table 8.1. It merges tables 6.1a coherence, and consistency of public expenditures on and 7.1a and integrates inputs, outputs, outcomes, RDI. Section 8.2 follows with a description of indicators and impact information to enable a comprehensive, that can be used to analyze the composition of public results-oriented review of public spending, following expenditures based on existing indicators. Section 8.3 the input-output-outcome-impact (IOOI) framework contains a brief discussion about the level of R&D. (chapter 3). Table 8.1 is the core instrument for the Section 8.4 discusses assessment questions for the proposed PER exercise. It reflects the intention to go governance structure. Section 8.5 concludes. beyond the pure review of public spending to involve a first assessment of their impact, linking disbursements/ budgeting and the outputs and outcomes. 8.1. POLICY RELEVANCE, COHERENCE, AND CONSISTENCY Table 8.1 can be complemented by table 8.2. Table 8.2 goes beyond R&D expenditures and links public spend- As discussed before (see chapter 3) there is no reason to ing to the results-oriented framework, thereby consoli- assume, a priori, that policies will target economic and dating the STI budget. It follows a previous structure social goals which are relevant for the country context. (described in table 5.1) and is filled with specific data to Nor is there reason to assume that that policy design enable a subsequent practical exercise. Together with and implementation will result in a coherent body of more disaggregated versions (see chapter 5), table 8.2 measures consistent with an intended public goal (com- provides for a comprehensive, results-oriented approach mon good). This analysis complements, therefore, the of public spending in STI. It reflects the intention to go assessment of the operational efficiency of expenditures beyond R&D data and cover other government expen- (chapter 6) and their effectiveness (chapter 7) with the ditures related to the innovation process. analysis of the “policy mix.” The underlying hypothesis is that the more balanced a policy mix is, the more it Let’s see a practical exercise starting with the data from will serve to maximize returns to public investment. It table 8.2, which is expected to be available at this stage follows that the interdependence of STI policies is a of the analysis. The analysis is not meant to be exhaus- major determinant of their impact. tive but rather illustrate the potential of the proposed framework. It is proposed that this issue is approached through three supplementary questions: Policy Relevance The ratio (H/J) in table 8.2 provides an approximation • Policy relevance: Do the composition and level of of the share of public expenditures with a potential STI public expenditure reflect the development needs impact on productivity growth—a possible indicator of and priorities of the country? the relevance of public spending on STI. In that case, • Policy coherence: Are the activities and programs more than one third of public spending on STI has a being implemented and financed complimentary? goal different from improving total factor productivity Are the activities and programs being implemented (TFP) or labor productivity. The reallocation of those and financed redundant? resources toward the productivity objective could, in • Policy consistency: Is the composition and level of principle, represent a potential source of improvement public expenditures consistent across existing outputs, in the quality of public spending in STI and thus a source outcomes, and higher-level goals? of economic efficiency gains. 84 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table 8.1: Policy Input, Outputs, and Intermediate Outcomes Policy/program Intermediate outcomes Possible indicator Output Budget/disbursement Total factor productivity (TFP), labor productivity in manufacturing and service sectors; yield per hectare; export of Development goal new products Citations per capita; publications Research projects funded National Science Foundation per capita in top journals Research Grants Research projects completed (number and value) Research excellence Note: Government-financed GERD and research would provide an aggregate figure productivity Gross domestic expenditure on Academy of Science’s Program for R&D (GERD)/triadic patents, GERD/ the Advancement of Research papers published in high impact factor journals Revenues from services provided Research projects with the private Research Institute for Marine to the market as share of total sector (number and value) Biology revenues Note: Distribution between basic and Institute for Agricultural Research Research experimental research funded by the commercialization, government is a first approximation science-industry (from budget) collaboration IPs licensed and spinoff companies Research Institute for Marine from PROs (number and value) Biology Institute for Agricultural Research Number of firms introduction Value of disbursements and number Innovation Vouchers Program (from new products or processes; share of firms covered Ministry of Economy) of firm revenues coming from innovation ; Business innovation, IP rights registered (trademarks, Value of business R&D or share of Tax breaks for business R&D business R&D, and patents) SMEs investing in R&D startups Survival and capitalization of Number and value of knowledge- Matching grants program for knowledge-based startups based startups created that received early-stage funding from Ministry (number and value of knowledge- funding from the program of Science based startups five years old or more) Quality certification for computer Number of firms assisted; number of Technology extension services and Technology adoption use by firms using the Internet firms certified. matching grants from Mininistry of by manufacturing, Industry agriculture, and service sectors Intensity in the use tractor, Number of individual trained Agricultural Extension Services from fertilizers (per hectare) Ministry of Agriculture Final Report 85 Table 8.2: Country Alfa Consolidated STI Sector Budget—Illustration (US$ ‘000s) Budget item Value (US$ ‘000s current) Expenditures on R&D (A) 100,000 Expenditures for Research Commercialization and Collaboration (B) 5,000 R&D and Technology Transfer Budget (C)= (A+B) 105,000 Tax Breaks for Business R&D (D) 15,000 Expenditures Supporting Business R&D and Startups (E) 3,000 Expenditures Supporting Technology Adoption by Firms (F) 50,000 Innovation Budget (G)= (D+E+F) 68,000 R&D and Innovation Sector Budget (H)=(C+G) 173,000 Other Expenditures (I) 87,000 Consolidated R&D Budget (J)= H+I 260,000 This is likely to be the case, for instance, if (i) the analysis lion) compared to efforts to commercialize that research from the Country Paper (chapter 4) indicates that low or foster science-industry collaboration (US$5 million). levels of agricultural productivity hinder poverty reduc- This is more likely to be the case if the indicators in table tion given a large share of the population employed in 8.1 suggest good performance in terms of research subsistence agriculture; and (ii) the information gener- excellence and poor performance on science-industry ated using the sources referenced in table 8.1 indicates collaboration and research commercialization. that very little public spending in STI is allocated to agricultural extension services. The situation would be less clear were the results from the analysis described in table 8.1 to show poor indica- Yet, the reallocation across categories should not be tors for both intermediate outcomes—research excel- taken for granted. It is important to keep in mind that lence and science-industry collaboration and research other objectives than economic efficiency may be driv- commercialization. If research outputs do not reach ing the allocation of resources —for example, improving reasonable levels of academic excellence, the possibil- public health. Note that it is often difficult to too com- ity of collaboration or commercialization is significantly pare and trade off efficiency with non-efficiency goals. reduced—and public investments in this area may not be warranted. Policy Coherence Table 8.1 also provides for the assessment of policy It is also possible to analyze the coherence of public coherence. This includes the existence of overlaps in spending by comparing the allocation across the four programs and activities, whether programs and activi- intermediate outcomes. This proportionality check ties contradict or complement each other, and whether could demonstrate, for example, that the country Alfa program and activities may be missing. The identifica- is overinvesting in R&D (A) activities while neglecting tion of missing activities and programs is similar to the to support technology transfer and science-industry assessment of the effectiveness conditions (chapter 7). collaboration (B). The comparison between the innovation (G) and the In the hypothetical situation illustrated in table 8.2, too research budgets (C) is an indicator of policy consis- much seems to have been spent on R&D (US$100 mil- tency—assuming that innovation is one of the develop- 86 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note ment goals (as in the proposed structure in chapter 3). In For example: resources invested in a quality certification the example of table 8.2, about 25 percent of the total program cover only a small number of exporting firms spending in STI is allocated to innovation, of which less in the food industry sector. This in turn enables only a than 5 percent is invested in direct support to business negligible increase in the food industry’s exports which, expenditures in R&D. Table 8.1, which provides refer- in turn, is unlikely to lead to the diversification of exports ences for more detailed information on inputs (budget (supposedly the developmental goal). supporting business R&D), outputs (such as the value of firms’ investments in R&D), and outcomes (innova- There are potentially other ways to use the notions tion), complements the analysis with indicators (on the of policy relevance, consistency, and coherence. The research and innovation performances), which may or purpose of this Guidance Note is to identify some of may not corroborate the initial assessment. them without the ambition of addressing all possible angles. Table 8.3 summarizes the exercise. The team Another approach to policy consistency is related to implementing the PER exercise is encourage to consider the level of public spending. Economic literature offers other possible aspects according to the institutional some empirical evidence on the level of public spend- specificities of the country being studied. ing in R&D but not in terms of innovation investments. Moreover, calculating the social rates of return and com- A final word of caution refers to benchmarking, which paring it with alternative allocation of public spending is implicit to some extent in the previous examples. For is often a tricky exercise (see next section). The analysis instance, in the case of policy relevance STI spending of consistency of the level of public spending across was lower than expected given the country’s level of the different stages of the analysis—input-output and development. This conclusion rests on the analyst’s output-outcome—may be a useful approach. In more ability to determine an optimal composition of public informal terms, one can think of the issue as “consis- spending (even if roughly), and compare the country’s tency between means and ends.” spending to this. In some cases even a rough and ready Table 8.3: Policy Relevance, Consistency, and Coherence Assessment criteria Application Example Policy Compare the information available in table 8.1 with The Country Paper indicates that low levels of agricultural relevance the assessment of country needs prepared in the productivity hinder poverty reduction given a large share of the Inception Report. population employed in subsistence agriculture. Yet, very little of public spending in STI is allocated to agricultural extension Use the main budget categories from table 8.2 to services. Moreover, overall spending on STI is low compared to complement the analysis. the country’s development level. Policy Identify possible overlaps in the programs and activities Proportionality issue. In the hypothetical situation illustrated in coherence described in table 8.1. table 8.2, too much seems to have been spent on R&D (US$100 million) compared to efforts to commercialize that research or Consider to what extent programs and activities foster science-industry collaboration (US$5 million). This is more contradict or complement each other. likely to be the case if table 8.1 indicates that research excellence Consider programs or activities that may be missing in indicators are comparatively high but indicators of research table 8.1. commercialization are comparatively low. Consider to what extent public spending among key categories in table 8.2 is proportional. Policy Consider whether the level of public expenditures Resources invested in a quality certification program only cover a consistency is consistent across existing outputs, outcomes, and small number of exporting firms in the food industry sector. This higher-level goals small number of beneficiaries generates only a small increase in food industry’s exports which, in turn, is unlikely to lead to the diversification of exports in the country. Final Report 87 version of this exercise may be impossible and the team the R&D performing firm, while indirect support applies will have to revert to the use of judgment calls. This is some form of tax relief. Direct support is often directed especially so in the case of inputs (public spending) and toward particularly activities and sectors with perceived outputs (activities and programs) and less so in terms high social returns whereas indirect subsidies are more of outcomes (for which STI statistics is a starting point). neutral in terms of sector preference (though this is not always the case). Direct support is often associated A better possibility of benchmarking public spending with the objective of supporting R&D projects in SMEs in STI refers to the use of the OECD data from the or startups (OECD 2010a). Budgetary data may be Policy Mix Database and other indicators that could considered a possible source: the category ‘transfer’, as be calculated using the 2001 GFS Manual (IMF 2001). defined in the 2001 GSF Manual. Data on tax breaks, The limitation, as discussed before, is that the OECD as mentioned before, need to be obtained separately. Policy Mix Database is restricted to R&D spending (see Figure 8.3 depicts the distribution between indirect and chapter 5). The next section describes some of those direct incentives for business R&D as a share of GDP for indicators and, when possible, how to generate them the 2008–09 period. from the budgetary information. The next section also addresses issues related to the R&D level. Basic versus Experimental Research Basic research is often seen as relating to fundamental 8.2. COMPOSITION AND phenomena and linked to the idea of curiosity-driven LEVEL OF R&D SPENDING research. Basic research is often riskier and involves potentially large positive externalities. In theory, the Distribution of Public Spending to more fundamental research is, the less willing industry Public versus Private Sectors will be to fund it, because it is hard to appropriate and monopolize the results. Hence, governments tend to STI policy in most countries has placed an emphasis on pay for most of the cost of basic research. Figure 8.4 supporting business innovation. Yet policy makers often shows the distribution of R&D at aggregate level by ignore the distribution of public spending between type of research. The result, which shows less devel- PROs and the business sector. As discussed before this oped countries investing a lower share of their budget is a critical first step for the proposed PER exercise. The in experimental research, is in principle questionable— analysis of the distribution by beneficiary can be gen- even though the policy implication will not always be erated by combining the Function of Government and to recommend less investment in basic research. If not Economic Classification of the GFS Manual (IMF 2001). available, an approximation may be obtained by using Categories 2511 and 2521 of the Economic Classifica- the budget data and the Classification of Functions of tion correspond to subsidies provided to nonfinancial Government (COGOF) classification following the defi- public corporations and nonfinancial private enterprises nition adopted by the OECD in the Policy Mix Database.1 respectively. Data on R&D activity is available at the level of Government Functions. By combining both Operating Costs versus Capital Investments classifications it is possible to obtain the amount of subsidies resources allocated to the public and private While salaries of researchers are often a large share sector (figure 8.2). of R&D expenditures, the lack of a clear strategy to maintain and update research equipment has been an Direct versus Indirect Support important bottleneck for the development of research to Business R&D excellence in developing economies. The allocation for fixed capital may be obtained by combining this Another angle of possible interest is the distribution of classification with the COGOF. The codes related to expenditures between direct and indirect support. Direct support usually involves a transfer of public funds to 1. http://stats.oecd.org. 88 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Figure 8.2: Distribution of Public Support between Public and Private Sectors (2010 or latest year) 100 80 60 Public Percent Private 40 20 0 Chile Czech Poland Estonia Rep.of Slovenia South Turkey OECD United Denmark Republic Korea Africa median States Country Source: OECD Policy Mix MetaData at http://stats.oecd.org. Figure 8.3: Direct and Indirect Support to Business R&D (2008-09 OECD Countries) Source: OECD, Main Science and Technology Indicators Database, March 2010. Note: BERD = business enterprise R&D; R&D = research and development. Final Report 89 Figure 8.4: Share of Country’s R&D Expenditure Based on Research Type (2008) 100 8 8 6 90 29 80 41 41 25 31 70 63 49 60 82 Experimental Percent 31 50 20 Basic 22 40 Applied 69 30 11 60 20 39 43 40 35 10 22 15 0 4 Australia Japan Romania Ecuador Israel Hungary Sri Lanka Uganda Country Source: Data from UNESCO. investments in Fixed Capital according to the 2001 The Level of Public Expenditures on R&D GSF Manual are (3111) Buildings and structures, (3112) How much public investment in R&D is enough? To an- Machinery and equipment, (3113) Other fixed assets, swer this question, it is necessary to calculate the social (31131) Cultivated assets, and (31132) Intangible fixed rates of return of public investment in R&D. Calculat- assets (23) consumption of fixed assets, for Deprecia- ing social rates of return of public spending in general tion). The challenges in balancing operational and capi- and R&D in particular is a complex task (see box 8.1). tal expenditures are illustrated with the case of Croatia Yet social rates of return serve as parameters for the (figures 8.5a and 8.5b). discussions about the optimal allocation of public ex- Figure 8.5a: Operating Costs and Salaries Figure 8.5b: Capital Investments and Project Financing in Croatia, 2006–14 (€ million) in Croatia, 2006–14 (€ million) 500 500 60 60 Total Total investments investments 50 50 450 450 Project Project Expenditures (€ million) Expenditures (€ million) Expenditures (€ million) Expenditures (€ million) Financing Financing 40 40 400 400 30 30 350 350 Operating Operating costs costs and and salaries salaries 20 20 Capital Capital 300 300 10 10 Investments Investments 250 250 0 0 2006 2007 2006 2008 2007 2009 2008 2010 2009 20102011 2012 2011 2013 2012 2014 2013 2014 2006 2007 2006 2008 2007 2009 2008 2010 2009 20102011 2012 2011 2013 2012 2014 2013 2014 Year Year Year Year Source: Elaboration of data provided by Croatia’s Ministry of Science, Education, and Sports (MSES). 90 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 8.1: Macro-Level Analysis on Social Returns to R&D, Infrastructure, and Human Capital The rate of return on gross R&D investment is computed for Croatia using World Development Indicators macro-level data. Following the methodologies used in Coe and Helpman (1995), Jones and Williams (1998), and Lederman and Maloney (2003), the R&D capital stock is computed and then the elasticity of various output measures is found with respect to R&D capital and computed gross rate of return (see table B8.2.1 below). Rate of return on R&D investment for GDP measured in constant local currency unit (LCU) is 73 percent. Based on these findings, optimal amount of R&D investment can be estimated. In Croatia this amounts to 9.23 percent of GDP. The real interest rate in Croatia between 1997 and 2010 was around 7.3 percent. The current gross rate of return was 0.73 percent, which is close to 10 times less than real interest rate. As a result, optimal amount of R&D expenditure would be 0.92 * 10 = 9.2 percent, where 0.92 is the average R&D expenditure (as a percent of GDP). Using macro-level data, there are several studies that estimate the rates of return to R&D. Among these studies, Leder- man and Maloney (2003) use cross-country data for the 1975–2000 period and find that returns to R&D investment are around 78 percent. They group countries according to their income levels and find: 20–40 percent as OECD average, 60 percent for medium-income countries, and around 100 percent for poor countries. Returns to R&D in developing countries are higher than the values for industrialized countries. Coe and Helpman (1995) estimate rates of return to R&D for the period 1991–90 and find 123 percent for the G7 and 85 percent for the remaining 15 OECD countries. The result for rate of return obtained for Croatia is in accordance with the studies in the literature. Table B8.1.1: Elasticity, Rate of Return, and Optimal Amount of R&D Investment Elasticity Gross rate of return (%) GDP (current LCU) 1.26 139 GDP (constant LCU) 0.66 73 GDP (constant 2000 US$) 0.66 73 GDP per capita (constant LCU) 0.66 73 GDP current LCU/labor force 1.34 147 GDP constant LCU/labor force 0.75 83 Optimal Amount of R&D Investment Real interest rate (percent) Lending interest rate (percent) 9.23 5.64 Canning and Bennathan (2000) compute the rate of return to infrastructure (electricity generation and paved roads) . They find that the average return to electricity generation capacity is 40 percent where the values vary significantly across countries. To give some examples, over their survey period, in Turkey, the rate of return was 32 percent, Portugal 7 percent, Mexico 51 percent, and Brazil 10 percent. They find the rate of return to paved roads was 30 percent over this period. In their study, elasticities of GDP per worker with respect to electricity services were 9 percent for a country at the median income level and 6 percent for a country in the lower quartile. Elasticity of GDP per worker with respect to paved roads was 9 percent for a median-income country and 5 percent for a lower-income country. In a recent study, Drezgic (2008) esti- mates the elasticity of GDP from 1996 to 2006 across Croatian counties with respect to transport and electricity sectors. The elasticity varied around 4.7 percent and 4.6 percent for transport and electricity sectors, respectively. In another specification where Drezgic combines transport, electricity, and construction sectors, elasticity of GDP with respect to this combined sector was 6 percent. Based on these estimates, rate of return on infrastructure in Croatia can be estimated using the methodology used in Canning and Bennathan (2000). Consider as the Cobb-Douglas production function where K is physical capital, L is labor, and X is infrastructure capital. Solving the aggregate production function, the following first order conditions are found: (continued next page) Final Report 91 Box 8.1 (continued) where px is rate of return to infrastructure capital and r is the rate of return to physical capital. Drezgic (2008) computes net capital stocks for each industry in Croatia from 1996 to 2006. He shows that electricity sector comprised around 19 percent of total private capital stock (X = zK where z is a multiplier). The transportation sector comprised 21 percent of total private capital stock. Return to private capital stock which is the real interest rate, was around 7.3 percent. Using this information, the average rate of return on electricity and transportation sectors can be calibrated as follows: According to this formula and using the estimates obtained from Drezgic (2008), the rate of return to infrastructure is 34 percent for electricity and 32 percent for transportation, and 24 percent when electricity, transport, and construction sectors are combined. The literature on rates of return to human capital is much older. Psacharopoulos (1994) estimate the social returns to investment in education level for a large number of countries. An updated study (Psacharopoulos and Patrinos 2004) that presents results for the same countries finds that average rate of return to an additional year of schooling is 9.7 percent. Returns for low, middle, and high-income countries are 10.9 percent, 10.7 percent, and 7.4 percent in respective order. The value for Croatia is likely to be around 10.7 percent. These findings on rates of returns on R&D, infrastructure, and human capital show that the returns to R&D (73 percent) are quite significant and higher than the returns of infrastructure (around 24-34 percent) or human capital (around 10 percent) in Croatia. Sources: Seker (2011) and World Bank (2012). penditures. Because those calculations are made under • Goney and Maloney (2014) show a more nuanced heroic assumptions, the robustness of results is to some picture. They found that the rates of return of R&D extent debatable. In this sense, it is recommended to expenditures follow an inverted U shape: they rise take them in as reference points. Among the authors with distance to the frontier and then fall there- who have recently estimated social rates of return for after, potentially turning negative for the poorest R&D in developing countries are Lederman and Maloney countries.2 (2003), Böke (2009), and Seker (2011). A related question refers to the amount of public • Lederman and Maloney (2003) use cross-country investment in R&D necessary to generate a certain data from the 1975–2000 period and find that re- target of R&D expenditures (GERD) at aggregate level. turns to R&D investment were around 78 percent (60 For instance, new EU member countries and to some percent for medium-income countries, and around extent EU access countries have been asked to estab- 100 percent for poor countries). lished R&D target levels for the 2014–20 period. Little • Böke (2009) used a calibration exercise and found a attention has been given to how those targets would social rate of return to R&D in Turkey at around 62 be generated and what would be the corresponding percent, which, in turn, implies that the R&D levels fiscal requirements. The answer depends essentially in 2009 were between a tenth to a sixth of what on the estimated elasticity of business R&D to public they should be. support (see box 8.2). • Seker (2011) estimated the rate of return on R&D investment in Croatia at about 73 percent, sig- 2. The findings are consistent with the importance of factors complementary to R&D, such as education, the quality of nificantly larger than the returns of investment in scientific infrastructure, the overall functioning of the national infrastructure (around 24–34 percent) or human innovation system, and the quality of the private sector, which capital (around 10 percent). become increasingly weak with distance from the frontier. 92 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note 8.3. GOVERNANCE ANALYSIS Governance issues are also present in the design of programs and management of PROs. Three main set This session focus on the institutionalized rules and of issues are addressed that affect the impact of public procedures (governance) through which policies and spending: (i) Policy Making, (ii) Policy Implementation, programs were selected and implemented (governance and (iii) Policy Learning and Adaptation. structure). The main objectives are to understand how existing governance structure affects the existing al- Horizontal and Vertical Coordination of Policy location of public spending and more broadly, the management of the STI policies. Box 8.3 summarizes By affecting the behavior of stakeholders, different the vision of a well-governed national innovation sys- governance structures induce the development of tem, as articulated in the EU 2020 Strategy document different “policy mixes” and therefore the quality of (European Commission, 2000). public expenditures in STI. The challenge is to set up Box 8.2: The Cost of Reaching the Two Percent of GDP Target for R&D in Turkey in 2009 A back-of-the-envelope exercise can be done to calculate the necessary variation of public sector support to accomplish the goal of a total R&D to GDP ratio of 2 percent. In order to do this, one has first to consider the interactions between business enterprise R&D (BERD) and government R&D (GERD), and assume two different ways of public R&D spending. The first finances specifically BERD and the latter refers to total R&D spending by the government, without consideration of the sector where R&D takes place (whether BERD or GERD). Accordingly, these two ways of public R&D spending define two kinds of business sector R&D elasticity to public sector R&D. The first is the elasticity of private sector financing of BERD to the government financing of BERD, which quantifies how much the business sector R&D performance financed by the government encourages the private sector to spend on its own R&D. The second is the elasticity of total R&D financed by business to the total R&D financing of the government; this elasticity informs how much the government financing of R&D activities in the country encourages business to invest in R&D. The two elasticities are estimated in a background paper (Böke 2009), which was based on annual data between 1997 and 2007 from 36 countries. The majority of these countries are European, while others are large economies (including Russia, China, the United States, and Japan).a The first elasticity was inferred to be of the magnitude 2, which implies that an increase in government-financed BERD doubles the private financing of BERD. The second elasticity drops to a range 0.3 to 0.5, which means that when one takes into account the correlation of the two sources of financing without differentiating between the sectors in which R&D is performed, the effectiveness of government financing in crowding in business financing is reduced. In order to finally calculate the necessary variation of government expenditure in R&D (as a percent of GDP), the report defines three elasticity scenarios: 2; 0.3; and 0.5. The first one focuses specifically on the public financing of BERD while the other two relate to the total R&D spending by the government. The nuance is important because it will inform the op- tions on how further increases in public investment in R&D could be allocated (supporting more BERD, the first scenario; or keeping the current pattern of expenditures, the second or third scenarios). Solving the point elasticity equation for these three elasticity values, it is possible to construct three different scenarios, as summarized in table B8.2.1.b. If all additional public investment is allocated to finance BERD (the first scenario), public support would have to increase by about half a percentage point of GDP (0.56 percent), which in turn would make business R&D account for the largest share of R&D in the country (58 percent of the total). If one assumes public expenditures increase regardless the supported sector of support (scenarios 2 and 3), then the variation of public support would be in the range of 0.98 percent of GDP (for a elasticity of 0.5) and 1.07 percent of GDP (for a elasticity of 0.3) and business R&D would account for 25–30 percent of total R&D. Concentrating further increases in public investments in R&D on the support of private R&D seems therefore the most effective way to reach a total R&D-to-GDP ratio of 2 percent. (continued next page) Final Report 93 Box 8.2 (continued) Table B8.2.1: Scenarios on Public R&D Expenditures Scenario 1: Public Spending to Finance BERD Elasticity = 2 GERD expenditure BERD expenditure Total Baseline 0.42% GDP 0.28% GDP 0.7% GDP Share to total R&D expenditure 60% 40% n.a. ∆ (variation) 0.56% GDP 0.74% GDP 1.3% GDP Final 0.96% GDP 1.04% GDP 2.0% GDP Share to total R&D expenditure 48% 52% n.a. Scenario 2: Public Expenditure to Finance Total R&D Activities Elasticity = 0.5 GERD expenditure BERD expenditure Total Baseline 0.42% GDP 0.28% GDP 0.7% GDP Share to total R&D expenditure 60% 40% n.a. ∆ (variation) 0.98% GDP 0.32% GDP 1.3% GDP Final 1.4% GDP 0.6% GDP 2.0% GDP Share to total R&D expenditure 70% 30% n.a. Scenario 3: Public Expenditure to Finance Total R&D Activities Elasticity = 0.3 GERD expenditure BERD expenditure Total Baseline 0.42% GDP 0.28% GDP 0.7% GDP Share to total R&D expenditure 60% 40% n.a. ∆ (variation) 1.07% GDP 0.23% GDP 1.3% GDP Final 1.49% GDP 0.51% GDP 2.0% GDP Share to total R&D expenditure 75% 25% n.a. Source: World Bank 2010. Note: n.a. = Not applicable. a. The data on R&D expenditure is obtained from the EuroStat while the data for control variables mostly come from the World Develop- ment Indicators. The R&D data includes the source of the financing as well as the target sector in which the R&D is undertaken. b. The following point elasticity formula is assumed: Elasticity = (∆BERDexp/∆GERDexp) x (GERDexpi/BERDEXPi), where GERDexp is the government R&D expenditure (as % of GDP), BERDexp is the same for private R&D expenditure, and GERDexpi (BERDEXPi) is the baseline level of the government (private) R&D expenditure (as % of GDP). Accordingly, the elasticity can then assume three values: 2; 0.3; and 0.5, and once knowing the values of GERDexpi (BERDEXPi), which are the same for all the three scenarios, it is possible to find ∆BERDexp and ∆GERDexp. 94 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Box 8.3: Features of Well-Performing National Innovation Systems, as Outlined in the EU 2020 “… Design and implementation of research and innovation policies is steered at the highest political level and based on a multi-annual strategy. Policies and instruments are targeted at exploiting current or emerging national/regional strengths... • An effective and stable center-of-government structure, typically steered by the top political level, defines broad policy orientations on a multi-annual basis and ensures sustained and properly coordinated implementation. This structure is backed up by networks involving all relevant stakeholders, such as industry, regional and local authorities, parlia- ments and citizens, thereby stimulating an innovation culture and building mutual trust between science and society. • A multi-annual strategy defines a limited number of priorities, preceded by an international analysis of strengths and weaknesses at national and regional level and of emerging opportunities and market developments, and provides a predictable policy and budgetary framework. The strategy duly reflects (country) priorities, avoiding unnecessary duplication and fragmentation of efforts, and actively seeks to exploit opportunities for joint programming, cross- border co-operation and exploiting the leverage effects of EU instruments. • An effective monitoring and review system is in place, which makes full use of output indicators, international benchmarking and ex-post evaluation tools.…” Source: Innovation Union, Annex I: Self -assessment tool: Features of well performing national and regional research and innovation systems (European Commission, 2010) a structure of incentives that that align the interest of Figure 8.6 illustrates the point. The circles represent the “principal” and the “agent” in both the process of different organizations within a government (such as policy making and policy implementation. Governance sector ministries) that are responsible for policies and structure should align the incentives of the policy makers programs affecting STI. These organizations are often (such as different ministers) with that of the taxpayers in at the same or a very similar hierarchical level. The the policy-making phase. During implementation, a se- vertical arrows indicate the number of stakeholders quence of principal-agent interactions occurs through- involved in the implementation process. Stakeholders out the command structure in an administration. may include, for example, a national innovation council, Figure 8.6: Horizontal and Vertical Coordination Challenges Information Asymmetries Agriculture Health IT Defense Metallurgy Vertical coordination Final Report 95 a ministry of higher education and science, the science council of ministries, national council, or other? Is the department of that ministry, the agency responsible for decision-making process considered transparent? Is funding research, and the internal departments up to there an institutionalized space for broad discussions the implementation unit. about STI policies and consensus building? • Does the government articulate a clear long-term In such governance structures, the level of information STI strategy? Are there measurable goals and de- available decreases both horizontally and vertically, fine corresponding means for their achievement? particularly in the absence of central coordinating agen- Is the strategy coherent, feasible, and adequately cies or mechanisms to reduce asymmetric access to funded? information. The result of such an opaque governance • Are there organizations in charge of coordinating STI structure is frequently a policy-making and implementa- policies across the government? If so, how are they tion system that lacks cohesion. The system may create composed? What is the legal basis for the operation misdirected policies and programs that suit the needs of those organizations? What is the scope of their of individual agencies or stakeholders and does not mandate? Are major stakeholders well represented? adequately improve the overall system. The challenge, How does the government balance common ten- therefore, is to set up a structure of incentives that sions and contradiction in setting innovation policy aligns the interest of the principal and the agent in both agenda? the process of policy making and policy implementa- tion. Box 8.4 summarizes recent lessons in addressing Policy Implementation horizontal coordination issues. The implementation of STI policy involves a number The following are some proposed assessment questions of different actors and organizations. Without proper for governance structure. attention to those issues, the process policy implemen- tation may lead to a final allocation or disbursement • How are the major decisions about policies, programs, of public funds for purposes that are not in line with and budget allocation taken—by a prime minister, intended goals. These issues concern how the pro- Box 8.4: Addressing Horizontal Bottlenecks in Innovation Policy An important international experience in addressing horizontal coordination issues in innovation policy is the development of innovation councils. The development of such councils is part of a move toward more comprehensive, integrative, in- novation policy making. Some countries that have moved toward such councils already had a history with organizations that were more narrowly focused on science and technology. For example, Finland and the Republic of Korea have had science and technology (S&T) councils established for decades. However, at present, no dominant structure seems to have emerged. Rather, the success of innovation councils depends largely on the council’s composition, mandate, and functioning. For example, all relevant stakeholders may be formally represented at the council, but the implementation structure of the council’s recommendations is not clear. Or it may be the case that the horizontal dimension has identified the right goals and cross-ministerial planning works, but the ministries or other agencies assigned with policy implementation in the vertical dimen- sion (such as S&T or the environment) are not implementing well. Stakeholders can also differ, or very little effort may be made to engage the relevant stakeholders in the design and implementation process. Finally, policy measures to achieve innovation policy goals may differ. The main innovation policy measure is resource allocation for R&D, but in many cases this may not be a sufficient measure to achieve the desired results. Therefore a number of other regulative and fiscal measures may be required. The composition of the innovation council can also be only a partial reflection of the relevant stakeholders, due to reasons of institutional tradition or otherwise. For example, S&T policies may monopolize the state budget’s allocations for R&D. Source: OECD 2005. 96 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note grams are designed and implemented. Poor planning Monitoring and Evaluation, Policy can systematically jeopardize consistency between the Learning, and Adaptability objective at the decision-making stage and the imple- Innovation policy is context specific and is particularly mentation and impact of the policy. prone to conflicts of interest. Therefore, creating mecha- nisms for systematic evaluation of programs, policy • Are organizations that design, implement, and learning, and policy adaptation is of crucial importance. benefit from the programs legally distinct, acting Such mechanisms can built around two key elements: (i) independently according to their own missions? quality information about programs and policies; and (ii) If not fully distinct, what is the overlap and how sound accountability rules for officials and organizations does it affect the incentives for and efficient and at the core of the policy making and implementation fair implementation of programs (if it does at all)? process, in addition to transparency and participation • Are human resources in public administration from stakeholders and beneficiaries. Transparency in properly trained to manage research and innova- this case is closely related to access to information and tion programs? Is there a specific career for public quality of data available. servants working on innovation policy? Is it capable of attracting and retaining the most qualified pro- • Have evaluations been done? If so, of what type fessionals? What is the predominant field and level (survey of beneficiaries, randomized control trials, of education among public servants managing in- peer reviews, or others)? Is monitoring and evalu- novation policies? Do they participate in existing ation (M&E) mandatory for the main programs? Is networks of research and innovation policy? there an organization responsible for M&E inno- • Are material conditions (physical and financial) vation programs? If so, is it properly staffed and commensurate with existing workload? Is staff en- equipped? Is routine monitoring occurring in the couraged to improve performance? If so, how? Is majority of programs (that is, are officials able to staff insulated from day-to-day political influence? track the flow of funds and their corresponding If so how? results?) (see Gorgens and Kusek 2009). • Are government officials obliged to communicate Stability and predictability of public funds to STI are to the public and facilitate access to the results of also aspects of implementation to be considered for evaluations of programs and policies? Is information a number of reasons. First, STI activities are often accessible to the public in general (within the bound- implemented over a period of few years. Interruptions aries of standards regulations of individual rights in the flow of funds may have very negative effects on to privacy or commercial/scientific secrecy)? Are the achievement of research results. Maintenance of stakeholders ‘heard”? Is the government somehow research infrastructure is another activity that requires obliged to act upon the findings of an evaluation or recurrent expenditures. Finally, returns from public R&D funded and generalized complains of stakeholder? investments will only be realized in the long run. • What is the quality of STI statistics? Does the country have regulation firm-level innovation surveys? How • Does the current governance structure provide for comparable they are with existing surveys? Is there the stability and predictability of public expenditures any explicit strategy to improve the STI statistics in on STI? Are there mechanisms for the implementa- the country? Does the statistical office have the ca- tion of multi-year planning and multi-year budgets? pacity to gather and process STI information accord- • Are STI expenditures earmarked? If so, how does it ing to international standards? What is the quality affect the quality of public spending in STI? of the available data on public finance? Final Report 97 8.4. INSTITUTIONAL REFORMS, Tables 8A.4.1 and 8A.4.2 in annex A to this chapter POLICY RECOMMENDATIONS, show the results from a process similar to figure 8.6. The AND STRATEGIC INVESTMENTS tables show the action plan from the Western Balkans Regional R&D Strategy for Innovation. The strategy was The analysis in the PER concludes with recommenda- designed to strengthen the region’s research capacity, tions for improved public expenditures on STI. It pro- enhance intra-regional cooperation, promote collabo- vides recommendations in terms of institutional and ration with business sectors, explore possibilities for policy reforms and strategic investments, as described financing R&D from EU funding schemes and other in chapter 3. The starting point for developing the external sources, and help integrate the region with the “draft plan” is clarifying the strategic goals for the NIS. European Research Area (ERA) and Innovation Union. These goals inform what interventions will be needed, what outcomes will be generated by these interven- 8.5. CONCLUSION tions, and what outputs are needed to produce these outcomes. This informs what resources are required for This chapter discussed the Final Report. It started by the plan in terms of funding and institutional capac- reviewing the analysis of the relevance, coherence, ity. If the resources or institutional capacity needed and consistency of NIS policies. This was followed by a to achieve the “draft plan” exceed what is available, discussion of the composition and level of R&D spend- then the “plan” will need to be adjusted. This may ing. An analysis of governance concluded the chapter. require changes in one rung in the process or perhaps Box 8.5 provides a number of useful readings. The next all along the chain. chapter concludes this guidance note. Box 8.5: The Final Report—Useful Readings A proposed structure for the final PER is provided in table 9.1. Useful reading: Flanagan, Uyarra and Laranja. 2010. “The ‘Policy Mix’ for Innovation: Rethinking Innovation Policy in a Multi-Level, Multi- Actor Context.” Manchester Business School. Working Paper, Number 599. Available at: http://www.mbs.ac.uk/ research/workingpapers. OECD. 2005. “Synthesis Report, Governance of Innovation Systems.” OECD, Paris. World Bank. 2008. “Chile: Toward a Cohesive and Well Governed National Innovation System.” World Bank, Washington, DC. 98 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note ANNEX A: ACTION PLAN ILLUSTRATION FROM THE WESTERN BALKANS REGIONAL R&D STRATEGY FOR INNOVATION Table 8A.1: Example: Excerpts from the Action Plan for the Western Balkans Regional R&D Strategy for Innovation Overall Strategic Example of a Example of a short term Example of a long Goal Metrics Strategic Sub-Goal intervention term intervention 1. Improve the Citations and citation impact; 1.1. Slowing down Promote the collaboration Eliminate any bias research base and co-publications within the region brain-drain supporting of local scientists and the against young conditions for and with external partners; share ‘brain-gain’; and scientific Diaspora researchers research excellence of young researchers employed; investing in human participation in Horizon 2020. capital 2. Promote Patenting and co-patenting 2.1. Improving the Simplify the legal Develop/Unify the Research-Industry activity locally and incentive regime for requirements for regulation regarding Collaboration and internationally; licensing and collaboration between collaboration between ownership and Technology Transfer spinoff companies (number and research institutes and public universities, management of IP value); volume of joint-research the private sector research centers, and the from publicly funded projects; share of services enterprise sector. research performed by provided to the business sector PROs in total revenues; share of innovative firms collaborating with public research organizations (as measured by the Community Innovation Survey.) 3. Enable business Share of innovative companies 3.1. Improving access Develop matching grant Promote the investments in (as measured by the ‘CIS’), BERD to innovation finance schemes for pre-seed development of seed research and (Eurostat); trade-marks and ISO- (pre-seed capital) and financing and the and venture capital innovation and certifications; volume of venture mentoring services provision of ‘mentoring’ industry. startup creation capital markets. services for new enterprises and SMEs. 4. Strengthen the Volume of R&D (GERD); 4.1. Completing the Consolidation of research Further integrate governance of distribution between basic and institutional reforms of institutes. Reform local universities to national research and applied research; distribution universities and research management of public the European Higher innovation policies between mission-related and institutes research institutes Education Area (EHEA) ‘curiosity’ driven; share of public towards increasing the and advance the research organization costs use of performance- implementation of the financed through competitive based contracts and more Bologna Process. funding; indicators related to the autonomy. productivity of the system (e.g. Patent/GERD). Final Report 99 Table 8A.2: Example: Action Plan for Regional Cooperation—Summary Total cost Expected Outputs Expected Outcomes (€ million) Research 80 international collaboration research projects Contributed to Improve the Research Base 55 Excellence funded and conditions for research excellence Fund (Strategic Objective 1) 50 young researchers projects funded Slowed brain drain, supported brain gain and 200 PhDs in science from leading universities investing in human capital Networks of Larger number of joint publications in high Improved research base and conditions for 55 Excellence impact journals research excellence Program Increased mobility of researchers Investing in human capital Better use and supply of research infrastructure, Improving access to modern research facilities and availability of research funding Increased number of post graduate students in the field Increased collaboration with the business sector through join research, licensing, training and technical assistance Technology 10 TT Organizations developed and 100 staff Research Industry Collaboration and 40 Transfer trained, Technology Transfer promoted Program 100 Joint projects between research and industry Soft support for collaboration and technology supported transfer provided 3 technology parks restructured Access and performance of technology and science and technology-parks improved 3 new parks created Early Stage 300 proof of concepts and prototypes tested Enable business investments in research and 40 Startup innovation and startup creation 100 business plans/bankable projects prepared Program More knowledge-based startups created 20 consultations with foreign and local investors Investments in startup companies increased Investments in R&D by the business sector increased Number of ‘innovative’ SMEs increased (as described by the Community Innovation Surveys) Regional Coordination of regional policy dialogue and Strengthening the Governance of Research 10 Technical promotion of reforms and Innovation Policies in the Western Balkans Assistance Technical advise for the R&D Pillar of the SEE2020 Improved public expenditures in R&D Facility (WISE) Capacity building activities (technical assistance and training) provided Total — Better research, more innovation for growth 200 and job creation 100 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note CHAPTER 9 CONCLUSIONS This Guidance Note proposes a framework for the which correspond loosely to three assessment criteria analysis of public spending in science, technology, and (chapter 3): (i) operational efficiency; (ii) effectiveness; innovation (STI) and discusses how to implement it in and (iii) relevance, coherence, and consistency of the practice. Our goal is to make it flexible enough to be policy mix (that is, composition and level of public adjusted to the needs and government requests of dif- spending). The logic of the proposed PER exercise is ferent countries. The goal is to generate a set of action- summarized in figure 9.1. The exercise starts with the able recommendations to improve the quality of public consolidation of an STI budget that encompasses in- spending in STI—that is, to improve the contribution novation expenditures, which are classified accordingly of public spending on STI to economic development. to that results-oriented framework (chapter 5). The proposed PER exercise is structured around a results- The underlying idea is that public spending can be oriented framework and a set of organizing questions, improved by (i) better design of programs and ac- Figure 9.1: The Proposed PER Exercise Economic Efficiency Operational Effectiveness Efficiency Country Inputs Outputs Outcomes Impact Context Efficient Program Conditions Research and outside the Policy (Activity) Design Mix & Innovation reach of and Governance Expenditure programs Implementation Revevance, Coherence, and Consistency Source: Elaboration of Technopolis (2009). Conclusions 101 tivities through which funds are disbursed; (ii) by While the analytical framework is presented in chap- improving conditions, beyond public spending, ter 3, the “how to” is concentrated in chapters 5–8. that affect the achievement of desired outcomes Chapter 5 is dedicated to a comprehensive description (effectiveness); (iii) through a more balanced policy of public spending in STI, chapter 6 discusses the imple- mix; and (iv) reforms in the national innovation system mentation of the operational efficiency assessment, (NIS) governance structure. Recommendations are chapter 7 addresses the conditions for effectiveness, provided on how to prioritize and design measures and chapter 8 analysis the policy mix and how the including reforms to policies and programs as well governance structure shapes it. as strategic investments. Each report builds on the previous one, with the goal A major challenge for implementation of the PER is of generating a unified report in the last stage of analy- access to both data on public spending and standard sis. Yet, they may be seen as partial deliverables and, STI statistics. For that reason, implementation requires in some cases, new deliverables are conceivable. For close collaboration with the beneficiary government example, chapter 5 can be combined with chapter 8 to and a good dose of realism in defining the scope of the generate an assessment of a policy mix based only on exercise—which must be congruent with the capacity budgetary analysis. This may be useful especially when to generate necessary information. The challenges are the emphasis of the country is on public spending on likely to be larger when addressing non-R&D innovation. R&D and most of data is readily available (as illustrated This issue is discussed extensively in chapters 3 and 4. in section 8.2). Table 9.1 illustrates a possible structure Chapter 4 also discusses other issues to be addressed for the final report, linking each chapter of the PER to at the inception stage. the chapter of this Guidance Note that would inform it. Table 9.1: Final PER Report: Possible Structure Section of PER Content of the PER Guidance Note: Main Inputs Chapter 1: Country’s Needs Country’s development level and associated Inception Report (chapter 4)–section 4.1, on Country developmental challenges (needs) related to Paper the STI policies Chapter 2: Functional Review Review of public spending on STI: how much Inception Report and Functional Review (chapter 5)– is spent, by whom, on what? tables 5.1, 5.5–5.7 Chapter 3: Operational Efficiency of programs and Efficiency Assessment (Chapter 6)–section 6.3., table activities funded by the government 6.3a. Chapter 4 Economic Effectiveness of public spending: Effectiveness Assessment (Chapter 7)—table 7.1. are public expenditures on STI generating the Each section discusses effectiveness conditions for selected intermediate outcomes? Assessment each of the four intermediate outcomes of effectiveness conditions Chapter 5 Policy Mix Final Report (chapter 8)—section 8.1, table 8.1 Governance Analysis Final Report (chapter 8)—section 8.3 Chapter 6 Conclusions and Main Recommendations Analytical Framework (chapter 3)—see section 3.3 on Recommendations from PER; tables 3.1–3.3 102 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note APPENDIX A DEFINITIONS Table A.1 includes a number of key definitions from the Frascati Manual (OECD 2002). Table A.1: Key Definitions Term Definition Abroad All institutions and individuals located outside the political borders of a country, except vehicles, ships, aircraft and space satellites operated by domestic entities and testing grounds acquired by such entities. All international organizations (except business enterprises), including facilities and operations within the country’s borders. Applied research Applied research is also original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily towards a specific practical aim or objective. Basic research Basic research is experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view. Business enterprise sector All firms, organizations and institutions whose primary activity is the market production of goods or services (other than higher education) for sale to the general public at an economically significant price. The private non-profit institutions mainly serving them. Capital expenditures Capital expenditures are the annual gross expenditures on fixed assets used in the R&D programmes of statistical units. They should be reported in full for the period when they took place and should not be registered as an element of depreciation. Experimental development Experimental development is systematic work, drawing on knowledge gained from research and practical experience, that is directed to producing new materials, products and devices; to installing new processes, systems and services; or to improving substantially those already produced or installed. Extramural expenditures Extramural expenditures are the sums a unit, organization or sector reports having paid or committed themselves to pay to another unit, organization or sector for the performance of R&D during a specific period. This includes acquisition of R&D performed by other munits and grants given to others for performing R&D. Government (for purposes of Central or federal government should always be included. GBAORD) Provincial or state government should be included when its contribution is significant. Local government funds (i.e. those raised by local taxes) should be excluded. (continued next page) Definitions 103 Table A.1 (continued) Government sector All departments, offices and other bodies which furnish, but normally do not sell to the community, those common services, other than higher education, which cannot otherwise be conveniently and economically provided, as well as those that administer the state and the economic and social policy of the community. (Public enterprises are included in the business enterprise sector.) NPIs controlled and mainly financed by government, but not administered by the higher education sector. Gross domestic expenditure on GERD is total intramural expenditure on R&D performed on the national territory during a given R&D (GERD) period. Higher education sector All universities, colleges of technology and other institutions of post-secondary education, whatever their source of finance or legal status. It also includes all research institutes, experimental stations and clinics operating under the direct control of or administered by or associated with higher education institutions. Intramural expenditures Intramural expenditures are all expenditures for R&D performed within a statistical unit or sector of the economy during a specific period, whatever the source of funds. Other supporting staff Other supporting staff includes skilled and unskilled craftsmen, secretarial and clerical staff participating in R&D projects or directly associated with such projects. Private non-profit sector Non-market, private non-profit institutions serving households (i.e., the general public). Private individuals or households. R&D personnel (initial coverage) All persons employed directly on R&D should be counted, as well as those providing direct services such as R&D managers, administrators, and clerical staff. Research and experimental Research and experimental development (R&D) comprise creative work undertaken on a systematic development (R&D) basis in order to increase the stock of knowledge, including knowledge of man, culture and society, and the use of this stock of knowledge to devise new applications. Researchers Researchers are professionals engaged in the conception or creation of new knowledge, products, processes, methods and systems and also in the management of the projects concerned. Technicians and equivalent staff Technicians and equivalent staff are persons whose main tasks require technical knowledge and experience in one or more fields of engineering, physical and life sciences or social sciences and humanities. They participate in R&D by performing scientific and technical tasks involving the application of concepts and operational methods, normally under the supervision of researchers. Equivalent staff perform the corresponding R&D tasks under the supervision of researchers in the social sciences and humanities. 104 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note APPENDIX B DATA AND DATA SOURCES ON SCIENCE, TECHNOLOGY AND INNOVATION REPORTS Content This section summaries and compares the main innova- The depth of analysis depends on data availability. For tion policy frameworks including (i) the OECD Reviews example, the review of Norway’s innovation policy is of Innovation Policy; (ii) ERAWATCH Country Policy Mix much more detailed than the studies of Chile or Peru. Reports; (iii) INNO-Policy TrendChart—Policy Trends and Nonetheless all studies have similar structure and their Appraisal Report; (iv) UNCTAD’s Science, Technology content covers: and Innovation Policy Reviews (STIP Reviews); and (v) UNESCO’s STI studies (Borowik 2012). • Overall assessment and recommendations • Economic performance and framework conditions OECD Reviews of Innovation Policy for innovation (e.g. macroeconomic stability, finan- Objective: A comprehensive review and assessment of cial markets and innovation, labor force, competition a country’s innovation system. in the product market, innovation system’s SWOT analysis (as in case of Norway); recommendations • A comprehensive assessment of the innovation sys- and identification of good practices for consideration tem of individual OECD member and non-member • STI main actors (business sector, public research countries, focusing on the role of government institutes, the higher education sector, intermediary • Strong orientation towards concrete recommenda- institutions) tions across a spectrum of innovation-related policies • The role of government (STI governance and policy on how to improve policies to have an impact on mix measures, Portfolio of instruments, innovation innovation performance, including R&D policies. It budget) does not attempt to conduct detailed policy design. • International benchmarking in innovation perfor- Each review identifies good practices from other mance countries • Builds on OECD work, especially on the links be- Data tween innovation and economic performance, and on best practice policies to foster innovation OECD reviews include extensive data on innovation expenditure (this may vary among studies). Nonethe- Data and Data Sources on Science, Technology and Innovation 105 less, the reports do not contain comprehensive data • Russian Federation (2011) analysis, but rather trends in R&D expenditures, their • Mexico (2009) reasons, comparison with other developing and the • Korea, Rep of (2009) OECD countries. • Hungary (2008) As in the comprehensive report on Norway, the review • China (2008) presents data on: • Norway (2008) • Chile (2007) • Policy mix: e.g.: S&T and innovation funds and programs, their budget over time, expenditures ac- • South Africa (2007) cording to loans, budget support, other resources; • New Zealand (2007) budget committed vs. executed; estimated revenue • Luxembourg (2007) losses due to R&D tax incentives as a percent of • Switzerland (2006) GBAORD; assessment of policy mix issues and bar- riers (with trends in financing and international ERAWATCH Annual Analytical comparison; budget of an instrument) Country Reports (since 2009) • Innovation budget: Estimates of total expenditures on STI activities by source of funds i.e. direct budget- Objective: Characterize and assess the performance of ary resources and to which institution (i.e. Multilat- national innovation systems and related policies eral financing institutions’ loans, private universities, business sector, other sources); estimated R&D • Focus on the national R&D investments targets, the appropriations by ministry, selected STI programs’ efficiency and effectiveness of national policies and budget and spending categories (i.e. financing HR, investments in R&D innovation , basic or applied R&D, scholarships, etc.); • The articulation between research, education and returns from the Norway’s research fund bottom-up innovation, and on the realization and better gov- funding of free basic research; State vs. business- ernance of ERA funded R&D as a proportion of GDP; Business sector • Focus on human resource mobilization, knowledge science, technology, and innovation patterns demand, knowledge production and science-indus- • Data on PROs: details under which ministry, budget try knowledge circulation with share of institutional funding, main focus areas, • Reports cover the ‘inter-linkage’ between research number of personnel and innovation, in terms of their wider governance • R&D expenditures in benchmarking: R&D and overall and policy mix innovation expenditures by sectors and main reasons • Reports across all countries have the same structure/ for that; R&D intensity (GERD/GDP) and wealth (GDP content per capita); Norwegian GERD/GDP compared with • As these are annual reports, each year reports build the largest OECD countries on the previous ones, therefore focusing on recent • Periodicity: Not specified, every year different coun- policy changes rather than repeating what has been try review already covered The OECD reviews in this series so far include: Content • Slovenia (2012) • Performance of the national research and innovation • Peru (2011) system and assessment of recent policy changes, also in relation to ERA 106 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note • Focus on Resource mobilization (provision for re- • Serves as benchmarking and the exchange of good search activities; Evolution of national policy mix practices geared towards the national R&D investment tar- • INNO-Policy TrendChart produced “2011 mini coun- gets, Providing qualified human resources), Knowl- try reports” for each of the 48 countries monitored edge demand, Knowledge production (quality of by the network of country correspondents in the the knowledge production, the exploitability of the second half of 2011 knowledge creation and policy measures aiming to improve the knowledge creation), Knowledge Content of 2011 mini country reports circulation (between the universities, PROs and busi- ness sectors, Cross-border knowledge circulation); • Innovation policy trends containing key challenges, Knowledge transfer; Interactions between national governance, changes in the innovation policy mix policies and ERA (Towards a European labor market • Innovation policy budget including innovation mea- for researchers; Research infrastructures; Knowledge sures and evidence on effectiveness of innovation transfer policies; Cooperation, coordination in ERA); policy, future challenges for funding innovation; Assessment of the policy mix. Departmental and implementing agency budgets for innovation policies Data and innovation expenditure analysis • Demand-side innovation policies (including sectoral • Data on GERD; GBAORD; BERD; GERD financed by specificities) and the governance challenges. abroad R&D performed by HEIs/ PROs/ businesses; GERD/GDP ratio; Data and innovation expenditure analysis • Data analysis from the perspective of the reasons, • Detailed data on departmental and implementing main barriers to R&D investments and respective agency budgets for innovation policies policy opportunities and risks • Description of trends in spending, with major rea- Periodicity: Annually since 2009 sons without a deeper data analysis Countries covered: EU 27 Member States, 11 Countries Periodicity Associated to FP7 and selected third countries • Mini country reports for 2011 only Website: http://erawatch.jrc.ec.europa.eu/erawatch/ • Since 2012, the INNO Policy TrendChart and opencms/information/reports/country_rep ERAWATCH policy monitoring activities are run as a single fully integrated operation INNO-Policy TrendChart “2011 Mini Country Reports” The 2011 mini country reports include studies on: Objective: Assess the innovation policy and identify • EU 27 Member States examples of good practice, thus improving the basis for decision making in innovation policy. • Albania • Bosnia • Serves the “open policy coordination approach” laid • Brazil down by the Lisbon Council in March 2000 • China • Pursues collection, regular updating and analysis of • Croatia information on innovation policies at national and European level • Faroe Islands MCR Data and Data Sources on Science, Technology and Innovation 107 • Macedonia, FYR • Inputs, results and evaluation of the national innova- • Iceland tion system (NIS) • India • NIS institutional and legislative framework, policy mix instruments, financing measures, and gover- • Israel nance • Japan • Analysis of sectorial innovation systems—sector • Korea, Rep. of analyzed vary among country cases • Liechtenstein • Conclusions and recommendations. • Moldova • Montenegro Data and innovation expenditure analysis • Norway • Very extensive data. Presentation of trends in R&D • Russian Federation expenditure and comparison to other countries. • Serbia Based on the Salvador study data include R&D ex- • Switzerland penditure; comparative trends in R&D expenditure, investment in science and technology activities • Turkey (STA), R&D expenditure by source of financing; • United States staff employed in R&D, expenditure on STA by socioeconomic objective, patent data, expendi- Website: http://www.proinno-europe.eu/inno-policy- ture on scientific and technological R&D area of trendchart/repository/country-specific-trends knowledge; etc. • Deeper analyses regard bibliometric analysis to UNCTAD’s Science, Technology and identify the strongest areas of research, and patent Innovation Policy Reviews (STIP Reviews) analysis Objective: Assist governments in developing national capacities in science, technology, and innovation. Periodicity: not specified, each year different study • UNCTAD’s reviews are intended to be an analytical STIP Reviews comprise the following countries: tool that examine a series of proposals from a neutral external viewpoint tool for learning and reflection, • El Salvador (2011) not a rating mechanism • Ghana (2011) • The goal of the reviews is to provide the Govern- • Peru (2011) ments with an up to date diagnostic analysis of the • Lesotho—An Implementation Strategy (2010) effectiveness of their STI -related policies and mea- sures, and strengthen these policies and measures • Mauritania (2009) by integrating them in the national development • Angola (2008) process. It also seeks to improve technological capac- • The Islamic Republic of Iran (2005) ity, encourage innovation, and incorporate greater • Colombia (1999) added value into production processes. • Jamaica (1999) Content (the level of detail vary among countries) Website: http://archive.unctad.org/templates/Page. • Economic background, structural conditions and asp?intItemID=5463&lang=1 performance in STI 108 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note UNESCO STI Studies • Include recommendations on how to approach set- ting up own strategy/modify it. In this context helps Objective: Guide countries on national policy reforms. select an appropriate development module in order to develop an accurate policy. • UNESCO assists countries in formulating or refor- mulating national STI policy; Periodicity: no specified, studies based on demand • Reports adjusted to country’s developing level and STI needs. In most cases the studies support creation UNESCO STI studies series are shown in table B.1. of STI strategy from scratch or based on some first policy documents that serve as a STI plan. Table B.1: UNESCO STI Studies Series Israel (interview): The high level of basic research and innovation promotes Israeli science-based industries (2012) Seychelles (ongoing): Seychelles preparing its first science, technology and innovation policy (2011) Botswana (ongoing): Botswana instigates policy dialogue on revised STI policy in Gaborone (2011) Azerbaijan (ongoing): UNESCO assisting Azerbaijan in reviewing its STI strategy (2011) Iraq (ongoing): UNESCO helping Iraq to draw up science policy (2011) Burundi Bref état des lieux du système national de recherche scientifique et technique de la République du Burundi (2009) Armenia Towards a Science, Technology & Innovation Policy for the Republic of Armenia (2009) Tanzania (ongoing): UNESCO’s work in Tanzania since 2008 within the One UN programme Congo (ongoing): Reform of the S&T system in Congo Nigeria (ongoing): Reform of the S&T system in Nigeria Mongolia Toward a Master Plan for Science and Technology Policy (2007) Nepal Science, Research and Technology in Nepal (2006) Lesotho Lesotho Science & Technology Policy (2006) Bosnia & Herzegovina Guidelines for a Science and Research Policy (2006) Lebanon Science, Technology and Innovation Policy for Lebanon (2006) Namibia New Directions for Namibia’s Science and Technology Sector (2005) Brunei Darussalam Review Science and Technology Capacity and Policy Options (2005) Albania The Development of Albanian S&T Policy (1996) Data and Data Sources on Science, Technology and Innovation 109 INDICATORS The section describes sources for a number of indicators. describes the indicators, and provides various sources These describe scientific performance, human capital, for data on these indicators. structural factors, and innovation diffusion. The table Table B.2: Indicators Related to Science, Technology, and Innovation Performance Indicator Measure/description Source Scientific performance Patent applications, Worldwide patent applications filed through the Patent Cooperation World Intellectual nonresidents Treaty procedure or with a national patent office for exclusive rights for an Property Organization invention. Provides protection for the invention patent to the owner of the (WIPO), WIPO Patent patent for a limited period, generally 20 years. Report: Statistics on Worldwide Patent Activity Patent applications, residents Patents in United States Number of patents filed in the U.S. by residents of a country. U.S. Patent and Trademark Office Patents in Europe Number of patents filed in the European Union by residents of a country. European Patent Office Royalty and license fees, Royalty and license fees are payments and receipts between residents International Monetary receipts (Balance of payments and nonresidents for the authorized use of intangible, nonproduced, Fund, Balance of [BoP], current US$) nonfinancial assets and proprietary rights (such as patents, copyrights, Payments Statistics trademarks, industrial processes, and franchises) and for the use, through Yearbook and data files. licensing agreements, of produced originals of prototypes (such as films and manuscripts). Data are in current U.S. dollars Licenses Share of establishments (in percent) in the country/sector that have World Bank Enterprise purchased either a foreign or local license Surveys Researchers in research and Researchers in R&D are professionals engaged in the conception or creation World Development development (R&D) (per of new knowledge, products, processes, methods, or systems and in Indicators million people) the management of the projects concerned. Postgraduate PhD students (ISCED97 level 6) engaged in R&D are included. Research and development Expenditures for research and development are current and capital expenditure (% of GDP) expenditures (both public and private) on creative work undertaken systematically to increase knowledge, including knowledge of humanity, culture, and society, and the use of knowledge for new applications. R&D covers basic research, applied research, and experimental development. Scientific and technical journal Scientific and technical journal articles refer to the number of scientific National Science articles and engineering articles published in the following fields: physics, biology, Foundation, Science and chemistry, mathematics, clinical medicine, biomedical research, engineering Engineering Indicators and technology, and earth and space sciences. 110 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Table B.2 (continued) Indicator Measure/description Source Trademark applications, direct Trademark applications filed are applications to register a trademark with a World Intellectual resident national or regional intellectual property (IP) office. A trademark is a distinctive Property Organization sign that identifies certain goods or services as those produced or provided by a (WIPO) Trademark applications, direct specific person or enterprise. A trademark provides protection to the owner by nonresident ensuring the exclusive right to use it to identify goods. Human capital Technicians in R&D (per Technicians in R&D and equivalent staff are people whose main tasks World Development million people) require technical knowledge and experience in engineering, physical and Indicators life sciences (technicians), or social sciences and humanities (equivalent staff). They participate in R&D by performing scientific and technical tasks involving the application of concepts and operational methods, normally under the supervision of researchers. Availability of scientists and To what extent do you agree that scientists and engineers in your country Executive Opinion Survey, engineers are widely available? World Economic Forum 1: Disagree strongly, 5: Agree strongly Enrollment in STEM disciplines Registered students in science, technology, engineering, or mathematics Country’s own statistics (STEM). School enrollment, tertiary (% The gross enrollment ratio is the ratio of total enrollment, regardless of United Nations gross) age, to the population of the age group that officially corresponds to Educational, Scientific, the level of education shown. Tertiary education, whether or not to an and Cultural Organization advanced research qualification, normally requires, as a minimum condition (UNESCO), Institute for of admission, the successful completion of education at the secondary level. Statistics (UIS) Share of population speaking Economic Growth Center English at Yale University % of tertiary-educated Docquier and Marfouk individuals in OECD countries 2004 Structural factors State of cluster development In your country, how extensive is collaboration among firms, suppliers, Executive Opinion Survey, partners, and associated institutions within clusters? World Economic Forum 1 = Collaboration is non-existent, 7 = Collaboration is extensive Local availability of In your country, to what extent are high-quality specialized training services specialized research and available? training services 1= not available, 7= widely available University-industry To what extent do business and universities collaborate on research and collaboration development (R&D) in your country? 1 = Do not collaborate at all, 7 = Collaborate extensively Quality of scientific research To what extent do you agree that your country has adequate scientific institutions research institutions available? 1: Disagree strongly, 5: Agree strongly Intellectual property How would you rate intellectual property protection, including anti- protection counterfeiting measures, in your country? 1 = Very weak, 7 = Very strong Data and Data Sources on Science, Technology and Innovation 111 Table B.2 (continued) Indicator Measure/description Source Non R&D innovation and technology adoption Business expenditures in non- Firm turnover (%). Sum of total innovation expenditure for enterprises, in Community Innovation R&D innovation national currency and current prices excluding intramural and extramural Survey (CIS) R&D expenditures. (Community Innovation Survey: European Commission (2008) question 5.2, sum of variables RMACX and ROEKX). FDI net inflows % of GDP World Development Indicators FDI in manufacturing % of total FDI Country’s own investment statistics FDI and technology transfer To what extent does foreign direct investment (FDI) bring new technology Executive Opinion Survey, into your country? World Economic Forum 1 = Not at all, 7 = FDI is a key source of new technology Royalty and license fees, BoP (current US$) International Monetary payments Fund, Balance of Payments Statistics Yearbook, and data files Imports of high-tech goods % of GDP CEPII BACI database Imports of high tech capital % of GDP goods Imports of intermediary % of GDP goods Foreign intermediate inputs % all inputs that are foreign by country/sector World Bank Enterprise Surveys International certifications % of establishments in the country/sector that have an International Organization for Standardization (ISO) certification Technology diffusion Electrical power consumption kilowatt-hours/capita World Development Indicators International outgoing minutes telephone traffic Air transport, registered carrier Domestic takeoffs and takeoffs abroad of air carriers registered in the departures worldwide country Agricultural machinery: per 100 hectares of arable land tractors Main lines per 100 inhabitants Internet users per 1,000 inhabitants World Development Indicators Personal computers per 1,000 inhabitants Cellular subscribers per 100 inhabitants Percentage of digital mainlines per 100 inhabitants 112 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note APPENDIX C INFORMATION ON COUNTRY PERFORMANCE AND INNOVATION BENCHMARKS The World Bank’s World Development Indicators is a THE GLOBAL comprehensive source of data on country’s economic COMPETITIVENESS INDEX and social performance. It compiles a host of indicators on science and technology. UNCTAD’s trade database The World Economic Forum’s Global Competitiveness and United Nations Industrial Development Organiza- Index (GCI) provides a detailed assessment for analyzing tion (UNIDO) industry database are potentially useful a country’s overall long-term economic competitiveness. sources for data on country economic performance. The It ranks countries according to three types of attribute. World Bank’s Enterprise Surveys provides firm-level data Basic requirements encompass institutions, infrastruc- on a broad range of firm performance and investment ture, macro-economic stability, health, and primary climate variables. education. Efficiency enhancers include higher educa- tion; and training, labor efficiency, financial market A growing number of organizations produce worldwide sophistication, market size, and technological readiness. reports on competitiveness and innovation based on a Innovation and sophistication factors include business composite index. The World Economic Forum is one sophistication and innovation. commonly used source for competitiveness. The World Bank has developed a Knowledge Assessment Meth- The GCI comprises 12 pillars, including, among other odology (KAM) to generate the Knowledge Economy considerations, institutions and the rule of law. The and Knowledge Indexes. The World Intellectual Property results for public institutions have a strong bearing Organization (WIPO), Institut privé d’enseignement su- on competitiveness and include measures on: (i) périeur (INSEAD), and Cornell University jointly publish ethics and corruption, (ii) burden of government the Global Innovation Index. Eurostat produces an in- regulation, (iii) efficiency of legal framework, and (iv) novation index called the Innovation Union Scoreboard. transparency of government policy making. Excessive The OECD Science, Technology and Industry database bureaucracy, red tape, overregulation, corruption, (http://stats.oecd.org/Index.aspx?DataSetCode=IPM_ dishonesty in dealing with public contracts, and a STIO) provides data for a comparative performance of lack of transparency and trustworthiness impose sig- national science and innovation systems with a focus nificant costs to businesses and have negative impacts on its member economies. on economic development. Information on Country Performance and Innovation Benchmarks 113 The GCI model for evaluating competitiveness has THE GLOBAL INNOVATION gone through several evolutions in recent years. Most INDEX (GII) recently, a “New Global Competitiveness Index” (WEF 2008) was published that took a more comprehensive The GII is depicted in figure C.1. The index relies on two approach to determining both microeconomic and sub-indices, the Innovation Input Sub-Index and the In- macroeconomic factors influencing productivity in a novation Output Sub-Index, each built around pillars. country. The new index also models the relative impact Five input pillars capture elements of the national econ- of micro and macro factors on economies in different omy that enable innovative activities: (1) Institutions, states of economic development. (2) Human capital and research, (3) Infrastructure, (4) Market sophistication, and (5) Business sophistication. Sources: WEF (2010) and World Bank (www.worldbank. Two output pillars capture actual evidence of innovation org/kam), and Dutto and Lanvin (2013) and Innovation outputs: (6) Knowledge and technology outputs and (7) Union Scorecard (http://ec.europa.eu/enterprise/ Creative outputs. Each pillar is divided into sub-pillars policies/innovation/policy/innovation-scoreboard/ and each sub-pillar is composed of individual indica- index_en.htm). tors (84 in total). Sub-pillar scores are calculated as the Figure C.1: The Global Innovation Index—Summary Structure Global Innovation Index (average) Innovation Efficiency Ratio (ratio) Innovation Input Innovation Output Sub-Index Sub-Index Human Knowledge and capital and Market Business technology Creative research Institutions Infrastructure sophistication sophistication outputs outputs Political Knowledge Knowledge Intangible Education environment ICT Credit workers creation assets Tertiary Regulatory General Innovation Knowledge Creative goods education environment infrastructure Investment linkages impact and services Research & Business Ecological Trade & Knowledge Knowledge Online development environment sustainability competition absorption diffusion creativity Source: Dutto and Lanvin (2013): www.globalinnovationindex.org. 114 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note weighted average of individual indicators; pillar scores their performance on the four Knowledge Economy (KE) are calculated as the weighted average of sub-pillar pillars: (1) Economic Incentive and Institutional Regime, scores (figure C.1). In 2013, the ranking covered 142 (2) Education, (3) Innovation, and (4) Information and economies, accounting for 94.9 percent of the world’s Communications Technologies. Variables are normalized population and 98.7 percent of the world’s GDP (in on a scale of 0 to 10 relative to other countries in the U.S. dollars). The report has been published once a comparison group. The KAM also derives a country’s year since 2007. Annual methodological adjustments overall Knowledge Economy Index (KEI) and Knowledge in the structure of the index limit comparisons within Index (KI). long period of time. The indicator is available in six different display modes: • Basic Scorecard uses 12 key variables as proxies to THE WORLD BANK’S benchmark countries on the aforementioned four KNOWLEDGE ASSESSMENT KE pillars and derive their overall KEI and KI indexes. METHODOLOGY (KAM) The scorecard allows comparisons for up to three The KAM (see figure C.2) is an interactive benchmark- countries for 1995, 2000, and 2012. ing tool help countries identify the challenges and • Custom Scorecards allow any combination of the opportunities they face in making the transition to a 148 variables and to compare up to three countries knowledge-based economy. It consists of 148 structural or regions for 2000 and the most recent available and qualitative variables for 146 countries to measure year. Figure C.2: The Knowledge Assessment Methodology Knowledge Economy Index Knowledge Index (KEI) (KI) Economic and Institution Regime Index Education Index Innovation Index ICT Index • Tariff & Nontariff Barriers • Regulatory Quality • Rule of Law • Average years of • Royalty Payments schooling • Telephones and Receipts • Secondary Enrollment • Computers • Patent Count • Tertiary Enrollment • Internet Users • Journal Articles Source: KI and KEI Indexes website: http://go.worldbank.org/SDDP3I1T40. Information on Country Performance and Innovation Benchmarks 115 • KEI and KI Indexes present performance scores of EUROSTAT’S INNOVATION all countries on the KEI and KI indexes, as well as UNION SCOREBOARD (IUS) on the four KE pillars, in a sortable table format. The IUS provides a comparative assessment of the • Over Time Comparison demonstrates countries’ research and innovation performance of countries and progress on Knowledge Economy pillars and indexes the relative strengths and weaknesses of their research from 1995, 2000, and the most recent year. and innovation systems (figure C.3). The Scoreboard • Cross-Country Comparison allows bar-chart com- covers innovation indicators and trend analyses for the parison of up to 20 countries on their KEI and KI EU-28 member states, as well as for Serbia, the former indexes while demonstrating the relative contribu- Yugoslav Republic of Macedonia, Turkey, Iceland, Nor- tion of different KE pillars to the countries’ overall way, and Switzerland. On a more limited number of knowledge readiness. indicators available internationally, it also covers Aus- • World Map provides a color-coded map for the tralia, Brazil, Canada, China, India, Japan, the Russian global view of the world’s KE readiness for 1995, Federation, South Africa, the Republic of Korea, and the 2000, and the most recent year. United States. The IUS replaces the European Innovation Scoreboard, which was published from 2001 to 2009. The IUS distinguishes between three main types of in- dicators and eight innovation dimensions, capturing in total 25 different indicators (see figure C.3). 116 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Figure C.3: The Innovation Union Index—Summary Structure Summary Innovation Index (SII) Enablers Firm Activities Outputs Open, excellent, Finance and Linkages and Human resources Firm investments Intellectual Innovators Economic effects attractive support entrepreneurship assets research systems R&D expenditure SMEs with International R&D expenditures PCT patent Employment in New doctorate in the SMEs innovating product or scientific in the applications knowledge- graduates public sector in-house process co-publications business sector intensive activities innovations Non-R&D SMEs with Population aged Top 10 percent Innovative SMEs PCT applications Medium and Venture capital innovation marketing or 30-34 with most cited collaborating in societal high-tech product expenditure organizational tertiary education publications with others challenges exports innovations Youth with at Non-EU Public-private High-growth Knowledge- least upper Community doctorate co-publications innovative firms intensive secondary trademarks students services exports education Sales of new to Community market and designs new to firm innovations License and patent revenues from abroad Information on Country Performance and Innovation Benchmarks Source: EC 2012. 117 REFERENCES Adams, James D. 1990. “Fundamental Stocks of Knowledge Buisseret, Timothy J., Hugh M. Cameron, and Luke and Productivity Growth.” Journal of Political Economy Georghiou. 1995. “What Difference Does It Make: Ad- 98(4): 673–702. ditionality in the Public Sector Support of R&D in Large Firms.” International Journal of Technology Management Audretsch, David B., Werner Bönte, and Stefan Krabel. 10(4-6): 587–600. 2010. “ Why Do Scientists in Public Research Institu- tions Cooperate with Private Firms?” DRUID Working Canning, David, and Esra Bennathan. 2000. “The Social Papers 10-27. DRUID, Copenhagen Business School, Rate of Return on Infrastructure Investments.” Policy Department of Industrial Economics and Strategy/ Research Working Paper Series 2390. World Bank, Aalborg University, Department of Business Studies. Washington, DC. Audretsch, David B., and Maryann P. Feldman. 1996. “In- Borowik, Iwona Maria. 2012. “Review of Innovation Policy novative Clusters and the Industry Life Cycle.” Review Frameworks, A Comparison.” FPSD August. of Industrial Organization 11: 253–73. Bart, Clarysse, Mike Wright, and Philippe Mustar. 2009. Audretsch, David B., Max C. Keilbach, and Erik E. Lehmann. “Behavioural Additionality of R&D Subsidies: A Learning 2006. “Entrepreneurship and Economic Growth.” New Perspective.” Research Policy 38: 1517–33. York: Oxford University Press. Coe, David T., and Elhanan Helpman. 1995. “International Baumol, William J. 1990 “Entrepreneurship: Productive, R & D Spillovers.” European Economic Review 39: Unproductive, and Destructive.” Journal of Political 859–87. Economy 98(5): 893–921. Cohen, Wesley M., and Daniel Levinthal. 1990. “Absorptive ———. 2002. “The Free-Market Innovation Machine: Capacity: A New Perspective on Learning and Innova- Analyzing the Growth Miracle of Capitalism.” Princeton tion.” Administrative Science Quarterly 35(1): 128–152. University Press. Comin, Diego, and Bart Hobijn, 2010 “An Exploration of Bell, Martin, and Keith Pavitt. 1993 “Technological Ac- Technology Diffusion.” American Economic Review 100 cumulation and Industrial Growth: Contrasts Between (December): 2031–59. Developed and Developing Countries.” Industrial and Comin, Diego A., and Marti Mestieri. 2010. “The Intensive Corporate Change 2 (1): 157–210. Margin of Technology Adoption.” Harvard Business Böke, Selin Sayek. 2009. “Social Returns to R&D in Turkey.” School BGIE Unit Working Paper No. 11-026. Harvard Unpublished background paper for World Bank Report. University, Cambridge, MA. Available at: http://ssrn. Bilkent University, May. com/abstract=1676302 or http://dx.doi.org/10.2139/ ssrn.1676302. Branscomb, Lewis M., and Philip E. Auerswald. 2001. “Tak- ing Technical Risks: How Innovators, Executives, and Correa, Paulo, and Irem Guceri. 2013. “Tax Breaks for R&D.” Investors Manage High-Tech Risks.” Cambridge, MA: ITE Policy Brief No.4. World Bank, Washington, DC. The MIT Press. 118 References Correa, Paulo, and Pluvia Zuniga. 2013. “Public Policies Eaton, Jonathan, and Samuel Kortum. 2001. “Trade in to Foster Knowledge Transfer from Public Research Capital Goods.” European Economic Review 45(7): Organizations.” Innovation, Technology and Entrepre- 1195–1235. neurship Global Practice, Washington, DC: March 2013 European Commission. 2000. “Europe 2020 A Strategy for Crépon, Bruno, Emmanuel Duguet, and Jacques Mairesse. Smart, Sustainable and Inclusive Growth.” Brussels, 1998. “Research, Innovation, and Productivity: An 3.3.2010 COM(2010) 2020 Econometric Analysis at the Firm Level.” Economics of ———. 2006 “Indicative Guidelines on Evaluation Methods: Innovation and New Technology 7(2). Monitoring and Evaluations Indicators” European Com- Crespi, Gustavo, Alessandro Maffioli, Pierre Mohnen, mission Directorate-General Regional Policy Working and Gonzalo Vázquez. 2011. “Evaluating the Impact Document No. 2 of Science, Technology and Innovation Programs: ———. 2008. “The Community Innovation Survey 2008, a Methodological Toolkit.” IDB Technical Note No. The Harmonised Survey Questionnaire.” http://epp. IDB-TN-333. Inter-American Development Bank, eurostat.ec.europa.eu/portal/page/portal/microdata/cis. Washington, DC. Available at: http://publications.iadb. org/bitstream/handle/11319/5396/Evaluation%20the%20 ———. 2010. “Europe 2020 Flagship Initiative Innovation impact%20of%20Science%20Technology%20and%20 Union.” Brussels, 6.10.2010 COM(2010) 546 final. Innovation%20Programs%20a%20Methodological%20 ———. 2012a. “Evaluation of Innovation Activities: Guid- Toolkit.pdf?sequence=1. ance on Methods and Practices.” European Commission Crespi, G., and E. Tacsir. 2012. “Effects of Innovation on Directorate-General for Regional Policy. Employment in Latin America.” Technical Note No. IDB- ———. 2012b. Innovation Union Scoreboard 2011 . TN-496. Inter-American Development Bank Institutions European Union. Available at: http://ec.europa.eu/ for Development (IFD), Washington, DC. enterprise/policies/innovation/files/ius-2011_en.pdf. Crespi, Gustavo, and Pluvia Zuniga. 2012. “Innovation and European Parliamentary Research Service (EPRS). 2014. Productivity: Evidence from Six Latin American Coun- “Measuring Scientific Performance for Improved Policy tries.” World Development 40(2): 273–290. Making.” Science and Technology Options Assessment, Debackere, Koenraad, and Reinhilde Veugelers. 2005. “The EPRS, European Parliament, PE 527.383. Role of Academic Technology Transfer Organisations Edler, Jakob, and Luke Georghiou. 2007. “Public Procure- in Improving Industry-Science Links.” Research Policy ment and Innovation—Resurrecting the Demand Side.” 34(3): 321–42. Research Policy 36: 949–63. Denzin, Norman K. 1978. The Research Act: A Theo- Edler, Jakob, Paul Cunningham, Abdullah Gok, and Philip retical Introduction to Sociological Methods. New York: Shapira. 2013. “Impacts of Innovation Policy: Synthesis McGraw-Hill. and Conclusions Compendium of Evidence on the Ef- Docquier, Frederic, and Abdeslam Marfouk. 2004. “Mea- fectiveness of Innovation Policy Intervention Project.” suring the International Mobility of Skilled Workers Manchester Institute of Innovation Research, Manches- (1990–2000).” Policy Research Working Paper Series ter Business School funded by NESTA. 3381. World Bank, Washington, DC. Ellis, S. 2008. “The Current State of International Science Drezgić, Saša. 2008. “The Effects on Public Sector Invest- Statistics for Africa.” The African Statistical Journal 6: ments on Economic Growth of Croatia.” Working Paper, 177–89. University of Rijeka, Croatia. European Parliamentary Research Service (EPRS). 2014. Dutta, Soumitra, and Bruno Lanvin. 2013. “The Global “Measuring Scientific Performance for Improved Policy Innovation Index 2013.” Cornell University, INSEAD, Making.” Science and Technology Options Assessment, and WIPO. EPRS, European Parliament, PE 527.383. References 119 Erdil, Erkan, and Dilek Çetin. 2014. “ERAWATCH Country Hall, Bronwyn H. 2002. “The Financing of Research and Report 2012: Turkey.” ERAWATCH. Available at: Development.” Oxford Review of Economic Policy http://erawatch.jrc.ec.europa.eu/erawatch/opencms/ 18(1): 35–51. information/reports/countries/tr/report_0006. Hall, Bronwyn H., and Jacques Mairesse. 2006. “Empirical Fagerberg, Jan, Martin Srholec, and Bart Verspagen. 2010. Studies of Innovation in the Knowledge Driven Econo- “Innovation and Economic Development.” In B. Hall and my.” NBER Working Paper No.12320. National Bureau N. Rosenberg, eds., Handbook of the Economics of In- of Economic Research, Cambridge, MA. novation, Vol. II, 833–872. North Holland. Hall, B., and N. Rosenberg. 2010. Handbook of the Econom- Fahrenkrog, Gustavo, Wolfgang Polt, Jaime Rojo, Alexan- ics of Innovation. Amsterdam and New York: Elsevier. der Tubke, and Klaus Zinocker. 2002. “RTD Evaluation Hall, Robert. E., and Charles I. Jones. 1999. “Why Do Some Toolbox—Assessing the Socio-Economic Impact of Countries Produce So Much More Output per Worker RTD-Policies.” Strata Project HPV 1 CT 1999–00005. than Others?” The Quarterly Journal of Economics 114 Flanagan, Uyarra and Laranja. 2010. “The ‘Policy Mix’ for (February): 83–116. Innovation: Rethinking Innovation Policy in a Multi-Level, Hodgson, G. M. 2006. “What Are Institutions?” Journal of Multi-Actor Context.” Manchester Business School. Economic Issues 40(1): 1–25. Working Paper, Number 599. Available at: http://www. mbs.ac.uk/research/workingpapers. Hsu, Fang-Ming, Der-Juinn Horng, and Chao-Chih Hsueh. 2009. “The Effect of Government-Sponsored R&D Pro- Fowler, Martin, Patrick Abbott, Stephen Akroyd, John grammes on Additionality in Recipient firms in Taiwan.” Channon, and Samantha Dodd. 2011. “Forest Sector Technovation 29(3): 204–17. Public Expenditure Reviews: Review and Guidance Note.” Program on Forests (PROFOR). World Bank, International Monetary Fund (IMF). 2001. Government Washington, DC. Finance Statistics Manual 2001. Washington, DC: IMF. Gök, Abdullah, and Jakob Edler. 2011. “The Use of Be- Indecon. 2008. “Value for Money Review of Science havioural Additionality in Innovation Policy-Making.” Foundation Ireland.” Department of Enterprise Trade Research Evaluation 21(4): 306–18. and Employment by Indecon International Economic Consultants. Goldstein, Markus. 2010. “An Introduction to Impact Evalu- ation.” World Bank Presentation PRMPR. World Bank, Independent Evaluation Group. 2013. “World Bank Group Washington, DC. Support for Innovation and Entrepreneurship: An Inde- pendent Evaluation.” World Bank Independent Evalua- Goñi, Edwin, and William F. Maloney. “Why Don’t Poor tion Group, September. World Bank, Washington, DC. Countries Do R&D.” Policy Research Working Paper 6811. World Bank, Washington, DC. Jaffe, A. 1989. “The Real Effects of Academic Research.” American Economic Review 79: 957–970. Gorgens, M., and J. Z. Kusek. 2009. Making Monitoring and Evaluation Systems Work. Washington, DC: World Bank. Jones, Charles I., and John Williams. 1998 “Measuring the Social Rate of Return to R&D.” Quarterly Journal of Guimón, Jose. 2014. “Standards and Certification.” World Economics 113(4): 119–35. Bank and OECD Innovation Platform. https://www.in- novationpolicyplatform.org. Kane, T. 2010. “The Importance of Startups in Job Creation and Job Destruction.” Kauffman Foundation Research Griffith, Rachel, Stephen Redding, and John Van Reenen. Series: Firm Formation and Economic Growth. Ewing 2003. “R&D and Absorptive Capacity: Theory and Em- Marion Kauffman Foundation, Kansas City, Missouri. pirical Evidence.” Scandinavian Journal of Economics 105(1): 99–118. 120 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note Kaplinsky, Raphael. 2010. “The Role of Standards in Global Organisation for Economic Co-operation and Development Value Chains and Their Impact on Economic and Social (OECD). 1997. “Policy Evaluation in Innovation and Upgrading.” World Bank, Washington, DC. Technology: Towards Best Practices.” OECD, Paris. Keller, Wolfgang. 2004. “International Technology Diffusion.” ———. 2002. Frascati Manual: Proposed Standard Practice Journal of Economic Literature 42(3): 752–82. for Surveys on Research and Experimental Development, 6th Edition. Paris: OECD. Available at: http://www.oecd. Kim, Linsu. 1997 “From Imitation to Innovation: The Dy- org/science/inno/frascatimanualproposedstandard namics of Korea’s Technological Learning.” Harvard practiceforsurveysonresearchandexperimentaldevelop Business School, Boston, MA. ment6thedition.htm. Lederman, Daniel, and William F. Maloney. 2003. “R&D and ———. 2003. “Turning Science into Business Patenting and Li- Development.” World Bank Policy Research Working censing at Public Research Organisations.” OECD, Paris. Paper 3024. World Bank, Washington, DC. ———. 2005. “Synthesis Report, Governance of Innovation Link, Albert N., and John T. Scott. 2003. “U.S. Science Systems.” OECD, Paris. Parks: The Diffusion of an Innovation and Its Effects on the Academic Missions of Universities.” Journal of ———. 2008a. “OECD Reviews of Innovation Policy, Industrial Organization 21: 1323–56. Norway.” OECD, Paris. Available at: www.oecd.org/sti/ innovation/reviews. Link, Albert N., John T. Scott, and Donald S. Siegel. 2003. “The Economics of Intellectual Property at Universities: ———. 2008b. “Inventory of National Science, Technology An Overview of the Special Issue.” International Journal and Innovation Policies for Nanotechnology.” OECD, of Industrial Organization 21: 1217–25. Paris. Louis Lengrand & Associés. 2006. “Smart Innovation: A ———. 2008c. “OECD Science, Technology and Industry Practical Guide to Evaluating Innovation Programmes.” (STI) Outlook 2008—Policy Questionnaire” OECD, A Study for DG Enterprise and Industry. Paris. Lybbett, Travis, and Daniel Sumner. 2010. “Agricultural ———. 2010a. “R&D Tax Incentives: Rationale, Design, Technologies for Climate Change Mitigation and Evaluation.” OECD, Paris. Adaptation in Developing Countries: Policy Options ———. 2010b. “The Innovation Policy Mix.” In Science, for Innovation and Technology Diffusion.” Issue Brief Technology and Industry Outlook 2010, 251–279. Paris: No. 6. International Centre for Trade and Sustainable OECD. Development (ICTSD). Available at: http://dspace. africaportal.org/jspui/handle/123456789/29144. ———. 2012a. “Measuring R&D in Developing OECD Countries.” OECD, Paris. STI/EAS/STP/NESTI(2011)5/ Miles, Matthew B., and Michael A. Huberman. 1994. Qualita- FINAL. tive Data Analysis: An Expanded Sourcebook, Second Edition. Thousand Oaks, CA: Sage. ———. 2012b. Science, Technology and Industry Outlook 2012. Paris: OECD. Mueller, Pamela. 2006. “Exploring the Knowledge Filter: How Entrepreneurship and University–Industry Rela- ———. 2013a. “Main Science and Technology Indicators.” tionships Drive Economic Growth.” Research Policy Volume 2013/1, June 2013. OECD, Paris. 35(10): 1499–1508. ———. 2013b. “OECD-NESTI Data Collection on Tax In- Nin-Pratt, Alejandro. 2011. “Agricultural R&D Investment, centive Support for R&D Expenditures.” OECD, Paris. Poverty and Economic Growth in Sub-Saharan Africa: Ozcelik, E., and E. Taymaz. 2008. “R&D Support Programs Prospects and Needs to 2050.” ASTI Conference Work- in Developing Countries: The Turkish Experience.” Re- ing Paper No. 9. Agricultural Science and Technology search Policy 37: 258–275. Indicators (ASTI), Washington, DC. References 121 Papaconstantinou, G., N. Sakurai, and A. Wyckoff. 1996. Findings from ATP’s First Decade.” Prepared for the “Embodied Technology Diffusion: An Empirical Analysis Economic Assessment Office Advanced Technology for 10 OECD Countries.” OECD Science, Technology Program, National Institute of Standards and Technol- and Industry Working Papers, 1996/01. OECD, Paris. ogy, Gaithersburg, MD 20899-4710. Parente, Stephen L., and Edward C. Prescott. 1994 “Bar- Saliola, F., and M. Seker. 2011. “Total Factor Productivity riers to Technology Adoption and Development.” The Across the Developing World.” Enterprise Surveys, En- Journal of Political Economy 102(2): 298–21. terprise Note Series, Productivity, 68273. World Bank, Washington, DC. ———. 2005. “A Unified Theory of the Evolution of Inter- national Income Levels.” In P. Aghion and S. Durlauf, Şeker, Murat. 2011. “Importing, Exporting, and Innovation eds., Handbook of Economic Growth, Vol I. Elsevier in Developing Countries.” World Bank Working Paper. Science Publishers. World Bank, Washington, DC. Patel, P., and K. Pavitt. 1994. “The Nature and Economic Scott, John. 2003. “Absorptive Capacity and the Efficiency Importance of National Innovation Systems.” STI Review of Research Partnerships.” Technology Analysis & No. 14, OECD, Paris. Strategic Management 15(2): 247–53. Crespi, Gustavo, and Pluvia Zuniga. 2010. “Innovation and Shackleton, Robert. 2013. “Total Factor Productivity in Productivity: Evidence from Six Latin American Countries.” Historical Perspective.” Working Paper Series 2013-1. Inter-American Development Bank, Washington, DC. Congressional Budget Office, Washington, DC. Prescott, Edward. C. 1998. “Needed: A Theory of Total Factor Siegel, Donald S., R. Veugelers, and M. Wright. 2007. “Technol- Productivity.” International Economic Review 39: 525–51. ogy Transfer Offices and Commercialization of University Intellectual Property: Performance and Policy Implications.” Psacharopoulos, George. 1994. “Returns to Investment Oxford Review of Economic Policy 23(4): 640–60. in Education: A Global Update.” World Development 22(9): 1325–43. Stiglitz, J. 2000. The Economics of the Public Sector. W.W. Norton & Company. Psacharopoulos, George, and Harry A. Patrinos. 2004. “Returns to Investment in Education: A Further Update.” Swann, Gavin P. 2010. “International Standards and Trade: Education Economics 12(2): 111–34. A Review of the Empirical Literature.” OECD Trade Policy Papers No. 97, OECD Publishing. Radas, S., I.-D. Anić, I. Božić, J. Budak, and E. Rajh. 2011. “Evaluation of the Innovation Programs Financed by Syverson, C. 2011. “What Determines Productivity?” Journal World Bank in Croatia.” Background Paper for the ICR of Economic Literature 49(2): 326–65. of the Croatia STP. Institute of Economics, Zagreb. Technopolis Group. 2009. “Evaluating the National Rogers. J. 2013. “Technology Extension Services.” IPP Innovation Strategy for Competitiveness.” Mimeo, 17 Policy Brief. World Bank, Washington, DC. June. http://biblioteca.cnic.cl/media/users/3/181868/ files/18813/Evaluating_the_National_Innovation_ Roper, Stephen. Hewitt-Dundas, Nola Margot, and James H. Strategy_for_Competitiveness.pdf. Love. 2004. “An Ex Ante Evaluation Framework for the Regional Benefits of Publicly Supported R&D Projects.” Technopolis Group & MIOIR. 2012. “Evaluation of Innovation Research Policy 33: 487–509. Activities. Guidance on Methods and Practices.” Study funded by the European Commission, Directorate for Rosenberg, N., and R. Nelson. 1994. “American Universities Regional Policy. and Technical Advance in Industry.” Research Policy 23: 323–48. Trajtenberg, M. 2006. “Innovation Policy for Development: an Overview.” WP 6-06, July. Foerder Institute for Eco- Ruegg, Rosalie, and Irwin Feller. 2003 “A Toolkit for Evalu- nomic Research, Tel Aviv. ating Public R&D Investment Models, Methods and 122 Public Expenditure Reviews in Science, Technology, and Innovation: A Guidance Note United Nations Conference on Trade and Development ———. 2009. “Turkey National Innovation and Technology (UNCTAD). 2012. “Science, Technology & Innovation System: Recent Progress and Ongoing Challenges.” Policy Review: Dominican Republic.” UNCTAD/DTL/ Europe and Central Asia Region. Report No. 48755-TR. STICT/2012/1. United Nations, New York, NY. World Bank, Washington, DC. United Nations Organization for Education, Science and ———. 2010. “Innovation Policy A Guide for Developing Culture, Institute for Statistics (UNESCO-UIS). 2009. Countries.” The International Bank for Reconstruction “Definitions of R&D, Innovation and S&T Activities.” and Development/The World Bank, Washington, DC. Training workshop on science, technology and innova- ———. 2011 “Romania Functional Review: Research, tion indicators, Cairo, Egypt, 28–30 September 2009. Development, and Innovation Sector.” World Bank, ———. 2013. “Summary Report of the 2012 UIS Innova- Washington, DC. tion Metadata Collection.” UNESCO-UIS, Montreal, ———. 2012. “Implementation Completion and Results Report Canada. (IBRD-73200) on a Loan in the Amount of Euro 30.0 Million United Nations Industrial Development Organization (UNI- (US$39.0 Million Equivalent) to the Republic of Croatia DO). 2009. “Breaking In and Moving Up: New Industrial for a Science and Technology Project.” World Bank, Challenges for the Bottom Billion and the Middle Income Washington, DC. Available at: http://www-wds.worldbank. Countries.” UNIDO Industrial Development Report 2009. org/external/default/WDSContentServer/WDSP/IB/2012 UNIDO, Vienna. /03/12/000333037_20120312003316/Rendered/INDEX/ ICR20700P080250IC0disclosed03080120.txt. World Economic Forum (WEF). 2010. Global Competitive- ness Report 2010/2011. www.gcr.weforum.org. ———. 2013a. “Trade and Competitiveness Toolkit.” World Bank, Washington, DC. ———. 2008. “New Global Competitiveness Index.” www. gcr.weforum.org. ———. 2013b. “Western Balkans Regional R&D Strategy for Innovation.” World Bank, Washington, DC. World Bank. 2007. “Spending for Development: Making the Most of Indonesia’s New Opportunities. Indonesia ———. 2014a. “Creating Your Own Angel Investor Group: A Public Expenditure Review 2007.” World Bank, Wash- Guide for Emerging and Frontier Markets.” World Bank, ington, DC. Washington, DC. ———. 2008. “Chile: Toward a Cohesive and Well Governed ———. 2014b. “Poland Mid-Term Review—Applied Re- National Innovation System.” World Bank, Washington, search Program.” World Bank, Washington, DC. DC. Zucker, G. Lynne, Darby R. Michael, and Brewer B. Mari- lynn. 1998. “Intellectual Human Capital and the Birth of U.S. Biotechnology Enterprises.” The American Economic Review 88(1): 290–306. References 123 The purpose of this Guidance Note is to help countries to assess the quality of public spending on STI. It adopts a results-oriented framework, combining the consolidation of STI expenditures with the analysis of their main outputs, intermediate outcomes, and developmental impact. The framework proposes WKHDQDO\VLVRIWKUHHPDLQVRXUFHVRIGHÀFLHQFLHV L SURJUDPGHVLJQ LPSOHPHQWDWLRQ LL LQVWLWXWLRQDOFRQGLWLRQVDQGWKH LLL FRPSRVLWLRQDQGOHYHO of public expenditure. The main product of this exercise is an integrated set of actionable measures combining institutional reforms with changes in the policy PL[ WKHFRPSRVLWLRQDQGOHYHORISXEOLFVSHQGLQJ DQGVWUDWHJLFLQYHVWPHQWV