Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005168 IMPLEMENTATION COMPLETION AND RESULTS REPORT 7599-AR ON A LOAN IN THE AMOUNT OF US$195 MILLION TO THE ARGENTINE REPUBLIC FOR AN Unleashing Productive Innovation Project (P106752) MARCH 2020 Finance, Competitiveness, and Innovation Global Practice Latin America And Caribbean Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective July 23, 2019) Currency Unit = Argentine Peso 1 ARS = 0.016 USD 1 USD = 63.69 ARS FISCAL YEAR July 1 - June 30 ABBREVIATIONS AND ACRONYMS AF Additional Financing ANPCyT Agency for the Promotion of Science and Technology BECAS TIC Becas Tecnologías de la Información y de las Comunicaciones – Scholarships for Information Communications and Technology BECAS Becas Jovenas Profesionales Tecnologías de la Información y de las Jovenas Comunicaciones – Scholarships for Young Profesionals in Information Profesionales Communications and Technology TIC CAS Country Assistance Strategy CONICET National Scientific and Technical Research Council CPF Country Partnership Framework CPS Country Partnership Strategy DTEC Doctores en Universidades para Transferencia Tecnológica – University Doctorate in Technology Transfer ERR Economic Rate of Return FECOAGRO Federation of Agricultural Cooperatives FM Financial Management FONARSEC Fondo Argentino Sectorial – Argentinian Sectoral Funds Program FONCYT Fund for Scientific and Technological Research FONSOFT Trust Fund for Promotion of the Software Industry FSAT Fondos Sectoriales – Sectoral Funds GDP Gross Domestic Product GTEC Programa de Gerentes y Vinculadores Tecnológicos – Technology Managers and Transfer Program ICT Information and Communications Technology IDB Inter-American Development Bank IPR Intellectual Property Rights ISR Implementation Status and Results report MINCyT Ministry of Science, Technology, and Productive Innovation M&E Monitoring and Ealuation NPV Net Present Value OECD Organisation for Economic Co-operation and Development PAD Project Appraisal Document PDO Project Development Objective PMT III Program for Technological Modernization PPD Public-Private Dialogue PPP Public Private Partnerships R&D Research and Development SME Small and Medium Enterprise SPU Secretariat of University Policies STI Science Technology and Innovation TBF Technology-based firm TOC Theory of Change TTL Task Team Lead UGSA Socio-Environmental Management Unit VCF Venture Capital Fund Regional Vice President: Humberto Lopez Country Director: Jordan Z. Schwartz Regional Director: Robert R. Taliercio Practice Manager: Zafer Mustafaoglu Task Team Leader(s): Tugba Gurcanlar ICR Main Contributor: Puja Guha, Mariano Pereira TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 10 II. OUTCOME .................................................................................................................... 12 A. RELEVANCE OF PDOs ............................................................................................................ 12 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 13 C. EFFICIENCY ........................................................................................................................... 20 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 21 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 23 A. KEY FACTORS DURING PREPARATION ................................................................................... 23 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 24 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 25 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 25 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 26 C. BANK PERFORMANCE ........................................................................................................... 27 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 31 V. LESSONS AND RECOMMENDATIONS ............................................................................. 32 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 37 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 50 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 52 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 54 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 57 ANNEX 6. SUMMARY OF IMPACT EVALUATION .................................................................... 58 ANNEX 7. CASE STUDIES – SECTORAL FUND AND EMPRETECNO - SUCCESS EXPERIENCES ...... 60 ANNEX 8. UNIVERSITIES RETAINING THE GTEC PROGRAM .................................................... 64 The World Bank Unleashing Productive Innovation Project (P106752) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P106752 Unleashing Productive Innovation Project Country Financing Instrument Argentina Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency The Argentine Republic Agencia Nacional de Promocion Cientifica y Tecnologica Project Development Objective (PDO) Original PDO The Objective of the Project is to expand the Borrower's capacity to generate productive innovation in knowledge-based areas by: (a) facilitating the creation of new knowledge-based companies; (b) developing specialized human capital; (c) supporting productive research innovation activities and their technology commercialization in the areas of biotechnology, nanotechnology and information and communication technology; (d) upgrading the research infrastructure in the areas of science, technology and productive innovation; and (e) strengthening the policy framework governing science, technology, and productive innovation. Page 1 The World Bank Unleashing Productive Innovation Project (P106752) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 150,000,000 150,000,000 150,000,000 IBRD-75990 45,000,000 32,830,826 32,830,826 IBRD-86340 Total 195,000,000 182,830,826 182,830,826 Non-World Bank Financing 0 0 0 Borrower/Recipient 80,000,000 80,000,000 76,660,000 Total 80,000,000 80,000,000 76,660,000 Total Project Cost 275,000,000 262,830,826 259,490,826 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 25-Sep-2008 21-Aug-2009 12-Nov-2012 30-Sep-2014 25-Sep-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 09-Dec-2009 8.51 Change in Components and Cost 24-Jan-2012 30.90 Change in Components and Cost 21-Feb-2014 91.81 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories 07-Jul-2015 128.95 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories 13-Jun-2016 149.63 Additional Financing Change in Results Framework Change in Loan Closing Date(s) 22-Aug-2018 172.15 Change in Loan Closing Date(s) Page 2 The World Bank Unleashing Productive Innovation Project (P106752) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 17-Oct-2008 Satisfactory Satisfactory .15 02 26-May-2009 Moderately Satisfactory Moderately Satisfactory .47 03 22-Dec-2009 Satisfactory Satisfactory 8.51 04 16-Jun-2010 Satisfactory Satisfactory 12.47 05 02-Jan-2011 Moderately Satisfactory Moderately Satisfactory 14.07 06 09-Aug-2011 Moderately Satisfactory Moderately Satisfactory 16.64 07 26-Mar-2012 Moderately Satisfactory Moderately Satisfactory 30.90 08 12-Oct-2012 Moderately Satisfactory Moderately Satisfactory 58.75 09 24-Apr-2013 Moderately Satisfactory Moderately Satisfactory 71.97 10 20-Nov-2013 Moderately Satisfactory Moderately Satisfactory 88.49 11 16-Jun-2014 Moderately Satisfactory Moderately Satisfactory 96.28 12 26-Dec-2014 Moderately Satisfactory Moderately Satisfactory 116.51 13 30-Jun-2015 Moderately Satisfactory Moderately Satisfactory 128.95 14 31-Dec-2015 Satisfactory Satisfactory 149.62 15 09-May-2016 Satisfactory Satisfactory 149.63 16 30-Jun-2016 Satisfactory Satisfactory 149.63 17 21-Dec-2016 Satisfactory Satisfactory 149.63 18 30-Jun-2017 Satisfactory Satisfactory 165.96 19 04-Jan-2018 Satisfactory Satisfactory 172.15 20 11-Jul-2018 Satisfactory Moderately Satisfactory 172.15 21 25-Feb-2019 Satisfactory Moderately Satisfactory 179.51 Page 3 The World Bank Unleashing Productive Innovation Project (P106752) 22 25-Sep-2019 Satisfactory Moderately Satisfactory 180.14 SECTORS AND THEMES Sectors Major Sector/Sector (%) Information and Communications Technologies 16 Public Administration - Information and 4 Communications Technologies ICT Services 8 Other Information and Communications Technologies 4 Education 15 Tertiary Education 15 Industry, Trade and Services 69 Public Administration - Industry, Trade and Services 15 Other Industry, Trade and Services 54 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Economic Policy 42 Trade 42 Trade Facilitation 42 Private Sector Development 53 Business Enabling Environment 43 Innovation and Technology Policy 43 Public Private Partnerships 10 Page 4 The World Bank Unleashing Productive Innovation Project (P106752) Human Development and Gender 16 Education 16 Access to Education 4 Science and Technology 4 Teachers 4 Standards, Curriculum and Textbooks 4 ADM STAFF Role At Approval At ICR Regional Vice President: Pamela Cox Ceyla Pazarbasioglu-Dutz Country Director: Pedro Alba Jordan Z. Schwartz Director: Marcelo Giugale Robert R. Taliercio Practice Manager: Lily L. Chu Zafer Mustafaoglu Task Team Leader(s): Esperanza Lasagabaster Tugba Gurcanlar ICR Contributing Author: Puja Guha Page 5 The World Bank Unleashing Productive Innovation Project (P106752) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. At the onset of the project, Argentina was experiencing a healthy recovery following the 2001- 2002 financial crisis with 8-9% growth. Several factors contributed to this, including high commodity prices for exports, improved domestic investment spending and consumption, and better fiscal performance. Fiscal and external surpluses supported macroeconomic stability and the accumulation of foreign reserves of approximately US$50 billion. Argentina’s economy however remained vulnerable due to its concentration in low value export products. 2. While most exports remained low-value, pockets of innovative SMEs had emerged. For example, a cluster of software producers had demonstrated exports valued at US$400 million in 2007. A strong research foundation in biology and biomedicine had also supported the development of 86 biotech enterprises with exports of US$75 million at this time. 3. Although public research institutions were strong, firm innovation remained low, with private R&D expenditures (0.14% of GDP) less than half of regional peers such as Brazil (0.36% of GDP) and Chile (0.30% of GDP). The currency revaluation in 2002 made equipment purchases more expensive, reducing both equipment purchases and overall private investment in innovation. Firms considered innovation investments to be risky and long-term, requiring a more stable macro environment. 4. Moreover, other weaknesses within the innovation ecosystem also persisted, meaning that despite strong public research capacity, the economy was characterized by low levels of commercialization, innovation, and productivity growth. These weaknesses included: i) a culture of siloed and supply-driven innovation, ii) human capital capacity that was poorly matched with private sector needs, iii) a lack of incentives for commercialization and productivity facing R&D, iv) poor informational and institutional linkages and coordination between research institutions, the private sector and other relevant actors, and v) limited available financing for innovation, particularly for start-ups. 5. Despite these hindering factors, rising public expenditures in Science, Technology and Innovation (STI) and the creation of the Ministry of Science, Technology, and Productive Innovation (MINCyT) demonstrated a strong public commitment. As part of its commitment to supporting innovation, the MINCyT was also in the process of creating sectoral funds to support biotech, ICT, and nanotech. The Agency for the Promotion of Science and Technology (ANPCyT), which would implement the project, had been established in 1996. 6. The Argentinian government requested World Bank support for the STI policy agenda, particularly development of new technology ventures, development of innovation funds, and strengthening of human capital for technological companies. In particular, the project aligned with the National Strategic Plan for STI 2006-2010 (Plan Estratégico Nacional de Ciencia, Tecnologia e Innovación). The Bank’s expertise in these areas was based on projects in Mexico, Uruguay, and Chile, along with Korea, Turkey, and Russia. The project design was coordinated with the IDB to ensure complementary support with IDB projects such as the Program for Technological Modernization (PMT III). Page 6 The World Bank Unleashing Productive Innovation Project (P106752) 7. The project aligned with the Country Assistance Strategy (CAS) at the time of appraisal, with its specific support for promoting sustainable private sector led growth and more productive employment. Theory of Change (Results Chain) Figure 1. Theory of Change Activities Outputs - Intermediate Indicators • Establish graduate programs for • Postgraduate program in technology brokering technology brokering and • No. of graduates in tech transfer and ICT management: development of careers curricula and scholarships • Tech transfer subprojects with action plan • ICT scholarships • Creation of CONICET tech transfer unit • Support to CONICET on tech • CONICET’s researchers and personnel transfer/commercialization trained in tech transfer/commercialization • Financial support for business plans, investment plans and/or Expand local capacity to technology prototypes • New business plans prepared generate productive innovation • Deal flow facilitation for new STI in knowledge-based areas business ideas • Universities with int’l norms • Skilled jobs created on IPR to facilitate innovation • Improved firm productivity • Technology grants for research • Upgraded firms, labs, and • Increased private sector consortia to commercialize research consortia that investment technology • No. of Consortia created innovate • Better conditions for • Innovation grants for individual • CONICET disclosures – more innovation and improved firms research available for access to technology commercialization • Refurbishments and specialized • New Technology-based firms equipment for research centers & Funding for start-ups • No. of upgraded labs and tech centers • Construction of national knowledge center for STI • Establishment of int’l advisory committee • Strengthening MINCyT and ANPCyT to MINCyT C1: Creating human capital for productive innovation capacity • No. of independent evaluations/studies • Research centers implementing shared C2: Support for knowledge-based companies • Support to tech transfer offices at universities best practices for IPR resource tool C3: Sectoral Funds • Analysis of nat’l innovation systems • No. of staff at universities trained on IPR and project dissemination • Beneficiary satisfaction surveys C4: Upgrading of research infrastructure • Positive beneficiary perception of ANPCyT • Investment catalyzed C5: Strengthening the policy framework Note: Prepared by ICR authors. *CONICET is the National Scientific and Technical Research Council. Project Development Objectives (PDOs) 8. The project development objective (PDO) was to expand Argentina’s capacity to generate productive innovation in knowledge-based areas by (i) facilitating the creation of new knowledge- based companies; (ii) fostering specialized human capital; (iii) supporting productive research innovation projects and their commercialization in three priority technology areas (biotechnology, nanotechnology, and Information and Telecommunication; (iv) upgrading research-infrastructure in the areas of science, technology and productive innovation; and (v) strengthening the policy framework governing science, technology, and productive innovation. The PDO was not revised during project implementation. Page 7 The World Bank Unleashing Productive Innovation Project (P106752) 9. The original PDO indicators used to assess achievement of the PDO were: a. Increase in the number of technology broker and technology manager students b. Increase in graduates from ICT related careers c. Number of technology companies that have received financing from “early stage fund” d. Number of consortia (public-private and private-private) formed e. Number of upgraded research laboratories and technology centers f. Number of universities that successfully updated / established norms on property rights according to international best practices Adjustments to the PDO indicators 10. Cancellation of one PDO indicator due to Project restructuring: The above noted PDO indicators remained consistent until Additional Financing in 2016 with one exception. In 2012, the Project was restructured to cancel the component associated with the Venture Capital Fund. As such, the PDO indicator related to this fund—“number of technology companies that have received financing from ‘early stage fund’” was eliminated. 11. Revision of PDO indicators at additional financing: In 2016, an additional loan of US$45million was added to the Project. As part of this additional financing, the Results Framework was adjusted to better capture project outcomes rather than outputs and four PDO indicators were revised as intermediate indicators. Following the additional financing, the PDO indicators are: a. Number of universities that successfully updated/established institutional changes or norms on international property rights according to international best practices b. Percentage of firms, labs and/or research consortia supported by the Project that introduced to existing or new markets new/improved products/protocols/service c. Disclosures made by CONICET Researchers d. Number of technology-based start-ups that formally register as companies e. Attraction of External Funding for tech start-ups 12. The full Results Framework provided in Annex 1 also includes a number of intermediate indicators which are linked to activities under specific Project components. Project Components 13. Component 1: Creating human capital for productive innovation. Under this component, the Project aimed to support (a) graduate program grants to establish/strengthen programs related to technology brokering and management, thus supporting a combination of a scientific background with business knowledge to commercialize technological ideas; (b) scholarship provisions for undergraduate students in ICT programs; and (c) Technology Transfer grants to solve technology-related problems and capitalize on local / regional socio-economic development opportunities. Parts (a) and (b) were part of the original project design, while part (c) was added during the 2014 restructuring. 14. Component 2: Support for knowledge-based companies. This component included (a) Venture Capital Fund; (b) support for operating deal flow facilities through basic and promoter fee financing. Part Page 8 The World Bank Unleashing Productive Innovation Project (P106752) (a), the Venture Capital Fund, was cancelled during the 2012 restructuring and replaced with knowledge- based grants for the development of business plans, investment plans, and/or technology prototypes. 15. Component 3: Fostering sector-specific technology capacity for productive innovation. This component focused on supporting sectoral funders under the MINCyT, specifically: (a) technology grants to carry out public-private, public-public, and/or private-private collaborative research and commercialization; and (b) financing of innovation activities by individual firms, primarily in the biotechnology, ICT, and nanotechnology sub-sectors. Part (b) was added during the 2010 restructuring. 16. Component 4: Upgrading of research infrastructure. Under this component, the Project financed (a) infrastructure grants to refurbish research institutions; and (b) construction of a national knowledge center for science, technology, and productive innovation, including refurbishment and remodeling work along with the purchase and installation of equipment. 17. Component 5: Strengthening the policy framework for innovation. Activities under this component focused on: (a) strengthening the capacity of MINCyT for policy making and monitoring and evaluation; (b) support to technology transfer offices at selected universities to improve intellectual property management based on international best practice models; (c) surveys to gather information on national and provincial innovation systems in Argentina; (d) Project dissemination activities; and (e) strengthening the operational capacity of ANPCyT. 18. Project overall. The operation in question was a Specific Investment Loan (SIL) in the amount of US$150 million with an additional financing of US$45 million from the World Bank to the Government of Argentina. Government investments of an additional US$80 million increased the overall Project value from US$195 million (original US$150 million loan with additional financing of US$45 million) to an estimated total of US$275 million across five components. 19. At closing, the Project investments totaled at US$259.23 million relative to the estimated US$275 million estimated as the combination of project preparation and additional financing. This difference is due to: (i) fiscal cuts ordered by Argentina’s Treasury due to macroeconomic crisis, and (ii) devaluation of the Argentinean peso against the US dollar. The following table provides the original estimated allocations for each component at additional financing versus the actual allocation. Page 9 The World Bank Unleashing Productive Innovation Project (P106752) Table 1: Estimated and Actual Costs by Component (US$ million) Total World Bank Gov of Argentina Component / Name Estimated Actual Estimated Actual Estimated Actual C1 - Developing Human Resources for 33.50 48.62 27.50 31.33 6.00 17.29 Productive Innovation C2 - Support for Knowledge-Based 70.00 29.07 40.00 19.41 30.00 9.66 Companies C3 - Sectoral Funds 110.00 80.71 85.00 64.52 25.00 16.19 C4 - Upgrading Research Infrastructure 35.99 71.80 24.99 51.86 11.00 19.94 C5 - Strengthening the Policy and Institutional Framework for Innovation 25.51 29.03 17.51 15.45 8.00 13.58 and Project Management Total 275.00 259.23 195.00 182.57 80.00 76.66 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) 20. The Project was restructured six times in total, in four occasions (2010, 2012, 2014, and 2015), followed by additional financing issued in 2016 and extension of the closing date in 2018. These restructurings resulted in cancellation of the Venture Capital Fund, along with component reallocations, and minor adjustments to the design and Results Framework to add new indicators and increase existing targets. Design adjustments were based on lessons learned during project implementation and identification of new opportunities which could be financed under the loan, in line with the PDO. Table 2 below provides a summary of each of these restructurings, with the paragraphs that follow providing further detail. Table 2: Summary of Project restructurings Year of restructuring Description 2010 Reallocation of US$10 million under Component 3 to be committed to Innovation matching grants due to extra funds (following Peso devaluation) 2012 Cancelation of Component 2.1: Venture Capital, with reallocation to Component 4 (Upgrading Research Infrastructure) 2014 1. Extension of closing date 2. Creation of Component 1c: Technology transfer grants 3. Reallocation of US$10 million from Component 3 (Sectoral funds) to Component 4 (Upgrading Research Infrastructure) 2015 1. Extension of closing date 2. Reallocation of US$10 million from Components 2 and 3 to Components 1 and 4b 2016 1. Additional financing to scale up Components 2, 3, and 5 2. Changes to the Results Framework to create new indicators and increase existing targets 3. Extension of closing date 2018 Extension of closing date Page 10 The World Bank Unleashing Productive Innovation Project (P106752) 21. The 2012 restructuring canceled Component 2.1, Venture Capital: Support to New Knowledge- Based Companies, and reallocated funds under this component (US$10 million) to Component 4 (Upgrading Research Infrastructure). This cancelation of Component 2.1 resulted from difficulties in setting up the Venture Capital Fund, discussed in more detail in subsequent sections of this ICR. 22. Additional financing of US$45 million was approved in May 2016 to scale up support for new technology-based companies, research consortia, SME innovation projects, and impact evaluations based on successful disbursement and continued demand for these Project activities. This additional financing also included a further extension of the closing date and minor changes to project activities with related modifications to the Results Framework. Revisions to the PDO indicators at additional financing are noted earlier in this document. The final restructuring in August 2018 was a final extension of the closing date to September 2019. Other Changes 23. The 2010, 2014, and 2015 restructurings consisted of small component reallocations and inclusion of a new sub-component under Component 1. a. Restructuring in 2010 reallocated US$10 million under Component 3 (Sectoral funds) to support matching grants for innovation projects conducted by individual firms in the biotechnology, nanotechnology, and ICT sectors (also under Component 3), along with minor procurement adjustments. b. The 2014 restructuring consisted of an extension of the original closing date, inclusion of a new sub-component under Component 1 to support technology transfer projects between universities and the local productive sector, and reallocation of US$10 million from Component 3 (Sectoral funds) to Component 4 (Upgrading Research Infrastructure). This reallocation was due to slow disbursement under Component 3 as a result of depreciation of the peso in relation to the dollar; reallocated funds were used to equip the national knowledge center for science, technology, and productive innovation. c. Restructuring in 2015 included another extension of the closing date and reallocation of US$10 million from Components 2 and 3 to Components 1 and 4b. This reallocation was due to slower disbursement under both Component 2 (Support for Knowledge-Based Companies) and Component 3 (Sectoral Funds) since both of these received fewer proposals than anticipated; however, the number of proposals increased following adjustments made by the ANPCyT which led to additional financing of both Components 2 and 3 in 2016. Rationale for Changes and Their Implication on the Original Theory of Change 24. The changes to Project design improved linkages to the Theory of Change with scaling up and creation of new activities based on the demand of potential beneficiaries, such as the innovation grant program. Since the PDO itself was not changed, these changes to project design helped to strengthen the link between activities and the desired outcomes. Page 11 The World Bank Unleashing Productive Innovation Project (P106752) II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDO and Rating 25. The PDO to expand Argentina’s capacity to generate productive innovation in knowledge - based areas remains highly relevant. The PDO was developed to support Argentina’s diversification of sources of growth against its persistent experience with volatile and commodity dependent growth cycles, an issue that is still highly relevant in Argentina. The PDO is consistent with the current World Bank Group’s Country Partnership Framework (CPF) for Argentina (FY 2019-2022), discussed by the Board of Executive Directors on September 9, 2014 (Report No. 81361-AR). CPF Focus Area 1, Supporting Argentina’s access to long-term private financing, specifically references the importance of leveraging private financing for development—a key outcome of this project for both the EMPRETECNO program and Sectoral Funds under Components 2 and 3. Component 5 (Strengthening the policy framework for innovation) also aligns with Objective 2 of the CPF, Fostering stronger market institutions, productivity- led growth, and increased exports, by providing support to strengthen capacity at MINCyT, ANPCyT, and universities to increase commercialization opportunities for innovation. Additionally, Objective 6 of the CPF, Building Skills for the Future, underscores the use of evidence-based public policy making, aligning with the impact evaluations financed under the additional financing, along with the skills support provided by the Project’s investments in human capital and research facilities. 26. The Project focus on leveraging private sector financing aligns with an important lesson from the Country Partnership Strategy FY15-FY18, noting the strategic value of crowding-in private financing given the fiscally constrained environment. The PDO indicator “Attraction of external funding for tech start-ups” demonstrates this Project focus and further emphasizes the relevance of the Project design to the current CPF. 27. The above-noted Additional Financing modified the PDO indicators to better reflect outcomes related to the PDO, and, as such, increased alignment of these indicators with objectives under the CPF. 28. The PDO also remained consistent with the National Strategic Plan for STI 2006-2010, as well as the current Strategic Plan – Argentina Innovadora 2020. Strategic Objective 3, Strengthening innovation and technological transfer to industry and agriculture, emphasizes the importance of supporting innovation activities and carrying out public-private collaborative research. These are key outcomes of Project activities under Components 2 and 3, including the EMPRETECNO program, matching grants for individual firms, and Sectoral Funds. In turn, Strategic Objective 4, Increase in the Scientific Basis and Technological Capacity, references the importance of fostering specialized human capital, the primary outcome of the different programs under Component 1. 29. Government institutions have been strengthened in alignment with the STI policy agenda, underscoring the continued relevance of the activities supported by the Project. For example, the MINCyT has become an important government institution to focus on the economic and social impact of science, innovation and firm support initiatives. The ANPCyT has also become an established player in Argentina’s knowledge-based economy and was recently elevated in status within the government to Page 12 The World Bank Unleashing Productive Innovation Project (P106752) become a more independent agency with budget autonomy outside of the MINCyT1. Additionally, CONICET, the largest and most important public research institution in Argentina, has moved toward more commercialization, including productivity-enhancing firm support—a major shift in focus away from pure siloed research, which was a shift designed and supported by the Project in line with its PDO. 30. The Project was fully appreciated even following a change in government administration, thus safeguarding the stability of STI policies—this policy stability demonstrates the role of micro- interventions to address constraints in the STI ecosystem and provide direct support for firm innovation despite the context of macro-volatility and low levels of private R&D and productivity. 31. Relevance of the PDO is rated as High. B. ACHIEVEMENT OF PDOs (EFFICACY) 32. Although the Project was restructured several times, most of these restructurings consisted of minor component budget reallocations and extension of the closing date; the most significant restructuring entailed cancelation of the Venture Capital Fund. As such, these restructurings do not demonstrate lack of efficacy of the Project—rather they were due to stringent supervision requirements and strict application of the original Legal Agreement. Additionally, these restructurings generally added new indicators/targets or increased existed targets. Also worth noting is that while the following section does discuss the issues encountered relevant to the different areas of the PDO, as a whole the Project was characterized by a strong level of commitment from the government and ANPCyT, along with high levels of achievement across the PDO areas. This government commitment includes US$76 million in counterpart financing; in addition, the ANPCyT raised an additional US$42 million for innovation matching grants and US$18 million in innovation subsidies for research consortia from the IDB to supplement programs initiated under this Project. 33. The Project achieved all of its PDO targets along with fourteen of sixteen intermediate indicators. The following tables summarize results and their achievement relative to their targets, with further detail provided in this section. Since the PDO incorporates several different areas, some of which are best assessed by indicators at the intermediate level, the discussion of efficacy incorporates the most relevant indicators under each PDO area. 34. The Project supported Argentina’s capacity to generate productive innovation in knowledge- based areas, as referenced in the first part of the PDO. Project activities contributed to the STI policy environment on the institutional side, along with providing direct support to address private sector and labor market challenges. Further detail on the different outcomes in the PDO are discussed below. 1 https://www.boletinoficial.gob.ar/detalleAviso/primera/225584/20200217 Page 13 The World Bank Unleashing Productive Innovation Project (P106752) Table 3: Achievement of PDO indicator targets End Final Level of PDO Indicators Baseline2 Target value achievement Number of firms that increased innovation investment and practices Percentage of firms, labs and/or research consortia supported by the project that introduced to existing or new markets new/improved products/protocols/services (Percentage) 0 40 47 Exceeded Number of technology based start-ups that formally register as companies (Number) 0 30 65 Exceeded Attraction of external funding for tech start-ups (Number) 0 10 15 Exceeded Number of public agencies that increased their linkages to STI and the private sector Disclosures made by CONICET researchers (Number) 0 25 25 Fully achieved Number of universities that successfully established norms on intellectual property rights according to international best practices 0 8 18 Exceeded Table 4: Achievement of intermediate indicator targets End Final Level of Intermediate Indicators Baseline3 Target value achievement Developing human capital for productive innovation Increase in number of technology broker and Substantially technology manager graduates (Number) 0 400 375 achieved Number of strengthened or established graduate programs for technology brokers (Number) 0 6 6 Fully achieved Number of technology transfer subprojects (DTEC) that have completed proposed action plan (Number) 0 20 54 Exceeded Training of CONICET researchers in technology transfer and commercialization (Number) 0 150 991 Exceeded Training of CONICET tech transfer personnel abroad (Number) 0 20 43 Exceeded CONICET Tech Transfer Unit (Yes/No) No Yes Yes Fully achieved Support to new knowledge-based companies Number of business plans prepared (Number) 0 45 79 Exceeded Support for Innovation Projects in Consortia and Individual Firms Number of research consortia created for the Project (Number) 0 27 33 Exceeded 2 Project indicators are reported only in reference to Project beneficiaries rather than making an assessment of the STI ecosystem as a whole. This strategy was chosen due to limited data at Project outset, as such most baselines are set at zero. For example, the indicator “Number of technology based start-ups that formally register as companies” applies only to those start- ups which were directly engaged with the ANPCyT team. Similarly, the indicator “Attraction of external funding for tech start- ups” applies only to direct beneficiaries as well. 3 As noted above, project indicators are reported only in reference to direct beneficiaries. For example, since the first graduate technology broker programs were established through the Project, the numbers reported here can be attributed directly to the Project interventions. Page 14 The World Bank Unleashing Productive Innovation Project (P106752) End Final Level of Intermediate Indicators Baseline3 Target value achievement Upgrading research infrastructure Number of upgraded research laboratories and technology centers (Number) 0 30 49 Exceeded Strengthening and informing the policy and institutional framework Satisfactory establishment of the international Not In In advisory committee to the Ministry established operation operation Fully achieved Increase in the number of independent evaluations of STI programs (Number) 0 7 9 Exceeded Percentage of Research Centers who implement tool to monitor and facilitate usage of shared research equipment (Percentage) 0 50 0 Not achieved Number of staff in technology transfer offices at universities trained on intellectual property rights 0 140 203 Exceeded Beneficiary satisfaction surveys implemented and utilized for policy making (Yes/No) No Yes Yes Fully achieved Percentage of beneficiaries who express positive perception on ANPCyT's activities during project implementation (Percentage) 0 70 70 Fully achieved Overall additional investment catalyzed / leveraged through project activities (USD million) 0 32 48.14 Exceeded 35. Impact evaluations of the different programs supported under the Project demonstrate the positive estimated return of the interventions, such as a 30% increase in the probability of creating a Technology-based firm after participation in the EMPRETECNO program and a 19% increase in sales based on participation in the innovation grants program. The following table outlines the results and methodologies of the impact evaluations conducted for several of the programs under the Project.4 Est. impact of program (Avg Component - Program Objective of intervention Methodology Outcome analyzed treatment on treated) Support the formation of tech Diff in Diff with Pr. of increasing skills like tech C1 - GTEC +11.5 % broker and tech manager FE brokers Pr. of being an ICT graduate (or Support for strengthening human Diff in Diff with +24.0 % C1 - ICT Scolarships related career) resources in the ICT sector FE Pr. of income increased +21.0 % Support the creation of PSM + Diff in Pr. of creating a new TBF +29.6 % C2 / EMPRETECNO Technology-Based Firms Diff Pr. of raising private capital +12.8 % R&D +70k annual dollars Support innovation by individual Diff in Diff with C3 / Innovation grants Employment +10.3 % firms FE Sales +19.1 % Innovation intensity per employee +30.9 % PSM + Diff in C3 / Sectoral Funds Support research consortias Employment +13.4 % Diff Sales +9.4 % 4Note: The impact evaluations conducted are included in detail in Innovation in Argentina: Can it increase its contribution to Growth and a Strong Economy?, World Bank, 2019, which was prepared as part of the World Bank ASA “Support for Policy Dialogue and Capacity Building in Innovation (P166964)”. A summary of these evaluations has also been included in Annex 6. Page 15 The World Bank Unleashing Productive Innovation Project (P106752) Outcome 1: “Facilitating the creation of new knowledge-based companies…” 36. The Project contributed to the creation of new knowledge-based companies. EMPRETECNO, one of the Project programs that promoted entrepreneurship, supported 102 sub-projects in total. Among the 77 finalized projects, 90% led to the creation of a Technology-Based Firm (TBF), and more than half of these developed a new product or service, with one third registering sales revenues, obtaining private capital contributions, and hiring staff5. 75% of the TBFs which reported sales revenues were able to reach break-even within their first 2 years of operation and 44% of them were able to export a portion of their production. These statistics are high relative to global standards for new and small firms across both developed and developing countries, and especially within the Argentinian context where many sectors experienced a net decrease in real annual sales of 5-10%. A recent impact evaluation of the program also identified positive results with a quasi-experimental evaluation method, demonstrating an increase in the number of TBFs created as a result of the program (29.6% higher probability of creating a new TBF), along with a higher likelihood of firm survival and improved ability to raise private capital (12.8% higher than without the program). Annexes 6 and 7 discuss the results of the impact evaluation and several case studies of beneficiaries under EMPRETECNO, respectively. Outcome 2: “Fostering specialized human capital” 37. At a high level, the Project contributed to developing human capital for innovation across several metrics. The Project supported six graduate programs for technology brokers under the GTEC program and 54 technology transfer sub-projects. Most of the university beneficiaries are retaining the GTEC program, with interest for continued financing by the private sector.6 The Project also increased the number of technology broker and technology manager graduates, with the final value of this indicator at 375, just under the end target of 400. Additionally, the Project trained 991 CONICET researchers in technology transfer and commercialization, along with 43 abroad, in far excess of the end targets of 150 and 20, respectively. This training focused on commercialization needs, how to engage potential partners, how to support a favorable environment for partnerships, etc., including modules such as evaluation and valuation of technologies, marketing strategies, quality management, etc.7 The Project also financed 4,066 ICT scholarships through the BECAS TIC program (under Component 1), and an impact evaluation confirmed a 21% likelihood of a higher salary for these scholarship recipients.8 38. However, the human capital component (Component 1) of the Project encountered a number of management difficulties, especially related to a lack of commitment from universities. Component 1 includes two scholarship programs for university students: i) “BECAS TIC”, that provides stipends to 5 These figures correspond to a study carried on in June 2019. The rate of creation of TBF has been growing since then. The latest data available indicates 70 new ventures (source: UEAC). This study is referenced and discussed in detail in the World Bank ASA “Support for Policy Dialogue and Capacity Building in Innovation (P166964)”, deliverable FY2020. 6 A full list of the universities planning to retain the GTEC program as of the impact evaluation conducted in 2017 is provided in Annex 8. 7 The full list of courses and seminars include: Evaluation and valuation of technologies, Intellectual Property and competitive intelligence, Marketing strategies and technological marketing, Project management and negotiation strategies, Technological surveillance and competitive strategy, Quality management, Trading strategy on intangibles, Training for technology management, Project evaluation and technological incubation, Marketing strategies and technological marketing, Responsible research and innovation, and Intellectual Property and Technology Transfer. Instructors included a mix of local and international experts. 8 Scholarship recipients were unwilling to reveal precise salary values; hence the impact evaluation could not assess the extent of salary difference for ICT scholarship recipients. Page 16 The World Bank Unleashing Productive Innovation Project (P106752) technology students that have completed at least two-thirds of their studies, and ii) “BECAS Jóvenes Profesionales TIC”, which supports specific ideas and business plans9 submitted by advanced students. The “BECAS TIC” program suffered from management issues that affected its efficacy. Management of the “BECAS TIC” program was the responsibility of FONCYT, one of the four funds under the ANPCyT. This management, however, was limited to receiving university reports and paying out student stipends, while working with scholarship recipients was the responsibility of each university. This resulted in a wide disparity in how non-financial support was provided to these students—some universities were able to do this well, but in most cases, the evidence suggests the opposite. This lack of commitment from universities affected the overall accountability of the BECAS TIC program. For instance, only 15 of the 35 universities that participated in the program provided a list of scholarship students that completed their academics. As such, the ANPCyT was unable to properly evaluate the impact of the BECAS TIC program. 39. In contrast, the ideas / business plans financed under Component 1 helped improve academic performance. FONSOFT, another fund under the ANPCyT, was responsible for each aspect of management of the “BECAS Jóvenes Profesionales TIC”. FONSOFT tracked the academic performance of student beneficiaries, allowing for better monitoring of results. An impact evaluation performed at mid- term found evidence that this program helped to improve student’s academic performance and increase the likelihood that a student was hired by an ICT firm by 24% relative to without the intervention. 40. Overall, ANPCyT encountered some difficulties in managing the breadth and scope of activities under Component 1, especially given the wide dispersion of beneficiaries across the country (also discussed in paragraph 38). Between both the BECAS TIC and BECAS Jovenes Profesionales TIC programs, over 2,400 students received scholarship funds nationally. ANPCyT had no prior experience in management of such a widespread program, which would have been better suited to an agency experienced in education and/or social development programs such as the Secretariat of University Policies (SPU) at the Ministry of Education. The SPU maintains ongoing relationships with universities and their academic management units and has extensive experience in managing national scholarship programs. 41. The Project financed training for CONICET researchers to improve CONICET’s institutional capacity for technology transfer and commercialization, but most of these researchers moved on to better paying jobs in the private sector. The Project was unable to address the lack of incentives within CONICET to retain such skilled workers, hindering the impact of this program and its sustainability. As such, while the Project did support training for 991 CONICET researchers, CONICET’s ability to facilitate technology transfer and commercialization is hampered by this brain drain. That said, this increase in the number of trained researchers would still promote overall research/innovation in the private sector, benefitting Argentina as a whole. Outcome 3: “Supporting productive research innovation projects and their commercialization in three priority technology areas (biotechnology, nanotechnology, and Information and Telecommunication” 42. Project activities facilitated an additional US$60 million in additional private investments to support innovation projects and their commercialization. This figure only accounts for financing raised 9Specifically, an Idea-Plan is a project related to: i) Development of new Products or Systems, ii) Generation of knowledge applicable to a Technological Solution, and iii) Development and innovation of existing products and processes within the field of ICTs. Page 17 The World Bank Unleashing Productive Innovation Project (P106752) immediately after the project interventions, and as such drastically underreports the amount of private investment generated through subsequent rounds of funding. Over half of the firms, labs, and research consortia supported by the Project (88 firms/labs/consortia, or 58%) introduced new or improved products or services to new or existing markets, in excess of the target of 40%. This high percentage of beneficiaries bringing new innovative products to market clearly illustrates the Project’s impact in supporting the commercialization of innovation projects. 43. The Sectoral Funds (FSAT) program under Component 3 supported 35 projects in total. Among the 24 public-private consortiums for which all project support has been completed, about half developed a new product/service and registered sales revenue. Annex 7 outlines several case studies of successful beneficiaries under the FSAT program. Additionally, a recent independent impact evaluation confirmed that the program positively contributed to innovation activities for beneficiary firms. Specifically, innovation per employee was 31% higher than in the absence of the program. The study also found that the FSAT program improved firm performance, increasing growth in terms of both employment (+13.4%) and sales (+9.4%). Details on this study are discussed in Annex 6. 44. The innovation matching grant program for SMEs in biotechnology, nanotechnology, and ICT supported 232 beneficiaries, contributing to an increase in R&D investment. Based on an impact evaluation conducted in 2017, the study confirmed that beneficiaries of the program showed an increase in both specialized employment and total sales, 10.3% and 19.1%, respectively, than they would have in the absence of the program. This study estimates the return on overall project investments under Component 3, including both the Sectoral Funds and innovation grant programs, to be ~26% in increased revenues, netting out the value of the grants provided. Outcome 4: “Upgrading research-infrastructure in the areas of science, technology and productive innovation” 45. The Project supported the upgrading of 49 research laboratories and technology centers, in excess of the end target of 30, providing important equipment and scientific facility upgrades. This Project financing, however, did not address long-term constraints in financing for these labs and technology centers, which could compromise their operations and maintenance. 46. Additionally, Project financing enabled reconstruction and refurbishment of the Polo Cientifico Tecnologico, which houses the ANPCyT and has become a public science, technology and innovation center in Buenos Aires. This center has become a cornerstone for supporting the STI policy agenda and has helped to improve public awareness and understanding of the agenda and its importance. Outcome 5: “Strengthening the policy framework governing science, technology, and productive innovation” 47. Project activities helped to strengthen key STI institutions including the MINCyT, the ANPCyT, and CONICET, along with universities to help improve the environment for commercializing new innovations. For example, the Project supported establishment of the international advisory committee to the MINCyT to provide recommendations and guidance on how to support the STI agenda, along with the tech transfer unit within CONICET. The Project also financed activities to improve transparency and informed decision-making processes such as independent evaluations and beneficiary satisfaction surveys for the ANPCyT. All of the management teams within the ANPCyT associated with the Project Page 18 The World Bank Unleashing Productive Innovation Project (P106752) (EMPRETECNO, FONARSEC, etc.) also remain functional, with well-developed capacity to support firms, universities, and other beneficiaries. 48. Both PDO indicators related to linkages between the public sector, STI, and the private sector met their targets. In fact, the Project target for number of universities that successfully established intellectual property norms in accordance with international best practices was set at eight; however, Project activities led to sixteen universities successfully establishing such norms.10 These norms help establish a more beneficial environment for universities to engage with potential commercial partners on their research, an environment which did not exist previously at these universities. Twenty-five disclosures were also made by CONICET researchers, in accordance with the target of 25.11 49. The Project helped facilitate a cultural shift which places a larger emphasis on productivity- enhancing innovation and its potential impact within the MINCyT, the ANPCyT, and the Argentinian government overall. While this cultural change is difficult to measure, government stakeholders underscored this point and its likely long-term impact on STI in Argentina. The ANPCyT’s recent elevation to a more independent agency reflects this cultural shift within the government, especially in an STI policy environment that has faced pressure with recent fiscal cuts and consolidation. Similarly, the Project’s support helped restore the importance of the MINCyT following budget cuts during the previous administration. Further, the STI Center in Buenos Aires places innovation front and center for the local population, so far entertaining 235,000 visitors since it opened in 2015. Rating of Overall Efficacy 50. Assessment of achievement of the PDO: Substantial. The Project was successful in achieving its PDO by generating productive innovation in knowledge-based areas across the different facets of the PDO as discussed above. All of the PDO indicators covering the creation of and support for technology-based companies, along with the commercialization of R&D, were surpassed. 58% of Project beneficiaries (firms, labs, and research consortia) were able to introduce new/improved products and protocols to the market, and the Project supported the creation of 70 technology-based start-ups.12 The Project also supported 18 universities to establish better norms on international property rights according to international best practices and 25 new disclosures from CONICET researchers. Finally, Project support enabled 15 start-ups to attract external funding, in excess of the target of 10, leveraging over US$60 million in additional investment.13 In addition to the PDO indicators, the Project also reached its targets for all but two of the sixteen intermediate indicators. 10 At the university level, these norms include creation and implementation of a policy for effective intellectual property management and knowledge transfer. This policy helps to provide a structure for and predictability in engaging industry and other types of outside partnerships. As such, these norms create a more beneficial environment for industry and researchers to access and share knowledge, technology, and intellectual property. Worth noting is that these norms are different from Intellectual Property Law—rather these focus on creating this beneficial environment that universities can use to engage potential partners on ongoing research. While universities retained some discretion on how to implement these norms, meeting this indicator was contingent upon these norms being in line with international best practices. 11 CONICET disclosures are a public release of research discovery as part of the process for seeking commercial partners. Essentially, the researcher indicates a development to be potentially patented, requesting advise from other researchers and potential partners. 12 These start-ups were still in operation prior to the COVID crisis. 13 As discussed later in this document, the US$60 million underreports the financing enabled by Project activities since this is limited to immediate capital raises and excludes follow-on financing from the same beneficiaries. Page 19 The World Bank Unleashing Productive Innovation Project (P106752) C. EFFICIENCY Assessment of Efficiency and Rating 51. As part of ICR preparation, we have conducted an economic and financial analysis to estimate the economic rate of return of the Project incorporating actual disbursement figures. Based on an analysis limited to the GTEC, EMPRETECNO and FSAT programs which supported technology transfer graduates, technology based companies and public-private research consortia, respectively, the NPV is estimated at US$81 million at a 15 percent discount rate with an ERR of 72% over the full Project implementation period. This analysis is based on improved cash flows of beneficiaries under these three programs (i.e. higher revenues for EMPRETECNO and FSAT beneficiaries and higher salaries for GTEC beneficiaries) and is discussed in further detail in Annex 4. Since our estimates terminate at the end of Project implementation, actual benefits are likely to be higher as EMPRETECNO and FSAT beneficiaries show continued growth with additional rounds of financing and product development as these businesses grow. The original Project Appraisal Document (PAD) did not calculate an ERR, so a comparison is not possible here. While extensive social benefits under the remaining components (e.g. upgraded public laboratories, improved university norms for intellectual property, etc.) should be strongly noted, the efficiency analysis of this ICR focuses only on these three programs (Components 1, 2 and 3) since the financial benefits in terms of increased revenues and salaries for beneficiary companies and individuals provide a clearer estimate of the Project value. 52. A detailed economic analysis was conducted in 2017 to assess the interim impact of the project, calculating an average economic rate of return of 27%, based on improved cash flows for beneficiaries across the Project components. While this analysis did not include results from the Additional Financing approved in 2016, the 2017 analysis provides a strong indication of the economic impact of the Project activities. The analysis is based on financial flows of Project beneficiaries estimated via meetings with beneficiaries and stakeholders involved in Project implementation. This analysis however excludes Component 5 (Strengthening the policy and institutional framework for STI) due to the difficulty in quantifying impacts of this component. 53. Project activities under Component 5 have strengthened the policy and regulatory environment for innovation, and thus are likely to amplify the impact of Project activities through positive externalities across knowledge-based sectors. 54. The Project did however encounter operational delays associated with the Venture Capital Fund, eventually leading to its cancellation, and difficulties in implementation of the Human Capital and deal flow facilitation activities; that said the Project closed at a disbursement rate of 94% including the Additional Financing. Delays associated with the VCF however were dealt with via restructuring, while deal flow facilitation was undertaken by the ANPCyT. While the EMPRETECNO and FSAT programs did encounter some initial delays due to less interest than expected during the initial calls for proposals, both programs were able to gain traction later in implementation. Additionally, these delays were mitigated by reallocation of some funds to Component 4 to allow for reconstruction of the ANPCyT’s Science and Technology Center in Buenos Aires—an investment which proceeded with disbursement and implementation more quickly. Page 20 The World Bank Unleashing Productive Innovation Project (P106752) 55. Considering the factors described above, Efficiency is rated as Modest. As discussed above, the Project demonstrated strong returns from the EMPRETECNO, FSAT and GTEC programs. The interim economic analysis also shows high returns across Components 1-4. Although these ERR estimates incorporate actual disbursement rates, the long implementation period reduced the Project efficiency, leading to a Modest rating. D. JUSTIFICATION OF OVERALL OUTCOME RATING 56. Considering the rates assigned to Relevance of PDO (High); Efficacy (Substantial) and Efficiency (Modest), the Overall Outcome is rated Moderately Satisfactory. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Institutional Strengthening 57. The operation contributed to building institutional capacity within the ANPCyT. At the onset of the project, this unit required regular guidance from the World Bank team to manage implementation, particularly fiduciary aspects such as procurement and financial management. As implementation continued, however, the unit became more apt at accomplishing responsibilities and actions, along with supporting the Project beneficiaries. All of the different management teams associated with the Project (EMPRETECNO, FONARSEC, etc.) remain functional, with strong capacity to support EMPRETECNO and FSAT beneficiaries especially. This capacity is of tremendous value and should be used to continue to support these programs, along with training future ministry staff. This continued support should also entail institutionalizing project management and financial advisory support that was provided to entrepreneurs. The Socio-Environmental Management Unit (UGSA) has also now been developed into a consolidated unit within the ANPCyT, with permanent staff and a network of external experts to cover the technical, procedural, management, and technical areas as needed. These areas include environmental assessment and management, occupational health and safety, biosafety, legal compliance, experimentation ethics, animal health, and supervision of minor civil works. 58. As noted earlier, the Project also supported a change in the culture around scientific research and innovation; in particular, activities under the Project helped to combat the negative mindset hindering technology transfer of research and partnerships between public research institutions and private partners. In addition, establishment of the Science, Technology, and Innovation Center in Buenos Aires provides a public space to keep this mindset front of mind for the local population. This new center has been able to entertain 235,000 total visitors since opening in 2015, and provides better conditions for innovation than its predecessor, the National Research Council. Mobilizing private sector financing 59. By addressing bottlenecks in product development, financing start-up costs, and providing technical and managerial support, the Project helped mobilize US$48-60 million in immediate direct private financing for beneficiaries—leveraging almost an additional 50% in private investments for these research consortia and individual firms. Specific beneficiaries were able to vastly increase this leverage, for example, one beneficiary was able to generate US$8 million in subsequent private financing following an initial Project injection of US$0.6 million. With secondary financing rounds included, 15 Page 21 The World Bank Unleashing Productive Innovation Project (P106752) companies and 6 PPPs were able to crowd in investments of over US$60 million for less than US$10 million invested by the Project. Given the success of beneficiaries, further investments are expected both for the individual beneficiaries and innovation investments overall. These figures considerably underestimate the Project’s broader impact—each successful beneficiary creates a demonstration effect that amplifies the importance of the STI agenda and helps to grow the different knowledge-based sectors. Poverty Reduction and Shared Prosperity 60. The Project supported shared prosperity by addressing both the demand and supply sides of knowledge-based sectors. On the supply side, Project activities supported training and education for researchers and technology brokers, while on the demand side, the Project enabled the creation of 69 Technology-based firms, supported 35 research consortia, and provided financing through matching- grants to 232 individual firms under the innovation grant program. As such, the Project contributed to growth within the knowledge-based economy, including employment growth for this area. Moreover, by focusing on both economic impact and social objectives, over and beyond project’s direct results, the Project supported poverty reduction and shared prosperity by enabling sustainable private sector solutions across a range of sectors including in agriculture, health and environmental risk management, among others. 61. For example, the DTEC program financed under Component 1 had a particularly important impact on poor communities in the San Juan region, benefiting small-scale agricultural producers and cooperatives. This program financed technology transfer activities including a sub-project to develop molecular markers for certifying the genetic quality in seeds and seedlings14. Researchers under this sub- project worked with two seed varieties sold by the Federation of Agricultural Cooperatives (FECOAGRO) of San Juan. The DTEC program financed training for FECOAGRO producers and technicians, leading to major yield and marketing improvements for products such as onions and pumpkins. These improvements benefited 650 small-scale producers which make up the 30 cooperatives under FECOAGRO, providing them with access to technology that would otherwise have been unattainable. As such, this program, through this one sub-activity alone contributed to improved livelihoods for over 650 farmer households (2,500 beneficiaries) in the San Juan region. 62. Similarly, other Project investments also supported development of new agricultural variants to improve livelihoods within poorer agricultural communities by improving drought tolerance and increasing quality and yields. For example, a publicly financed CONICET technology that resulted in the development of 8 new alpha alpha breeds, was successfully developed for on-field use through a PPP under the Project. This breed enhances drought and salinity tolerance, demonstrating a 20-30% increase in yields in the dairy and cattle industries in trials to date. As part of the commercialization process, the PPP has filed patents in the US, Australia, Paraguay, and Argentina, with others in process, meaning that this technology could have far reaching impacts on agricultural communities around the world and contribute to export revenues. Another similar example is a wine producer supported under the Project who started an Ag-tech company to improve and diversify wine production. Through better traceability and crop management via this company’s innovations, cost-efficiency and incomes for small scale farmers in the region have increased by 20%. 14 This Project being jointly run by the National University of San Juan and CONICET. Page 22 The World Bank Unleashing Productive Innovation Project (P106752) 63. Project financing also supported the development of generic cancer drugs which resulted in a US$54 million annual reduction in public health spending since 2017. One of the Project beneficiaries, a firm focused on developing cancer drugs, developed two new products at a 30% cost reduction relative to the equivalent products on the market and in addition to capturing the market share of the existing treatment, expand the number of total cases treated by 15,000 a year to date. As such, this firm was able to capture a 70% national market share, leading to an additional US$91 million in currency savings a year. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 64. Most of the Project components were designed as a collaboration between the ANPCyT and the World Bank team; however, this process encountered some difficulties. In particular, the design of both the venture capital fund and the deal flow facilitation sub-components under Component 2 were subject to financial management requirements which did not align well with the local context. These difficulties included: a. Process of collaboration: The process of collaboration between the World Bank team and government counterparts encountered some friction due to tight timelines and differing views of how some Project components should be structured. Overall, however, both sides were generally satisfied with the scope of the original design. b. Venture capital fund: The venture capital fund was originally identified as a high-risk component of the Project design. The fund was designed to include both public and private participation; however, the private participants had to be identified as part of Project implementation, along with the hiring of a fund manager who would also contribute private capital. This design required these two steps to occur in tandem, meaning that private participants had to commit capital to the fund before the fund manager had been identified. As such, the design failed to account for the lack of trust amongst potential investors in the Argentine context, meaning that private investors refused to commit capital without knowing who the fund manager would be. c. Deal flow facilitation: During Project design, government counterparts expressed concerns regarding local private capacity for deal flow facilitation. Although implementation for this sub- component was initially planned for private deal facilitators, it was later moved to FONARSEC due to poor implementation by the originally identified facilitators. d. Pipeline of beneficiaries: The Project contended with some delays in identifying and supporting beneficiaries under Component 3; these could have been mitigated had a pipeline of beneficiaries been identified as part of preparation. 65. Although the PDO indicators were generally aligned with the PDO, the PDO indicators at the original design were primarily output-based as opposed to outcome-based. These indicators were revised as part of the additional financing to better reflect the Project’s desired outcomes. 66. The ANPCyT had limited experience in World Bank operations and the resulting fiduciary requirements. Individual firms, universities, and research consortia beneficiaries under the Project were also subject to Bank procurement requirements causing difficulties and delays in implementation since these beneficiaries had no experience in complying with these requirements. While the implementation Page 23 The World Bank Unleashing Productive Innovation Project (P106752) unit did receive training and support from consultants and the World Bank team, this lack of knowledge resulted in initial implementation delays and a steep learning curve. B. KEY FACTORS DURING IMPLEMENTATION 67. Despite some challenges, most Project components led to strong outputs and follow-on achievements, as discussed in-depth in Section II.B. Successful implementation can be attributed to a strong implementation unit within the ANPCyT, which exhibited a problem-solving mentality when they encountered bottlenecks. For example, when faced with inefficiencies associated with Deal Flow Facilitators, the ANPCyT led the scouting of projects using different initiatives, including a large event with entrepreneurs and research team in different provinces, and an advertising campaign with mass media notices. Additionally, since the academic community historically failed to engage with the private sector, the ANPCyT, initiated a dialogue with researchers and CONICET to break down barriers to their participation. Factors subject to client control: 68. The multidiscipline team from ANPCyT in charge of implementation and execution of the Project activities was well-qualified and committed to the Project. The breadth of the Project activities required coordination across a wide number of public and private organizations, including the implementation unit, ANPCyT, MINCyT, CONICET, research institutes, private beneficiaries, and the various fund structures such as FONARSEC, FSAT, etc. 69. Federal budget cuts, difficulties in coordination, and frequent policy shifts caused hindrances to implementation. While the ANPCyT was able to manage the large number of institutions involved, this coordination was difficult and cumbersome, with a steep learning curve over the course of Project implementation. Fiscal consolidation also impacted ANPCyT’s importance within the government—for example, 2017-2018 budget realignments disproportionately affected Science Technology and Innovation (STI) policies, which were reduced from 1.5% of total spending in 2015 to 1.1% in 2018. 70. The ANPCyT encountered difficulties in managing the beneficiaries of the human capital component due to lack of experience in administering scholarship programs, the geographic dispersion of beneficiaries, and difficulties in working with universities. The design of the BECAS TIC and BECAS Jovenas TIC programs under this component planned to work in close collaboration with the SPU, but in practice this collaboration was more difficult than planned. Factors subject to WB control: 71. The fiduciary and technical team which implemented the project was well-qualified, including both internal and external technical expertise. The process of collaboration between the World Bank team and government counterparts functioned well for the most part, despite occasional delays in World Bank approvals. After implementation began, the turnover rate for World Bank staff was low, with appropriate reporting and transition measures. Additionally, the ANPCyT had to provide procurement- related support since beneficiaries (e.g. under EMPRETECNO and FSAT) were required to meet World Bank standards. This support had to be given on a case-by-case basis by the ANPCyT because of the wide range of equipment being procured across the beneficiaries, meaning that these procurement needs could not be addressed through larger-scale workshops or trainings. One particular issue raised with the World Bank Page 24 The World Bank Unleashing Productive Innovation Project (P106752) team, however, was the advice provided to the government around Additional Financing (AF): the AF allowed for only an additional two years of implementation, which the government wanted to extend further. As such, the World Bank team could have considered a follow-on operation as opposed to AF. Factors outside the control of government/implementing entities: 72. Devaluation of the Argentine peso over the course of the Project had serious repercussions on local purchasing power using project funds. In particular, many beneficiaries of the FSAT and innovation grant programs had to reduce equipment purchases since program financing could no longer cover all of their equipment needs. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 73. The original M&E framework established a link between Project activities and the PDO, but needed to be revised to be more outcome-based and assess achievement of the PDO. To this end, the Project revised the Results Framework during the process for Additional Financing in 2016 to put in place more outcome-based indicators at the PDO level. This updated Results Framework provides a better assessment of the PDO and links into the TOC. 74. Even after revision of the indicators at Additional Financing, a number of the indicators remained too broad. For example, the PDO indicator “Percentage of firms, labs, and/or research consortia supported by the project that introduced to existing or new markets new/improved products/protocols/services” incorporates a wide range of aspects, making it more challenging to measure and understand. This indicator could have been separated out by specific parts such as “Percentage of firms supported by the project that introduced new/improved products”. 75. Based on the above, M&E Design is rated Modest. M&E Implementation 76. The implementing agency and the Bank systematically reported on key indicators. The Project implementation unit monitored the progress of the Project and prepared semi-annual Project reports that were shared with the WB. The reports included a summary of the activities performed during the period per each component of the Project, the updated table with the performance of the PDO indicators and Intermediate Results Indicators, along with important activities and actions undertaken during that time period. ANPCyT has developed a much stronger M&E culture related to its activities and is likely to keep collecting information and calculating Project indicators. Additionally, the Project incorporated a number of impact evaluations into the design at Additional Financing to ensure better documentation of impact in addition to the Results Framework. The ANPCyT is also committed to building out the number of impact evaluations conducted as part of its activities to ensure that they both make and can demonstrate high Page 25 The World Bank Unleashing Productive Innovation Project (P106752) quality investments. Further the monitoring system contributed to strategic and operational decision making which led to better credibility for both the MINCyT and the ANPCyT within the government. 77. Based on the above, M&E Implementation is rated High. M&E utilization 78. M&E was used to inform project management and decision-making periodically. The M&E data on performance and results progress periodically reported were used and incorporated in the Implementation Status and Results report (ISR). M&E fed into decisions related to subsequent Project activities. For example, the initial structure of the innovation matching grant program limited financing only to R&D activities, which resulted in a low take-up rate of beneficiaries. Based on discussions with potential beneficiaries, the ANPCyT broadened potential uses of the matching grant to include all innovation-related activities, drastically increasing take-up of the program. Similarly, the FSAT and EMPRETECNO programs dealt with disbursement delays because beneficiaries encountered difficulties in meeting Bank procurement requirements for acquiring equipment; this led to the ANPCyT adjusting the type of support provided to beneficiaries to support them on meeting these fiduciary requirements. 79. Project indicators were monitored against their targets to measure the accomplishment of objectives, but PDO indicators needed to be modified to better capture Project outcomes, and were adjusted at Additional Financing. Moreover, the Project’s M&E utilization initially weighed towards monitoring, as opposed to an economic and financial evaluation. It focused more strongly on evaluation, with some comprehensive analyses of project’s high-level economic impact and implications for the future in the last 2 years of the project. 80. Based on this discussion, M&E Utilization is rated Substantial. Justification of Overall Rating of Quality of M&E 81. The Overall Rating of the Quality of M&E is Substantial. Despite minor shortcomings at design and utilization, the M&E system’s implementation and utilization were overall adequate. The M&E system was able to provide a basis upon which to assess the achievement of the objectives and to test the links in the results chain; and was also reviewed and improved over time. M&E findings were disseminated and used to inform the direction, evolution and perspectives of the project. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Safeguards 82. Given the nature and scale of the Project’s activities related to science and technology and innovation research, no significant or irreversible potential negative impacts were anticipated or arose for the Project. Based on that, the Project triggered the World Bank Environmental Assessment OP/BP 4.01 and was classified as Category B according to this Policy. No adverse impacts or issues that were not previously identified as of potential occurrence appeared during Project implementation; moderate to low risks and non-significant potential impacts associated with sub-projects were successfully managed with the safeguard instruments developed for the Project. As a result of the performance of the UGSA Page 26 The World Bank Unleashing Productive Innovation Project (P106752) and proper application of safeguard instruments, the Project environmental compliance was consistently rated satisfactory through Project implementation. Financial management 83. Financial Management (FM) performance was rated as Satisfactory and the FM risk rating as Moderate, throughout the Project’s life. 84. All of the audit reports under the Original Loan were received with a delay, except for the 2016 audit report. Performance continued to improve after 2016 under the Additional Financing, with all audit reports being received on time. All the audit reports expressed unmodified (i.e. clean) opinions, except for the 2011 audit report, which was qualified due to an amount of AR$ 2.500.000 (USD 568.000 approximately) that was received from the IADB Loan Nº 2180 and erroneously registered as counterpart funds in the Project Financial Statements (this was subsequently corrected). The final audit report covering the FY19 period (January 1, 2019 to December 31, 2019 – 12 months) and FY20 period (January 1, 2020 to January 25, 2020- 1 month) is due by June 30, 2020. 85. All IFRs received during the life of the Project were considered acceptable and with the majority (50%) received by the due date. The Project’s Designated Account is fully documented and there are no instances of ineligible expenditures identified to date. 86. The Borrower’s FM team was well qualified and highly competent and the ANPCyT’s planning, budgeting, accounting, internal controls, funds flow, financial reporting, and auditing arrangements: (a) correctly and completely recorded all transactions and balances relating to the project; (b) facilitated the preparation of regular, timely, and reliable financial reports/statements; (c) safeguarded the project’s assets; and (d) were subject to auditing arrangements acceptable to the Bank, and therefore provided reasonable assurance that the proceeds of the loan were used for the intended purposes. C. BANK PERFORMANCE Quality at Entry 87. The Project design novel and forward-looking, although complex and broad in scope, and identified specific and important entry points to support innovation across a disperse public and private landscape to leverage maximum impact. Support to innovation and R&D was well grounded in the high potential returns of these types of investment: Lederman and Maloney (2003) estimated the return on R&D investment to range 20-40% for OECD countries, with returns for medium-income countries such as Argentina of approximately 55%. The design also included support to a new ministry and aligned incentives for public, private, and academic actors in a manner that had never been done under a Project previously. The following table maps the problem / constraint identified during preparation with the sub- component designed to tackle it. Page 27 The World Bank Unleashing Productive Innovation Project (P106752) Problem / constraint identified Linked Project sub-component during Project preparation Lack of advanced highly-skilled 1a. Graduate grants for technology brokering and management human capital 1c. Technology transfer grants Shortage of ICT advanced human 1b. Scholarships for undergraduates in ICT capital Low private investment in innovation 3a. Technology grants for collaborative research and commercialization 3b. Financing of innovation activities by individual firms Limited capital for commercialization 3a. Technology grants for collaborative research and commercialization of high potential research 3b. Financing of innovation activities by individual firms Lack of innovation financing on the 2a. Venture Capital Fund market (especially for start-ups) 2b. Support for deal flow facilities 3a. Technology grants for collaborative research and commercialization 3b. Financing of innovation activities by individual firms Poor culture of innovation, focused 4b. Construction of national knowledge center for STI only on public sector research 5b. Support to technology transfer offices at universities on intellectual property management 5d. Project dissemination activities Lack of linkages between innovation 3a. Technology grants for collaborative research and commercialization actors Poor incentives for public research 4a. Infrastructure grants to refurbish research institutions and technology centers Lack of public sector capacity for 5a. Strengthening capacity of MINCyT supporting STI policy 5c. Surveys on national / provincial innovation systems 5e. Strengthening operational capacity of ANPCyT 88. The operation was designed based on a collaborative process between the ANPCyT, the overall MINCyT, and the World Bank team, and with a foundation of strong analytics. The design of the operation also benefitted from thorough and rigorous analyses of the growth dynamics and opportunities in the Argentine economy and best practices of similar operations from around the world as well as previous experience in Argentina. As such the operation was able to identify critical levers through which project interventions could achieve maximum impact—specifically addressing the demand and supply constraints to knowledge-based sectors—and supported these with appropriate governance arrangements. While there was occasional friction during this design process, both the government and Page 28 The World Bank Unleashing Productive Innovation Project (P106752) the World Bank were generally satisfied with the original design. Despite these strengths, however, the VCF and Human Capital components had some design flaws. 89. The complexity of the Project design did however result in implementation delays and had to be adjusted during the different restructurings; in particular, the design of the VCF resulted in implementation difficulties which eventually led to its cancellation. The original design could also have benefitted from incorporating a definition for “productive innovation”, which could have helped streamline some of the design complexity. 90. The PDO was broad and multi-faceted to cover both the supply and demand sides of innovation support, and the Project results framework had to be revised at Additional Financing because the PDO indicators at design were output-based rather than outcome based. The original PDO indicators should have been set as outcome-based during preparation and could have more clearly framed the desired outcome and relevant beneficiary group(s). 91. The requirements incorporated into the Component 2 design did not adequately account for local conditions, particularly for the Venture Capital Fund. That said, the Venture Capital Fund (VCF) was seen as risky from the get-go. Especially given this context, however, the intended management structure for this component was rushed rather than carefully planned. The VCF was intended to be managed by a private fund manager who would also be a major investor; however, local private investors were unwilling to commit any financing prior to identification of this fund manager due to a lack of trust. Additionally, the required use of multiple private deal facilitators also under this component did not account for the fact that such deal facilitators were not available in the local market. 92. The design of Component 1 (human capital component) did not adequately account for private sector needs, implementation capacity and institutional requirements. Component 1 was intended to address the quality of labor available for knowledge-based sectors, however the focus on scholarships under the BECAS TIC program put it more in line with a social program rather than a private sector-focused engagement. While activities under this component were incorporated into the PDO, the Project activities did not address the mismatch between university curriculums and the needs of the private sector; the BECAS TIC program provided direct financing for students to complete their studies based on the already existing curriculums. As such, the program design did not really address labor market requirements for private firms in ICT. Moreover, the implementation arrangements agreed upon for the human capital component did not account for ANPCyT’s lack of experience in education programs, especially given the potential geographic spread of student beneficiaries and universities 93. Additionally, the support provided to CONICET under Component 1 helped to create the tech transfer unit to better prioritize productivity-focused activities, but did not address the cultural, financial, and institutional constraints within CONICET. While the creation of the tech transfer unit demonstrates a lasting shift in institutional focus away from pure siloed research, many of the researchers trained under the Project to bolster CONICET’s capacity were later hired by the private sector due to insufficient salaries and the improved but continuing cultural mismatch within CONICET around technology transfer and commercialization. 94. Based on this assessment, Quality at Entry is rated Moderately Satisfactory. Quality of Supervision Page 29 The World Bank Unleashing Productive Innovation Project (P106752) 95. The project was implemented with close supervision by the Bank. Over the course of implementation, the Bank team put together twenty-two Aide Memoires and ISRs following supervision missions which included beneficiary roundtables and site visits. The Bank Team also commissioned ten mid-term evaluations, five impact evaluations, two economic analyses and two case study reports (with 12 deep dive interviews of beneficiaries) to assess the impact and or aide with the course correction of project activities. Additionally, the TTLs maintained close contact with Projects' beneficiaries, through a combination of field visits (at least five throughout the whole country) and roundtable discussions for feedback, which were a consistent part of supervision missions. 96. Supervision proactively identified improvements in Project design and how to maximize the use of resources. Restructurings focused on both new and additional activities, such as creation of the innovation matching grant in the 2010 restructuring, and scaling up well functioning ones including the DTEC technology transfer, EMPRETECNO, and FSAT programs under Components 1, 2, and 3, respectively. The innovation matching grant program created during the 2010 restructuring was developed using resources freed up due to currency devaluation, and these funds were reallocated to this program in line with project objectives. Additional financing also introduced design improvements to finance better data collection and impact evaluations, producing datasets such as the ENDEI 1 and 2, now used across the government. 97. The Bank team supervised Project implementation consistently and with high levels of engagement and was responsive to client needs despite minor delays in project management processes. As part of this process, the WB was proactive in identifying the challenges and risks to the achievement of the PDO and PDO indicators. Lack of movement in the Venture Capital Fund for example later led to restructuring and reallocation of these funds to better functioning components. The ANPCyT was broadly satisfied with the support from and exchange of ideas between the WB team and their staff, and continued to consider the World Bank team as a critical partner. Restructurings and funding reallocations were discussed and agreed to between the ANPCyT and the World Bank team, and better incorporated adjustments to the local context. 98. The results framework was updated as part of additional financing to address the fact that the original PDO indicators were output-based; however, the updated results framework continued to include several multi-faceted indicators which are harder to measure. For example, the PDO indicator “Percentage of firms, labs and/or research consortia supported by the Project that introduced to existing or new markets new/improved products/protocols/service” incorporates many different beneficiaries and product types, making it harder to measure. That said, since this indicator was limited only to direct project beneficiaries, the ANPCyT was able to appropriately measure and report on it. 99. One issue however was some of the impractical fiduciary and procurement requirements placed upon the ANPCyT in some components, which hampered project implementation. For example, procurement thresholds for ex-ante World Bank review were set extremely low, requiring individual non- objections for even small-scale equipment purchases by individual Component 3 beneficiaries, causing difficulties and delays. While the World Bank team did respond to these requests, the project design and implementation could have been better adjusted to accommodate for needs and realities of such interventions, and the Project timeline. 100. Based on this assessment, Quality of Supervision is rated Satisfactory. Page 30 The World Bank Unleashing Productive Innovation Project (P106752) Justification of Overall Rating of Bank Performance 101. Based on the assessment of Quality at Entry and Quality of Supervision, the overall World Bank performance is rated Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 102. The Project focus on private sector solutions with an immediate demand help bolster the credibility, and thus sustainability, of its programs. For example, results from the EMPRETECNO and FSAT programs clearly demonstrate a healthy return on these investments, high firm survival rates, and ability to leverage private financing, with beneficiaries demonstrating strong positive sales growth despite economic turmoil and a net decrease in real annual sales across sectors. The resilience of these outcomes also creates a demonstration effect to help ensure continued support for these programs. 103. Innovation has become more front-of-mind within the Argentinian economy due to Project support and stakeholders are clear on the need for this to continue. As mentioned earlier, both the MINCyT and the ANPCyT have become more established and important players within the government in support of the STI policy agenda, especially with the establishment of the STI Center in Buenos Aires. The recent change in status to support increased independence of the ANPCyT also demonstrates how much the government now prioritizes this agenda. Additionally, the Project supported the upgrading of research laboratories and technology centers which will continue past the life of the Project. Support to CONICET has also helped to improve its capacity, along with the establishment of intellectual property norms at sixteen universities. Moreover, the masters programs created by the GTEC program for technology brokers under Component 1 will be continued by these universities, and with continued financing from the private sector under discussions. Finally, the capacity built within the ANPCyT to manage the different Project components (FSAT, EMPRETECNO, etc.) remains intact, and is becoming increasingly institutionalized with strong capacity to support beneficiaries. 104. However, some policy and macro-level risks are also present. For example, STI-related policy uncertainty can undermine the importance of ANPCyT’s programs. Some of the progress in promoting industry linkages and firm-level productivity are hampered by fiscal consolidation and budget realignment efforts. As mentioned earlier, 2017-2018 budget realignments drastically reduced spending associated with STI policies. Similarly, tax incentives for STI investment have been cut by 64%, with export promotion and entrepreneurship receiving the highest reductions. Since these policy shifts essentially devalue the importance of productive innovation within the context of government policy, this policy environment could hurt ANPCyT’s sustainability. 105. Ongoing difficulties within the macroeconomic environment in Argentina can compromise the development of knowledge-based sectors and programs financed by the Project. Argentina’s economy is crisis-prone, having spent one third of the last 60 years in 14 episodes of recession. As such, the economy and government-supported programs are subject to recurrent boom and bust cycles which are likely to place continued fiscal pressure and reduce financing for programs that support innovation and knowledge-based sectors. 106. The scholarship programs under Component 1 are unlikely to continue in the absence of the Project. While effectiveness of the BECAS TIC program is unclear, the BECAS Jovenes TIC program which Page 31 The World Bank Unleashing Productive Innovation Project (P106752) provided scholarships for specific ideas and business plans improved academic performance and ICT hiring. Without financing through the Project, the activities under this Component are not self-sustaining. 107. The upgraded labs and technology centers financed under Component 4 constitute real assets with continued long-term value; the Project did not however address long-term funding constraints for these institutions, placing the ongoing maintenance and further equipment / other investments at risk. By not incorporating programs for maintenance into the Component design, the Project placed these investments at considerable risk. More sustainable Project activities could have incorporated organizational changes for a smaller number of labs to allow for them to become more financially self- sustaining as opposed to remaining dependent on central or regional government funding. 108. The tool to monitor and facilitate usage of shared research equipment, which was supported by the Project, would have somewhat addressed this funding constraint for research centers had it been fully implemented. Installation and implementation of this tool however remains in process, reducing the longer-term sustainability of these research centers. 109. Despite the focus on improving CONICET capacity through training programs in technology transfer and commercialization, many trained personnel have been lost to the private sector—as such, CONICET’s capacity in these two areas remains limited. Structural and institutional changes would be required to address this concern with CONICET in the long-term. 110. Continued weakness in ANPCyT’s M&E systems, including limited capacity to conduct impact evaluations will constrain their ability to advocate for financing, especially given continued macroeconomic pressure and Argentina’s limited fiscal resources. While the ANPCyT recognizes the need for improved M&E capacity, the inadequate baseline and underlying data will make it more difficult for them to scale up and build on existing programs. The lack of data will also restrict their ability to build consensus and credibility amongst government institutions, affecting the long-term sustainability of ANPCyT’s programs. 111. Based on this assessment, Risks to the development outcome are rated Moderate. V. LESSONS AND RECOMMENDATIONS Lessons for the global agenda to support sustainable growth through STI 112. Strong results of the Project demonstrate the role of innovation-focused micro-interventions in environments that suffer from macro volatility. Despite the macro-uncertainty in Argentina, the Bank was able to support firms and research consortia, and thus mobilize additional private capital. The Project shows that interventions focused on STI can improve local capacity for productive innovation and bolster the economy as a whole. 113. Ensuring a smart and flexible intervention which tackles STI from several different angles is especially important. The performance of this Project indicates that market-based instruments may not function well in a context of macro-volatility—both the Venture Capital Fund and the Deal Flow Facilitation components were closer to the market and did not work well. This is an important lesson to take into consideration for future operations. Macro volatility, capital restrictions and constraints within the overall business environment can be binding constraints that hamper market mechanisms; however, Page 32 The World Bank Unleashing Productive Innovation Project (P106752) the success of the other Project components also demonstrates that other hurdles to the innovation agenda (and broader economic growth) can still be tackled in Bank operations. Lessons for Argentina 114. The Project underscores the importance of the STI policy agenda as a source of long-term growth for Argentina. Despite poor macro conditions and a change of government, the Project was able to support the creation of technology-based firms, improve firm productivity, create skilled jobs, and commercialize new products, including those which helped increase incomes for vulnerable communities. 115. A more regional approach to Project design and implementation could help combat regional capacity discrepancies within Argentina. ICT graduation rates in different Argentinian states vary from zero to 90 percent, and a regional approach would have helped to better roll out the program to support ICT graduates. Calls for proposals for EMPRETECNO and FSAT could also have benefitted from a more regional approach to better account for what sectors are present in specific regions. This approach would also help better address specific technical needs for beneficiaries depending on the resources they are able to access at a regional level. 116. The focus on biotechnology, nanotechnology and ICT was narrow and a broader technology focus would have allowed for faster implementation and a wider response to calls for proposals. The initial calls for proposals for the innovation grant program were limited to these three sub-sectors and responses were limited; this low response rate eventually led to the program’s scope being widened to include all technology sub-sectors. Limiting knowledge-based programs to specific sub-sectors may also exclude important innovations from benefitting from Project activities; as such, future projects focused on supporting knowledge-based companies should consider allowing all types of technology firms to become Project beneficiaries. Institutional Lessons 117. Project activities emphasize the importance of strengthening key institutions within the innovation space alongside private sector focused interventions. These institutions are critical to the sustainability of Project interventions. In this case, the Project supported key institutions within the national innovation system, including the ANPCyT and the MINCyT. Both of these institutions now have more well-developed capacity and higher standing within the government, which will continue to support the STI agenda. Project financing also enabled creation of FONARSEC, the ANPCyT fund to promote technology-based entrepreneurship and collaborative research consortia, and CONICET Tecnologia, the technology transfer unit under CONICET to support commercialization of new research. Most importantly, the performance of the Project was crucial to re-establishing the MINCyT after budget cuts during the previous administration, along with elevating the status of the ANPCyT to a semi-autonomous agency. 118. Projects should incorporate activities which tackle cultural hindrances to the innovation ecosystem. For example, support to rebuild the Science and Technology center in Buenos Aires helps to reshape the mindset around research, especially given the center’s focus on productivity and commercialization. Project communications can also incorporate success stories to support a mindset that values innovation. Page 33 The World Bank Unleashing Productive Innovation Project (P106752) 119. Addressing structural issues at key institutions plays an important role in the success of the overall program. In this case, the Project was likely able to provide longer-term support to the knowledge- based economy by leveraging and/or strengthening key institutions: ANPCyT, MINCyT, and CONICET, and tapping into their existing strengths and assets. However, support to CONICET should have been allocated from the onset of the Project, with institutional reforms to maximize structural changes—for example, by promoting more sustainable financing structures that better exploit CONICET’s participation in productive research. Structural institutional support could also help to address long-term funding constraints for labs and technology centers to allow for better operations and maintenance alongside equipment upgrades. Similarly, Project support to address structural issues for operation of research centers could have enabled full implementation of the tool to monitor and facilitate usage of shared research equipment, alongside financing the tool itself. 120. Sustained government ownership and commitment are critical for successful implementation and depend in part on (a) a sustained, long-term strategy/plan/legislative agenda into which the project fits; (b) strong leadership and capacity for project oversight, management and constant evaluation; and (c) critical and stable institutional/implementation arrangements. In this case, the Project benefitted from leadership continuity at the Minister and technical levels over its 10 years and across two government administrations, along with strong implementation teams set up with appropriate incentive structures. However, it still suffered from policy reversals and unpredictability at a higher level. Better top-down support for a well-developed ANPCyT strategy for technology companies and start-ups, based on a clear understanding of the innovation system, constraints, capacity gaps, etc., and clear articulation of economic impact and financial sustainability would have helped to streamline the direction of Project activities. Lessons for private sector engagement 121. Components requiring private sector participation should incorporate market testing at the design stage to ensure implementation-ready design and fiduciary requirements. During design, both the Bank team and government counterparts recognized the risk associated with the VCF, for example; however, the fiduciary arrangements and requirements for this component were designed in a rush during negotiations rather than carefully planned and market tested. This market testing could have engaged with potential investors during Project preparation to identify binding conditions for investment, incentives, and other mechanisms to encourage participation. Such a process would have allowed for adjustments in the design prior to the start of implementation, most likely mitigating some of the implementation risks and reducing the associated difficulties. 122. Public-private dialogue and communications can play an important role in combatting cultural hurdles—in this case, better publicity, communications, marketing, and public-private dialogue to engage the innovation ecosystem would have amplified Project activities . While the project’s public- private dialogue at the activity level proved strong and effective, its high-level communication and policy advocacy could have benefitted from stronger engagements. Wider PPD would have also helped to increase dispersion of calls for participation in the innovation grant program and could have identified the need to widen the scope of this program beyond biotechnology, nanotechnology, and ICT at an earlier stage. 123. Calls for proposals should better emphasize the potential value of receiving Project support as a signal to the market and incorporate longer-term mentorship to engage wider participation. Page 34 The World Bank Unleashing Productive Innovation Project (P106752) Beneficiaries of the EMPRETECNO and FSAT programs underscored the “signaling value” of their participation in these programs, which could be better incorporated into Project communications to entice larger-scale participation. This point is likely to be of even greater importance as these programs grow and engage start-ups, companies, and consortia at scale. Additionally, previous beneficiaries also expressed a commitment to providing mentorship to future beneficiaries, which could help ensure long- term support for program beneficiaries and increase the value of participation. 124. Now that some of Project activities have gone through market-testing, similar project designs can better incorporate long-term sustainability and scalability through provisions such as royalty-based funding schemes, size and lifecycle-tailored firm support, and asset recycling mechanisms. For example, FSAT and EMPRETECNO beneficiaries supported the possibility and feasibility of royalty-based matching grants in future programs, and also expressed interest in serving as an advisory body and providing in- kind support for the subsequent rounds of beneficiaries of similar programs. These types of features could be incorporated at the design stage in the future, depend on growth performance of the beneficiaries, and over time contribute to revolving funds to bolster sustainability. A delayed royalty-based or limited fee structure could also help to ensure sustainability of programs such as the BECAS Jovanas TIC program which provided scholarships for specific ideas and business plans. Recycling of product development stage R&D assets could also be included to help early stage businesses reduce upfront technology acquisition costs. Stakeholders agreed that the EMPRETECNO and FSAT programs should be continued, while scaling up the number of beneficiaries; other considerations to be incorporated going forward include further developing services to cater to different types and sizes of firms at different growth stages, and introducing measures to increase the programs’ financial and operational sustainability and efficiency. Lessons for Project preparation and implementation 125. The project scope was vast, incorporating five different components to tackle innovation from different angles. Future operations should consider more focused support and simplify the project design to incorporate fewer institutions and implementation agencies, along with simplifying fiduciary requirements to streamline implementation. Procurement arrangements which apply to beneficiaries could also be simplified to reduce their associated burden. The Project scope incorporated major human capital interventions, direct engagement of potential start-ups, government agencies being supported on processes, norms, and the regulatory environment, and private sector beneficiaries of different sizes. Covering such a wide scope was taxing for the ANPCyT, especially given the number of beneficiaries and their lack of experience in working with universities. While the ANPCyT was able to reach the required capacity eventually, complexity in the project design resulted in procurement and FM delays, especially due to stringent requirements placed on beneficiaries’ procurement processes. Future projects could consider how to simplify these procurement requirements for sub-executors such as possible bundling of procurements. The extent of requirements for the VCF also contributed to difficulties in implementation, ultimately leading to its removal during Project restructuring. 126. Complex Project components should be designed with a focus on the implementation arrangements and the pipeline of beneficiaries with an implementation plan and timeframe, based on clear agreement between the Bank team and government counterparts. For example, a better survey on potential fund managers for the Venture Capital Fund at the design stage could have identified the difficulties in attracting private financing without prior identification of the fund manager. Similarly, the ANPCyT identified the lack of qualified deal flow facilitators as an issue in the design of activities under this sub-component, a flaw which could have been mitigated during Project preparation. Page 35 The World Bank Unleashing Productive Innovation Project (P106752) 127. Ensuring a good match between Project activities and the specific implementation agency plays a key role in success of these activities. For example, the SPU at the Ministry of Education could have been better suited to implement the BECAS TIC program under Component 1 than the ANPCyT due to their experience in overseeing scholarship programs around the country. 128. Support for M&E and impact evaluation activities should be given a more central focus. A stronger M&E system would help document program impacts and allow for more timely adjustments as needed to ensure program effectiveness. Additionally, better M&E systems would help improve the ANPCyT and MINCyT’s credibility, thus helping to secure further financing from both within and outside the government. Finally, stronger evaluation, coupled with better communication mechanisms discussed above would support the importance of innovation within Argentinian culture and ensure continued support for the STI agenda from the national government. 129. M&E design at Project preparation should emphasize the importance of gathering baseline data, especially when accurate statistics are limited. Due to limitations in M&E systems and time constraints, this Project was unable to gather baseline data across all its components, leading to difficulties in documenting the impact of Project activities. This also hinders institutional credibility for the ANPCyT and MINCyT, along with the overall STI and innovation agenda. Including baseline studies plays a key role in assessing the impact of interventions and measuring the scale of impact. 130. Flexibility in design should be built into the Project design to allow for some reallocation of funds within and between components without a full restructuring. For example, the 2010 restructuring, which broadened the possible usage of funds under Component 3 to allow for US$10 million to be committed to innovation matching grants under the same component, could have been avoided with a more flexible design in the original the legal agreement. This would have saved time and allowed the Project to proceed more smoothly with implementation. 131. Project preparation should include adequate time to lay out an appropriate design and provide project management training as needed, especially with innovative projects such as these, to maximize implementation readiness, and identify a beneficiary pipeline. At the design stage for example, this entails ensuring a pipeline of interest for required private sector participation—e.g. for potential participants in the Venture Capital Fund and beneficiaries of the innovation grant program. Pressures for project approval, along with inadequate time, limited staff, and lack of sufficient mechanisms to offset limits in capacity, are likely to create delays in project implementation. . Page 36 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Number of firms that increased innovation investment and practices Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of firms, labs Percentage 0.00 40.00 47.00 and/or research consortia supported by the project that 13-May-2016 24-Sep-2019 30-Aug-2019 introduced to existing or new markets new/improved products/protocols/services Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of technology based Number 0.00 30.00 65.00 Page 37 The World Bank Unleashing Productive Innovation Project (P106752) start-ups that formally 17-May-2012 24-Sep-2019 30-Aug-2019 register as companies Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Attraction of External Number 0.00 10.00 15.00 Funding for tech start-ups 13-May-2016 24-Sep-2019 31-Jul-2019 Comments (achievements against targets): Objective/Outcome: Number of public agencies that increased their linkages to STI and the private sector Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Disclosures made by CONICET Number 0.00 25.00 25.00 Researchers 13-May-2016 24-Sep-2019 30-Aug-2019 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 38 The World Bank Unleashing Productive Innovation Project (P106752) Number of universities that Text 0 8 18 successfully established norms on intellectual 31-Dec-2008 31-Jul-2019 31-Jul-2019 property rights according to international best practices Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Developing human capital for productive innovation Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in number of Number 0.00 400.00 375.00 technology broker and technology manager 31-Dec-2008 24-Sep-2019 30-Aug-2019 graduates Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of strenghtened or Number 0.00 6.00 6.00 Page 39 The World Bank Unleashing Productive Innovation Project (P106752) established graduate 31-Dec-2008 24-Sep-2019 25-Feb-2019 programs for technology brokers Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of technology Number 0.00 20.00 54.00 transfer subprojects (DTEC) that have completed 01-Feb-2014 24-Sep-2019 31-Dec-2018 proposed action plan Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Training of CONICET Number 0.00 150.00 991.00 Researchers in technology transfer and 01-Aug-2015 24-Sep-2019 31-Dec-2018 commercialization Comments (achievements against targets): Page 40 The World Bank Unleashing Productive Innovation Project (P106752) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Training of CONICET tech Number 0.00 20.00 43.00 transfer personnel abroad 01-Aug-2015 24-Sep-2019 31-Dec-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion CONICET Tech Transfer Unit Yes/No N Y Y 01-Aug-2015 24-Sep-2019 31-Dec-2017 Comments (achievements against targets): Component: Support to new knowledge-based companies Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of business plans Number 0.00 45.00 79.00 prepared 31-Dec-2008 24-Sep-2019 02-Sep-2019 Comments (achievements against targets): Page 41 The World Bank Unleashing Productive Innovation Project (P106752) Component: Support for Innovation Projects in Consortia and Individual Firms Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of Research Number 0.00 27.00 33.00 Consortia created for the project 31-Dec-2008 24-Sep-2019 30-Aug-2019 Comments (achievements against targets): Component: Upgrading research infrastructure Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of upgraded Number 0.00 30.00 49.00 research laboratories and technology centers 31-Dec-2008 24-Sep-2019 30-Jun-2016 Comments (achievements against targets): Component: Strengthening and informing the policy and institutional framework Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 42 The World Bank Unleashing Productive Innovation Project (P106752) Satisfactory establishment of Text Not Established Committee in Committee in the international advisory Operation Operation committee to the ministry 31-Dec-2008 31-Dec-2018 31-Dec-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increase in the number of Number 0.00 7.00 9.00 independent evaluations of STI programs 31-Dec-2008 24-Sep-2019 31-Dec-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of Research Percentage 0.00 50.00 0.00 Centers who implement tool to monitor and facilitate 14-May-2016 24-Sep-2019 02-Sep-2019 usage of shared research equipment (Of Centers that receive training) Comments (achievements against targets): Page 43 The World Bank Unleashing Productive Innovation Project (P106752) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of staff in Text 0 140 203 technology transfer offices at universities trained on 31-Dec-2008 31-Dec-2018 02-Sep-2019 intellectual property rights Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Beneficiary satisfaction Yes/No N Y Y surveys implemented and utilized for policy making 14-May-2016 24-Sep-2019 30-Jun-2018 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percentage of beneficiaries Percentage 0.00 70.00 70.00 who express positive perception on ANPCyT's 14-May-2016 24-Sep-2019 31-Dec-2018 activities during project Page 44 The World Bank Unleashing Productive Innovation Project (P106752) implementation Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Overall additional Amount(USD) 0.00 32.00 48.14 Investment catalyzed/leveraged through 14-May-2016 24-Sep-2019 02-Sep-2019 project activities (USD million) Comments (achievements against targets): Page 45 The World Bank Unleashing Productive Innovation Project (P106752) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: Facilitating the creation of new knowledge-based companies 1. Number of technology-based start-ups that formally register as Outcome Indicators companies 2. Attraction of External Funding for tech start-ups 1. Number of business plans prepared Intermediate Results Indicators 2. Overall additional Investment catalyzed/leveraged through project activities (USD million) Component 2: Support to new knowledge-based companies: 1. Number of business plans prepared Key Outputs by Component Component 5: Strengthening and informing the policy and (linked to the achievement of the Objective/Outcome 1) institutional framework 1. Overall additional Investment catalyzed/leveraged through project activities (USD million) Objective/Outcome 2: Fostering specialized human capital 1. Disclosures made by CONICET Researchers Outcome Indicators 2. Number of universities that successfully established norms on intellectual property rights according to international best practices 1. Increase in number of technology broker and technology manager graduates 2. Number of strengthened or established graduate programs for Intermediate Results Indicators technology brokers 3. Number of technology transfer subprojects (DTEC) that have completed proposed action plan Page 46 The World Bank Unleashing Productive Innovation Project (P106752) 4. Training of CONICET Researchers in technology transfer and commercialization 5. Establishment of CONICET Tech Transfer Unit Component 1: Developing human capital for productive innovation 1. Increase in number of technology broker and technology manager graduates 2. Number of strengthened or established graduate programs for Key Outputs by Component technology brokers (linked to the achievement of the Objective/Outcome 2) 3. Number of technology transfer subprojects (DTEC) that have completed proposed action plan 4. Training of CONICET Researchers in technology transfer and commercialization 5. Establishment of CONICET Tech Transfer Unit Objective/Outcome 3: Supporting productive research innovation projects and their commercialization in three priority technology areas (biotechnology, nanotechnology, and Information and Telecommunication 1. Percentage of firms, labs and/or research consortia supported by Outcome Indicators the Project that introduced to existing or new markets new/improved products/protocols/service 1. Number of Research Consortia created for the project Intermediate Results Indicators 2. Overall additional Investment catalyzed/leveraged through project activities (USD million) Component 3: Support for Innovation Projects in Consortia and Individual Firms 1. Number of Research Consortia created for the project Key Outputs by Component Component 5: Strengthening and informing the policy and (linked to the achievement of the Objective/Outcome 3) institutional framework 1. Overall additional Investment catalyzed/leveraged through project activities (USD million) Page 47 The World Bank Unleashing Productive Innovation Project (P106752) Objective/Outcome 4: Upgrading research-infrastructure in the areas of science, technology and productive innovation 1. Percentage of firms, labs and/or research consortia supported by Outcome Indicators the Project that introduced to existing or new markets new/improved products/protocols/service 1. Number of upgraded research laboratories and technology centers 2. Percentage of Research Centers who implement tool to monitor Intermediate Results Indicators and facilitate usage of shared research equipment (Of Centers that receive training) Component 4: Upgrading research infrastructure 1. Number of upgraded research laboratories and technology centers Component 5: Strengthening and informing the policy and Key Outputs by Component institutional framework (linked to the achievement of the Objective/Outcome 3) 1. Percentage of Research Centers who implement tool to monitor and facilitate usage of shared research equipment (Of Centers that receive training) Objective/Outcome 5: Strengthening the policy framework governing science, technology, and productive innovation Outcome Indicators 1. Satisfactory establishment of the international advisory committee to the ministry 2. Increase in the number of independent evaluations of STI programs 3. Percentage of Research Centers who implement tool to monitor and facilitate usage of shared research equipment (Of Centers that Intermediate Results Indicators receive training) 4. Number of staff in technology transfer offices at universities trained on intellectual property rights 5. Beneficiary satisfaction surveys implemented and utilized for policy making Page 48 The World Bank Unleashing Productive Innovation Project (P106752) 6. Percentage of beneficiaries who express positive perception on ANPCyT's activities during project implementation 7. Overall additional Investment catalyzed/leveraged through project activities (USD million) Component 5: Strengthening and informing the policy and institutional framework 1. Satisfactory establishment of the international advisory committee to the ministry 2. Increase in the number of independent evaluations of STI programs 3. Percentage of Research Centers who implement tool to monitor and facilitate usage of shared research equipment (Of Centers that Key Outputs by Component receive training) (linked to the achievement of the Objective/Outcome 2) 4. Number of staff in technology transfer offices at universities trained on intellectual property rights 5. Beneficiary satisfaction surveys implemented and utilized for policy making 6. Percentage of beneficiaries who express positive perception on ANPCyT's activities during project implementation 7. Overall additional Investment catalyzed/leveraged through project activities (USD million) Page 49 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Esperanza Lasagabaster Task Team Leader Julio Sanjines Gonzales Procurement Specialist Alejandro Roger Solanot Financial Management Specialist Luz Maria Meyer Financial Management Specialist Natalia Cecilia Bavio Financial Management Specialist Paula Agostino Di Crocco Financial Management Specialist Alejandro Alcala Gerez Team Member Alejandro Caballero Team Member Andrea Patton Team Member Diego Ambasz Team Member Hernando Enrique Daniel Gomez Team Member Jose C. Janeiro Team Member Jose L. Guasch Team Member L. Keta Ruiz Team Member Lilian Pedersen Social Specialist Maria Elizabeth Grandio Team Member Maria Pia Cravero Team Member Michael F. Crawford Team Member Micky O. Ananth Team Member Page 50 The World Bank Unleashing Productive Innovation Project (P106752) Natalia Cecilia Bavio Team Member Qahir Dhanani Team Member Rekha Reddy Team Member Reynaldo F. Pastor Team Member Ricardo Eduardo Lugea Team Member Robert Hodgson Team Member Tanja Gabriele Faller Team Member Vanessa Uchiyama Team Member Yevgeny N. Kuznetsov Team Member Supervision/ICR Tugba Gurcanlar Task Team Leader (ICR) Martin Ariel Sabbatella Procurement Specialist Miguel-Santiago da Silva Oliveira Financial Management Specialist Elba Lydia Gaggero Environmental Specialist Santiago Scialabba Social Specialist Ana Maria Kuschnir Procurement team Ana Paula Cusolito Team Member Ellen Olafsen Team Member Maria Elizabeth Grandio Procurement team Maria Emilia Sparks Procurement team Micky O. Ananth Procurement team Paula Agostina Di Crocco Team Member Paula Marcela Houser Team Member Xavier Cirera Team Member Cristian Quijada Torres Task Team Leader (Supervision) Lily L. Chu Task Team Leader (Supervision) Page 51 The World Bank Unleashing Productive Innovation Project (P106752) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY08 33.374 190,554.00 FY09 18.723 78,302.33 Total 52.10 268,856.33 Supervision/ICR FY09 12.846 80,060.87 FY10 16.983 130,705.12 FY11 15.263 113,575.32 FY12 11.925 95,385.51 FY13 18.438 134,861.18 FY14 21.288 120,293.58 FY15 15.598 100,402.57 FY16 19.553 123,600.41 FY17 12.873 112,116.34 FY18 17.358 148,749.29 FY19 14.063 136,889.97 FY20 8.483 94,196.83 Total 184.67 1,390,836.99 ANNEX 3. PROJECT COST BY COMPONENT Actual at Project Amount at Approval Components Closing (US$, Percentage of Approval (US$, millions) millions) C1 - Developing Human Resources for Productive 33.50 48.62 145% Innovation C2 - Support for Knowledge- 70.00 29.07 42% Based Companies Page 52 The World Bank Unleashing Productive Innovation Project (P106752) C3 - Sectoral Funds 110.00 80.71 73% C4 - Upgrading Research 35.99 71.80 199% Infrastructure C5 - Strengthening the Policy and Institutional Framework 25.51 29.03 114% for Innovation and Project Management Front-end fee Total 275.00 259.23 94% Page 53 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 4. EFFICIENCY ANALYSIS 1. This annex provides an economic and financial analysis for this project as part of the ICR. To the extent possible, this analysis is based on actual data gathered as part of the M&E efforts of the program. Since a detailed economic analysis was conducted in 2017, the updated analysis focuses only on the GTEC, EMPRETECNO, and FSAT programs under Components 1, 2, and 3, respectively. Because the program was extended several times, thus modifying the parameters of the original economic and financial analyses developed at project design and during additional financing, the team has focused on the full period of implementation rather than comparing the current estimates for the program valuation with those generated during the design stage. 2. The total ERR across these two programs is estimated at 76 percent based on improved revenues and salaries for beneficiaries over a 15-year period including Project implementation. The NPV is estimated at US$185 million at a 15 percent discount rate. 3. This valuation estimate is based on the actual number of beneficiaries under the GTEC, EMPRETECNO, and FSAT programs. Given the considerable growth associated with beneficiaries of both programs, we have assumed a growth rate of 30% for business beneficiaries of EMPRETECNO and FSAT extending for five years after the Project support. We assume a 10% increase in salaries and a 15% increase in employment for GTEC beneficiaries. 4. Sensitivity analyses of the valuation estimates noted above are: a. Reducing the assumed growth rate of EMPRETECNO and FSAT beneficiaries from 30% to 20% reduces the ERR to 45% b. Increasing the assumed growth rate of EMPRETECNO and FSAT beneficiaries from 30% to 40% increases the ERR to 99%. c. Reducing the additional salary of GTEC beneficiaries from the assumed 10% to 5% reduces the ERR to 73%. d. Increasing the additional salary of GTEC beneficiaries from the assumed 10% to 15% increases the ERR to 81%. 5. The following table provides disaggregated results from the 2017 analysis by component, which, however, excludes Component 5 (Strengthening the policy and institutional framework for STI) due to the difficulty in quantifying impacts of this component. Net Present Value Internal NPV / Component / Name Rate of ERR: 12% ERR: 8% Investment Return C1 - Developing Human Resources 106.49 170.10 2.51 27% for Productive Innovation C2 - Support for Knowledge-Based 107.64 159.74 1.85 30% Companies Page 54 The World Bank Unleashing Productive Innovation Project (P106752) C3 - Sectoral Funds 59.87 98.62 0.76 22% C4 - Upgrading Research 63.79 90.07 1.77 34% Infrastructure Total 337.79 518.53 1.57 27% 6. Key assumptions. The valuation estimates are based on Project investments supported, with key assumptions as follows: (a) Context: The valuation assumes a sound macroeconomic outlook in the medium to long- term, along with a favorable institutional context reflected in the National Strategic Plan “Argentina 2020” and the absence of crowding-out effects. (b) Component 1: The analysis of Component 1 assumes an increase in firms’ productivity through the engagement with technology brokers and participation in the DTEC program, higher wages for scholarship students, and the creation of new dynamic ventures within the ICT sector. (c) Component 2: The Component 2 analysis is based on higher sales for new technology-based firms and sectoral growth, with an annual assumption of 3% higher growth. (d) Component 3: Analysis of Component 3 assumes higher productivity growth for supported firms and consortiums at +12% and +20% respectively, along with overall sector growth in biotech, nanotech, and ICT. (e) Component 4: The analysis of Component 4 is based on cost savings for lab users because of the services provided through the new research infrastructure, increased revenues from sale of these services, and an increase in the overall supply of research equipment. 7. The literature studying the impact of improved productivity and innovation supports the investments under this project. While the exact additional impact depends on the location and sector, the following studies demonstrate the impact of infrastructure improvements, in support of the key assumptions: (a) Modern growth theory argues that productivity improvements account for half of gross domestic product (GDP) growth (Easterly and Levine 2001). (b) Innovation is the essence of creative destruction that has come to characterize transformational growth (Schumpeter 1934, and later 2018) across the world. (c) Andrews and Criscuolo (2013) identified three stages in the innovation process: new ideas and technologies, their commercialization, and the ensuing dynamic benefits through changes in the reallocation of resources to growing firms. Combined, these three combined can account for as much as 50% of economic growth, depending on the country, the level of economic development and the phase of the economic cycle. Page 55 The World Bank Unleashing Productive Innovation Project (P106752) (d) Recent empirical estimates from Argentina based on ENDEI II survey results from 2630 firms and 27 manufacturing sectors (2014-2016) also find an impact on productivity; Argentine firms that report innovative activities achieve a higher total factor productivity in comparison to those that do not. (Galiani et al. 2019) (e) A recent set of seminal studies of income divergence among countries suggest that technology adoption and investment in innovation play an important role in countries’ ability to move to higher income levels. Comin and colleagues, for example, argue that the differences in the rate of adoption for new technologies drives the magnitude of the “Great Divergence” of incomes between developed and developing economies (Comin and Hobijn 2004; Comin and Ferrer 2013). Maloney and Valencia Caicedo (2017) further suggest that the ability to identify, absorb, and adapt technologies, as represented by the number of engineers per capita, is indeed a key part of the divergence story (see figures below). Accordingly, they show that a country’s capability for innovation in the 1900s drives their income levels today. Figure 2: Capability for innovation in Figure 3: Returns from innovation the 1900s drives income levels Page 56 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Page 57 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 6. Summary of Impact evaluation High Technology Sectorial Fund – Executive summary To carry on the impact evaluation, we considered only those calls for proposals that had already ended. So, we study eighteen consortiums integrated by 50 firms and 30 public actors. According to the administrative registers of FONARSEC, these projects began -on average- during 2012 and ended between 2015 and 2016. The treatment group includes private firms that integrate the beneficiary consortiums. To build the control group we focused on granted firms by the ANR TEC program. As with FSAT, the ANR TEC program was managed during the same period of time and financed projects oriented towards Biotechnology, Nanotechnology and ICT. The main difference between both programs refers to the amount of public support and the type of beneficiary: in the FSAT the beneficiaries were public/private consortiums while in the ANR TEC the beneficiaries are isolated firms. The database is a panel data integrated by 111 firms with 222 observations. Among them, 34 firms correspond to the treatment group and the remaining 77 to the control group. This panel data includes information for two points in time: before and after the program. As regards to the empirical strategy, we use a Difference in Difference (DD) model combined with Propensity Score Matching (PSM). Firstly, we analyze the impact of FSAT program on the innovative effort. Specifically, innovation intensity per employee grew at a more intense rate than the one that would have been registered in the absence of a program. As regards the firms’ R&D intensity, estimated results did not allow us to confirm an effect of the program. These results can be explained by the public/private associativity promoted by the program, and especially by the synergy between firms and research groups. In other words, research groups were responsible for the R&D activities. They used the subsidy to train and incorporate human resources, acquire machinery and improve their infrastructure. In turn, beneficiary firms relied on capabilities developed by the research groups and oversaw other production-related activities. Finally, we estimated the effect of the program on the firm’s performance. We focused on the trajectory of Employment, Sales and the ratio of Sales per Employee. Results confirm that having participated in any of the beneficiary consortiums led to greater growth, both in the employment and sales. On the other hand, we cannot confirm that the program has affected firm’s productivity. However, this result must be taken with caution due to the short time that has passed since the end of the program. Support for knowledge-based companies – Executive summary The EMPRETECNO launched three calls during 2009, 2016 and 2017. A total of 304 projects were submitted and 126 of them were selected to receive financial support. However the last two calls are still in progress and their projects cannot be evaluated for this paper. Therefore only the EMPRETECNO PAEBT 2009 was considered in the impact evaluation. A total of 209 projects were submitted and 82 of them were financed. On average, the financed projects started in 2013 and finalized in 2016. The treatment group consists of a team of entrepreneurs that have obtained a financial support from EMPRETECNO PAEBT. In turn, the control group consists of those teams who have requested but did not Page 58 The World Bank Unleashing Productive Innovation Project (P106752) obtain a subsidy from the program. To evaluate the impact we considered three outcome variables: i) Rate of creation of new TBFs, ii) Age of the TBFs, and iii) Rate of success raising private capital. The database is a panel data made of 209 teams of entrepreneurs and 418 observations. As regards to the empirical strategy, we use a Difference in Difference (DD) model combined with Propensity Score Matching (PSM). The conclusion of this report is that the EMPRETECNO program exerts a relevant role in the argentinean entrepreneurship ecosystem. Estimated results indicates that the PAEBT contributed to a positive change on the demographic indicators of the TBFs. Particularlly, as a consecuence of the program, we can confirm that the creation of new TBFs has increased, were longer-lived and more prone to raise private capital. This last result shows an additional contribution of the program. In a context of asymetric information, the selection of projects made by FONARSEC provided the necessary signaling to stimulate the flow of investments from private venture capitals. Page 59 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 7. Case studies – SECTORAL FUND AND EMPRETECNO - Success Experiences Empretecno Biocodices - R&D for diagnosis and consultancy in medical genomics. Hernán Dopazo is a Biologist specialized in Genetics and Evolution and has a Ph.D. in Biological Sciences. Because of the economic crisis in Argentina in 2001, he immigrated to Spain, only to return to Argentina in 2011. During that time, he worked as a researcher in a biomedical laboratory, specifically in the area of genomics and bioinformatics. This experience allowed him to understand that the creation of Technology-Based Firms is part of the academic career. While he lived in Spain, Argentinean colleagues told him that genomic studies in Argentina were carried out abroad due the absence of local firms. This prompted him to visit different Spanish TBF to understand their business model. In 2011 he applied to enter into the CONICET research career. Additionally, he postulated to the EMPRETECNO program as a member of the Faculty of Exact and Natural Sciences (UBA). The project supported the development of innovative products and services in genomics and bioinformatics for the analysis and manipulation of biological information codices. They use molecular techniques, genomic machinery, bioinformatics tools and statistical concepts. However, due to specific market demand, they have specialized in reproductive medicine. The search for partners took three years (from 2013 to 2015). They attend to several business rounds looking for private investors. It represented a considerable effort for the team because they were all researchers and did not have the means or the capital to self-finance. As a result, they raised one million dollars from private funds. Additionally, these investors have become involved in the TBF and are actively engaged in business development and commercial aspects. The primary obstacles to the development of this project have been those regarding financing of equipment and certain regulatory aspects. Firstly, the local market is reduced since Healthcare Management Organizations do not cover the cost of genetic treatments. Secondly, local regulations to import samples reduce the competitiveness of Argentinean firms: entering a sample into the country costs more than the test. Additionally, taking samples from the country is free and has no restrictions, leaving the company in a vulnerable situation concerning its main competitors in Europe, USA and China. Despite this, the company had success in selling its services in the local market. Currently, they explain about 20% of the market related to tests for reproductive medicine. According to the consortium members, the CONICET seal is an added value to win new customers. Finally, they estimate reaching the breakeven point this year. Empretecno Enercraft - Innovation for the airline industry from Santiago del Estero Eduardo Urquizú is an electronic engineer from the Universidad Nacional del Sur. He has worked for the airline industry for more than 20 years. In this field he has developed products for private and public sector (for instance the Argentinean Air Force). He also integrates a company dedicated to maintaining aircraft and ground support equipment in Santiago del Estero called AV2. Working there, he discovered Page 60 The World Bank Unleashing Productive Innovation Project (P106752) that the equipment which was on the market at that time was not qualified enough. Mainly because of the strong technological backwardness the sector had. In addition to this, existing equipment did not help detecting ground fails that the aircraft could experience during the flight. All this prompted him to dream of developing a small and medium-sized ground support made for aircraft which also works as a monitoring unit. But with the private contribution of AV2 it was not enough to carry out the project. The banking system did not provide the financing that Eduardo needed. The Ministry of Science and Technology from Santiago del Estero told him about the EMPRETECNO program and he decided to apply. Eduardo acknowledges that being selected helped him solve a financing problem. But he also points out that the support structure of FONARSEC helped him to stay focused on the project and accelerate the development process. Last but not least, he says that receiving the subsidy gave him a strong boost by feeling that the country supported him in the development of his idea. From the beginning, he sought to develop a different and competitive product worldwide. For this, the creation of a work team with technological knowledge and experience in the development of products for the airline industry was the key. An electronic engineer who had worked for the French airline industry, and an aeronautical engineer with extensive experience in the local market, joined the project. They managed to develop the PDU, a ground artefact that in addition to fulfilling the power supply function also works as a monitoring and controlling system which can monitor the status of the different systems and devices that integrates the aircraft. There were added two other equipment. The ultra- starter, a starter for turbines that for it is light, portable and adaptability to low temperatures is indicated to improve the engine start in helicopters and airplanes. And the GPU, best known as auxiliary units that can provide energy to airplanes while them are on the ground. ENERCRAFT was created in 2017 (www.enercraft.com.ar). Meant to compete in the international market, they sought to partner with companies from other countries and produce their equipment there. Thus they founded ENERCRAFT Canada Inc. (www.enercraft.ca), and currently there are in negotiations with another company in Europe. A short time after starting activities, they received requests from Mexico, the army of Chile, Bolivia, Uruguay and Europe. In the future, the main challenge is to develop its own sales team. Exports achieved so far were thanks to the participation in international exhibitions and events. They are currently among the 18 finalists of ARGENTINA CHALLENGE. It is one of the most important events of technology, creativity and entrepreneurship. There they will present their projects to potential investors, from the private and public sector and also VIP guests. FSAT MAGenTA - First steps in precision medicine to detect and treat tumors in Argentina The Map of Actionable Genome in Tumours from Argentina (MAGenTA, the Spanish acronym for its name) was built as an initiative of the Italian Hospital of Buenos Aires (HIBA for its acronyms in Spanish) and the Multidisciplinary Institute of Cell Biology (IMBICE–CONICET for its acronyms in Spanish). During 2017, a professional team was formed aiming at applying precision medicine to diagnose tumours. To develop the project, it was crucial to replace the old sequencer that HIBA had. With the financing of FONARSEC, a cost-effective device was purchased, in line with the volume of HIBA patients. During 2019 Page 61 The World Bank Unleashing Productive Innovation Project (P106752) a second generation sequencer started operating, which can obtain information from multiple genes in the same assay. This new equipment, along with the efforts of the research group, allowed the creation of a sequencing platform. This platform is very versatile as it can work on a large group of tumours and provides information about 52 clinically relevant genes. Many of these genes are accompanied by a specific drug to treat them. Hence, the treatment the patient receives is more precise than the traditional one. Today, a "one-to-one" detection method is employed. That is, lung cancer is looking for a single gene. Thus, the treatment is much more generic. Evidence shows that precision medicine significantly improves the survival rate in cancer patients. Thanks to the project, the consortium develops a map of actionable tumour genomic on adult subjects diagnosed with solid tumours in Argentina. Tumour frequency data and their molecular alterations were generated and made available to the medical community (Biocomputer data system: https://magenta.hospitalitaliano.org.ar/cbioportal/index) The sequencing platform has already passed the validation stage and is all ready to start the production stage and detect the presence of tumours in patients. The Commercial Management of HIBA is negotiating with different Healthcare Management Organizations to offer this new method of detection to its associates. The adoption of new diagnostic methods is always slow. However, considering that precision medicine is a reality in the developed world, there is room to be optimistic. Considering the HIBA patient population, the platform could test the presence of different tumours in up to 20 patients per week. But these figures would be greater since the platform would allow working with patients from other institutions. This new method of detection will provide significant foreign exchange savings to the nation. Today, when Healthcare Management Organizations authorizes this test, it must be ordered abroad for a cost close to US$4000. Consortium members estimate that they can perform this diagnosis for a quarter of that value. Consortium members have held several conferences on precision medicine, and two initiatives emerged that promise to increase the impact of the project. On the one hand, the medical community has expressed interest in receive training on precision medicine. Today, HIBA is offering a subject for its Post-graduate Programme and is preparing a PhD in medicine precision. Thus, the projects help to build capabilities on precision medicine. On the other hand, these conferences allowed other public health institutions to know the platform. Garrahan Hospital offer to put their paediatric patient data into the platform. This way, the platform will be expanded to other patients of the public health system. FSAT Mabxience - Development of Monoclonal Antibodies for therapeutic use. The private companies involved are part of the Insud Group. This group has different characteristics in terms of the role assigned to R&D in generating value for its products. Regarding this project, the group was interested in developing an alternative biological product to those already existed in the market called biosimilar. The intention was to compete against imported products, for which there was only one supplier in the market. To do so, a technological platform had to be developed that could allow the development of a range of biosimilar products. Page 62 The World Bank Unleashing Productive Innovation Project (P106752) Although the project began before FONARSEC was officially convened, the existence of this instrument accelerated the process. Despite the fact that FONARSEC's economic support represented a relatively smaller percentage of the total investment, participation in the High Technology Sectorial Fund gave it recognition in terms of quality and hierarchy that reinforced the work plan. The actors who participated in the project already had a previous link. In the case of private firms that belonged to the same investment group; in the case of public institutions from the scientific and health systems, they had already collaborated on other projects for which they knew each other. In addition, they looked for complementary actors in the different expertise needed for the project. So far, two products and several pipeline products were developed. The first allowed to assemble the platform and to acquire the needed complementary capabilities. The first was Rituximab, a monoclonal antibody for the treatment of Non-Hodgkin Lymphoma and Rheumatoid Arthritis. The second product developed was Bevacizumab, a monoclonal antibody for the treatment of lung cancer and colorectal metastatic cancer. The first one took about six years to commercialize since launched in 2015. However, concerning the second, it managed to reduce those times by half and was launched in 2017. At present, both products have a market share higher than 70% in the local market and exports to more than eight countries. Although in their projections, they plan to triple the export destinations. Within these objectives are the European and U.S. markets. This project presents an important impact on the Argentine STI system. Employment in the company grew from 12 initial people to 160 at present, and with projections to continue increasing the endowment, the price of products offered is 30% lower than the original product, which represents a significant savings for health systems. The project promoted the private investments in infrastructure for more than 35 million dollars. It was built a new plant that increases production capacity by eight times over the previous one. Page 63 The World Bank Unleashing Productive Innovation Project (P106752) ANNEX 8. Universities retaining the GTEC program Page 64 The World Bank Unleashing Productive Innovation Project (P106752) Project Name Universities Code GTEC 09 Programa Litoral de Formación de Universidad Nacional del Litoral Gerentes y Vinculadores Universidad Nacional de Gral. Sarmiento Tecnológicos. Universidad Autónoma de Entre Ríos UTN Paraná UTN Rafaela UTN Santa Fe UTN Reconquista Universidad Nacional de Córdoba GTEC 11 Programa de formación en recursos Universidad Nacional de Rosario humanos e infraestructura de apoyo Universidad Nacional del Nordeste de la Pcia. de Bs. As en gestión de la innovación y la UTN San Nicolás vinculación tecnológica UTN Rosario UTN Venado Tuerto Universidad Nacional de Entre Ríos GTEC 10 Programa de Formación de Recursos Universidad Nacional de Tucumán Humanos en Gestión y Vinculación Universidad Nacional de Catamarca Tecnológica del Norte - UNT Universidad Nacional de Jujuy Universidad Nacional de Misiones Universidad Nacional del Nordeste Universidad Nacional de Salta Universidad Nacional de Santiago del Estero GTEC 06 Programa de Formación de Gerentes Universidad Nacional de Río Negro y Vinculadores Tecnológicos de la Universidad Nacional de la Patagonia Austral Patagonia. Universidad Nacional de la Patagonia San Juan Bosco UTN Bahía Blanca UTN Río Grande UTN Chubut Page 65