Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) Report Number: ICRR0022133 1. Project Data Project ID Project Name P106752 AR Unleashing Productive Innovation Country Practice Area(Lead) Argentina Finance, Competitiveness and Innovation L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-75990,IBRD-86340 30-Sep-2014 182,830,825.84 Bank Approval Date Closing Date (Actual) 25-Sep-2008 25-Sep-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 150,000,000.00 0.00 Revised Commitment 182,830,825.84 0.00 Actual 182,830,825.84 0.00 Prepared by Reviewed by ICR Review Coordinator Group Nestor Ntungwanayo John R. Eriksson Christopher David Nelson IEGFP (Unit 3) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As per the Loan Agreement (LA) on p. 6, "the objective of the Project is to expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by: (a) facilitating the creation of new knowledge- based companies; (b) developing specialized human capital; (c) supporting productive research innovation activities and their technology commercialization in the areas of biotechnology, nanotechnology and information and communication technology; (d) upgrading the research infrastructure in the areas of science, Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) technology and productive innovation; and (e) strengthening the policy framework governing science, technology, and productive innovation." The statement of the project development objective (PDO) in the project appraisal document (PAD) on page 8 is identical to the one in the LA and was unchanged until the project closing date. This review will assess the project performance by parsing the overarching objective into the five sub-objectives as follows: 1. To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by facilitating the creation of new knowledge-based companies; 2. To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by developing specialized human capital; 3. To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by supporting productive research innovation activities and their technology commercialization in the areas of biotechnology, nanotechnology and information and communication technology; 4. To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by upgrading the research infrastructure in the areas of science, technology and productive innovation; and 5. To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by strengthening the policy framework governing science, technology, and productive innovation. Parsing the overarching objective into sub-objectives will allow the review to unpack the specific project outcomes, and to define the theory of change underlying the achievement of the outcomes under each sub- objective. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The original and revised components are described below based on the 2010 Loan Agreement (LA) on p.1- 2, the 2012 Amendment to LA on p.4-6, the 2014 Amendment to LA on p.2-3, and the ICR on p. 8-10. Component 1: Creating human capital for productive innovation: (cost at appraisal of US$27.50million from the World bank, and US$6.0 million from the Government; actual cost of US$31.33 million from the World Bank, and US$ 17.29 million from the Government). Under this component, the Project aimed to support (a) graduate program grants to establish and strengthen programs related to technology brokering and management, thus supporting a combination of a scientific background with business knowledge to commercialize technological ideas; (b) scholarship provisions for undergraduate students in ICT programs; and (c) technology transfer grants to solve technology-related problems and Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) capitalize on local and regional socio-economic development opportunities. Parts (a) and (b) were part of the original project design, while part (c) was added during the 2014 restructuring. Component 2: Support for knowledge-based companies: (Cost at appraisal of US$ 40.00 million from the World bank, and US$30.00 million from the Government; Actual cost of US$19.41 million from the World Bank, and US$9.66 million from the Government). This component included (a) a Venture Capital Fund; (b) support for operating deal flow facilities through basic and promoter fee financing. The sub-component (a) -the Venture Capital Fund-, was cancelled during the 2012 restructuring and replaced with knowledge-based grants for the development of business plans, investment plans, and/or technology prototypes. Component 3: Fostering sector-specific technology capacity for productive innovation: (Cost at appraisal of US$85.00 million from the World bank, and US$25.00 million from the Government; Actual cost of US$64.52 million from the World Bank, and US$16.19 million from the Government). This component focused on supporting sectoral funders under the Ministry of Science, Technology, and Productive Innovation (MINCyT), specifically: (a) technology grants to carry out public- private, public-public, and/or private-private collaborative research and commercialization; and (b) financing of innovation activities by individual firms, primarily in the biotechnology, ICT, and nanotechnology sub- sectors. Part (b) was added during the 2010 restructuring. Component 4: Upgrading of research infrastructure: (Cost at appraisal of US$24.99 million from the World Bank and US$11.00 million from the Government; Actual cost of US$51.86 million from the World Bank, and US$19.94 million from the Government). Under this component, the Project financed (a) infrastructure grants to refurbish research institutions; and (b) construction of a national knowledge center for science, technology, and productive innovation, including refurbishment and remodeling work along with the purchase and installation of equipment. Component 5: Strengthening the policy framework for innovation. (Cost at appraisal of US$17.51 million from the World Bank, and US$8.00 million from the Government; Actual cost of US$15.45 million from the World Bank, and US$13.58 million from the Government). Activities under this component focused on: (a) strengthening the capacity of MINCyT for policy making and monitoring and evaluation; (b) support to technology transfer offices at selected universities to improve intellectual property management based on international best practice models; (c) surveys to gather information on national and provincial innovation systems in Argentina; (d) Project dissemination activities; and (e) strengthening the operational capacity of the Agency for the Promotion of Science and Technology (ANPCyT). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost: The actual project's cost totaled US$259.23 million or 94 percent of the US$275 million, which was the the combination of (i) the original World Bank project amount, (ii) the World Bank additional financing, and (iii) the Borrower contribution. Respective actual cost was US$182.57 million by the World Bank, and US$76.66 million by the Borrower. The difference between the estimated and actual cost arises from fiscal Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) cuts ordered by Argentina’s Treasury due to macroeconomic crisis, and the devaluation of the Argentinean peso against the US dollar. Financing: The project financing was from a Specific Investment Loan (SIL) in the amount of US$150 million, with an additional financing of US$45 million from the World Bank to the Government of Argentina. Borrower Contribution: Government investments at appraisal amounted to US$80 million, out of which an mount of US$76.66 million was disbursed. Dates: The project was approved on September 25th, 2008, and made effective August 21, 2009. The project underwent six level-2 restructuring operations as follows: (i) on December 09, 2009 and January 24th, 2012 to make changes in components and costs, (ii) on February 21, 2014, and July 07, 2015 to make change in results framework, in components and cost, in loan closing dates, and in reallocation between disbursement categories, (iii) on June 13, 2016 to provide additional financing, and make change in the results framework, and the loan closing date, and finally (iv) on August 22, 2018 to make the last change in the loan closing date until September 25, 2019. The project was closed on the latest closing date, five years after the original closing date. 3. Relevance of Objectives Rationale The PDO was consistent with the country's priorities until the closing date (ICR para 28). At appraisal, the PDO was in line with the National Strategic Plan for Science Technology and Innovation (STI) 2006-2010, and remained consistent with the current Strategic Plan – Argentina Innovadora 2020. The third strategic objective aimed at "strengthening innovation and technological transfer to industry and agriculture" emphasizes the importance of supporting innovation activities and carrying out public- private collaborative research. This objective is in synchrony with the outcomes generated and expected from the project's activities related to the EMPRETECNO program, the matching grants for individual firms, and the sectoral funds. Similarly, the fourth strategic objective aimed at "increasing the Scientific Basis and Technological Capacity, emphasizes the importance of fostering specialized human capital, the primary outcome of the different programs under Component 1. The PDO was consistent with the World Bank's strategies in the country throughout the period of project implementation (ICR para 25&26). The Project focus on leveraging private sector financing aligns with an important lesson from the Country Partnership Strategy FY15-FY18, noting the strategic value of crowding-in private financing given the fiscally constrained environment. The PDO indicator “Attraction of external funding for tech start-ups” demonstrates this Project focus and further emphasizes the relevance of the Project design to the current CPF. The PDO was in coherence with the current World Bank Group’s Country Partnership Framework (CPF) for Argentina (FY 2019-2022), (Report No. 81361-AR). The first CPF focus area aimed to "supporting Argentina’s access to long-term private financing". It specifically referenced the importance of leveraging private financing for development, which was a key outcome expected from both the EMPRETECNO program and Sectoral Funds. The second objective of the CPF aimed at "fostering stronger market institutions, productivity-led growth, and increased exports" was also consistent with the goal of the project's component 5 aimed at strengthening the policy framework for innovation. Finally, the sixth CPF objective of the CPF aimed at building skills for the future, underscoring Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) the use of evidence-based public policy making, and aligning with the impact evaluations financed under the additional financing, and the skills arising from project's investments in human capital and research facilities. Because the PDOs were realistic and consistent with the country's priorities and the World Bank's partnership strategies with Argentina, and given the comprehensiveness of the supported program, the relevance of objectives is rated as high. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by facilitating the creation of new knowledge-based companies. Rationale Theory of change The constructed results chain (ICR, Figure 1: Theory of Change) toward the above objective was underpinned by a set of activities aimed at providing (i) financial support for business and investment plans and/or technology prototypes, (ii) deal flow facilitation for new STI business ideas; (iii) technology grants for research consortia to commercialize technology; and (iii) innovation grants for individual firms. The above activities were to help firms to prepare business plans, create consortia, and attract external funding for tech start-ups. Expected immediate and long-term outcomes included (i) upgraded firms, labs, and research consortia that innovate, (ii) improved firm productivity, (iii) increased private sector investment, (iv) better conditions for innovation and improved access to technology, and (v) an expanded local capacity to generate productive innovation in knowledge-based areas. The constructed theory of change did not discuss the assumptions underlying the achievement and sustainability of outcomes, but this review opines that for the outcomes to be achieved and sustained under each sub-objective, the following policies should be in place or continued: (i) improved macroeconomic stability helping the generation of sufficient fiscal space to fund labs and technology centers, and (ii) salary incentives to researchers and experts in productive innovation in order to resist attraction from the private sector. Outputs Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752)  The target for prepared business plans was exceeded, reaching the number of 79 against of a target of 45;  The performance toward attracting external funding for tech start-ups was exceeded, as external funding was mobilized in favor of 15 firms, against a target of 10;  The target for the number of consortia created was exceeded, reaching 33, against a target of 27;  The target for the number of public agencies that increased their linkages to Science Technology and Innovation (STI) and the private sector was met as disclosures made by CONICET researchers reached the target of 25. Outcomes The salient results reported in the ICR (para 36) are as follows:  The target of firms, labs and/or research consortia supported by the project that introduced to existing or new markets, new and improved products, protocols and services was exceeded, reaching 47 against a target of 40;  The target of number of technology-based start-ups that formally registered as companies was largely exceeded, reaching 65, against a target of 35;  A program initiated by the project (EMPRETECNO) completed 77 sub-projects, out of which 90% led to the creation of a Technology-Based Firm (TBF), and more than half of them developed a new product or service, with one third registering sales revenues, obtaining private capital contributions, and hiring staff;  75% of the TBFs which reported sales revenues were able to reach break-even within their first 2 years of operation and 44% of them were able to export a portion of their production;  A recent impact evaluation of the above program showed an increase in the number of TBFs created as a result of the program (29.6% higher probability of creating a new TBF), along with a higher likelihood of firm survival and improved ability to raise private capital (12.8% higher than without the program); and  The overall additional investment catalyzed and leveraged through project activities was exceeded, reaching the amount of US$48.14 million, against a target of US$32.00 million. Rating High OBJECTIVE 2 Objective To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by developing specialized human capital; Rationale Theory of change The constructed theory of change (ICR, Figure 1. Theory of Change) toward the above objective relied on a set of project activities aimed at (i) launching postgraduate programs in technology brokering and Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) management; (ii) developing curricula and scholarships; (iii) providing ICT scholarships; and (iv) supporting CONICET on tech transfer/commercialization. The above activities were to generate the following outputs: (i) enhanced graduate programs for technology brokering, (ii) increased number of graduates in tech transfer and ICT careers, (iii) tech transfer subprojects with action plan, (iv) the creation of CONICET tech transfer unit, and (v) a higher number of CONICET’s researchers and personnel trained in tech transfer/commercialization. Outputs  The target of increased number of technology broker and technology manager graduates was partially achieved reaching 375, against a target of 400;  The target of 6 strengthened or established graduate programs for technology brokers was fully achieved;  The target of technology transfer subprojects (DTEC) that have completed their proposed action plan was exceeded, reaching the number of 54, against a target of 20;  The target of trained CONICET researchers in technology transfer and commercialization was largely exceeded, reaching the number of 991, against a target of 150;  The target of trained CONICET tech transfer personnel abroad was exceeded, totaling the number of 43, against a target of 20;  The target of setting up and strengthening the CONICET Tech Transfer Unit was fully achieved. Outcomes  The project financed 4,066 ICT scholarships through the BECAS TIC program, and an impact evaluation confirmed a 21% likelihood of a higher salary for these scholarship recipients;  Business plans financed by the project helped improve academic performance. A Trust Fund for Promotion of the Software Industry (FONSOFT) under the Agency for the Promotion of Science and Technology (ANPCyT) tracked the academic performance of beneficiary students allowing for better monitoring of results. An impact evaluation performed at midterm found evidence that this program helped to improve student’s academic performance and increase the likelihood that a student was hired by an ICT firm by 24% relative to without the intervention; However, there were shortcomings (para 38&40) in achieving some of the expected outcomes as detailed below: (i) the commitment from universities was proportionate to their ability to provide in-house support, and this affected the overall accountability of the BECAS TIC program, as only 15 of the 35 universities that participated in the program provided a list of scholarship students that completed their academics; (ii) ANPCyT did neither have nor acquire the expertise needed to manage the breadth and scope of identified activities, given the wide dispersion of beneficiaries across the country and the lack of prior experience in management of a widespread program better suited to an agency experienced in education and/or social development programs; and finally (iii) most of the National Scientific and Technical Research Council (CONICET) trained researchers were expected to improve CONICET’s institutional capacity for technology transfer and commercialization, but moved on to better paying jobs in the private sector, as insufficient incentives within CONICET to retain skilled workers hindered the impact of this program and its sustainability. Rating Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) Modest OBJECTIVE 3 Objective To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by supporting productive research innovation activities and their technology commercialization in the areas of biotechnology, nanotechnology and information and communication technology. Rationale Theory of change The constructed theory of change (ICR, Figure 1. Theory of Change) toward the above objective was underpinned by a set of activities geared to: (i) strengthening MINCyT and ANPCyT capacity, (ii) supporting to tech transfer offices at universities best practices for Intellectual Property Rights (IPR), and (iii) analysis of national innovation systems and project dissemination. The above activities were to trigger the creation of upgraded research laboratories and technology centers, the launching of international advisory committees, the completion of independent evaluations/studies, and beneficiary satisfaction surveys, and an increased number of staff at universities trained on intellectual property rights. The expected outcomes included the introduction of of new or improved products or services to new or existing markets, improved firm performance, increasing growth in terms of both employment and sales. Outputs  The innovation matching grant program for small and medium enterprises (SMEs) in biotechnology, nanotechnology, and ICT supported 232 beneficiaries, contributing to an increase in R&D investment;  The target of upgraded research laboratories and technology centers was exceeded, reaching the number of 49, against a target of 30; and  The target of number of staff in technology transfer offices at universities trained on intellectual property rights was exceeded by 45 percent, reaching the number of 203, against a target of 140. Outcomes  Over half of the firms, labs, and research consortia supported by the Project (88 firms/labs/consortia, or 58%) introduced new or improved products or services to new or existing markets, in excess of the target of 40%. This high percentage of beneficiaries bringing new innovative products to market clearly illustrates the Project’s impact in supporting the commercialization of innovation projects;  The Sectoral Funds (FSAT) program supported 35 projects in total. Among the 24 public-private consortiums for which all project support has been completed, about half developed a new product/service and registered sales revenue;  A recent independent impact evaluation (Annex 7 of the ICR) confirmed that the program positively contributed to innovation activities for beneficiary firms. Specifically, innovation per employee was 31% higher than in the absence of the program. The study also found that the FSAT program improved firm performance, increasing growth in terms of both employment (+13.4%) and sales(+9.4%);  An impact evaluation conducted in 2017 confirmed that beneficiaries of the above program showed an increase in both specialized employment and total sales, 10.3% and 19.1%, respectively, than they Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) would have in the absence of the program. This study estimates the return on overall project investments, including both the Sectoral Funds and innovation grant programs, to be about 26% in increased revenues, netting out the value of the grants provided;  The above evaluation found that the project implementation led to an increased probability of (i) skills like tech brokers (+11.5 %), (ii) being an ICT graduate or related career (+24.0 %), (iii) increased income (+21.0 %); (iv) innovation intensity per employee (+30.9 %), (v) employment (+13.4 %), and (vi) Sales (+9.4 %); and  Finally, the percentage target of beneficiaries (70 percent) who expressed a positive perception of ANPCyT's activities during project was fully achieved. Rating Substantial OBJECTIVE 4 Objective To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by upgrading the research infrastructure in the areas of science, technology and productive innovation. Rationale Theory of change The constructed theory of change (ICR, Figure 1. Theory of Change) was articulated around the goal of upgrading the research infrastructure in the areas of science, technology and productive innovation. Key activities were the supply of infrastructure grants to refurbish research institutions; and the construction of a national knowledge center for science, technology, and productive innovation, including refurbishment and remodeling work along with the purchase and installation of equipment. Expected outputs and outcomes included (i) upgraded research laboratories and technology centers, (ii) trained staff in technology transfer, and (iii) the establishment of norms on intellectual property rights according to international best practices. Outputs  The target for upgrading research laboratories and technology centers was exceeded, reaching the number of 49 in excess of the end target of 30, including the provision of important equipment and scientific facility upgrades. However, long-term constraints in financing the labs and technology centers might compromise their operations and maintenance;  The project enabled the reconstruction and refurbishment of the Polo Cientifico Tecnologico housing the ANPCyT and has become a public science, technology and innovation center in Buenos Aires, and a cornerstone for supporting the STI policy agenda as well; Outcomes Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752)  The project performance target toward the number of universities that successfully established norms on intellectual property rights according to international best practices was more than doubled, reaching 18 against a target of 8; Rating Substantial OBJECTIVE 5 Objective To expand the Borrower’s capacity to generate productive innovation in knowledge-based areas by strengthening the policy framework governing science, technology, and productive innovation. Rationale Theory of change The constructed theory of change (ICR, Figure 1. Theory of Change) was to achieve the goal of strengthening the policy framework governing science, technology, and productive innovation. Key activities that were to be deployed included (i) strengthening the capacity of MINCyT for policy making and monitoring and evaluation; (ii) supporting technology transfer offices at selected universities to improve intellectual property management; (iii) undertaking surveys to gather information on national and provincial innovation systems; and (iv) strengthening the operational capacity of ANPCyT. Outputs The ICR reported (para 47-49) the achievement of the following outputs:  An international advisory committee to the Ministry of Science, Technology, and Productive Innovation (MINCyT) was established and was operational;  The target of number of staff in technology transfer offices at universities trained on intellectual property rights was exceeded, reaching 203 against a target of 140;  The project helped to strengthen key STI institutions including the MINCyT, the ANPCyT, and CONICET, along with universities to help improve the environment for commercializing new innovations;  The Project also financed activities to improve transparency and informed decision-making processes such as independent evaluations and beneficiary satisfaction surveys for the ANPCyT. All of the management teams within the ANPCyT associated with the Project (EMPRETECNO, FONARSEC) also remained functional, with well-developed capacity to support firms, universities, and other beneficiaries;  Beneficiary satisfaction surveys were completed and utilized for policy making, although not quantified in the ICR;  However, the percentage performance target of Research Centers that implemented tools to monitor and facilitate usage of shared research equipment was missed, as achievement was nil against a target of 50. Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) Outcomes  The percentage target of project beneficiaries (70 percent) who expressed positive perceptions about ANPCyT's activities during project implementation was fully achieved;  Both PDO indicators related to linkages between the public sector, STI, and the private sector met their targets: (i) the project target for number of universities that successfully established intellectual property norms in accordance with international best practices reached 16, against a target of 8, (ii) twenty-five disclosures were made by CONICET researchers, in accordance with the target of 25;  The project performance target toward the number of universities that successfully established norms on intellectual property rights according to international best practices was more than doubled, reaching 18 against a target of 8;  The above norms helped establish a more beneficial environment for universities to engage with potential commercial partners on their research, an environment which did not exist previously at these universities;  The performance toward increasing the number of independent evaluations of STI programs was exceeded, reaching the number of 9 against a target of 7;  The ANPCyT’s recent elevation to a more independent agency reflects this cultural shift within the government, especially in an STI policy environment that faced pressure with recent fiscal cuts and consolidation; and  Further, the STI Center in Buenos Aires places innovation front and center for the local population, so far entertaining 235,000 visitors since it opened in 2015. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall efficacy by sub-objectives is summarized below:  Toward facilitating the creation of new knowledge-based companies, achieved outcome includes (i) the introduction to existing or new markets, new and improved products, protocols and services, (ii) increased number of technology-based start-ups that formally registered as companies, (iii) the creation of Technology-Based Firms (TBFs), which developed a new product or service, some registering sales revenues, obtaining private capital contributions, and hiring staff; (iv) some TBFs reached break-even within their first 2 years of operation and some of them exported a portion of their production; and (v) an increase of the overall additional investment catalyzed and leveraged through project activities;  Toward developing specialized human capital, achieved outcomes included the following: (i) the ICT scholarships led to a 21% likelihood of a higher salary for these scholarship recipients; and (ii) the FONSOFT program increased the likelihood that a student would be hired by an ICT firm by 24% relative to without the intervention. Challenges were limited commitment from universities, difficulties Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) in managing the breadth and scope of identified activities, and that trained researchers moved on to better paying jobs in the private sector;  Toward supporting productive research innovation activities and their technology commercialization in the areas of biotechnology, nanotechnology and information and communication technology, achieved outcome include the following: (i) introduction of new or improved products or services to new or existing markets, (ii) among public-private consortiums supported by FSAT, about half developed a new product/service and registered sales revenue; (iii)the FSAP program positively contributed to innovation activities for beneficiary firms. Specifically, the FSAT program improved firm performance, increasing growth in terms of both employment and sales, and (iv) the return on overall project investments is estimated to be about 26% in increased revenues, netting out the value of the grants provided;  Toward upgrading the research infrastructure in the areas of science, technology and productive innovation, key achieved outcomes were as follows: (i) the number of universities that successfully established norms on intellectual property rights according to international best practices was more than doubled, and (ii) the target of universities with international norms on intellectual property rights to facilitate innovation was exceeded;  Toward strengthening the policy framework governing science, technology, and productive innovation, achieved outcome include the following: (i) the majority of project beneficiaries express positive perception on ANPCyT's activities, (ii) the number of universities that successfully established norms on intellectual property rights according to international best practices was more than doubled, (iii) the ANPCyT’s recent elevation to a more independent agency reflects this cultural shift within the government, and (iv) the STI Center in Buenos Aires places innovation front and center for the local population. Overall Efficacy Rating Substantial 5. Efficiency Economic efficiency At appraisal, an economic efficiency analysis was completed, but no ERR or NPV were calculated, as this was a predominantly research, development (R&D) and innovation operation. While productive innovation can have an important impact on growth, numerous market failures generally hinder or delay investments in R&D and innovation more broadly. The simulations conducted at appraisal suggested that the returns on R&D projects for higher income countries were in the 20-40 percent range, while for medium-income levels, such as Argentina, the average return was about 55 percent. As part of the ICR exercise, an economic analysis (ICR, Annex 4) to estimate the economic rate of return of the project incorporating actual disbursement figures was limited to the GTEC, EMPRETECNO and FSAT programs, which supported technology transfer graduates, technology based companies and public-private research consortia. The above analysis resulted in an estimated NPV of US$81 million at a 15 percent discount rate, with an ERR of 72% over the project implementation period. The analysis was based on improved cash flows of beneficiaries under the three programs, with higher revenues for EMPRETECNO and FSAT Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) beneficiaries and higher salaries for GTEC beneficiaries. However, the ICR estimates did not factor in actual benefits which will likely be higher for the program (EMPRETECNO and FSAT) beneficiaries, following continued business growth, arising from additional financing and product development. Moreover, the ICR noted (para 53) that the project activities to strengthen the policy and regulatory environment for innovation were likely to amplify the impact of the project through positive externalities across knowledge- based sectors. Based on the above estimates and considerations, the project's economic efficiency is rated as substantial. Administrative and operational efficiency The project faced operational delays associated with the Venture Capital Fund, which was eventually canceled, and hurdles in the implementation of the human capital and deal flow facilitation activities. However, disbursement performance rate was high as it reached 94%, including the additional financing. On the positive side, all implementation challenges were handled flexibly through six restructuring operations which deployed mitigating measures through reallocation of funds and cancelation of no longer essential activities. However, the long implementation period (five years after original closing date), and the six restructuring operations reduced the overall project efficiency which is rated as Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate  72.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of project objectives was high, due to the comprehensiveness of the supported program, and the congruence among the country's priorities, and the World Bank partnership strategies with Argentina. Efficacy was substantial, because of significant results achieved under four sub-objectives, and modest results under one sub-objective. Efficiency was modest due to very long implementation period. On balance, the overall outcome is rated as Moderately Satisfactory. Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR found (para 102-103) that the project's achievements provided sufficient evidence that private sector solutions with an immediate demand can bolster the sustainability of the programs (EMPRETECNO and FSAT) with a healthy return on investment, high firm survival rates, and an ability to leverage private financing. Both the MINCyT and the ANPCyT have shown strong management capacity, and have become important players within the government in their support of the STI policy agenda, notably with the establishment of the STI Center. Additionally, the project achieved results in the upgrading of research laboratories and technology centers, including the GTEC program for technology brokers, the improvement of the CONICET's capacity, and the establishment of intellectual property norms at sixteen universities. The ICR identified several weaknesses (para 104-105) that could endanger the achieved outcome: (i) first, the ICR asserted (para 104-105) that policy and macro-level risks might undermine achieved progress. For instance, the progress made in promoting industry linkages and firm-level productivity might suffer from from fiscal consolidation, budget realignment efforts and tax incentives for STI investment. Since these policy shifts essentially devalue the importance of productive innovation within the context of government policy, this policy environment could hurt ANPCyT’s sustainability, (ii) second, the ICR revealed that Argentina’s economy is crisis-prone, as it has spent one third of the last 60 years in 14 episodes of recession. The economy and government-supported programs are subject to recurrent boom and bust cycles which might place continued fiscal pressure and reduce financing for programs that support innovation and knowledge- based sectors. The ICR noted (para 106-107) that some launched programs and activities are not self-sustaining without the external financing: (i) while the BECAS Jovenes TIC program provided scholarships for specific ideas and business plans, improved academic performance and ICT hiring, this program is unlikely to continue in the absence of the project, and (ii) the upgraded labs and technology centers constitute real assets with continued long-term value; but there were long-term funding constraints for these institutions. More adequate activities could have been incorporated organizationally to allow them to become more financially self- sustaining as opposed to remaining dependent on central or regional government funding. Finally, the ICR found (para 110-111) that CONICET's capacity improvements achieved through training programs in technology transfer and commercialization were dwarfed by the fact that many trained personnel were lost to the private sector. Moreover, the continued weakness in ANPCyT’s M&E systems may constrain their ability to advocate for financing, given the continued macroeconomic pressure and Argentina’s limited fiscal resources. The inadequate baseline and underlying data might make it difficult to scale up and build on existing programs, and the lack of data may restrict their ability to build consensus and affect the long-term sustainability of ANPCyT’s programs. 8. Assessment of Bank Performance Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) a. Quality-at-Entry The project was complex and had five different components to tackle innovation from different angles. The project scope incorporated major human capital interventions, direct engagement of potential start- ups, government agencies being supported on processes, norms, and the regulatory environment, and private sector beneficiaries of different sizes. The PDO was broad and multi-faceted and covered both the supply and demand sides of the innovation support. The project results framework had PDO indicators designed as output-based rather than outcome based. The ANPCyT had limited experience in World Bank operations and the resulting fiduciary requirements. Individual firms, universities, and research consortia beneficiaries under the project were also subject to Bank procurement requirements causing difficulties and delays in implementation since these beneficiaries had no experience in complying with these requirements. While broad and complex, the project targeted key players in the public and private landscape to leverage the maximum impact in launching the modernization of the productive innovation. The project design drew on a strong analytics of the country's growth dynamics and opportunities and the support from a new ministry, which led to the identification of development constraints and the matching activities to address them in a creative manner. Moreover, the project design aligned incentives for public, private, and academic actors, including a collaborative process between the ANPCyT, the overall MINCyT, and the World Bank team. The multidisciplinary team from ANPCyT in charge of implementation and execution of the project activities was qualified and committed to delivering the project outcome. The breadth of the project activities required coordination across a wide number of public and private organizations, including the implementation unit, ANPCyT, MINCyT, CONICET, research institutes, private beneficiaries, and the various fund structures. However, fiduciary and procurement arrangements were weak and needed to be improved The ICR reported (para 88-93) that collaboration between the ANPCyT and the World Bank team in the design of some of the components was generally smooth, but there was also disagreement due to tight timelines and differing views of how some project components should be structured. The activities which had weaknesses at design are as follows:  the venture capital fund was originally identified as a high-risk component as it failed to account for the lack of trust amongst potential investors, as private investors refused to commit capital without knowing who the fund manager would be;  the deal flow facilitation was later moved to FONARSEC due to poor implementation by the originally identified facilitators, as the required use of multiple private deal facilitators did not account for the fact that such deal facilitators were not available in the local market;  the prior identification of the pipeline of beneficiaries was not made, which could have been mitigated had a pipeline of beneficiaries been identified as part of preparation;  the design of the human capital component did not adequately account for private sector needs, implementation capacity and institutional requirements, nor did it address as intended the quality of labor available for knowledge-based sectors; and  while the support provided to CONICET helped to create the tech transfer unit intended to better prioritize productivity-focused activities, it did not address the cultural, financial, and institutional Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) constraints within CONICET. Moreover, many of the researchers trained to bolster CONICET’s capacity were later hired by the private sector due partly to insufficient salaries. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision As reported in the ICR (para), the project complexity resulted in implementation delays and had to be adjusted during different restructuring operations. The Bank closely implemented the project, and prepared twenty-two Aide Memoires and ISRs following supervision missions which included beneficiary roundtables and site visits. The Bank team also commissioned mid-term evaluations, impact evaluations, economic analyses and case study reports to assess the impact and help in the course correction of project activities. Additionally, the TTLs maintained close contact with projects' beneficiaries, through a combination of field visits and roundtable discussions for feedback. Supervision proactively identified improvements in project design and how to maximize the use of resources. Restructurings focused on both new and additional activities, such as the creation of the innovation matching grant in the 2010 restructuring, and scaling up well functioning ones. The innovation matching grant program created during the 2010 restructuring was developed using resources freed up due to currency devaluation, and these funds were reallocated to this program in line with project objectives. Additional financing also introduced design improvements to finance better data collection and impact evaluations. The Bank team was generally responsive to client needs despite minor delays in project management processes, and was proactive in identifying the challenges and risks to the achievement of the PDO and PDO indicators. For instance, the lack of movement in launching the Venture Capital Fund led to the restructuring and reallocation of these funds to better functioning components. Restructurings and funding reallocations were discussed and agreed to between the ANPCyT and the World Bank team before their incorporation in ongoing or new activities. Fiduciary and procurement requirements placed upon the ANPCyT in some components hampered project implementation. The project design and implementation could have been better adjusted to accommodate for needs and realities of such interventions, and the project timeline. The fiduciary and technical team implementing the project was well-qualified, and had both internal and external technical expertise. The collaboration between the World Bank team and the government counterparts functioned well for the most part, despite occasional delays in World Bank approvals. After implementation began, the turnover rate for World Bank staff was low, with appropriate reporting and transition measures. The ANPCyT worked productively with the WB team and their staff, and continued to consider the World Bank team as a critical partner in addressing procurement challenges. ANPCyT had to provide procurement related support since beneficiaries were required to meet World Bank standards. Because of the wide range of equipment being procured for beneficiaries, the procurement needs could not be addressed through larger-scale workshops or trainings. Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The ICR provided little information on the project's original monitoring and evaluation (M&E) system. But the monitoring and evaluation of outcomes and results was given priority in the PAD in its Annex 3, which presented the results framework and the monitoring and evaluation mechanisms. The original M&E framework established a link between project activities and the PDO, and the PDO indicators were generally aligned with the PDO, but they were primarily output-based as opposed to outcome-based. The ANPCyT had previously acquired extensive experience in monitoring results and outcomes, and there were activities devoted to enhancing the MINCyT’s capacity to monitor the impact of STI policies, including the impact of the proposed operation. b. M&E Implementation As noted by the ICR (para 76), the implementing agency and the Bank systematically reported on key indicators. Project indicators were monitored against their targets to measure the accomplishment of objectives. The project implementation unit monitored the progress of the project and prepared semi- annual reports that were shared with the World Bank team. The reports included a summary of the activities performed during the period per each component, an updated table with the status of the PDO and Intermediate results indicators, along with important activities and actions undertaken during that time period. ANPCyT developed a much stronger M&E culture that continued to collect information and calculate project indicators. The project incorporated a number of impact evaluations into the design at Additional Financing to ensure better documentation of impact in addition to the results framework. The project revised twice the results framework during the process in 2015 and in 2016 to put in place more outcome-based indicators at the PDO level. This updated results frameworks provided a better assessment of the PDO and links into the theory of change. The results framework was updated as part of additional financing to address the fact that the original PDO indicators were output-based; however, the updated results framework continued to include several multi-faceted indicators which were more difficult to measure. Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) c. M&E Utilization The ICR reported (para 78-79) that M&E was used to inform project management and decision-making periodically. Further, the monitoring system contributed to strategic and operational decision making which led to better credibility for both the MINCyT and the ANPCyT within the government. The M&E data on performance and results progress periodically reported were used and incorporated in the Implementation Status and Results report (ISR). M&E fed into decisions related to subsequent project activities. For example, the initial structure of the innovation matching grant program limited financing only to R&D activities, which resulted in a low take- up rate of beneficiaries. Based on discussions with potential beneficiaries, the ANPCyT broadened potential uses of the matching grant to include all innovation-related activities, drastically increasing take-up of the program. Similarly, the FSAT and EMPRETECNO programs dealt with disbursement delays because beneficiaries encountered difficulties in meeting Bank procurement requirements for acquiring equipment; this led to the ANPCyT adjusting the type of support provided to beneficiaries to support them on meeting these fiduciary requirements. The project’s M&E utilization initially weighed towards monitoring, as opposed to an economic and financial evaluation, but it focused more strongly on evaluation for the future in the last 2 years of the project. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The ICR noted (para 82) that at appraisal, it was assessed that there would be no significant or irreversible potential negative environmental and social impacts, given the nature and scale of the project’s activities related to science and technology, and innovation research. The project triggered the World Bank Environmental Assessment OP/BP 4.01 and was classified as a Category B project. There were no adverse impacts that were not previously identified that occurred during project implementation. Moderate to low risks and non-significant impacts associated with sub-projects were successfully managed using the safeguard instruments developed at appraisal. The ICR reported that following the performance of the Socio-Environmental Management Unit and proper application of safeguard instruments, the project environmental compliance was consistently rated satisfactory throughout the project implementation period. b. Fiduciary Compliance Financial management The ICR reported (para 83-86) that the Borrower’s FM team was competent and handled correctly and satisfactorily the ANPCyT’s financial management function. All IFRs received during the project's life were Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) considered acceptable, with the majority of them received by the due date. The project’s designated account was fully documented and there were no instances of ineligible expenditures identified to date. Financial Management (FM) performance was rated as Satisfactory and the FM risk rating as Moderate, throughout the project’s life. Most audit reports under the original loan were received with a delay, except for the 2016 audit report, and performance continued to improve after 2016 under the Additional Financing, with all audit reports being received on time. All the audit reports expressed unqualified opinions, except for the 2011 audit report. The final audit report covering the FY19 period and FY20 period is due by June 30, 2020. Procurement The ICR did not provide a systematic assessment of the project's procurement management. However, procurement issues that occurred during project implementation were discuused in many places in the ICR as summarized below:  First, the FSAT and EMPRETECNO programs faced disbursement delays because beneficiaries encountered difficulties in meeting Bank procurement requirements for acquiring equipment, forcing the ANPCyT to adjust the type of support provided to beneficiaries on meeting these fiduciary requirements.  Fiduciary and procurement requirements placed upon the ANPCyT in some components hampered project implementation. For example, procurement thresholds for ex-ante World Bank review were set extremely low, requiring individual nonobjections for even small-scale equipment purchases by individual beneficiaries, causing difficulties and delays. While the ANPCyT was able to reach the required capacity eventually, complexity in the project design resulted in procurement and FM delays, especially due to stringent requirements placed on beneficiaries’ procurement processes.  Future projects could consider how to simplify these procurement requirements for sub-executors such as possible bundling of procurements. The extent of requirements for the VCF also contributed to difficulties in implementation, ultimately leading to its removal during Project restructuring. c. Unintended impacts (Positive or Negative) Two developments reported in the ICR appear to be noteworthy unintended impacts: (i) The Project achievements led to a change in the culture around scientific research and innovation; the activities supported by the proejct helped to combat the negative mindset hindering technology transfer of research and partnerships between public research institutions and private partners. (ii) The project financing supported the development of generic cancer drugs which resulted in a US$54 million annual reduction in public health spending since 2017. A firm focused on developing cancer drugs, developed two new products at a 30% cost reduction relative to the equivalent products on the market and in addition to capturing the market share of the existing treatment, expanded the number of total cases treated by 15,000 a year to date, and the firm was able to capture a 70% national market share, leading to an additional US$91 million in currency savings per year. Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) d. Other The ICR did not identify any other significant issues. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory There were significant Bank Performance Satisfactory Moderately Satisfactory shortcomings in the quality at entry. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Several lessons documented in the ICR (para 112-131) reflect the experience of project design and implementation, and the prominent ones are summarized below: 1. Micro-interventions focused on STI can generate results in a context of macro- volatility; on the contrary, market-based instruments may not function as well. Despite the macro-uncertainty in Argentina, the project supported the creation of technology-based firms, improved firm productivity, created skilled jobs, and commercialized new products, including those which helped increase incomes for vulnerable communities. However, both the Venture Capital Fund and the Deal Flow Facilitation components were closer to the market and did not work well, stressing that market-based instruments supporting productive innovation may not function as well in a context of macro-volatility. 2. Projects focused on supporting knowledge-based companies may consider allowing diverse technology firms to become project beneficiaries: The initial calls for proposals for the innovation grant program were limited to focus on biotechnology, nanotechnology and ICT. A broader technology focus would have allowed for faster implementation and a wider response to calls for proposals. The initial low response rate eventually led to the program’s scope being widened to include all technology sub-sectors. Limiting knowledge-based programs to specific sub-sectors may exclude important innovations from benefitting from project activities. The focus on biotechnology, nanotechnology and ICT was narrow and proposals for the innovation grant program were limited; this low response rate eventually led to the program’s scope being widened to include all technology sub-sectors; 3. Private sector focused interventions need to emphasize the strengthening of key institutions within the innovation space. The project supported key institutions within the national innovation system, including the ANPCyT and the MINCyT. Project financing also enabled the creation of FONARSEC, the ANPCyT fund to promote technology-based Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) entrepreneurship and collaborative research consortia, and the National Scientific and Technical Research Council (CONICET Tecnologia), the technology transfer unit under CONICET to support commercialization of new research. Structural institutional support could address long-term funding constraints for labs and technology centers to allow for better performing operations and maintenance alongside equipment upgrades. 4. A deeper public-private dialogue and communication is essential to engage the innovation ecosystem. Key results arose from a strong and effective public-private dialogue at the activity level. In the best case scenario, components requiring private sector participation should incorporate market testing at the design stage to ensure implementation- ready design and fiduciary requirements. Such a process would have allowed for adjustments in the design prior to the start of implementation, most likely mitigating some of the implementation risks and reducing the associated difficulties. Beneficiaries of the EMPRETECNO and FSAT programs underscored the “signaling value” of their participation in these programs, which could be better incorporated into project communications to entice larger-scale participation. Additionally, previous beneficiaries also expressed a commitment to providing mentorship to future beneficiaries, which could help ensure long-term support for STI program beneficiaries and increase the value of participation. 5. Streamlining the project design increases the probability of achieving the identified project objective: Future operations should consider more focused support and simplify the project design to incorporate fewer institutions and implementation agencies, along with simplifying fiduciary requirements to streamline implementation. Procurement arrangements which apply to beneficiaries could also be simplified to reduce their associated burden. Complex Project components should be designed with a focus on the implementation arrangements and the pipeline of beneficiaries with an implementation plan and timeframe, based on clear agreement between the Bank team and government counterparts. Ensuring a good match between Project activities and the specific implementation agency plays a key role in success of these activities. 6. A large-scope innovation project requires a strong M&E system for a successful implementation and assessment: Due to limitations in M&E systems and time constraints, the project was unable to gather baseline data across all its components, leading to difficulties in documenting the impact of project outcome. This also hindered institutional credibility for the ANPCyT and MINCyT, along with the overall STI and innovation agenda. Including baseline studies could have plays a key role in assessing the impact of interventions and measuring the scale of impact. A stronger M&E system would help document program impacts and allow for more timely adjustments as needed to ensure program effectiveness. M&E design at project preparation would have emphasized the importance of gathering baseline data, especially when accurate statistics were limited. 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AR Unleashing Productive Innovation (P106752) This is a transformational project which brought changes in the structural and institutional foundations of the public and private stakeholders active in the productive innovation area. Most intended results are likely to show up only with a significant lag. A PPAR prepared a couple of years after the project closing date would shade light on the impacts of the project on the diverse sectors of the economy. 14. Comments on Quality of ICR The ICR is comprehensive and generally consistent with guidelines, and the analysis is broadly evidence- based. The ICR constructed an adequate theory of change for the project, which was missing in the Project Appraisal Document and Restructuring Papers. The ICR’s assessment of efficacy is candid, and provides contextual information about the project's outputs and outcomes, although the material could have been streamlined. The ICR suggests several thoughtful lessons learned from the project, which mostly reflect the experience of project design and implementation. There were moderate deficiencies in the ICR as follows: (i) there was no attempt to attribute outcomes to sources of financing and associated substantive emphases, such as the contribution of both the Argentina Government and the Inter-American Bank (IDB), which supported productive innovation (ii) there was no systematic assessment of the procurement function. a. Quality of ICR Rating Substantial Page 22 of 22