Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) Report Number: ICRR0023536 1. Project Data Project ID Project Name P156250 Peru Innovation Country Practice Area(Lead) Peru Finance, Competitiveness and Innovation L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 31-Dec-2021 41,909,640.38 Bank Approval Date Closing Date (Actual) 11-Jan-2017 14-Oct-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Nestor Ntungwanayo Fernando Manibog Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As stated in the loan agreement on page 5, "the objective of the Project is to strengthen the Science, Technology and Innovation (STI) System to improve research skills and firm-level innovation." The Project Development Objective in the Project Appraisal Document (PAD) was identical and was unchanged throughout the project implementation period. Applying the OPCS guidelines (World Bank Guidance of December 2021) suggesting that compound PDOs with multiple outcomes linked together in a single Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) sentence are unpacked and each outcome assessed separately, the PDO will be unbundled and assessed as follows: PDO-1: To strengthen the Science, Technology and Innovation (STI) System to improve research skills, and PDO-2: To strengthen the Science, Technology and Innovation (STI) System to improve firm-level innovation. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components Component 1: Improving the Institutional Framework of the National STI System (appraisal amount of US$4.27 million, actual amount of US$4.17 million) : This component had two sub-components as follows: (i) the first subcomponent focused on strengthening the institutional design and governance capacity of the STI system for it to become more integrated, efficient, coordinated, and effective, (ii) the second subcomponent was composed of technical assistance (TA), training, and financing of equipment to build capacity, and digitalize and modernize the management and monitoring capacities of CONCYTEC, PROCIENCIA, and STI institutions. Component 2: Strategic Programs: Productivity & Innovation Fund and Competitiveness Reinforcement Initiatives for Productive Innovation: (appraisal amount of US$7.95 million, actual amount of US$5.14 million) (a) Original sub-components:  Component 2 was originally composed of four subcomponents as follows: i) identification of technological innovation opportunities through academia-industry linkages, ii) formulation of Competitiveness Reinforcement Initiatives to strengthen academia-industry linkages (Iniciativas de Vinculación Academia Industria-IVAI, in Spanish), iii) implementation of IVAIs, and iv) evaluation and monitoring of IVAIs. (b) Revised sub-components: The 2018 restructuring reorganized the activities under the 4 original sub-components around 2 separate programs as detailed below:  Window I: “Go-to-market Program: The objective of this program was to help firms finalize, introduce, and scale up their innovations by supporting their acquisition of expertise, knowledge, and insights on the markets they are targeting. The program comprised a package of three specific support services to a hundred innovation-seeking firms as follows: i) training for firms’ managers to Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) enhance their positioning in the market and scale their operations, ii) the provision of peer-to-peer learning by industry-specialized mentors, and iii) subsidized hiring of Chief Revenue Officers to help firms create a marketing strategy and increase market penetration.  Window II: “IVAI” Program: This program focused on (i) enhancing private-public dialogues in targeted clusters and value chains based on a robust, market-oriented, and analytically underpinned approach, (ii) formulating action plans to strengthen eight value chains, (iii) subsidizing support to beneficiary firms interested in implementing the action plans; and iv) all along, building the capacity of CONCYTEC and related STI institutions to replicate this experience on other value chain. Component 3: Research and Innovation Capacity (appraisal amount of US$31.8 million, actual amount of US$32.03 million)  This component provided full or co-financed grants to strengthen the capacity of the supply side of the STI system to generate knowledge and contribute to productive innovation and had three subcomponents: (i) the first focused on strengthening human capital for STI by providing grants for teams of senior, postdoctoral, and associate researchers, as well as the reinforcement of ten new PhD programs, (ii) the second provided grants to modernize and strengthen research equipment in universities and research centers, and finally (iii) the third consisted of grants to support so-called “integrated projects” for basic and applied research and technological development projects. Component 4: Project Management and Monitoring & Evaluation (appraisal amount of US$0.85 million, actual amount of US$0.58 million)  This component was to fund the implementing agency (CONCYTEC first, and PROCIENCIA later) to cover expenditures related to project coordination, technical, procurement, financial, and M&E specialists. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost: : At appraisal, the estimated project cost amounted to US$100.0 million, of which 45 percent from IDA, and the remaining 55 percent from the Government of Peru. Actual project cost totaled US$83.66 million, with Bank disbursements amounting to US$41.9million, while US$3.1 million were cancelled. Financing: The project was financed by a total of US$100 million, including an IDA loan of US$45 million, and a Borrower contribution in the amount of US$55 million. Actual disbursements totaled US$83.66 million, or about 84 percent of initial financing commitments. Borrower Contribution: The Government of Peru's disbursements amounted to US$41.75 million, or about 76 percent of its commitments at appraisal. Dates: The project was approved on January 11, 2017, and became effective on July 06, 2017. A mid-term review was completed in July 2019. The project was restructured five times and its duration extended once as delineated below, and was closed on October 14, 2022, nine months and a half after the original closing date. Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) (i) On September 10, 2018, the first restructuring modified the contracting scheme used to hire foreign researchers for Component 3 and corrected an inconsistency between the PAD and the Loan Agreement with respect to the type of matching grants for Component 2. (ii) On April 30, 2020, the second restructuring clarified implementation arrangements to reflect changes in administrative and fiduciary procedures resulting from the approval of Law 30806 in 2018, and reallocated resources from Component 3 to Component 2 and between disbursement categories, to ensure the necessary funding was available for Component 2 in support of the Ministry of Economics and Finance's (MEF) National Plan of Competitiveness and Productivity 2019-2030. This restructuring also made minor adjustments to certain targets in the results framework intermediate indicators. (iii) On October 20, 2021, the third restructuring extended the closing date from December 31, 2021, to October 14, 2022. (iv) On March 15, 2022, the fourth restructuring renamed FONDECYT as PROCIENCIA in legal documents and the attribution of all project management functions to PROCIENCIA, based on the changes in Peru’s Innovation Law. (v) On September 20, 2022, the fifth restructuring canceled an amount of US$3,090,360 from the loan, as requested by the Ministry of Economy and Finance. 3. Relevance of Objectives Rationale The relevance of objectives was discussed in the paragraphs 22-26 of the ICR and is summarized below. This review found that the PDO was substantially relevant. While the PDO was in congruence with the Borrower priorities and the World Bank country strategies, the project ambition was pitched low, because this project was a preliminary step aimed to address challenges in a sector which had major institutional weaknesses as discussed under Section 8, thus requiring the Bank to provide close support to the PIU and the implementing agency. The PDOs were consistent with the Borrower priorities. The PDO was aligned with the Peru’s National STI Policy Plan 2016-2020, and the MEF’s National Plan of Competitiveness and Productivity 2019- 2030. For instance, the Objective 1 of the National STI Policy Plan, “Promote the generation and transfer of scientific knowledge”, and Objective 2, “Promote and develop new incentives that increase STI activities by actors in the STI system”, were reflected in the activities of Component 2, which promoted higher innovation and technology adoption by firms and increased collaboration between firms and academia. The objective 3, “Promote the creation of human capital qualified for STI”, and the objective 4, “Improve the quality levels of research centers”, were in congruence with the activities of Component 3, which aimed to increase human capital specialized in STI as well as upgrading research equipment. Finally, the project was in congruence with the Ministry of Economy and Finance's (MEF) National Plan of Competitiveness and Productivity 2019-2030, specially with three objectives: “Strengthen Human Capital” (objective 2), “Create capacities for innovation, adoption, and transfer of technological improvements” (objective 3), and Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) “Strengthen the country’s institutions (objective 8)”. The Borrower's commitment was further illustrated by its elevated co-financing of the project activities. The PDO was in congruence with the WBG strategies in Peru. The PDO was consistent with Pillar 1, “Productivity for Growth”, of the World Bank Group’s Country Partnership Framework (CPF) for Peru (FY 2017-2021), discussed by the Board of Executive Directors on May 2, 2017 (Report No.112299). In particular, the PDO was aligned with Objective 3 (under Pillar 1) of the CPF “Facilitating absorption of skills and technology, especially by small and medium-sized businesses”. Component 2 addressed constraints to firms’ adoption of innovation and/or technology, by supporting firms and value chains through business development services that incentivize innovation and increase their competitiveness, while Component 3 focused on innovation supply constraints by increasing research capacity, in terms of human capital and research centers, for which the results were to be evaluated by measuring PhD enrollment and research publications. Finally, Component 1 facilitated productivity-led growth by improving the efficiency of the innovation system in Peru through higher institutional capacity of key public sector actors. The PDO remains substantially relevant, as Objective 2 of the new CPF for Peru (FY 2023- 2027) is focused on “Improving innovation and technical skills”. However, the PDO was pitched low, because the goal to modernize the STI system was a daunting one, and this project aimed to pave the way for a long process that might take many years and a series of operations. That is why a second operation building on the achievements of this project was approved in February 2022 (P176297). Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective PDO-1: To strengthen the Science, Technology and Innovation (STI) System to improve research skills. Rationale Theory of change The PAD developed a results chain which was updated in the ICR on page 7 and summarized hereafter. Toward improving research skills, key project activities aimed to: (i) improve the institutional framework of the national STI system, (ii) strengthen the country's research and innovation capacity, and (iii) ensure adequate project management and monitoring & evaluation. Expected outputs included the following: (i) an increased number of CONCYTEC's and other officials trained, (ii) a higher number of active researchers supported by the project, (iii) a higher number of PhD programs in areas covered by the Strategic Programs Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) on Innovation, (iv) a higher number of laboratories upgraded and fully operational, and (iv) a higher number of University-Industry Research Partnerships supported by the project. Expected outcomes included (i) an increased number of research papers submitted for publication to internationally indexed journals, (ii) an increased number of students enrolled in PhD programs, and (iii) an increase in scientific research and publication in Peru. Overall, there was a consistency between the theory of change and the results framework, and a logical and causal link between the selected activities, and the expected outputs and outcomes. The identified activities were appropriate and realistic. They were first generation activities and expected outcomes were reasonable. However, the ICR did not discuss the underlying critical assumptions and the attribution aspects underpinning the theory of change. In particular, respective contribution from the Government's entities and other donors was not specifically highlighted. Outputs  The target for the number of programs evaluated under the PER was exceeded reaching 55 against a target of 20.  The target for the number of PER recommendations incorporated in official Government documents to make public expenditure more efficient was missed; only one recommendation was incorporated against a target of 6.  The target for the number of CONCYTEC's and other officials trained was exceeded, reaching the number of 136 against a target of 60.  The target for the number of active researchers supported by the project during each year of implementation was exceeded, reaching the number 30 against a target of 20.  The target number for the PhD programs in areas covered by the Strategic Programs on Innovation supported by the project was fully achieved, as the target of 6 was achieved as planned.  The number of laboratories updated and fully operational supported by the project was slightly exceeded, reaching the number of 32 against a target of 30.  The number of University-Industry Research Partnerships supported by the project during each year of implementation was slightly exceeded, reaching the number of 28 against a target of 20. Outcome  The number of research papers submitted for publication to internationally indexed journals was largely exceeded, reaching 644 against a target of 80.  The number of students enrolled in PhD programs supported by the project was exceeded, reaching 94 against a target of 56.  The combination of the policy actions funded by the project led to a sharp increase in scientific research and publication in Peru as illustrated for the following: (i) the Peruvian scientific production in the Scopus database more than quadrupled by 2021, compared to the year the law was passed in 2015, and (ii) the number of students that enrolled in Project-funded PhD Programs was 94, that is 28 above the target of 54, (iii) through focus groups, students showed satisfaction with the PhD programs and scholarships, especially female and non-Lima based researchers. Overall, the efficacy toward improving research skills was substantially achieved, as illustrated by increased number of research papers submitted for publication to internationally indexed journals, as well as increased number of students enrolled in PhD programs. The combination of the policy actions funded by the project led Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) to a sharp increase in scientific research and publication in Peru. However, the attribution of these achievements was not sufficiently discussed in the ICR. Rating Substantial OBJECTIVE 2 Objective PDO-2: To strengthen the Science, Technology and Innovation (STI) System to improve firm-level innovation. Rationale Theory of change The PAD developed a results chain which was updated in the ICR on page 7 and summarized hereafter. Overall, there was a consistency between the theory of change and the results framework, and a logical link between the selected activities, and the expected outputs and outcomes. Toward the PDO-2, key activities were organized around three programs: (i) the “Go-to-market Program which comprised training for firms’ managers to enhance their positioning in the market, the provision of peer-to-peer learning by industry- specialized mentors, and subsidized hiring of Chief Revenue Officers, and (ii) the “IVAI” Program focused on enhancing private-public dialogues in targeted clusters and value chains, formulating action plans to strengthen eight value chains, subsidizing support to beneficiary firms interested; and iii) building the capacity of CONCYTEC and related STI institutions. Expected outputs included the following: (i) SMEs' owners and academia needed to be engaged in strategic Public-Private Dialogue, (ii) increase in the private sector capital leveraged, (iii) new or upgraded processes created by beneficiary firms, (iv) new or upgraded products created by beneficiary firms, and (v) the number of patents submitted by beneficiaries. Expected outcomes included the number of beneficiary firms that have introduced new or upgraded processes and products. The theory of change for the PDO-2 could have been improved. While the link between some of the activities, outputs and outcomes was logical and causal, it was not clear how firms could introduce new and upgraded process and products using just the project resources and without undertaking new investments. Moreover, the ICR did not discuss the theory of change's critical assumptions and the attribution aspects. Outputs  The target for the number of SMEs' owners that have been engaged in strategic Public-Private Dialogue (PPD) was exceeded, reaching the number of 89 against a target of 50.  The target of the number of Academia engaged in PPD was exceeded, reaching the number of 20, against a target of 5.  The target for the private sector capital leveraged was partially achieved, as only an amount of US$ 1.2 million was leveraged against a target of US$2.0 million. The ICR indicates (paragraph 36) that the shortfall is mainly attributable to the COVID-19 pandemic which created a context of lockdowns and a looming recession. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250)  The target for the number of new or upgraded processes created by beneficiary firms was exceeded reaching a total of 66 against a target of 30.  The target for the number of new or upgraded products created by beneficiary firms was largely exceeded, reaching a total of 112 against a target of 20.  The target for the number of patents submitted by beneficiaries was largely exceeded, reaching a total of 39 from a target of 10. Outcome  The target for the number of beneficiary firms that have introduced new or upgraded processes was exceeded, totaling 49 against a target of 25.  The target for the number of beneficiary firms that have introduced new or upgraded products was exceeded, reaching the number of 65 against a target of 15. Efficacy toward improving firm-level innovation was substantially achieved, as firms introduced new or upgraded processes and products. However, the attribution of these achievements was not sufficiently discussed. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Efficacy toward improving research skills was substantially achieved, because there was an increased number of research papers submitted for publication to internationally indexed journals, as well as increased number of students enrolled in PhD programs. The combination of the policy actions funded by the project led to a sharp increase in scientific research and publication in Peru. Efficacy toward improving firm-level innovation was also substantially achieved, because firms introduced new or upgraded processes and products. However, the attribution of achieved results under the two sub-objectives was not sufficiently discussed. Overall Efficacy Rating Substantial 5. Efficiency Economic efficiency: The ICR team updated the economic and financial analysis presented in the PAD using an 8 percent discount rate and found that the net present value of the project is positive at US$14,8 Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) million. The estimated ERR at the project’s closure was practically unchanged (11.39 percent), implying that the final benefits were similar to those expected at preparation, showing that the return to these investments was higher than the assumed discount rate. The benefits of the project were only calculated for the support programs under Components 2 and 3, thus reflecting a conservative estimate of the project’s value. Social benefits were estimated as the gains in consumer surplus obtained from lower prices and higher quality goods resulting from new or improved products and processes associated to the innovation and investigation support programs. Social benefits also included the gains in salaries for PhD graduates funded by Component 3. This result was obtained despite the fact that less resources were invested (US$83.7 million instead of US$100 million). The lower amount of total investments were attributable to lower costs, higher success rates for firms (61% instead of 50%), and more estimated future revenues from the higher number of PhD students (94 instead of 56). Operational efficiency. There were implementation challenges that delayed disbursement of funds until 2018 and halted the implementation of various activities. Counterpart funding represented 55% of the total project cost, and the project had to be incorporated in the public budget of the year of effectiveness. There were delays in assigning the budget with the needed resources for the PIU to launch activities until 2018. Furthermore, the COVID-19 pandemic, political and institutional changes, and a high turnover rate in the implementation unit’s staff affected the pace of investments. In addition, as the implementation of certain activities was significantly delayed due to the previously mentioned factors, adequate data collection on the impact of the project’s programs on beneficiaries was constrained. Overall, the project efficiency is rated substantial, based mainly on positive ERR and the NPV in line with appraisal projections. Regarding operational efficiency, the project team succeeded to keep the project on course and achieved key outcome performance indicators, despite weak implementation institutions. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 84.00 Appraisal  11.40  Not Applicable 84.00 ICR Estimate  11.40  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Based on the conclusions and the results presented under Sections 3 and 4 respectively, the PDO relevance was high, while the performance of the project's efficacy and efficiency were both rated substantial, resulting in an overall satisfactory of the project's outcome. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) a. Outcome Rating Satisfactory 7. Risk to Development Outcome From the material presented in the paragraphs 86-89 of the ICR, the risk to development outcome is modest. While there are lingering institutional weaknesses affecting public entities supporting the country's research and innovation system, the borrower is committed to build on and expand the achieved outcomes. There are potential risks that could negatively impact the project's long-term results and the efficacy of new projects. CONCYTEC does not have sufficient capacities to manage the supply and demand of STI. For instance, the Ministries of Production and Agriculture have not entirely integrated CONCYTEC’s oversight, especially when various ministries implement programs financed by different development partners with limited coordination with CONCYTEC. The Borrower ensured that the progress achieved by this project acts as a building block to strengthen the innovation capacity of research institutions, firms, and public actors in the STI system. The Borrower is implementing the new Peru innovation project (P176297), which allocated resources towards innovation policy, despite the changing governments and presidencies, which provides reasonable confidence in the consistent and broad-based support for this sector by the Borrower. 8. Assessment of Bank Performance a. Quality-at-Entry The performance of quality-at-entry is assessed as Satisfactory, based on the ICR material presented in the paragraphs 51-53 and 78-81. Overall, the project design was strategically sound, although some weak implementation and M&E arrangements were not sufficiently mitigated. The project was strategically relevant as all activities were consistent with key lessons learned from the Country Partnership Strategy FY12-FY16, which highlighted that Peru needed to further rely on increasing productivity given the weakened external conditions at project approval. The theory of change underpinning the PDO aimed to promote productivity-led growth through higher innovation. The support to the sector’s main overarching governance agency (CONCYTEC) aimed to improve the overall STI system in Peru and the sustainability of its results. The Bank supported the formulation of a theory of change which addressed issues pertaining to the three axes of the innovation ecosystem: the supply of innovation, the demand of innovation, and the complementary factors that affect the allocation of innovation. The World Bank regional and global technical teams leveraged the institution’s knowledge Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) and experience in innovation projects to help the implementing unit to identify activities addressing the main issues associated with Peru’s STI system. The implementation arrangements in the PAD entailed the collaboration of many actors across the STI System. At preparation, the PIU did not have the appropriate capacity to implement all project components and significant capacity building efforts would be required. The design of the project’s components accounted for the PIU’s lack of experience in implementing investment and TA projects financed by multilateral agencies, and its lack of experience in working with firms in the innovation sector. The project’s components were identified in collaboration with the PIU and the World Bank regional and global teams. This impacted on the level of ambition of the indicators’ targets included in the Results Framework, as the risk was substantial and the likelihood for the PIU’s capacity to achieve impacts was low. To help the PIU and CONCYTEC assume a more comprehensive and balanced role across the entire STI system, World Bank teams provided significant support for the preparation of activities that were new to the PIU and CONCYTEC (Component 2). The technical and economic aspects of the project at appraisal were thoroughly discussed in the PAD and achieved results are assessed at closure under the efficacy section. Identified components reflected the country's context and the World Bank's sector knowledge and lessons learned from similar projects implemented in other contexts. During negotiations, the Government requested to repay the debt in full nine months after the loan closing date, and this Loan Agreement tenet affected project implementation. The above Government's commitment affected the implementation of some activities under the IVAI action plans, which were ready to be rolled out in a 12-to-14-month period had the extension been possible. Finally, M&E arrangements at approval are presented under Section 10.a and were overall attuned to the country's context and conditions of implementation, including mitigation measures to ensure that identified outcomes can be achieved. Quality-at-Entry Rating Satisfactory b. Quality of supervision The performance of the Bank supervision quality is rated as Satisfactory based on the ICR material presented in paragraphs 54-63 and 82-85. Overall, the World Bank team was proactive in addressing implementation challenges and delivered the gist of the identified activities and expected outcomes. The ICR did not discuss specifically to which extent each project stakeholder was committed to the development impact during the period of project implementation, but this can be derived from the sustained efforts of the World Bank's project team involved in the project implementation as detailed below. The World Bank team closely supervised the PIU throughout implementation, leveraging its experience to mitigate the consequences of the adverse circumstances that arose. Before the COVID-19 pandemic, the Bank team undertook regular implementation support missions to resolve specific issues and delays, and acted upon feedback, evaluation reports, and other project documents and regular communication with the PIU. The supervision activities included members of the Bank’s global and regional teams, with specific expertise in different topics and interventions as needed. When the COVID-19 pandemic hit, the Bank Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) teams transitioned to a virtual approach to supervision, organizing weekly meetings with the counterparts to help address different issues that made implementation difficult. The PIU faced significant staffing issues that affected the implementation of the Project, especially for activities under component 2, and overall project management. Many highly skilled members of the Project only lasted a few months, and the remaining vacancies were not always filled on time. The areas that were most affected by this turnover were procurement, financial management, administrative management, and monitoring. These personnel shortages led to an overload of work for the remaining staff. Procurement staffing, in particular, underwent significant and repeated changes, leading to high demand for hands-on training by World Bank teams. As a result, the PIU strongly depended on World Bank assistance in procurement aspects during prolonged periods where the procurement specialist position was vacant, or for intensive support and training for the rotating specialists. The Bank’s regional and global technical teams, as well as procurement, legal, and financial management teams heavily invested in training and coaching the constantly changing staff of the regulating and implementing agencies, so that critical functions associated with implementation remained operational. The main external factor that affected the implementation of the project was the Covid-19 pandemic, which was particularly acute in Peru. The policy response to the health crisis entailed extraordinary lockdowns and restrictions that hindered the implementation of various Project components. The PIU and world Bank team collaborated intensely to resolve surging problems and issues in during the entire period of project implementation. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The ICR discussion of the M&E design in paragraphs 64-66 is summarized below. Overall, the link between the theory of change and the results framework was valid, but the quality of the results framework and M&E arrangements needed to be improved during the restructurings. The ICR revisited the theory of change prepared at appraisal, which developed a logical and causal link between the project’s activities and the PDO, following a detailed results framework to track the performance of the project’s components and monitor the achievement of targets of outputs and outcome indicators as summarized in the theory of change under Section 4. Building on the M&E system developed by CONCYTEC and FONDECYT, the project developed a data collection and management system to register all project beneficiaries and track the outputs and outcomes achieved by the project. The M&E system was detailed and the PIU leveraged it to prepare semiannual reports. In hindsight, the level of ambition of some of the indicators and targets and the methodology used to measure them could have been improved. For instance, criteria used to determine the targets for the number of new or upgraded Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) products created by beneficiary firms were not explicit. On a methodological level, the attribution aspect was not discussed, as new and upgraded products by firms could result from a conjunction of own and/or donor resources, in addition to the project’s resources. b. M&E Implementation The performance of the M&E implementation was discussed in the ICR in paragraphs 67-69. Overall, M&E implementation benefitted from the multiple restructurings which adjusted the results framework to take into account the constraints and developments of project implementation on the ground. During implementation, the design of the monitoring system showed some shortcomings. For instance, subprojects under Component 3 were monitored through a separate IT system, different from the overall financial monitoring system. This design flaw eventually meant that data on disbursement and implementation progress in the subprojects had to be manually transcribed to the project’s overall monitoring system, limiting the ability of the project and World Bank supervision teams to obtain information in real time during implementation. Using these reports, the World Bank team prepared annual Implementation Status and Results reports and provided support to the client to enhance implementation. To robustly evaluate the impact of the programs under Component 2 and to increase learning from these policy interventions, a Randomized Control Trial-Impact Evaluation (RCT-IE) was designed and implemented, with results expected by April 2024. The RCT-based evaluation will provide evidence- based findings on the appropriateness and effectiveness of these specific interventions on promoting innovation. This not only introduced an innovation in Peru’s STI system, but also contributed more broadly to the global effort to use experimental IE when supporting firm-level productivity enhancing activities. The implementing agency and the Bank systematically reported on key indicators. The PIU monitored the progress of the Project and prepared semi-annual Project reports that were shared with the World Bank. The reports summarized the activities performed during the previous six months for each component of the Project, an updated performance tracking of the PDO and results indicators, and any potential issues that the Project’s components faced. The Project leveraged the PIU’s M&E system, which was most useful for the programs under Component 3, given the PIU’s previous experience in monitoring and evaluating programs that support research institutions. The initial IT system designed to monitor implementation progress and disbursements was not fully apt to track in live time project activities. The project monitoring was carried out by a system which was previously developed by CONCYTEC and mainly focused on evaluating innovation activities associated with research institutions. Hence it was only adequate to monitor Component 3’s programs. Having access to data that was automatically updated, quasi instantly, would have been particularly helpful in the last months of the project, as all activities were racing to a close. Nonetheless, no flaws in data accuracy were identified, and the IT system was eventually updated to ensure the automatic transfer of data, but this was only effective for the implementation of the follow-on Project (P176297) To implement the RCT-based impact evaluation for Component 2’s programs, a globally renowned and specialized NGO, Innovations for Poverty Action (IPA), was hired to collect information from beneficiaries to assess results. This data collection process consists of surveying the beneficiaries in the experimental Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) group and non-beneficiaries in the control group. This data collection is underway, also financed by the new project. The conclusions of the impact evaluation are expected in April 2024. c. M&E Utilization Throughout the lifetime of the project, the M&E framework informed project management and decision- making. The M&E data on performance and results were periodically incorporated into the Implementation Status and Results report (ISR). However, the M&E system could have been timelier to optimize revisions of targets. In practice, as previously mentioned in the Achievement of PDOs section, this was not the case at the start. Lastly, the data collected from the impact evaluation will help inform the design of future innovation programs by PROCIENCIA. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as a Category B operation, based on the relatively low environmental, health and safety risks and impacts associated with applied research, technological development, equipment management, and experiment-based subprojects, and no social safeguard policies were triggered. The OP 4.01 was triggered and relevant national environmental regulations were triggered, including the preparation of an Environmental Management Framework (EMF). A full-time environmental specialist and a Safeguards Analyst, with the necessary profile and experience were hired to closely manage and monitor the Project according to the provisions of the program’s EMF. The continuity of both specialists until project closure contributed to ensuring an adequate and smooth management of E&S issues throughout. During project implementation, no relevant environmental and/or social aspects delaying or preventing the project closure were evident. The Grievance and Redress Mechanism (GRM) unearthed a number of complaints as follows: (i) non- payment by one of the beneficiary universities of the wages of two researchers; (ii) sexual harassment of one of these female researchers by a male supervisor; and (iii) delays in the disbursement of promised budget to a research team affiliated with a public university. By the project closure, the PIU had addressed all the complaints satisfactorily. b. Fiduciary Compliance Financial Management The ICR concisely discussed financial management issues in paragraph 77 and found that financial management was overall good, which included detailed agreements associated with organization and Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) staffing, planning and budgeting, accounting and information systems, internal controls and internal auditing, financial reporting, auditing, flow of funds, disbursements, counterpart funds, bank supervision, and procurement. At completion, the project had complied with all these obligations. Internal and external audits showed compliances with the financial management arrangements and the external auditors issued a clean opinion of the financial statements of the project. However, financial reporting shortcomings arose in 2020, following the COVID-19 crisis coupled with the high staff turnover of the PIU, which directly affected fiduciary processes as critical staff specialized in administrative management and treasury left and their vacancies were not easily filled, and the World Bank procurement and financial management teams supported the PIU to address these shortcomings. Procurement The ICR did not specifically discuss procurement issues. Upon request, the Bank project team provided additional material which is summarized hereunder. The procurement plan was fully executed during project implementation and deficiencies in the updated Systematic Tracking of Exchanges in Procurement (STEP) were duly addressed as follows: (i) problems identified in some procurement processes were mostly overcome with ongoing training and mitigation actions implemented, (ii) delays in some contract execution were addressed through the implementation of the necessary controls in the execution of contracts and payments, and (iii) the PIU and the World Bank team collaborated intensively to respond appropriately to the COVID-19 pandemic. c. Unintended impacts (Positive or Negative) Results related to gender-based inclusivity were exceeded. The initial target for this indicator was to have at least 20 percent female researchers. At the project’s completion, 34 percent of researchers were women, amounting to more than 150 researchers, and beneficiary female scientists particularly praised the project for improving their access to R&D funding and PhD programs. The project incorporated an RCT in one of its flagship activities. The results were expected 18 months after the closing of the Go-to-market program, but because implementation was delayed, the data collection process is still ongoing, and findings are expected in April 2024. This will not only improve the measurement of efficacy and efficiency of the project, and related innovation policy actions, but it will also inform the design of the new project. In addition, this activity was specifically designed to also build CONCYTEC’s capacity to conduct future impact evaluations of innovation programs based on a more robust approach. d. Other No other impacts were identified in the ICR. Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The ICR main text rates the project outcome as Satisfactory. There were minor shortcomings Outcome Highly Satisfactory Satisfactory in the achievement of project's objectives, in its efficiency and its relevance. Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR identified a list of useful lessons and recommendations arising from the implementation of this project presented in the paragraphs 90-98. This review highlights two lessons which are rephrased below. (i) A comprehensive and coordinated approach when strengthening the STI system needs to focus on supply of innovation, demand of innovation, and an ecosystem that limits misallocation of resources. Achievements from this project indicate the benefits of innovation might occur if a country’s innovation ecosystem has: 1) the necessary conditions for firms to demand innovation, 2) qualified human capital to supply innovation, and 3) an efficient allocation and accumulation of knowledge capital. This project’s different components addressed these three aspects, that were primarily focused on the demand side, and adding pilot programs that overall promote Peru’s ability to foster long-term innovation-driven productivity growth. To ensure structural impact, the experience in this project suggests that these interventions require a strong collaboration between all actors of the STI system. (ii) Strong and coordinated institutions are critical to (i) a more efficient allocation of knowledge capital and investments within the innovation ecosystem and (ii) sustain policy in the innovation space long enough to achieve tangible results. The impact of innovation policy is contingent upon the institutional strength and stability of the governing body of the STI system, as these institutions are critical to ensuring the sustainability of policy interventions. All project interventions transferred the capacity to CONCYTEC to conduct these activities independently in the future. Through these activities, relevant institutions now have an improved set of tools to use for the effective and efficient design and implementation of innovation programs. Many of the activities need significant time to show their impacts. As the benefits of innovation programs tend to materialize in the medium to long term, it is key that Governments design policy agendas that are forward looking. 13. Assessment Recommended? Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Peru Innovation (P156250) No 14. Comments on Quality of ICR The ICR is comprehensive and consistent with the World Bank guidelines. It provides a detailed narrative of the project context and the achieved results and is generally evidence-based and internally consistent. The results orientation and quality of analysis drew from a theory of change prepared at approval and updated during the ICR exercise, and the quality of analysis was underpinned by the latest data, tables, and annexes that cover key areas of the project interventions.  The procurement function was not sufficiently discussed in the ICR, which is a minor shortcoming. There was also a discrepancy in the outcome rating in the data sheet and the main text. Overall, the quality of the ICR is rated as Substantial. a. Quality of ICR Rating Substantial Page 17 of 17