Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005073 IMPLEMENTATION COMPLETION AND RESULTS REPORT ON AN IBRD LOAN IN THE AMOUNT OF (US$ 95 MILLION EQUIVALENT) TO THE REPUBLIC OF INDONESIA FOR THE RESEARCH AND INNOVATION IN SCIENCE AND TECHNOLOGY PROJECT June 30, 2022 Education Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective Feb 01, 2021) Currency Unit = IDR IDR 9,515.65= US$1 FISCAL YEAR January 1- December 31 Regional Vice President: Manuela V. Ferro Country Director: Satu Kristiina Jyrintytar Kahkonen Regional Director: Daniel Dulitzky Practice Manager: Tobias Linden Task Team Leader(s): Ratna Kesuma ICR Main Contributor: Mar Mar Thwin, Rythia Afkar ABBREVIATIONS AND ACRONYMS ATP Agro Techno Park LPNK Lembaga Pemerintah Non Kementerian (Non-ministerial State Institutes) BAPETEN Badan Pengawas Tenaga Nuklir M&E Monitoring and Evaluation (Nuclear Energy Regulatory Agency) BAPPENAS Badan Perencanaan Pembangunan Nasional (National Development MOF Ministry of Finance Planning Agency) NDP Non-Degree Program BATAN Badan Tenaga Nuklir Nasional OFP Overseas Fellowship Program (National Nuclear Energy Agency) BIG Badan Informasi Geospasial ORAF Operational Risk Assessment (Geospatial Information Agency, Framework formerly National Mapping and Survey Agency) BLU Badan Layanan Umum (General PCN Project Concept Note Service Agency) BMBF Bundesministerium fur Bildung und PDO Project Development Objective Forschung (German Federal Ministry of Education and Research) BPPT Badan Pengkajian dan Penerapan PhD Doctor of Philosophy Teknologi (Agency for Assessment and Application of Technology) BSN Badan Standardisasi Nasional (National PHRDP Professional Human Resource Standardization Agency) Development Project BTC Business Technology Center PMO Project Management Office CPS Country Partnership Strategy PNBP Penerimaan Negara Bukan Pajak (the state’s non-tax revenue) DA Designated Account POM Project Operation Manual DIPA Daftar Isian Pelaksanaan Anggaran Puspiptek Pusat Penelitian Ilmu (Budget Activity List) Pengetahuan dan Teknologi (Center for Research in Science and Technology) DRN Dewan Riset Nasional (National QUE Quality of Undergraduate Research Council) Education DUE Development of Undergraduate RENSTRA Strategic Plan Education RISET Research and Innovation in FDI Foreign Direct Investment Science and Technology FM Financial Management RISTEK The State Ministry of Research and Technology RPJM/N Rencana Pembangunan Jangka FY Fiscal Year Menengah Nasional (Medium- term National Development Strategy) GDP Gross Domestic Product RTI Research and Technology Institute GoI Government of Indonesia R&D Research and Development GTZ Gesellschaft für Technische S&T Science and Technology Zusammenarbeit GmbH (German Technical Assistance Agency) IBRD International Bank for Reconstruction STI Science, Technology, and and Development Innovation IDA International Development Association S1 Bachelors Degree IG Inspector General S2 Masters Degree IMHERE Indonesia Managing Higher Education S3 Doctoral Degree for Relevance and Efficiency KADIN Indonesian Chamber of Commerce and STAID Science and Technology for Industry Industrial Development KIN Komite Inovasi Nasional (National STMDP Science and Technology Innovation Commission) Manpower Development Program KPI Key Performance Indicator TA Technical Assistance LAPAN Lembaga Penerbangan dan Antariksa TF Trust Fund Nasional (National Institute of Aeronautics and Space) LBME Lembaga Biologi Molekular Eijkman URGE University Research for (Eijkman Institute for Molecular Graduate Education Biology) LIPI Lembaga Ilmu Pengetahuan Indonesia WBI World Bank Institute (Indonesian Institute of Sciences) Monevrisbang SINAS Sistem Inovasi Nasional/National Innovation System TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 Higher Level Objectives to which the Project Contributes ............................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9 II. OUTCOME .................................................................................................................... 15 A. RELEVANCE OF PDOs ............................................................................................................15 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................15 C. EFFICIENCY ...........................................................................................................................21 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................25 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................26 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 27 A. KEY FACTORS DURING PREPARATION ...................................................................................27 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................28 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 28 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................28 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................29 C. BANK PERFORMANCE ...........................................................................................................30 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................31 V. LESSONS AND RECOMMENDATIONS ............................................................................. 32 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 34 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 46 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 47 ANNEX 5. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 52 The World Bank Research and Innovation in Science and Technology Project (P121842) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Research and Innovation in Science and Technology P121842 Project Country Financing Instrument Indonesia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Republic of Indonesia BRIN Project Development Objective (PDO) Original PDO To create an enabling policy environment for research and development in science and technology, to improve thepublicresearchanddevelopment institutes’ performance, and to improve science, technology and innovation human resource capacity. Revised PDO To support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). Page 1 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 95,000,000 69,584,160 69,584,160 IBRD-82450 Total 95,000,000 69,584,160 69,584,160 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 95,000,000 69,584,160 69,584,160 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 29-Mar-2013 12-Jun-2013 09-Feb-2016 31-Dec-2020 31-Dec-2021 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 30-Jan-2017 29.55 Change in Implementing Agency Change in Project Development Objectives Change in Results Framework Change in Components and Cost Cancellation of Financing Change in Financing Plan Reallocation between Disbursement Categories Change in Institutional Arrangements 19-Nov-2020 67.58 Change in Results Framework Change in Loan Closing Date(s) Cancellation of Financing Reallocation between Disbursement Categories Change in Implementation Schedule Page 2 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 18-Jun-2013 Satisfactory Satisfactory 0 02 11-Jan-2014 Satisfactory Moderately Satisfactory 4.46 03 02-Aug-2014 Moderately Satisfactory Moderately Satisfactory 7.26 Moderately 04 05-Mar-2015 Moderately Unsatisfactory 8.54 Unsatisfactory Moderately 05 09-Sep-2015 Moderately Unsatisfactory 12.04 Unsatisfactory Moderately 06 11-Apr-2016 Moderately Unsatisfactory 18.85 Unsatisfactory Moderately 07 07-Dec-2016 Moderately Satisfactory 24.55 Unsatisfactory 08 22-Jun-2017 Moderately Satisfactory Moderately Satisfactory 37.45 09 08-Dec-2017 Moderately Satisfactory Moderately Satisfactory 42.45 10 11-Jun-2018 Moderately Satisfactory Moderately Satisfactory 52.26 11 11-Dec-2018 Satisfactory Moderately Satisfactory 56.95 12 17-Jun-2019 Satisfactory Moderately Satisfactory 64.45 13 24-Dec-2019 Satisfactory Satisfactory 66.45 14 25-Jun-2020 Satisfactory Satisfactory 66.45 15 22-Dec-2020 Satisfactory Satisfactory 67.58 16 25-Jun-2021 Satisfactory Satisfactory 71.17 17 08-Dec-2021 Satisfactory Satisfactory 71.17 Page 3 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration - Education 10 Tertiary Education 74 Workforce Development and Vocational Education 11 Other Education 5 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 101 Gender 9 Education 92 Access to Education 23 Science and Technology 23 Teachers 23 Standards, Curriculum and Textbooks 23 ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Manuela V. Ferro Satu Kristiina Jyrintytar Country Director: Stefan G. Koeberle Kahkonen Director: Xiaoqing Yu Daniel Dulitzky Practice Manager: Luis Benveniste Tobias Linden Task Team Leader(s): Dandan Chen Ratna Kesuma ICR Contributing Author: Mar Mar Thwin Page 4 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Country Context 1 1. The Government of Indonesia (GoI) had made innovation a priority area and was already taking important steps towards developing an effective Science, Technology and Innovation (STI) system. At the policy level, the creation of the National Innovation Commission2 (KIN) and the “Master Plan for Acceleration and Expansion of Indonesia’s Economic Development 2011-2025” set the stage for the development of an effective innovation system. By setting “technological and science capacity” as one of three main strategies of the Master Plan (the other two are Economic Corridors and National Connectivity), the Government of Indonesia (GoI) showed its understanding for the need for a coherent STI policy. The creation of the KIN as a high-level independent advisory body that reports directly to the President, highlights the commitment at the highest political level. At the implementation level, KIN had already produced a draft strategy for Science, Technology and Innovation (STI), which had been submitted to the President. In addition, the State Ministry of Research and Technology (RISTEK) had already initiated a nationwide survey of all public research centers (including those in other line Ministries) to identify challenges and devise actions for reform, and a revision of the legal framework for the “National Innovation System” led by the Deputy Minister for STI Institutions at RISTEK. All these initiatives fall under the “Science and Technology” chapter of the Government’s medium-term national development strategy (RPJM), which highlights the importance of strengthening “the quality and utilization of science and technology to support the overall goal of improving the country’s competitiveness and a move towards a more knowledge-based economy”. Sectoral and institutional context 3 2. The Indonesian STI system was underdeveloped, suffering from low availability of resources (especially advanced human capital), low productivity, and low quality/irrelevance of research outputs. The reasons for this low system performance were mainly institutional: i) lack of a coherent policy framework; ii) fragmented institutional set up and fragmented, non-competitive funding that is not linked to performance; and iii) poor performance of the main implementers of Research and Development (R&D) policy (public research institutes or LPNK). 3. The overall resources devoted to R&D in Indonesia were insufficient, and high-level human capital was scarce. Indonesia’s overall public budget allocation to R&D was very low, estimated at only 0.08 percent of GDP, much lower than other countries in the region such as China (1.47 percent), Malaysia (0.6 percent) and Thailand (0.26 percent). 1 This section was obtained from the Project Appraisal Document (PAD) for the Indonesia RISET project dated March 1, 2013 (Report Number 61982-ID). 2 Composed of members from the Ministry of Research and Technology (Kementerian Riset dan Teknologi, RISTEK), non- ministerial research institutes (Lembaga Pemerintah Non Kementerian, LPNK), academia, and private sector representatives. 3 This section was obtained from the Project Appraisal Document (PAD) for the Indonesia RISET project dated March 1, 2013 (Report Number 61982-ID). Page 5 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Higher Level Objectives to which the Project Contributes 4 4. The project aimed to contribute to the higher-level objective of improving Indonesia’s competitiveness in the knowledge economy. The project was aligned with the GoI’s medium term priorities to strengthen the national innovation, by creating a “National Science and Technology System to make a real contribution to people’s welfare and advance civilization through institutional strengthening, network strengthening, research and development improvement and science and technology application system improvement,” as stated in the GoI’s Medium-Term Strategy Document (RPJM). 5. The proposed project aimed to support one of the Country Partnership Strategy (CPS) objectives of enhancing skills and technology, with focus on supporting the reforms of public institutes that are playing important governmental functions related to science and technology or having leading roles in research and innovation. This project also aimed to help GoI in enhancing institutional incentives to drive research and innovation activities into the national priority areas. Key challenges related to Research Development in Science and Technology field 6. As stated in the sectoral and institutional context, the key challenges related to Research Development in Science and Technology field were deterring the development of human resources and the technology transfer in the R&D field. Moreover, these barriers had hampered the country’s economic growth and competitiveness. The RISET project was designed to address these challenges to contribute to the country’s priorities. 7. The low resource allocation for R&D contributed directly to the low investment in human resources and infrastructure for research. The country had a low science base, with 90 researchers per million people, while The Republic of Korea produced 5,802 researchers per million5 After the discontinuation of the large-scale scholarships for graduate studies abroad over a decade ago, the shortage of high-quality researchers, especially of young researchers, was evident. At the public research institutes, LPNKs, less than 5 percent of staff held a Ph.D. degree (S3 level), and only 15 percent hold master’s degrees (S2 level). Most research staff were college graduates (S1 level) and some held even lower qualifications. The picture was only somewhat better for university professors, with 6 percent holding a PhD and 37 percent a masters. Low availability of high-quality human capital was not only a problem of LPNK, but a national problem. This further contributed to the low R&D productivity and quality in Indonesia. 4 This section was obtained from the Project Appraisal Document (PAD) for the Indonesia RISET project dated March 1, 2013 Report Number 61982-ID). 5 https://knoema.com/atlas/Republic-of-Korea/Number-of-researchers-in-RandD?mode=amp Page 6 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Theory of Change (Results Chain) Activities Outputs PDO Outcome Long-Term Impact Component 1 -National Innovation Strategy and Action Plan -NISAP developed, development Technical and policy -Technical and policy study completed, study Legal framework Enabling policy -National Science and reviewed and environment Technology Info system revised, for R&D in -Review and revise legal Institutional Science and framework assessment of LPNKs, Technology -Institutional Institutional assessment of LPNK development plan research centers completed, -Institutional LPNKs reforms development plan piloted, -Piloting reforms by LPNKs Improved country’s Component 2 -Feasibility study for competitive -Feasibility study for competitive funding ness competitive funding system completed, Improved system Research network public R&D + -Establish research established, Institutes’ network -M&E improved, performance More -Improve M&E, Budget system knowledge- Improve effectiveness of improved, based budget system economy + Component 3 -Non-Degree Accelerated -Non-degree Improved STI economic Professional course human professional course opened, growth -Fellowship program resource -Scholarship program capacity implementation being implemented -SC and Technical meeting Component 4 organized regularly, -Steering committee, -Project M&E activities Technical committee, started, PMU, FM, Key assumptions: -project M&E, PMU, FM, The scholars come back after studies abroad and Procurement coordinated, apply knowledge and skills in respective fields. procurement, audit National Audit Office Legends_ blue arrows: directly related, shared audit reports as a red dashes: contributing to longer term impact regular basis Page 7 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 8. As seen in the TOC figure of the original project design, the Component 1 outputs were not quick wins. It required a longer implementation period to achieve the desired outputs. As the Component 1 activities were heavily involved system level inputs, delivering system level outputs was too ambitious to achieve within the project timeframe. The component activities were intended to only reform several Research Agencies/Centers, while the targets of the PDO Indicator 1 was: ‘Increased percentage of LPNK (the Public Research Institution) reform milestones reached’. Components 2, 3, and 4 showed more clarity in the respective result chains. Component 3 was straight forward, and Component 4 focused the project management of Component 1, 2 and 3 to be able to deliver related outputs and immediate outcomes. Therefore, it was clear that Component 1 needed major design changes when after two years of implementation it was not demonstrating achievement of any milestones. Project Development Objectives 9. The Project Development Objective (PDO) was “to create an enabling policy environment for R&D in science and technology, to improve the public R&D institutes’ performance, and to improve STI human resource capacity”. The main beneficiaries were the Ministry of Research and Technology (MoRT or Kementerian Riset dan Teknologi or RISTEK) and seven LPNKs (Public Research Institutes) under its coordination including: BAPETEN (Nuclear Energy Regulatory Agency), BATAN (National Nuclear Energy Agency), BIG (Geospatial Information Agency), BPPT (Agency for Assessment and Application of Technology), BSN (National Standardization Agency), LAPAN (National Institute of Aeronautics and Space), and LIPI (Indonesian Institute of Sciences). The RISET Project was approved on March 29, 2013, and became effective on June 12, 2013, with a closing date of December 31, 2020 and financing of US$95 million IBRD. Key Expected Outcomes and Outcome Indicators 10. The original PDO level result indicators were as follows: i. Increased percentage of LPNK reform milestones reached, ii. Increased share of total LPNK R&D revenue generated from external partners (industry, local community, foreign grant), and iii. Increased percentage of Master’s or Ph.D. degree holders in LPNK and RISTEK. Original Components 11. Component 1: Improving Innovation Policy Framework and Performance of Public Research Centers (US$ 4 million) This component aimed to focus on 1) Improving Innovation Policy Framework, and 2) Revitalization of Public Research Centers. The project aimed to support Improving Innovation Policy Framework activities by RISTEK focusing on coordination and consensus building on a national innovation strategy and action plan, technical and policy study, capacity building, development of national science and technology information system, and review/revise the legal framework for science, technology, and innovation. The project aimed to support Revitalization of Public Research Centers activities undertaken by RISTEK through institutional assessments of selected LPNKs’ research centers, developing institutional development plans, and piloting reforms by selected LPNKs. Page 8 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 12. Component 2: Strengthening Public Research Funding (US$ 4 million) This component supported RISTEK to undertake a feasibility study for, and design of, a competitive national research funding system, to strengthen existing institutional research funding program, to establish industrial and global research linkages for science and technology researchers, to improve monitoring and evaluation of publicly funded research, and to improve the effectiveness of science and technology planning formulation and budgeting system. 13. Component 3: Developing Science and Technology Human Resources Capacity (US$ 80 million) This component supported RISTEK to implement (a) Fellowship Program; and (b) Non-degree Professional Course and Training Program through a competitive selection of candidates, provision of pre-departure programs, and provision of Scholarships for Ph.D. Degree studies and Master’s degree studies. 14. Component 4: Project Management (US$ 7 million) This component supported the operation of the Steering Committee, Technical Committee and Project Management Office through project coordination, project monitoring and evaluation, procurement, financial management and other fiduciary aspects of project implementation, project advisory activities, and providing a re-entry program for scholarship recipients. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION 15. On February 21, 2017, a Project Level 1 restructuring was undertaken to improve the project focus of the PDO and specific outcomes, and to respond to institutional changes. As a result, the restructuring (i) revised the Project Development Objective (PDO); (ii) revised the results framework and associated indicators to align them with the revised PDO; (iii) modified the project components to reflect a reduced scope of project activities; (iv) introduced changes as a result of the merger of the MoRT and the Directorate General of Higher Education of the Ministry of Education and Culture (MoEC) to form the Ministry of Research Technology and Higher Education (MoRTHE), which was the Executing Agency of the project, and (v) cancelled US$15 million of the loan, reducing it to US$80 million, to reflect modifications to project activities. The cancellation was requested by the Government of Indonesia because some of the allocated budget for Component 1: Strengthening the Innovation System and Technology Transfer and Component 2: Strengthening Public Research Funding was not going be used as planned. 6 16. In November 2020, the RISET project experienced a Level 2 restructuring that mainly focused on the following: (a) extending the closing date from December 31, 2020 to December 31, 2021 to allow the implementation of NDP program for the additional 500 new participants; and (b) cancelling US$ 5,236,621.27 of unused funds. With this one-year extension, the total period of the project implementation was nine years. The revised PDO and the Result indicators remained unchanged during the second restructuring. 17. The ICR recognizes the Government’s willingness to acknowledge the shortcomings of the original 6 Second Restructuring Paper, Report No. RES43948, November 2020 Page 9 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) design and their flexibility to change the design for the success of achieving the project outcomes. It was appropriate to make the course correction as required. Otherwise, the RISET project may not have achieved its original outcomes. Revised PDOs and Outcome Targets 18. The revised PDO statement was to support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). The revised PDO was designed to better reflect the scope of the project's activities within the project timeframe and dropped the original objective to create the broader enabling policy environment. It also highlighted that the main objective of the project is to improve human resource capacity at LPNKs and at MoRTHE, as this constituted 84 percent of the loan. The main programs under the refocused Components 1 and 2 were related to: (i) technology transfer, in terms of the development of TTOs and the design of Science and Technology Parks (STP); (ii) development of a management information system (MIS) and database for STI; (iii) support for the implementation of the government's improved output-based unit cost methodology; and (iv) helping to increase coordination among research funding agencies and to improve international networking. The revised PDO was not committed to, but contributed to the broader policy environment and institutional changes which were beyond the project scope. In alignment with the revised PDO, a set of new and revised PDO indicators were defined with a focus on measurablity. Revised PDO Indicators 19. First restructuring: Among the three original PDO indicators, two were revised, one dropped, and four new indicators were added in alignment with the revised PDO statement. See Table 1 for rationale of the changes. Out of 14 IR indicators and 8 sub-indicators, one indicator remained unchanged, 13 indicators were dropped, and 8 sub-indicators were dropped. The scope was relatively reduced for the revised Results Framework and the sub-component activities were refocused in alignment with the revised RF indicators. Table 1. Changes in PDO Indicators and Rationale at the First Restructuring PDO Indicators Rationale Original in the PAD Revised as “Increased % of Masters’ or “Increased % of Masters’ and To focus on the fellowship Ph.D. degree holders in LPNK Ph.D. degree holders in LPNK program within the project and RISTEK” and MoRTHE supported by scope the RISET Project” “Minimal annual share of “Improved regulation on the To narrow the focus on more RISTEK’s institutional research output-based cost standard tangible project outcomes funding to projects with for the STI research funding rather than a longer-term industry co-implementation” system is issued on an annual impact which is not feasible basis” within the time frame of the project Page 10 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) New PDO Indicators added Rationale “LPNK research agency reforms are institutionalized” by To focus activities on December 2020 institutionalizing the reform of selected units – TTOs being the front office of the LPNKs and research agencies being the back offices “Masters degree holders who participate in applied science To strengthen the outcome of research programs increased” from zero baseline to 90 percent STI human resource capacity end target enhancement programs “Ph.D. degree holders who manage or coordinate applied To support the objective of science research programs increased” from zero baseline to 90 improving STI human resource percent end target capacity Dropped Indicators Rationale Percentage of LPNK reform milestones reached LPNK reform was too large to be achievable within the project life because each of the 7 LPNKs consisted of 6-25 research agencies. Share of total LPNK R&D revenue generated from external The statement indicated a partners (industry, local community, foreign grant) longer-term impact which was not feasible within the time frame of the project. Revised Components, Costs and Fund Cancellation First restructuring 20. In alignment with the revised PDO statement, the objectives/outcomes of the Components 1 and 2 were focused on supporting the government in strengthening technology transfer and institutional functioning and data management of LPNKs respectively). A new set of activities were identified as agreed during the MTR including: a. Technology transfer, in terms of the development of TTOs and the design of Science and Technology Parks (STP) (Component 1); b. Development of a management information system (MIS) and database for STI (Component 1); c. Support for the implementation of the government's improved output-based unit cost methodology (Component 2); and d. Helping to increase coordination among research funding agencies and to improve international networking (Component 2). 21. A portion of the loan in the amount of US$15 million was cancelled to reflect modifications to project activities. The cancellation was requested by the Government to reflect scaled back activities under components 1 and 2 where the fund allocation was decreased US$1.5 million each. For Component 3, the fund allocation was reduced to US$68 million to reflect the reality because it was calculated at a Page 11 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) higher cost at the time of preparation. But the project retained the allocated percentage for Component 3 as 85 percent of total funding. See Table 2 for the changes of component percentage in the PAD and Restructurings. Second Restructuring 22. Through MoRT, GoI proposed a project extension of one year until December 31, 2021, with a focus on financing NDPs (Component 3b) and management support for Component 4. This was a level 2 restructuring, and the project was able to increase the NDP target with 500 additional people and to support COVID-19 pandemic mitigation. 23. From the US$10,886,621.27 of estimated unspent funds by December 31st, 2020, the project used US$ 5,650,000.00 to finance NDPs and project management activities, including M&E, reporting, and dissemination activities during GoI FY of 2021. The amount of fund cancellation at the second restructuring was US$5,236,621.27. The total fund cancellation in both restructurings reached US$ 20,236,621.27. Table 2: Revised Components and Costs Components Component Component Actual Component Components (Revised at the Costs (First Costs (Second Expenditure Costs (PAD) (Original PAD) First Restructuring) Restructuring) (8 Dec 2021 (US$) Restructuring) (US$) (US$) Latest IFR) 1) Strengthening 1) Improving the innovation Innovation Policy system and 4,000,000.00 2,500,000.00 2,371,238.74 2,376,397.47 Framework and technology (4.2%) (3.1%) (3.2%) (3.4%) Performance of Public transfer at Research Centers public research agencies 2) Strengthening Public 4,000,000.00 2,500,000.00 2,067,145.29 1,902,354.47 Research Funding (4.2%) (3.1%) (2.8%) (2.7%) 3) Developing Science 80,000,000.00 68,000,000.00 64,753,924.85 60,338,721.42 and Technology Human (84.2%) (85%) (86.6%) (86.7%) Resources Capacity 7,000,000.00 7,000,000.00 5,571,069.85 4,966,687.08 4) Project Management (7.4%) (8.8%) (7.4%) (7.1%) Total project 95,000,000.00 80,000,000.00 74,763,378.73 69,584,160.44 cost Fund 15,000,000.00 Cancellation 5,236,621.27 Page 12 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Other Changes Institutional Arrangements 24. The merger of the Ministry of Research and Technology with the Directorate General of Higher Education of the Ministry of Education and Culture in 2015 formed MoRTHE. For this reason, the term RISTEK (Ministry of Research and Technology) was revised to MoRTHE (Ministry of Research, Technology, and Higher Education) in the project documents and in the amended Loan Agreement. Specifically, the MoRTHE director general (DG) offices have been overseeing the following components: Component 1 was under the DG of Institution, Component 2 was under DG of Research and Development, and Components 3 and 4 were under the DG of Resources for Science, Technology, and Higher Education. The Project Management Office was also under the supervision of the DG of Resources for Science, Technology, and Higher Education. Rationale for Changes and Their Implication on the Original Theory of Change 25. The project’s Theory Change was primarily to address the key challenges of the country’s R&D field scarcity of the high-level human capital and low performance to be able to contribute towards Indonesia’s longer-term goal of improved country’s competitiveness, more knowledge-based economy and accelerated economic growth. 26. The project’s original design aimed to create enabling policy environment that will support the system level innovation and changes in R&D along with the increased STI human resource capacity in R&D in Science and Technology. During the initial project years, it became evident that the policy and institutional changes were beyond the purview of MoRTHE and the LPNKs. Therefore, the project required a first restructuring as discussed above. 27. The justification for the RF changes was to ensure that: (i) the indicators were properly aligned with the revised PDO, ensuring that the indicators could adequately track and report progress towards achieving the project's objective; (ii) the new results framework reflected the revised project activities and reduced scope as a result of initial project delays and the US$15 million cancellation; and (iii) results could be reported within the given project timeline. 28. Although the scope of components 1 and 2 were narrowed with funding reduced from 4.2 percent of the project total cost to 3.1 percent, the revised activities and the expected results will still contribute towards the enabling policy and institutional environment in a long run. The project’s primary focus on STI human resource capacity development (component 3) was the largest investment of the RISET project with 84.2 percent of the project total cost allocated at the project preparation and increased to 85 and 86.6 percent at the first and second restructurings. The actual expenditure was 86.7 percent at the project closing. Component 4 fund allocation was originally 7.4 percent of the total cost, and the actual expenditure percentage was 7.1. The revised PDO and RF indicators, revised project Page 13 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) components and activities were still attributable to the long-term impact of the accelerated economic growth and knowledge-based economy. In conclusion, the project’s Theory of Change was still valid and not seriously affected by the restructurings, but the revised PDO outcomes became more specific and realistic. The ICR notes the first restructuring brought about the right course correction at the right time. Page 14 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 29. This ICR assigns the rating of the Relevance of the PDOs as High because of the following. 30. The 2020 SCD Update 7 clearly stated that nurturing world-class human capital and strengthening the competitiveness and resilience of the economy were two pathways out of four to poverty reduction and shared prosperity in Indonesia. The RISET project’s objectives of improving STI human resource capacity and strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs) were directly attributed to the World Bank’s twin goals. 31. At appraisal, this project was in alignment with the Country Partnership Strategy (CPS) 8 objectives of enhancing skills and technology, with a focus on supporting the reforms of public science and technology institutes having leading roles in research and innovation. This project has been supporting GoI in enhancing institutional incentives to drive research and innovation activities into national priority areas. Besides, the project was still relevant with the new CPF for 2021-2025 (IBRD/IFC/MIGA). 9 In alignment with the government’s medium term Development Plan 2020-2024, the World Bank’s CPF Engagement Area 3 included “Nurture Human Capital” and it was one of the core agenda of the government development strategy “Increasing the Quality and Competitiveness of Human Resources”. 32. In the MoEC’s roadmap for education 2020-2035 and in the National Research Master Plan (2017- 2045), it stated that the Indonesian education system will be adapted to change for the future jobs that demand disruptive technology and the excellent human resources. The RISET project served as the forerunner of the government priorities and policies in education and economic growth. 33. The RISET project’s objectives were aligned with the GoI’s National Development Plan (which was published in 2017 after the project was approved) 10 as it was meant to increase the productivity and increase competitiveness in international markets so that the people of Indonesia can match economic advancement in other Asian nations. B. ACHIEVEMENT OF PDOs (EFFICACY) 34. This ICR assigns the split rating for the achievement of the PDO as the project experienced significant changes in the project components, activities and the project funding after the first restructuring. The ICR assesses PDO outcomes before and after the first restructuring: the achievement of the original 7 Systematic Country Diagnostic Update 2020, Indonesia 8 Report No. 99172, Country Partnership Strategy FY 2015-2020) 9 Report No. 157221-ID, Country Partnership Framework for the Republic of Indonesia (FY 2021-2025) 10 Regulation of the Minister of National Development Planning/Head of National Development Planning Agency, 2017 Page 15 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) PDO indicators for the period of 2013-2016, and the revised PDO indicators for the period of 2017- 2021 after the first restructuring which occurred in 2017. Project Achievement during the period of 2013-2016 before the first restructuring 35. The original PDO was “To create an enabling policy environment for research and development in science and technology, to improve the public research and development institutes’ performance, and to improve science, technology and innovation human resource capacity.” Three key outcomes were unpacked from the PDO statement: 1) enabling policy environment for R&D created, 2) the performance of LPNKs improved, and 3) STI human resource capacity improved. The achievement of the PDO was assessed through PDO indicators: PDO indicator 1 was to be assessed for the enabling policy environment, and PDO indicator 3 was to be assessed for the STI human capacity. The ICR noted that PDO outcome 2 -the performance of LPNKs improved- was not attributable to the PDO indicator 2 which was “Share of total LPNK R&D revenue generated from external partners (industry, local community, foreign grant)”. PDO outcomes unpacked PDO indicators Remark 1) enabling policy “Percentage of LPNK reform PDO outcome and PDO environment for R&D milestones reached” indicator were relevant and created matched. 2) the performance of “Share of total LPNK R&D PDO outcome and PDO LPNKs improved, revenue generated from indicators were mis-matched. external partners (industry, The performance of LPNKs to be local community, foreign assessed through other outputs. grant)” 3) STI human resource “Increased % of Masters or PDO outcome and PDO capacity improved Ph.D. degree holders in LPNK indicator were relevant and and RISTEK” matched. Outcome 1: Enabling policy environment for R&D created 36. The ICR assessed the achievement of this outcome through PDO indicator 1 which was Percentage of LPNK reform milestones reached. No specific target was set for 2016, but the 2017 target was 60 percent which was 40 percent higher than the baseline. Supportive activities were carried out then, even though they were limited in number. Since a LPNK was a very large institution consisting of many research agencies, it was hard to report in 2016. But some activities were implemented, and the ICR gathered key outputs as follows: Improving the Innovation Policy Framework The SINAS (Sistem Inovasi Nasional/National Innovation System) Concept was strengthened through discussions formulating the SINAS model involving all stakeholders (MoRT, LPNKs, relevant ministries, local government, universities). It was agreed that the SINAS covered three areas: (i) Science and Technology (S&T) Parks, (ii) Consortia, and (iii) Local Innovation System development (SIDa) in 2014. In alignment with the vision, mission, and targets in STI of the new elected Government, the Page 16 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) establishment of 100 STPs was completed by 2014. In 2015, the project started to support the reform of STP initiating the development of the STP model in Agriculture. Outcome 2: The performance of LPNKs improved 37. This was reported as an output, not as a PDO indicator. With regard to the revitalization of public research centers, the project supported the transformation of two research agencies in BATAN into Centers of Excellence. These are the Center for Radio Isotope Technology and Radio Pharmacology (Pusat Teknologi Radio Isotop dan Radio Farmaka – PTRR) and the Center for the Application of Radiation Isotopes (Pusat Aplikasi Isotop Radiasi – PAIR). PTRR has now been classified as a Center of Excellence. The project also supported the development of IDPs for revitalization of TTOs and their Back Office (BO) units in LAPAN, BATAN, BPPT, and LIPI. 38. PDO Indicator 2: The data collected showed that the actual revenue generated by the LPNKs with external partners was lower than the baseline, even though in-kind collaboration of LPNKs with external parties had been increasing. The fluctuation of the income generated was influenced by the extent and willingness of the private sector to collaborate with LPNKs, which was beyond the control of the project. Outcome 3: STI human resource capacity improved 39. PDO Indicator 3: Progress against this indicator was evident since 2013. The project had already selected the candidates for scholarship program and 279 scholars (84 Master’s and 195 Ph.Ds) were studying in universities abroad. In October 2016, 9 Masters and 1 Ph.D. graduates have completed their studies. The baseline and end target for PDO Indicator 3 at the time of appraisal was considered to include graduates from all scholarship programs who were working in the LPNKs and RISTEK. In 2013, 15% of the staff in LPNKs and RISTEK were the Masters’ and Ph.D. graduates. The reporting of 24% achievement represented the diverse scholarship programs. Therefore, the revision of this indicator was required to focus the results within the RISET project scope. Table 3: PDO Indicators and Achievements (Original PAD) PDO Indicator Baseline 2015 Actual in Sub-Rating Target 11 Oct 2016 PDO Indicator 1: “Percentage of LPNK reform 0% 20% 0% Modest milestones reached” PDO Indicator 2: “Share of total LPNK R&D revenue <5% No 4% generated from external partners (industry, local change Negligible community, foreign grant)” PDO Indicator 3: “Increased % of Masters or Ph.D. 15% 17% 24% Modest degree holders in LPNK and RISTEK” 11 Baselines and 2015 targets were obtained from the original PAD Result Framework Page 17 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 40. Corrective measures were taken as part of the first restructuring to achieve the desired results including developing project action plans for the remaining life of the project, assigning staff and consultants to support project implementation, and revising the Project Operation Manual (POM) to accelerate overall activity implementation. Monitoring and Evaluation (M&E) improved by undertaking the pre- Mid Term Review (MTR) work and providing timely project evaluation results. The disbursement total before the first restructuring was US$29,551,344.88 (as of 19 December 2016). Project Achievement after revision of the PDO and its indicators (2016-2021) 41. The revised PDO aimed to support the government in: (1) improving STI human resource capacity; and (2) strengthening technology transfer, institutional functioning, and data management of public research agencies (LPNKs). It was unpacked into four outcomes to be assessed for the achievement of the project: (i) technology transfer strengthened, (ii) institutional functioning strengthened, (iii) STI human resource capacity improved, and iv) data management of public research agencies (LPNKs) strengthened. Among the four outcomes, strengthening of data management of public research agencies (LPNKs) was reported as a Component 1 Intermediate indicator, and not as a PDO indicator. Table 4: Revised PDO Indicators and Achievements (PDO Indicators after 1st Restructuring) PDO Indicator Baseline End Target End Comment Achieve- ment Outcome 1: Technology transfer strengthened PDO Indicator 1: “LPNK research agency reforms are 0 Institutiona Institution Achieved institutionalized” lized alized Outcome 2: Institutional functioning strengthened PDO Indicator 2: “Improved regulation on the 0 Refined Complete Achieved output-based cost standard for the STI research regulation d in funding system is issued on an annual basis” is issued December based on 31, 2020. annual review covering research priorities Outcome 3: STI human resource capacity improved PDO Indicator 3: “Increased percentage of Masters’ 0 100 % 83 % Mostly and Ph.D degree holders in LPNKs and MoRTHE achieved supported by the RISET Project” PDO Indicator 4: “Master’s degree holders who 0 90% 100% Over- participate in applied science research programs” achieved PDO Indicator 5: “Ph.D. degree holders who manage 0 90 % 96 % Over- or coordinate applied science research programs” achieved Page 18 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Outcome 4: Data management of public research agencies (LPNKs) strengthened Component 1, IR 2: National Information System for 0 STI data is STI Achieved STI is established and integrated into ministry’s continuous database database system ly being was collected, expanded, and the utilized database is and being integrated expanded into to include Ministry’s national, system. local governmen t, and university STI data. Assessment of the achievement of each outcome Outcome 1: The technology transfer strengthened (sub-rating as Substantial) 42. This ICR notes the key achievements for this outcome through PDO Indicator 1 activities. This outcome was fully achieved as the LPNK research agency reforms were institutionalized as envisioned. This enabled the LPNKs to strengthen the institutions that serve the function for technology transfer, through preparation and implementation of regulations relevant to technology commercialization in LPNKs. The number and quality of inventions increased. The key achievements of this outcome included: a) Policy papers on STI legal framework and innovation funding system as well as commercialization carried out and have informed a number of regulations and implementation b) Reforms institutionalized in five Science Techno Parks (in three LPNKs) with the issuance of MoRTHE Regulation No. 25/2019 on STP Governance. These STPs were: i) STP Polewali Mandar, West Sulawesi (BATAN), ii) STP Pelalawan, Riau (BPPT), (iii) STP Cibinong, West Java (LIPI); (iv) STP Pasar Jumat Jakarta, DKI (BATAN); and v) STP Cimahi, West Java (LIPI). The two STPs in Pasar Jumat Jakarta and Cimahi had especially requested to be included in the pilot. c) TTO Management Guideline and the technology valuation and incentive system guideline and SOP revised and aligned with the new Law No. 11/2019 on National System of Science, Technology and Innovation. A deeper analysis of this law has been carried out, particularly on relevant articles on technology commercialization process and implementation. d) Thirty (30) TTO staff were certified as Registered Technology Transfer Professionals (RTTP) by ATTP (Alliance of Technology Transfer Professionals). e) Policy papers informed the Law No. 11/2019 on National System of Science, Technology and Innovation. f) Strategy action plans were submitted for consideration to the new national research and innovation master plan. Page 19 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Outcome 2: Institutional functioning strengthened (sub-rating as Substantial) 43. This outcome was achieved through PDO indicator 2 activities that brought the improvement of a regulation on the output-based cost standard (PMK 106/2016, DG of Budget Regulation No. Per- 15/PB/2017, Permenristekdikti No. 42/2016, and Perpres No: 16/2018 and Permenristekdikti No: 20/2018). DG Regulation No: PER-7/PB/2019 has been issued to replace DG Regulation No: PER- 15/PB/2017. Outcome 3: STI human resource capacity improved (sub-rating as Substantial) 44. PDO Indicator 3 was not achieved at the time of ICR reporting but will likely be achieved in 2023 after the project closing date. Because of COVID pandemic, their studies were disrupted. As of November 2021, of 457 scholars, 378 (83 percent) have successfully completed their studies, and 17 (4 percent) have dropped out. 62 (14 percent) scholars remain active and will aim to complete their studies after project closure, using university scholarships or their own funding. Reasons for delayed study progress include change of thesis/dissertation supervisors; lengthy approval of thesis/dissertation topics; COVID- 19 pandemic challenges (disruption of classes and labs); and health and family reasons. Reasons for drop- out include failing in some subjects, health problems, and non-approval of thesis/dissertation topics by supervisors. 45. PDO indicator 4 was over-achieved with 236 Master’s degree holders (100 percent) taking research posts aligned with their fields of study within their respective LPNKs. 46. PDO indicator 5 was also over-achieved with evidence that 131 Ph.D. graduates out of 137 (96 percent) were leading research works in their fields of study within their respective LPNKs. The remaining six PhD Degree holders have been waiting for job assignment and positions that are aligned with their fields of study. 47. The total number of scientific publications was 699 articles (480 produced by Ph.D. graduates and 220 by Master’s graduates), through:  382 indexed publications (294 Ph.D. and 88 Master’s).  318 unindexed publications (186 Ph.D. and 132 Master’s).  27 Intellectual property/patents (9 Ph.D. and 18 Master’s).  62 publications approved for the Publication Allowance, including 45 in Scopus indexed international journals, 7 in non-globally indexed international journals, and 10 as international proceedings. Outcome 4: Data management of public research agencies (LPNKs) strengthened (sub rating as Substantial) 48. STI data was continuously being collected, and the database was being expanded to include national, local government, and university STI data. The process was continued through the establishment of Monevrisbang. The database has been utilized and integrated into Ministry’s data base system. Key results included increased number of research programs in LPNKs and MoRTHE with focus on higher ranks of the Technology Readiness Level (TRL), strengthened R&D monitoring & evaluation capacity, increased number of research programs of higher TRL, and twelve independent evaluations of multi-year research activities were completed. Page 20 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Justification of Overall Efficacy Rating 49. The ICR assigns a Modest rating for the original PDO achievement for the period of 2013-2016 as discussed in paragraphs 34, 35, 36, 37 and 38. The ICR notes that the original PDO statement was quite ambitious and too broad to achieve within the project time frame, and the indicators were subject to external influences that were beyond project interventions. The ICR assigns a Substantial rating for the project period of 2017-2021. After the first restructuring, the RISET project made steadfast progress under the revised components and activities. By the end of the project in December 2021, out of Five PDO indicators, PDO indicators 1 and 2 were fully achieved, PDO indicator 3 was not achieved, and PDO indicators 4 and 5 were over-achieved. Although the project experienced implementation delays and shortcomings during the initial years, the project objectives were largely achieved after two restructurings. 50. Based on the World Bank’s ICR guidance note, this ICR requires a split rating because the original PDO and its outcome indicators were revised at the first restructuring. The ICR takes into consideration both the original and revised objectives and key associated outcome targets in deriving the project’s overall outcome rating by assessing achievements across the entire project timeframe. C. EFFICIENCY Assessment of Efficiency and Rating 51. An economic analysis using cost-benefit analysis (CBA) was performed to determine the level of efficiency attained by the project. The CBA compares economic benefits and costs “with” and “without” the RISET Project interventions to arrive at the present value of net benefits (NPV) and internal rate of return (IRR). The benefits include the social return from a strengthened national innovation strategy, strengthened public research funding, and the increased human resources capacity in science and technology. The costs consist of direct costs in financing the project activities under different components which also includes project management, and indirect costs that include human capital depreciation, costs associated with drop out students, and opportunity cost associated with undisbursed/cancelled project funds. 52. Realized data on economic benefits for different R&D interventions is not systemically available since there is a lag of several years of after the project closure to have such data available. Hence, this economic analysis leverages from the findings of the economic research literature that estimate social rates of return to R & D spending between 200 to 300 percent 12. The recent estimate of gross returns to public R&D in Southeast Asian countries is also found to be significantly large, being about 252 percent 13. 12 Griffith, Redding, and Van Reenen (2004), Goni and Maloney (2017). 13 The estimates of gross returns to public R&D in Southeast Asian countries is significantly large, ranging from 252% to 334%. Note that the difference results from using different estimation techniques and assumptions placed on main variables (SAHEP Project Appraisal, 2017). To be conservative, we choose the lowest bound of the estimates. Page 21 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) The analysis uses the low-range value of 200 percent return. This economic analysis also assumes that 30 – 40% of financing for scholarships and fellowships will successfully generate social return after four years (achievement rate). This assumption is considered conservative given the number of achievements of the graduates that have been identified at the project closure 14. 53. Meanwhile, the costs of the project consist of the direct cost in financing the project activities under different components which also include project management. The current value of direct costs of the disbursed funds is about US$88.6 million. There are also indirect costs which come from human capital depreciation, costs associated with drop out students, and opportunity costs associated with undisbursed/cancelled project funds. The largest amount of the indirect costs is the opportunity cost of the cancelled amount, which is assumed to be directed to other productive projects. In the base case scenario, the opportunity cost is US$21.1 million, the cost due to human capital depreciation is US$ 4.6 million, and the costs associated with drop-out students is US$2.6 million, with the total of indirect costs is US$28.4 million. 54. The analysis shows that in a base-case scenario under conservative assumptions, the RISET project investment generated NPV benefits of about US$18.5 million and IRR of 12 percent, thereby providing evidence that the RISET Project was productive and economically justifiable. The details of the economic analysis appear in Annex 4. The estimates are considered to be conservative and low-bound given that other benefits associated with spillover effects of R&D products such as positive externalities on socio- economic indicators such as more shared prosperity (equity) and increased future salaries of the human resources are not reflected in the analysis. Nevertheless, the economic analysis provides strong evidence that the RISET project investments have been productive and economically profitable. 55. Besides the conservative assumption, the analysis looks at two additional scenarios under different assumptions. The risks and the expected NPV of the project are assessed with the assumption of different social rates of returns of the RISET project investments, different social rates of return of the restructured funds in alternative projects, and different achievement rates of the scholarship and fellowship financing. The first scenario assumes that the achievement of graduates in generating social return of their research is 10 percent higher than the base-case scenario. With this assumption, RISET project investments result in NPV of US$31.2 million and IRR of 16 percent. The second scenario assumes that the RISET restructured funds would be redirected to highly productive projects of which the social rate of returns would be 70 percent. This assumption increases the NPV to US$27.1 million and IRR to 14 percent. 14 RISET alumni (both in degree and non-degree programs) in LIPI, BPPT, and LAPAN have initiated and involved in various research projects such as marine biology, creating earthquake resistance building material, digital transformation for COVID response, creating bone implants, coordinating the national aircraft development (among many others). Page 22 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Table 5: Summary of Economic Costs and Benefits of the RISET project (US$, present value) Costs and Benefits (PV, USD) Base-case Case 1 Case 2 scenario 47,045,130 59,697,901 47,045,130 Gross economic benefits of RISET R&D investments 28,480,388 28,480,388 17,855,865 Economic costs - of project activities 88,686,299 88,686,299 88,686,299 - of depreciation 4,613,857 4,613,857 4,613,857 - of undisbursed/cancelled funds (opportunity 21,249,044 21,249,044 10,624,522 costs) - drop out students 2,617,487 2,546,245 2,546,245 NPV 18,564,743 31,217,514 27,115,504 IRR 12% 16% 14% Social rate of return of FIRST R&D investments 200% 200% 200% Social rate of return of cancelled funds in alternative projects 35% 35% 70% Achievement rates of the scholarship and fellowship financing 30% 40% 30% 56. The external benefits of the project include social benefits that spillover to benefit others in the society and future generations. A number of alumni now teach and have become academic supervisors for post- graduate students in universities across Indonesia such as in ITB (Institut Teknologi Bandung/Bandung Institute of Technology), ITERA (Institut Teknologi Sumatera/Sumatera Institute of Technology), IPB (Institut Pertanian Bogor/Bogor Agricultural University), Sampoerna University, UGM (University of Gajah Mada Yogyakarta), and UNILA (Lampung University) which would support the improvement in quality of teaching in university. Furthermore, a number of alumni have actively become resource people in scientific seminars/webinars such as for Indonesian Statistic Agency, UNESCO forum for assistive technology for people with disability, and G-20 events. 57. Other aspects to efficiency that need to be considered is aspect of design and implementation. While some aspect of design and implementation are hard to quantify (such as frequent staff turn-over, structural change in executing agency, slow procurement and disbursement), the efficiency analysis presented here has taken into account design and implementation aspects, such as the opportunity costs that comes from loan cancellation and drop-out students. 58. During the implementation, the project has taken important decisions which promote efficiency. Fund cancelations that were part of the first restructuring were intended to accommodate the changes in the project design and outcomes. Two PDO indicators were dropped because of external factors that were beyond the scope of the project. New indicators that are more relevant and measurable were added to ensure the project could achieve its objectives and outcomes. The experience in the first years of project implementation has also allowed for more accurate calculations of the funding that was needed to achieve the objectives of the project. After the first restructuring, the project progressed significantly and achieved most of its targets. Moreover, the project extension that was done through the second restructuring was Page 23 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) not because of inefficient implementation but was intended to optimize project funds by increasing the number of Non-Degree participants in response to the COVID pandemic. 59. Furthermore, the largest proportion of project funds was disbursed under Component 3 which provided financing for scholarships for degree and non-degree programs which covered tuition fees, travel, living and book allowances, and research and journal publication allowances (as approved by research supervisor/university). All cost items followed the regulation and guidance from the Ministry of Finance on the government standardized unit cost 15. The scholarship amount of RISET is shown comparable with scholarship amounts under the current government scholarship scheme, LPDP (Lembaga Pengelola Dana Pendidikan or Indonesia Endowment Funds for Education) as well as other scholarship programs such as the Australian award scholarship, Fulbright scholarship, and Erasmus mundus program (Table 6). Furthermore, in the appraisal, Component 3 was allocated US$80 million to finance 1,264 beneficiaries. While at project closure, Component 3 spent US$60 million providing scholarships and fellowships to 3,026 beneficiaries. This shows that during the implementation, the project efficiently reached more beneficiaries with lower costs. In addition, the government gets more value added with the scholarships under this project since it brought about a new way of design and administration, and more importantly, its impacts were better and consistently monitored through the result framework. Table 6. Comparison of scholarship amount of RISET and other scholarship programs (*source 16) Country Currency RISET LPDP Other scholarship programs Living Settling Living Settling Living Settling (monthly) (once) (monthly) (once) (monthly) (once) Australia AUD 2,291 3,894 2,000 4,000 2,300 5,000 England (London) GBP 1,569 2,668 1,600 3,200 1,134 1,100-1,400 (49,000 for the France/Germany/ full 2-year program include Belgia/Italy EUR 1,593 2,708 1,100-1,300 2,200-2,600 tuition fee) Japan JPY 192,649 327,503 170,000 340,000 143,000-145,000 USA 35,000/year for master student NYC/Boston USD 1,798 3,057 2,200 4,400 & 40K/year for PhD student (including for tuition fees) Other cities USD 1,667 2,834 1,500 3,000 Switzerland CHF 2,198 3,737 2,050 4,100 15MoRTHE decree no. 180-A/D3.1/PG/RISET-Pro/VII/2017 16https://www.scholarsofficial.com/beasiswa-aas/penjelasan-rincian-pembiayaan-beasiswa-aas-australia-awards-scholarship/, https://www.simonev.lpdp.kemenkeu.go.id/assets2/file/Buku_Panduan_Keuangan_LPDP_01032021.pdf, https://www.aminef.or.id/grants-for-indonesians/faq/, https://www.studyinjapan.go.jp/en/planning/about-scholarship/, https://fulbrightscholarship.net/erasmus-mundus-scholarship-2022/, https://erasmus-plus.ec.europa.eu/programme- guide/part-b/key-action-2/erasmus-mundus Page 24 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 60. Another aspect that is important to highlight is that the financial report of the project has always been reviewed by BPK (Audit Board of the Republic of Indonesia) each year. Despite the audit findings of ineligible expenditures in FY2017 and FY2018, the project’s financial report consistently received a clean opinion from BPK. The Project Management Unit had taken immediate follow up actions which included replacement of the assistant treasurer and the decision-making officer and assigning additional verification staff. All ineligible expenditures 17 were refunded. The status of clean audit report can be considered as a proxy of proper use of resources and overall efficiency of the project. 61. With the economic analysis and aspect of design and implementation explained above, the project efficiency is rated substantial. D. JUSTIFICATION OF OVERALL OUTCOME RATING 62. The overall outcome rating for project is assigned as Moderately Satisfactory. The split rating application is explained in the table below. Table 6: Application of Split Rating 18 Original objectives Revised objectives and targets without and/or targets with restructuring restructuring Relevance of PDO High Efficacy (PDO) Modest Substantial Indicator 1 Modest Indicator 2 Negligible Indicator 3 Modest Indicator 1 Substantial Indicator 2 Substantial Indicator 3 Modest Indicator 4 Substantial Indicator 5 Substantial Efficiency Substantial 1 Outcome Rating Moderately Unsatisfactory Satisfactory (H + M + S) (H + S + S) 2 Numerical value of the 3 5 outcome ratings 3 Disbursement US$29,551,344.88 US$40,032,815.56 (Disbursed as of 19 Dec (Disbursed from 19 Dec 2016 to 2016) 3 June 2021) 19 17 The total amount of ineligible expenditures is US$ 1,908,949 (Source: annual project audit reports by BPK & CMU communications to MoF on refund request of ineligible expenditures) 18 Bank Guidance, ICR for IPF Operations, March 2020 19 Data obtained from the portal Page 25 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Original objectives Revised objectives and targets without and/or targets with restructuring restructuring 4 Share of disbursement 0.42 (42.47%) 0.57 (57. 53%) 5 Weighted value of the outcome 3 x 0.42 = 1.26 5 x 0.57 = 2.85 rating (Row 2 X Row 4) 6 Final Outcome Rating 1.26 + 2.85 = 4.11 (4) Moderately Satisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 63. At the time of appraisal, the percentage of women researchers in the LPNKs was 30. Therefore, the end target of women participation for this project was set as 28%. The actual percent the project reached was 32.51% which was beyond expectation. When the announcement for application was made for scholarship program, it was mentioned that women were encouraged to apply. They did apply and got the scholarships through competitive process. By the end of the project, out of (148) women graduate (Master’s and Ph.Ds.), 25 women (17%) have taken higher positions such as the lead researchers and head of sections and (69%) and have been working in their expertise fields. These were the good examples of women empowerment in the high-level professional environment. Male % Female % All Degree Program Master's 169 67.87% 80 32.13% PhD 140 67.31% 68 32.69% 457 Non-degree Program 1,551 67.46% 748 32.54% 2,299 Total 1,860 67.49% 896 32.51% 2,756 Institutional Strengthening 64. The RISET project experienced significant institutional changes throughout the project life. This has impacted project management. The PMO has encountered frequent changes of supervising departments: a merger of MoRT and MoECRT into MORTHE in 2014; and five years later in 2019, it was separated back to MoRT. Therefore, institutional strengthening during project implementation was disrupted frequently. Mobilizing Private Sector Financing 65. Indonesia Research Funding Forum (IRFF) was a forum where all public and private institutions to Page 26 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) discuss strategic directions of the Indonesian research way forward. There were 21 members from public and private institutions. It was important to share a common vision, mission, and overarching goals among these institutions to be able to optimize and harmonize the collaboration. The first step was to develop various funding schemes and collaborative multi-stakeholder research funding after the establishment of BRIN, and secondly to increase the synergy of Research and Innovation Funding from various funding institutions that were members of IRFF. It was an ongoing dialogue in the forum and private financing was not yet mobilized. Poverty Reduction and Shared Prosperity 66. The long-term objective of the project itself was to accelerate economic growth through the knowledge- based economy by improving the country’s competitiveness. The project has been building the knowledge and capacity of researchers that foster more productive industries and improve labor productivity which is one of key drivers in reducing poverty. In addition, with more innovative products, poorer households could be protected from paying too much for consumption products and leave room for other household expenditures such as for health, education, and other household expenditures. Innovation that will be produced by researchers and graduates of this project is also expected to make a difference in addressing urgent developmental challenges that are mostly experienced by the poor such as providing access to drinking water, reducing hunger, etc. Other Unintended Outcomes and Impacts 67. The RISET project was able to support the MoRT in COVID-19 pandemic mitigation. Through NDPs, the project has been building the knowledge and capacity of researchers and engineers to support research and development into vaccines and medical devices (e.g.: ventilators and other pandemic-related devices). III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 68. Project preparation took two years to be able to finalize the PAD with the defined objectives, associated activities and outcome indicators. As the RISET project was the second WB funded project for Science and Technology field in the education sector, MoRT was overly ambitious given government experience in this area. The RISET project’s Component 3 was quite similar to the previous WB funded scholarship program_ “Overseas scholarship Program”. But Components 1 and 2 were envisioned to support enabling policy environment and funding mechanisms for R&D activities that were not part of the previous project. Since components 1 and 2 were new, some activities were not designed to bring about quick results. The key factor that influenced project design was the uncertainties around government commitment to the project at the time of project preparation. MoRT has specific administrative and technical linkages with the LPNKs and it was not assessed comprehensively then. This may explain why the Component 1 design was beyond the project scope, and the achievement of the Component 1 would Page 27 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) be influenced by the external factors. As a result of the over ambitious design, the project had to be restructured after 3-years of implementation. B. KEY FACTORS DURING IMPLEMENTATION 69. The primary challenge during implementation of the RISET project was the prolonged reorganization of the newly formed Ministry of Research, Technology, and Higher Education (MoRTHE) that took place after the new government took office in October 2014 (i.e., 18 months after project approval). This process led to: (i) frequent changes to the project management team during 2014-15; (ii) lack of proactive oversight from the Ministry on project implementation in 2014-15; (iii) slow implementation of activities under Components 1 and 2 leading to low disbursements, (iv) slow payment processes for departing participants of Component 3 that were exacerbated by long and cumbersome project fund withdrawal processes from the Ministry's internal finance unit; and (v) inaccuracy in the project's annual budget document coupled with the government's cuts that frequently required revisions to the budget document (again requiring lengthy processes). By the end of 2015, only 20 percent of loan had been disbursed, compared to the planned target of 55 percent. The project's yearly expenditures and implementation average rate also lagged behind forecasts with only 36 percent expensed (by 2015) against the annual budget allocated in the government's annual budget document. These factors have led to the first restructuring of the project in 2016. After the first restructuring, the project design was simplified with project activities that were doable and achievable, and project targets achievable. When it was under MoRTHE, the approval process went down to Director level, not Minister level. In addition, intensive support and control from MoP and MoF plus consultants and assistants also played a vital role. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 70. The original PDO was ambitious, broad beyond the project scope, and its Result Framework indicators were not consistent with the desired outcomes. Following on this, the project required a level one restructuring after three-years of implementation. During the first restructuring, a significant revision of the PDO, RF indicators and the project components was made to be able to achieve the desired results of the Theory of Change. Thus, the revision of the M&E design allowed the project to be more specific, measurable, and achievable. 71. The M&E framework was built around three mechanisms for timely establishment of project progress within the context of developments in the sector. STI surveys, Independent Assessment Panels, Mid- term Review, and routine monitoring by PMO officers have contributed to improving the quantitative and qualitative data of the project. Page 28 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) M&E Implementation 72. At the sector level, an annual STI survey was carried out to establish and update key system indicators comparable to STI performance indicators defined by UNESCO or OECD. Independent Assessment Panels have carried out the annual review and evaluation of LPNK performance based on the targets set through the Institutional Development Plan by each LPNK. At the project level, routine monitoring was carried out by the PMO, through compiling administrative data to compute and update the PDO, as well as intermediate result indicators. The project Mid-term Review was carried out in 2016 for the assessment of the following: (i) overall progress in project implementation; (ii) results of monitoring and evaluation activities; (iii) progress on implementation of annual work plans and budgets; (iv) progress on procurement and disbursement; (v) implementation arrangements; and (vi) the need to make any adjustments to the project and reallocate funds to improve performance. The findings of the MTR have contributed lessons learned and ways forward to keep the project on track through project restructurings. M&E Utilization 73. The lessons learned and recommendations from this project are of interest to similar projects in the region. The Ministry of Religious Affairs is implementing the LPDP (Lembaga Pengelola Dana Pendidikan or Indonesia Endowment Funds for Education), a MoF program that manages endowment funds used to provide scholarship programs for MoRA staff to take Degree program and non-Degree programs. MoRA LPDP program is utilizing the lessons learned and recommendations from this project. MoRA also is using the guidelines for Degree Program and Non-Degree program of RISET project as a reference in developing the technical guidelines of the LPDP program. Justification of Overall Rating of Quality of M&E 74. The quality of M&E was assigned as Substantial as discussed above. Although the project faced design issues at the entry stage, necessary measures were taken to get the project restructured including the M&E mechanisms which enabled the project outcomes to be measured effectively. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 75. Social compliance: A Stakeholder Engagement Survey was conducted and showed positive perceptions of project implementation. A national symposium was organized to disseminate the project results. The project didn’t trigger environmental compliance as the project implementation was focused on scholarships and capacity building of the stakeholders. Though the scholarship awards were selected via competitive process, significant complaints from some of the non-awardees could have been an issue. An electronic Redress Mechanisms was never set up but all complaints were addressed manually throughout the project. The number of complaints and how they were addressed was not well documented. 76. Financial Management and Procurement: During the initial years (2013 – 2015), the project experienced significant weaknesses on financial management aspects, including continued delays in the payments of Page 29 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) scholarship-related expenditures, and weaknesses in the handling of soft expenditures, such as travel. Steps were taken to strengthen financial management capacity and to reimburse funds that were used for activities not eligible for project financing. 20 The project hired FM and Procurement consultants to assist the PMO in managing financial reports and procurement processes with due diligence. Project Management has continuously followed up on the audit findings and when ineligible expenditure was reported, the necessary measures were taken including refunding of ineligible expenditures. An example of a corruption case demonstrated the proactivity of project management: when it was revealed in the National Audit report, MoF requested repayment for misused monies by a retired staff. Both the National Audit Office and the Ministry didn’t report the incident as a corruption case, but instead categorized it as a misused/mishandling case. Taking the responsibility and accountability of the project funds, MoF refunded the misused money to the Bank. It was documented that a total of US$ 1,908,949 was refunded in the disbursement records. For the project extension at the second restructuring, MoF required MoRT to submit a statement of accountability for the Ineligible Expenditures. 77. The total project disbursed amount was US$71,671,344.88. The total undisbursed amount was US$5.8 million. The total refund amount for ineligible expenditure was US$1,908,949. C. BANK PERFORMANCE Quality at Entry 78. The project design took advantage of lessons learned from successful and unsuccessful experiences in reforming public Research and Technology Institute (RTI) in South and East Asia, as well as in Eastern Europe and Central Asian regions. Key lessons learned on overseas training for researchers about competitive research grants and STI system strengthening regionally and globally were taken into account when designing the project components. 79. For the implementation arrangements, a Steering Committee was established to provide guidance and oversight of the project, chaired by the RISTEK Executive Secretary and the Deputy- Minister for S&T Resources of RISTEK as the secretary. A Technical Committee was also established to provide technical advice and a Project Management Office (PMO) was formed to be responsible for the coordination of the overall project implementation, carrying out the activities under the project according to the agreed administrative arrangements, financial management and procurement practices, and applicable safeguard policies. The ICR noted that the implementation arrangements were adequately prepared, including for financial management and procurement for the RISET project. 20 First Restructuring Paper, Report No: RES19262, January 2017 Page 30 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 80. Project related key risks were identified thoroughly in the Operational Risk Assessment Framework (ORAF) and the mitigation measures were built into the project design. The Implementation Support Plan was focused on risk mitigation measures, for example, the implementation agency risk was rated as Substantial at the appraisal as the RISTEK was the new ministry for the Bank’s engagement, unfamiliar with the Bank’s operation policies and procedures. High financial management risk (related to soft expenditure) and high procurement risk (related to the large amount of technical assistance) were addressed by the implementation support to help the PMO core staff recruit FM and Procurement experts. Additionally, the Bank team’s hands-on training support to the PMO team on all aspects of the project helped the risk mitigation throughout the project life. Overall risk rating was noted as Substantial at the time of appraisal but in the latest ISR, it was rated as Low because the risk mitigation measures were continuously addressed during implementation. Quality of Supervision 81. An implementation support plan has been in place for technical, procurement, fiduciary, and procurement inputs to the client departments. Bank supervision and implementation support missions were carried out on a regular basis. 82. Throughout the eight-year implementation period, the RISET project was supervised by the same TTL based in Indonesia. The Bank’s expert teams also provided inputs on FM, procurement, STI, anti- corruption and others. 83. As the Task Team was always ready to support the client departments, the project has made intensive efforts to implement NDP activities despite pandemic-related challenges. Justification of Overall Rating of Bank Performance 84. This ICR assigns overall rating for Bank Performance as Satisfactory. The quality of supervision and implementation support was satisfactory especially in the later phases of the project. Although the project experienced initial weakness at project entry, the Bank’s proactivity and supervision throughout the project life was able to improve the project in terms of achieving the results and bringing a big change in the institutional capacity. The Bank team has worked closely with the PMO and the high-level leadership of MoRT. Adequate supervision and course correction during project implementation when required allowed for the successful completion of the project. D. RISK TO DEVELOPMENT OUTCOME 85. To sustain the STI capacity strengthened in this project, it was recommended first to develop various funding schemes and collaborative multi-stakeholder research funding after the establishment of Ministry of Research Technology (BRIN), and secondly to increase the synergy of Research and Innovation Funding from various funding institutions that are members of the Indonesia Research Funder Forum (IRFF) to strengthen the research and innovation ecosystem. Page 31 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 86. The government’s affordability to continue to invest in future scholarship programs on R&D is worth pursuing. Under the MoF, Indonesia Endowment Funds for Education (LPDP) 21has been operating the scholarship programs targeting tertiary level graduates in diverse field of studies. The government’s investment plan for LPDP should have ring-fencing thresholds of Masters and Ph.D. graduates. To increase from the current 90 researchers per million people, multiple resource mobilizations should be initiated not only through public resources but also through public private partnerships: to increase the current 0.08% of GDP on R&D, to collaborate more with research funding institutes locally and internationally, and to continue ongoing dialogue for resource mobilization during the IRFF forum meetings. V. LESSONS AND RECOMMENDATIONS 87. The RISET project experienced valuable lessons learned during the project preparation, implementation, and monitoring. Among them, some key lessons learned, and relevant recommendations are highlighted in this section. 88. To keep informed public research institutions on the role and function of the innovation ecosystem development helps them to proceed a structured product commercialization process. It is recommended to bring in the industrial support early in the process, and to initiate agreement on the commercialization process by different stakeholders. 89. Research institutions tend to focus on their own mandates which prevent them from disclosing data and information. It is recommended to have shared vision, missions and policy guidelines among research funding institutions to collaborate more on broader national research priorities, rather than the individualized approach on limited research areas. 90. Without having an orientation on different fields of study available in foreign universities prior to the application process, the selected candidates found it difficult to choose the universities that offer the specialization or the specific field of studies. It was recommended that future scholarship programs should consider budget allocation for hiring education agencies or individual agents to liaise between international universities and the selected candidates. 91. To manage a large-scale scholarship program, especially when it is new, responsibilities for administration and for finance should be split between different teams or departments. The selection of candidates through a competitive process, supporting the candidates to get orientated to enroll in the foreign universities, and helping candidates overcome problems during their studies needs different skills than managing and monitoring the fund transfers to a large number of scholars over several years. 92. The Masters and Ph.D. degree scholars took positions in the R&D in the LPNKs on their return and applied their knowledge and skills in the respective science research programs. The government 21 https://lpdp.kemenkeu.go.id/en/ (LPDP is committed to preparing Indonesian future leaders and professionals through scholarships and encouraging research innovation through research funding. LPDP continues to move towards an organization with high competitiveness, not only on a local scale, but also on a regional and even international scale. Page 32 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) ministries treated the scholars as a talent pool and planned to provide professional networks and to organize professional development programs for the scholars such as seminars, conferences, symposiums, joint research works, and pilot projects domestically, regionally and globally. This good practice should be maintained in a long run to contribute towards the country competitiveness and knowledge transfer. . Page 33 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To support the government in improving STI human resources capacity Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 3. Increased % of Master's Percentage 0.00 100.00 27.00 and Ph.D. degree holders in LPNK and MoRTHE supported 30-Sep-2013 31-Dec-2020 15-May-2019 by the RISET Project Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 4. Master’s degree holders Percentage 0.00 90.00 95.00 who participate in applied science research programs 30-Sep-2013 31-Dec-2020 15-May-2019 Comments (achievements against targets): Page 34 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 5. Ph.D. degree holders who Percentage 0.00 90.00 97.00 manage or coordinate applied science research 30-Sep-2013 31-Dec-2020 15-May-2019 programs Comments (achievements against targets): Objective/Outcome: 2. To support the government in strengthening technology transfer Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. LPNK research agency Text NA In progress In progress reforms are institutionalized 30-Sep-2013 15-May-2019 15-May-2019 Comments (achievements against targets): Objective/Outcome: 3. To support the government in strengthening institutional functioning and data management of LPNKs Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Improved regulation on Text NA In progress In progress Page 35 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) the output-based cost 13-May-2011 15-May-2019 15-May-2019 standard for the STI research funding system issued on an annual basis Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Strengthening the Innovation System and Technology Transfer at Public Research Agencies Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. National innovation Text NA Strategy action plans Strategy action plans strategy is developed have been submitted have been submitted (Component 1) in the form of a in the form of a technocratic paper for technocratic paper for RPJMN 2020-2024. RPJMN 2020-2024. 30-Sep-2013 15-May-2019 15-May-2019 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. National Information Text 1.00 1.00 1.00 Page 36 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) System for STI is established 30-Sep-2013 15-May-2019 15-May-2019 and integrated into ministry’s database system (Component 1) Comments (achievements against targets): STI data is continuously being collected, and the database is being expanded to include national, local government, and university STI data. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 3. Existing regulations in the Text NA Modifications are Modifications are STI legal framework are discussed and discussed and reviewed and modifications validated with validated with proposed (Component 1) relevant ministries relevant ministries and institutions. and institutions. 30-Sep-2013 15-May-2019 15-May-2019 Comments (achievements against targets): Component: Strengthening Public Research Funding Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. Increased number of Text NA Number of research Number of research Page 37 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) research programs in LPNKs programs of higher programs of higher and MoRTHE focused on TRL are being TRL are being higher ranks of the identified. identified. Technology Readiness Level (TRL) (Component 2) 30-Sep-2013 15-May-2019 15-May-2019 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Strengthened R&D Number 0.00 12.00 6.00 monitoring & evaluation capacity (Component 2) 30-Sep-2013 31-Dec-2020 15-May-2019 Comments (achievements against targets): Component: Developing Science and Technology Human Resources Capacity Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. Number of LPNK, MoRTHE, Number 0.00 457.00 457.00 Research Agency staff who have studied overseas as 30-Sep-2013 15-May-2019 15-May-2019 supported by RISET-Project (Component 3) Page 38 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Number of participants in Number 0.00 1,670.00 1,670.00 RISET-Project’s Non-Degree professional courses and 30-Sep-2013 15-May-2019 15-May-2019 training (Component 3) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 0.00 2,387.00 2,127.00 30-Sep-2013 31-Dec-2020 15-May-2019 Female beneficiaries Percentage 0.00 28.00 31.00 Comments (achievements against targets): Component: Project Management Page 39 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. Project implementation Yes/No No Yes Yes and monitoring reports (include financial reports) are 30-Sep-2013 31-Dec-2020 15-May-2019 completed (Component 4) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Computerized project Yes/No No Yes Yes monitoring database is established, updated, and 30-Sep-2013 31-Dec-2020 15-May-2019 key information is placed on MoRTHE project website (Component 4) Comments (achievements against targets): Page 40 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: To support the government in strengthening technology transfer Outcome Indicators LPNK research agency reforms are institutionalized 1. National innovation strategy is developed 2. National Information System for STI is established and integrated Intermediate Results Indicators into ministry’s database system 3. Existing regulations in the STI legal framework are reviewed and modifications proposed 1. Policy papers have informed the Law No. 11/2019 on National System of Science, Technology and Innovation. 2. Strategy action plans are submitted for consideration to the new national research and innovation master plan Key Outputs by Component 3. STI data is collected through Monevrisbang system, which is being (linked to the achievement of the Objective/Outcome 1) used and integrated into Ministry’s data base system 4. Policy papers on the STI legal framework and innovation funding system, as well as commercialization, have been published and have informed a number of regulations and implementation Objective/Outcome 2: To support the government in strengthening institutional functioning and data management of LPNKs Improved regulation on the output-based cost standard for the STI Outcome Indicators research funding system issued on an annual basis 1. Increased number of research programs in LPNKs and MoRTHE Intermediate Results Indicators focused on higher ranks of the Technology Readiness Level (TRL) 2. Strengthened R&D monitoring & evaluation capacity Key Outputs by Component 1. Increased number of research programs of higher TRL (linked to the achievement of the Objective/Outcome 2) 2. Twelve independent evaluations of multi-year research activities Page 41 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) were completed. The target was achieved in 2019. Objective/Outcome 3: To support the government in improving STI human resources capacity 1. Increased % of Master's and Ph.D. degree holders in LPNK and MoRTHE supported by the RISET Project 2. Master’s degree holders who participate in applied science Outcome Indicators research programs 3. Ph.D. degree holders who manage or coordinate applied science research programs. 1. Number of LPNK, MoRTHE, Research Agency staff who have studied overseas as supported by RISET-Project 2. Number of participants in RISET-Project’s Non-Degree Intermediate Results Indicators professional courses and training 3. Direct project beneficiaries 4. Students benefiting from direct interventions to enhance learning 1. 457 staff studied overseas 2. 2,098 participated in non-Degree professional courses and Key Outputs by Component training (linked to the achievement of the Objective/Outcome 2) 3. 2,555 direct beneficiaries of which 32% are female 4. 2,460 students benefiting from direct interventions to enhance learning Objective/Outcome 4: Project Management Outcome Indicators N/A 1. Project implementation and monitoring reports (include financial reports) are completed Intermediate Results Indicators 2. Computerized project monitoring database is established, updated, and key information is placed on MoRTHE project website Page 42 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 3. Implement a beneficiary survey to get feedback from stakeholders 1. Submitted the annual reports and financial reports. 2. Computerized project monitoring database is established and updated. The website link is: http://risetpro.ristekdikti.go.id and Key Outputs by Component https://www.youtube.com/watch?v=C1eOCIm3DrI&t=6s (linked to the achievement of the Objective/Outcome 2) 3. Feedback from stakeholders were used as inputs in the development of Implementation Completion Report (ICR) by MoRT. Page 43 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Ratna Kesuma Task Team Leader(s) Budi Permana Procurement Specialist(s) I Gusti Ngurah Wijaya Kusuma Financial Management Specialist Neil Butcher Team Member Francisca Melia. N Setiawati Social Specialist Sylvia Njotomihardjo Team Member Sri Hastuti Procurement Team Noah Bunce Yarrow Team Member Ruwiyati Purwana Team Member Petra Wiyakti Bodrogini Prakosa Team Member Krisnan Pitradjaja Isomartana Environmental Specialist Theodore Manggala Amarendra Counsel B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 35.279 219,501.69 FY12 43.931 415,568.97 FY13 29.312 312,338.10 FY14 3.900 105,701.94 Page 44 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Total 112.42 1,053,110.70 Supervision/ICR FY13 2.520 49,041.51 FY14 27.652 244,453.41 FY15 28.272 120,260.19 FY16 31.626 111,681.11 FY17 9.850 48,225.33 FY18 3.585 30,620.42 FY19 18.592 86,302.82 FY20 65.413 239,244.98 FY21 0 - 72.68 Total 187.51 929,757.09 Page 45 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Strengthening the Innovation System and Technology 2371238.74 2376397.47 100% Transfer at Public Research Agencies Strengthening Public 2067145.29 1902354.47 92.03% Research Funding Developing Science and Technology Human 64753924.85 60338721.42 93.18% Resources Capacity Project Management 5571069.85 4966687.08 89.15% Total 74,763,378.73 69,584,160.44 0.00 Page 46 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) ANNEX 4. EFFICIENCY ANALYSIS Introduction 1. A cost-benefit analysis (CBA) framework is used for the economic analysis of the RISET project. The benefits include the social return from a strengthened national innovation strategy, strengthened public research funding, and the increased human resources capacity in science and technology. The costs consist of direct cost in financing the project activities under different components which also includes project management, human capital depreciation, costs associated with drop out students, and opportunity cost associated with undisbursed/cancelled project funds. Economic Benefits 2. The RISET project provided financing to a number of beneficiaries and achieved the results as follows. Component 1 financed five Science and Technology Parks (STPs) and four Technology Transfer Office (TTOs), implemented through: (i) selection of STPs programs of excellence; (ii) developing road map and action plans; (iii) identification of industrial needs and STP management capacity building; and (iv) preparation for STP networking and training. 3. Component 2 provided funding to improve regulation on the output-based cost standard for the Science and Technology Innovation research funding system issued on an annual basis. Technical inputs to documents were provided and shared to inform a number of regulations on output-based cost standard. 22 4. Component 3 provided financing for scholarship to increase percentage of Master’s and Ph.D. degree holders in LPNKs and MoRTE. A total of 457 staffs have received scholarship, of which 253 (55.4%) undertook Master’s degrees, and 187 (45.6%) for PhD programs. All of the Master’s graduates (236) have involved in research, appointed as research coordinators, continue further study or have been allocated posts aligned to their fields of study. Meanwhile, 131 out of 137 PhD graduates have returned and become research team leaders and allocated posts aligned to their fields of study. The other six graduates completed their studies by 2021 and their re-entry process is ongoing. Component 3 also provided funding to non-degree professional courses and training. A total of 2,289 participants have taken part in this program, and an additional 55 proposals were submitted involving 464 researchers, which will make the total participants of this program become 2537. 5. The funding for component 4 was provided to support the operation of the Steering Committee, Technical Committee and Project Management Office in the functions of: Project coordination, Project monitoring and evaluation, procurement and financial management and other fiduciary aspects of Project implementation; and Project advisory activities. Component 4 also provided a re-entry program for returning Scholarship Recipients under component 3. 22PMK 106/2016, DG of Budget Regulation No. Per-15/PB/2017. Permenristekdikti No. 42/2016, and Perpres No. 16/2018 and Permenristekdikti No. 20/2018. In addition, DG Regulation No. PER-7/PB/2019 was issued to replace the one No. PER- 15/PB/2017. Page 47 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 6. Ideally, the cost and benefit analysis should look at realized data on benefits for different interventions. However, such data is not available systematically particularly since there is a lag of several years of after the project closure to have such data available. Hence, this economic analysis leverages from the findings of the economic research literature that estimate social rates of return to R & D spending between 200 to 300 percent 23. The recent estimate of gross returns to public R&D in Southeast Asian countries is also found to be significantly large, being about 252 percent 24. Specifically, this economic analysis assumes that 30 – 40% of financing for scholarships and fellowships will successfully generate social return after four years (achievement rate). This assumption is considered conservative given the number of achievements of the graduates that have been identified at the project closure 25. The analysis uses the low-range value of 200 percent return and takes an annual discount rate of 5 percent when estimating present values. 7. The data suggests that the RISET project funding invested about USD 3 million to strengthening national innovation strategy, USD 2.4 million to strengthening public research funding, and USD 76.8 million to developing S&T human resources capacity. The present value of gross benefits, using the assumptions on social returns and discount rate, is estimated at USD 47 million (table 1). Table 1. Gross economic benefits of RISET project (USD, present value) SPENDING on R &D through RISET project Year of 2013 2014 2015 2016 2017 2018 2019 2020 2021 Total spending Strengthening 197,289 436,788 375,652 264,043 936,512 357,639 344,300 180,977 - 3,093,204 national innovation strategy Strengthening 1,560 51,884 77,596 836,883 765,902 442,718 162,770 90,021 - 2,429,337 public research funding Developing 2,157,346 4,194,031 7,014,066 19,371,212 14,758,540 14,962,339 9,802,172 1,946,197 S&T human 2,563,529 74,804,648 resources capacity SOCIAL BENEFIT OF RISET PROJECT INVESTMENT Year social 2017 2018 2019 2020 2021 2022 2023 2024 2025 Total returns realized 23 Griffith, Redding, and Van Reenen (2004), Goni and Maloney (2017). 24 The estimates of gross returns to public R&D in Southeast Asian countries is significantly large, ranging from 252% to 334%. Note that the difference results from using different estimation techniques and assumptions placed on main variables (SAHEP Project Appraisal, 2017). To be conservative, we choose the lowest bound of the estimates. 25 RISET alumni (both in degree and non-degree programs) in LIPI, BPPT, and LAPAN have initiated and involved in various research projects such as marine biology, creating earthquake resistance building material, digital transformation for COVID response, creating bone implants, coordinating the national aircraft development (among many others). Page 48 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) Strengthening national innovation strategy 324,622 718,693 618,101 434,459 1,540,943 588,461 566,513 297,781 - 5,089,574 Strengthening public research funding 2,567 85,371 127,678 1,377,013 1,260,219 728,451 267,824 148,122 - 3,997,244 Developing S&T human resources capacity 1,064,912 2,070,264 3,462,294 9,562,047 7,285,133 7,385,732 4,838,563 960,685 1,328,684 37,958,313 Gross economic benefits 1,392,101 2,874,328 4,208,073 11,373,518 10,086,295 8,702,645 5,672,900 1,406,587 1,328,684 47,045,130 Assumptions: Social rate of returns to RISET R&D Investment=200%, Discount rate=5% Economic Cost 8. There are two types of direct costs associated with the RISET project. First, investment made to research and development (R&D), either through creating enabling policy environment of R&D in science and technology, improving public R&D institutes’ performance, or improving STI human resources capacity. Another direct cost is the spending used for project management. 9. This economic analysis also considers indirect costs due to human capital depreciation of the scholarships and fellowships recipients which assumed to be 6 percent 26, and costs associated with drop-out students and undisbursed/cancelled funds of the originally committed funds of the RISET project. The total of the cancelled funds was USD 25.4 million, of which USD 15 million cancelled in 2017, USD 5.2 million cancelled in 2020, and USD 5.1 million cancelled in 2021. The analysis considers the opportunity cost if the cancelled funds were allocated/invested in alternative projects which might have had lower social rate of returns than 200 percent social rate returns estimated for the RISET project investment. Hence, the difference between the RISET project investment in R&D and the alternative projects assumed to generate 35 percent social return with a lag of 1 year. Details on the project costs associated with this opportunity costs are presented in Panel A and Panel B of Table 2. The direct costs of the disbursed funds is about USD 88.6 million, while the total opportunity cost was estimated at US$21.2 million. The total economic cost of the RISET project is equal to about US$28.4 million. Table 2. Costs under RISET project (USD, present value) 2013 2014 2015 2016 2017 2018 2019 2020 2021 Total Panel A. Direct costs of RISET project disbursed funds Strengthening national innovation 197,290 436,788 375,653 264,044 936,513 357,639 344,300 180,977 - 3,093,204 26Human capital depreciation ranges from 1 to 6 percent (Lentini and Gimenez, 2019; Dinnerstein et al, 2020). To be conservative, we choose the highest bound of the estimates Page 49 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 2013 2014 2015 2016 2017 2018 2019 2020 2021 Total strategy Strengthening public research funding 1,560 51,885 77,597 836,884 765,902 442,718 162,771 90,021 - 2,429,338 Developing S&T human resources capacity 2,157,346 4,194,031 7,014,066 19,371,212 14,758,540 14,962,340 9,802,172 1,946,197 2,691,706 76,897,611 Project Management 32,045 375,220 570,542 1,120,348 1,916,279 883,197 783,808 252,206 332,502 6,266,147 Total direct cost 2,388,241 5,057,924 8,037,858 21,592,488 18,377,234 16,645,894 11,093,052 2,469,401 3,024,207 88,686,299 Panel B. Indirect costs Depreciation cost 129,441 251,642 420,844 1,162,273 885,512 897,740 588,130 116,772 161,502 4,613,857 Opportunity cost because of cancelled 21,249,044 funds 13,400,956 4,041,362 3,806,725 Costs of drop-out 2,546,245 students Total Economic Costs 129,441 251,642 420,844 1,162,273 14,286,469 897,740 588,130 4,158,134 3,968,228 25,862,901 Assumptions: Depreciation rate of human capital=6%, Social rate of return of cancelled funds in alternative projects = 35%, and Discount rate=5% Cost and Benefit Analysis 10. The comparison between the cost and benefits associated with the RISET project is presented in Table 3. Overall, the RISET project investment generated NPV benefits of about USD 18.5 million and IRR of 12 percent in a base-case scenario with conservative assumptions. The estimates are considered to be conservative and low-bound given that other benefits associated with spillover effects of R&D products such as positive externalities on socio- economic indicators such as more shared prosperity (equity) and increased future salaries of the human resources are not reflected in the analysis. Nevertheless, the economic analysis provides strong evidence that the RISET project investments have been productive and economically profitable. Sensitivity Analysis 11. Besides the base-case scenario above, the analysis looks at two additional scenarios under different assumptions. The risks and the expected NPV of the project are assessed with the assumption of different social rates of returns of the RISET project investments, different social rates of return of the restructured funds in alternative projects, and different achievement rates of the scholarship and fellowship financing. Page 50 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) 12. The first scenario assumes that the achievement of graduates in generating social return of their research is 10 percent higher than the base-case scenario. With this assumption, RISET project investments result in NPV of USD 31.2 million and IRR of 16 percent. 13. The second scenario assumes that the RISET restructured funds would be redirected to highly productive projects of which the social rate of returns would be 70 percent. This assumption increases the NPV to USD 27.1 million and IRR to 14 percent. Table 3: Summary of Economic Costs and Benefits of the RISET project (USD, present value) Costs and Benefits (PV, USD) Base-case Case 1 Case 2 scenario 47,045,130 59,697,901 47,045,130 Gross economic benefits of RISET R&D investments 28,480,388 28,480,388 17,855,865 Economic costs - of project activities 88,686,299 88,686,299 88,686,299 - of depreciation 4,613,857 4,613,857 4,613,857 - of undisbursed/cancelled funds (opportunity 21,249,044 21,249,044 10,624,522 costs) - drop out students 2,617,487 2,546,245 2,546,245 NPV 18,564,743 31,217,514 27,115,504 IRR 12% 16% 14% Social rate of return of FIRST R&D investments 200% 200% 200% Social rate of return of cancelled funds in alternative projects 35% 35% 70% Achievement rates of the scholarship and fellowship financing 30% 40% 30% Page 51 of 52 The World Bank Research and Innovation in Science and Technology Project (P121842) ANNEX 5. SUPPORTING DOCUMENTS (IF ANY) 1. MoRT ICR Summary 2. RISET Restructuring Paper dated January 30, 2017 3. RISET Restructuring Paper dated November 19, 2020 4. RISET Project Appraisal Document dated March 1, 2013 Page 52 of 52