Document of The World Bank FOR OFFICIAL USE ONLY Report No: 126482-VN IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA-52490 ON A CREDIT IN THE AMOUNT OF SDR36.7 MILLION (US$ 55 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR THE VIETNAM INCLUSIVE INNOVATION PROJECT May 22, 2018 Finance, Competitiveness and Innovation Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective as of May 20, 2018) Currency Unit = VND VND 1= US$0.00 US$1 = SDR 22,771 FISCAL YEAR July 1 - June 30 Regional Vice President: Victoria Kwakwa Country Director: Ousmane Dione Senior Global Practice Director: Ceyla Pazarbasioglu-Dutz Practice Manager: Ganesh Rasagam Task Team Leader(s): Smita Kuriakose ICR Main Contributor: Brian G. Mtonya ABBREVIATIONS AND ACRONYMS ADB Asian Development Bank BoP Base of the Pyramid DA Designated Account DANIDA Danish Development Program DFID U.K. Department for International Development DPL Development Policy Loan DPM Deputy Prime Minister EDA Enterprise Development Agency, MPI FIL Financial Intermediary Loan FIRST Fostering Innovation through Research, Science and Technology GDP Gross Domestic Product GoV Government of Vietnam HUST Hanoi University of Science and Technology ICT Information and Communications Technology IDA International Development Association IPR Intellectual Property Rights ISR Implementation Status and Results Report M&E Monitoring and Evaluation MIC Middle-Income Country MoST Ministry of Science and Technology MPI Ministry of Planning and Investment MTR Mid Term Review NAFOSTED National Foundation for Science and Technology Development NDC National Development Challenge NHTM National Hospital of Traditional Medicine NIMM National Institute of Medicine Materials NIS National Innovation System OOG Office of the Government OECD Organization for Economic Cooperation and Development PDO Project Development Objective PFI Participating Financial Intermediaries PIA Project Implementation Agency PIU Project Implementation Unit PMU Project Management Unit PSC Project Steering Council R&D Research and Development RDI Research and Development Institution SEDS Socio-Economic Development Strategy SNV Netherlands Development Organization STI Science, Technology and Innovation THM Traditional Herbal Medicine UNDP United Nations Development Program VAST Vietnam Academy of Science and Technology VIIP Vietnam Inclusive Innovation Project WB World Bank WHO World Health Organization TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9 II. OUTCOME .................................................................................................................... 10 A. RELEVANCE OF PDOs ............................................................................................................ 10 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 11 C. EFFICIENCY ........................................................................................................................... 11 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 12 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 12 A. KEY FACTORS DURING PREPARATION ................................................................................... 12 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 12 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 16 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 16 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 17 C. BANK PERFORMANCE ........................................................................................................... 17 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 19 V. LESSONS AND RECOMMENDATIONS ............................................................................. 19 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 22 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 30 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 32 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 33 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 34 ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 35 The World Bank Vietnam Inclusive Innovation Project (P121643) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P121643 Vietnam Inclusive Innovation Project Country Financing Instrument Vietnam Specific Investment Loan Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Enterprise Development agency, Ministry of Planning and Investment, National Foundation of Science and SOCIALIST REPUBLIC OF VIETNAM Technology Development (NAFOSTED), Ministry of Science and Techno Project Development Objective (PDO) Original PDO The Project Development Objective (PDO) is to adopt, upgrade and develop inclusive innovations for the benefit of the Base of Pyramid population. This will be achieved by strengthening Vietnam's capacity to undertake inclusive innovation, including financing the development, adaptation, adoption, scaling up and commercialization of inclusive technologies, and improving RDIs' and SMEs' technological and innovation capabilities. Page 1 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 55,000,000 6,416,664 1,384,026 IDA-52490 Total 55,000,000 6,416,664 1,384,026 Non-World Bank Financing Borrower 625,000 0 0 Total 625,000 0 0 Total Project Cost 55,625,000 6,416,664 1,384,026 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 13-May-2013 06-Dec-2013 11-May-2016 30-Nov-2018 30-Jun-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-Apr-2017 1.61 Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Unsatisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 23-Oct-2013 Satisfactory Satisfactory 0 Page 2 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 02 08-Jul-2014 Satisfactory Moderately Satisfactory 1.00 03 28-Jan-2015 Satisfactory Moderately Unsatisfactory 1.00 Moderately 04 23-Sep-2015 Moderately Unsatisfactory 5.00 Unsatisfactory 05 28-Mar-2016 Unsatisfactory Unsatisfactory 5.61 06 01-Aug-2016 Unsatisfactory Unsatisfactory 5.61 07 22-Jan-2017 Unsatisfactory Unsatisfactory 1.61 08 19-Jul-2017 Unsatisfactory Unsatisfactory 1.38 SECTORS AND THEMES Sectors Major Sector/Sector (%) Agriculture, Fishing and Forestry 4 Other Agriculture, Fishing and Forestry 4 Public Administration 18 Other Public Administration 18 Financial Sector 45 Banking Institutions 34 Other Non-bank Financial Institutions 11 Health 5 Health 5 Industry, Trade and Services 28 Other Industry, Trade and Services 28 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Page 3 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Private Sector Development 68 Business Enabling Environment 36 Investment and Business Climate 10 Innovation and Technology Policy 26 Enterprise Development 32 MSME Development 32 Finance 32 Financial Infrastructure and Access 32 MSME Finance 32 ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Victoria Kwakwa Country Director: Victoria Kwakwa Ousmane Dione Senior Global Practice Director: Tunc Tahsin Uyanik Ceyla Pazarbasioglu-Dutz Practice Manager: Hormoz Aghdaey Ganesh Rasagam Task Team Leader(s): Nancy Chen Smita Kuriakose ICR Contributing Author: Brian G. Mtonya Page 4 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country Context: Vietnam has experienced rapid and consistent economic growth since launching its transition towards a market based economy in 1986. Political and economic reforms (Ä?ổi Má»›i) launched in 1986 transformed Vietnam from one of the poorest countries in the world, with per capita income below US$100, to a lower middle-income country within a quarter of a century. The level of development and growth experienced by Vietnam had been accompanied with impressive poverty reduction – the poverty rate declined from 58 percent in the early 1990s to 14.5 percent in 20081. Yet the gap in the poverty headcount ratio between rural and urban areas remained substantial, at 10.5 percent. This disparity was a clear indication that certain segments of the society were not benefiting fully from Vietnam’s growth. Vietnamese authorities were concerned with the widening disparities, especially between urban and rural populations, with 72 percent of the population residing in rural areas and some 60 percent are employed in the agricultural sector (accounting for 20.6 percent of GDP). 2. To address widening social disparities, Vietnam wished to improve its human capital (education), technological and innovation capacity and infrastructure, which are key drivers to competing at a higher level. Further, it acknowledged the need to adopt national policies which would utilize technological advances to improve the quality of the lives of the Base of Pyramid (BoP) population2 and reduce disparities. “Inclusive Innovationâ€? was identified by senior officials as an appropriate measure to reduce growing disparities in society and to improve the competitiveness of Vietnam by lowering the costs of products, and expanding markets and opportunities for businesses. The objective of inclusive innovation is to develop and deliver, through public and private sector initiatives, high-performance products and solutions at a low cost that are affordable to, and benefit, the resource-poor or BoP population. The Government decided to launch a pilot project on Inclusive Innovation with the objective of developing a sustainable ecosystem for inclusive innovation, which would involve strong collaboration among government agencies, private sector enterprises, entrepreneurs, research and development institutions and global partners. 3. Sectoral Context: The Socio-Economic Development Strategy (SEDS) 2011-2020 of Vietnam clearly identified the need to harness science and technology development in improving peoples’ lives and supporting sustainable and shared economic growth. It calls for major reform of the National Innovation System (NIS) to enhance the capacity of Research & Development Institutions (RDI), strengthen collaboration among RDIs and enterprises, establish national standards, measurement and assessment, enhance enterprises’ capacity to innovate and improve productivity and competitiveness, and develop appropriate public and private financing mechanisms to support innovation activities. Vietnam’s National Innovation System has undergone many reforms; however, 1 A new poverty line was estimated for 2010 by the General Statistics Office (GSO) and the World Bank (referred to as the GSO- WB poverty line). 2 Base of the pyramid (also referred to as economic “Bottom of the Pyramidâ€?) is the world’s largest but poorest socio-economic group subsisting on less than USD 2 a day (PPP). Page 5 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) several major challenges remain to make it efficient. Despite a less-than-optimal innovation system, Vietnam has made notable accomplishments in some sectors that demonstrate the country is poised to adopt an inclusive innovation agenda. 4. Agricultural innovation and traditional medicine sector are areas where Vietnam demonstrates strong tradition, capacity and strategic interest. Agricultural innovation in Vietnam has been positive, as government policies encouraged farmers to apply new knowledge in production activities and helped farmers generate better income, thus contributing to poverty reduction. However, contribution of technological innovation in agriculture had been at levels less than the country’s potential. In the context of Vietnam’s integration into the global economy, the agricultural sector must deepen the use of technological innovation to further improve productivity and enhance competitiveness in international markets. The importance of innovation in agriculture is vital to the concept of inclusive innovation – it can potentially further improve the lives of the BoP by raising the incomes of rural society, who predominantly depend on agriculture for their livelihoods, and are a target of social development in Vietnam. Vietnam also has a long history of using traditional herbal medicine (THM) in the national health care system. Vietnam not only has rich resources for traditional herbal medicine development, but has developed a medical professional training system that includes traditional remedies education as a stream for specialization for undergraduate medical students. 5. Rationale for World Bank Assistance: Over the last decade, government and donors have financed innovation- related projects in areas of promoting technology adaptation and transfer, developing and customizing technology applications for Vietnam (e.g., SNV and ADB biogas projects), technical assistance to R&D institutions (WHO, OECD and UNDP projects), strengthening national innovation system (Finland), and supporting Small and Medium Enterprise (SME) technology upgrading (DFID, DANIDA). The Inclusive Innovation Project aimed to complement these efforts and focus on developing an ecosystem for an inclusive innovation system based on the strong collaboration among RDIs, enterprises, innovators and global research partners to develop, adapt, adopt, upgrade and commercialize inclusive technologies/solutions to the targeted BoP population in Vietnam. 6. Recognizing that the World Bank has several decades of successful worldwide experience in providing support for Science Technology and Innovation (STI) system strengthening, the government requested World Bank assistance to pursue its STI reform and SME development agenda, including in the area of Inclusive Innovation. The Bank was supporting the government’s request through a number of analytical products and lending operations on innovation system development and SME development, including the policy paper, “Vietnam: Innovation Policy for Competitiveness, Economic Growth and Social Equityâ€? (December 12, 2009), which was prepared in response to the Deputy Prime Minister’s (DPM) request; and several projects in the education and STI fields. The projects included: (i) Education Development Policy Loans (DPLs), focusing on the reform of the education sector, including higher education; (ii) Fostering Innovation through Research, Science and Technology Project (FIRST), aimed at improving the national policy framework for S&T, and supporting restructuring of RDIs; (iii) Vietnam Inclusive Innovation Project (VIIP, this project), focusing on supporting inclusive innovation, including strengthening SMEs’ technological and innovation capacity, with the twin objectives of helping to reduce social disparities and improving enterprise competitiveness; and (iv) the Regional Inclusive Innovation Workshop in Hanoi in May 2011, to share global experiences on successful development and commercialization of inclusive technologies. Page 6 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 7. The Vietnamese authorities, as well as other stakeholders (including RDIs, major universities and the private sector), indicated a strong desire and commitment to pursuing the STI reform agenda, especially with a view to promoting inclusive innovation. Subsequent discussions during project preparation missions between October 2010 and December 2012, reinforced the government’s commitment to the proposed project. The formal project preparation process from concept to board approval was 27.9 months, a rather long period but normal by Vietnam portfolio standards due to the country’s lengthy approval procedures. Theory of Change (Results Chain) 8. Based on the PDO of the project (see section below), the theory of change in figure 1 illustrates the result chain by explaining the links between the project’s interventions, outputs, intermediate results and the desired outcomes based on the PDO statement. Fig.1: Theory of Change: Vietnam Inclusive Innovation Project Activities Outputs PDOs/Outcomes Long Term Number of inclusive Strengthening the Inclusive Development of Inclusive technologies Recipient's capacity to Innovation Technologies developed/upgraded undertake Inclusive for the likely to benefit the BoP Innovation, including benefit of financing development, the Base of adaptation, adoption, the Scale up and scale-up and Pyramid Commercialization of commercialization of population Inclusive Technologies Inclusive Technologies adopted, Number of inclusive upgraded technologies Improving RDIs' and and commercialized or SMEs' technological and Capacity Building and developed scaled up by enterprises innovation capabilities Global Knowledge Transfer likely to benefit the BoP. Project Development Objectives (PDOs) 9. The Project Development Objective (PDO) is to adopt, upgrade and develop Inclusive Innovations for the benefit of the Base of the Pyramid population, by: (i) strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies, and (ii) improving RDIs' and SMEs' technological and innovation capabilities. Page 7 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Key Expected Outcomes and Outcome Indicators 10. The Inclusive Innovation Project has the following two expected outcomes; (i) Strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies; and (ii) Improving RDIs' and SMEs' technological and innovation capabilities. 11. The following outcome indicators were to be used to measure the development objective: i. Number of inclusive technologies developed/upgraded that are likely to benefit the BoP. ii. Number of inclusive technologies commercialized or scaled up by enterprises that are likely to benefit the BoP. 12. It was expected that affordable quality technological solutions developed through this project would be made available to improve the quality of life of the BoP (low-income) population. The beneficiaries of the project included: i. RDIs, enterprises and grassroots innovators: strengthened capacity to develop, upgrade and adopt inclusive technologies. ii. SMEs: supported to adopt and absorb technology and innovations or to work closely with R&D institutions to upgrade their technological capabilities and enhance competitiveness. iii. Agencies responsible for SME development policy and inclusive innovation policy: strengthened capacity in program development, policy coordination, program implementation, monitoring and evaluation. iv. The BoP population: to benefit both directly and indirectly during and beyond the project life. Grassroots innovators would benefit during the life of the project. In the long-term, the BoP population would be the end- users of the inclusive technologies developed under the project. Components 13. The Inclusive Innovation Project, with an IDA Credit of SDR 36.7 million ($D 55 million equivalent), consisted of the following four components. Component I: Development of Inclusive Technologies ($12 million). The component aimed to support the development of inclusive technologies that would help address the problems/needs of the BoP population, through competitive grants, based on public calls for proposals and transparent selection criteria. Component 1 aimed to provide support for: (i) developing technological solutions to address a few National Development Challenges (NDCs) specific to Vietnam; and (ii) developing, acquiring, adapting and upgrading inclusive technologies by R&D institutions, enterprises and grassroots innovators. The Component 1 aimed to focus on key priority themes which could have a quick and significant impact on the BoP population. Under Page 8 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Component I, the priorities for support were to be selected by a National Development Challenges Council in Vietnam within the areas of: (i) Traditional Herbal Medicine (THM); (ii) Information and Communications Technologies Applications (as a tool for the delivery of services to the BoP); and (iii) Agriculture/Aquaculture applications. While developing actual solutions, the component aimed to provide ancillary benefits by strengthening the capacity (see Component III) of key participating institutions. Component II: Scale up and Commercialization of Inclusive Technologies ($33 million). This component aimed to provide loans and matching grants to private enterprises for upgrading, scaling up and commercialization of inclusive technologies. This was to include: (i) support to enterprises (SMEs and large enterprises) for scaling up, commercialization and sustainable production of inclusive technologies dealing with manufacturing of products and delivery of services for the BoP; and (ii) funding to private sector SMEs for the acquisition, adoption and use of technology and innovations in priority areas with significant potential for technological upgrading and growth. The general eligibility criteria for sub-project selection under Component I and II included: a) existing or new technologies applicable to Vietnam; b) readiness for adoption/adaptation and cost reduction potential; c) affordable, durable, and environmentally friendly solutions that address the needs of the BoP; d) time-bound deliverables with 2-3 years to reach the prototype stage, or having the potential to commercialize in Vietnam within 2 years; e) multi-disciplinary applications of new ideas/technologies; and f) collaboration among RDIs, enterprises, local and international partners, researchers/innovators to ensure quality and commercialization potential. Component III: Capacity Building and Global Knowledge Transfer ($7 million). Component III aimed to provide capacity building support to key Vietnamese national institutions such as the National Institute of Medicine Materials (NIMM), the National Hospital of Traditional Medicine (NHTM), NAFOSTED, Vietnam Academy of Sciences and Technology (VAST), Hanoi University of Science and Technology (HUST) for the sustainable development and delivery of inclusive technologies in the priority areas supported under the project. These included: a) enhancing technology, quality and clinical trials of traditional herbal medicine products; b) enhancing innovation grant management capacity; and c) enhancing capacity in technology transfer and commercialization, and intellectual property rights protection. Component IV: Project Management, Monitoring and Evaluation ($3 million). Component IV aimed to support the Ministry of Planning and Investment (MPI) Project Management Unit (PMU) and the Project Implementation Unit (PIU) in the National Foundation for Science and Technology Development (NAFOSTED), and the Participating Financial Intermediaries (PFIs) in project management, coordination, capacity building, oversight, monitoring and evaluation, reporting and audits. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Page 9 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Other Changes 14. A restructuring process was completed in April, 2017. Through this restructuring, the project closing date was moved from the original date of November 30, 2018 to 30 June 30, 2017 and the project cancelled. Rationale for Changes and Their Implication on the Original Theory of Change 15. The decision to cancel the project – a decision that was shared between the client and the bank – was a result of lack of progress in implementing project activities. As is outlined in more detail in the section on key factors that affected implementation and outcome, the project experienced a number of implementation issues including: (a) lack of enforcement by MPI in implementing institutional and legal agreements; b) implementation capacity constraints at the level of MPI and the technical level of NAFOSTED within the Ministry of Science and Technology (MoST) the main implementing agency for the matching grants components of the project; and (c) weak inter-ministerial collaboration between MPI and MoST and other implementing agencies. 16. The Bank undertook a Mid-Term Review (MTR) in May 2016. During this MTR, restructuring proposals were discussed with the PMU-with a view to find solutions to the above described issues. The MTR proposed to move the project, in its entirety to MoST, which had the technical expertise on innovation policy. This solution would have helped streamline the decision-making process, but was not accepted by MPI. The Bank team explored other options including changing the policy instrument into an IPF with Disbursement Linked Indicators (DLIs). However, an agreement could not be reached on a concrete restructuring proposal. In light of continued lack of implementation progress- and the reduced credibility and likely interest in this project amongst potential beneficiaries after the delays experiences, the Bank recommended the project be cancelled by advancing the closing date of the credit. Canceling project funds prior to June 30, 2017 allowed for the funds to be recommitted towards other IDA projects in Vietnam (IDA funds reallocation) – particularly important as Vietnam graduated from IDA on July 1, 2017. 17. While the project theory of change and the PDO remained relevant (see section below), the use of the matching grant instrument became a major issue during implementation. This was primarily a result of: (a) a lack of adherence by the client to the agreed selection and approval processes, which were outlined in the legal agreement and project manual and (b) the government view that as private sector would benefit from government funds, the government should decide what was to be considered an innovation sub-project. Other Innovation projects around the world have also suffered this issue. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 18. The original development objective “to adopt, upgrade and develop Inclusive Innovations for the benefit of the Base of the Pyramid population, by: (i) strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies, and (ii) improving RDIs' and SMEs' technological and innovation capabilitiesâ€? is consistent with the Page 10 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) current Vietnam Country Partnership Framework (CPF) FY 18-22, specifically Focus Area 1: Enable Inclusive Growth and Private Sector Participation and Focus Area 2: Invest in People and Knowledge. The development objective is also consistent with the Vietnam 2035 Report’s chapter on Innovation and the government’s SEDS. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 19. The development objective of the project has two outcomes: (i) Strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies This part of the PDO was not achieved by the closing of the project as evidenced by the lack of progress in the two PDO indicators: (a) Number of pilot inclusive technologies developed/upgraded that are likely to benefit the BoP (end target was twenty-five but no pilots were done by project closing); and (b) Number of pilot inclusive technologies commercialized that are likely to benefit the BoP (end target was eight but no pilots were done by project closing). (ii) Improving RDIs' and SMEs' technological and innovation capabilities. This part of the PDO was also not achieved by the closing of the project. While not having a specific PDO level indicator, all related intermediate results level indicators were unachieved at project close. Justification of Overall Efficacy Rating 20. The rating for the overall Efficacy of the project is Negligible. At project close, four years after project approval, none of the intended project activities had been implemented except activities under component four on Project Management, Monitoring and Evaluation. As mentioned above, no progress was made towards achievement of the two PDO level targets and related intermediate level results targets by project close. Only the end project target intermediate level results indicators related to component IV on Project Management, Monitoring and Evaluation were partially or fully achieved. A PMU/PIU maintained, M&E system had been established and rated at 50 percent achieved and financial management and procurement systems were established and maintained both rated at 100 percent achieved. C. EFFICIENCY Assessment of Efficiency and Rating Page 11 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 21. An economic and financial analysis of the project with net present value and financial rate of return was not conducted by the Bank team at appraisal. This was justified by the team in the PAD at the time as follows; (a) the nature of innovation/technological projects and the difficulties in measuring the externalities related to innovation made an economic and financial analysis not feasible; (b) as the project was to implement sub- projects which were not pre-identified, the economic benefits could only be presented qualitatively. It is understood that a full economic and financial analysis is now required for innovation projects just as with other types of projects. 22. From an Ex Post perspective, as the planned pilot sub-projects and other activities were not implemented at project closure, an economic analysis would not be feasible or meaningful. However, given that $ 1.38 million was spent on Project Management, Monitoring and Evaluation and Supervision and that there was no actual expenditure on planned activities thus no impact or outcomes, the economic analysis would most likely result in a negative rate of return. 23. Based on the above, the Efficiency of the project is rated as Negligible. D. JUSTIFICATION OF OVERALL OUTCOME RATING 24. Based on a combined assessment of project efficacy and efficiency, both assessed as Negligible, the overall outcome of the project is rated as Unsatisfactory. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 25. As noted in the PAD for the Inclusive Innovation Project, the Bank had a number of high level (Deputy Prime Minister and Minister level) and technical (RDI, major universities, government agencies, donors and private sector) discussions between 2009-2012. Strong buy-in was expressed by all stakeholders for the Science, Technology reform agenda with a special focus on promoting inclusive innovation. All key stakeholders had been consulted on the project concept during the preparation of the project and strong support was expressed for the project. 26. It should be noted that inclusive innovation was a new concept in Vietnam at project design and there was thus need for enhanced information sharing and knowledge transfer among key stakeholders to ensure greater ownership of the project. In this regard, the Bank supported a number of analytical and capacity building initiatives including a policy paper, “Vietnam Innovation Policy for Competitiveness, Economic Growth and Social Equity (December, 2009), OECD Report on â€?Vietnam Science, Technology and Innovation Reviewâ€? and World Bank Institute, “Rapid Innovation Assessment of Vietnam Innovationâ€?. In addition, a number of stakeholder workshops were held and a 2010 study tour to India for South-South knowledge sharing on Innovation undertaken. B. KEY FACTORS DURING IMPLEMENTATION 27. Lack of enforcement of agreed institutional and legal agreements: Soon after the implementation commenced, Page 12 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) the project suffered from a lack of enforcement by MPI in implementing the institutional and legal agreements that had been agreed to during project preparation. The project also experienced implementation capacity constraints at the level of MPI and at the technical level with NAFOSTED in Ministry of Science and Technology (MoST) the main implementing agency for the grants components of the project. This was further exacerbated by weak inter-ministerial collaboration between MPI and MoST and other implementing agencies. The Project Steering Committee (PSC) which could have helped bridge this lack of coordination and was supposed to meet semi-annually to resolve implementation bottlenecks, as agreed during project preparation, was not able to meet as planned or conduct this function as proposed. The section below provides a detailed account of problems faced with during implementation and the factors that led to the cancellation of the project. 28. The project was approved by the World Bank Board of Executive Directors on May 13, 2013. The Financing Agreement was signed on September 6, 2013. There were no effectiveness conditions, and the project became effective on December 6, 2013. Implementation started off at a slow pace due to a number of factors: First, the PSC was only officially established on October 3, 2013 with Deputy Prime Minister as Chairman. Second, EDA (MPI) internal procedures required all decisions to be approved by the MPI Vice Minister, something that made implementation cumbersome. Third, NAFOSTED did not as specified in the implementation arrangements of the project, have autonomy. This meant activities that should have been directly under the purview of NAFOSTED (e.g. hiring PIU staffing, hiring of Technical Advisor, selection of the technical panel, engaging peer reviewers) were delayed. 29. Nineteen months after reaching effectiveness, in February 2015, the first call for proposals was launched under the project. The call received 70 applications and the evaluation was completed by the technical panel. The evaluation report was submitted to the World Bank for No Objection (NO) in October 2015, clearance of which would be required before any sub project can be funded. The Bank provided its NO to 11 subprojects. The first NO was for nine sub projects, the second was for a additional two sub-projects (with a condition to change the sub project’s category from C1 to C2). The procedures required that the list be approved by the NAFOSTED chairman. However, there was a disagreement voiced by the MPI PMU concerning the selection process, stalled the process and an attempt to re-select sub-projects. This was done after the technical panel had reviewed and selected 11 sub-projects, jeopardizing the legitimacy of the process that had been established. It was recommended by MPI PMU that some of the ideas proposed should be advertised on local television to solicit better ideas. 30. The Bank team strongly advised against the PMU recommendation to advertise research/business ideas from any of the 11 proposed sub-projects with an objective to attract more applicants for those ideas for re-appraisal. The Bank viewed these as ideas of a proprietary nature that and could not be offered to other candidates, without undermining confidentiality and fiduciary requirements. The Bank advised MPI that taking this route would potentially damage the credibility of the VIIP and would be deemed unfair to the selected applicants, particularly considering the perception of weaknesses in the protection of intellectual property rights in Vietnam. The PMU recommendation also differed from the independent processes set up for the grants approval process. The Bank objected to the proposal to advertise and thus this course of action was not carried out. 31. In light of the deadlock faced in implementing the project, two high level meetings were held. The first meeting took place in December 2015 between the Country Director and Minister of MoST to seek the Minister’s guidance on the pending approval of the 11 sub-projects (six weeks after the meeting, on March 2, 2016, two Page 13 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) sub-projects out of the eleven were approved by MPI). A second meeting was held in February 2016 between the Country Director and the Deputy Prime Minister with a view to clarifying the roles and responsibilities of the MPI and MoST in project implementation. It was agreed that MoST should take the lead on the matching grants component but a letter was subsequently issued by the Office of Government (OOG) that MoST should not approve any sub projects without MPI acceptance. 32. In August 2015, the MPI PMU launched the second call for proposals without the first call being finalized. By September 2015, 106 applications had been received, none of which could be sent to the peer reviewers and the technical panel. This was a result of no agreement between NAFOSTED and the MPI PMU relating to a set of proposed revisions to the forms and processes that the international advisor had worked on with NAFOSTED with a view to streamline them in light of the lessons learnt from the first round. 33. There were also delays in the implementation for Components 2 and 3. Component 2, the largest component, was designed to provide loans and matching grants to private enterprises for upgrading, scaling up and commercialization of inclusive technologies. The selection criteria were outlined in the Project Manual. The first and second calls received a small number of applications. To reach a larger number of potential applicants and benefits to SMEs strengthened, it was recommended that more PFIs be accredited. In December 2015, the MPI PMU therefore agreed to expedite the accreditation of more PFIs to the Project. However, the revised due diligence report for accrediting the new PFIs submitted on June 24, 2016 (after the May 2016 MTR) did not address the comments that were provided by the Bank team on a previous draft. 34. Delays in Procurement: Component 3 was designed to support capacity building within key Vietnamese national institutions such as the National Institute of Medicine Materials (NIMM), the National Hospital of Traditional Medicine (NHTM), NAFOSTED, Vietnam Academy of Sciences and Technology (VAST), and Hanoi University of Science and Technology (HUST) for the sustainable development and delivery of inclusive technologies in the priority areas supported under the project. Specifically, NIMM and NHTM would focus their efforts on pilot lab level THM raw materials farming, R&D and pilot THM production, while transferring knowledge and technologies to the private sector for the commercialization and scaling up of actual traditional medicine manufacturing and sale to final consumers. This involved procuring specialized medical equipment for these pilot lab facilities. The technical specifications for the four packages for the medical equipment (three packages for NIMM and one for NHTM) were shared with the Bank team that reviewed and provided comments. After a delay of more than two years, these technical specifications would need to be incorporated into bidding documents that would require Bank review and clearance prior to the bidding process being initiated. Thereafter, the PMU required close to 10 months to procure the equipment once the specifications are finalized, seriously questioning the viability of this component benefiting the institutions during the life of the project. 35. Attempts at Restructuring: Given the lack of progress despite agreements being reached with the Deputy Prime Minister, and with only two years left for project implementation, the PDO, and implementation progress ratings were downgraded to Unsatisfactory. Further, lack of progress in implementing any of the components led to the downgrading for each of the subcomponents to Unsatisfactory. The Bank team undertook a MTR in May 2016 which was used to discuss restructuring proposals with the PMU. Options considered included transferring the project in its entirety to MoST, which was considered to have the technical expertise on innovation policy. It was also believed that transferring responsibilities to MoST would help streamline the decision-making process. However, MPI did not agree with this proposal. The Bank team explored other options including changing the policy instrument. However, agreement could not be reached on a concrete restructuring Page 14 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) proposal. In light of the continued lack of implementation progress, and the reduced credibility and interest in the project among potential beneficiaries as a result of implementation delays, the Bank recommended the project be cancelled by advancing the closing date of the credit. This allowed the Government of Vietnam to recommit the remaining IDA resources before the end of FY17 when Vietnam graduated from IDA. 36. Cancellation of Project: The project was discussed at meetings between the World Bank and the OOG and MPI officials highlighted the need to expedite the process to cancel the project in order to allow for remaining resources to be recommitted to FY17 IDA projects. On Aug 5, 2016 a letter (ref 219/ TB-VPCP) was sent from the OOG documenting the decision made to cancel the project. There were several official letters exchanged in the following months, and on March 31, 2017, the Bank received the formal request to cancel the undisbursed funds and close the project. 37. On April 18, 2017 upon completion of the internal Bank restructuring process, which was also brought the closing date to June 30,2017, a letter was sent to the State Bank of Vietnam (SBV) confirming the cancellation of SDR 36,700,000. On June 30, 2017, a second cancellation in the amount of SDR 168,528.53 took place, cancelling funds from Category 3 which had been refunded to the Bank by the MPI PMU. All cancelled funds were recommitted to other Vietnam IDA projects. 38. To summarize the events described above; Key Events Dates Board approval of project May 13, 2013 Project Financing Agreement signed September 6, 2013 Project Effectiveness December 6, 2013 First call for proposal launched February, 2015 Second call for proposals launched August, 2015 Technical Panel Report sent to the Bank for No objection October, 2015 MPI convened a meeting with all 70 applicants from Call 1 November, 2015 WB Country Director met with Minister MoST December 5, 2015 WB Country Director met with DPM- Decision made that NAFOSTED would lead the selection February 4, 2016 process for the grants component of the project NAFOSTED Chairman (also VM, MoST) approved 2 out of 11 sub projects that were approved by March 2, 2016 VM MPI MTR mission – no concrete restructuring proposal agreed to. The Bank recommended the project May 2016 be cancelled and remaining funds reallocated to other projects Project put forth by OOG and WB to be included in omni bus restructuring July 12, 2016 Letter (ref 219/ TB-VPCP) was received from the OOG confirming the decision made to cancel the Aug 5, 2016 project. Letter from MPI was sent to DPM (Ref 9011/BKHDT-KTDN) reporting the cancellation of Oct 28, 2016 undisbursed funds of approximately USD 50M. SBV request to MPI (in response to a letter sent by WB on Dec 5, 2016) to finalize the procedures Dec 15, 2016 and submit to the government for approval the cancellation of the five selected projects (VIIP being one of them). The Bank received the formal request to cancel the undisbursed funds and close the project March 31, 2017 First restructuring completed with cancellation of SDR 36,700,000 of project funds April 18, 2016 Page 15 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Second cancellation was undertaken. Cancellation of SDR 168,528.53 from Category 3 which had June 30, 2016 been refunded to the Bank by the MPI PMU. Closing date advanced to June 30, 2017 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 39. The project’s theory of change as laid out in the PDO is clear and well-articulated. However, the two PDO level indicators only cover the first outcome of Strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies whereas the second outcome of Improving RDIs' and SMEs' technological and innovation capabilities is not clearly covered. The intermediate level indicators were comprehensive and relevant. M&E Implementation 40. Under the project management arrangements, the MPI Enterprise Development Agency (EDA) was responsible for the project M&E. In the last Implementation Status and Results Report (ISR) of July 2017, M&E was rated as Unsatisfactory and the intermediate result indicator on M&E indicated that only 50 percent of the end target was met (M&E system established and M&E reports delivered on time). M&E Utilization 41. Since planned activities under the project were not implemented, the PDO level outcome and intermediate level results indicators target not reached. Further, M&E systems and mechanisms were never put to the test or implemented. The lack of actual implementation of project activities meant M&E data was not collected or analyzed, nor could M&E data be used to inform project management, decision making or implementation. The only data for intermediate level indicators under Component four (Project Management, Monitoring and Evaluation) was, in fact, collected and analyzed including intermediate indicators regarding the establishment of M&E system and reports (rated at 50 percent achieved at project closing; PMU and PIU maintained (100 percent achieved at the closing of the project); and financial management and procurement systems established and maintained (100 percent achieved at project closing). Justification of Overall Rating of Quality of M&E 42. The M&E system designed for the project is considered appropriate. However, it was never really implemented or tested apart for the data collected and utilized for M&E under Component Four (Project Management, M&E). Page 16 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 43. The overall quality of M&E is rated as Modest in recognition that there were significant shortcomings in the M&E system’s implementation and utilization. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 44. The project had no environmental, social and fiduciary compliance issues. The project was a category B (Partial assessment). The only Bank safeguard policies triggered was OP 4.01 due to the potential adverse environmental and social impacts associated with planned innovative sub-projects under Components one and two and upgrading activities of laboratories two institutions under Component 3. An Environmental and Social Management Framework (ESMF) was prepared by the client and approved by the Bank. No Social reviews were triggered. The project met all fiduciary compliance requirements. C. BANK PERFORMANCE Quality at Entry 45. The project design reflected the strong interest and commitment of the stakeholders (including RDIs, major universities and the private sector) in Vietnam to provide innovative solutions to the Base of the Pyramid (BoP). The project was strategically relevant as it was aligned with two of the three pillars of the Vietnam CPF at appraisal and as well as the current CPF (see relevance section). The technical aspects at entry were well addressed by the team including following best practice of other innovation projects and key lessons learned in implementing similar projects in other countries. Given the risky nature of the project, the team is commended for taking the risk in the design of this pioneering project. However, a financial and economic analyses was not conducted by the team at appraisal due to reasons mentioned in the efficiency section of the ICR, an omission that negatively effects the overall rating of Bank performance for the project. In addition, a financial management assessment and procurement capacity and risk assessment of the implementing agencies were undertaken and risk mitigation measures proposed. A ESMF was prepared and approved before project effectiveness. The M&E framework was rated as Satisfactory during the first ISR in 2013. 46. The project may have benefitted from a more streamlined design. Implementation arrangements, straddling two ministries, were not optimal. However, this set-up was the preference of the client. Regular implementation support missions were conducted where the Bank team worked with NAFOSTED to strengthen their capacity and with the MPI PMU to share key lessons learnt and selection criteria used in other countries. Quality of Supervision 47. The Bank team provided hands on support to NAFOSTED (implementing agency for matching grants component) by sharing knowledge on how such matching grants schemes have been managed in other countries and through international experts who provided capacity support. The Bank also engaged an international consultant to work with NAFOSTED to revise forms and make procedures more practical. Page 17 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 48. A number of the risk mitigating measures outlined in the PAD were implemented during implementation including: (a) have the Deputy Prime Minister chair the Project Steering Committee; (b) provide TA to PFI’s; (c) timely establishment of PMU and PIU and appointment of qualified staff; (d) appointment of a technical review panel to provide independent review of proposals according to agreed sub-project criteria. 49. The team documented implementation status through regularly updated Implementation Status Reports (ISRs) and detailed mission Aide Memoires. In addition, as soon as it was clear that the implementation arrangements and project design were not working, the Bank team used implementation support missions to bring these delays to the attention of the stakeholders and identify options for addressing issues. Finally, the team used the MTR to explore new implementation modalities as well as different instruments (e.g. Investment loans with disbursement linked indicators). Acknowledging the fact that there was no credible restructuring option, the Bank team worked diligently to cancel and advance the closing date of the project such that the IDA funds could be recommitted to other projects as Vietnam was graduating to an IBRD country and if not recommitted in FY 17, would lose these IDA funds. The timely closure and cancellation of the unutilized funds enabled these funds to be recommitted towards other IDA funds for Government of Vietnam. Justification of Overall Rating of Bank Performance 50. The Bank team identified and brought to the management attention the emerging problems and concerns in implementation as early as the second ISR (July 2014) where the team downgraded the IP rating from S to MS. Recommendations such as delegating the chairmanship of the Steering Committee from the Deputy Prime Minister to MPI Minister to ensure issues were handled speedily, was agreed to and implemented with the client as early as during the second ISM. Subsequent downgrades of the IP rating to MU and then to U as well as the continued highlighting of continuing implementation coordination, governance and other issues were clearly highlighted by the Bank team and brought to the attention of management and both Bank and client senior management held a number of meetings to attempt to resolve key issues. 51. Based on the recognition of problems in implementation of the project, CMU provided increased supervision budget to the team of $72k (FY14); $75k (FY15); $119k(FY16) and $91k (in the last year of implementation, FY17). The team used this increased budget to, for example, engage international experts to provide hands on support to NAFOSTED staff to build their capacity, engage an international consultant to work with NAFOSTED to revise and simplify forms to make procedures more practical and efficient and also provided additional TA to PFIs. 52. As mentioned, elsewhere in the ICR, during a number of ISMs, high level meeting at Bank Country Director Level and Deputy Prime Minister/Minister level were held and issues agreed only for these agreements to be not honored or implemented. Due to this lack of progress, several options, including cancellation were considered during the 5th ISR in March 2016 ahead of the Mid Term Review. As the role of the Bank is to provide implementation support and the counterpart/client is responsible for actual implementation, the Bank team performed as best as they could under the circumstances. 53. Based on the above, the overall Bank performance is rated as Moderately Satisfactory. Page 18 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) D. RISK TO DEVELOPMENT OUTCOME 54. The risk to Development Outcome is the risk, at the time of completion, that development outcomes, which have been achieved or are expected to be achieved, will not be maintained. In the case of the Inclusive Innovation Project, the following development outcome indicators were used: (a) number of inclusive technologies developed/upgraded that are likely to benefit the BoP and (b) number of inclusive technologies commercialized or scaled up by enterprises that are likely to benefit the BoP. 55. Both design and implementation risk were rated Substantial at the project approval stage, reflecting the Bank team’s recognition of the risk to the development outcome due to the project design and implementing arrangements. The implementation risk did indeed materialize despite the team’s risk mitigation measures. The fact that project management related intermediate level indicators were all fully or partially achieved while the none of the implementation PDO level and intermediate result level indicators were achieved indicates that the project’s design may not have been suitable to its operating environment. V. LESSONS AND RECOMMENDATIONS (a) Design complexity 56. Implementation readiness: Given that this was a pilot project in Vietnam, the implementation readiness could have been emphasized further during project preparation evaluating the capacity of the various implementing agencies and conducting a demand analysis of the grants among prospective applicants. This could have helped prevent some of the delays faced during implementation. 57. Pilot before Scaling: A key lesson that the Bank could consider is that when designing Inclusive Innovation projects in future, TTLs should consider conducting smaller, more rigorous experiments in future before launching a larger operation. 58. Application of international best practice: Due assessment should be made of local institutional and pollical economy conditions when applying international best practice. 59. Suitable and realistic design. The Inclusive Innovation Project demonstrates the need for Bank teams to realistically assess the suitability of project design to its operating environment. While the Bank team acknowledged the cumbersome approval procedures and processes in Vietnam, the team seemed to view the DPM chairing of the Project Steering Committee as a sufficient mitigation measure. 60. Inclusive Innovation in Socialist Economies: While there is limited literature in this area, there are important lessons to be learnt when designing and implementing Inclusive Innovation projects in economies with socialist foundations (such as Vietnam or ECA region) as funding innovation directly to enterprises is counter-intuitive, especially for ministries traditionally in charge of science, technology and research. (b) Client Ownership Page 19 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 61. Lack of a strong and consistent champion: During the project design, the DPM and the Vice Minister MoST were strong champions for the project. However, once the project was approved there were a change of leadership and hence a shift in the priorities. The lesson learnt was that while strong champions are important, the project implementation needs to be simple with clear communication and accountability on the part of all the relevant implementation agencies. 62. Clear definition of new concepts. Implementation of the Vietnam Inclusive Innovation Project shows the importance of broad support, buy-in and common understanding of key concepts amongst all key stakeholders for projects that introduce new models such as “Inclusive Innovationâ€?. While the PAD mentions that Inclusive Innovation was identified by “Senior Officialsâ€? as an appropriate measure for reducing growing disparities and improving competitiveness in Vietnam, there was not a common understanding between the PMU (MPI) and PIU (MoST) as to what was to be considered as “Inclusive Innovationâ€?. This resulted in the PMU not approving some sub projects as they were not considered innovative or inclusive. New concepts and models should be clearly defined and agreed to by all parties (the most logical place being in the Project Manual and/or in the Financing Agreement). (c) Implementation arrangements 63. Practical implementation arrangements: The main reason that led to the lack of implementation progress and cancellation of the project was the fact that this project was straddling two ministries and this was exacerbated by clear a lack of coordination between MPI and MoST. Innovation projects are multi-sectoral by their nature and therefore require inter-ministerial policy and implementation coordination. The ministry that had the technical know- how was MoST, where the implementation agency NAFOSTED was housed. However, in spite of the legal project documents stating that NAFOSTED is the lead implementation agency for the matching grants component, there was a lack of agreement between MPI PMU and NAFOSTED PIU on the decisions and the procedures to be followed for the grants selection process. While this arrangement of having, inter-ministerial projects is not unique to this project, the lack of adequate implementation capacity in MPI and the complete lack of coordination between MPI PMU and NAFOSTED PIU made this implementation arrangement an impediment to effective project implementation. The decision to have MPI as the implementing agency was made at the design stage as MoST was designated as the implementing agency for another Innovation related project, the Fostering Innovation through Research, Science and Technology (FIRST) Project, which was being designed at the same time and it was agreed that having MoST implement the two projects at the same time would be risky. 64. Adherence to agreed-upon implementation arrangements: The PAD clearly stated that project would adhere to a decentralized implementation approach involving the professional science and technology program management institutions such as NAFOSTED to manage the selection of sub-projects with the support of an independent Technical Review Panel. Selection of sub-projects was to follow the detailed eligible criteria developed in line with the project development objectives and laid out in the Project Manual. At the same time, procurement, supervision, monitoring and evaluation for other project components was centralized at EDA to ensure fund management meets the fiduciary requirements of the World Bank. However, these basic principles were not followed during project implementation leading to the breakdown of the processes put in place. Thus, the lack of adherence to abide to agreed selection criteria, honoring the recommendations of the independent technical review panel and the advice of the Bank’s oversight function pointed to clear governance issues. 65. Matching grants as instruments of support to the private sector: Though government had agreed during preparation, to the project structure and processes, the government appeared uncomfortable about giving matching Page 20 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) grants to private sector actors. The government view was that as the private sector were beneficiaries of funds from the government and therefore government should decide on what was to be considered an innovation sub-project and who the beneficiaries should be, despite the project having clearly established and transparent procedures for this purpose (including an independent technical evaluation panel). In the context of Vietnam, the matching grants instrument may require more client buy-in upfront, through sharing of learning from other countries where matching grants have been used successfully to strengthen the private sector). . Page 21 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. Number of pilot inclusive Number 0.00 0.00 25.00 0.00 technologies developed/upgraded that are 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 likely to benefit the BoP Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Number of pilot inclusive Number 0.00 0.00 8.00 0.00 technologies Commercialized that are likely to benefit the 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 BoP Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Page 22 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Objective/Outcome: Improving RDIs' and SMEs' technological and innovation capabilities Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 1. Number of pilot inclusive Number 0.00 0.00 25.00 0.00 technologies developed/upgraded that are 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 likely to benefit the BoP Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion 2. Number of pilot inclusive Number 0.00 0.00 8.00 0.00 technologies Commercialized that are likely to benefit the 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 BoP Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. A.2 Intermediate Results Indicators Component: Component II: Scale up and Commercialization of Inclusive Technologies Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 23 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Intermediate Result indicator Number 0.00 0.00 8.00 0.00 One: Number of pilot inclusive technologies 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 commercialization sub- projects funded Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Number 0.00 0.00 30.00 0.00 Two: Number of SMEs technology upgrading 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 projects funded Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Component: Component III: Capacity Building and Global Knowledge Transfer Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Number 0.00 0.00 2.00 0.00 One: Number of laboratories/ plants 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 upgraded Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Page 24 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Number 0.00 0.00 3.00 0.00 Two: A network of domestic and /or global organizations 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 formed Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Component: Component IV: Project Management, Monitoring and Evaluation Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Percentage 0.00 0.00 100.00 100.00 One: PMU and PIU maintained during the 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 project implementation and staffed with required technical and professional experts according to agreed TORs Comments (achievements against targets): Achieved at project closing. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 25 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Intermediate Result indicator Percentage 0.00 0.00 100.00 100.00 Two: Financial management and procurement 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 management systems established and maintained with FM and procurement professionals engaged throughout the project Comments (achievements against targets): Achieved at project closing. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Percentage 0.00 0.00 100.00 50.00 Three: M&E system established and M&E reports 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 delivered on time Comments (achievements against targets): The end target was partially achieved at project closing (50%). Component: Component I: Development of Inclusive Technologies Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Number 0.00 0.00 6.00 0.00 One : Number of NDC Solutions funded 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 Page 26 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Number 0.00 0.00 25.00 0.00 Two: Number of pilot inclusive technology 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 solutions funded Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Intermediate Result indicator Number 0.00 0.00 30.00 0.00 Three: Number of grassroots innovations 06-Dec-2013 06-Jan-2017 30-Jun-2017 30-Jun-2017 funded Comments (achievements against targets): End target at project closing was not achieved due to non implementation of planned activities. Page 27 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 1. Number of inclusive technologies developed/upgraded that are likely to benefit the BoP. Outcome Indicators 2. Number of inclusive technologies commercialized or scaled up by enterprises that are likely to benefit the BoP. 1. Number of NDC Solutions funded. 2. Number of pilot inclusive technology solutions funded Intermediate Results Indicators 3. Number of grassroots innovations funded. 4. Number of SMEs technology upgrading projects funded Component 1: Developing Inclusive Technologies There were no outputs at project closing due to non-implementation of activities and thus the outcome was not achieved. Key Outputs by Component (linked to the achievement of the Objective/Outcome 1) Component 2: Upgrading, Scaling up and Commercialization of Inclusive Technologies. There were no outputs at project closing due to non-implementation of activities and thus the outcome was not achieved. Objective/Outcome 2 1. Number of inclusive technologies developed/upgraded that are likely to benefit the BoP. Outcome Indicators 2. Number of inclusive technologies commercialized or scaled up by enterprises that are likely to benefit the BoP. Intermediate Results Indicators 1. Number of laboratories/plants upgraded. Page 28 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) 2. A network of domestic and/or global organizations formed. Component 3: Capacity Building and Global Knowledge Transfer Key Outputs by Component (linked to the achievement of the Objective/Outcome 2) There were no outputs at project closing due to non-implementation of activities and thus the outcome was not achieved. Page 29 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Smita Kuriakose Task Team Leader(s) Kien Trung Tran Procurement Specialist(s) Ha Thuy Tran Financial Management Specialist Ngan Thuy Nguyen Team Member Viet Quoc Trieu Team Member Lien Anh Pham Team Member Linh Thi Anh Vu Team Member Son Van Nguyen Environmental Safeguards Specialist Thuy Cam Duong Social Safeguards Specialist B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 32.625 233,307.85 FY12 27.373 162,557.80 FY13 58.891 300,505.19 Total 118.89 696,370.84 Supervision/ICR FY11 0 0.00 Page 30 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) FY12 0 0.00 FY14 22.001 72,594.14 FY15 18.050 75,440.65 FY16 27.176 118,810.05 FY17 15.050 90,879.45 FY18 3.425 20,610.41 Total 85.70 378,334.70 Page 31 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Developing Inclusive 0 0 0 Technologies Upgrading, Scaling up and Commercialization of 0 0 0 Inclusive Technologies Capacity Building and Global 0 0 0 Knowledge Transfer Project Management, 0 1,384,026.00 0 Monitoring and Evaluation Total 55,625,000.00 1,384,026.00 2.49 Page 32 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) ANNEX 4. EFFICIENCY ANALYSIS N/A Page 33 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The borrower was formally requested to provide comments but none were received. Page 34 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) ANNEX 6. SUPPORTING DOCUMENTS 1. Concept Stage Public Information Document / Integrated Safeguards Data Sheet: Aug 2.2012 2. Program Appraisal Document, April 12, 2013 3. Financing Agreement for Credit 5249-VN: 6-Sep-2013 4. Project Agreement for Credit 5249-VN: 6-Sep-2013 5. Implementation Status & Results Report (ISR) ISR1: 23-Oct-2013 ISR2: 8-Jul-2014 ISR3: 28-Jan-2015 ISR4: 23-Sep-2015 ISR5: 28-Mar-2016 ISR6: 01-Aug-2016 ISR7: 22-Jan-2017 ISR8: 19-Jul-2017 Page 35 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Page 36 of 38 The World Bank Vietnam Inclusive Innovation Project (P121643) Page 37 of 38