Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) Report Number : ICRR0021261 1. Project Data Project ID Project Name P121643 Vietnam Inclusive Innovation Country Practice Area(Lead) Vietnam Trade & Competitiveness L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 30-Nov-2018 55,625,000.00 Bank Approval Date Closing Date (Actual) 13-May-2013 30-Jun-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Malathi S. Malathi S. Jayawickrama IEGEC (Unit 1) Jayawickrama 2. Project Objectives and Components a. Objectives According to the Financing Agreement and the Project Appraisal Document (PAD, p. 5), the objective was "to adopt, upgrade and develop Inclusive Innovations for the benefit of the Base of the Pyramid (BoP) population, by: (i) strengthening the Recipient's capacity to undertake Inclusive Innovation, including financing development, adaptation, adoption, scale-up and commercialization of Inclusive Technologies; and (ii) improving Research and Development Institutions' (RDIs') and small and medium enterprises' (SMEs') technological and innovation capabilities." Page 1 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) The BoP population was defined as the 2.6 billion people worldwide, a majority of whom live in Asia, who subsist on less than US$ 2/day, and who lack access to basic services and reliable markets to buy and sell goods and services. "Inclusive Innovation" was defined as development and delivery, through public and private sector initiatives, of high-performance products and solutions at a low cost that are affordable to, and benefit, resource-poor or BoP people (Project Appraisal Document (PAD), p, 2). b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project contained four components: 1. Development of Inclusive Technologies (appraisal: US$12 million; actual US$0). This component was to support sub-projects for: (i) developing technological solutions to address National Development Challenges specific to Vietnam, including traditional herbal medicine, information and communications technology applications, and agriculture/aquaculture applications; and (ii) developing, acquiring, adapting, and upgrading inclusive technologies by RDIs, enterprises, and grassroots innovators. 2. Scale-up and commercialization of inclusive technologies (appraisal: US$33 million; actual US$0). This component was to provide loans and matching grants to private enterprises, to support SMEs and large enterprises for scaling up, commercialization, and sustainable production of inclusive technologies dealing with manufacturing of products and delivery of services for the BoP population; and to support SMEs in the acquisition, adoption, and use of technology and innovations in priority areas with significant potential for technological upgrading and growth. Under both the first and second components, the eligibility criteria for sub-project selection were to include: (a) existing or new technologies applicable to Vietnam; (b) readiness for adoption/adaptation and cost reduction potential; (c) provision of solutions to address the needs of the BoP population that were affordable, durable, and environmentally friendly; (d) the specification of time-bound deliverables within 2-3 years of the prototype stage, or with the potential for commercialization in Vietnam within two years; (e) the multi-disciplinary application of new ideas and technologies; and (f) collaboration among RDIs, enterprises, local and international partners, and researchers/innovators to ensure quality and commercialization potential. 3. Capacity building and global knowledge transfer (appraisal: US$7 million; actual US$0). This component was to support capacity building for key Vietnamese national institutions such as the National Institute of Medicine Materials, National Hospital of Traditional Medicine, National Foundation for Science and Technology Development (NAFOSTED), Vietnam Academy of Sciences and Technology, and Hanoi University of Science and Technology, for the sustainable development and delivery of inclusive Page 2 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) technologies in: (a) enhancing technology, quality, and clinical trials of traditional herbal medicine products; (b) enhancing innovation grant management capacity; and (c) enhancing capacity in technology transfer and commercialization, and in intellectual property rights protection. 4. Project management, monitoring, and evaluation (appraisal: US$3 million; actual US$1.38 million). This component was to support the Project Management Unit (PMU) in the Ministry of Planning and Investment's (MPI's) Enterprise Development Agency (EDA), Project Implementation Units (PIUs) in NAFOSTED, and participating financial intermediaries for project management, coordination, capacity building, oversight, monitoring and evaluation (M&E), reporting, and audits. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Costs: Total project cost at appraisal was US$55.625 million. Actual costs were reported only for project management, in the amount of US$ 1.38 million. Financing: The project was to be financed by a US$55 million Specific Investment Loan from the International Development Association (IDA), with about US$20 million supporting technology development through grant financing, US$25 million channeled through a financial intermediary loan to participating financial institutions for on-lending to eligible enterprises (mainly SMEs), and US$10 million for capacity building, project implementation, and M&E. At an April 2017 restructuring, the project was cancelled after US$1.38 million had been disbursed. All remaining funds were cancelled and committed to other Vietnam IDA projects. Borrower contribution: The government's planned contribution was US$0.625 million, to be allocated toward the fourth component on project management. None of this planned contribution was made. Dates: The project was approved on May 13, 2013 and became effective on December 6, 2013. A mid- term review (MTR) was held on May 11, 2016. A restructuring on April 17, 2017 moved up the closing date from the originally planned November 30, 2018 to June 30, 2017 and cancelled the project. 3. Relevance of Objectives Rationale The project's objective was highly relevant to country conditions, government strategy, and Bank strategy at closing. At appraisal, Vietnam was experiencing rapid and consistent economic growth accompanied by overall poverty reduction, yet inequities were increasing. The government had identified "Inclusive Innovation" as a measure to reduce growing disparities and improve overall competitiveness by lowering the costs of products, as well as expanding markets and opportunities for businesses. This project was designed as a pilot in this area. Vietnam's national 2011-2020 Socio-Economic Development Strategy identified a need to harness science and technology development to improve people's lives and Page 3 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) support sustainable and shared economic growth, specifically by reforming the National Innovation System to enhance the capacity of RDIs; strengthen collaboration between RDIs and enterprises; establish national standards, measurement, and assessment; enhance enterprises' capacity to innovate and improve productivity and competitiveness; and develop appropriate public and private financing mechanisms to support innovation activities. Furthermore, the government's June 2009 decree on financial and technical support to SMEs encouraged and supported SMEs to meet new requirements in integration and development. The objectives were directly aligned with two focus areas of the current Country Partnership Framework (FY 2018-22): the first, Enable Inclusive Growth and Private Sector Participation, and the second, Invest in People and Knowledge. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Adopt, upgrade and develop Inclusive Innovations for the benefit of the Base of the Pyramid population Rationale The project's theory of change plausibly linked the project's interventions of targeted grant and sub-loan support to the development, scale-up, and commercialization of inclusive technologies, underpinned by capacity building and global knowledge transfer. This improved innovation and commercialization capacity in areas specifically focused on development of affordable goods and services for the BoP population would eventually lead to tangible benefit for that population. However, the project experienced a number of early implementation challenges, many of which were exacerbated by a change in government leadership (new Deputy Prime Minister and Vice Minister for Science and Technology) shortly after project approval (ICR, p. 20): lack of enforcement by the MPI of institutional and legal agreements; capacity constraints at MPI and NAFOSTED; and weak inter-ministerial collaboration between MPI and the Ministry of Science and Technology (MoST). The Project Steering Committee, which was supposed to meet semi-annually to bridge these kinds of coordination issues, did not function as intended. Use of the matching grant instrument also became an issue, due to lack of adherence by the client to the agreed selection and approval processes, and the government's view that, as the private sector was to benefit from government funds, the government should decide what should be considered an innovation sub-project (ICR, p. 10). In February 2015, nineteen months after effectiveness, the project launched its first call for proposals. 70 applications were received and evaluated by a technical panel. The Bank provided its No Objection to 11 approved sub-projects. However, the MPI PMU disagreed with the selection process, stalled it, and attempted to implement a re-selection, including advertising some of the proposed ideas on local television to solicit better approaches. The Bank team strongly disagreed with the latter recommendation, viewing the Page 4 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) proposals as proprietary and therefore not available for offer to others without undermining confidentiality and fiduciary requirements. Due to the Bank's objection, the advertising was not carried out, but this first round of proposals remained stalled. The MPI then launched a second call for proposals in August 2015, producing 106 applications, but no selection procedure was executed because of disagreement between NAFOSTED and MPI on forms and processes. Similar procedural and institutional coordination issues and delays prevented implementation of the loans and matching grants to private enterprises (under the second component), as well as the capacity building to key national institutions (under the third component). As a result of the deadlock on implementation strategy, none of the intended project activities were implemented except some under the fourth component on project management (the PMU and PIU were established and maintained, and, according to the ICR (p. 26), 50% of the M&E system was established, though no detail is provided on what that means). Specifically, no sub-projects, pilot technologies, innovations, or laboratories/plants were funded/upgraded. There was therefore no progress along the two key outcome indicators: number of pilot inclusive technologies developed or upgraded that were likely to benefit the BoP population (achievement zero, against a target of 25); and number of pilot inclusive technologies commercialized that were likely to benefit the BoP population (achievement zero, against a target of 8). With no substantive outputs or outcomes reported, achievement of the objective is rated Negligible. Rating Negligible PHREVDELTBL PHOVRLEFFRATTBL Rationale No outcomes toward the project's objective were achieved. Overall Efficacy Rating Primary reason Negligible Low achievement 5. Efficiency The PAD did not conduct a formal economic analysis, explaining (pp. 12-13) that externalities in technology development efforts made it infeasible to measure project-specific rates of return or accurate estimates of benefits. Benefits expressed in qualitative terms were expected to accrue to the BoP population, project implementation agencies, enterprises, participating financial intermediaries, and government entities. Page 5 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) As none of the planned activities were implemented, an ex-post economic analysis was not feasible or meaningful. Given that US$ 1.38 million was spent on project management with no resulting outcomes or benefits, any such economic analysis would surely indicate a negative rate of return. Although the Bank team cancelled the project in time for funds to be reallocated to other IDA projects in Vietnam, and this timing was expeditious (as Vietnam graduated from IDA status on July 1, 2017), the cancellation was done only after four years. Efficiency is rated Negligible. Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives was High, as the objective aligned tightly with country conditions, Bank strategy, and government strategy. Achievement of the objective to adopt, upgrade and develop Inclusive Innovations for the benefit of the BoP population is rated Negligible, as no outputs or outcomes were reported. Efficiency is rated Negligible; with no outcomes achieved, any economic analysis would have found a negative rate of return. These ratings indicate severe shortcomings in the project's preparation and implementation, and therefore Outcome is rated Highly Unsatisfactory. a. Outcome Rating Highly Unsatisfactory 7. Risk to Development Outcome As no outcomes were achieved, there is no risk to be assessed. Page 6 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) 8. Assessment of Bank Performance a. Quality-at-Entry During the time of project preparation, the Bank was already providing support for Vietnam's science and technology sectors through analytic products (including a 2009 policy paper, "Vietnam: Innovation Policy for Competitiveness, Economic Growth and Social Equity") and several projects in the education and innovation fields. Strong buy-in had been expressed by all stakeholders for the science and technology reform agenda, with a special focus on inclusive innovation, and for this specific project. The PAD (p. 8) reflected several lessons learned from innovation projects implemented in other parts of the world, covering the needs to: (i) recognize the inherent complexity of science/technology projects, as they involve multiple sectors and require coordination among many stakeholders; (ii) have a strong champion and the commitment of government and implementing agencies; (iii) decentralize implementation to autonomous professional institutions that have appropriate mandate and capabilities; (iv) define clearly the roles and responsibilities of each participating institution; (v) develop a strong M&E framework with independent and periodic impact evaluation; (vi) support sub-projects that are both technologically and commercially sound; (vii) develop capacity of participating institutions at an early stage; and (vii) ensure collaboration between relevant stakeholders, including global partners. Project design intended to incorporate these lessons: (i) the Deputy Prime Minister in charge of Science and Technology, as well as other key agencies, were strong champions of the project; (ii) a Project Steering Council for policy coordination was formed; (iii) the project focused on just a few priority themes that could have a quick impact on the BoP population and in which Vietnam had a strong tradition; (iv) the project adopted a pilot approach to reduce risk and achieve a few early successes; and (v) collaboration with appropriate global research and development organizations was built into the early stages of the project, to expedite the development of high-performance and cost-effective solutions. Overall risk was rated Substantial due to the existence of multiple stakeholders and complex inter-governmental agency coordination, the novelty of the concept of inclusive innovation in Vietnam, and lack of the project implementing agencies' experience with Bank-funded projects. Mitigation measures included efforts to strengthen information sharing among stakeholders and to improve institutional capacity. A number of stakeholder workshops were held, as was a 2010 study tour to India for South-South knowledge sharing on innovation. However, there were shortcomings. No financial or economic analysis was conducted at appraisal. The results framework did not include outcome indicators adequate to measure all elements of the project's objective (Section 9a). Implementation arrangements, straddling two ministries, were not optimal, but this was the client's preference. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank team provided hand-on support to NAFOSTED, including engaging the services of an international consultant to work on revisions of forms and procedures. Implementation support missions were used to attempt to resolve bottlenecks and delays, and Implementation Status Reports were Page 7 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) candid. Following implementation challenges, the Bank scheduled two high-level meetings: first in December 2015 between the Country Director and the Minister of MoST, and the second in February 2016 between the Country Director and Deputy Prime Minister. During the May 2016 MTR, restructuring proposals were discussed to deal with the significant implementation issues that had arisen. The MTR recommended moving the entire project to the MoST, where there was technical expertise on innovation policy, but the MPI did not accept this proposal. Given the impasse on determining on a path forward, as well as the likely reduced interest among potential beneficiaries, the Bank team recommended cancelling the project by advancing its closing date. Although cancellation took four years, and arguably the Country Director could have engaged the client even earlier than was the case, the Bank team for the most part proactively identified opportunities and resolved threats in a challenging context. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design According to the PAD (p. 10), the EDA PMU was to develop a comprehensive M&E framework to measure project outputs and outcomes/impact. This framework was to include regular reporting on implementation progress and compliance with financial management and procurement guidelines and environmental safeguards, as well as periodic beneficiary surveys (including a baseline survey). NAFOSTED and the participating financial intermediaries were responsible for reporting on grant and sub-loan disbursement under the first and second components. EDA was to complete an impact evaluation within 24 months of project closure. The project's theory of change was sound, with comprehensive output indicators logically linked to intended outcomes. However, the key outcome indicators -- number of pilot inclusive technologies developed/upgraded that were likely to benefit the BoP population, and number of pilot inclusive technologies commercialized that were likely to benefit the BoP population -- were not complete measures of the objective. Although they were appropriate indicators to assess development and upgrading of inclusive innovations, they did not capture adoption of the new technologies or the technologies' benefit to the BoP population (commercialization does not necessarily imply adoption or benefit to the BoP population). This means that, even if project implementation had proceeded as planned, there would have been challenges in assessing achievement of the full scope of the objective under the existing results framework. b. M&E Implementation The PMU was established and maintained. However, there were no implementation or results data to collect Page 8 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) or analyze. c. M&E Utilization As no outputs or outcomes were reported, and implementation was stalled, there was no opportunity for project M&E to be used to inform project management, decision making, or implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was rated Environmental Assessment category B and triggered the Environmental Assessment (OP/BP 4.01) safeguard policy. Potential adverse environmental impacts were associated with the supported sub-projects under the first two project components, and the upgrading of laboratories under the third component. An Environmental and Social Management Framework was approved, with appropriate public consultation, and included in the Project Operation Manual. According to the ICR (p. 17), there were no safeguard compliance issues. b. Fiduciary Compliance The ICR (p. 14) notes extensive delays in the procurement of specialized medical equipment for pilot laboratory facilities, such that the procurement was never completed. It also (p. 13) notes that, had the MPI moved forward with its planned television advertising of some of the ideas proposed in the first round under the first component, fiduciary requirements would have been violated; since this advertising ultimately did not occur, the potential violations were not an issue. Other than these observations, the ICR (p. 17) states only that the project met all fiduciary compliance requirements. c. Unintended impacts (Positive or Negative) None reported. d. Other --- Page 9 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment There were severe shortcomings in the Outcome Unsatisfactory Highly Unsatisfactory achievement of outcomes. Efficiency is also rated Negligible. Moderately Moderately Bank Performance --- Satisfactory Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR (pp. 19-21) offered many important and insightful lessons, including: Deep and careful consideration of local context is crucial, especially for innovative projects. In this case, inadequate attention was paid to the need for smaller, more rigorous piloting in a new environment; local institutional and political economy conditions; the adequacy of proposed mitigation measures within the operating environment; and, importantly, the special considerations around innovation projects in countries with socialist foundations (where funding innovation directly to enterprises remains counterintuitive to ministries responsible for science, technology, and research). New concepts require clear definition and explicit agreement by all relevant entities on that definition, including codification in the Project Manual and/or Financing Agreement. In this case, the new model of "inclusive innovation" was not clearly understood by the PMU (MPI) and PIU (MoST), resulting in the rejection of some sub-project proposals by the PMU because they were not considered innovative or inclusive. Complicated, inter-ministerial projects require practical and clear implementation arrangements and commitment to adhere to those arrangements. In this case, disagreement between the PMU (MPI) and PIU (MoST) on procedures for sub-project selection completely derailed implementation progress, with the MPI in particular not honoring the independent technical review decisions made by the professional science and technology program management institutions (NAFOSTED). 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 10 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Vietnam Inclusive Innovation (P121643) The ICR is clear and concise. It candidly and effectively explains the project's implementation challenges, reporting carefully on the storyline explaning the lack of outputs and outcomes achieved. Its lessons are comprehensive and draw deeply from the project's experiences, demonstrating the importance of learning from failed as well as successful projects. a. Quality of ICR Rating Substantial Page 11 of 11