Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Report Number : ICRR0020424 1. Project Data Project ID Project Name P117590 CO - Science Technology & Innovation Country Practice Area(Lead) Colombia Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79440 31-Dec-2013 41,500,000.00 Bank Approval Date Closing Date (Actual) 15-Jul-2010 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 25,000,000.00 0.00 Revised Commitment 25,000,000.00 0.00 Actual 20,647,714.52 0.00 Prepared by Reviewed by ICR Review Coordinator Group Kimberly Kamlesh Judyth L. Twigg Joy Behrens IEGHC (Unit 2) Parekh 2. Project Objectives and Components a. Objectives According to the original Loan Agreement and the Project Appraisal Document (PAD), the development objectives of the Project were to: (i) "strengthen COLCIENCIAS’ [Administrative Department of Science, Technology and Innovation] ability to promote human capital for the knowledge economy, R & D and innovation"; and (ii) "raise awareness of science, technology, and innovation in Colombian society" (Loan Agreement, 2010; PAD, 2010). Although the ICR applies a split rating, this ICR review does not apply a split rating and instead assesses the project based on the revised indicators for the following reason: Changes made to the indicators at Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) restructuring amounted to improvements in the indicators (for example, removing an indicator that had been designated as a "PDO-level indicator" but did not actually assess project outcomes), and the indicators that better measured the PDO remained in force or were added. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components According to the Loan Agreement and PAD, the project components were: Component 1 (Planned US$5.8 million; Actual US$2.5 million): Strengthening COLCIENCIAS' operational and policy making capacity & institutional strengthening of the Science, Technology and Innovation National System. This component intended to strengthen the capacity of COLCIENCIAS and its science, technology, and innovation (STI) enabling environment by enhancing organization, human resource, and business processes; improving strategic and sector policy and planning; and improving managing, monitoring, and evaluating of the project. The component comprised provision of goods and services; training for staff; technical assistance; and operating costs for this component (PAD, p.12). Component 2 (Planned US$14.6 million; Actual US$6.8 million): Strengthening COLCIENCIAS' capacity to promote development of human capital for science and technology. This component intended to promote human capital by financing pilots to: provide stipends to Colombian and non-Colombian scientists outside of Colombia working for knowledge management organizations and the private sector to hire doctoral graduates; strengthen linkages between Colombian and non-Colombian scientists and the Colombian diaspora; support building scientific skills for basic and secondary education students; and provide technical assistance (PAD, p.12). Component 3 (Planned US$17.6 million; Actual US$15.7 million): Strengthening COLCIENCIAS' capacity to promote research and innovation; investment for research and innovation. This component intended to strengthen COLCIENCIAS to promote research and development (R&D) and innovation through financing, including matching grants, by developing innovation capacity in firms; strengthening R&D and innovation subprojects in strategic areas; strengthening revisions of subprojects under existing COLCIENCIAS instruments for matching grants to finance R&D and innovation; and providing technical assistance for development and implementation of plans and carrying out evaluations. Component 4 (Planned US$3.5 million; Actual US$3.0 million): Promoting social dissemination of science, Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) technology and innovation and institutional communication. This component intended to finance investments in order to increase awareness and knowledge of science, technology, and innovation in both the public and private sectors; and increase COLCIENCIAS' visibility in Colombia. A September 5, 2013 Level Two Restructuring provided changes to Components 2, 3, and 4. The following changes were made: Component 2 was amended to: (1) include capacity building for science, technology and innovation with a focus on supporting regional development and implementation of science, technology, and innovation projects; and (2) include pre-school education through a pilot in five departments. Component 3 was modified to: (1) change the financing and evaluation of the matching grants for research and innovation, to finance a review of these two instruments based on international standards and provide recommendations for their improvement; and (2) add a new pilot of an instrument intended to promote innovation both from private and academic providers. Component 4 was modified to: (1) eliminate the acquisition and operation of vehicles for mobile classrooms due to high recurrent costs; and (2) add a new subcomponent intended to recognize successful examples in innovation; develop new social innovation policy; and pilot incentives for social entrepreneurship among youth (Restructuring Paper, 2013). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total cost of the project was expected to be $41.5 million, which included $25.0 million from IBRD and $16.5 million from the Borrower (PAD, p. 57). The actual cost of the World Bank portion of the project was US$20.65 million, or 83% of the original estimate of US$25 million (ICR, p.25). (The ICR's Data Sheet give a slightly different disbursed amount, $21.32 million). The project was not fully able to spend funds because of implementation delays based mostly on significant political and budgeting changes in the country, specifically with STI. Financing: This US$25 million project was to be financed by IBRD, and an additional $16.5 million was to be financed by counterparts including universities, firms, and other private entities (PAD, 2010). The project was proposed as an Adaptable Program Loan (APL) with two phases over 9 years. The first phase was to be financed through a loan of US$25 million from 2010-2013, and the second phase was to be financed through a loan of US$225 million from 2014-2019. Phase I was to focus on strengthening the capacity of COLCIENCIAS to promote human capital for the knowledge economy, research and development, and innovation, and Phase II was intended to scale those efforts. Financing for Phase II was contingent on meeting specific milestones and two specific conditions: (1) the design and/or redesign of three new or existing financing instruments approved by the COLCIENCIAS Board; completion of evaluations and lessons learned of Phase I investment pilots; and disbursement of 75% of Phase I loans; and (2) the completion of at least two of four actions: (a) 50% of COLCIENCIAS staff trained in new competencies; (b) 80% of COLCIENCIAS management using performance management tools; (c) revised COLCIENCIAS project management processes such as calls for proposals, subproject monitoring, etc.; Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) and (d) completion of at least one research paper by internal policy staff (PAD, 2010). Under the project, both Condition 1 and Condition 2 were met, but changes in government priorities and introduction of regional STI funds led the government not to pursue the second APL (ICR, 2016). Borrower Contribution: The Borrower contribution was expected to be US$16.5 million (PAD, p. 57). The actual cost of the Borrower portion of the project was US$7.35 million, or 45% of the originally planned US$16.5 million (ICR, p. 25). Dates: The project was approved on July 15, 2010 and was originally expected to close on December 31, 2013. A September 5, 2013 Level Two Restructuring extended the closing date to August 31, 2015 because the project needed "to finance COLCIENCIAS' continued institutional strengthening, investment pilots, commitments to small grant beneficiaries, R&D subprojects, and planned evaluations of project activities that will contribute to the development of the second phase of the Adaptable Program Loan (APL)" (Restructuring Paper, 2013). A June 3, 2015 Level Two Restructuring extended the new closing date of August 31, 2015 to December 31, 2015 to complete key activities in Component 3, including a biodiversity commercialization pilot that was considered a key priority for Colombia and the project, and to complete evaluations of pilot programs in priority areas in Component 3 and social appropriations in Component 4, both of which were considered critical for knowledge management of the project (Restructuring Paper, 2015). 3. Relevance of Objectives & Design a. Relevance of Objectives The project's objectives were substantially relevant to country conditions and priorities. At appraisal, the Colombian National Council for Economic and Social Policy (CONPES) had issued two policy documents on the Colombian STI sector since 2008, and the World Bank issued a policy note in 2006 that included an analysis of STI. These documents identified the following issues in STI: (1) budget fluctuations, institutional fragmentation, and weak sector leadership; (2) limited degree of development of scientific skills at the lower levels of the education system; (3) inadequate stock of advanced human capital; (4) low economic relevance and limited international linkages of existing public research; (5) weak links between local researchers and the Colombian diaspora; (6) weak links between the private sector and knowledge institutions; (7) low capacity of firms to undertake commercially oriented innovations leading to the introduction of new products and services; (8) low levels of awareness of STI in society; and (9) limited diffusion of current public incentives (PAD, pp. 2-5). These policy documents laid the groundwork for the policy framework that influenced STI. The Government of Colombia's National Competitiveness and Productivity Policy and Productive Transformation Strategy, established STI as key pillars (PAD, p. 6). Finally, the Government of Colombia established a legal framework to support STI in 2009. COLCIENCIAS prepared a policy document that proposed a new national STI policy, and CONPES finally approved a new STI policy in 2009. The legal and institutional environment was very relevant for STI in general. The project was also substantially relevant to Bank strategy, both at appraisal and at closing. Its focus on STI was in alignment with the Bank's Country Partnership Strategy (CPS) for 2008-2011 (PAD, pp. 7-8), Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) which was in line with Colombia's former and current National Development Plan (2014-2018). The CPS supported sustained equitable growth aimed at improving Colombia's competitiveness and private sector productivity. The Bank's current Country Partnership Framework (CPF, 2016-2021) contains a sixth objective that supports "improved business environment and innovation to boost productivity" (Colombia CPF 2016, p. 27). Specifically, the CPF plans for finalizing a revision of public innovation programs and support for authorities in the implementation of a new innovation policy (Colombia CPF 2016, p. 27). While the rationale for supporting STI is very well established, the project's second objective, which focused primarily on raising awareness, could have been more precisely defined. Rating Revised Rating Substantial --- b. Relevance of Design The project's objectives were substantially linked with the project's original components and activities. The first objective "to strengthen COLCIENCIAS' capacity to promote human capital for the knowledge economy, R & D and innovation" was envisioned to be achieved through Component 1 (strengthening COLENCIENCIAS' institutional and operational development), Component 2 (strengthening COLENCINCIAS to promote the development of human capital) and Component 3 (strengthening COLCIENCIAS to promote research and innovation). Component 1 activities, i.e. enhancing organizational, human resource, and business process capacity; improving strategic and sector planning; and improving monitoring and evaluation, were envisioned to strengthen COLCIENIAS' institutional capacity. Component 2 activities, i.e. providing stipends to public and private organizations to hire doctoral graduates and strengthening linkages between Colombian and non-Colombian scientists with the Colombian diaspora through matching grants for projects, training, and technical assistance and providing grants for students and teachers for research were envisioned to demonstrate COLCIENCIAS' capacity to promote human capital. Component 3 activities, i.e. creating investment pilots, including building the capacity of firms to design, manage, and evaluate such grants and create projects in strategic areas were intended to further research and innovation. The second objective of raising awareness of STI was linked with Component 4 (promoting general science, technology, and innovation and improving COLCIENCIAS' visibility). Component 4 activities, i.e. increasing awareness of STI by furthering dissemination through mass media, mobile classrooms, conferences, etc. were intended to further STI awareness. The September 5, 2013, Level Two Restructuring created changes in activities and in the results framework. Changes in activities reflected changes in the legal, policy, and operational framework since the approval of the project. For example, in 2011 and 2012, the COLCIENCIAS Board approved new pilots creating collaborative projects with the Colombian diaspora; promoting innovation management in firms; and including doctoral graduates in the labor market. New laws also required funds allocated for STI in specific industries. Accordingly, the Restructuring changed Component 2 to include support for regional development, which also supported new funding in specific industries, and changed Component 3 to include piloting a new instrument to support innovation in the private sector and academia. These activities remained substantially relevant to the project's objectives. Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Rating Revised Rating Substantial --- 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To strengthen COLCIENCIAS' capacity to promote human capital for the knowledge economy, research and development (R&D) and innovation. Rationale Outputs Outputs related to institutional strengthening (training, strategic and organizational development; grant proposal and monitoring; and research) included: • 94 staff were trained according to a revised competencies model, exceeding the 2013 target of 80 staff; • 683 regional stakeholders completed short courses on STI project development and implementation, exceeding the target of 420 regional stakeholders. • 10 management unit staff used newly-developed Balanced Scorecard which monitors strategic performance, organizational development, communications, etc., meeting the target; • COLCIENCIAS' grants call-for-proposal processes revised, meeting the target; • 10 research or policy papers completed by COLCIENCIAS' Policy Analysis Unit, not meeting the original target of 15 papers but exceeding the revised 2013 target of 8 papers; and • Design and Evaluation of Public Policy Unit within COLCIENCIAS created. Outputs related to development of human capital for science and technology included: • 1,536,469 students were exposed to cognitive skills, non-cognitive skills, and content knowledge on research and development (formulation of questions, observation, data collection, analysis) by Ondas, an existing school-based STI program promoting the development of scientific skills in pre-primary, basic, and secondary education, exceeding the target of 1,200,000 students; • 28 doctoral graduates were employed in the labor market with placement lasting between 12 and 36 months for 24 companies, exceeding the original target of 15 graduates but slightly missing the revised 2013 target of 29 graduates; • 54 subprojects involving Colombian researchers residing abroad participated in collaborative research projects with Colombian scientists/engineers, exceeding the original target of 10 and the revised 2013 Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) target of 37 projects. Outputs related to investment for research and innovation included the following: • 128 proposals were financed under an Innovation Management call for proposals, exceeding the target of 100. This program aimed to improve innovation capacity among Colombian companies by hiring a consulting firm to provide support in building innovation capacities and processes within the firm. 94 companies (out of a total of 705 companies in the applicant pool) were selected to participate. The financing of such proposals could also be considered as reflecting COLCIENCIAS' capacity to finance research and innovation. • 5 R&D projects in specific priority areas financed, not meeting the original target of 10 but exceeding the revised 2013 target of 2. • One firm committed to finance $6.5 million in R&D and innovation, exceeding the $4 million target. • Three evaluations for R&D and innovation pilots completed, meeting the target of 3. The "Innovation Management" pilot built the capacity of firms to implement product, process, or service innovation. An evaluation of the Innovation Management pilot examined differences pre and post participation of the program. The evaluation found participating firms to increase their STI budgets by 113%; increase employees with PhDs by 44%; increase STI activities by 68%; and finance STI projects by 87% (ICR, p. 30). The "Diaspora" pilot involved the financing of Colombian diaspora to collaborate with domestic research groups and the financing of the development of specific innovation subprojects. An evaluation of the former pilot resulted in two published articles, 13 graduate thesis projects, three experiments, and 11 workshops while an evaluation of the latter pilot resulted in 14 articles (6 published articles and 8 articles intended for publication), 10 graduate thesis projects, and three technological models (ICR, p.28). The "Priority Area" pilot involved financing R&D and innovation subprojects in strategic knowledge areas. This was a process evaluation and but the findings were not highlighted in the ICR. Outcomes The project achieved or exceeded targets on all outcome indicators as defined in the project's results framework. In terms of institutional strengthening of the STI system, the percentage of COLCIENCIAS' annual investment budget committed by the end of the second quarter of the calendar year was 70% by project closure, exceeding the 60% target. In addition, the number of STI financing/support instruments designed or redesigned and approved by COLCIENCIAS' Board of Directors was four, exceeding the target of three instruments. These indicators provided less insight, however, on strengthened capacity on the part of COLCIENCIAS (an outcome as defined by the PDO). For example, there were no explicit indicators measuring COLCIENCIAS' technical; operational; systemic; adaptive; and influencing capacities. Related to development of human capital for science and technology and investing in research and innovation, 60% of companies involved in the labor market initiative stated that inclusion of doctoral Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) graduates through the Labor Market Insertion program resulted in new products, access to knowledge networks, and alliances with universities and research, but only 30% stated that these students led to increased productivity. (The ICR does not provide comparison figures provided for other similar programs.) Less than half the companies (46%) hired the doctoral graduates. (The ICR does not provide contextual information on the labor market for doctoral graduates, or comparison information on how many companies hired doctoral graduates without assistance from the program.) The ICR does indicate that doctoral graduates are credited to have created nine innovations, many of which have social impact. For example, one innovation developed a tool to measure the neurological impact of rehabilitation therapies for epileptic patients (ICR, p. 28). Regarding COLCIENCIAS' capacity to promote human capital for the knowledge economy, a preliminary study showed statistical difference with positive effects in test scores among students in Ondas-participating and non-participating schools and a lower per-student cost for Ondas-participating schools. Also, the project financed 128 product, process, and organizational innovations, some of which are currently under patent application. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To raise awareness of science, technology and innovation in the Colombian society. Rationale Outputs COLCIENCIAS' Social Appropriation Program financed research and innovation sub-projects amongst civil society groups in partnership with universities and research organizations. These sub-projects involved 2,512 participants (community leaders, local cooperative members, etc.). In addition, in 2013 and 2015, COLCIENCIAS' Open Innovation competitions engaged civil society in two strategic areas -- conservation of hydro resources and agro production for food security -- and awarded innovation proposals. The competition drew interest from over 25,000 participants. Outcomes In terms of social dissemination of science, technology, and innovation and institutional communication, the project's engagement resulted in a 50% increase in the number of citations of COLCIENCIAS and/or its activities in the media (ICR, p.15-16). The ICR states that COLCIENCIAS had 5 million visits to its website, 60,000 followers on its Facebook page and 100,000 followers of its Twitter account (ICR, p. 16). Comparison information is not provided. Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Achievement of key outcome targets for awareness is unclear. According to a 2015 nationally representative perception awareness survey, 53% of the Colombian population agreed with the statement that Colombia creates science, and 52% of the Colombian population agreed with the statement that Colombia creates technology, not meeting the targets of 76% and 73% respectively. These survey results compare to a 2012 nationally representative perception awareness survey in which 75% of the Colombian population agreed with the statement that Colombia creates science and 72% of the Colombian population agreed with the statement that Colombia creates technology. While both surveys were nationally representative, budget constraints resulted in a much smaller sample size in 2015. The ICR does not give the margin of error in the 2015 survey, but it was large enough that the 2015 findings and 2012 findings could not be compared meaningfully. Rating Modest PHREVDELTBL PHREVISEDTBL 5. Efficiency The PAD did not provide an ERR/IRR but provided economic data on various dimensions of the project (Annex 9), including on the value of STI for economic growth; linkages between innovation and productivity; the rationale for public interventions in STI; and a calculation of public and private investment necessary to achieve the a ratio of STI investment to GDP of 2% by 2019 (ICR, Annex 3). According to the ICR, based on reasonable assumptions and a discount rate of 12%, the Net Present Value is US$9.79 million and the Internal Rate of Return is 29% in an optimistic scenario, while the NPV is US$0.3 million and the IRR is 13% in a pessimistic scenario (ICR, pp. 38-39). The ICR's economic and financial analysis included two elements that could suggest efficiency improvements related to human capital development (Ondas) and research (Diaspora): • A preliminary study showed lower per-student cost for Ondas-participating schools vs non-Ondas participating schools. The ICR notes the average cost of Ondas per geographical region is US$14,500, with unit cost per student US$1.10 (ICR, p. 34); a background note gives costs per student in Colombia in 2011 of US$3,491 (pre-school), US$2,041 (primary), and US$2,207 (secondary). The ICR states that "The low cost of Ondas per student as compared with the current costs of educating a student suggests a potential cost-effective opportunity to enhance [student] learning outcomes, increase [student] transition to higher educational levels, and increase [student] interest in science and technology" (ICR, p. 35). • The first round of financing of 17 Diaspora projects, totaling US$1.3 million, resulted in 49 published scientific articles, or US$26,000 per output, while the second round of financing, totaling US$100,000, resulted in five published articles and a book chapter, or US$17,000 per output (ICR, p.35-36). Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Financial management was rated moderately satisfactory throughout most of the project's lifetime, with a few occasions of moderately unsatisfactory, due to: (1) key financial management position vacancies for long periods of time; (2) cumbersome internal controls for financial follow-up of subprojects; (3) insufficient or inadequate staff follow-up in a timely manner; (4) subproject contracts that could not be met by the project closing date; and (5) delay in contracting a new fiduciary agency. (ICR, pg. 9). Early delays in project implementation were due primarily to political and budget changes in the country and its STI sector, and the project's two-year extension allowed for significant scale-up of activities. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 100.00 ICR Estimate  29.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of project objectives is rated substantial due to the objectives' alignment with Colombia's and COLSCIENCIAS' legal, institutional, and operational policy frameworks for STI. The relevance of project design is rated substantial due to strong linkages between objectives, components, and activities. Achievement of the first objective is rated substantial because the project met targets for investment, capacity building, and development of human capital. However, the project could have had a stronger results framework and created better output and outcome indicators. Achievement of the second objective is rated modest because the project did not achieve targets for population awareness of science and technology. Efficiency is rated modest. Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, and therefore an outcome rating of moderately satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) The sustainability of achieved outcomes is uncertain, primarily due to the volatility of investment for STI. For example, the Government announced an increase in public investment for STI from 0.7% of GDP in 2015 to 1.5% by 2025. In 2016, however, the Government announced a decrease in public spending, including a decrease in public investment for STI by 4% points. In addition, since 2013, the COLCIENCIAS budget has steadily decreased (ICR, p. 20). Despite the financial and budgetary decreases, the draft national STI policy (2015-2025) continues to include strengthening capacity to promote human capital for R&D and innovation as one of its main policy objectives. The draft STI policy also includes other project components, such as institutional strengthening. Specifically, COLCIENCIAS' efforts in this area resulted in greater effective implementation of STI policies; greater geographical coverage of programs; improved ability to reach marginalized populations; and better financial sustainability of activities, all of which is likely to be sustained (ICR, p. 19). a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry Project preparation was informed by lessons from other IBRD education projects during the project preparation phase, though none of these projects focused exclusively on science, technology, and innovation. The project drew upon lessons learned from COLCIENCIAS' implementation of pilot investments. These pilots allowed COLCIENCIAS to better understand how to finance and provide technical assistance to such investments in the future. The project also collaborated with the Inter-American Development Bank (IDB) during the project preparation phase. IDB intended to finance two independent operations in Colombia's STI sector. Collaboration ensured coordination of activities. IDB's projects were intended to support institutional strengthening, human capital, and research and innovation that would complement Components 1, 3, and 4 of the project (PAD, 2010). The results framework had some shortcomings: indicators connected well to project components, even those indicators identified as outcome indicators were not as well-connected to the PDO itself. Also, the institutional strengthening component lacked a baseline capacity assessment. Indicators for the project's second objective were assessed to be weak, and the June 2013 restructuring resulted in improvements to PDO-level indicators and intermediate outcome (output) indicators. Targets for the indicators identified as outcome indicators appear appropriate given the context. COLCIENCIAS is the relevant government agency for STI. However, the project may have been "optimistic" in setting the original closing date and had to extend it twice (ICR, p.5). Also, the Bank did not identify the risk associated with potential changes in leadership at COLCIENCIAS. Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision As science and technology engagement with Colombia was relatively new, the Bank team had a high level of engagement throughout the project implementation period. There were at least two mission visits annually, and appropriately specialized technical assistance was continuously provided. The engagement incorporated lessons learned from the 2012 mid-term review that led to the key 2013 Restructuring. Also, COLCIENCIAS is said to have valued Bank Staff technical assistance. Coordination with IDB is considered to have been strong (ICR, p. 20). However, there were a few minor shortcomings. Specifically, coordination could have been improved between the project team and the Bank's Private Sector & Competitiveness team also working on STI issues in Colombia; however, these coordination issues did not impact project outcomes (ICR, p. 21). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The Government was actively engaged during the project and was open to modifying the project to adapt to new rules and regulations in science and technology, i.e. the introduction of a regional financing modality for STI investment. However, COLCIENCIAS' continuous change in leadership and fluctuations in allocation of resources remained challenging throughout the project period. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance COLCIENCIAS is credited with significant achievements, particularly under the project's first objective, despite changes in leadership, staff turnover, and changes in allocation of resources. As an organization, COLCIENCIAS is credited as learning from experience and modifying the project as time passed. It had a strong commitment towards developing a monitoring and evaluation system. The agency was also successful in coordinating with other STI actors and coordinating with other donors, primarily IDB. There were some shortcomings in financial management (see Section 5). Implementing Agency Performance Rating Moderately Satisfactory Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The original M&E design adequately measured progress in implementing the components, but a results chain that traced how the activities under the project components would generate the intended outcomes at the PDO level was not completely explicit, and the results framework had few indicators that would speak directly to achievement of the PDO. M&E design was characterized by imperfect alignment between the project's first objective on strengthening COLCIENCIAS' capacity and the associated output and outcome indicators. The project's second objective had significant weaknesses with indicators, and the indicator on raising awareness turned out to be difficult and relatively costly to measure. The June 2013 restructuring resulted in improvements in the associated indicators. b. M&E Implementation M&E data was collected regularly, although data collection was regarded as time-consuming and not always automatic, sometimes producing delays. The delay in the COLCIENCIAS online Integrated Information System, an activity financed by the IDB, resulted in a lack of automated systems. The evaluation of some aspects of the project, e.g. Diaspora and Social Appropriation Programs, focused more on process than on outcome. Finally, the science and technology climate changed over the course of the project, leading to changes in the M&E framework. The M&E framework was therefore modified in the first restructuring. The new M&E framework eliminated one PDO indicator. In terms of intermediate outcome (output) indicators, five were changed, three were added, and one was eliminated. c. M&E Utilization In general, the M&E framework was used to measure progress in the project. Lack of progress towards achieving some output targets was factored into the modifications at restructuring. Many of the indicators in the results framework were incorporated to monitor COLCIENCIAS management's performance through the project-created Balanced Score Card System. M&E information was transparent and was shared in internal workshops and among staff. According to the project Task Team Leader, the M&E framework is considered one of the project's more important achievements, and the data is widely used by COLCIENCIAS officials. The ICR also notes that "[COLCIENCIAS] has demonstrated a renewed effort in improving its M&E capacity, as well as systematizing and documenting the experience to date in order to create an institutional memory" (ICR, p. 22). M&E Quality Rating Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Modest 11. Other Issues a. Safeguards The ICR (p. 9) indicates compliance with safeguard policies. The project was environmental category B and triggered OP/BP 4.01 Environmental Assessment (EA) and OP/BP 4.10 Indigenous Peoples. For the EA, the project required all sub-projects to ensure low environmental impact. The Bank reviewed the environmental assessment through a sub-sample and found the project compliant with all requirements. An Indigenous Peoples Planning Framework was developed that required: (1) elimination of discriminatory factors for research and innovation grants; (2) special outreach to indigenous groups; and (3) technical assistance to indigenous groups. The ICR (p. 9) notes that the IPPF was satisfactorily carried out by COLCIENCIAS, though no further information is provided. The M&E framework did not disaggregate data based on indigenous groups, and this may have been a missed opportunity. b. Fiduciary Compliance Procurement: Procurement was performed in accordance with World Bank guidelines (ICR, p. 10). Financial Management: Financial management experienced challenges throughout the project mostly because of: (1) key financial management position vacancies for a long time; (2) cumbersome internal controls for financial follow-up of subprojects; (3) insufficient or inadequate staff follow-up in a timely manner; (4) subproject contracts that could not be completed by the project closing date; and (5) delays in contracting a fiduciary agency (ICR, pg. 9). Nevertheless, the auditors issued a timely and unqualified (clean) opinion of the project's financial statements (ICR, p. 10). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Moderately --- Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) Satisfactory Satisfactory Risk to Development Modest Modest --- Outcome Moderately Moderately Bank Performance --- Satisfactory Satisfactory Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR (p. 23) notes several lessons learned: (1) Implementing agencies may need guidance at the beginning of a project to formulate clear roles and clear allocation of resources. In this project, there were early instances where COLCIENCIAS's role was not clear in relation to other entities. (2) Implementing agencies hoping to create new programming can achieve maximum impact through specific articulation of financing and technical assistance. In this project, COLCIENCIAS supported pilot investments through every step of the process, including calls for proposals, design and implementation of research/business plans, allocation of innovation subsidies to support technical assistance of projects, etc. (4) Bringing in a broad range of stakeholders can achieve awareness and outreach beyond the traditional range of partners. This lesson stems from this project's experience with engagement with non-traditional partners such as local actors, vulnerable communities, indigenous cultures, etc., all of whom provided bottom- up understanding of the importance, role, and impact of STI. (5) Guidelines for institutional strengthening, including concrete contextual examples and continuous technical assistance, are essential in order to help an implementing agency with institutional development. (6) Unintended impacts of policies can be positive. In this case, for example, the Government's introduction of regional financing for student support led to increased coordination among regional actors in the STI sector and institutions. 14. Assessment Recommended? No 15. Comments on Quality of ICR Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO - Science Technology & Innovation (P117590) The ICR is clear, candid, and concise. It describes well the introduction of systems and institutional support to a new implementing partner. It also lays out logically and succinctly the linkages between the project's objectives, implemented activities, intermediate outcomes (outputs), and outcomes. It specifies clearly the revisions to components, indicators, and output targets that took place during project restructurings. It includes a discussion of the project's inclusion of and impact on vulnerable groups. a. Quality of ICR Rating Substantial Page 16 of 16