Document of The World Bank Report No: 31731 IMPLEMENTATION COMPLETION REPORT (SCL-45720) ON A LOAN IN THE AMOUNT OF US$5.0 MILLION TO THE BOLIVARIAN REPUBLIC OF VENEZUELA FOR THE MILLENNIUM SCIENCE INITIATIVE PROJECT June 20, 2005 Human Development Sector Management Unit Country Management Unit for Bolivia, Ecuador, Peru and Venezuela Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 14, 2005) Currency Unit = Bolivares (Bs.) Bs. $1.00 = US$ 0.00052 US$ 1.00 = Bs. 1915.20 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS CAICET Núcleo Centro Amazónico para la Investigación y Control de Enfermedades Tropicales Nucleus Amazon Center for the Research and Control of Tropical Diseases CAS Country Assistance Strategy CE Centers of Excellence CONICIT Consejo Nacional de Investigaciones Científicas y Tecnológicas National Council for Scientific and Technological Research FONACIT Fondo Nacional de Ciencia, Tecnología e Innovación National Fund for Science, Technology and Innovation GoV Government of Venezuela IADB Inter-American Development Bank IBRD International Bank for Reconstruction and Development IMF Centro de Excelencia en Dinámica y Control de Aludes Torrenciales Center of Excellence in Dynamics and Control of Landslides and Flooding IMU Implementation and Management Unit IVIC Núcleo del Laboratorio de Biología de Virus Nucleus Viral Biology Lab LIL Learning and Innovation Loan M&E Monitoring and Evaluation MS&T Ministry of Science and Technology MSI Millennium Science Initiative NER Nuclei for Excellent Research NIS National Innovation System PAD Project Appraisal Document R&D Research and Development S&T Science and Technology Vice President: Pamela Cox Country Director: Marcelo Giugale Director/Sector Manager: Evangeline Javier/Eduardo Vélez Bustillo Task Team Leader/Task Manager: Lauritz Holm-Nielsen VENEZUELA MILLENNIUM SCIENCE INITIATIVE PROJECT CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 4 5. Major Factors Affecting Implementation and Outcome 11 6. Sustainability 12 7. Bank and Borrower Performance 13 8. Lessons Learned 14 9. Partner Comments 14 10. Additional Information 17 Annex 1. Key Performance Indicators/Log Frame Matrix 18 Annex 2. Project Costs and Financing 19 Annex 3. Economic Costs and Benefits 21 Annex 4. Bank Inputs 23 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 25 Annex 6. Ratings of Bank and Borrower Performance 26 Annex 7. List of Supporting Documents 27 Project ID: P066749 Project Name: MILLENNIUM SCIENCE INITIATIVE PROJECT Team Leader: Lauritz B. Holm-Nielsen TL Unit: LCSHE ICR Type: Core ICR Report Date: June 22, 2005 1. Project Data Name: MILLENNIUM SCIENCE INITIATIVE L/C/TF Number: SCL-45720 PROJECT Country/Department: VENEZUELA Region: Latin America and the Caribbean Region Sector/subsector: Tertiary education (89%); Central government administration (11%) Theme: Education for the knowledge economy (P); Technology diffusion (P) KEY DATES Original Revised/Actual PCD: 11/09/1999 Effective: 04/10/2001 04/10/2001 Appraisal: 02/14/2000 MTR: 05/15/2004 05/15/2004 Approval: 06/26/2000 Closing: 06/30/2003 12/31/2004 Borrower/Implementing Agency: BOLIVARIAN REPUBLIC OF VENEZUELA/MINISTRY OF SCIENCE AND TECHNOLOGY Other Partners: STAFF Current At Appraisal Vice President: Pamela Cox David de Ferranti Country Director: Marcelo Giugale Andres Solimano Sector Director/Manager: Evangeline Javier \Eduardo Xavier Coll \Jamil Salmi Velez Bustillo Team Leader at ICR: Lauritz B. Holm-Nielsen Lauritz B. Holm-Nielsen ICR Primary Author: C. Cesar Yammal and Kristian Thorn 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: SU Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: S Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The objective of this Learning and Innovation Loan (LIL) was to demonstrate the effectiveness of transparent, merit-based allocation procedures and investigator autonomy in improving the quality and efficiency of scientific research and training, and thereby demonstrate how to revitalize Venezuela's Science and Technology (S&T) system. The project aimed to energize Venezuela's S&T system by supporting advanced training of human capital by world-class scientists and to encourage investigator autonomy by improving the quality and efficiency of scientific research and training and by rewarding research system productivity and efficiency. An important objective of the project was to show that, under proper procedures, Venezuela can perform S&T research at high international quality and productivity standards, and that high quality, international level scientific research could be expanded and sustained within Venezuela's S&T research budget. Cutting-edge scientific research and knowledge production are key for a country to take part in global knowledge production and the project was designed to provide vibrant research environments and thereby stem "brain-drain" and assist Venezuela in developing into a knowledge-based economy. Improved performance was specifically expected in the areas of (i) quality of the selection process for research projects, (ii) human resources training opportunities, (iii) international collaboration among researchers, (iv) administrative efficiency, (v) allocation of resources, and (vi) perception of S&T policy in Venezuela. General Assessment: The development objective and general design were appropriate, in accordance with best international practice and with proper consideration of circumstances at appraisal time in Venezuela. In particular at that time, there was too much rigidity in the S&T system on use of funds granted to researchers, and a fragmentation of funding "windows", each with their own limitations. This situation meant that a researcher wanting to carry out a single research project had to prepare different proposals and present them to different small funds for specific types of expenditures. Thus, small equipment/ consumables, scholarships for graduate students, participation in conferences, and large equipment, each and all needed to be requested and won separately competing with other proposals, at different times in the year and following different rules. Other types of expenditures, also essential for carrying out excellent research, such as small refurbishing of research facilities (for maintenance or to accommodate new equipment), or vehicles and boats for field research, were not eligible under current funding mechanisms. The project provided researchers with autonomy to carry out their projects, so they could be more ambitious in their research, and group different types of expenditures needed (even those that were not allowed before) in one "package". It is also important to point out the relevance of promoting advanced training of human capital, in a country that has not yet reached a critical mass of researchers and technologists, and where Research and Development (R&D) activities in some instances are carried out by professionals that have not completed or are not completing their scientific or technological training, and only have their first university degree or a master's degree. 3.2 Revised Objective: The objective was not revised. 3.3 Original Components: The project consisted of two components: - 2 - Project Component A - Capacity building: Ministry of S&T, CONICIT (later transformed into FONACIT) and the MSI Directorate (amended budget US$1.94 million): This component would provide (i) capacity building for the new Ministry of Science and Technology, with technical support to establish national S&T policies, introduce incentives for formation of high quality researchers and modernize the then National Council for Scientific and Technological Research (Consejo Nacional de Investigaciones Científicas y Tecnológicas, CONICIT); (ii) the establishment and operation of a Board of Directorsto oversee the Millennium Science Initiative (MSI) Project; a Program Committee, tasked with selecting research projects; and an Implementation and Management Unit (IMU) performing all necessary support activities; (iii) technical assistance for the selection of Centers of Excellence (CE) and Nuclei for Excellent Research (NER); and (iv) Monitoring and Evaluation Studies. This component would also finance studies, publications, remuneration of Program Committee and IMU's personnel, and administrative costs. Project Component B ­ Competitive Fund for Scientific Excellence: This component was designed to promote research activities nationwide. There were two subcomponents: B1. Centers and Nuclei of Scientific Excellence (amended budget US$5.52 million). This component was designed to provide funding for research projects at three Centers of Excellence (CE) and 8-12 Nuclei for Excellent Research (NER). The CE and NER would carry out the following activities: (i) scientific research; (ii) expansion of master, doctoral, and post-doctoral training programs; and (iii) networking, outreach and special activities to promote scientific excellence. This component would finance high-level scientific equipment, infrastructure rehabilitation (including laboratories), fellowships for doctoral and post-doctoral students and publications. B2. Network for the Promotion of Scientific Excellence (amended budget US$0.49 million). Networking activities would include: (i) research visits to establish formal and informal connections to high level international institutions; (ii) coordination of appropriate activities with Directors of CE, NER and principal investigators; (iii) programs for exchange of researchers, post-graduate, and graduate students; (iv) design and delivery of international advanced courses; and (v) dissemination of lessons learned. This component would finance remuneration for researchers, fellowships for master and doctoral students, travel expenditures and publications. 3.4 Revised Components: The Loan Agreement was amended once and the Closing Date was extended. Although these modifications did not change the project objective, the amendment reduced the overall funding and a smaller number of Centers and Nuclei were supported. Even with the reduction in project size, the expected demonstration effect was achieved (see section 4.2, Outputs by components). In addition, a complementary activity called "Innovative Ideas" (Ideas Innovadoras) was included under sub-component B1. This activity allowed research groups that had been selected during the second round of funding--a round that was ultimately cancelled due to the budget reduction--to explore new avenues of research. Six innovative ideas were funded. Extension of the closing date On June 24, 2003 the Loan Closing Date was extended for an additional period of 18 months until December 31, 2004. This extension allowed for the implementation and finalization of on-going project activities whose start had been postponed due to disbursement delays caused by (i) the fiscal and political crisis the country went through during 2001, 2002, and part of 2003; and (ii) a necessary amendment to the new Constitution (approved after loan approval) regarding procurement regulations, which eliminated contradictions between the Government legislation and the Bank's procurement guidelines. - 3 - Amendment to the Pari-passu On September 5, 2003, the loan cost sharing was changed from 70/30 (Borrower/Bank) to 30/70 to address a shortfall in counterpart funding stemming from the fiscal and political crisis. The modification in cost sharing allowed for the acceleration and completion of remaining agreed activities and ensured that the project was completed within a total budget of US$8.0 million of which US$5.0 million came from Loan proceeds (See section 5.1 for more details on the national crises that affected project implementation). 3.5 Quality at Entry: The project's quality at entry is rated satisfactory and the choice of lending instrument was appropriate. The LIL allowed, with relatively modest resources, piloting of a new approach to science funding in the Venezuelan context, and produce valuable learning that has been incorporated into the Venezuelan administration of S&T. The project addressed the World Bank's Country Assistance Strategy (CAS) dated May 19, 1997 (Report No. 16471-VE) objective of enhancing sustainable growth through the upgrading of Venezuela's human capital. It supported the Government's efforts to develop a national long-term vision for S&T and to establish a coherent set of policies. The project complemented an Inter-American Development Bank-financed project, which supported actions of the then CONICIT to strengthen the National Innovation System (NIS) through capacity building and training of new researchers in priority fields, and interaction and cooperation among the various stakeholders in the NIS. In addition, the project was part of a region-wide initiative and drew on experiences and lessons learned from the first MSI Project in Chile, which was completed in 2002. Throughout project preparation significant stakeholder consultation was undertaken and the counterpart and Bank teams identified well the major sector issues facing Venezuelan scientists. These included: weak and uncoordinated planning of scientific research, scarce and fragmented resources (both human and capital), lack of collaborative research at the national and international levels, weak procedures and practices for the implementation of research, and non-existence of private S&T groups. It was recognized from the outset that a time frame of ten years or more was necessary to make a substantive change in the S&T sector. Therefore, and as stated in the Project Appraisal Document (PAD), the project was seen as a first phase of a long-term strategy. The LIL provided relevant learning and a foundation over which to build the second phase of such a strategy. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: Achievement of objectives was satisfactory. The project was successful in demonstrating how to revitalize Venezuela's S&T system and made a significant contribution to strengthening the scope, quality and relevance of research in supported Centers of Excellence and Nuclei. The project's objective was met in the context of serious macroeconomic and political instability with negative implications for the availability of resources for research and the ability of beneficiaries and the IMU to implement selected sub-projects. Below the main outcomes of the project are listed: Demonstrated how to reform funding of research in Venezuela. The project introduced comprehensive research grants based on competitive procedures with clear eligibility and selection criteria. Rather than - 4 - awarding small grants in different lines to individual researchers ­ as has been the traditional approach of CONICIT and later FONACIT ­ the project awarded large multi-purpose grants for research teams. One grant that funded research, equipment, training of advanced human capital, and participation in conferences and networks facilitated the adoption of a coherent and long-term approach among supported Centers of Excellence and Nuclei. Supporting teams through multi-purpose grants also proved effective in promoting multi-disciplinary research. Moreover, the grants allowed Centers and Nuclei to improve the scope, quality and relevance of research in a sustainable way by balancing investments and recurrent expenditure. In the words of the researchers, the MSI allowed supported researchers "to pursue new, more ambitious and relevant research problems that represent big challenges" and "to make a qualitative jump in the use of methodologies to identify and tackle research problems."Based on the positive lessons learned, FONACIT has begun a process to incorporate MSI-like elements into its lines of funding. Notably, there is an increased emphasis on using research funding to stimulate the formation of scientific networks in priority areas, encourage interdisciplinary research and strengthen the transfer of scientific discoveries to potential users. Provided guidance on how to build advanced human capital. With the resources and stability that came with research grants, Centers and Nuclei were successful in showing how to upgrade the educational level of researchers and taking on additional graduate students. As a result of targeted investments, the average age of researchers in Centers and Nuclei fell from 46 to 41 years between 1999 and 2004. Moreover, the project raised the proportion of researchers with a Ph.D. from 44 to 50 percent and lowered the number of researchers with only a bachelor's degree from 24 to 2 percent in the same period. Finally, the number of researchers involved in supporting graduate training and thesis work increased from 26 percent in 1999 to 34 percent in 2004 (N. Núñez, 2005). Table 1 shows the shift in qualifications from first degree to enrollment in master's and doctoral programs. Table 1. Educational Level of Researchers in the Millennium Science Initiative (MSI) Educational Level % of MSI % of MSI Researchers in 2001 Researchers in 2004 First university degree completed 24 2 Master student 7 15 Master degree completed 17 13 Ph.D. student 9 17 Ph.D. degree completed 44 50 Source: N. Núñez, 2005 Showed how to leverage resources and build a critical mass of researchers through networks. The project demonstrated how to strengthen Venezuela's science base by enhancing inter-team/center/nuclei research collaboration. Notably, the project provided evidence that formal and informal networks of researchers and their students have the potential of improving the quality of research, reducing the isolation of researchers outside the capital area, and providing for mentoring opportunities for master's and Ph.D. students. Thus, for instance, a master candidate in health sciences working at beneficiary CAICET, in Puerto Ayacucho (Amazon State), was mentored by a scientist at beneficiary IVIC in Caracas, benefiting also through access to sophisticated molecular biology techniques and equipment not available in CAICET. An impact assessment of supported Centers and Nuclei (Núñez, N., 2005) shows that six research networks were created during project implementation, and that existing networks were strengthened: the number of researchers per network increased from an average of 4 researchers in 1999 to 38 researchers in 2004. Table 2 enumerates supported national and international networks. - 5 - Table 2. Networks strengthened by the Millennium Science Initiative Name of the Network Health related Networks Leishmaniasis Network Dengue Network National Network of Virology Iberian American Network for Emerging Viral Infections (international) Network for the identification and Molecular Diagnostics of Hemoparasites with economic impact in the agricultural sector Malaria Research Network (international) Regional Oncocercosis Network Hepatitis Network Yanomami Health Plan and Network International "Ecohealth Approach" community (international) Environment related Networks Network for the Sustainable Managements of Plains Iberian American Network of Hydraulic Research Institutes (international) Guárico River Basin Network (international) OVUM Econetwork - National Ornithology Network Iberian American Network for Organic Agriculture (international) Soil Biology Network Source: N. Núñez, 2005 Illustrated how to improve the relevance of S&T for society. The project was instrumental in showing that high-quality research can have a strong impact on society. Advances were made in stimulating participatory research and focusing research on real-life problems. Specific achievements illustrative of the project's societal impact include: (i) establishment of a warning system for landslides, which pose a serious risk to, particularly, poor areas on hills around the capital; (ii) development of models which guide efforts to prevent flooding in urban areas; (iii) advances in the prevention of dengue and other vector diseases; and (iv) improved understanding on how to combat endemic tropical diseases among indigenous populations. Enhanced the capacity of the Ministry of S&T for sector oversight and management. At the beginning of the project a Ministry of S&T had just been established. The project contributed to consolidating the Ministry and its operations by providing funding for policy studies and key equipment. Moreover, the project contributed to clarifying roles in the S&T system by refocusing FONACIT activities on the funding of research and supporting the Ministry in formulating sector policy. Finally, the project supported a move by the Ministry to raise awareness of science and technology in order to induce students to consider a career as a scientist and build public support for the funding of research. Established how the Government of Venezuela (GoV) can improve accountability for the use of public subsidies. The project successfully introduced a number of innovations in the oversight of grants for S&T. Notably, supported researchers were required to provide frequent reports on research findings, achievements in the training of master's and doctoral graduates and networking activities. By collecting and providing feedback on these reports, the project's Program Committee gave researchers a sense that they were accountable for the use of project's funds. Moreover, an independent, external review and an impact assessment introduced two new ways in FONACIT of receiving credible feedback and adjusting financing instruments based on collected lessons learned. - 6 - 4.2 Outputs by components: In May 2004 three international S&T experts conducted an external evaluation of the project. Their final report ­ evaluating results in supported Centers and Nuclei and synthesizing lessons learned ­ provides invaluable insights for assessing the outputs of the project (Barreto de Castro, L., C. Wernli and N. Núñez, 2004). Moreover, an impact assessment comparing Centers and Nuclei before and after the project was conducted in March 2005 (Núñez, N., 2005). Although this assessment does not apply a design that allows the establishment of causal relationships, it provides important information on changes in the productivity and working practices of supported researchers. Project Component A - Capacity building: Ministry of S&T, CONICIT/ FONACIT and the MSI Directorate: The implementation of this component was satisfactory. The project promoted capacity building and S&T awareness both at the Government level and at the level of the general public. The project financed Venezuela's first international conference on S&T policies held in October 2004. The conference covered themes such as approaches to the formulation of national S&T strategies, technology forecasting and linkages between regional and national innovation plans. Moreover, a host of national and international case studies were presented. There were about 200 participants including representatives from Argentina, Brazil, Colombia, Costa Rica, Cuba, Mexico, Peru, Spain and Uruguay. At the general public level, the project financed a survey of public perception of S&T and the adaptation of a bus into a mobile science lab. This bus visits schools around Venezuela, and students have a chance to learn about science and engage in experiments. This was an important contribution to raising science awareness and encouraging students to pursue a career in research. Activities that strengthened the day-to-day activities at FONACIT included the provision of essential office equipment and the development of a project database which allows FONACIT and the Ministry of S&T to overview and manage funded research projects. The project also funded studies on specific sectors of importance to Venezuela's economic development that otherwise would have been very difficult to undertake. For instance, the project financed a study on the S&T capacity of the petroleum sector, which identified ways in which research can be strengthened in this area. Finally, the project established working organizational bodies that allowed not only for effective project implementation, but also for showcasing a governance model for S&T initiatives: a Board of Directors, a Program Committee, and an Implementation and Management Unit with the attributions described in the PAD. Project Component B ­ Competitive Fund for Scientific Excellence: The implementation of this component was satisfactory. After a slow start due to external factors activities under component B were implemented according to schedule. Two calls for proposals for Centers and Nuclei were made. Due to a shortfall in counterpart funding only proposals selected from the first round were funded. B1. Centers and Nuclei of Scientific Excellence. Two Centers of Excellence and five Nuclei for Excellent Research were established to conduct socially relevant research in the areas of health and environment: Centers of Excellence: v Center of Excellence in Dynamics and Control of Landslides and Flooding (Centro de Excelencia - 7 - en Dinámica y Control de Aludes Torrenciales, IMF), carrying out hydrological research and analyses of Venezuelan river basins and building capacity to assess risk and deal with natural disasters caused by flooding and related phenomena, including the elaboration of flooding risk maps using sophisticated mathematical models (see Box 1 for more details); and v Center of Excellence in Biomedicine (Centro de Excelencia Instituto de Biomedicina), carrying out research, treatment and control of several endemic diseases, particularly Hanseniasis, Leshmaniasis and Oncocercosis. The Center also works on child health issues such as parasitosis and nutrition and conducts research on environmental health, for instance related to potable water and safe disposal of solid waste. Nuclei for Excellent Research: v Nucleus Viral Biology Lab (Núcleo del Laboratorio de Biología de Virus, IVIC), used advanced molecular biology tools to research viral illnesses such as Dengue, Hepatitis, Viral Diarrhea and Yellow Fever; v Nucleus for the Treatment of Industrial and Urban Waste using the Fenton Process (Núcleo para el Tratamiento de Efluentes Industriales y Urbanos mediante el Proceso Fenton), focused on treatment of effluents from petroleum distilleries and paper industries to reduce environmental pollution; Box 1. Center of Excellence in Dynamics and Control of Landslides and Flooding Project funding financed the upgrading of the Institute of Fluid Mechanics (located at the Central University of Venezuela in Caracas) into a Center of Excellence that performs studies based on mathematical modeling with the aim of predicting landslides and flooding, developing methodologies, and designing equipment for prevention, control, and risk reduction. Some key implemented research projects were: "Proceda," "Caracas," "Presas 2," and "Orinoco." Sub-project "Proceda" established a remote observation and monitoring system in the San José de Galipan River Basin in northern Venezuela. This area witnessed severe torrential avalanches in December 1999 with an estimated toll of 30,000 fatalities, and total economic losses estimated at US$1.79 billion. Several monitoring stations, transmitting real time data, were installed to study landslide generation mechanisms and to serve as an early alert system. Sub-project "Caracas" consisted of the elaboration of risk maps for fluvial and torrential floods in the Caracas Valley. These maps are essential in attempts to prevent fatalities from natural disasters among the Venezuelan poor that live in the hills surrounding the Caracas Valley, resembling the Brazilian "favelas." Sub-project "Presas 2" defined flood risk probabilities in urban areas related to dam failures, using methodologies developed with a pilot dam built with project funds. Finally, sub-project "Orinoco" examined the dynamics of sediment transportation in the Orinoco River ­ the third largest in the world ­ to establish criteria to improve navigation. This sub-project may lead to important economic benefits, given the high dredging costs and the fact that the Orinoco river is not navigable all year long. Essential inputs for this research, in addition to sensors, probes, and samplers; were a 4x4 vehicle and a boat, which could not have been acquired under previous FONACIT criteria for R&D expenditures. For more information, see http://imf.ing.ucv.ve/ v Nucleus for the Consolidation of "La Iguana" Agro-Ecological Experimental Station (Núcleo de Consolidación de la Estación Experimental "La Iguana" como un Centro Nacional e Internacional de Estudios Agroecológicos en Sabanas), demonstrated new technologies in agriculture, such as alternatives for soil use in poor ecosystems, helping the conservation of biodiversity in the Venezuelan Savannah; v Nucleus for the Management of Guárico River's High Basin (Núcleo para el Manejo Integral de la Cuenca Alta del Río Guárico), carried out research to define technical criteria which serve as a basis for - 8 - strategic planning of land management to achieve optimal use of water in the area; and v Nucleus Amazon Center for the Research and Control of Tropical Diseases (Núcleo Centro Amazónico para la Investigación y Control de Enfermedades Tropicales, CAICET), combines research, prevention, monitoring and treatment of Oncocercosis, Intestinal Helmintiasis, Hepatitis, Tuberculosis and Malaria; particularly in indigenous communities (see Box 2 for more details). Each Center of Excellence was funded at the level of US$1.0 million per year for two years, and each Nucleus at the level of US$150,000 per year for two years to support investments in research, equipment, training of advanced human capital and network activities. The two years of funding have consolidated physical and human capital inputs but has not yet led to an increase in the publication productivity of funded researchers. New equipment in Centers and Nuclei has given researchers access to new and more relevant areas of research. Notably, supported Centers of Excellence and Nuclei have attracted young researchers by providing better working conditions and more possibilities for training and interaction with international peers (see Table 1 for details on increased qualifications as a result of the MSI). The funding for two additional nuclei (not listed above) was discontinued due to insufficient progress towards the accomplishment of their proposed objectives. The strengthening of the M&E system was, therefore, another important dimension of institutional learning. The M&E mechanisms put in place were evolving over time, and gave the IMU and the Program Committee a strong basis for evidence-based decision making. Box 2. Nucleus Amazon Center for the Research and Control of Tropical Diseases (CAICET) Project funding allowed CAICET (located in Puerto Ayacucho, in the Amazon State, site of indigenous communities) to accomplish its objectives of institutional strengthening by (i) introducing a systemic approach to health problems; (ii) strengthening research and service activities; and (iii) transferring knowledge applicable to community needs. A key institutional achievement was the formulation of a decree that would grant CAICET autonomy from the national Health Ministry, allowing CAICET to diversify its sources of funding and improve its management to focus better on research and service activities. Furthermore, CAICET was able to hire scientific and technical personnel, and complement skills in the areas of environment, anthropology, epidemiology and management. Also, CAICET was strengthened by an increase of researchers with master's and doctoral degrees, the training of technical personnel, the refurbishing of installations and the procurement of equipment, goods and consumables. For the first time in its twenty years of existence, CAICET was able to introduce a systemic approach to the research and control of endemic diseases, having now a holistic vision, and forming truly multidisciplinary teams of researchers and professionals from all of the center's groups (including physicians, epidemiologists, anthropologists, and biochemists). As a result, the social and anthropological aspects of the work were strengthened. Research and control of malaria, onchocerchiasis, dengue, viral hepatitis, and tuberculosis, which are prevalent among populations of urban areas and indigenous communities, were improved. In addition, there were notable improvements in the communications of results to the indigenous communities, resulting in increased participation, not only as subjects of the studies, but also as contributors to the definition of the research agenda. There was also increased collaboration with other research centers (e.g., MSI beneficiary IVIC) and other related institutions (e.g., UNICEF), in addition to active participation in the consolidation of the malaria, tuberculosis, leishmaniasis, and anthropology networks. Finally, CAICET was able to conduct knowledge transfer activities in areas of need pointed out by the served communities. An example was the use of the bromathology lab and personnel to support small pineapple producers in identifying ways to process frozen pulp. - 9 - Innovative Ideas. The project supported six innovative pre-proposals to pursue a novel research concept by conducting preliminary analysis and developing a full proposal to be implemented through research networks. This activity was not foreseen in the PAD, and its inclusion emphasizes the learning value and flexibility provided by the LIL. The following Innovative Ideas received a one-time funding of between US$50,000 and US$91,000: v Design and building of a system to manufacture cocoa at medium scale; v Improvement of metabolism studies; v Network for the molecular study of dengue; v Identification and molecular diagnostic of micro-organisms; v Simultaneous multi-diagnostics to detect infectious diseases; and v Research, cooperation and innovation network for agricultural technological improvement. B2. Network for the Promotion of Scientific Excellence. The project facilitated the formation and strengthening of a number of formal and informal networks among researchers in Venezuela and abroad, as listed in Table 2. Table 3 below exemplifies the diverse membership of some of the networks, both national and international. Table 3. Membership of Selected Research Networks supported by the MSI Leishmaniasis Network: Venezuelan universities, Ministry of S&T, Ministry of Health, 28 regional dermatology offices, and agricultural communities Malaria Research Network: Belgium, United Kingdom, and Venezuela Iberian American Network of Hydraulic Research Institutes: Argentina, Brazil, Bolivia, Chile, Colombia, Costa Rica, Cuba, Ecuador, España, Guatemala, Mexico, Peru, Portugal, United Status, Uruguay, and Venezuela Guárico River Basin Network: Costa Rica, France and Venezuela Source: N. Núñez, 2005 4.3 Net Present Value/Economic rate of return: In addition to expected benefits due to proper investments in R&D (see Annex 3 for details), the MSI has been cost effective, particularly in the context of the general crisis described in section 5 below. This crisis brought the funding for S&T projects in general to a halt. Therefore the MSI allowed for continuity in funding for the sector, concentrating scarce resources into well selected and monitored Centers and Nuclei whose objectives were aligned to relevant socio-economic needs in the health and environment fields. The MSI showcased cost-effectiveness criteria in an S&T system not used to close monitoring of the research project portfolio. It was a novelty for the S&T system that research projects (in this case two nuclei) were suspended for not meeting their goals. Freed resources were allocated into other worthwhile activities, such as the Innovative Ideas program. The project also opened avenues for potential socio-economic benefits arising from supported R&D activities. These potential benefits include a lower number of labor hours lost to tropical diseases, more sustainable rural production and lives saved to preventable diseases and natural disasters such as landslides and flooding. 4.4 Financial rate of return: Not applicable. 4.5 Institutional development impact: Institutional development impact is substantial, at the Government level and at the research institutions level through capacity building and by the adoption of new methodologies for funding and execution of - 10 - R&D activities. At the Government level the Bank assisted in the set up of the Ministry of S&T. Moreover, the project contributed to increasing FONACIT's capabilities to make effective use of Venezuela's human and financial resources. FONACIT experimented with a new methodology for S&T funding, elements of which it is adopting in its regular funding programs (see "General Assessment" in section 3.1 for an elaboration on methodologies). Moreover, the project allowed FONACIT to introduce yet another, albeit smaller initiative to fund innovative ideas. At the same time, FONACIT is making provisions for further funding of the different ongoing projects (Centers, Nuclei and Innovative Ideas) based on evaluations by the Program Committee. At the level of individual beneficiary institutions, capacity related to project management was gained, at the same time that they were able to define and tackle more complex scientific problems. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: In 2001, 2002, and part of 2003, Venezuela underwent severe fiscal and political crises, which at its worst, provoked difficulties to cover even basic government operating expenditures, such as public sector salaries and health services. Early causes for the crisis were external shocks due to declining oil prices, coupled with "capital flight", and extreme polarization of Venezuelan politics. These problems prompted currency devaluation and tight exchange controls. In this environment, and amidst massive protests, a frustrated coup d'état was carried out in April 2002, followed by a two-month long general strike that almost brought the country to a standstill. These events had an impact on the MSI, delaying implementation particularly due to a shortfall of counterpart funding. It was against this background that the project's cost sharing was changed. 5.2 Factors generally subject to government control: The GoV codified national practices for procurement into law following negotiations of this project. The proposed legislation did not clearly state that provisions such as those of the Loan Agreement take precedence over national procurement legislation. As a result, the Bank was not in a position to sign the negotiated Loan Agreement. The time required to align Government legislation and Bank policy delayed the project by almost a full year. 5.3 Factors generally subject to implementing agency control: The dialogue and cooperation with the implementation unit were excellent. Despite a challenging context, agreed improvements were carried out in an effective and timely manner. The project's Program Committee had an important role in emphasizing the social relevance of research. However, it went too far in demanding process related tasks from researchers with little or no administrative assistance. As a result, important time for research went into writing lengthy reports. Moreover, the Committee demanded item-by-item approval for international network activities and as such did not provide sufficient autonomy to principal researchers for managing inputs and activities. 5.4 Costs and financing: Total project costs estimated in the PAD were US$15.0 million, including US$5.0 million to be provided by the Bank loan. Due to the factors affecting project implementation (described in sections 5.1 and 5.2) total project budget was reduced to US$8.0 million, while maintaining the loan amount. The reduction in budget did not affect the accomplishment of project objectives, but meant a reduction in the number of Centers and Nuclei supported. Two Centers and 5 Nuclei were established, as opposed to 3 Centers and 8-12 Nuclei as originally foreseen. The expected demonstration effect and learning objectives were - 11 - achieved nevertheless as lessons learned are now being integrated into other S&T policies in Venezuela (See Section 6). Final project costs were US$7.81 million (98% of revised budget) and final loan amount was US$4.81 million (96% of original amount), in line with the revised costs (See Annex 2 for details on project figures). The Government plans to continue funding for those projects whose final reports are deemed satisfactory by the MSI's Program Committee. 6. Sustainability 6.1 Rationale for sustainability rating: Sustainability of the project is likely. The sustainability assessment is based on the following observations: l The project made a significant institutional development impact by strengthening the Ministry of S&T, and showing the benefits of awarding large multi-purpose grants for research. FONACIT has responded by allocating grants (in regular funding programs) based on the principles showcased by the MSI. l The training of advanced human capital has increased the pool of advanced human capital. This has given an important boost to the research capacity of Venezuela. However, the impact is contingent on the ability to retain young researchers by offering them appropriate research opportunities and working conditions. l The project has provided researchers with cutting-edge equipment. The improved research infrastructure has opened new possibilities for conducing high-quality research and interacting with international peers. This capacity will remain even if supported researchers do not receive funding targeted at investments in the near future. l Continuation of project activities. FONACIT has opened a funding line that would allow supported beneficiaries (Centers, Nuclei and Innovative Ideas) of the project to send in a proposal for activities that would help maintain the investment made. The budget for these grants would come out of the 2005 FONACIT budget. The impact assessment (Núñez, N., 2005) finds that two-thirds of researchers consider sustainability likely: 24 percent of researchers in supported Centers and Nuclei are convinced that their research would continue without funding from the project; 31 percent state that the project would be able to continue with difficulty; and 12 percent consider sustainability likely if project objectives are reformulated. About a third of researchers do not think they would be able to continue their project activities without further support. 6.2 Transition arrangement to regular operations: As noted, many of the project features are in the process of being institutionalized by FONACIT. The GoV is seeking further support from the Bank to continue strengthening the S&T sector, as envisaged at the preparation stage (See Section 3.5). - 12 - 7. Bank and Borrower Performance Bank 7.1 Lending: Lending performance is rated satisfactory. The project was prepared in a short time, taking full advantage of discussions with Venezuelan stakeholders and lessons learned from the Chile MSI (World Bank, 2002b). Although it would have been impossible to anticipate external events as those described in section 5, a more flexible implementation schedule would have been more appropriate for a less than stable political landscape. 7.2 Supervision: Overall, supervision efforts were satisfactory, including close monitoring, as reflected by a mid-term review (Barreto de Castro, L., C. Wernli and N. Núñez, 2004) and a final evaluation (Núñez, N., 2005) in a relatively short implementation time. The technical team responded quickly and positively to numerous requests derived from the crisis situation, including the processing of extension of the closing date and the amendment to the cost sharing percentages. The Bank's Resident Representative served as an indispensable facilitator. Moreover, the project Task Manager worked intensely to keep the project on track until completion despite adverse external conditions. The borrower highlighted the Task Manager's traits that proved to be useful for project implementation: deep sector and project knowledge, thorough understanding of the country's difficulties, realistic problem solving skills, and continuous willingness to help. 7.3 Overall Bank performance: Overall Bank performance is rated satisfactory. Borrower 7.4 Preparation: Borrower preparation performance is rated satisfactory. The borrower had high level support and commitment, which provided the right framework to start the MSI, under FONACIT and within the newly created Ministry of Science and Technology. The project was well aligned to the Ministry's vision to "(i) establish coherent policies in the national science and technology sector; (ii) consolidate Venezuela's national innovation system; (iii) introduce incentives for the formation of a critical mass of high quality researchers; and (iv) support research and development in priority technological areas" (World Bank, 2000). 7.5 Government implementation performance: The implementation performance of the government is rated satisfactory. Despite the difficulties described in section 5, the Government sustained the project, giving it high priority. 7.6 Implementing Agency: The performance of the implementing agency is rated highly satisfactory. The IMU experienced several changes in the coordinator role at the beginning of the project, but soon formed as a very cohesive group that worked closely with project beneficiaries, the Program Committee and the World Bank to ease implementation under difficult circumstances. The IMU also facilitated external reviews of the project and ensured excellent communication among project stakeholders. 7.7 Overall Borrower performance: Overall Borrower performance is rated satisfactory. Although the overall implementation performance was - 13 - satisfactory, performance could have been more effective if the roles ("division of labor") of the different project implementation bodies (IMU, Program Committee, and Board of Directors) were clearly defined. 8. Lessons Learned l Research excellence and relevance can go hand in hand. The project featured a number of promising examples of how to align research with national priorities and the needs of indigenous people and the poor. l Initial proposal should contain a plan for transition to regular activities. Support for Centers and Nuclei should only be provided based on a clear strategy of how activities will be continued after the end of the grant period. Expecting a follow-on Bank-financed project, the Institute of Fluid Mechanics used grants to finance recurrent expenditures on a non-declining basis. As a consequence, the Center faces the challenge of sustaining the investment in the training of highly-skilled researchers. l Extra effort is needed for human resource formation in countries with no critical mass. Developing countries are often faced with a shortage of university graduates willing to start a career in sciences and many existing researchers do not have adequate qualifications. The project showed that advances can be made by providing research staff with opportunities to obtain advanced degrees, customized to their needs; and to attract students through adequate training programs. l Investigator autonomy is key to research success. The best research results are achieved by "packaging" different categories of expenditures into one "multi-purpose, multi-year" research grant, including resources to pay salaries of graduate students and staff, and networking and diffusion activities. To ensure accountability, a clear and simple set of results indicators should be established ex-ante and validated with the researchers. l The importance of supporting network and dissemination activities tends to be underestimated. Networking and dissemination are often perceived by researchers and S&T administrators as add-on activities. However, as the project showed, such activities are important. Through S&T dissemination politicians and society at large become aware of the importance of building a capacity for research, and through the establishment of networks resources are leveraged and some of the disadvantages of low numbers and geographical dispersion are mitigated. l External controls and accountability can defer time from core activities. Although monitoring and controls are needed, excessive administrative demands, such as the request of bi-annual reports from research teams with little or no administrative support staff, required researchers to spend a significant amount of time on writing reports documenting their results. This took away time from conducing research, guiding students and writing articles. 9. Partner Comments (a) Borrower/implementing agency: The Millennium Science Initiative (MSI) Program was designed to support the work of important research teams, grouped around centers and nuclei of excellence, producing science relevant to national interests and of international quality. The achievements listed below were reached in spite of significant challenges posed by a need to adjust implementation capacities to promote the transformation of the S&T sector and ensure its involvement in the solution of national problems. Financing of 2 Centers and 5 Nuclei, established to strengthen research activities in the areas of health biotechnology, risk management, food security and sustainable production of drinking water. The Centers and Nuclei thus contributed to the national strategic objectives of improving effectiveness and social inclusion, indispensable requirements for development and the well-being of all citizens. The research capacity was strengthened through the incorporation of new laboratory equipment and research personnel and by increased emphasis on new research problems, and on the dissemination and transfer of knowledge. - 14 - The grants helped train a new generation of researchers, although the relative short duration of the project did not allow for a complete evaluation of this component. "Innovative ideas": 6 proposals were funded, each oriented to produce a final proposal, including a pilot phase, for the conformation of innovation networks. The results include: (i) design and manufacturing of six prototypes, aimed at increasing productivity in the processing of cocoa; (ii) development of multi-diagnostic techniques for endemic infectious diseases; (iii) application of bio-molecular techniques for the identification and diagnostic of microorganisms in health, agriculture, oil and environment; (iv) establishment of a network for the molecular study of dengue, with expected impact in the health and education sectors; (v) creation of a scientific research, cooperation and innovation network for the improvement of agricultural technology; and (vi) the inclusion of ten new techniques in the National Center for the Study of Metabolic Defects. These results have an important potential for the successful development of the respective proposals in some cases, and for their scale-up in the short term in other cases, given the importance of the selected areas. The subcomponent Network for the Promotion of Scientific Excellence was implemented within the framework of Project's objectives related to the strengthening of the S&T sector. Ten networks were funded in the health area and six in the area of environment. The networks are fundamental for the continuity of research and the accomplishment of long term objectives, beyond the funding from the MSI. The project also financed researcher exchange, attendance to conferences, promotion visits, and internships that lead to the building and strengthening of alliances with national and international institutions, favoring collaborative work and access to a wider knowledge pool. Unfortunately, this component was only developed in depth starting in 2003. As to the strengthening of the Ministry of Science and Technology, and the support to its strategies to improve the S&T sector, the project financed, among others, the following efforts: l The "Route of Science" mobile unit (a refurbished bus) that will fulfill tasks of knowledge dissemination and scientific promotion while traveling all over the country; l A feasibility study for the creation of the National Chemistry and Catalysis Center; l A Public Perception of Science Survey. Its results were an important input for the National Science, Technology and Innovation Plan 2005 ­ 2024, as well as for the definition of short and medium term strategies and policies; and l An International Seminar on Public Policies and Planning in Science and Technology, carried out in Caracas in October 2004. Experts from diverse countries (in the region and beyond) met to discuss different aspects of sectoral planning and its importance to leverage national development strategies. In general, greater autonomy was provided to researchers for the management of their resources, facilitating administrative processes to a maximum. The Project contributed to strengthening the research capacity and funded important research inputs. The researchers showed motivation, responsibility and commitment towards the sub-projects, particularly after the external evaluation of May 2002 and the monitoring and control performed by the Program Committee. The latter allowed, through memoranda of understanding, the execution of needed actions for the accomplishment of Project's objectives. In summary, the researchers are enthusiastic about the future of their projects. It can be concluded that all selected Institutions and their research lines are high-priority for the country. With respect to Bank performance, we recognize the excellent project management by the Task Manager and his team, as well as their very high capacity, technical and scientific qualifications. We rate the Bank performance as highly satisfactory. However, some Bank administrative and bureaucratic aspects related to - 15 - procurement and disbursement must be reviewed as they constituted important obstacles in some occasions. In addition, and since 2003, it was fundamental the unrestricted support from the authorities at FONACIT and the Ministry of Science and Technology, who were committed to the carrying out of previous agreements, and effectively contributed to their implementation. Important lessons from this pilot phase of the MSI in Venezuela include: l Greater researcher autonomy, in the form of direct resource transfer to the researchers, allows for a quicker verification of results and better decision-making with respect to resource allocation; l Close and adequate sub-project monitoring allows timely changes in planned activities, and reduces the risk of "losses" due to inefficiencies in the use of resources; l Greater learning was achieved by scientific teams in their quest for a better inter-disciplinary and inter-institutional integration; l Institutional learning was achieved on the need to keep close monitoring and evaluation; l The technical strength of the implementation unit team and the good coordination with FONACIT, the researchers, and the Bank team constitute a good guarantee for success; and l The commitment of authorities, through concrete actions to ease processes and eliminate bottlenecks, is crucial. But other lessons should also be recognized and considered for future experiences: l The analyses during the design phase should be exhaustive and based on as much updated data as possible, in order to diminish the risk of false perceptions that may lead to misinterpretation of the reality, and therefore, to inexact or vague strategies, objectives and goals; l Aspects that in the end are key for successful implementation, such as evaluation procedures and the role of the Program Committee and the different organizational bodies must be fully developed during the design phase and should include precise indications for their implementation. Also, fundamental concepts like excellence and autonomy should be defined without ambiguities; l Coherence between the design of monitoring and evaluation processes and key project variables aiming at the identification of suitable results and the measurement of the level of innovations should be ensured during a pilot phase. The researchers have indicated that agreeing to output and management indicators at the start of sub-projects would have facilitated the process both for the Program Committee and for themselves; l It should be considered during the design phase the time needed to master operating mechanisms and Bank's norms, which affected the achievement of planned milestones; l Researcher autonomy, coupled with direct funds transfer, makes it possible the rapid verification of results and allow for opportune decision making on resource management. However, it does not necessarily facilitate an adequate integration of the different research teams that constitute the Centers and Nuclei; and in some cases facilitated non-inclusiveness; l Weaknesses were detected among researchers regarding the planning of activities related to relationships with users of their research, in addition to some lack of clarity in sub-project design, during the preparation of the Project; l It is fundamental that the initial project document includes greater details with respect to key implementation procedures, including the consideration of realistic execution times; l The peculiarities of the national system of science and technology, its size, ways of operation, labor calendar, work culture, etc.; should be taken into account when defining strategies, objectives and goals; l Each project component must be extensively described from the beginning so that the implementation of all components can begin simultaneously to guarantee a better project implementation rhythm; and - 16 - l A pari-passu relation of 70/30 (Government-Bank), as initially established, is not operative nor realistic. Considering the advances of the MSI, the country has officially requested the Bank to include a new project in the Country Assistance Strategy to capitalize and harness the experience and gains obtained in this learning stage. A follow-on project should incorporate aspects initially not pondered but strategically important for national development, such as endogenous development, productive diversification and social appropriation of knowledge for sustainable local development, from the perspective of the innovative municipality. (b) Cofinanciers: N/A (c) Other partners (NGOs/private sector): N/A 10. Additional Information N/A - 17 - Annex 1. Key Performance Indicators/Log Frame Matrix Indicator/Matrix Baseline Actual/Latest Estimate (2000) (2004) Average number of MA students per 1.07 MA students 1.41 MA students researchers in supported Centers and Nuclei Average number of PhD students per 0.59 PhD students 1.26 PhD students researchers in supported Centers and Nuclei Average number of post-docs 0.22 post-docs 0.41 post-docs collaborating with each researchers in supported Centers and Nuclei Annual number of publications in indexed 73 publications 99 publications journals by researchers in supported Centers and Nuclei Annual number of international visitors to15 international visitors 33 international visitors Centers and Nuclei Annual number of international visits by 3 international visits 39 international visits members of Centers and Nuclei Total number of virtual research networks 0 research networks 6 research networks implemented by Centers and Nuclei in supported Centers and Nuclei Annual amount of resources mobilized US$ 1,731 US$ 222,134 from other organizations to projects at Centers and Nuclei - 18 - Annex 2. Project Costs and Financing Table 1. Project Cost and Financing by Components (in US$ million equivalent) Appraisal Budget (US$ Amended Budget (US$ Actual Budget (US$ Actual total Components millions) millions) millions) costs as % GoV Bank Total GoV Bank Total Gov Bank Total of amended budget A. Capacity building: Ministry of S&T, CONICIT/ 0.50 0.50 1.00 0.96 0.98 1.94 0.94 1.00 1.94 100% FONACIT and the MSI Directorate: B1. Centers and Nuclei of Scientific 8.53 3.48 12.00 1.91 3.61 5.52 1.89 3.46 5.35 97% Excellence B2. Network for the Promotion of Scientific 0.75 0.75 1.50 0.17 0.32 0.49 0.17 0.30 0.47 96% Excellence Front End Fee 0.05 0.05 0.05 0.05 0.05 0.05 100% Unallocated 0.23 0.23 0.45 Totals 10.00 5.00 15.00 3.04 4.96 8.00 3.00 4.81 7.81 98% Table 2. Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) Procurement Method Expenditure Category ICB NCB Other1 Total Cost (1) Grants 0.00 0.00 12.00 12.00 (2) Goods 0.00 0.00 0.15 0.15 (3) Consultant Services 0.00 0.00 2.30 2.30 (4) Operating Costs and Front- 0.00 0.00 0.55 0.55 end-fee Total 0.00 0.00 15.00 15.00 - 19 - Table 3. Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) Procurement Method Expenditure Category ICB NCB Other1 Total Cost (1) Grants 0.00 0.00 5.35 5.35 (2) Goods 0.00 0.13 0.39 0.52 (3) Consultant Services 0.00 0.00 1.67 1.67 (4) Operating Costs and Front- 0.00 0.00 0.27 0.27 end-fee Total 0.00 0.13 7.68 7.81 1/ Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. - 20 - Annex 3. Economic Costs and Benefits Economic analyses consistently show high social and economic rates of return to research and development (R&D) and innovation. It is well established in the literature that a broad definition of technological progress (which includes scientific research) accounts for a large part of economic growth. For example, Esterly and Levine (2000) using the growth regression methodology, estimate that up to 80 percent of cross-country variation in economic growth can be attributed to technological progress. For the Latin American and Caribbean region, the World Bank study "From Natural Resources to the Knowledge Economy" (De Ferranti et al, 2002) finds that knowledge and technology is a critical determinant for growth in the region, especially because of the countries' dependence on natural resources. Lederman and Maloney (2003) estimate returns for a panel of countries explaining economic growth by control variables ­ capturing the impact of initial development, physical capital and labor growth ­ and by policy variables related to the national innovation system. They find that the size, function and efficiency of the innovation system strongly influence economic performance. Some of the key findings are: l Social Returns to R&D exceed by far the return to investment in physical capital by a factor of 6 to 10 depending upon the initial level of income per capita; l Returns to R&D increase with the intensity of natural resources in the economy. Countries that rely heavily on natural resources, enjoy higher returns on R&D than more manufacturing-dominated economies; and l Formation of advanced human capital as measured by tertiary education is a prime driver of technical progress, and appears to be more important than basic education. This argument can be extended to graduate education. Efforts to strengthen the science base do not inevitably reflect macro-growth findings. The most common methodologies usually measure either (i) the impact of publications and citations resulting from the research or (ii) the economic value of outputs of research, in terms of production function analysis or social rates of return. However, the best available methodologies are often unable to measure the outputs completely from fundamental research (Popper, 1995) which may include: l Training new scientific and technical workers; l Enriched expertise and experience of current researchers; l Development of "schools" of researchers who represent an informal community focused on a particular system; l Exploration of new experimental designs, instrumentation, and research protocols; l Increased numbers of "guesses" about the nature of the system being explored; l Lowering the notional price of an "option" on a prospective technological application which may or may not have some value in the future and for which the basic research finding provides an input; l Negative results which are important in more effectively directing subsequent efforts which may then lead to positive results; and l Intangible benefits conferred on society and the public imagination by the search itself, and/or its contribution to a culture of science that will facilitate future discoveries. Some of these outputs are not amenable to empirical testing, or are only amenable to certain forms of qualitative assessments. In addition, time frames for the appearance of output from fundamental research are idiosyncratic and non-linear. Important findings from fundamental research have remained unused for decades until being "discovered" by happenstance. Even when results are utilized immediately after publication, time horizons for the development of applications can run ten years or more. For these reasons, and consistent with the guidelines for LILs, no formal rate of return analysis was undertaken for the MSI - 21 - project. In addition to expected benefits due to proper investments in R&D, the MSI has been cost effective, particularly in the context of the general crisis described in section 5. This crisis brought the funding for S&T projects in general to a halt. Therefore the MSI allowed for relative continuity of funding in the sector, concentrating scarce resources into well selected and monitored Centers and Nuclei, whose objectives were aligned to relevant socio-economic needs in the health and environment fields. Actually, the MSI showcased cost-effectiveness criteria in an S&T system not used to close monitoring of the research project portfolio. It was a novelty for the S&T system that research projects (in this case two nuclei) were suspended for not meeting their goals. Thus, now freed resources could be allocated to other worthwhile activities, such as the "Innovative Ideas" program. Another important dimension of cost-effectiveness corresponds to the "Network for the Promotion of Scientific Excellence" sub-component, which, with a relatively modest amount (5% of final project costs), provided great leverage for project investments, not only by strengthening existing networks and creating new ones, but also making use of these networks to share human and physical infrastructure, and to achieve increased collaboration across scientific teams. Finally, the project opened avenues for potential socio-economic benefits deriving from the supported R&D activities. These potential benefits include a lower number of labor hours lost to tropical diseases, more sustainable rural production and lives saved to preventable diseases and natural disasters such as landslides and flooding. - 22 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation May 16, 1999 3 Task Team Leader, Lead S&T Specialist (1); Lead Specialist Education (1); Consultant, Resident Mission Staff (1) August 22, 1999 2 Task Team Leader, Lead S& T Specialist (1); S&T Specialist (1); October 6, 1999 4 Task Team Leader, Lead S& T Specialist (1); S&T Specialist (1); R&D Specialist (1); Consultant, Resident Mission Staff (1) Appraisal/Negotiation February 14, 4 Task Team Leader, Lead S& 2000 T Specialist (1); S&T Specialist (1); Financial Management Specialist (1); Procurement Specialist (1) Supervision April 2, 2001 5 Task Team Leader, Lead S& S S T Specialist (1); S&T Specialist (1); Financial Management Specialist (1); Procurement Specialist (1), Consultant (1) August 30, 2001 2 S&T Specialist (1); Consultant S S (1) April 18, 2002 2 Task Team Leader, Lead S& T S S Specialist (1); Consultant (1) October 4, 2002 3 Task Team Leader, Lead S& T S S Specialist (1); Consultant, Economist (1); Languague Program Assistant (1) May 5, 2003 2 Task Team Leader, Lead S&T S S Specialist (1); Resident Mission Representative (1) February 3, 2004 2 Task Team Leader, Lead S&T S S Specialist (1); Consultant, Monitoring & Evaluation Specialist (1) October 13, 2004 2 Task Team Leader, Lead S&T S S Specialist (1); Consultant, Monitoring & Evaluation Specialist (1) - 23 - ICR March 14, 2005 2 Consultant, Monitoring & S S Evaluation Specialist (1); Consultant (1) (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation 10 56,104.84 Appraisal/Negotiation 5 28,052.40 Supervision 44.76 376,337.16 ICR 9.78 50,138.98 Total 69.54 510,622.38 - 24 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA - 25 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU - 26 - Annex 7. List of Supporting Documents Barreto de Castro, L., C. Wernli and N. Núñez, 2004. External Evaluation of the Project: Millennium Science Initiative, BM-4572-VE. De Ferranti, D. et al, 2002. From Natural Resources to the Knowledge Economy. -World Bank Latin American and Caribbean Studies, World Bank, Washington, DC Easterly, W and R. Levine (2000). It is not factor accumulation: stylized facts and growth models. World Bank Economic Review 15: 177-219. Washington D.C. FONACIT, 2005. Final Reports submitted by the Centers of Excellence and Nuclei for Excellent Research. Caracas, Venezuela. Lederman, D. and W. Maloney (2003). R&D and Development. The World Bank. Washington D.C. Núñez, N., 2005. Impact Assessment of the Millennium Science Initiative in Venezuela. "Hechos significativos y lecciones aprendidas: El trabajo individual de los investigadores beneficiarios en la evaluación de impacto del programa Iniciativa Científica del Milenio ­tal como éste fue implementado-." Caracas, Venezuela. Popper, S. (1995). Economic Approaches to Measuring the Performance and Benefits of Fundamental Science. RAND, Critical Technologies Institute, Washington D.C. World Bank, 1997. Memorandum of the President of the International Bank for Reconstruction and Development to the Executive Directors on a Country Assistance Strategy of the World Bank Group for the Republic of Venezuela. Report Number 16471-VE. Washington, D.C. _______, 2000. Project Appraisal Document: Bolivarian Republic of Venezuela Millennium Science Initiative Project. Report Number 19853. Washington, D.C. _______, 2001-2004. Aide-Memoires and Back to Office Reports corresponding to project missions carried out on April 2001; August-September 2001; April, 2002; May 2003; September 2003, and October 2004. _______, 2002a. Memorandum of the President of the International Bank for Reconstruction and Development to the Executive Directors on an Interim Country Assistance Strategy for the Bolivarian Republic of Venezuela. Report Number 25125-VE. Washington, D.C. _______, 2002b. Implementation Completion Report: Chile Millennium Science Initiative Project. Report Number 24069. Washington, D.C. - 27 - - 28 -