Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) Report Number : ICRR0020234 1. Project Data Project ID Project Name P082520 CO Sustainable Development Inv Project Country Practice Area(Lead) Additional Financing Colombia Environment & Natural Resources P115639 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-73350,IBRD-81330 31-Dec-2009 19,050,000.00 Bank Approval Date Closing Date (Actual) 27-Oct-2005 29-Apr-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 17,000,000.00 0.00 Revised Commitment 13,761,652.80 0.00 Actual 13,761,652.80 0.00 Sector(s) Central Government (Central Agencies)(78%):Water Supply(10%):Sub National Government (Central Agencies)(5%):Urban Transport(5%):Sanitation(2%) Theme(s) Environmental policies and institutions(33%):Pollution management and environmental health(33%):Water resource management(17%):City-wide Infrastructure and Service Delivery(17%) Prepared by Reviewed by ICR Review Coordinator Group Isabelle Daverne George T. K. Pitman Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The purpose of the Sustainable Development Investment Project was to support the design and implementation of policy reforms and related investments in line with the framework of the Development Policy Loan Program for Sustainable Development, particularly those policies and plans that address environmental problems affecting the quality of life and wellbeing of the Colombian population. The DPL Program 2005-2009, that cost US$800 million through three operations, was to support the government’s efforts to move towards achievement of the Millennium Development Goals, particularly MD7 that seeks to ensure environmental sustainability. According to the 2005 Loan Agreement (page 18), the objectives of the Sustainable Development Investment Project were: Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) “to support the Borrower in the development and implementation of policy reforms and related investments in line with the Program’s framework, particularly those polices and investments that address environmental problems affecting the quality of life and well-being of the Borrower’s population.” The Project Appraisal Document’s statement of objectives (PAD, page 7) is more detailed: “To support the Borrower in the development and implementation of policy reforms and related investments in line with the Program’s framework, particularly those polices and investments that address environmental problems affecting the quality of life and well-being of the Borrower’s population. In particular, the project will support (i) the incorporation of environmental considerations in the policies of the sectors that result in a high cost of environmental degradation (e.g., health, transport, and water); (ii) the implementation of policy measures for integrated water resource management; (iii) the optimization of urban development policy instruments to improve urban quality of life; and (iv) the strengthening of the planning, monitoring, and oversight of national, regional and local environmental management and policies.” Additional Finance was added to the project in 2012 and neither the 2005 Loan Agreement’s nor the PAD’s statement of objectives was changed (Additional Financing Project Document, page 2). b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components The original project had four components and these were amended at the time of the 2012 Additional Financing: A: Environmental Health (Appraisal cost estimate US$ 3.22 million; Additional Financing (AF) added US$ 1.5 million. Disbursement of the AF at closing was US$ 0.52 million, disbursement of the original Loan by subcomponent is not reported, and total actual costs are not known). 1. Design and implementation of an air pollution control policy through: (a) the provision of equipment and technical assistance for air quality management and monitoring, supporting improvements in the monitoring of those pollutants that are most harmful to human health; (b) the development and implementation of a consistent protocol for air quality monitoring; (c) the collection and transfer of air quality information gathered by selected air monitoring networks, including the development of a national, unified report on air quality; (d) the carrying out of studies to characterize particulate matter and develop a protocol for emission inventories; (e) the characterization of meteorological conditions to provide a greater understanding of these conditions on pollutant dispersion and increase knowledge of the health impacts of air pollution; and (f) the carrying out of studies to support the design of mechanisms and incentives for local authorities to control air pollution in selected cities and the preparation of a national guide for air quality modeling. 2. Design and implementation of a program to prevent hygiene-related diseases among the most vulnerable segments of the population in selected areas through: (a) the design and implementation of a hand-washing campaign at the national and local levels; and (b) the monitoring and evaluation of the effectiveness of said campaign. The Additional Financing of 2012 significantly changed the focus of activities supported under Environmental Health. • Subcomponent (1) was made more specific to include installation of an emissions testing laboratory (homologation center) for vehicles and motorcycles in order to ensure accurate measurement of moving vehicle emissions in the specific range of conditions found in Colombia such as the high altitude of many cities. In addition the revised component supported (a) Development of a prototype Unified Environmental Health Information System; (b) strengthening of the Air Quality Information System; and (c) strengthening of the capacity of Regional Environmental Agencies (Corporacions Autonomas Regionales (CARs) and Urban Environmental Agencies (Autoridades Autonomas Urbanas (AAUs)) to implement regulations on noise pollution, monitor air quality and implement technical norms on point source emissions. • Subcomponent (2) to support prevention of hygiene-related diseases was dropped. B: Sustainable Urban Development (Appraisal cost estimate US$ 0.93 million; Additional Financing added US$ 0.0 million; disbursement of the original Loan by subcomponent is not reported, and total actual costs are not known). 1. Support implementation of the Borrower’s sustainable urban development policy through: (a) development of guidelines on inter- sectoral coordination of public services and the carrying out of workshops to increase regional knowledge of the guidelines and promote inter-sectoral coordination; (b) design and implementation of mechanisms that regulate and provide incentives for improved urban drainage infrastructure; (c) strengthening of public space indicators; (d) assessment of management instruments related to public transport systems in cities with less than 600,000 inhabitants; (e) development of a methodology to integrate mobility plans within municipal ten-year development plans; (f) improvement of planning for urban densification and renewal, through the design of a legal framework to promote more effective use of public property and promotion of private sector involvement in re urban densification and renewal programs; and (g) Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) improvement of the planning process for areas subject to urban expansion through the development of guidelines that establish procedures and responsibilities of the CARs and other environmental authorities in reviewing expansion plans. 2. Provision of support for the implementation of the Solid Waste Decree through: (a) development of associated regulations that take into consideration legal, economic and financial instruments; and (b) provision of technical assistance to 200 municipalities through workshops and the dissemination of materials advocating closure of open dumps and the opening of sanitary landfills. The Additional Financing of 2012 excluded this component. C: Integrated Water Resource Management (IWRM) (Appraisal cost estimate US$ 0.40 million; Additional Financing added US$ 6.50 million. Disbursement of the AF at closing was US$ 4.65 million, disbursement of the original Loan by subcomponent is not reported, and total actual costs are not known). This financed three subcomponents: 1. Updating the national legal framework for water resource management through: (a) the provision of training to develop wastewater discharge standards and regulations for pollution fees; and (b) the carrying out of workshops to promote regional knowledge and consultations on a draft water law. 2. Strengthening of the water quality laboratory accreditation process through: (a) the carrying out of an assessment of the analytical capacity of regional laboratories responsible for monitoring water quality; (b) the development and implementation of an action plan to improve laboratory accreditation; and (c) the provision of technical support to Institute of Hydrology, Meteorology and Environmental Studies (Instituto de Hidrología, Meteorología y Estudios Ambientales IDEAM) in conducting audits and accrediting laboratories. 3. Strengthening knowledge on water resources at the regional level through the development of a guide for modeling the availability of water resources. The Additional Financing of 2012 significantly enhanced the focus and scope of this component by scaling-up activities and adding one new subcomponent: 1. Institutional Strengthening of MADS in the Area of Water Resources Management (US$1.0 million). This had two sub-components: (a) Strengthening of the capacity of the Integrated Water Resources Management Directorate of MADS to allow the Directorate to implement the National Water Resources Policy and the Project; and (b) carrying out capacity building activities and training programs for CARs in technical topics related to water resources management, including, inter alia, integrated water resources management, the Borrower’s integrated water resources system, and the water regulations introduced to implement the National Water Resources Policy. 2. Integrated and Participatory River Basin/Aquifer Planning and Management (US$3.4 million). Support for the implementation of the National Water Resources Policy through the development of priority planning and technical instruments for integrated water resources management and testing the application of integrated water management tools in pilot watersheds, including: • Update of a participatory groundwater management plan, including the updating of the integrated participatory plan for the aquifer of la Sabana (Bogota), and increasing the technical understanding of the aquifer in order to facilitate its improved management, hydrological modeling, water quality monitoring, and developing inventories of water users; • Development of a participatory methodology to legalize water use rights and associated discharges through the carrying out of two pilot technical assistance (TA) projects in the Chinchina and Guali watersheds. This was to include the development of registers of water users and water polluters in each of the two watersheds, preparation of water balances, and ensuring adherence to the consultation process agreed in the Environmental and Social Management Framework (ESMF); • Carrying out a public communications strategy at the national level consisting of a media campaign to raise awareness of the need for, and benefits from, the legalization of water-use rights and associated wasterwater discharges; • Strengthening of technical norms and strengthening of the implementation capacity of the Regional Environmental Agencies (CARs) in Colombia, in coordination with IDEAM. This included (i) development of guides and manuals for environmental zoning, definition of ecological flows and discharge monitoring protocols to be used to strengthen the capacity of CARs to prepare watershed management plans (Planes de Ordenamiento y Manejo de Cuencas), and the administration of water rights and discharge permits; and • Updating and/or development (in a participatory manner through consultations with main water consumers) of new proposed technical norms on water quality criteria for various water uses, and wastewater discharge to the soils and coastal areas. 3. (New) Management of Climate Risks in the Water Resources Sector (US$0.5 million). This aimed to provide technical support to CARs to implement a pilot project, in coordination with IDEAM, to help them fulfill their new responsibility of generating information about flood risk area through (a) the development of a methodology to identify flood risk areas, and (b) development of a methodology to elaborate flood risk maps at regional and local levels, and (c) validation of this methodology. 4. Strengthening Water Resources Monitoring and Information (US$1.5 million). This included (a) development of additional modules for IDEAM’s information system for water resources management, including the state of water resources, risks, and management indicators; and (b) strengthening of IDEAM’s national monitoring network through the acquisition, assembly and installation of 14 hydrological stations and other equipment necessary for the strengthening of the national water-monitoring network in selected areas. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) D: Planning and Oversight of Environmental Management (Appraisal cost estimate, US$ 1.86 million; Additional Financing added US$ 2.0 million. Actual AF disbursement at closing was US$ 1.6 million; total cost is not known). This component aimed to improve the effectiveness and efficiency of the National Environmental System (Sistema Nacional Ambiental - SINA) at national and regional levels and it financed goods, consultants’ services, training, and operating costs to support: 1. Planning, monitoring, and oversight instruments to facilitate policy implementation and to monitor indicators established in Three-Year Annual Plans (Decree No. 1200) of the Regional Environmental Agencies (CARs); 2. The institutional strengthening of the National Planning Department (Departamento Nacional de Planeacidn - DNP) and the Ministry of Environment, Housing and Territorial Development (Ministerio de Ambiente, Vivienda y Desarrollo Territorial – MAVDT) to design and implement policies, as well as to improve, MAVDT’s coordination with CARs and other SINA actors; 3. The implementation of two strategic environmental assessments on environmental health and fuel quality; and 4. Assist the MAVDT in project implementation and oversight. The Additional Financing of 2012 supported three revised components at the national level providing goods, consultants’ services and training, and operating costs: 1. Strengthening the National Planning Agency’s (DNP’s) capacity to issue the Environmental Health Policy through for inter-sectoral coordination including establishment of thematic round tables (permanent discussion fora with a wide range of stakeholders), undertaking diagnostic studies at national and local levels, and the preparation of dissemination strategies and communication of environmental health information; 2. Enabling the National Planning Agency’s ’s integrated water resources management to establish institutional pathways, implementation mechanisms and an action plan for the implementation of the National Water Resources Policy. This included strengthening the DNP's capacity to support the national information system for water resources management and environmental information or decision-making, and formulation of an institutional and financing strategy to upgrade water resources monitoring and information systems; 3. Development of the second-phase DPL Program to reduce the Borrower's fiscal vulnerability to disasters. This included inclusion of risk management criteria for public investment projects, establishment of mechanisms to guide reconstruction efforts in response to national disasters, and the development of a strategy for the inclusion of risk management in sectoral planning and investment programs; and 4. Development of an environmental information management strategy for decision-making in the areas of water resources and air quality management. 
 d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs: At approval baseline costs were US$ 8.05 million including physical and price contingencies of US$ 1.64 million. With Additional Financing total project costs were expected to be US$17.00 million excluding some contingencies. At closing project cost was US$6.99 million for the original Loan and US$7.81 million for AF. Total cost by component is not stated in the ICR because costs of the original Loan were by the type of disbursement (goods, TA etc.,) those of the AF are classified by its project components and only US$6.76 million of its costs are allocated. The largest share of the AF budgetary resources was allocated to the Integrated Water Resources Management component that accounted 72% percent of the total estimated AF cost. While the Environmental Health component accounted for only 35% of its cost estimate, Planning and Oversight of its Environmental Management component accounted for 79% of its cost estimate. No reasons are given for the cost variations. Financing: The first IBRD Loan of US$ 7.00 million increased to US$8.05 million due to exchange rate variations of Special Drawing Rights against the US$ of which US$7.66 million was disbursed. For similar reasons the Additional Financing IBRD Loan of US$ 10.00 million increased to US$10.03 million of which US$6.79 million was disbursed; the undisbursed balance was cancelled. Borrower Contribution: At appraisal the Borrower agreed to finance US$1.05 million and agreed to a further US$1.05 million at the time of Additional Financing. Only half this total amount of US$2.1 million was provided. Dates: The Mid-term Review took place in November 6, 2007 at which time the original closing date was December 31, 2009. It was subsequently extended three times in order to complete ongoing, and additional planned activities under the AF, for a total of 5 years and 4 months; it finally closed April 29, 2015. In October 2014 the Borrower made a request to restructure the project and further extend the closing date to October 2016 and this request was rejected by the Bank. In all there here were 5 restructurings: • October 2008 to incorporate 2 changes to the Loan Agreement (procurement of consultants and a changes to activities under component C); • November 2009 to extend project closing by one year to December 31, 2010; • December 2010 to extend the project by 10 months to October 31, 2011 to allow reallocation of Loan proceeds among components and give additional time to process the AF because of change in the administration; • December 2011 by 6 months to April 30, 2012 to allow additional time to achieve project objectives and process the AF request; and Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) • February 2012 at which time the approval of the AF extended the closing date by three years to April 29, 2015. 3. Relevance of Objectives & Design a. Relevance of Objectives While the statement of objectives in the Loan Agreement is very general and could be universally relevant in almost any country, the four specific sub-objectives stated in the Project Appraisal Document and the Project Paper for Additional Financing were and remain substantially relevant. As estimated by the 2005 Country Environmental Analysis (CEA) the costs of environmental degradation amounted to 3.7% of Colombia’s GDP. Urban air pollution, and inadequate water, sanitation and hygiene were identified as priority areas, associated with the most significant costs that compromised Colombia’s potential for sustainable economic growth. Clearly, improving environmental policies and capacity of national institutions to address these problems was relevant Colombia is highly urbanized with 72% of its population living in cities, half of which have more than 100,000 inhabitants. At the time of appraisal about 9% of the population did not have access to an improved water source, and 20% of urban centers lacked sewerage, leading to crowded and often unsanitary conditions, and the costs associated with intestinal morbidity from contaminated water and inadequate hygiene in both children and adults was high. Air pollution was and is one of the most widespread and serious problems in Colombia’s cities and industrial corridors. Outdoor air pollution, especially fine particulate matter such as sulfur in transport fuels, was and is an important cause of respiratory illness and deaths from pneumonia. The 2006 CEA found that every year 6,000 people died in Colombia prematurely (nearly half of them in Bogotá); and there were 7,400 new cases of bronchitis as a result of urban air pollution. Another 1,100 fatalities each year were associated with exposure to indoor air pollution, largely affecting women and children, and resulting from the use of fuel wood, charcoal, and other solid fuels as primary sources of cooking fuels. High vulnerability to natural disasters and degradation of water resources—scarcity in hotspot areas with arid conditions and pollution from urban and rural wastewater and agricultural sources—had also been identified as priority environmental action areas. The subsequent CEA of 2012 and Advisory and Analytical Analysis of 2014 indicated that the specific objectives remained relevant at project closing. In 2012 the total costs of urban air pollution, indoor air pollution, and inadequate water supply, sanitation and hygiene remained unchanged at about 2% of GDP. The 2014 AAA found air pollution accounted for 52% of environment-related deaths and 0.95% of GDP. While the government introduced broad-based reforms and policies to mainstream environmental management and integrate principles of sustainable development in key sectors, in line with its commitment to meet Millennium Development Goal 7 (to integrate the principles of sustainable development into national policies and programs, and reverse the loss of environmental resources), the 2002-2006 National Development Plan (NDP) did not include a specific chapter on the environment. This was rectified in the 2006-2010 NDP that included the environment and incorporated the findings and recommendations of the CEA. The 2012-2014 NDP had three main objectives: (1) Sustainable Growth and Competitiveness, (2) Equality of Opportunities for Social Prosperity, and (3) Consolidation of Peace. There were four cross-cutting themes: (a) relevance of international relations, (b) environmental and disaster risk management, (c) good governance, and (d) regional development and integration. Specific project objectives are relevant to the Country Partnership Strategy FY12-16 that notes while environmental degradation and adverse climatic events continues to affect public health and welfare, and compromises the country’s potential for economic growth, its support is arranged around three strategic themes: (a) Expanding Opportunities for Social Prosperity; (b) Sustainable Growth with Enhanced Climate Change Resilience; and (c) Inclusive Growth with Enhanced Productivity. Environmental risks were given a priority because extraordinary rainfall that plagued the country during 2010 and 2011 highlighted the country‘s continued vulnerability to an array of disaster risks and the likelihood that adaptation to climatic events could become a priority concern. Rating Substantial b. Relevance of Design Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) The general results framework covers the overall and specific sub-objectives. The project actions are linked to the DPL Program identified in the PAD (pages 26-30) that would enable the incorporation of environmental and social considerations into policies and programs of the water, transport, health and energy sectors. The expected outcome was ‘improved functioning of SINA to set priorities related to sustainable development and key results of the 2nd and 3rd phases of the DPL.” There is only general definition of what the “improved functions” would be and the “key results” expected of the DPL Program. This lack of a detailed results chain also applied to sub-objectives. For example. the urban development component included "improved urban living and management of urban development” but gave no detail of what the inputs were, who was to do it, what exactly was to be produced. Integrated water resources management was that the “Water Unit adapted to support implementation of the Water law" - there is no definition of what "adapted" means. Within the planning and oversight of environmental management sub-objective, one outcome was “accomplishment of at least three of six of [Colombia's] sustainable development objectives”; the six objectives had a tremendously wide scope and design seemed to be of the 'pick-an-choose' variety with no criteria to focus inputs or their scope. Generally, the range of expected outcomes was very large given the relatively small resources contributed by the project and some are very clearly over-ambitious in the four years planned. Thus it was expected that the project‘s support for the development of an air quality monitoring protocol, as well the development of software to transfer and monitor air quality data collected by municipal environmental authorities, would be used to generate a national, unified report on air quality in Colombian cities and support the analysis of information collected. It was also expected that the information collected by regional and municipal authorities would measure those pollutants that are most harmful to human health and be used to help increase understanding of the pollutants’ impacts on the health of the population. Similarly, while the definition of the inputs to the hand-washing program were missing, there was to be an evaluation to assess whether appropriate messages and mediums were being utilized to increase awareness of the health and the behavioral changes generated by the project. At the time of Additional Financing the scope of the project design was reduced through elimination of the urban development component, dropping of the hand-washing component, and a very substantial scaling-up of the integrated water resources management. The environmental health component focused almost entirely on the building of an emissions testing laboratory for vehicles and motor cycles and dropped any links to human health or efforts to better manage urban pollution more generally, including disposal of solid waste. It appears that project activities on which few or no advances were made under the original project were eliminated. As in the original project, the links with the overall objective are poor. In contrast, the integrated water resources management component became much more detailed and also, at the same time, more ambitious including for example the development of guides and manuals for environmental zoning, the definition of ecological flows, and discharge monitoring protocols for watershed management planning. Again, as with the original design, the range of detailed activities was huge for the three years during which they were expected to be achieved. A similarly wide and ambitious series of project activities supported the planning and oversight of environmental management, a task made more difficult by radical reorganization of the government’s environmental and planning ministries/agencies. While all the activities covered were relevant to the overall objective, the redesign was effectively a new project. But like the original project, the results chain and outcomes expected were similarly vague. Rating Modest 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective “To support the Borrower in the development and implementation of policy reforms and related investments in line with the Program’s framework, particularly those polices and investments that address environmental problems affecting the quality of life and well-being of the Borrower’s population. In particular, the project will support (i) the incorporation of environmental considerations in the policies of the sectors that result in a high cost of environmental degradation (e.g., health, transport, and water); (ii) the implementation of policy measures for integrated water resource management; (iii) the optimization of urban development policy instruments to improve urban quality of life; and (iv) the strengthening of the planning, monitoring, and oversight of national, regional and local environmental management and policies.” Rationale The overall objective had no specific outcomes defined, thus this Review assesses the outcomes under the four specific sub-objectives. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) There is a major problem also with attributing any outcomes because most of the policies, regulations and other improved institutions developed and supported by the project were also the subject of the DPL Program. Additionally, almost all the project achievements described in the ICR are ad hoc outputs and any outcomes are difficult to trace directly to project inputs. (i) The incorporation of environmental considerations in the policies of the sectors that result in a high cost of environmental degradation Outputs: Environmental health was supposed to be improved by improving air quality, particularly in cities, increasing household and individual hygiene through national hand-washing program, and the closure of municipal solid waste dumps. • The project provided 21 air-quality monitoring stations that are at varying levels of operation. This included air quality monitoring equipment to measure particulate and gaseous matter. The Institute of Meteorology and Environment Studies received the most equipment, but the equipment provided by the project is out of operation due to budget constraints. Reports were prepared on 20 stations in 2008 and 14 stations in 2012. No air quality reports have been produced since 2012. • The equipment to measure pollutants for road vehicles at the Vehicle Homologation Center was extensively over-designed and was not procured by project completion. • The Strategic Environmental Assessment on fuel quality was not conducted. Instead an environmental assessment of policies, plans and programs for biofuels in Columbia, with an emphasis on biodiversity, was completed in 2008. Even so, several studies were conducted: on characterization of particulate in several Colombian cities; draft reports for the development of emission inventory controls; a study to characterize meteorological conditions and better understand their impact on pollutant dispersion; and a protocol for air pollution control from fixed sources. However, the information used was often incomplete or out of date. In addition, according to the ICR the reports did not focus on the most relevant criteria/ particulate matters. • A SUISA (Sistema Unificado de Información sobre Salud Ambiental) was designed. • The hand-washing program was implemented in 13 departments. A mass media campaign (Public-Private Partnership) was designed and launched. The program was discontinued in 2012. The hand-washing program was confined mainly to dissemination of messages and there were no surveys on how hand-washing and hygiene messages were accepted or adopted by individuals and households. There is no measurement of the share of the population benefitting from this program and what the benefits were although the ICR states that 10.2 million people heard the messages compared with 9.6 million people targeted. • Solid waste management outputs were inconclusive. 122 municipalities were given technical assistance to improve their solid waste disposal compared with the 200 targeted. The ICR reports that 286 dump-sites were closed by December 2008 but states it is very unclear how these closures related to the 122 municipalities that received project assistance. • The mid-term review indicates that the project supported the preparation of some Decrees that included references to human settlements in urban areas at risk. The ICR reports, however, that it can trace no linkage to project activities. Outcomes: It is difficult to determine how much the project contributed to incorporation of environmental policies in the ill-defined sectors the project was supposed to support. Most of the references in the ICR are related to outputs produced before 2009 that were produced at the end of the 3rd DPL as reported in its ICR dated 2012. Columbia has a wealth of environmental legislation and the project could have had some impact on this - but there is no evidence to indicate that it did. Rating Modest PHREVISEDTBL Objective 2 Objective (ii) The implementation of policy measures for integrated water resources management. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) Rationale Outputs: 66 water quality laboratories were accredited for 19 procedures, exceeding the target of 5. None of the 14 hydrological stations planned were procured and it is uncertain how water resources monitoring and information systems were strengthened and improved. No other outputs attributable to the project can be traced. One pilot project on aquifer management and legalization of water use was implemented (target 3), and one pilot on flood risk management was developed. Eight consultants were hired under the IWRM Directorate to support the implementation of the National Water Resources Policy and the project. Training was provided to personnel; the program format allows for virtual and on-campus presentations in four cities (Bogota, Medellin, Barranquilla, Cali). 10 regional workshops on regional water resources management were held. Specialized training by consultants was given through an integrated water resources management diploma to 394 individuals. Diagnostic work on some ground water modeling was conducted, but there are no discernible policy outcomes that can be linked to this work. Integrated Participatory River Basin/ Aquifer Planning and Management: Diagnostic work on the aquifer of Sabana de Bogotá (including modeling and definition of critical areas) and a pilot on legalization of water use rights in the Guali river basin was developed; there is no information on their implementation. A participatory process involving the IDEAM and CARs and other stakeholders reportedly developed several guidelines, including guides for environmental zoning, discharge monitoring protocols for data collection of water users, and other sub-processes. The Aquifer Management Plan for the Sabana de Bogotá and the pilot in Chinchina river basin as well as the participatory and socialization mechanisms were undertaken. Management of Climate Risks in the Water Resources Sector: preparation of a methodology and guidelines for incorporating natural risks in the formulation of POMCAs (Anexo Gestión de Riesgo, MADS 2013) was undertaken. With respect to flood risk, an approach was defined and a pilot was conducted for Barrancabermeja with detailed flood maps and risk maps but there is no information of their application. Outcomes: The project was supposed to support the implementation of a national Water Law. Subsequently due to delays in Congress, the loan agreement was amended in 2008 to refer to water resource management reforms instead of the Water Law. According to the government, the original loan supported the preparation and adoption of the National Policy for Integrated Water Resources (2010) and three decrees covering the creation of water resources information systems; creation of water use registry; and prioritization of some watersheds for planning management and intervention. The ICR reports that it could find no evidence linking project activities to these achievements. Rating Modest PHREVISEDTBL Objective 3 Objective iii) The optimization of urban development policy instruments to improve urban quality of life. Rationale Outputs: The project was supposed to develop four regulations to support the urban development law and in particular to prevent human settlements in areas at risk. There are no clearly defined outputs from the large volume of technical assistance that aimed to support sustainable urban development. • An unspecified number of diagnostic reports, guidelines and instruments for improving coordination between national policies for urban development, housing, drinking water and basic sanitation is reported in the ICR. Topographic surveys were completed in 6 districts covering 62 hectares. There were 15 regional workshops. A pilot-project on improved public space was designed for Buenaventura and a manual for standards in urban planning were produced. However, the policy instruments and regulations that had been targeted, were not issued. • A document on integrated management of hazardous wastes was approved, and a consultancy to produce a guide on the regionalization of landfills was contracted - but the final report, according to the ICR, cannot be found. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) Outcomes: • None can be traced to the project according to the ICR. Rating Negligible PHREVISEDTBL Objective 4 Objective (iv) The strengthening of the planning, monitoring, and oversight of national, regional and local environmental management and policies Rationale Outputs: According to the ICR, there is no evidence that the outputs anticipated were achieved by the project: (a) consolidation of methodologies and procedures for evaluation of environmental policies; (b) preparation of sustainable development reports (such as the reporting on the attainment of the MDGs); (c) adjustment of the electronic reporting systems of sustainability by CARs in accordance with updated regulations; and (d) development of information systems and provision of training in support of better monitoring and reporting of sustainability indicators by CARs and by environmental research institutes. The MADs coordinated a workshop on the SIUR (Sistema de Información de Uso de Recursos), which included developing indicators of natural resources use by the manufacturing sector. The project supported the reengineering of the information subsystem for several sub-national entities - but specific outputs are not described in the ICR. Outcomes There are no evidence attributable to the project that the planning, monitoring, and oversight of national, regional and local environmental management and policies were strengthened. Rating Modest 5. Efficiency Economic and Financial Efficiency No economic or financial analysis was conducted during the preparation of the parent project or Additional Finance loan. Economic justification of the project was based primarily on mitigating the high costs of environmental degradation. The project’s main benefits were to be derived from the environmental health component, specifically the activities to improve household and personal hygiene through hand-washing at critical times in the daily cycle. Significant benefits were also to come from reduction of air and water pollution. Other benefits, such as improved urban planning and better environmental policies and management more generally, were not subject to economic analysis. There was no evidence-based analysis of economic or financial efficiency at project completion. Institutional and Administrative Efficiency Many extensions were necessary and it took almost 7 years to complete the original 4-year project; however the AF extension was closed as scheduled. Even so, at closing, there was an undisbursed amount of US$ 3.24 million and many of the AF components were either not implemented or completed. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated substantial. Relevance of design is rated modest. Three of the four sub-objectives are rated modest and one objective is rated negligible. Efficiency is rated modest. Thus there were major shortcomings in the project’s design, achievement of its objectives and efficiency, and outcome is rated unsatisfactory. a. Outcome Rating Unsatisfactory 7. Rationale for Risk to Development Outcome Rating • As it is unclear what the project outcomes actually were, it is very difficult to specifically identify the risk to them. However, a principal risk is continued uneven government financing of the environmental sector and the governance challenges related to natural resources utilization. • A major risk, based on the project experience, is that mechanisms to ensure institutional accountability for environmental management are weak. In particular, MADS is struggling with how to engage in public-private partnerships aimed at supporting and monitoring compliance with the many environmental policies and regulations promoted by the DPL. Given these challenges, there is risk that the MADS may focus on non-controversial biodiversity and similar problems and not those of environmentally sustainable urbanization and licensing and regulation of natural resource utilization. • Financing of institutional capacity that may have been assisted by the project in MADs and elsewhere is very uneven and below the levels required to maintain sustainability. For example, IDEAM’s budget for air quality monitoring and regulation has been cut-back, so that staff has been lost and project equipment sits unused in storage. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The project was appraised as a typical small technical assistance and investment Loan to support the DPL program. It was appraised on a very small budget and in just over five weeks. While the design was linked to the DPL policy framework, key studies to inform the selection Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) and design of investment projects, or the methodologies to evaluate them, were not completed. There was no economic appraisal of the investments envisaged. Costs of the inputs expected to produce intermediate and final outcomes were woefully under-estimated. The results chain was very weakly defined and project activities were scattered across several sectors and agencies. Although many national and regional institutions were to be involved in project implementation, there were only general proposals for inter-sectoral coordination. Key stakeholders below the national level, while supposedly being also recipients of the technical assistance, were not included in the consultation process. And even at the national level, appraisal did not make very clear institutional responsibilities for key project deliverables, and how they related to intended outcomes. Because of the weak appraisal of institutions, the priorities for capacity building were not identified and this was particularly a problem below the national level. It is clear also that the appraisal team made unrealistic assessment of the political environment because the approval of the Water Law was effectively stymied in Congress and the financing agreement had to be subsequently modified to reflect this change. The original loan did not trigger any significant environmental social safeguards, and while fiduciary and procurement risks appeared to be adequately mitigated, there were subsequent problems with procurement. Attention to monitoring and evaluation arrangements was totally inadequate. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision Overall, supervision of the project had several significant shortcomings and these were increased by new and additional activities at the time of AF. The rationale for the AF is not clear and no account was taken of the inadequate progress on expected outputs and the only criteria seems to have been disbursement of the 81% original loan by mid-2008. Even then, it took four years to get the AF approved and project activities effectively stalled. At the time of AF key project activities (e.g. hand-washing) were dropped or changed (e.g. air quality monitoring and regulation) with no explanation or any idea of the results produced. The task team did not include sustained and adequate technical expertise to supervise specialized topics, a task made more difficult by the ad hoc and new activities, the many institutions involved and the lack of precise indicators. Only two of the 21 supervision missions visited the field where many of the technical support activities were located. Poor terms of reference for the few specialists on the project contributed to project inefficiencies, as did lack of knowledge about institutional mandates and responsibility. Thus, for example, the Vehicle Homologation Center was overdesigned and the estimated costs were 13 times the allocated budget of US$1 million and it could not be implemented. Implementation issues were not brought to management’s attention in a timely manner, reporting on progress toward achievement of objectives was not candid, and a number of reports from supervision were not filed. Lack of realism contributed to the delay in downgrading the development outcome rating of the project. The Mid-Term Review (MTR) under the original loan was not used for the purpose of identifying or mitigating risks to the achievement of project objectives or making course corrections in design or implementation. At the time of AF the project environmental rating was changed appropriately from category ‘C’ to ‘B’. No MTR was carried out under the AF. The Bank’s performance both on substantive and procedural aspects is considered (ICR page 34) to have been a key limiting factor to the performance of the Borrower. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance While the government’s commitment to addressing priority environmental challenges had been demonstrated by the broad-based reforms to mainstream environment in national policy consistent with its commitment to meet Millennium MDG 7, subsequent ownership was uneven. The weaknesses resulting from the 2003 merger of most of the functions of the Ministry of Economic Development (housing, water, sanitation, and zoning) with the Ministry of Environment to create the MAVDT (Ministerio de Ambiente, Vivienda y Desarrollo Territorial) were not corrected. Most of the institutional capacity was reassigned to housing and water supply topics. Coupled with this the environment unit in the DNP (Departamento Nacional de Planificación), which was a key factor in design and implementation of environmental policy in Colombia, was dismantled. Subsequently, the MAVDT was superseded by the Ministry of Environment and Sustainable Development (MADS) with another reshuffling of environmental responsibilities and management. Government commitment to Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) the sub-objectives of the project waned as demonstrated by lack of adequate budgets for environmental pollution monitoring, analysis and regulation. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The set-up for implementing the project was very complex and implementing agency (IA) performance was extremely difficult to measure.The former MAVDT (Ministerio de Ambiente, Vivienda y Desarrollo Territorial), through its Vice Ministry of Environment, Vice Ministry of Housing and Regional Development, and Directorate of Planning, Information, and Regional Coordination, supported by its Multilateral Bank Group, had overall responsibility for implementing the project under the first loan that was spread over several co- implementing institutions and the Ministry of Social Protection (Ministerio de Protección Social). Under the AF approved in 2012, overall responsibility for project implementation was assumed by the newly-created MADS (Ministerio de Ambiente y Desarrollo Sostenible) through its Vice Ministry of Environment and Sustainable Development and its Multilateral Bank Group. In addition, IDEAM (Instituto de Hidrología, Meteorología y Estudios Ambientales de Colombia) had responsibility for the subcomponents on water resources monitoring and information systems and the Single Environmental Registry (Registro Unico Ambiental, RUA). The DNP (Departamento Nacional de Planeación) had responsibility for the subcomponent on its institutional strengthening. Actual performance of the IAs was hindered by institutional reorganization and variable capacity, unclear and changing mandates of what the project institutions were supposed to achieve, and lack of adequate monitoring and evaluation systems. Inter-sectoral coordination by MADS and arrangements for consolidated reporting on the project were inadequate. Follow-up on project activities was generally poor and some reports of consultancies commissioned by the IAs are missing (e.g., the consultancy output on the disposal of solid waste is unaccounted-for). Procurement was at times problematic as illustrated when equipment for air quality monitoring failing contract specifications was not returned. Planning could have been improved and inefficiencies caused many delays. Many of the more tangible outputs were dropped, not achieved or were untraceable. In spite of several extensions of closing date and restructurings, the project closed with a sizeable undisbursed amount. Implementing Agency Performance Rating Unsatisfactory Overall Borrower Performance Rating Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The results framework was very complex and very poorly designed, as was the M&E system to support it that spanned many institutions. Design of the indicators was faulty because the specific outcomes were unclear. There was no consistent chain linking the inputs and outputs to specific outcomes. The results framework was incomplete and baselines were generally obscure or absent. In many cases a single indicator (e.g., the number of regulations) was used to measure multiple parameters that should have had separate indicators. b. M&E Implementation This was very uneven over time and fragmentary in coverage of projects inputs, outputs and outcomes. c. M&E Utilization The information provided was of limited use as a tool to monitor progress toward the achievement of project objectives, and even Bank records of the progress reports are incomplete. It was not possible to use this information to fully identify implementation problems, find Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) solutions or introduce mitigating actions. M&E Quality Rating Negligible 11. Other Issues a. Safeguards The project was classified Category ‘C.’ under OP/BP 4.01 Environmental Assessment. Subsequently at AF the project classification was uprated to Category ‘B’ on the precautionary principle and OP/BP4.04 Natural Habitats was triggered. An Environmental and Social Management Framework, including public participatory processes and mechanisms, was prepared to identify prevention and mitigation measures to enable compliance with safeguards. Overall, the ICR (page 16) reports that the project complied with the environmental safeguards that were triggered. b. Fiduciary Compliance Financial. From 2006 to 2009, the project was audited by the Contraloría General de la Republica and from 2010 to the end of the project by a private audit firm. The financial statements received a clean unqualified opinion, except in 2007, when they received a negative opinion, and in 2008 when they received a qualified opinion. Procurement. Inconsistencies were detected in the compliance of some procurement and planning procedures in the selection processes: (a) procurement planning was misaligned with the project time-table; (b) the development of technical specifications or terms of reference or expressions of interest were not prepared in a timely fashion; (c) limited technical oversight; (d) weaknesses in the administration of contracts; (e) weaknesses in the reports on short lists, financial proposal evaluations, and negotiation meetings; and (f) in a few cases, the project choose the selection method solely based on the method’s short duration and not on meeting the assignment’s technical demands as should have been the case. According to the ICR (page 17) Bank supervision missions made recommendations for corrective actions after which procurement management was strengthened, and generally procurement performance was rated moderately satisfactory c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory --- Risk to Development Outcome Substantial Substantial --- Bank Performance Unsatisfactory Unsatisfactory --- Inadequate accounting for use of Borrower Performance Moderately Unsatisfactory Unsatisfactory project inputs, poor records of outputs and almost no Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CO Sustainable Development Inv Project(P082520) discernment of outcomes attributable to project investments and inputs. Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons This ICR presents 11 lessons, most of which are covered by the Bank’s Operational Policies. The most important is summarized and edited below: • Investment projects supporting a DPL need to have well-defined and traceable outcomes. When a project has a very broad objective and difficulty in defining specific outcomes, care has to taken that the causal chain between inputs, outputs and outcomes/ impacts are clear and monitorable. This is a particular problem with relatively small projects developed to support a DPL program. • A good ICR cannot be written if the Bank does not keep thorough and realsitic records of what was happening and holding the Borrower to account. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is comprehensive and covers much of the detail of a long and complex operation. In addition, the ICR has sought to build sufficient evidence through analysis to make accurate judgments with regards to the deficiencies of the project. At 40 pages (excluding annexes) it is overly repetitive, particularly with regards to details on activity level information. Although candid, the project ratings do not always follow from the evidence presented excepting those of the Bank and Borrower. This makes it difficult to link the narrative performance story to the associated ratings. In addition, the report would benefit from a more concise and consistent results orientation with greater focus on how the collection of outputs sought to contribute towards higher level outcomes. a. Quality of ICR Rating Substantial