ICRR 14921 Report Number : ICRR14921 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/30/2016 Country : Colombia Project ID : P082429 Appraisal Actual Project Name : Natural Disaster US$M): Project Costs (US$ 551.3 102.8 Vulnerability Reduction - First Phase APL L/C Number : L7293 Loan/ US$M): Loan /Credit (US$ 260.0 102.8 Sector Board : Urban Development US$ Cofinancing (US $M): Cofinanciers : Board Approval Date : 05/10/2005 Closing Date : 06/30/2010 12/15/2013 Sector (s): Flood protection (35%); General public administration sector (23%); Roads and highways (20%); General agriculture fishing and forestry sector (16%); General education sector (6%) Theme (s): Natural disaster management (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Roy Gilbert Christopher David Christopher David IEGPS1 Nelson Nelson 2. Project Objectives and Components: a. Objectives: "To assist the Borrower in the strengthening of its national capacity for reducing the fiscal vulnerability to national disasters and mitigate the negative impact of possible effects derived from such disasters ." (Loan Agreement - LA Schedule 2 p. 20) As per the standard IEG approach, the present review is based upon the assessment of the achievement of the project objective as formulated in the LA . The ICR did the same thing. The Project Appraisal Document (PAD) presented a somewhat different formulation of the project objective, namely: "To reduce the fiscal vulnerability of the state to adverse natural events by strengthening national capacity to manage disaster risk and by reducing vulnerability in key municipalities which combine high exposure to disaster risk and high contributions to national income and productivity." (PAD p. 9). While the substance of the PAD version of the objective is mostly the same as that of the LA, its content is presented in a different order. One substantive difference, however, is the PAD version's call for attention only to "key municipalities" for receiving disaster risk management (DRM) assistance. By following the LA version of the objective, the present review therefore takes a broader view, as did the ICR, that project DRM assistance intended going beyond just "key municipalities". b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets ? No c. Components: (original components listing compiled from LA Schedule 2 and PAD pp. 13-16) Part A : Disaster Risk Identification (appraisal cost US$6.2 million; actual cost US$22.0 million) including: (A.1.) development of a comprehensive information system on disaster vulnerability , risk evaluation and risk reduction programs, including scientific, geographic and technical data , maps, reports on mitigation programs and activities, and local best practices, through the purchase and installation of hardware and software and other necessary equipment, as well as the provision of specialized training on disaster prevention and management of disaster emergencies; (A.2) strengthening data quality and analytic capacity for early warning and risk mapping related to hydrological , seismic and volcanic events , through the provision of technical assistance , necessary equipment and installation works needed for the updating of existing systems for monitoring catastrophic events . Part B : Risk Reduction (appraisal cost *US$242.4 million;; actual cost US$82.0 million) - including: (B.1.) Improvement of the current practices of territorial planning by including better risk assessments , risk analyses, and preventive planning, through the provision of technical assistance to municipalities to guide risk reduction activities in municipal investment plans , preparation of studies, and the provision of training and workshops ; (B.2.) carrying out of risk reduction investments by territorial entities , infrastructure agencies such as Colombia 's National Roads Institute (INVIAS), and the Colombian Petroleum Company (ECOPETROL) in: (a) reforestation and management of micro-watersheds; (b) flood defense measures; (c) flood plain management—irrigation, drainage, and land recovery; (d) mitigation and prevention measures in urban and rural areas identified as high disaster risk ; (e) provision of fire-fighting and disaster response equipment ; (f) disaster prevention; (g) disaster response; (B.3.) strengthening municipalities’ hazard risk reduction and management capabilities , through the provision of technical assistance, training and workshops, and the establishment and operation of an information sharing system to link Municipalities, municipal institutions and the national agencies as members of Colombia 's National System of Disaster Response and Prevention (SNPAD). This subcomponent will support monitoring and evaluation of disaster risk reduction investments, using the findings in the construction of a system of disaster vulnerability indicators to measure impact of investments in the development of guidelines for design of hazard risk reduction measures , and in the identification and promotion of best practices . * The PAD project cost table (p. 50) gives the figure of US$102.4 million as the appraisal cost of this component--the value reproduced in the ICR (p. 23). This does not include US$140.0 million of "country financing" of Part B2,, an amount that is correctly counted as part of the US$550 million total appraisal cost of the project reported by the Pad cost table. Part C : Institutional Development (appraisal cost US$ 0.8 million; actual cost US$4.0 million) - including: (C.1.) strengthening the institutional capacity of SNPAD , through the provision of technical assistance , specialized training and workshops leading to an institutional analysis of SNPAD 's strengths and weakness , new areas of SNPAD work, an organizational plan for SNPAD, and a policy framework defining the role and the limits of the State in responding to natural disaster risk emergencies (that is, the level of risks it will assume.); (C.2.) strengthening the Borrower’s national capacity to coordinate , implement, monitor and evaluate risk reduction , through the provision of technical assistance , necessary goods and specialized training covering financial management and administration, and technical management and reporting . Part D : Awareness and Preparedness (appraisal cost US$ 0.2 million; actual cost US$0.7 million) including: (D.1.) development and integration of disaster risk awareness activities as part of MAVDT 's the environmental education program covering: (i) environmental education for risk awareness ; (ii) emergency preparedness; (iii) and emergency and contingency planning . LA ) or Risk Financing (in PAD ) (appraisal cost *US$300.6 million; actual cost US$0.9 Part E : Risk Transfer (in LA) million) including: (E.1.) strengthening the Borrower’s capacity to define strategies that improve and optimize the financial coverage of its risks, through the provision of technical assistance and preparation of studies to assist project implementing agencies (PIUs) in the evaluation of risk exposure and the formulation and implementation of cost -effective risk transfer arrangements; (E.2.) financing of critical imports required due to a disaster as identified in a recovery plan for up to two month prior to and six months following the declaration of a national disaster emergency . * For this component cost the PAD project cost table (p. 50) gives the figure of US$150.3 million, the value reproduced in the ICR (p. 23). This is not correct as it does not include US$150.0 million of "country financing" of Part E2, disaster emergency assistance. The same PAD project cost table nevertheless correctly counts the US$150.0 million into its US$550.0 million total appraisal cost. (principal revisions to components ) Part E.2 was cancelled as part of the May 2009 restructuring, as was US$300 million of the APL's financing--US$150 million from the Bank Loan plus US$150 million "country financing". It was replaced by a Development Policy Loan with a Catastrophe Deferred Drawdown Option (DPL Cat DDO - P113084) financed by a Bank Loan in an amount of US$150 million. A DPL Cat DDO, a new Bank financing instrument had not been available at the time of the appraisal of the APL in 2006, became the preferred option of both the Bank and the Government to provide more ready access to Bank financial assistance .. With the 2009 restructuring, Part B.2., carrying out risk mitigation investments through the national revenue sharing program funding of US$61 million, was cancelled. One reason was that the Government lacked a suitable mechanism to oversee investments such as these carried out by municipalities . Part B.2. also introduced a new subcomponent of disaster risk reduction investments called "lifeline infrastructure" that was to use the US$16.13 million reallocated from Category 2 (risk reduction) to Category 1 (operating cost) expenditures. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost The total project appraisal cost as reported by the PAD cost table (p. 50) was US$551.3 million (Bank - US$260 million; Borrower - US$290 million). By not including the US$290 million government counterpart commitments , the ICR incorrectly reports the total appraisal costs of the project as being only US $260 million (ICR p. 23). Regarding the actual project costs at completion , ICR reporting is imprecise. The total actual costs incurred by completion , reported as US$110 million by the ICR, cannot exceed the total funding of the project financed exclusively by the Bank Loan whose total disbursements were US $102.8 million according to the Bank's Client Connection database. At completion, the undisbursed balance of U $7.2 million remaining in the Loan account was cancelled at closing . While the correct appraisal costs can be deduced from the PAD , the figures reported by the ICR raise doubts about the accuracy of the actual project costs down to the level of components , since, together, these add up to a total that is incorrect and unfeasible given the amount of financing available . Financing The Bank's originally committed Loan of US$260 million would have financed 47.2% of the cost of the project as estimated at appraisal. An amount of US$150 million of this Loan was cancelled as part of the main project restructuring in 2009, and, informally, its proceeds used to finance the US $150 DPL cat DDO approved that year. As mentioned above, US$7.2 million of the remaining undisbursed balance of the APL 1 Loan was cancelled at project closing. Thus, a total of US$102.8 million was actually disbursed against this project 39.5% of the amount originally committed. As a result of these changes , by completion a smaller Bank loan financed 100% of a much smaller project, instead of a larger Loan financing 47.2% of a much larger project, as originally intended. Borrower contribution By completion, the Borrower made no counterpart contribution to the financing of the project , although the appraisal had pledged US$290 million, that would have funded 52.7% of the expected total costs . The Borrower's withdrawal from financing this operation was through project restructuring agreed by both Bank and the Borrower . The 2009 restructuring cancelled most of the Part B 2 component and all of Part E2 that would have benefited from this counterpart funding, US$140 million and US$150 million respectively. Dates For this review and In keeping with the norm of prioritizing the legal agreement over the PAD , IEG uses the June 30 2010 closing date reported by the Loan Agreement (LA p. 21). This is different from the December 31, 2011 date reported by the PAD (PAD p. 1) and used by the ICR. The LA date indicates that the project closed three and a half years behind schedule. On five separate occasions the project was restructured at the following times : 1. 05/21/2009 - cancellation of the project's US$150 million contingent credit that was informally reallocated to the Colombia Disaster Risk Management DPL (P113084) approved in an amount of US$150 million in 12/08/2008 and closed 01/31/2012. 2. 03/12/2010 - reallocation of US$5.5 million to strengthen technical assistance (TA) to municipalities and the seismic and volcanologic monitoring network at request of Government and propose changes to the project 's results framework. 3. 12//13/2011 - reallocation of US$16.13 million from disbursement category 2 (risk reduction) to disbursement category 1 (operating costs), extend closing date by 20 months, and propose changes to indicators and targets in the project results framework. 4. 05/10/2012 - re-issuing of revised amendment letter covering the 2011 restructuring, since the original copy had not been countersigned by the government . 5. 07/11/2013 - extension of closing date by 3.5 months to 12/15/2013. 3. Relevance of Objectives & Design: a. Relevance of Objectives: IEG introductory note on the project objectives Strengthening national capacity is the starting point for this project 's objective, but it is not clear exactly what the stronger national capacity is supposed to achieve . The objective formulation itself refers to reducing the fiscal vulnerabilities of the state to national disasters and mitigating the negative impact of possible effects from such disasters. To better understand these (two) intended results, IEG looked for the operational concepts of them used by the project, as well as baseline values of their situation before the project , appraisal targets and resulting endline values at completion. The scarcity of such information in the project documentation makes an assessment 's identification of positive achievements by this operation very difficult . To be credible, such an assessment should robustly demonstrate measurable progress in reduced fiscal vulnerabilities and risk mitigation attributable to project interventions to strengthen national capacity . While not defining fiscal vulnerability or offering measurements of it, the PAD notes that the idea of it originated in a key policy/strategy document on disaster risk management (Conpres 3146 of 2001) by Colombia's National Planning Department (NPD), a key government partner of the present APL 1 project. IEG was unable to find clarification or even references to fiscal vulnerability in that document, however. Risk mitigation is more widely, even if not precisely referenced, but still not measured. For this review, the project team clarified to IEG that measurements , such as those of fiscal vulnerability, disaster risk and risk mitigation, are still work in progress. Other evidence corroborates this. The ICR of the DPL Cat DDO (P113084) associated with this APL1, claims that the definition of a framework for contingent financing for disasters was achieved in 2011 as means toward reducing fiscal vulnerability but a risk financing strategy still had to be developed (ICR DPO Cat DDO pp. 11 and 20). The question for IEG's review, therefore, is how an operation appraised in the year 2005 proposed to reduce the fiscal vulnerability of the state, a condition that still had not been measured six years later ? For the present review, however, IEG interprets the key concepts of the project objective as follows : reducing fiscal vulnerabilities of the state to national (*) disasters - means smaller financial liabilities of the national government to pay for post disaster recovery ; - ex post results in reduction could be measured by a time series of amounts actually paid out year -by-year. (*) IEG assumes that the PAD reference to this means natural disasters. mitigating risks - means lowering the costs incurred through the impacts of disasters ; - ex post results in mitigation could also be measured by a time series of the actual costs incurred year -by-year. - resumption of : a. Relevance of Objectives : rated : Substantial The project objectives are consistent , with the priorities of Colombia's current National Development Plan entitled Prosperity for All , one of whose four cross cutting themes is Environmental and Disaster Risk Management (CPS 2012-2016 p. 6). In the same vein, IEG recognizes some consistencies of the project objectives with the 2002 Country Assistance Strategy (CAS) in force at the time of this project 's appraisal. This 2002 CAS called for "efficient policies and procedures also need to be developed to anticipate , prevent and mitigate the effects of natural disasters which strike Colombia with regularity and disrupt basic infrastructure services , making as much use as possible of private initiative and insurance market instruments ." (CAS 2002 p. 12). The project objectives remain relevant to current Bank priorities laid out in the 2012-2016 Country Partnership Strategy (CPS) that specifically incorporates the project's own (output) aim of ensuring that 750-800 municipalities become DRM capable by 2014 (CPS 2012-2016 p.39). b. Relevance of Design: rated : Modest Except under its section on Bank Performance , its note of (unspecified) "drawbacks of the Results Framework " (ICR p. 17), the ICR does not have material assessing the relevance of the project design in terms of the consistency of the project activities and components with the achievement of the operation 's objectives. IEG nevertheless recognizes some consistencies in this respect . Thus Part B (Risk Reduction) and Part E (Risk Financing) that together accounted for 98.5% of the total appraisal cost were in line with at least some parts of the objectives. Part B Risk Reduction could have contributed to municipalities becoming stronger partners in disaster mitigation, possibly reducing national government financial liabilities for post disaster reconstruction . Part E Risk Financing might have helped transfer post -disaster financing responsibilities away from the national government to the private sector and subnational administrations , although the project did not make clear how this would happen . Two other components, to which only 1.1% of appraisal funding was allocated , were less relevant. These were Part A (Disaster Risk Identification) and Part D (Disaster Awareness and Preparedness ), which, by unearthing previously unknown risks, might even increase vulnerability to disasters . There is an analogy in public health campaigns that can lead to an increase of the incidence of diseases through better reporting of cases , especially of existing ones that had previously been unknown . Part C (Institutional Development) was also less relevant, but for a different reason. Its focus, SNPAD had been in operation since 1998 and become a model for Latin America (PAD p.3) was perhaps not a priority target for institutional strengthening . Since only 0.1% of planned funding at appraisal was allocated to this component, this was not a major shortcoming of the relevance of the project design . 4. Achievement of Objectives (Efficacy): "To assist the Borrower in the strengthening of its national capacity for reducing the fiscal vulnerability to national disasters and mitigate the negative impact of possible effects derived from such disasters ." The output-oriented ICR for this project does not evaluate the achievement of the operation 's objective in terms of lower national government liabilities for financing post disaster recovery and lower costs of the natural disaster impacts themselves. It does record project component deliveries , however, but without explaining how these outputs contribute to the outcomes sought . For this review, IEG assembles the ICR's output evidence and explores possible links of these outputs to the achievement of the objectives , parsed into two separate parts for the purpose of this assessment. (*) possible double counting of results as similar outputs also claimed by the DPL cat DDO (P113084) - part 1 of objective : To assist the Borrower in the strengthening of its national capacity for reducing the fiscal vulnerability to national disasters . rating - Modest Outputs (*) 824 municipalities prepared hazard risk management programs , higher than the target of 632, and up from the pre-project baseline of 210. How many of these plans were adopted and implemented was not reported by the ICR. Avoiding local disaster damage through implementing them could reduce reconstruction expenses at the local level, but the ICR does not report evidence of reduced liabilities that the project sought on behalf of the national government. (*) Policy recommendations to reduce government exposure were made in the form of policy guidelines for Financial Protection Mechanisms against disasters and a final action plan adopted by the Ministry of Finance and Public Credit (MHCP in its acronym in Spanish, and MOF, the acronym in English, as used by this review). In the absence of reports of specific actions , it is not clear how these recommendations and the action plan can reduce the government's financial liabilities for post disaster recovery . (*) According to the ICR, MOF has developed the Colombian Policy Strategy for Public Financial Management of Financial Risk. The ICR also acknowledges that the modelling and estimation of financial contingencies associated with natural disaster risk were not fully delivered (ICR p. 12). On the other hand, the ICR reports that there has been some progress with a study on catastrophic risk insurance mechanisms . The Risk Transfer/Risk Financing part E of this project , the principal component designed to achieve this objective, was not implemented. Resources for this were re-assigned to the DPL Cat DDO (P113084). Outcomes Evidence of the achievement of this outcome is lacking in the ICR . ICR evidence of outputs referred to above refers to results that may strengthen some capacity , but which may not be connected to the intended outcome of this project of reducing fiscal vulnerability and mitigating risks for which direct evidence is not available . Municipalities adopting DRM programs is not by itself an assurance of smaller national government post disaster financial liabilities . Evidence of action by the state to strengthen its own capacity to reduce fiscal vulnerability is weak . Government policy recommendations were adopted by MOF 's action plan without specifying how they would be implemented . The ICR does not report about actual implementation . Finally, the project's main instrument designated to reduce fiscal vulnerability, the part E Risk Transfer - Risk Financing component, was cancelled, and with that project financing commitments of US$300 million. While there was some contribution here in terms of other supporting functions and interventions including the understanding of risk dimensions and the identification of financial instruments that can be applied to these circumstances , there is little to illustrate just how important these steps were in dealing with the strengthening of capacity . - part 2 of objective : To assist the Borrower in the strengthening of its national capacity for mitigating the negative impact of possible effects derived from disasters . rating - Modest Outputs (*) 824 municipalities prepared hazard risk management programs , higher than the target of 632, and up from the pre-project baseline of 210. How many of these plans were adopted and implemented was not reported by the ICR. The ICR also does not provide evidence of the negative impacts of disasters avoided or mitigated through the project's capacity strengthening activities --that a time series of the actual costs incurred per year might constitute Studies were completed into the disaster risk assessments of INVIAS and ECOPETROL . SIGPAD, Colombia's National Disaster Risk Information System was set up , but did not work as an open online information management system, as originally conceived. Of the seven institutions expected to take part in SIGPAD, only one, the Institute of Hydrology Meteorology , and Environmental Studies - IDEAM, did so (ICR p. 12). (*) Against a target of 100%, only 14% of the SNPAD was integrated into the SIGPAD , meaning that only one of the seven information sharing agreements planned by SNPAD was implemented (ICR p 10). (*) For Increased disaster awareness and preparedness , the environment ministry MADS made educational materials available to municipalities, but funding was insufficient to start to apply them and sustain their use .(ICR p. 12). Only 6% of municipalities received in-depth environmental education for disaster risk awareness , against a target of 20%. Nevertheless the ICR also reports that 95% of municipalities were "educated", "trained" and made aware of hazard risk reduction , against a target of 60%. The PAD (p. 41) and ICR (p. 10) reports that municipalities themselves would be "educated" or "trained" are unusual. IEG assumes that, as is normally the case, municipal staff would instead have been educated , although no details about the trainees are provided by the ICR. National policy defining government responsibility for DRM , based upon a national risk management analysis (ICR p. 12), was launched and incorporated into Law 1523 in 2012. This Law also enshrined the sharing of DRM between the environment ministry MADS and the housing and urban planning ministry MVDT --and the risk of the duplication of DRM responsibilities between them that contributed to project implementation delays at the municipal level especially (ICR p. 7). Outcomes Most of the ICR evidence cited above is output -based. Much of it reveals significant shortcomings in implementation and delivery of these outputs . Some policy and legislative moves inspired by the project helped draw attention to DRM, but also contributed to the fragmentation of responsibilities between two ministries , whereas before the project these responsibilities had remained with just one . The ICR is short on evidence of outcomes that would point to a stronger achievement of this part of the objective . Such evidence would include a baseline measure of that year 's cost of disaster impacts, the target(s) set and an annual time series of these costs until completion . Rigorous measurements of disaster risks too would have provided valid evidence , although as IEG learned from the task team , reliable evidence of this type has only recently become available . Exogenous factors undermining project outcomes : The scale of natural disasters of flooding and landslides arising from the 2010-2011 La Niña phenomenon was the worst in Colombia for decades, and may have had two contrary effects upon the outcome of this project . First, and positively, it could have brought attention to DRM to the forefront , with favorable results for increased awareness and preparedness, DRM institutional strengthening and development of a risk financing strategy (ICR p. 7). Second, and negatively, it could have increased the cost of disaster impacts and possible increased the financial liabilities of the state to have to address them . 5. Efficiency: Rating – Modest . The project appraisal explained that “in economic terms , the benefits from investment in disaster management are the avoided losses in lives and property , which are multiplied as they translate in turn into macroeconomic losses to the country”, losses that are often cited to be more than a half billion US dollars per year in Colombia (PAD p. 27). While the PAD acknowledges the lack of empirical data in Colombia showing a relationship between such investments and consequent reductions in disaster losses , it also affirms that “while no formula exists for measuring the returns on investments in emergency preparedness , there is little room for doubt that the returns exist and that they are positive (PAD p. 77). Project appraisal did not provide an estimated economic rate of return (ERR) of the project. The PAD's Annex 9 pays more attention to the costs and benefits of DRM investment in general than to the analysis of this project's own performance. The ICR reports that Parts A-D of the project, as must all public sector investments in Colombia , have to go through a regular economic cost benefit analysis demonstrating feasibility to be approved . But the report does not provide details or summary of the results of such analyses of the project investments (ICR p. 12). The ICR reports that none of the intended annual assessments by the project—of annual losses of exposed public properties and infrastructure , actual damage reduction through DRM , and comparative costs of risk litigation investment—were carried out (ICR p. 12). There was no technical oversight or reporting requirements to guide ECOPETROL and INVIAS investments through a “cost recognition” approach espoused by the project (ICR p. 13). There was some data provided on the potential benefits of avoiding loss with regards to improvements made in the sector, but overall the lack of concrete evidence about project benefits means only a modest rating is possible . ERR )/Financial Rate of Return (FRR) a. If available , enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: With relevance Substantial and Modest on design , efficacy Modest on both objectives , and efficiency Modest, this project encountered significant shortcomings in trying to achieve its difficult to measure objectives . The lack of data demonstrating results relating to the achievement or otherwise of these objectives and to the project 's efficiency in so doing was also a shortcoming. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR notes that, thanks to the project, implementing agencies have developed knowledge management products that they can use to help local stakeholders pursue DRM . (ICR p. 15). Although not specifying which , the ICR reports that some ministries are integrated into an operationalized DRM framework , in terms of decision making, implementation, human resources etc. that should be important factors in sustaining project outcomes (ICR p. 16). The ICR expects that an unspecified “part” of the project investments will be continued into the future . Undue reliance on overseas consultants does increase the risk to development outcome , however, according to the ICR (ICR p. 16). a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: At appraisal the Bank opted for an Adjustable Program Loan (APL) instrument to be applied in three phases over a ten year period. The first phase APL1, the project reviewed here, was designed to address disaster related issues at the national level--by itself extended to over eight years . The second phase APL2 (P085727) aimed directly at Bogota only, at a total planned cost of US$145 million, was approved just ten months after APL 1. APL2 is not covered by this review . The planned APL3, to be aimed at other key municipalities , did not go ahead because the APL3 trigger (set out at the appraisal of APL 1) (APL1 PAD p. 34) that at least three municipalities should propose to take part in an APL 3 was not met. The ICR calls the APL1 project design ambitious and innovative , one that would "have to learn from its own experience, with no reference point" since there were no precedents to follow (ICR p. 5). One innovative component was a large contingency component under Part E , that was cancelled as part of the 2009 restructuring. It was replaced by a DPL Cat DDO (P113084) in an amount of US$150 million approved in December 2008. Interestingly the APL1 appraisal (in 2004) had then considered using a DPL DDO instrument (first offered by the Bank in 2001) for its design, but had rejected it on the grounds that "it could not be used to finance eligible expenditures needed for investments " (PAD p. 17). A DPL Cat DDO instrument (evolved from the earlier DPL DDO, but specifically with natural disasters in mind ) was not available at the time of the APL 1 appraisal. It was first offered by the Bank in 2008. Another APL1 innovation was a cost recognition arrangement for ECOPETROL under Part B. That proved to be unworkable , however (ICR p. 5-6), since it came with risks that had not been identified or mitigated during appraisal (ICR p. 17). Innovations such as these introduced undue complexity that would have required nurturing coordination and implementation arrangements beyond those that had actually been set up (ICR p. 16). Innovations aside, the project design, in response to the complex and risky circumstances in which the Borrower operated, was complicated. It involved many executing agencies unfamiliar with Bank rules and procedures (ICR p. 17) and multiple modalities of implementation that increased delays and transaction costs (ICR p. 6). In project preparation, local technical capacity was assumed without an ex ante assessment of the strengths and weaknesses of the particular local institutions involved (ICR p. 16). The project design chose MAVDT , that had some DRM experience, as the institution to lead project implementation . There was no alternative in Colombia at that time. The project’s Results Framework was not conceived as a tool that would facilitate management and monitoring of such a complex operation (ICR p. 17). It also did not establish linkages between the project components and the achievement of reduced fiscal vulnerability highlighted by the project objective . Still on fiscal vulnerability , whose intended reduction was a central aim of the project , IEG finds that the project design lacked an operational concept of it, one that would make clear exactly what the project intended to achieve through its institutional capacity strengthening. This clarity would have allowed measurements of the baseline conditions of fiscal vulnerability that would tell us the nature and scale of the problem to solve , as well as establish targets of reduction to be met. Against these the success of the project could be assessed . Project preparation provided none of this and the ICR did not focus on it thereafter . at -Entry Rating : Quality -at- Unsatisfactory b. Quality of supervision: According to the ICR, the Bank was proactive, flexible and responsive in identifying , addressing and resolving implementation issues (ICR p. 17). The report’s Basic Data (Table G.) shows 17 supervision missions all showing fully satisfactory ratings for the achievement of the development objective and all but four of the 17 showing fully satisfactory ratings for implementation progress --the four being rated Moderately Satisfactory . The ICR nevertheless notes that over the eight years of implementation , project supervision became “more difficult” because of the lack of “corrective action” to fix the discrepancies of the formulations of the objective that had not been revised, implying that the ICR thought it should have been . IEG concurs that a revision of this project 's objective was necessary , and probably best carried out as part of the 2009 restructuring. The ICR also criticizes "the drawbacks of the Results Framework " implying that it too should have been revised (ICR p. 17). IEG’s understanding is that Bank supervision itself should have been proactive in proposing and bringing about the necessary corrections through Bank management , and not passively awaiting action by others . There were four different sector managers and four different task managers over the eight years of this project , partly explaining, according to the ICR, why supervision had to rely partly on the Borrower (ICR p. 17). The ICR concludes, and IEG concurs, that closer, more hands on supervision by the Bank might have better guided project investments and enhanced project outcomes (ICR p. 17). Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government, Colombia's MOF and NPD, were committed to the aims and methods of the project . These included the use of cost recognition by ECOPETROL and INVIAS which would, in the words of the ICR, increase MOF’s “fiscal space” (ICR p. 18). But according to the ICR, the Borrower failed to identify the institutional constraints to implementing it, ultimately leading to its failure (ICR p. 18). NPD not only drove the design phase of the project, but was also active during the implementation and evaluation phases of the project . The project design stage was too short , however, to allow for adequate consultations with the many stakeholders that a cross cutting theme like DRM assembled. At the outset, most stakeholders lacked a general understanding of the project rationale, objectives and strategy (ICR p. 5). Also, a national financial crisis put the Government under pressure, allowing "very little fiscal space" for sub national entities like municipalities to function as key project players (ICR p. 5). Government ownership of the project strengthened when President Santos took office in 2010 and embraced a DRM agenda that was incorporated into the 2010-2014 National Development Plan (ICR p. 6). A 2011 restructuring of government by the new administration , however, split DRM responsibilities between the two new ministries MVCT (housing and urban planning) and MADS (environment) created out of the dismemberment of the single MAVDT (environment, housing, and urban development). MAVDT had been this project's original implementation agency, and formerly had full charge of government DRM in Colombia . After 2011, the overlap of DRM responsibilities undermined the efficacy and efficiency of the project 's risk reduction efforts at the sub national level (ICR p. 7). By completion, there had been no Government counterpart contributions to the project . This is in contrast to the US$290 million Government spending pledged to the project at appraisal . While the cancellation of this amount was fully agreed to by the Bank , principally via the 2009 restructuring, it will be interpreted by many as an indicator of a significant loss of Government commitment to , and ownership of the APL1 project. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: Between 2006 and 2011, the large single ministry, MAVDT, was in charge of project implementation . After 2011, this became the responsibility of one of the two ministries dismembered from it , namely MVCT. Their respective project implementation units (PIUs) sometimes lacked a technical coordinator to guide the unit 's work (ICR p. 6). The project's Technical Reference Group, foreseen in the design to support the PIUs , was not established, while the PIU preferred instead to deal ad hoc with their own consultants (ICR p. 6). A high turnover of implementing agency (IA) staff meant that the effects of formal project training were disbursed , but MVCT staff involved in the project nevertheless said that they had learned a lot on the job from the experience . During project execution, MAVDT (and later MVCT) had to supervise up to nine other IAs at the national level , who, in turn, had to oversee their numerous others at the municipal level . The ICR does not specify how many of the 1,044 municipalities targeted by project TA also had IAs responsible for implementing disaster reduction investments . Leaving aside the municipalities , most IA’s (at the national level) found it difficult to “comply with Bank rules especially during the first half of the project (ICR p. 18). Some of them, the ICR does not specify which , "lacked the general vision of the project ", but things improved later. The ICR notes that “only towards the end of the project was the coordination of actions and project components more consistent .” (ICR p. 18). Among the stronger performing IAs were Colombia’s Geological Service (SGC), and the Hydrology, Meteorology and Environmental Studies Institute (IDEAM). Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Reducing fiscal vulnerability to disasters , highlighted by the project objective , features (graphically) at the top of the PAD's Results Framework (PAD pp. 38-39), but is not mentioned by the appraisal 's "Arrangements for Results Monitoring" (PAD p. 40). The Results framework offers only one indicator , the share of municipalities with hazard risk management programs, to measure the achievement of this most important objective . As a performance indicator, this is not an adequate this project 's key outcome. Between the adoption by municipalities of DRM plans and programs and the intended reduction in fiscal vulnerability of the state, there are missing links each deserving performance indicators not included in the project 's M&E system. The implementing and financing of the municipal programs; local program substitution of state liabilities ; actual past experience of national government payments against such liabilities are among the missing links . The M&E design for this project had no performance indicators to monitor changes in such variables . As the ICR correctly recognizes , nearly all of this project's performance indicators, more process and output related than outcome oriented , could not capture progress in achieving the intended results of the project (ICR p. 7). In terms of organization, M&E design suffered from two weaknesses . First, it depended upon data collected at the local municipal level across the country for a project whose prime drivers were national institutions. Second, it did not make clear who was responsible for the collection and analysis of the data , who should present the M&E results and who should be the recipient of the reports . b. M&E Implementation: The ICR does not describe how M&E was implemented. or assess how well this was done . c. M&E Utilization: The principal use of M&E was to help track disbursements and project implementation itself . In this way it helped highlight the strong demand for project TA by municipalities , redirecting more funding toward them. Without providing information on the actual achievement of project objectives , though, M&E could not be used to provide feedback , as it should, on whether implementation was taking the project towards its intended outcomes or not , and whether any adjustments of execution might be needed to ensure it reach such an outcome . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: The ICR does not assess project compliance with environmental safeguards during implementation in detail . IEG notes from the PAD that APL1 was an Environmental Category B project , particularly due to the environmental impact of risk reduction works under Part B of the operation . Apart from this, the only other safeguard triggered was in relation to Cultural Property. (PAD p. 31). The ICR does not report on compliance with this safeguard . On the other hand, the ICR informs (p. 8), as does the PAD, that Involuntary Resettlement, OP 4.12, was not triggered. b. Fiduciary Compliance: MVCT ensured that this was in place during implementation . Financial information was made available in an accurate way and in a timely fashion. The ICR notes that " it can reasonably be assumed that project funds were used for the intended purposes" even though no external audit since 2011-2012 had been conducted at the time of that statement (ICR p. 8). The ICR does not report qualifications in the findings of the earlier audit reports . c. Unintended Impacts (positive or negative): According to the ICR, the project complied with Bank procurement rules despite the high turnover of procurement staff and the multiplicity of agencies with procurement responsibilities . In response to these difficulties , the PIU was strengthened with procurement specialists who were subject to in situ retraining . There were no reports in the ICR of misprocurement. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The project lacked a clear operational Unsatisfactory concept of reducing fiscal vulnerability at the heart of the project’s objective . The project provided no evidence of a project induced stronger capacity reducing fiscal vulnerability or mitigating disaster risks in Colombia during the eight years of its implementation. The project also lacked assessments of efficiency both at appraisal and at completion. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Unsatisfactory ICR did not factor into its rating several Satisfactory reported shortcomings, including: (i) risk to design not identified (re. ECOPETROL); (ii) complexity of design; iii) local technical capabilities were assumed to exist instead of appraising them properly; (iv) poor Results Framework; (v) project design making supervision difficult; (vi) insufficient hands on Bank supervision , that was shared with the Borrower; (vii) failure to revise the project objective after key components to achieve it had been cancelled.. Added to these was the lack of an operational concept , and baseline and target values of fiscal vulnerability and risk mitigation , the project’s main focus. Borrower Performance : Moderately Moderately ICR rating did not factor into the Satisfactory Unsatisfactory several shortcomings that it reported : (i) Borrower failing to identify institutional constraints to implementation; (ii) most stakeholders lacked a general understanding of the rationale, strategy and objectives behind the project; (iii) dividing DRM responsibilities between two ministries , thereby weakening its effectiveness Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: from ICR (with some IEG editing ): • Demand exists at the municipal and sub -national levels for in-depth and continuous technical support for DRM and disaster risk mitigation actions focused on the following : (i) stronger disaster risk identification and analysis by municipalities at all levels : (ii) coordination and reinforcement of municipal DRM by governing bodies at the regional level in Colombia (CARS); (iii) need for better communication on DRM among the national , regional and local levels; • Lack of financing and investment capacity at the local level may have impinged on more tangible outcomes from the project. Because of the disconnect between the TA provided to local institutions and their actual (in)capacity to finance the necessary follow -up actions and investments , the assistance provided for risk identification and vulnerability reduction tools had a limited impact on investment decisions at the municipal level . In the case of this project , the cancellation of the municipal investments (under Component B) removed an incentive for municipalities to bring risk mitigation theory into practice . • The Bank partnered well with other donors active in DRM assistance to Colombia . GFDRR grants targeted specific areas of TA , thereby enabling the delivery of additional assistance . IDB’s “Regional Insurance Facility for Central America” helped MHCP explore risk transfer mechanisms for seismic risk . The United Nation’s Economic Commission for Latin America (CEPAL), through its “Integrated Risk Management and Agriculture Insurance”, provided similar support to the project . from Borrower ICR : • Exogenous effects, such as the “La Niña” phenomenon can strengthen the political commitment at the national level to mainstream DRM into development plans . • Incorporating disaster risk mitigation measures into existing (by retrofitting) and future projects can significantly increase the initial investment cost—the ICR estimates by up to 80 percent. This means that special funding associated with DRM will normally have to be mobilized , something that DRM entities at the national level especially will have to bear in mind before they embark on these kinds of operations . • Large DRM operations funded by multilateral development banks , such as the IBRD, raise new opportunities as well as having implications for national governments , who should only embark on the implementation of such projects if their objectives contribute to national institutional goals and priorities . This applies in particular when dealing with procurement . • Despite the constant availability and diligence of the Bank task team , there were occasions when the Bank was slow to respond to the needs of the project , presumably in communicating its “no -objections” that delayed procurement especially . Having a back up to the task team leader (TTL) empowered to take such decisions could help the more timely delivery of a project . Added by IEG :. • A self evaluation ICR exercise should focus primarily upon the assessment of the achievement of the objectives as the project promised . In the case of this project , the ICR gave most attention to the delivery of the project’s components, making for an output-oriented review that provided little evidence of what was actually achieved in terms of the outcome of more capacity to reduce fiscal vulnerability and mitigating the negative impact of natural disasters. • Appropriate self-evaluation by an ICR should be seen as an opportunity to assemble empirical evidence of the impacts of the kind of activities sponsored by the project upon the ultimate outcomes sought , namely reduced fiscal vulnerability to disasters and lessened effects of disaster , as sought by this project . Instead of providing this additional evidence , the ICR rested its case upon the assumption that DRM actions would yield the desired effects. 14. Assessment Recommended? Yes No Why? Why ? For IEG to follow up with evidence of project outcomes and to reconcile the links with the DPL project intended to contribute to the same ends . . 15. Comments on Quality of ICR: This is a less than satisfactory ICR with several errors and shortcomings , including the following: - instead of assessing the actual outcomes of the project in comparison with the intended outcomes , the ICR's assessment is output oriented based , focused principally upon the delivery of the project components . - in the absence of evidence of the actual outcomes , the ICR instead relied on assumptions that these outputs would produce the desired impacts on the outcome , without assembling evidence to demonstrate this . - assessment of relevance of objectives is very thin and assessment of relevance of design is non -existent. - several errors in reporting project costs and financing : (i) total appraisal cost of project according to the PAD was US$551.3 million (PAD p. 50). ICR reports a total cost of US$260 million that does not include government counterpart commitment of US$290.0 million; (ii) total actual loan disbursement according to the Bank 's Client Connection was US$102.8 million (after the cancellation of an undisbursed amount of US $7.2 million. ICR reports a total actual loan disbursement of US $110.0 million that does not take into account the Loan amount of US $7.2 million cancelled at completion; and (iii) the US$110 million final actual cost of the project reported by the ICR could not exceed US$102.8 million, the value of the loan disbursements that became the only source of funding for the project . This inconsistency raises doubts about the accuracy of the actual cost figures of all the project components that add up to the incorrect total. a.Quality of ICR Rating : Unsatisfactory