ICRR 13800 Report Number : ICRR13800 IEG ICR Review Independent Evaluation Group 1. Project Data : Date Posted : 10/11/2013 Country : Mexico Is this Review for a Programmatic Series? Yes No Series ID : First Project ID : P110849 Appraisal Actual Project Name : Mexico - Climate US$M ): Project Costs (US$M): US$M501,25 US$M501.25 Change Development Policy Loan L/C Number : L7535 Loan /Credit (US$M): Loan/ US$M ): US$M501.25 US$M501.25 Sector Board : Environment US$M ): Cofinancing (US$M): Cofinanciers : Board Approval Date : 04/08/2008 Closing Date : 05/30/2011 05/30/2011 Sector (s): General water sanitation and flood protection sector (50%); Forestry (25%); General energy sector (25%) Theme (s): Environmental policies and institutions (50% - P); Climate change (50% - P) Evaluator : Panel Reviewer : ICR Review Group : Coordinator : John R. Eriksson Robert Mark Lacey Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The Program Development Objective as stated in the “Operation Description� (Section V) of the Program Document (PD) is "to recognize and support the government's efforts under its National Climate Change Strategy to mainstream climate change considerations in public policy " (p.23). The objective stated in the PD Policy Matrix (Annex 3) is to "mainstream climate considerations in public policy " (p.66). This is shorter than the formulation in Section V, but the substance is virtually identical . A third definition at the beginning of the the main text of the PD states that the objective is to "to assist the GoM in integrating climate change considerations in public policy ." (p.1) The Policy Matrix definition of the objective will be used for evaluation purposes since it is more succinct . b. If this is a single DPL operation (not part of a series), were the project objectives/ key associated outcome targets revised during implementation? No c. Policy Areas: At the time of loan approval, Mexico met the policy requirement of OP 8.60 for a sound macroeconomic framework that had succeeded in moderating output volatility and creating economic growth . As described in the table below, the Program Document (p.28) stipulates five policy areas (called “Policy Action Areas� in the PD and ICR) and “Expected Outcomes� within 24 Months. Monitoring indicators and achievements corresponding to these “outcomes� are reported in section 4 below (most of the so-called “outcomes� are more properly designated as outputs in section 4). Policy Areas Expected Outcomes within 24 months (“Policy Action Areas� in the PD and ICR) Mexico defines its voluntary emission reduction commitments in the (1) Mitigation context of its 2007-2012 National Development Plan and its 2030 planning framework, with major sectors contributing to the national emission reduction targets (2) Domestic Carbon Mexico acquires information about policy options to internalize the effect of Pricing GHG emissions (“internalize� through a market mechanism) (3) Adaptation A number of state governments prepared to confront climate change impacts A number of city governments prepared to confront climate change Impacts (4) Knowledge Base Improved information basis for assessing progress towards reducing climate vulnerabilities and contributions to GHG emissions. Costs of specific mitigation measures available in priority sectors Mexico’s authorities able to make informed choices about policy measures (5) Institutional Improved coordination and accountability in climate change policy Strengthening implementation. Climate change policy is further mainstreamed in sectoral policy and Investments d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The Loan was disbursed in one tranche on 12/09/2008, two weeks after effectiveness . There was no Borrower contribution. The operation closed on May 30, 2011 as scheduled. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Modest . The overall objective of the operation of ‘mainstreaming’ is not defined in the document, and describes a process rather than an outcome . All operations are to be assessed against their outcomes . Moreover, the overall DPL objective of 'mainstreaming' is not given an operational meaning that would provide monitoring of its achievement by objective, measurable criteria. Consequently, the relevance of the overall objective is rated modest. b. Relevance of Design: Substantial Given the lack of a monitorable definition of 'mainstreaming' and the consequent lack of explicit linkages between this overall objective and the project's specific objectives and targets, we looked at key design features (the policy action areas) and their linkages with key associated outcome targets to figure out what to assess . The two anticipated outcomes of the strategy are climate change mitigation (reduction in Greenhouse Gasses ) and adaptation (managing risks). The Operation Policy Matrix (ICR, pp.4-5) seeks to align the Loan’s three prior actions and corresponding “outcome indicators� with the program objective to “mainstream climate considerations in public policy .� The targets set were however mainly at the level of outputs (in terms of actions) rather than intermediate outcomes – that the policies are actually being implemented and behavior has changed . • The first prior action was the Mexican submission in 2007 of the Third National Communication to the UN Framework Convention on Climate Change . This updated the 2002 inventory of Greenhouse Gas (GHG) emissions and submitted a number of climate change mitigation and adaptation studies . The target was to provide a basis for designing national public policies on climate change in all sectors, and for formulating the National Climate Change Strategy and the Special Climate Change Program . • The second prior action was the approval in 2007 of the National Climate Change Strategy by the Intersecretarial Commission on Climate Change . The target was to incorporate in the National Development Plan Mexico’s voluntary commitment to reduce GHG emissions associated with economic growth . The Strategy identified four “Policy Action Areas� (i.e., Policy Areas) that were addressed by the Loan – Mitigation, Domestic Carbon Pricing, Adaptation, and Knowledge Base, as well as a fifth area added by the Loan, Institutional Strengthening . Given the state of knowledge about climate change, knowledge building and institutional strengthening were seen as important in developing and refining policy, and carbon pricing was seen as the instrument of choice through the Clean Development Mechanism under the international Kyoto Protocol for reducing carbon emissions . • The third prior action was initiation of sector programs for energy, environment and natural resources, including policy incentives to reduce the carbon -intensity of the Mexican economy . The target was to initiate a process to reduce GHG emissions in the economic growth of key sectors and adopt measures to increase resilience to reduced emissions. No date is provided for this prior action . While the operation could have been strengthened by setting a clear and outcome-oriented objective, the actions supported by the DPL are highly relevant for taking climate considerations into account in public policy and are likely to contribute to climate change mitigation (reduction in Greenhouse Gasses) and adaptation (managing risks) in the longer run. 4. Achievement of Objectives (Efficacy): Since the objective of mainstreaming has not been defined in the documentation, IEG will base the efficacy discussion upon an interpretation of the overall objective being that to which output and outcome targets were achieved and in turn linked to policy action areas . The Results Framework Matrix at the beginning of the ICR identifies eleven “Program Development Objective Indicators� grouped under the five Policy Action Areas . Most of these achievements were at the output level . Baselines, targets and achievements were reported in the ICR for each Indicator . As shown below, targets were for the most part attained and in some cases exceeded . In each case, the baseline is 04/08/2008 and targets refer to achievement as of 05/30/2011. However, baselines, targets, and /or achievements are vague or not defined for 4 of 11 indicators, which makes assessment of achievement problematic . Climate change mitigation : Substantial Outputs Support implementation of critical steps required for the Government to collect, process, analyze and build consensus on the voluntary emission reduction targets it was committed to announce to the Parties to the UN Framework Convention on Climate Change . • Mexico adopted the Special Climate Change Program (Target: to adopt the Program against baseline of "Program being defined") • Several studies were completed that defined sectoral contributions to climate change . (Target: to define sectoral contributions to climate change against baseline of no studies ). (Note: The extent to which “several studies� constitute “define sectoral contributions to climate change � is not explained. “Several� is not a satisfactory definition of an achievement.) • A quantitative study on the physical and economic impacts of climate change plus four complementary studies were completed (Target: "Quantitative Characterization of Physical and Economic Impacts of Climate Change" against baseline "no precise data available"). • National emissions targets were defined through 2012 and longer term goals were set (Target: emissions to be set against baseline of no emissions set . Longer term goals were not defined ). • The Fourth National Communication was delivered to the Framework on December 14, 2009, and updated on April 20, 2010. (Target: Fourth Communication to be submitted 05/30/2011 against baseline of Third Communication submitted as prior action ) Support carbon trading system development . • A national carbon trading system was developed and a trading fund was established . Outcomes • National carbon trading system functioning (Note: Twelve projects were registered with the Clean Development Mechanism between September 2008 and August 2009, resulting in Mexico ranking fifth among countries in terms of Certified Emission Reductions obtained (ICR, p.14). • The Inter-secretarial Commission on Climate Change delivered its first progress report on August 28, 2009. (Target: first report to be delivered on May 5, 2011 against baseline of the Special Climate Change Program not having been adopted as of April 28, 2008 ) • There was no baseline for, or measurement of the longer term outcome indicator of GHG reductions . Adaptation : modest Ouputs Completed "multiple" studies describing the feasibility of priority mitigation actions . (Target: mitigation activities defined for priority sectors against baseline of some studies to be carried out, but further definition is required since the baseline, target and degree of achievement are vague .) Support creation of conditions for sub -national agents to prepare themselves to confront climate change impacts . • Climate change action plans were completed or under preparation in 26 states (of a national total of 32 states, including the Federal District ). Five states have completed and adopted their action plans (Federal District, Veracruz, Puebla, Nuevo León and Guanajuato) (Target: at least five states to commit to an action plan against a baseline of two states in the process of defining plans .) • Ten cities completed action plans that were incorporated in their respective state action plans (Target: at least 10 cities among the most vulnerable were to be committed to a climate change plan against a baseline of only Mexico City having a climate change action plan ). Outcomes There were no intermediate or longer term outcome indicators reported on for adaptation actions . 5. Efficiency (not applicable to DPLs): 6. Outcome: The relevance of the overall objective is rated modest in view of its lack of monitorability. Relevance of design is rated substantial, as the DPL supported actions are highly relevant to improve the country ’s performance in climate change mitigation and adaptation . The efficacy of achieving targets contributing to GHG emission reduction is rated substantial, while for adaptation it is rated modest due to the lack of evidence of implementation of action plans or of changed behavior. Outcome is rated Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Mexico is expected to be disproportionately affected by the impacts of climate change as a result of such phenomena as intensifying hurricanes, rising temperatures, and more variable precipitation . The Government recognized the need for action and took steps that were intended to take fuller account of climate change considerations in public policy, including the National Climate Change Strategy, the Special Climate Change Program, and related institutional coordination mechanisms . The global economic crisis affected Mexico most severely in 2009, several months after the Loan became effective, but this did not hinder the Government in pursuing policy actions under the Loan . One factor that could impact sustainability is the change in administration that took place at the end of 2012, since both the National Development Plan and the Special Climate Change Program are for the 2009-2012 period. But the magnitude of programs and projects that embrace climate change that are under way or designed and the institutionalization of the climate change adaptation and mitigation agenda at the Federal and State levels make it unlikely that a future administration would backtrack on what the Development Policy Loan supported . Thus, from technical, political, economic, financial, government, and institutional perspectives it would appear that the risk to the actions that have been taken so far is negligible to low . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: An analysis of the macroeconomic situation was undertaken prior to approval of the Loan . The Loan was prepared in tandem with the program of knowledge provision and advisory services under the Memorandum of Understanding (MoU). The Loan and activities under the MoU played complementary roles in supporting the Government's climate change program. However, the analytical activities supported by the MoU served to leverage a wider engagement across states and sectors . At entry the DPL lacked a monitorable definition for its stated overall objective, to mainstream climate change considerations in public policy, which precluded an explicit linkage between the overall objective and the project's specific objectives and targets . The Loan is part of a series of environment and climate change -related assistance the Bank provided Mexico over the eight-year period from 2004-2011, including 11 analytical and technical assistance activities, 10 investment project loans and 7 Development Policy Loans, including the Climate Change Loan . The timing of preparation and approval of the latter Loan in 2008 permitted the Bank to engage with an incoming Mexican administration that had put climate change high on its agenda . The Loan was designed, according to the Program Team, by a large team in 4 weeks. The Team also indicated that the Loan attracted the attention of sectors that had not been focusing on environmental issues but became a catalyzing vehicle for garnering support for the Government ’s climate change agenda. During preparation of this operation, the Bank team held several meetings with the Inter American Development Bank (IADB) to discuss cooperation on the costs of the planned climate change study . The PD states that “collaboration with IADB and other donors will continue through the implementation of this DPL and continued policy dialogue with the Government of Mexico � (p.19). Neither the PD nor ICR contain any other reference to other donors (except for a PD reference to the IMF ). More information about other donor support for climate change in Mexico might have helped assess attribution of climate change policies and programs to this DPL . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank conducted periodic missions throughout a two -year period, focused on progress towards results in accordance with the eleven monitoring indicators . Bank implementation support was focused on working with the Government, in particular with and through the Ministry of Environment and Natural Resources in the delivery of knowledge services as inputs to expected results, e .g., the Mexico Low-Carbon Study, and preparation of plans for climate change at state level in Michoacan and Campeche that subsequently fed into the design of policies in programs at the national sectoral level . Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government's commitment to climate change generally and to the DPL, was strong and consistent, according to the ICR (p.18 - "unwavering"). Despite constraints brought about by the economic crisis, budgetary allocations to the program were unaffected, and the Government, from the highest level provided strong support to the program. The hosting of the 16th Committee of the Parties under the Kyoto Protocol (COP16), in end-2010, reinforced Mexico’s position and visibility as a leader in the field of climate change . Government Performance Rating : Satisfactory b. Implementing Agency Performance: The Environment and Natural Resources Ministry played a key role in the formulation of climate change policies and in implementing the Special Climate Change Program . The Ministry was closely engaged in monitoring progress and providing assistance, with periodic corrections throughout the operation . Implementing Agency Performance Rating : Satisfactory Overall Borrower Performance Rating : Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The “Monitoring Indicators� to be achieved in 24 months, as set out in a table in the Program Document (p.28, Table 3), roughly parallel those in the ICR Results Framework as discussed in Section 4 above. Some of the indicators have the same limitations as in the ICR; namely, lack of quantification or clear definition . The same table shows “Expected Outcomes within 24 months.� These are quite vague. For example:, • “a number of state governments � • “a number of city governments� • “authorities able to make informed choices � • “improved information basis for assessing progress � • “improved coordination and accountability � • “climate change policy is further mainstreamed .� These “outcomes� are all pertinent in varying degree to the program objective but it is impossible to assess their degree of achievement without greater clarity and precision . Their usefulness as indicators is therefore limited . b. M&E Implementation: The Government and the Bank reviewed progress and the Ministry of Environment and Natural Resources collected data and provided updates to the monitoring indicators, as agreed at appraisal . c. M&E Utilization: No utilization of the M&E system is reported . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: No safeguard policies were triggered . b. Fiduciary Compliance: No fiduciary issues were identified . c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Relevance of the Program Objective is Satisfactory rated modest, and that of Design is substantial. With regard to efficacy, no definition of “mainstreaming� is provided and no evidence presented on which to base a judgment of the extent to which it has been achieved . While Efficacy of the mitigation objective is substantial, Efficacy of the adaptation objective is rated modest. Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Moderately A moderate shortcoming in preparation Satisfactory was the lack of precision and measurability for the operation's development objective and the lack of any related indicators. Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The first two lessons have been adapted from the ICR with some modification . The third is from IEG. 1. A Memorandum of Understanding can provide an integrated and flexible approach for the provision of analytical support, technical assistance and training to complement climate change Development Policy Loans . 2. Recognizing that many actions, especially with respect to adaptation to climate change in a large and diverse country, are best addressed at the sub -national level, a national government can establish a broad framework and incentives to engage sub -national governments as well as mechanisms to strengthen the capacity of weaker ones . 3. A Development Policy Loan requires clearly defined objectives and a logical results framework . In particular the main intended outcome – in this case “mainstreaming� -- should be clearly defined and measurable . All outcomes and outputs should be relevant to the Program Objective and have operational definitions with stated baselines and targets, so that achievement can be objectively assessed . Any indicator that can be stated in quantitative terms should have its baseline, target and achievement expressed in those terms . 14. Assessment Recommended? Yes No Why? There have been several environment -related DPLs for Mexico. Their total magnitude and relatively unique character would make them a strong candidate for a cluster PPAR . 15. Comments on Quality of ICR: The ICR is strongest in its presentation of evidence of achievements at the output level (including one at an intermediate outcome level), based on the 11 monitoring indicators, although there is insufficient information to assess fully the outputs of four of the indicators . But more concrete evidence could have marshaled and been more systematically organized. While the definition of the overall objective, “mainstreaming climate considerations in public policy� can be inferred, the PD and ICR lack an operational definition and corresponding indicators of the operation ’s intended overall outcome. There is also a lack of explicit linkage between the overall development objective and its specific objectives and targets There are four different places in the ICR where results information is found, spanning the Results Framework Analysis at the beginning to Annex 9 at the end. These sections often contain different information . Assertions tend to be stated with superlative adjectives, with little or no evidence to back them up . Some assessments are quite sparse and would have benefitted from specific evidence and concrete examples (e.g. sections on Relevance of Design, Bank Performance, Borrower Performance, M&E ). . The terms used to describe results are inconsistent and confusing: the 11 performance indicators are variously called “Project Development Objective Indicators, � “Monitoring Indicators� and “Expected Outcome Indicators.� Moreover, the three performance measures for the three Prior Actions are also called “Outcome Indicators,� but after being presented in Table 1 (pp.4-5), they do not subsequently reappear. All further references to results are in terms of the 5 “Policy Action Areas� and the 11 “PDO Objective Indicators� (or “Monitoring� or “Expected Outcome�). Section 5 of the ICR implies that “Policy Areas� and “prior actions� are the same. Other than this, the consistency of the ICR with guidelines is generally adequate . a.Quality of ICR Rating : Satisfactory