Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) Report Number : ICRR0020653 1. Project Data Project ID Project Name P121195 MX Efficiency Improvement Program Country Practice Area(Lead) Mexico Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79730 31-Dec-2014 162,000,000.00 Bank Approval Date Closing Date (Actual) 09-Nov-2010 30-Jun-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 100,000,000.00 0.00 Actual 100,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan John R. Eriksson Christopher David Nelson IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 6) and in the Project Appraisal Document (PAD) for the Water Utility Efficiency Improvement Project (known in Spanish as “PROMME”) was: "To improve the efficiency of Participating Water Utilities through the Provision of Technical Assistance and Financing". Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component 1. Water Supply and Sanitation (WSS) Sector information and knowledge management improvement. (estimated cost at appraisal US$4.75 million. actual cost at closure US$0.00 million: see Subsection 2e below for explanation of actual cost at closure). This component aimed at improving the National Water Commission’s (CONAGUA in Spanish) capacity with regard to efficiency improvements in the WSS sector. Activities included: (i) management of information (such as developing standardized indicators for measuring the WSS's performance, harmonizing activities on WSSU information systems and making this data accessible to the public through internal and external benchmarking. and (ii) management of knowledge (such as developing norms and standards for monitoring performance, developing manuals on physical and energy efficiency improvements, documentation and dissemination of good practices, training CONAGUA staff and documenting experience learned under the project in particular under the output-based disbursement window (discussed below in component 2c). Component 2. Modernization of the services of participating water utilities. (estimated cost at appraisal US$157.00 million. actual cost at closure US$161.75 million). There were three sub-components. Sub-Component 2a. Technical Assistance. (estimated cost at appraisal US$7.00 million. actual cost at closure US$10.09 million). Activities in this sub-component aimed at providing support for carrying out diagnostic studies, preparing investment plans and support to the Participating Water Utilities (PWCs) for financing technical assistance activities (such as studies on tariff and subsidies, governance structure or specific operational or commercial efficiency issues). Sub- Component 2b. Classical Efficiency Investments. (estimated cost at appraisal US$145.00 million. actual cost at closure US$149.66 million). Activities in this sub-component aimed at providing support for physical and commercial efficiency improvements such as establishing District Metering Areas, active leakage control, source and consumer water meter installation and supply and installation or upgrade of the billing system. Sub- Component 2c. Pilot Output-Based Efficiency Investments. (estimated cost at appraisal US$5.00 million. actual cost at closure US$0.00 million; see Subsection 2e below for explanation of actual cost at closure). Activities in this sub-component aimed at providing support for carrying out physical and commercial efficiency improvements through using outputs, rather than inputs, as the basis for disbursement. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. (estimated cost at appraisal US$161.75million. actual cost at closure US$161.75 million). Although the actual project at closure was as estimated at appraisal, actual costs of Sub-component 2b activities (classical efficiency investments) were higher than estimated. Activities associated with Sub-- component 2c (Output based efficiency) component were cancelled due to the lack of demand on the part Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) of the participating utilities and Component one activities (WSS Sector Information and knowledge management improvement) were completed by the National Water Commission (CONAGUA) using their own funds. Funds earmarked for these components were reallocated to finance the increase in cost of Sub-component 2b activities. Project Financing. The project was financed by an IBRD loan of US$100.00 million. Amount disbursed was as planned at US$100.00 million. Borrower Contribution. Appraisal estimate US$62.00 million. Their contribution at closure was as planned. Dates. The following changes were made following a Level 2 restructuring on 12/01/2014: (i) Component 2c activities were cancelled due to lack of demand on the part of participating water utilities: (ii) Funds were reallocated between components: (iii) The results framework and some intermediate indicators were modified: and, (iv) The project closing date was extended by 18 months for completing ongoing activities. The project closed a year and half behind schedule on 06/30/2016. 3. Relevance of Objectives & Design a. Relevance of Objectives Before appraisal, many water supply and sanitation utilities in Mexico were not making efficient use of water resources. Average Non Revenue Water (NRW) losses (such as physical losses due to leakages and overflows at the utility's storage tanks caused by poor maintenance, lack of active leakage control and poor quality of underground assets and commercial losses due to customer meter under registration, data- handling errors and theft of water) were over 38%. More than a third of utility customers did not have their consumption metered, over 20% of bills went uncollected and average tariff collection rate was 79%. Overall efficiency (an indicator used in Mexico for measuring operational and commercial losses) was 44% and the sector as a whole was not generating sufficient revenues for covering full costs. The PDO was relevant to the government's Program for Potable Water, Sewerage and Sanitation in Urban Areas (APAZU in Spanish) which had been under implementation since 1990 with the objective of improving and expanding potable water services through financial and technical support for federal and municipal agencies and utilities. The PDO was consistent with the Government's National Development Plan (NDP) for the 2007-2012 period. This plan established objectives and strategies which were to be implemented by a set of sectoral Programs. The water sector commitments of the plan were aligned with the National Water Program (PNH) which highlighted the need for strengthening the operational and commercial efficiency of water utilities and also set a goal of increasing the utilities' overall efficiency from 36% to 44% by 2012. The PDO was consistent with the objective of the National Infrastructure Program for the 2014-2018 period, which highlighted the goal of "expanding water infrastructure to ensure the supply of water for human consumption." At appraisal, the PDO was consistent with the Bank's Country Partnership Strategy (CPS) for Mexico for the 2008-2010 period. A key theme of the CPS was "Developing Infrastructure and Assuring Energy Security and Environmental Sustainability". The PDO was consistent with thematic area four of the CPS for the 2008- 2013 period, which focused on financial sustainability and efficiency in the infrastructure sectors. One of the pillars of the current CPS for the 2014-2019 period underscored the need for "Promoting Green and Inclusive Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) Growth" including through increasing efficiency in management systems at the sub-national level. Rating High b. Relevance of Design The statement of the PDO was clear and the causal links between the project activities, their outputs and outcomes were logical. Activities were associated with efficiency improvements (such as establishing District Metering Areas, active leakage control, consumer water meter installation and supply and installation or upgrade of the billing system) and Output-based disbursements. These activities, in conjunction with technical assistance activities to the participating water utilities and capability building to the National Water Commission (CONAGUA), were expected to aid in improving the efficiency of participating water utilities. However, there were shortcomings in design. The project was envisioned as a large scale up of a prior Bank financed project (discussed in section 9a). Unlike in the previous project where participating water utilities were mostly "hand-picked", this project was open to any utility which had fulfilled the eligibility conditions. However, with no limit on the number of participating utilities, 91 utilities chose to participate in the program as compared to the 50 utilities estimated at preparation. This nearly doubled the scope of work on M&E and general coordination of financial management and procurement oversight. This was further constrained by the limited available staff time for completing Component One activities associated with knowledge and information systems to bolster national benchmarking capacity. The latter activity was subsequently dropped from the scope of the project, although it was advanced in part by CONAGUA using its own resources. The design incorporated selection criteria to prioritize utilities that participated for more than one year. While this aspect of design could be expected to safeguard against inefficient use of funds, 44% of the utilities (40 out of 90 municipalities) participated for just one year and this in turn meant that robust conclusions could not be drawn regarding the increased efficiency established in the results framework for the PDO (Borrowers ICR, page 54). There was a disconnect between the scale of the investments, which had to conducted in one year given Mexico's one-year budget cycle (which mandates that actions must be designed, procured and executed between January and December) and the expected outcomes which could take a longer time. The design focused more on physical efficiency and did not target enough funds for activities associated with commercial efficiency aimed at financial viability of participating utilities. It is not clear if pilot activities relating to output-based disbursement efficiency investments were designed in consultation with the National Water Commission (CONAGUA), given that, as per the Borrower’s ICR, page 55), “From the start, CONAGUA did not accept the component on output-based disbursement, given that the national regulation did not allow it and the available time to implement it, but the Bank insisted to include it in the design” (ICR, page 55). It was a major shortcoming in the relevance of design not to have identified this issue at preparation and worked out a solution with the Government or dropped this subcomponent before approval. The absence of diagnostic assessments establishing the initial efficiency situation (baseline data) in the utilities contributed to imprecise measurement of the impact of resources under the program (Borrower's ICR, page 54). It is not clear if the actions implemented by the utilities had a medium or long term plan (Borrowers Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) ICR, page 54), given that a subsequent a Mid-Term Evaluation of 143 actions across 27 municipalities and 10 states, showed that: (i) investments did not follow any prioritization criteria: (ii) many utilities had undertaken investments for very small amounts: and, (iii) half of the utilities lacked a comprehensive investment plan. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve the efficiency of Participating Water Utilities through the Provision of Technical Assistance and Financing. Rationale Outputs. • 670 staff of the National Water Commission (CONAGUA) including management and technical specialists, administrators and commercial and operational staff were trained as compared to the target of 800 on areas associated with strengthening commercial systems, theory and practice of increasing electromechanical efficiencies and Water and energy savings. This was short of the target of 800. (ICR, Datasheet, Intermediate Indicator Number One and ICR, page 22). • 492 actions had been implemented across 25 states and 91 municipalities. No targets were set for this indicator. (ICR, page 11). • Publications pertaining to efficiency issues were not made available as intended at project closure, as targeted. (ICR, Datasheet, Indicator Number Two). • 95% of implemented actions achieved their set targets. This exceeded the original target of 60%. (ICR, Datasheet, Intermediate Indicator Number Four). • No water utilities participated in the Output-Based Disbursement (OBD) at project closure as compared to the target of One (ICR, Datasheet, Intermediate Indicator Number Five). • The project supported 91 utilities at project closure. This exceeded both the original and revised targets of 50 and 60 respectively. (ICR, Datasheet, Intermediate Indicator Number Six). • One workshop was undertaken to disseminate program results and best practice utility case studies at project closure as compared to the target of two (ICR, Datasheet, Intermediate Indicator Number Seven). • There was no documentation of best practice case studies at project closure as compared to the target of five (ICR, Datasheet, Intermediate Indicator Number Eight). Outcomes. Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) • The revenues collected in Mexican pesos per cubic meter for water produced by 25 water utilities increased by 5 percent in real terms at project closure. This was short of the target of 30 water utilities. (ICR, Datasheet, PDO Indicator Number One). • The energy consumption of seven water utilities decreased by 5 percent as compared to the target of eight water utilities (ICR, Datasheet, PDO Indicator Number Two). • 12 water utilities showed commercial efficiency improvement of five percent or more due to improvements in commercial system hardware and software, installation of micrometers and re- engineering of billing and collection systems. This was short of the target of 30 water utilities (ICR, Datasheet, indicator Number Three). This was partly due to external factors outside the control of the project such as Mexico's one year budget cycle, which mandates that actions must be designed, procured and executed between January and December. • The global efficiency (encompassing both physical and commercial efficiency) of eight water utilities increased by 2 percent at project closure. This exceeded the target of five water utilities. (ICR, Datasheet, PDO Indicator Number Four). In 2014, the project accounted for just 4% of federal funds to the National Water Commission (CONAGUA) and the participating utilities received support from other domestically funded federal programs. Therefore, it is difficult to assess the extent to which this project contributed to the PDO. Given this and the fact that targets were not met in two cases, the IEG assessment of efficacy is rated as Modest. Rating Modest PHREVDELTBL PHREVISEDTBL 5. Efficiency Financial Analysis. An ex ante financial analysis was conducted on an indicative group of three utilities at appraisal in the “Classical Efficiency Investments” category (Subcomponent 2b) activities. This subcomponent accounted for 87% of the estimated cost at appraisal. The financial benefits of the project were assumed to come from decrease in water losses (due to increase in collected revenue by the utilities and reduction in operating cost) of the utilities. The financial analysis was conducted using a 9 percent discount rate which was consistent with market values (PAD, page 35) and yielded an estimated Financial Rate of Return (FIRR) for the three utilities of 36%, 19% and 23%, respectively. Economic analysis. No economic analysis was conducted at appraisal on account of lack of reliable data on opportunity cost and economic cost of water saved. An economic cost-benefit analysis and a financial analysis were carried at closure for a sample of four operators that participated in the project, which accounted for 18% of the total project cost. The economic benefits of the project were assumed to come from savings in operating economic costs (due to reduction in physical losses and improvements in energy usage). The average Economic Internal Rate of return (EIRR) was 13.6% and the Net Present Value of Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) US$11.00 million at a discount rate of 9%. The financial benefits of the project were assumed to come from savings of operating financial benefits and increased revenue. The average FIRR was 13% for the four operators in the sample and the NPV at 9% discount rate was US$12.00 million. Administrative and Operational Issues. The project which was scheduled to begin in early 2011 became effective 11 months later in November 2011. Insufficient federal budget allocations to the program contributed to the limited implementation progress during the first three years of the project and this led to the hasty implementation of the project during the final three years (ICR, page 6). There were time overruns with the project closing 18 months behind schedule due to a combination of factors including procurement delays and external factors outside the control of the project (discussed in section 4). Thus, at project closure, less than a quarter (24%) - 22 of the participating utilities reported progress toward indicator targets. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objective both for the government strategy and the Bank strategy for Mexico was rated as High. Relevance of design was rated as Modest. Efficacy of the single objective - to improve the efficiency of Participating Water Utilities through the provision of technical assistance and financing - was rated as Modest, as targets for two of the key outcome indicators were not met and it is difficult to determine the extent to which this project contributed to realizing the PDOs. Efficiency was rated as Substantial, given the positive returns to infrastructure investment despite administrative and operational inefficiencies. Taking into account the constituent ratings and the fact that the “efficacy of the single objective was rated as Modest, pursuant to the Harmonized Criteria agreed by IEG and OPCS, the overall development outcome is rated Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) 7. Rationale for Risk to Development Outcome Rating Financial Risk. Most municipalities provide water supply and sanitation in Mexico through the Service Providers (OOs - the Spanish acronym). There is substantial financial risk to development outcome, given the fact that the Service Providers in municipalities lack financial autonomy and are highly dependent on state and federal government for financing, and tariffs approved by states which often do not cover costs (ICR, page 2, para 7). Technical Risk. Although some utilities improved their financial footing through increasing physical and commercial efficiency, it is not clear if all of the utilities would have the technical capacity as well as the ability to cover the costs associated with additional physical investments. Institutional Risk. There is substantial risk to ongoing benefits from this project, since it is if the water utilities particularly in smaller towns have the required capacity (in terms of technical staff) to manage similar undertakings. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry This project was a scale up of a pilot program financed by the Bank (Mexico Water Sector Technical Assistance Project (PATME in Spanish) which introduced a performance-based efficiency improvement approach in 18 water utilities and developed 12 standard performance indicators (ICR, page 2, para 9). The implementing agency chosen for this program had prior experience with implementing Bank-financed and other donor-funded programs (discussed in section 9b). Appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in section 11). • The team underestimated the risks associated with the capacity constraints of the National Water Commission (CONAGUA) to undertake the PROME program. Unlike the prior Bank-financed PATME project where the participating utilities were mostly handpicked, this project was open to utilities that met the eligibility conditions. 91 utilities chose to participate in the program as compared to the 50 utilities estimated at appraisal. This nearly doubled the scope of work of CONAGUA on monitoring and evaluation as well as financial management and procurement oversight (ICR, page 5, para 24). • There were shortcomings in the project's M&E (discussed in section 10a). Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The supervision team helped in preparing model technical specifications and this aided in improving the quality of bid documents presented by utilities during implementation. • There were delays associated with authorizing investments and with providing no objections on the part of the Bank team (Borrower's ICR, page 55). This was further exacerbated by the participation of different Bank procurement specialists applying different criteria during the life of the project (Borrower's ICR, page 56). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance The Government's commitment towards traditional utility investments was demonstrated by its counterpart funding (coming from a mix of State, municipal and utility cash-flow resources), which accounted for about 38% (US$62.00 million) of the total project cost. The performance of the Bank of National Savings and Financial Services (BANSEFI), as the financial agent in keeping records of all project activities, was deemed to be satisfactory (ICR, page 18). • Insufficient federal budget allocations to the Potable Water, Sewerage and Sanitation Program in Urban areas (APAZU in Spanish) contributed to the limited implementation progress during the first three years of the Project. This was partly due to the impact of the global financial crisis on the country's fiscal space, but was rectified and funds were made available, which eventually aided in completing the project activities in the remaining three years of the project. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The federal National Water Commission's (CONAGUA's) Office of Water Utilities Strengthening was in charge of implementing this project with the Bank of National Savings and Financial Services (BANSEFI) as the financial agent. CONAGUA had experience implementing Bank-financed projects and other donor- funded programs and had the required capacity for addressing financial, procurement and M&E issues. Despite the delays in the initial years of the project, which delayed disbursements, CONAGUA could Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) disburse all the funds in about half the time allocated. There was compliance with safeguards (discussed in section 11) • Lack of adequate technical staff in CONAGUA contributed to inadequate supervision as the financial and procurement aspects of the program absorbed practically all available staff time (Borrower's ICR, page 53). This contributed to the unsatisfactory performance of the water utilities in smaller towns. The implementing agency lacked capacity for monitoring the indicators for the results framework. • The more innovative output-based disbursement project component was not implemented within the time-frame and was eventually cancelled in 2014. Implementing Agency Performance Rating Moderately Unsatisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The four key outcome indicators, increase in revenue collected by the utilities, decrease in energy consumption of water utilities, improvements in the commercial efficiency of water utilities and number of utilities which showed increase in global efficiency (defined as encompassing both physical and commercial efficiency) were appropriate. Given that the project was part of a much broader federal program providing support to utilities of which the Bank supported activities represented just four percent of total funding, it is not clear the extent to which the realized outcomes could be attributed to this project. Lack of baseline data and the initial situation in the utilities in terms of efficiency in the utilities contributed to the imprecise measurement of the impact of resources (Borrower's ICR, page 54) and there were no mechanisms envisaged at appraisal for gathering the information necessary to calculate the indicators. b. M&E Implementation The data for monitoring performance was to be collected both by the participating water utilities and by the National Water Commission (CONAGUA). Intermediate indicators pertaining to Component One activities were not modified following the dropping of these activities. Although CONAGUA's weak capacity to track and monitor the indicators was acknowledged at appraisal, plans to bolster the M&E team were not carried out in a timely fashion, due to the three-year implementation delay associated with insufficient federal budgetary allocation to the program. Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) c. M&E Utilization The indicators were used for monitoring project performance. The team clarified that the indicators were originally designed with the intention of feeding into the national information system under Component One, as they were a continuation of the pilot indicator work started under the predecessor project. However, the indicators were not incorporated into the national information system at project closure. M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was classified as a Category B project. Other than environmental assessment (OP/BP 4.01), two social safeguard policies were triggered: Involuntary Resettlement (OP/BP 4.12): and, Physical Cultural Resources (OP/BP 4.11). At appraisal, activities aimed at improving efficiency-related infrastructure (such as installing electromechanical equipment and minor construction activities) were expected to have low environmental impacts. The safeguard on involuntary resettlement was triggered as a preventive measure as no land expropriation or resettlement was anticipated as a result of project activities. The safeguards associated with Physical Cultural Resources were triggered as the construction works entailed excavations near the site of physical cultural resources. An Environmental and Resettlement Framework was prepared and publicly disclosed as required at appraisal to address environmental and social safeguard issues (PAD, page 14). The ICR (page 8) reports that the project's construction works had no negative environmental, health or safety impacts, including land acquisition or resettlement, and that there was compliance with safeguards. b. Fiduciary Compliance Fiduciary Management. Although the project was to be implemented at federal and sub-national levels, most financial management and procurement activities were to be coordinated by the National Water Commission's (CONAGUA's) central office (ICR, page 23). CONAGUA had extensive experience with implementing Bank- financed and other donor-funded projects (PAD, page 22). Financial Management. A Financial Management Assessment of CONAGUA was conducted at appraisal (PAD, page 12). The assessment concluded that the financial management arrangements of CONAGUA were satisfactory. The ICR (page 8) reports that a slowdown in financial processing due to change in personnel in 2014 was rectified during the implementation phase with additional training of new staff and following this, annual good quality audits were submitted in a timely fashion. Procurement. The procurement arrangements for the project were similar to the ones used for the already implemented and closed Bank-financed (PATME) project (PAD, page 26). The ICR (page 8) notes that despite the constraints of the one-year budget cycle in Mexico, there was compliance with procurement issues (ICR, Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) page 9). c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Unsatisfactory Unsatisfactory Risk to Development There were financial, technical Modest Substantial Outcome and institutional risks. Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Government performance was rated as Moderately Satisfactory and implementing agency performance was rated as Moderately Unsatisfactory. Based on Harmonized Criteria agreed by IEG and OPCS, when the Moderately Moderately Borrower Performance outcome for one dimension is Satisfactory Unsatisfactory in the satisfactory range and the rating for the other dimension is in the Unsatisfactory range, the overall rating for Bank and Borrower performance depends on the Outcome rating. Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) 13. Lessons The ICR (pages 18-19) draws the following main lessons from the experience of implementing this project. (1) Project design needs to take into account the limitations in the legal framework to introduce innovative delivery mechanisms (such as Output-based disbursement) to ensure Borrower's Ownership. In the case of this project, the National Water Commission (CONAGUA) did not accept the component on Output-based disbursement, given that national regulations did not allow it. Consequently, this project activity was dropped due to lack of demand on the part of the water utilities. (2) Adequate mechanisms for reporting and capacity for monitoring are pre-requisites for utility performance improvement programs. In the case of this project, lack of baseline data and a proper assessment of the initial situation in the utilities in terms of efficiency contributed to the imprecise measurement of the impact of resources (Borrower's ICR, page 54) and there were no mechanisms envisaged at appraisal for gathering the information necessary to calculate the indicators. Further, in instances where other investments are underway in the same sector, the project should include measures to track attribution of towards project- specific results. (3) Projects when launched on a large scale should be sufficiently structured to prioritize investments for achieving the best results. Spreading funding among a wide range of utilities of different size, capacities and objectives could dilute the effectiveness and efficiency of Bank resources. Proper enforcement of eligibility criteria could help in narrowing the number or type of sub-national entities to be supported. (4). Utility performance improvements require investments across multiple types of efficiencies. Well- performing utilities can achieve better results with the same level of funding as other utilities. Such utilities can also be targeted to pilot output-based performance programs but it is critical to have the support of implementing agencies before designing innovative components. (5) The Bank's added value for countries with high capacity is in bringing global best experiences and practices to improve sector performance. In the case of this project, resources both in terms of time and effort were expended on processing a number of small procurement packages, instead of focusing on knowledge sharing and sector policy dialogue. To advance innovative approaches that require small-scale procurement, the Bank could consider more flexible procurement rules with higher thresholds that promote efficiency without compromising quality. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is concise and well written. It candidly discusses the problems that were encountered (such as drawbacks in design and problems associated with M&E). There is convincing information on safeguard Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Efficiency Improvement Program (P121195) compliance. The ICR could have provided more details on the wider program of which this project was a part and more details on the envisaged output-based disbursement activity. a. Quality of ICR Rating Substantial Page 14 of 14