Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) Report Number : ICRR0020145 1. Project Data Project ID Project Name P104034 HN Power Sector Efficiency Enhancement Country Practice Area(Lead) Honduras Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-45360 31-Dec-2013 42,300,000.00 Bank Approval Date Closing Date (Actual) 22-Jan-2009 30-Jun-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 30,000,000.00 0.00 Revised Commitment 27,564,762.40 0.00 Actual 27,992,017.40 0.00 Sector(s) Transmission and Distribution of Electricity(67%):Energy efficiency in Heat and Power(30%):Public administration- Energy and mining(3%) Theme(s) Urban services and housing for the poor(57%):Corporate governance(28%):City-wide Infrastructure and Service Delivery(15%) Prepared by Reviewed by ICR Review Coordinator Group Richard L. Berney Robert Mark Lacey Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The project development objective (PDO) is to improve the National Power Company’s (Empresa Nacional de Energía Eléctrica -- (ENEE) operational and financial performance, thus contributing to the sustainability of the power sector in Honduras. ( Financial Agreement page 5) The PDO in the Project Appraisal Document (PAD) was the same. b. Were the project objectives/key associated outcome targets revised during implementation? Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) No c. Components There were three components. Component 1: improving ENEE’s commercial and corporate resource management, (Appraisal US$18.5 million; Actual US$26.3 million) through four activities: • Installation of automated meter reading system (AMR) equipment (Appraisal US$4.8 million; Actual US$4.8 million) that allows remote metering of consumption and disconnection or reconnection of the clients, including: (1) completing ENEE's ongoing effort to install AMR systems for the existing consumers in the "industrial" and "high consumption" categories; and (2) extending the installation of AMR to low voltage consumers (residential and commercial) starting with customers with a monthly consumption above 400 kWh; • Installation of a new automated commercial management system (CMS) (Appraisal US$ 7.3 million; Actual US$7.9 million) that included: (1) the acquisition and implementation of the new CMS; (2) the creation and maintenance of a constantly updated and reliable customer database; and (3) the creation of a call center; • Implementation of incidence recording management system (IRMS) (Appraisal US$3.7 million; Actual US$6.2 million) that included: (1) the acquisition and implementation of the new IRMS; (2) the creation of three distribution operations; and (3) the building of an installations and supply database which would include data on medium-voltage networks and on each customer's connection to the corresponding transformation station; and • Development and implementation of a corporate resources management information system (CRMIS), (Appraisal US$ 2.7 million; Actual US$7.3 million) that included specific modules for: (1) accounting and finance; (2) integrated human resources management; (3) purchases and contracts; (4) logistics; (5) asset management; (6) corporate planning; and (7) regulation. Component 2: Rehabilitation of distribution networks, (Appraisal US$ 14.6 million; Actual US$10.3 million) including: • Replacement of transformers, purchase of related distribution equipment, and disposal of polychlorinated biphenyl (PCB) found in replaced PCB-containing transformers (Appraisal US$4.0 million; Actual US$2.3 million) ; and • Purchase of maintenance equipment, spare parts and materials for distribution networks (Appraisal US$1.06 million; Actual US$8.0 million). Component 3: Strengthening ENEE’s institutional capacity and corporate governance (Appraisal US$6.2 million; Actual US$2.7 million) through four sub components: • Strengthening of the governance in the Recipient's power sector, (Appraisal US$1.8 million; Actual US$0.8 million) including specific actions aimed at: (1) enhancing transparency, accountability and attention to ENEE's customers; (2) improving ENEE's corporate governance; and (3) strengthening the fiduciary aspects of ENEE's operations. • Carrying out an analysis of the long-term financial sustainability of ENEE, (Appraisal US$0.5 million; Actual US$0.1 million) including an analysis as to the required tariff and subsidy structure to support such sustainability; and development of a strategy and business plan for a new distribution and commercialization unit. • Implementation of a communication, outreach and participation program, (Appraisal US$1.4 million; Actual US$0.1 million) including: (1) outreach to the consumer and consumers' participation to save energy and reduce electricity thefts and illegal connections; (2) communication and transparency efforts to renew communication procedures establishing quality standards and an information system that will enable users to monitor and provide feedback on ENEE's performance; and (3) institutional strengthening through the provision of training and technical assistance to improve coordination among all areas in ENEE, both central and regional, in the area of customer service. • Project Management and Project Executing Unit (PEU): (Appraisal US$2.5 million; Actual US$1.7 million) There were three restructurings. The first, on June 17, 2010, involved a change in the structure of implementation institutions, transferring in essence all responsibilities to the Project Executing Unit (PEU) in ENEE. The second, in October 14, 2011, was in response to the Government's request for partial cancellation of funds, and had important implications for project implementation. In order to release IDA funds for a Development Policy Credit, project financing was reduced by SDR1.52 million (US2.3 million equivalent). Because of this, and also because of design deficiencies (see Section 3b below), the targeted number of AMRs to be installed was reduced from 50,000 to 5,000. So as to limit the impact on the development objectives, the installation of new AMRs was redirected to the largest consumers Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) representing around 45 percent of billing, which is similar to the appraisal goal of targeting around 50 percent of consumption. The third, on December 19, 2013, extended the closing date by 18 months. None of the restructurings involved changes in the development objectives. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: At appraisal, the project cost was estimated at US$42.3 million, which included US$3.0 million in contingencies. The actual cost was US$39.3million. Financing : The project was supported by an IDA credit of US$30.0 million. Actual disbursement was US$28.0 million. US$2 million was cancelled during the second restructuring at the Government’s request. Borrower Contribution : At Appraisal, the Government and ENEE were estimated to contribution of US$12.3 million. At closing their total contribution was US$11.4 million. Dates: There was one 18 month extension of the closing date from December 31, 2013 to June 30, 2015 in order to complete implementation of the systems and start the operation of the distribution and commercialization unit. 3. Relevance of Objectives & Design a. Relevance of Objectives The state-owned integrated national power utility, ENEE, was the dominant company in Honduras’s power sector, owning all transmission and distribution systems and about 40 percent of generation, with private independent power producers (IPPs) owning the remaining share of power plants. In 2007 it had financial losses representing over 2 percent of GDP (approximately US$141 million), thus constituting a threat to macroeconomic stability and a constraint to poverty reduction efforts. The Government issued an Emergency Decree in June 2007 to restore sustainable electricity supply and ENEE’s financial health, and strengthen the institutions of the electricity sector. The Government’s strategy, supported by the findings and recommendations in the Bank’s “Power Sector Issues and Options” report (July 2007) were the basis for the project design. The objectives remain highly relevant to the World Bank Group’s (WBG) strategy. IEG’s Review of the 2009-2014 Country Partnership Strategy (CPS) notes that weaknesses in core infrastructure, especially transportation and energy, inhibit economic growth (para.72). Energy sector investment remains a major WBG concern. The 2016-2020 Country Partnership Framework, states that “(the) WBG will continue to promote growth and competitiveness by reducing the cost of energy and increasing reliability of energy supply at the national level.” (para 76). Rating High b. Relevance of Design There was a clear causal chain between the activities supported by the project and the objectives as articulated. ENEE’s operational performance was to be improved through the rehabilitation of distribution networks, the establishment of quality standards, and the provision of an information system that would enable users to monitor and provide feedback on its performance. Its financial performance was to be improved in the medium term by the completion of an AMR system for all customers with monthly consumption above 400 kWh, and in the longer term through an analysis of tariff and subsidy structures needed to maintain its financial integrity, as well as by taking actions to enhance its financial transparency and accountability. However, design also embodied some important weaknesses. First, the project timetable was over-ambitious given the country context and ENEE’s known capacity limitations. For example, with regard to the latter, ENEE had previous experience in traditional distribution investment and AMR, but had none with regard to procurement of the systems of the scope and sophistication financed by the project. Second, implementation arrangements involving two agencies proved unworkable and had to be remedied through a restructuring. Third, insufficient account was taken of the risk of changes in government priorities and of the consequences of new sector policies and associated institutional reconfigurations. Fourth, the implications for the project’s impact of other, larger investments in the sector were not adequately analyzed. Fifth, during implementation, ENEE found that the AMR devices assessed at appraisal did not have the technical features that were needed to collect accurate measurement of consumption of the largest electricity users, and that the more sophisticated devices that were needed were considerably more expensive. It was also found that these new AMR devices required an entirely new communications platform. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) Rating Modest 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective Improving ENEE’s operational performance. Rationale Outputs: • The targets for implementation of the project’s physical investment needed for the rehabilitation of the distribution system were exceeded. • The target for the number of transformers to be replaced was 1000. The goal was achieved, with 1044 replaced. • An Incident Recording Management System (IRMS) was installed and 37 staff were trained in its use. It is working satisfactorily. Outcomes: Initially, the investments contributed to lower technical losses (although the ICR does not specify the extent of the improvement) and some improvements in quality of service. In 2013, one year after project investments, the system average interruption duration index (SAIDI) fell to 795 minutes per year (mpy), compared to 813 mpy in 2012. However, the average for the period 2009-2012 was 647 mpy, and the index rose again in 2014 to 929. The system average interruption frequency index (SAIFI) fell in 2013 to 19.4 interruptions per customer per year (icy) from 20.2 in 2012 and from an average of 22.1 icy over the period 2009-2012. Again, however, the index rose again in 2014 to 20.9. According to the ICR (page 21), the deterioration in the indices was due to a lack of continuity in investments in the grid. There is moreover, an attribution issue, in that it is not possible to separate the effects of the project-financed investments from those of other, larger rehabilitation investments financed by the IDB and CABEI, which are described in the PAD. Rating Modest PHREVISEDTBL Objective 2 Objective Improving ENEE’s financial performance. Rationale Outputs: • Given the project’s limited resources, it was necessary to reduce the number of ARM devices financed (see Sections 2c and 3b above). As part of the 2013 restructuring, the original target of 50,000 ARMs was reduced to 5000, 1,400 of which had been installed before the project, and 3,600 were purchased under the project, These 5,000 ARMs were provided to only the largest electricity users. An analysis by the Project Executing Unit (PEU) estimated that the revised target would cover 45% of total electricity consumption, not far short of the 50% envisaged at appraisal. By closure, 4810 meters had been installed. • A Commercial Management System (CMS) was installed, 450 staff have been trained in its utilization, and it is reported to be working satisfactorily. • A Corporate Resource Management System was installed, 250 staff have been trained in its utilization, and it is reported to be working Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) satisfactorily. • Legal approval was granted for the establishment of a separate Business Unit and a separate Distribution and Commercialization Unit within ENEE. Key staff for the Units were chosen. However, the Units had still not entered into operation at closure (as had been the intention at appraisal) since their funding had not yet been approved. Outcomes: • Utilization of these management systems has been considerably lower than anticipated. Some systems are not in use, such as the balance of energy module of the CMS, requiring management decisions to become operative, and only 40% of ENEE’s customers have been integrated into the two databases, compared with a target of 66%,. The IRSM final deployment was completed only in 2015, just before the Project closing, and the subcontractor in charge of measuring, billing and collecting electricity bills has not adopted the new systems yet. The operational interface needed to share information with the Ministry of Finance system is not yet fully operational. Separate business units have been legally approved, but are not yet operational. • The financial target was for an increase in the Cash Recovery Index to 10% from the baseline level of 0.63% . This target was not achieved. Cash recovery improved to only 0.67%. • The PEU reported that the target for the improvement in consumer satisfaction and perception of ENEE had been achieved, but the Bank’s task team has questioned the methodology and the consistency of the data presented to support this conclusion. Rating Modest 5. Efficiency According to the ICR, the project had an ex-post economic internal rate of return (EIRR) of 37% and a financial rate of return (FRR) of 60%, compared to an ex-ante EIRR of 14% (PAD, paragraph 44). The ex-post analysis is based on some adjustments to the projections made in the PAD, involving (i) the assumed levels of reductions in consumption of AMR-monitored large consumers as a result of better billing and fraud detection; and (ii) an assumed 2% reduction in technical losses (the same as in the PAD). However, these projections do not appear to be based on proven results or actual experience. There were significant operational and administrative inefficiencies stemming from the design of institutional arrangements and of procurement packages (see Section 11b below). These inefficiencies required a major reallocation of project resources and led to an implementation delay of two years. In addition, the complex political situation in Honduras between June 2009 and January 2010 delayed project effectiveness and project implementation of institutional arrangements. There were five changes in ENEE's management over the project’s life cycle, requiring the Bank team to rebuild support for the project each time through a process of educating the new management about its long term benefits. There were significant delays in ENEE decision-making on critical project implementation issues due to each new management's need to develop a full understanding of the project’s requirements. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) 93.00 ICR Estimate  37.10 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated high and that of design modest. The efficacy of both objectives was rated modest. Regarding the first objective -- improving ENEE’s operational performance – project-supported investments in the rehabilitation of the distribution system contributed to lower technical losses and may initially have helped to enhance service quality. However, the indices of both number and duration of interruptions, after showing some initial improvement, subsequently deteriorated due to lack of continuity of investments. Moreover, part of any improvements realized was partly attributable to support from other development institutions that also financed distribution system rehabilitation. Concerning the second objective -- improving ENEE’s financial performance – investments in the incidence recording management system (IRMS), the Commercial Management System (CMS) and distribution management software have provided ENEE with the tools to fight non-technical losses. However, these tools were not yet fully operational at closure, and the project’s financial targets were not met. Efficiency is rated modest. Although the estimated ex-post ERR is relatively high (37%), the projections used in the analysis do not appear to be based on proven results or actual experience. There were considerable operational inefficiencies resulting in delays and the need for major reallocation of IDA resources. Overall, these are considered to be significant shortcomings, and outcome is rated moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating There are several risks at the sector, institutional, and operational level that can mitigate the project’s achievements. At the sectorial level, there is considerable uncertainty about ENEE’s future structure and its relationship with future private sector operators of the new unbundled entities. This is a result of the Electricity Law introduced in 2014, and a set of decrees that established specific trustees to manage the areas of generation, distribution, transmission, and public lighting with a mandate to restructure and unbundle ENEE into a holding company with subsidiaries for each activity. The mandate also incudes finding an international investor with capacity to operate the unbundled entities. It is not clear whether the commercial management system introduced under the project would survive such a restructuring of the sector. Private investors would likely have their own information systems that might, or might not, be compatible with those the project financed. In order to ensure sustainability of the project's investments, the Bank has insisted on the need to ring-fence the investments, and the Government has stated its commitment to this endeavor. The practical implications, however, remais unclear. At the institutional level, it is uncertain whether ENEE will have the necessary funding and skilled resources (due to staff reductions) to carry out maintenance, improve and continue implementing the systems. At the operational level, the tools introduced under the project have the capacity to improve performance, while the unit established for the sustainable recovery of revenues based on the automatic meter reading system has developed expertise in managing the system and evaluating results. However, the Government’s decision (for reasons unexplained in the ICR) to keep using the existing outsourced meter reading system in parallel with the new in house commercial management reading and billing system leaves ENEE without control of the billing and still dependent on a third party for client information and losses management. However, core investments implemented through the project are not at risk. Irrespective of the final structure of the sector and arrangements for the management of ENEE, the measurement devices installed should eventually reduce the need to rely on the outsourced metering and invoicing company; hazardous PCBs have already been adequately treated; the sector will continue to benefit from project-supported enhancements in the distribution network; and the installed system for Enterprise Resource Planning is compatible and widely used. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) a. Quality-at-Entry Although project preparation was grounded in solid analytical work and coordination with other development partners, there were four significant shortcomings. First, the use of two institutions to implement the project proved unworkable and had to be changed in an early restructuring, Second, the implementation schedule was over ambitious and did not provide sufficient mitigating measures to manage risks. Third, the AMR devices assessed at appraisal did not have the technical features to meet ENEE’s requirements. Fourth, there were important weaknesses in M&E design: the lack of baseline values in relevant indicators hindered supervision and a timely response to implementation issues; and. outcome targets for improvements in loss reduction and cash recovery did not take into account the relatively small impact that the operation would have compared to projects funded by IDB and CABEI. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Despite the difficulties related to changes in both Government and ENEE’s management, the supervision team continued efforts to ensure that changes in the structure of the power sector would be consistent with the objectives of the project. There was a close and comprehensive supervision of environmental safeguards in the activities related to PCB. The Bank worked closely with the IMF to ensure that the IMF’s and Bank’s programs in the power sector were mutually supporting. However, there were two significant shortcomings in the Quality of Supervision. First, the Implementation Status Reports did not always adequately reflect the status of the project or inform management that indicators were not being achieved, and were unlikely to be achieved, and that, as a result, the project needed substantive review and restructuring. Second opportunity was not taken during the third restructuring to adapt the results framework and indicators to feasible outcomes with a clear view on what could be done in the time remaining. Instead an optimistic expectation that all outcomes would be achieved with the 18 months extension was presented. In general, the Bank should have taken advantage of the opportunities presented to propose deeper and more thorough restructurings. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance The Government changed ENEE Management five times during the project life cycle, with resulting delays in decision making and the need for the Bank to re-build support for the project with each change. At times, the lack of sufficient counterpart funds delayed project implementation. For instance, during 2013 the project need around US$6 million of counterpart funds, but the Government allocated only US$ 2 million. The Government did provide stronger support during the final implementation period, prioritizing the implementation of the remaining programs. It prioritized the implementation of the remaining project activities in the months immediately prior to closure. However, this did not fully compensate for the negative impact of lack of continuity in ENEE´s management support and changes in Government. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance ENEE was the implementing agency, assisted administratively by a Project Executing Unit (PEU). The initial institutional arrangements had two implementing agencies -- a project coordination unit (PCU) in the Presidential Commission for State Modernization (CPME) which Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) was responsible for procurement, monitoring, evaluation, and fiduciary matters; and a Project Execution Unit (PEU) in ENEE, which was responsible for technical matters. This structure was ineffective, but once all responsibilities had been transferred from the Unit In the Presidential Commission to the PEU in ENEE, the Unit’s performance was generally satisfactory . It actively sought ENEE management backing for the project's implementation, but its reports to the Bank proved to be overly optimistic in their assessment of its ability to address effectively the effects of management delays and lack of support. Implementing Agency Performance Rating Moderately Unsatisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design Due to ENEE´s lack of reliable information systems, the appraisal team decided to postpone the setting of baselines until investments under Component 1 were carried out and systems implemented. The intention expressed in the PAD was that monitoring data would be collected from those systems. However, as a result of delays and changed priorities, several baselines remained unquantified throughout most of the implementation phase. b. M&E Implementation Lack of quantitative baselines resulted in meaningless indicators, preventing adequate monitoring of progress in implementation. The project was blind regarding the movement of key performance indicators. For example, monitoring activities focused on the number of AMR procured but not on the number installed. As a result there was some confusion regarding targets being declared as achieved by the PEU. c. M&E Utilization The M&E framework was never used as a supervision tool or for policy purposes. M&E Quality Rating Negligible 11. Other Issues a. Safeguards The project was classified as Catefory “B” for environmental assessment purposes. The only safeguard policy to be triggered was OP 4.01 (Environmental Assessment) An Environmental Management Plan (EMP) was prepared and found acceptable by the Bank (ICR, p. 19) OP 4.01 was triggered because of activities related to the construction of a storage place for contaminated PCB material. The ICR reports that the Bank closely reviewed the design and construction of the storage works and provided examples according to international standards. During implementation, the Bank also provided guidance on improving ENEE´s environmental management practices related to PCBs and other rehabilitation residues. Storage facilities were completed by closure. The ICR reports that the processes and manuals established for the disposal of PCBs go beyond the life of the project, and would be expected to have a significant positive impact on the environmental and on health. During a routine site visit, the Bank supervision team discovered that the works were not being carried out in accordance with the Bank´s standards on Health and Safety (H&S) for the workers. This was reported to ENEE and its PEU. ENEE provided additional resources to Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) monitor the works better, and requested the contractor to implement additional H&S measures and to have an H&S specialist on site. The PEU also intensified their supervision and the Bank contracted additional support from a safeguards consultant to improve its own control. b. Fiduciary Compliance There were moderate shortcomings in financial management at the start of the project . Performance was downgraded in 2012 due to the inability of the PEU to prepare acceptable financial statements, and the first audit report for the project was delayed by more than five months. The ICR reports that capacity building and Bank support addressed these issues, and allowed the project to close with clean audits and no pending financial management issues. Furthermore, ENEE’s financial statements could not be satisfactorily audited in 2010. The lack of reliable information and acceptable internal control procedures were the central arguments raised by the auditor for not subscribing to the statements. The ICR does not state whether the external auditor’s opinions regarding project accounts were qualified or not. Procurement: Procurement efficiency was adversely affected by weak design of implementation arrangements and of procurement packages. Initial institutional arrangements (see Section 9b above) created unnecessary administrative complexities, which led to major procurement delays. In the project’s first restructuring, all responsibilities were shifted to ENEE’s PEU. The ICR states that procurement of goods and services was carried out more efficiently after this transfer (para. 55). With regard to the procurement packages, all three computer management systems were initially designed as a single package (in accordance with Bank recommendations). This required highly sophisticated documentation. The requirement that a single bidder have all the expertise required led to the failure of the initial tender. Re-launching the bidding with separate packages involved a two year delay. Moreover, in order to be useful to ENEE, the Management Information System needed a broader scope than originally designed as well as additional complementary equipment, leading to higher costs and a reallocation of IDA financing. c. Unintended impacts (Positive or Negative) None. d. Other None. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Unsatisfactory Moderately Unsatisfactory --- Risk to Development Outcome Substantial Substantial --- Bank Performance Moderately Unsatisfactory Moderately Unsatisfactory --- Borrower Performance Moderately Unsatisfactory Moderately Unsatisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HN Power Sector Efficiency Enhancement (P104034) IEG draws the following lessons from the experience of this project: • A critical eye on the feasibility of projected implementation schedules is essential for evaluating project progress and the need for additional Bank interventions. The Bank’s team had an overly optimistic view of project development, even though implementation delays meant that some outcomes could not be achieved in the time left. As a result, opportunities were missed to make changes that could better reflect the project’s situation and improve the likelihood of achievements. • A successful implementation of modern consumption monitoring systems is necessary, but not sufficient, to achieve a reduction in commercial losses. Outcomes, such as reduction in non-technical losses and CRI, require complementary intensive loss reduction and collection programs that go beyond an investment or technical assistance that can be provided in a project. They require sustained and continued high level support and commitment by the Government and power company management. • An adequate project design needs to include baseline data on which to judge progress towards desired outcomes. The indicators and result framework need to build on what already exists and use historical data to assess reasonable targets. Even low quality data prior to implementation would have been more helpful in judging a project’s progress than having no baselines and having to withhold critical analysis until reliable indicators to track progress became available. • Improvements in operational indicators need to be consistent with the activities that the project supports. To achieve this end, operational indicators should include only the areas covered by the project, and where the outcome can be reasonably attributed to the project’s investment, rather than to a larger set of investments outside the project’s control, even when defining indicators by project area is more data intensive and requires the power company to have and implement adequate data collection and validation. The overly inclusive nature of outcome indicators in this project made them unusable for judging the project’s outcomes. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is well written and comprehensive. It clearly and candidly identifies the shortcoming in implementation and the risks to achieving and maintaining the project's development objectives. It puts considerable focus on the fact that during the restructurings there was a lack of attention to updating the outcome targets, and suggests that the project outcome could have been improved if these targets had been modified (reduced). This reflects the ICR's view that the Bank team had been over-optimistic during both preparation and implementation, and that the indicators included in the results framework were inadequate to monitor progress. It would, in addition, have been useful to have discussed the need for (and possibilities of) refocusing the implementation of the project in light of the changing circumstances. The discussion of the performance of the “implementing agency” could usefully have included a cross reference to the analysis of the entity's management performance and continuous managerial changes considered elsewhere in the ICR. As noted in the ICR, the coordination between ENEE management and the PEU encountered many obstacles. There is no discussion of external audits of the project accounts. a. Quality of ICR Rating Substantial