ICRR 14513 Report Number : ICRR14513 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/30/2014 Country : Haiti Project ID : P098531 Appraisal Actual Project Name : Ht Electricity Project US$M ): Project Costs (US$M): 6.00 11.00 L/C Number : CH251; CH510 Loan /Credit (US$M): Loan/ US$M ): 6.00 11.00 Sector Board : Energy and Mining US$M): Cofinancing (US$M ): Cofinanciers : Board Approval Date : 08/03/2006 Closing Date : 02/28/2010 08/31/2013 Sector (s): Power (100%) Theme (s): Infrastructure services for private sector development (67% - P); Regulation and competition policy (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Ranga Rajan Roy Gilbert Christopher David IEGPS1 Krishnamani Nelson 2. Project Objectives and Components: a. Objectives: �To assist the Recipient in achieving sustainable improvement in the quality of electrical services to customers and strengthening the financial and operational performance of its power utility .� Financing Agreement, (Schedule 1, page 4) and Project Appraisal Document (PAD, page 13). b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 11/04/2009 c. Components: EDH) management systems and practices towards a more customer 1. Improvement in Electricite d'Haiti ’s (EDH) oriented approach : (original appraisal cost US$ 2.64 million: additional financing cost US$ 3.68 million: actual cost at closure US$ 4.56 million). Activities included, implementing a corporate Customer Management System (CMS) and a Technical Service Management System (TSMS), and implementing the new rules and procedures for supporting EDH's billing practices and quality of customer service .. 2. Improvement in quality and reliability of services and increased revenue collection of EDH for selected groups of customers : (original appraisal cost US$ 3.2 million: additional financing cost US$ 3.51 million: actual cost at closure US$ 1.84 million). This component financed the costs of implementing the new approach in two focus areas of the socio-economically mixed zone, which included several poor neighborhoods and large commercial customers . Activities included installing remote meters for large customers, installing remote meters for accurately measuring the production of the Independent Power Producers, and strengthening the anti fraud unit of the EDH for protecting commercial losses due to electricity theft .. 3. Participatory approach, project management, monitoring and impact evaluation, and replication strategy (original appraisal cost US$ 1.58 million: additional financing cost US$ 1.08 million: actual cost at closure US$ 1.36 million). Activities financed the costs of communication, coordination and monitoring indicators, and the costs of technical assistance to the Ministry of Transport, Public Works and Communication . An additional component was added following the approval of additional financing to the project on July 29, 2009. (discussed in section 2d). 4. Management Support to EDH (original appraisal cost US$ 0.00 million: additional financing cost US$ 2.57 million; actual cost at closure US$ 2.83 million). Activities financed a program of technical assistance to the commercial, technical, financial/administration and planning divisions of EDH through recruiting four international experts . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs : The appraisal cost was US$ 6.00 million. Additional financing of US$ 5.00 million was approved on July 29, 2009 following a project restructuring (discussed below). Actual cost at completion was US$ 11.00 million, 183.3% of the original appraisal estimate. Project Financing : The project was 100% financed by the Bank grant . There was parallel financing from other donors for other complementary sector activities by the EU, the Private Public Infrastructure Advisory Facility, the Inter-American Development Bank (IDB), and the USAID (PAD, pages 11-12). Borrower's Contribution : None was planned, and nothing materialized . Dates : The project was restructured thrice . The first restructuring on July 29, 2009, approved additional financing and extended the project closing date by 24 months (from February 28, 2010 to February 28, 2012). An important change was also made regarding project outcome indicators with this restructuring . Unlike in the original project design, where outcome indicators were to be measured for the project targeted area with 9,000 customers, outcome indicators henceforth were to be measured on EDH's customer base of 185,000 customers. The second restructuring was made in June 2010, in the wake of the earthquake in January 2010 (discussed in section 4). This restructuring, besides reallocating funds between disbursement categories, reset the indicator's baseline from 2006 to 2011, revised target values and extended the project closing date by 12 months (from February 28, 2012 to February 28, 2013). The third restructuring in February 2013, reallocated funds between components and extended the project closing date by an additional six months (from February 28, 2013 to August 31, 2013). The project closed 42 months behind schedule on August 31,2013. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Rating : Substantial Relevance at appraisal: The government's Interim Cooperation Framework for the 2004- 2007 period identified the goal of "improving access to basic services", through "establishing continuous and reliable electricity services .", as a priority.(PAD, page 8). The country assistance strategy issued in 1996 highlighted the need for "laying the base for sustainable growth, ", through among other things, investments in the power sector . It is not clear whether the project objectives were relevant to the Bank priorities at the appraisal stage since there was no country assistance strategy at that time . The project development objectives were relevant to the Bank strategy for Haiti during the early part of the project implementation phase. The first pillar of the Country Assistance Strategy (CAS) of May 2009 - the first CAS to be prepared since 1996 - aimed at promoting growth and development through among other things, increasing electricity generation capacity and through increasing the sector's productivity through the rehabilitation and maintenance of its existing installations . Current relevance: The project development objectives namely, (i) better quality of electrical services, and (ii) stronger financial and operational performance of the power utility were relevant to the government’s 2010 Action Plan for National Recovery and Development of Haiti issued in the wake of the earthquake for the following 18 months. The plan identified the need for the reconstruction of the energy sector . The development objectives remain relevant to the Bank's FY2013-FY2014 Interim Strategy Note for Haiti. The note identifies the need for "sustainable reconstruction through improved access to electricity and support for the commercial viability of the sector ." b. Relevance of Design: Rating : Modest Assessment of the results framework (as per PAD) For the most part, the linkages between the project activities, their outputs and the intended outcomes (i.e. objectives) of project were logical. These intended outcomes themselves were clear . The project results framework covered the necessary linkages for achieving the development objectives . To help achieve the first objective, " achieving sustainable improvement in the quality of electrical services to customers", the. design incorporated installing a corporate commercial management system (CMS) and a technical service management system (TSMS) customer database for EDH. The outputs of these project activities can be expected to improve the quality of services rendered by EDH to customers .. To help achieve the second objective of "strengthening the financial and operational performance of the power utility ." the design incorporated improved billing practices, through installing remote meters for large customers (commercial and industrial enterprises ) who accounted for about 70% of EDH total revenues (PAD, page 59), and through training of EDH staff. These activities can be expected to contribute to improving the financial and operational performance of EDH. 4. Achievement of Objectives (Efficacy): The project development objectives were : (1) To assist the recipient in achieving sustainable improvement in the quality of electrical services to customers .(2) To assist the recipient in strengthening the financial and operational performance of its power utility. The two objectives are assessed separately . The ICR (page 10) notes that there were two distinct phases of the project . This was due to earthquake in Haiti in January 2010 which caused severe destruction to the government's physical and institutional infrastructure, EDH's dispatch center, some power distribution infrastructure (including the anti-fraud unit) were destroyed. The EDH's customer database was rendered obsolete, since 1.5 people were either placed in refugee camps or moved to the provinces (ICR, page 12), and in March 2010, EDH could collect only about a third (30%) of its revenue entitlement (ICR, page 10). The assessment of efficacy takes account of this unexpected event .. Objective 1: To assist the recipient in achieving sustainable improvement in the quality of electrical services to customer s: Negligible The two database management systems established by the project (Customer Management System and the Technical Service Management System ], were installed for improving the EDH service delivery to customers, reflected through the number of customers who received electricity for at least 6 hours per day (ICR, page 15). The ICR (page 15) reports that although the two systems were in place, they were not operating at full potential at the project closure stage . 224,052 customers were in the database of the two management systems at the project closure stage, as compared to the revised target of 185,315. The number of customers in the database was higher than the originally targeted figure of 9,000 customers, as indicators were measured on the expanded nationwide customer base (discussed in section 2d). (ICR, Results Framework Analysis, Data Sheet ). The customer per employee ratio went up from 80 to 98, as compared to the revised target of 100 (ICR, Results Framework Analysis, Data Sheet).. Technical assistance was provided as targeted to EDH's management for implementing the new procedures in the areas of financial management, internal control, human resources and technical management .(ICR, Results Framework Analysis, Data Sheet). An original aim of installing prepaid electricity meters in slums, residential areas, and medium sized commercial areas was initially subject to implementation delays and then dropped in the wake of the earthquake (ICR, page 24). Outcomes : The EDH;s customer database was rendered obsolete by the earthquake in January 2010. Towards the end of 2011, the project conducted an extensive field survey spread over four months for updating the customer database. The ICR (page 22) notes that the compilation of this database which is ongoing, had current information on 70,000 customers at the project closure stage, and that this database was particularly useful for assessing the funding needs for rebuilding the power system . [Although the ICR informs that 100% of all customers in the EDH;s power distribution circuit currently receive electricity for at least 6 hours a day at project closure up from a 2005 baseline of 20% (ICR, Results Framework Analysis, Data Sheet)--the ICR (page 15) also reports that this "improvement in service levels resulted mostly from exogenous factors, most importantly the commissioning of the E -Power oil fired power station." For these reasons, this result cannot be attributed to the project or counted as one of its outcomes .] Objective 2: To assist the recipient in strengthening the financial and operational performance of its power utility : Modest . Of the 500 remote control meters supplied to EDH for monitoring its performance in supplying power to large customers who account for 70% of EDH's revenue, 72 were installed, and 6 remote meters were installed for verifying the power generation of the Independent Power Producers (IPP's) at the project closure stage (ICR, page 16). Supplies, furniture and equipment for the Anti -Fraud Unit's offices were purchased, and the unit's staff was trained, as targeted (ICR, page 24). 296 members of EDH staff were trained as compared to the target of 435. (Results Framework Analysis, The Data Sheet).. Financial tools (an accounting system, a manual of procedures, and tools for implementing inventory of assets ) were provided to EDH for improving its financial performance, as targeted (ICR, Results Framework Analysis, Data Sheet). The power utility company recruited four deputy directors (in the departments of planning, administration, finance and commerce) for strengthening the special unit in charge of the Energy Sector Management in the Ministry of Public Works, Transport and Communications (ICR, page 27). Outcomes : EDH recovered 26.7% of its cost in 2013 as compared to 11.5% of its costs in 2011 due to the installation of meters. This represented a 67% and 53% cost recovery as compared to the revised and original target, (ICR, page 15) a modest contribution to strengthening its financial performance . Despite the efforts to strengthen the anti fraud unit, there was a negative outcome with the technical losses due to electricity theft increasing from 45.9% to 58.3% between 2007 and 2012-2013 (ICR, page 24). The ICR (page 15) uses an inappropriate indicator - the number of staff trained- for measuring improvements in the operational performance of EDH at the project closure stage . A more appropriate one would have one showing how, after the training, the staff contributed more effectively to EDH's financial and operational performance. 5. Efficiency: The assessment of efficiency takes account of the effects of the earthquake . Negligible : The ICR (page 16) reports an ex post economic Internal Rate of Return (EIRR) of 15% (ICR, page 31). The ICR does not provide information on the ex ante EIRR . The PAD (page 102) reports that economic evaluation was done for two separate projects activities at the appraisal stage -installing pre paid electricity meters in poor neighborhoods and installing remote meters to major customers . The ex ante EIRR for the two activities were 15% and 42%. The former activity was dropped in the wake of the earthquake, and it is not clear from the ICR, how the ex post EIRR compares with the ex ante EIRR. The ICR (page 33) reports that the financial Internal Rate of Return (FIRR) was 191%, as compared to the ex ante FIRR of 91%. There were significant administrative inefficiencies in the initial years of the project, which in turn contributed to implementation delays and cost overruns . The ICR (page 11) notes that the project became effective a year after the Bank approval, and there were no disbursements for the first year. The ICR (page 11 and page 14 ) notes that the actual cost of installing the management systems was much higher than estimated . For instance, the actual cost in 2009 was US$ 3.5 million. This represented 132% of the estimated cost for this component (ICR, page 11). The ICR (page 25) notes that the project underestimated the risk premium that suppliers placed on EDH . Two tendering process for installing remote metering systems were unsuccessful, since the smallest offer, US$ 3.6 million, was more than twice the estimated budget . A pilot of installing prepaid electricity meters in the poor neighborhoods was dropped in the wake of the earthquake (ICR, page 24).. The ICR (page 10) notes that the cost overruns were magnified by the implementation delays in the wake of the four tropical hurricanes that hit Haiti over a period of two months in 2008, as the EDH staff were mobilized for post emergency repairs. These delays were exacerbated by delays after the earthquake in 2010 that rendered the customer database obsolete (ICR, page 12). At closure, the project spend twice as much to try to achieve the originally targeted results, and the project closed 42 months behind schedule. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 15% 44% ICR estimate Yes 15% 41.4% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The objectives remain were substantially relevant to the country priorities and the World Bank Group's strategy for Haiti. The relevance of design is rated as Modest . The first objective - achieving sustainable improvement in the quality of electrical services to customers - is rated Negligible as the outcome could not be attributed to the project . The second objective - strengthening the financial and operational performance of the power utility company is rated Modest. The financial performance of EDH did not improve to the extent anticipated, there was a negative outcome, and the project uses an inappropriate indicator. Efficiency is rated as Negligible. There were significant administrative inefficiencies that in turn resulted in cost overruns and implementation delays even prior to the earthquake . The overall outcome rating is Unsatisfactory . a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Country level Risk : Ongoing benefits from the project are at risk in view of the weak public sector management capacity (ICR, page 9), and it is not clear whether the government will remain committed to energy sector reforms, particularly in view of its discontinuing the public disclosure of information regarding the energy sector . (discussed in section 10b). Haiti also continues to be vulnerable to natural disasters . High . The risk to the development outcome is assessed as High. a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: The ICR (page 7) reports that this project was built on the basis of both previous studies which analyzed EDH's issues, and on the basis of lessons learned from the implementation of five power projects in Haiti from 1976-1999 (Haiti: First, Second, Third, Fourth and Fifth Power Project ). The project was prepared in five months and in coordination with other donors (ICR, page 7). The ICR (page 7) notes, that at the preparation stage, modernizing EDH's information systems and its billing practices was important, both for improving service delivery to customers, and for improving its own financial and operational performance. Modernizing EDH was thus reflected in the project design, but it needed more attention than initially given. The initial project design focused mostly on technical aspects, and underestimated the management changes that were required to accompany the installation of information systems , so that the project offered insufficient support to EDH management. It eventually led to the creation of a new component to address this gap (ICR p.8). But the appraised costs of installing the management systems and remote control meters were significantly underestimated. At the outset, the project also “set targets that could not be reached by a utility evolving in a fragile environment ." Parts of the technical design were not appropriate for the real conditions in Haiti in which the project would have to operate. The technical equipment funded under the project relied extensively on functional communication systems (internet, mobile network, radio frequency ) that required continuous and reliable supply of electricity that was not available in Haiti at the time--precisely one of the problems that the project itself intended to solve . This forced the EDH to purchase backup power generation from Independent Power Producers for communicating with its field agencies (ICR, page 11). The project appraisal also did not properly assess the limited availability of data for the monitoring indicators and the challenges of collecting it . This, in turn, undermined the implementation of M&E.(discussed in section 10) at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: According to the ICR (Data Sheet ), there were 13 missions in all. The team noted that the missions were more frequent in the wake of the earthquake in 2013. At key moments during implementation, the Bank supervision team demonstrated flexibility in response to government requests . It added an additional project component to address the technical assistance needs of the utility company , for example. There were problems associated with the engineering and cost estimates of some procurements . For instance, the first two remote metering systems were unsuccessful, and the cost estimates for the information systems and the remote metering system were well below the actual costs (ICR, page 14) The evidence provided in the ICR is very sparse, and the ICR does not provide information on whether the supervision inputs and processes were adequate, and the extent to which the supervision of this project contributed to ensuring adequate transition arrangements . Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: Although the project development objective was relevant to borrower /government's priorities at the time of appraisal, there is a contradiction in the ICR statement (page 18) which reports that the "government did not take a proactive approach either when the project was prepared, or during its execution ." The government did not take enough steps to ensure the transparency of the energy sector and efficiently manage its power utility (ICR, page 18) For instance, the lack of involvement on the part of the government was evidenced by the delays in the publication of the Electricity sector Dashboard . The dashboard intended making the electricity sector transparent, through public disclosure of information on sector variables (such as the amount of energy generated and distributed by EDH, and its financial performance). The irregularities and the eventual discontinuation of public disclosure of information through the dashboard was due to the lack of coordination between EDH, the Ministry of Public Works, Transport and Communication and the Ministry of Finance. The ICR (page 18) notes that only about a dozen of these dashboards were published during the project implementation period . Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The Project Coordination Unit (PCU). located within the Direction Generale of EDH, was in charge of project implementation from start up to completion. The PAD (page 25) notes that the project coordination unit was housed in EDH headquarters to ensure coordination of their activities with that of the EDH Board . The EDH board was chaired by the Minister in charge of Transport, Public Works and communication, and the Board included representatives of the Ministry of Finance, Justice and Planning (ICR, page 10) The ICR (page 10) notes that there were organizational and management issues with EDH . The ICR (page 10) notes that six different Managing Directors headed EDH during the project period, and project execution was subject to temporary delays with each change . The main management issues that affected implementation were : (i) lack of accountability of the different sub project leaders, managers and Directors at EDH, and (ii) the EDH Board meetings to endorse important decisions linked to project were very infrequent .. The ICR (page 19) notes that the staffing of the PCU itself maintained continuity, and this enabled the main elements of the project to be put in place . The ICR provides no clear information or assessment about the implementing agencies performance, and there is no evidence of its commitment to achieving the project objectives, how well its implementation arrangements worked, and its effectiveness in coordinating among stakeholders, contractors and beneficiaries . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: To monitor the first PDO "To achieve sustainable improvement in the quality of electrical services to customers ." M&E Design included two outcome indicators - improved quality of service provided by EDH in the project zone and improved quality of service provided to large customers ." To monitor the second PDO." To strengthen the financial and operational performance of its power utility ." M$E design included two outcome indicators - Increases in the revenues billed by EDH and increases in the revenue collected by EDH. The M&E design had an additional output indicator - Endorsement by EDH and the Government of a final proposal on how to replicate the project pilot to reduce losses in the other areas served by EDH . The monitoring arrangements included the design of a power sector performance dashboard for improving the transparency in the electricity sector that was set up (discussed in section 9a). The ICR does not provide information on the date when the dashboard was set up . The overall quality of M&E design was modest . b. M&E Implementation: The project restructuring of June 2009 reduced the number of outcome indicators from 5 to 2 and altered the interpretation of the outcome indicators . The monitoring indicators for the first PDO was replaced by " improvement in the quality of service provided by EDH " and the indicators for monitoring the second PDO was replaced by "improvement in the cash recovery index ." by EDH. The number of output indicators was reduced from 20 to 12.. The ICR (page 12) notes that the "the absence of a reporting culture in EDH " was the main issue for M&E implementation. As the ICR does not explain, however, what this "lack of culture" was, it is not clear what its implications were and how it hindered M$E implementation . Data collection was weak. It is not clear from the ICR (page 13) what were the constraints associated with data collection were, and whether the data was collected or not . The ICR (page 13) however reports that there were "difficulties in confirming project implementation progress .". The ICR (page 13) notes that the publication of information of the energy sector variables through the power sector performance dashboard was uneven right through the implementation period, and eventually the publication was discontinued. c. M&E Utilization: The ICR (page 13) notes that the customer database collected by the Bank proved very helpful in the preparation of the Post-Disaster Needs Assessment (PDNA) after the earthquake as EDH was not able to provide the required baseline data to the team in charge of assessing the funding needs for rebuilding the power system. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The ICR (page 13) reports that the project did not trigger the Bank's social or environmental safeguards [ICR, page 13).The PAD (page 37) notes that the project components were of a managerial and financial nature with no expected adverse social or environmental impacts . The ICR (page 13) notes that there were no adverse social or environmental impacts during the project implementation phase . b. Fiduciary Compliance: Procurement : The ICR (page 12) notes that project complied with the Bank's procurement and consultant guidelines, and that the procurement reviews were deemed to be satisfactory . No cases of misprocurement were reported by the ICR Financial Management : The ICR (page 13) reports that although the submission of financial audits was late by several months for the first reports, all annual audit reports since 2011 were submitted before the required deadline s, and all of them were deemed to be satisfactory. Although the ICR does not provide information on whether the financial reports were qualified or not, the ICR (page 13) reports that audit reports were satisfactory and of high quality . The ICR (page 13) notes that EDH's financial accounts had not been audited since 2006 and the lack of EDA accounts remains a problem to date.. Procurement : c. Unintended Impacts (positive or negative): d. Other: 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory Design was modest. Efficacy of the first Unsatisfactory objective - "achieving sustainable improvement in the quality of electrical services to customers" is negligible as the outcome could not be attributed to the project. Efficacy of the second objective." "strengthening the financial and operational performance of the power utility company" was modest, as the financial performance did not improve as much, there was a negative outcome and an inappropriate indicator was used. Efficiency is rated as Negligible. There were significant administrative inefficiencies that resulted in cost overruns and implementation delays even prior to the earthquake. Risk to Development High High .. Outcome : Bank Performance : Moderately Moderately There were significant shortcomings in Satisfactory Unsatisfactory Quality at Entry including overestimating the clients capacity to implementing the technical equipment, flaws in technical design, underestimation of costs of the technical equipment and inadequate attention to M&E. The ICR provides very little evidence about the quality of supervision. Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The following lessons can be drawn from this project . Better recognition and knowledge of the implementing agencies' capacity to execute sophisticated technical equipment is particularly important, while designing projects for fragile countries . And if the lack of capacity is identified as a concern, it would be helpful to address this at the design stage itself through adding a technical assistance component. When addressing the issues of poorly run state institutions, it is necessary to focus not only on the capacity of lower level of management, but also on the commitment of higher management . In the case of this project, although the (lower level) Direction team were committed to the project, the higher level management was not, and this in turn delayed execution . Selection of realistic indicators, and more importantly addressing the issue of data collection, is essential for M&E implementation. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is well-written and candidly discusses the problems encountered at the execution phase, particularly in the wake of the earthquake. It also presents a candid picture of the procurement and financial management issues . The ICR’s discussion of efficacy, efficiency and M&E is not always clear . Some other sections of the report, notably on Implementing Agency performance, are very short on information needed for a basic evaluation . There is a contradiction in the ICR. While it says that the project objective was relevant for the government and the Bank strategy, it also reports that the "Government did not take a proactive approach either when the project was prepared nor during its execution." One major shortcoming of the ICR is in not drawing proper lessons from what was an extraordinary project in a challenging environment . The ICR also provides very little information about or assessment of Bank supervision of the project . Despite these shortcomings, the remaining coverage of the ICR, with some candid assessments, sustains an overall positive rating of the report . a.Quality of ICR Rating : Satisfactory