Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) Report Number : ICRR0020053 1. Project Data Project ID Project Name P086379 DJ-Power Access and Diversification Proj Country Practice Area(Lead) Additional Financing Djibouti Energy & Extractives P112253,P130493 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-41200,IDA-H5740,IDA-H7830 31-Dec-2009 18,500,000.00 Bank Approval Date Closing Date (Actual) 01-Nov-2005 31-Dec-2014 IBRD/IDA (USD) Grants (USD) Original Commitment 18,200,000.00 0.00 Revised Commitment 17,768,509.80 0.00 Actual 17,899,392.25 0.00 Sector(s) Other Renewable Energy(76%):Transmission and Distribution of Electricity(22%):Water supply(2%) Theme(s) Urban services and housing for the poor(50%):Climate change(25%):Infrastructure services for private sector development(25%) Prepared by Reviewed by ICR Review Coordinator Group Richard L. Berney Robert Mark Lacey Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The original development objectives (Development Credit Agreement, page 15) were to: • increase the access to, and reliability of, the power sector; and • improve the efficiency of the electricity utility. The objectives were revised as part of a Level 1 restructuring in January 2009 (Project Paper page 2) as follows: Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) • increase the access to the power sector; • ensure emergency reliability of the power generation; and • provide some diagnostic tools to improve the efficiency of the utility. There were two Additional Financings (AFs). The first, a Grant of US$6 million, was approved on June 15, 2010. The second, a Grant of US$5.2 million, was approved on June 12, 2012. While the first AF did not change the development objectives, the second added a further objective: • to reduce the negative effects of drought on water pumping in both rural and urban areas by strengthening the country’s power supply resilience to natural catastrophes, through the creation of heavy fuel oil (HFO) and diesel security stocks. For the purposes of evaluation, this additional objective will be added to those of the restructured project. • At the time of the Level 1 restructuring US$1.93 million of IDA funding had been disbursed, 10.6% of the total final amount disbursed of US$18.2 million (ICR, Annex 1, Table b). This Review will undertake a split evaluation, with the results weighted in accordance with this percentage. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 15-Jan-2009 c. Components There were three original components: • Expansion of Electric Distribution in Balbala (Appraisal US$1.2 million; Actual US$1.2 ). This component covered the extension of the distribution network of Djibouti City to reach several unserved public and residential neighborhoods of Balbala, a large low-income area in the Western outskirts of Djibouti City. It was to support the installation of 2,000 new connections, expected to provide electricity to about 3,150 low income households. • Pilot Wind Farm in Arta (Appraisal US$4.9 million; Actual US$0.0). This component was to support a pilot wind farm near Arta, a small town west of Djibouti City that is connected to the capital city’s grid. The lessons and experience gained during this small scale pilot were to contribute to determining whether or not Djibouti had the potential to develop viable wind-based electricity generation on a larger scale. This component was cancelled. • Technical Assistance (Appraisal US$0.5 million; Actual US$2.2). This component focused on efficiency and performance improvement in the power sector. It included an Electricity Tariff Study that was, inter alia, to look at the financial issues related to access for low-income consumers; an Electricity-Water Joint Commercial Management Study, the aim of which was to reduce the respective costs of these utilities by establishing a joint billing service, and by reducing the accumulation of inter-enterprise arrears; and an Electricity Loss Reduction Study. The 2009 Level 1Restructuring substantially changed several components: • The Pilot Wind Farm component was formally dropped because the received bids were several times higher than the project funds available to finance this component. • An Emergency Heavy Fuel Oil (HFO) procurement support component (Appraisal US$4.9 million; Actual US$4.9 million) was added. This component used the funds formerly assigned to the cancelled pilot wind farm. • The Tariff Study was dropped because the African Development Bank had already financed and initiated a similar study. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) • The Electricity-Water Joint Commercial Management Study was replaced with a Water and Sanitation Client Management Study for the water utility. The ICR - explains that the joint study was dropped because one of its objectives was to identify staff redundancies in the two organizations, which would not have been politically feasible to implement (ICR para. 33). The First Additional Financing (AF1) in 2010 [US$6.0 million] introduced two new components, both of which were designed to improve EDD’s efficiency through loss reduction measures through: • Piloting the deployment of about 3,000 smart meters (Appraisal US$2.0; Actual US$2.0 million) • Replacing the emergency 20 kilovolt (kV) line from Djibouti to Arta (Appraisal US$3.9 million; Actual US$3.8 million.) The 2012 Second Additional Financing (AF2)[US$4.5 million] introduced a new project component to support implementation of the revised development objectives: • Creation of a “Petroleum Security Stock” (Appraisal US$4.5 million; Actual US$4.3 million). This component was originally designed to establish a five day buffer stock of Heavy Fuel Oil (HFO) and Diesel Fuel Oil (DFO). The HFO was to be used for emergency supply for urban pumping stations during exceptional droughts, and the DFO was to be used for an emergency supply for rural pumping stations during exceptional droughts. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates PROJECT COSTS: The total estimated project cost (including the two Additional Financings) was US$21.5 million. Actual project cost was US$21.1 million. FINANCING: The originally approved IDA Credit was for US$ 7.0 million, including US$0.5 million as a Project Preparation Facility. US$0.4 million of the Credit were undisbursed at closure and cancelled. The first Additional Financing (AF) Grant, in the amount of US$6 million was approved on June 15, 2010, and the second, also a Grant, in the amount of US$5.2 million, on June 12, 2012, bringing the total IDA financing to US$18.2 million. The AF Grants were fully disbursed. BORROWER CONTRIBUTION: The Government provided a cash contribution US$ 0.3 million to cover (i) the installation of the electric distribution component and some of the PMU’s operating costs. EDD contributed US$3.1 for network rehabilitation in the project area and to expand the remote metering program. DATES: There were four extensions of the original closing date of December 31, 2009: in December 2009, by six months to June 30, 2010,to provide time for the completion of the ongoing activities (fuel purchases, grids rehabilitation, studies), and for the preparation of the first AF; in December 2010 by 36 months, to June 30, 2013, to align the original project deadline to that of the AF1, and to utilize the undisbursed savings of the original project; in March 2012 by 9-months to March 31, 2014; and in March 2014, by another nine months to December 31, 2014, when the project finally closed. Both the last two extensions were needed to complete project activities. 3. Relevance of Objectives & Design a. Relevance of Objectives Original Objectives: The power sector, with its high cost and limited opportunities for lower cost new generation facilities was, and still is, one of the key constraints to poverty alleviation in Djibouti. In the Bank's 2005 Country Assistance Strategy (CAS), the third pillar of Djibouti’s Poverty Reduction Strategy focuses on development through reducing the costs of production factors through improved efficiency of investments in the infrastructure sectors and through investments in providing water and basic services in poor neighborhoods (para. 4.3). These priorities were supported by sector strategy notes on the power and water utilities, which focused on the key objectives (inter alia) of addressing key service-delivery constraints through rehabilitation of networks and improving the poor’s access to services (para. 6.8). In its Poverty Reduction Strategy Paper, the Government identified an action plan for the power sector to address key service delivery constraints through rehabilitation and extension of networks and administrative improvements and to explore renewable resources for power generation. The Balbala area was of particular importance for extension of the network because of the new private sector opportunities created by the completion in 2006 of the new port of Doraleh. The 2009 CAS, also identified as key challenges the limited access to electricity, and the sector's high cost relative to the income that it generated (Para 39). The CAS themes were, inter alia, to improve the access to and quality of service of the electricity supply, as one of the Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) important pillars of supporting private sector investment (para. 72). The 2012-2014 Country Partnership Strategy (CPS) continues to support the energy sector objectives of the 2004 and 2009 CASs. The overarching objective of this CPS is to support the Government’s “Vision 2035”, the focus of which is to reduce extreme poverty and build the foundations for shared growth. The second pillar of the CPS is to support the consolidation of gains achieved in infrastructure services, with a renewed emphasis on strengthening the energy and telecommunications programs in support of the government’s efforts to reduce poverty and boost shared prosperity (Para 83). Revised Objectives: The first three revised objectives – increasing access to the power sector, ensuring emergency reliability of the power generation, and providing diagnostic tools to improve the efficiency of the utility – were, for the most part, less ambitious formulations of the original objectives and share the latter’s relevance. However, ensuring emergency reliability of generation was to be achieved through supplementing the Government’s financial support to EDD (which amounted to US$26 million in 2009) and hence easing the consequent strain on the Government's budget brought about by the rapid rise in the cost of imported oil. However, the 2012-2014 CPS no longer considers supporting the budget allocation to EDD to be a priority especially in the light of IDA resource constraints. The fourth revised objective – to reduce the negative effects of drought on water pumping in both rural and urban areas by strengthening the country’s power supply resilience to natural catastrophes, through the creation of heavy fuel oil (HFO) and diesel security stocks – was in order to maintain agricultural output for the poorer rural population during periods of water shortage. Although the drought dimension is not specifically mentioned in the 2012-2014 CPS, the objective is nonetheless substantially relevant to the third Strategic Priority of “Expanding Opportunities Through Reducing Vulnerabilities,” in particular enhancing rural productivity and addressing climatic vulnerability (CPS, pages 23 and 25). Rating Revised Rating High Substantial b. Relevance of Design Original Objectives: There was a reasonably clear causal connection between the activities incorporated in the original design and the achievement of the corresponding objectives, though design could have benefited from more thorough research and analytical work. Increasing access to the power sector was to be realized through the expansion of electricity distribution in Balbala, a low-income area in the western outskirts of Djibouti City. The distribution extension covered about 7.3 km of primary lines (20kV) and 19.7 km of secondary lines (400/220V) and installation of 1,360 kVA of transformer capacity. It was expected to provide electricity to about 3,150 households. In addition, the tariff study (subsequently dropped because a similar exercise was financed by another development partner) included an examination of the financial issues related to access for low-income households. Increased reliability was to be achieved through an additional generating source in the form of wind power. A pilot wind farm was to be constructed. However, this activity was not thoroughly thought through in all its institutional, financial and operational dimensions. The cost of the pilot wind farm was significantly underestimated. It was subsequently dropped after two unsuccessful tenders. Except for joint water-electricity, joint commercial management and electricity loss reduction studies, design included few activities that could convincingly be expected to improve the efficiency of the utility. Moreover, one possible impact of the joint water-electricity study was not fully considered. Insufficient account was taken of EDD’s social role as one of the country’s largest employers, which would have led to political unacceptance of any downsizing recommendations. Revised Objectives: With the funding made available by the first Additional Financing, design incorporated two new activities that would more convincingly contribute to achieving the first two original objectives – increasing access to the power sector and improving the efficiency of the utility: replacing the emergency 20 kilovolt (kV) line from Djibouti to Arta; and piloting the deployment of about 3,000 smart meters. The financing of additional supplies of heavy fuel oil (HFO) was clearly relevant to the objective of ensuring emergency reliability of power generation, since otherwise the Government would have been obliged to use increasingly scarce foreign exchange resources for this purpose. It is, however, noteworthy that no further conditions were attached to this financing, and hence there was no attempt to use any possible leverage to support long term development objectives. The establishment – funded by the second Additional Financing -- of security stocks of HFO and diesel was relevant to the aim of reducing the negative effects of drought in both urban and rural areas. However, the attainment of such a result would be difficult to measure, given the impossibility of segregating the use of electricity for water pumping from other uses within an integrated electricity distribution system. Rating Revised Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) Modest Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Increase access to the power sector Rationale Outputs: • 7.3 km of primary distribution lines (20kV) and 19.7 km of secondary distribution lines (400/220V) were built in the low income district of Balbala along with 6 transformers (20kV / 400V / 200V). • 360 street lights were built along the power distribution lines in underserved areas (supported by the first Additional Financing) and were complemented by 61 street lights funded by EDD • 350 meters of MV network were built in the Weah village (supported by the first Additional Financing). • 72 km of transmission line (20kV) and 127 pylons were rehabilitated in the Arta region (supported by the first Additional Financing) Outcomes: • 3,828 Balbala households and businesses received new electricity connections, compared to an original target of 500 and a revised target of 2,462. • 26,796 people were provided with electricity access; there was no original target; the revised target was 22,800. • The new users replaced kerosene lamps or alternative forms of lighting and other uses of energy by electricity. This enabled them to benefit from considerable improvement in everyday comfort and community activities. • Street lights provided safer neighborhoods and enabled shops to stay open in the evening. The ICR reports that the number of outages in the Arta line corridor fell from 58 in 2009, to 9 in 2010, to zero in 2014 and the first quarter of 2015. Rating Substantial Revised Objective Increase access to the power sector Revised Rationale This objective is the same as the first original objective, and shares its rationale and rating. Revised Rating Substantial PHEFFICACYTBL Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) Objective 2 Objective To increase the reliability of the power sector. Rationale Outputs: • An engineering and grid-connection feasibility study, and environmental assessment for a pilot wind energy farm were implemented; • Bidding and contractual documents for the wind farm were drafted for the Ministry of Energy. Outcomes: • Two tenders were launched. There were no offers for the first, and only one for the second. The component was dropped when the cost of the lone bidder’s proposal was found to be significantly above the funds available for the project. The much higher than anticipated cost made the project uneconomic. Rating Negligible Revised Objective To ensure emergency reliability of the power generation. Revised Rationale Outputs: • US$4.3 million of project funds enabled EDD to purchase of 12,057 metric tons of heavy fuel oil (HFO) Outcomes: • With the HFO purchased through the project, EDD, facing an emergency situation, was able to maintain an acceptable level of power generation. The Government saved US$4.3 million that they would otherwise have had to transfer to the power company to purchase the needed fuel. The aversion of a major energy or fiscal crisis is clearly important for development. There is, nevertheless, an issue as to whether this type of expenditure, standing alone, constitutes an appropriate use of IDA funds. This type of stop-gap solution, when supported by the World Bank Group, would normally be accompanied by policy or institutional actions or reforms designed to prevent or mitigate the recurrence of the crisis situation. In this case, there is no indication of such measures being supported by the project. There is, moreover, little indication in the ICR of the alternatives to the purchase, or whether alternative sources of financing for the purchase were considered. It is noteworthy that further IDA support for Government transfers to EDD would seem to be ruled out in the 2012-2014 CPS. Revised Rating Modest PHEFFICACYTBL Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) Objective 3 Objective To improve the efficiency of the electricity utility. Rationale Outputs • An electricity tariff study, the purpose of which was to review the structure and level of electricity prices and to examine easing financial access to electricity for low income households, was cancelled. It had been initially agreed with the Government and the African Development Bank (AfDB) that IDA would finance the study as part of the project under review, and that the AfDB would support the implementation of the recommendations. However, as it took longer than anticipated to reach project effectiveness, it was decided that AfDB should finance the study. • An electricity loss reduction study designed to identify where major losses were taking place and to recommend appropriate action, was carried out. • A joint electricity-water joint commercial management study, to assess the feasibility of joint management of EDD and the National Water Authority (ONED) consumer bases, and hence reduce operating costs for both utilities, was dropped. According to the ICR, any recommendations to reduce the size of the total labor force in one or both utilities would not have been politically acceptable. Instead, the funding was used for a Water and Sanitation Client Management Study, which was provided to ONED. • With the support of the first Additional Financing, economic and technical feasibility and viability studies were conducted for the pilot smart metering component and for its potential generalization to all customers. • 3,300 smart meters and the associated information system were installed; 20 commercial and nine of the utility’s IT staff were trained on system operation and maintenance. Outcomes • The ICR does not provide information on the recommendations of the AfDB-financed tariff study, nor on the utilization of the loss reduction study. The Water Management study focused only on the operations of the water utility, and contained no recommendations relevant to the operation under review. • EDD achieved a 3.5 percent improvement in the volume of electricity billed and collected during the 2013-2014 period for the 3,300 clients equipped with smart meters. The utility has decided to expand the program, and had purchased 5,000 additional smart meters by project closure. • Technical and commercial losses as a percentage of total electricity billed fell from 21% in 2005 to 12% in 2014. The target for project closure was 3%. Since no information is provided on the possible contribution of the loss reduction study, it is unclear how much of this improvement, if any, can be attributed to the project. Rating Modest Revised Objective Provide some diagnostic tools to improve the efficiency of the utility. Revised Rationale This refers to the studies and technical assistance already evaluated under the second original objective and shares their assessment and rating. Revised Rating Modest Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) PHEFFICACYTBL Objective 4 Objective To ensure emergency reliability of the power generation. Rationale Output: • The US$4.3 million in transferred project funds enabled the purchase of 12,057 metric tons of heavy fuel oil. Outcome: • The Government saved the US$4.3 million that they would have had to transfer to the power company to purchase the fuel needed to maintain an acceptable level of power generation. • EDD's financial position did not improve. • There was no long-term development outcomes. Rating Modest Revised Objective To reduce the negative effects of drought on water pumping in both rural and urban areas by strengthening the country's power supply resilience to natural catastrophes, through the creation of heavy fuel oil (HFO) and diesel fuel oil (DFO) security stocks. Revised Rationale Outputs: • The initial aim was to establish an HFO reserve corresponding to five days of consumption by EDD (urban water pumps are fed by EDD through the grid) and five days of DFO for rural water pumps in villages not connected to the grid (ICR, page 27). This reserve was to be used exclusively to generate electricity to satisfy the needs of water-pumping stations in the event of a major drought. A procedure was to be established to replenish the emergency stocks once the drought crisis was over. However, it subsequently became clear that it would not be possible to establish regulations that would ensure the exclusive allocation of electricity generated by the HFO reserve to water pumping stations. Electricity is fungible. Power generated by the reserve could be deployed for purposes that do not correspond to the priority usage of IDA Grant resources (for instance, air conditioning). It was, therefore, decided to abandon the HFO reserve and instead to use the project funds to purchase diesel fuel oil (DFO), which would be directly supplied to stand-alone local water pumping stations in the event of local drought conditions. A petroleum security stock (PSS) of 4,500 tons of DFO was established for this purpose and given legal stature through a Law approved by Parliament in June, 2014; regulations for the storage and use of the PSS were prepared and implemented; and four Ministry of Energy staff members were trained in fuel stocks management. Outcome: • Due to the infeasibility of earmarking centrally generated electricity, the activity became irrelevant to urban areas which are supplied from the grid. Regarding rural areas, although it is estimated that the 4,500 ton DFO reserve would be sufficient to establish a 360 day security stock, no actual experience of deployment under drought conditions has yet been reported. There is, moreover, some doubt as to the sustainability of the arrangements after project closure. The ICR reports that a cross-subsidy mechanism is being implemented, whereby a tax on large, luxury motor vehicles would finance at least three months of free diesel supply for drought-affected villages. However, no details are provided concerning how much the tax would yield or the procedures to be put into place to ensure its intended use. The ICRR does not discuss EDD's ability to continue paying for the necessary storage facilities. Nor does it discuss whether the Government will allow EDD to gradually reduce this stock to a level more Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) appropriate for an emergency stock of DFO. Revised Rating Modest 5. Efficiency At appraisal, an economic cost-benefit analysis was carried out of the electricity distribution activities (Balbala) and the pilot windfarm, totaling 83.5% of estimated project cost. An ex-post cost-benefit analysis was conducted of electricity distribution (Balbala), the transmission grid rehabilitation (Arta) and the smart metering system, totaling 39.2% of actual project cost. The benefits of the distribution system were the consumption at current tariffs (not the social tariff applied to poor areas); those of the windfarm, avoided cost; those of the grid rehabilitation, the value of the extra consumption made possible by the improvements, and avoided maintenance costs; and those of the smart metering, billing and bill recovery rates for the 3,300 clients equipped with the smart meters. The results may be summarized as follows: EX-ANTE EX-POST ERR (%) % OF ESTIMATED ERR (%) % OF ACTUAL PROJECT COST PROJECT COST Electricity Distribution 21.0 16.4 24.1 6.6 Pilot Windfarm 19.0 67.1 Not applicable Not applicable Transmission Grid Rehabilitation Not applicable Not applicable 18.0 21.3 Smart Metering System Not applicable Not applicable 25,6 11.3 US$4.8 million (22.7% of final project cost) were used to purchase an emergency supply of fuel oil, representing about 2-3 months of the generation plant’s average annual fuel consumption. According to estimates subsequently provided by the project team, this quantity of fuel would serve to produce 43.5 GWh of electricity (about 14% of Djibouti’s annual generation output of 310 GWh). The value placed on this production, and hence an assessment of the economic efficiency of the purchase, would depend on a number of assumptions concerning, inter alia, the cost of obtaining electricity from alternative sources were the generating plant to lack fuel, consumer willingness to pay this cost, the likelihood of obtaining alternative sources of financing, and whether the Government would have purchased fuel in any case in order to avoid an energy crisis. All of these assumptions would need to be made explicit and carefully explained in the ICR. There were administrative and operational inefficiencies in both design and implementation, including major difficulties related to the pilot windfarm (see Sections 8 and 9 below for further details). These led to “a period of inactivity and delays due to issues in the wind component implementation, which lasted roughly three years, until end-2008” (ICR, page 12). In addition, there were political objections to the probable conclusions of the joint power/water management study, which eventually led to it being dropped. The tariff study had also been dropped and taken over by the AfDB. As a result, less than US$2 million was disbursed by early 2009, three years into what was originally scheduled to be a four year implementation schedule. Given these inefficiencies, a quantitative analysis that covers under 40% of actual project costs, and lack of clarity concerning both the efficiency and sustainability of the fuel oil purchases, efficiency is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) 0 Appraisal 0 Not Applicable 39.00 ICR Estimate  21.20 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original Objectives: Relevance of objectives is rated high and that of design modest. The first objective – to increase the access to the power sector – was substantially achieved thanks to the expansion of the electric distribution system in Balbala and the replacement of the emergency 20 kilovolt line from Djibouti to Arta. The efficacy of the second objective – to increase the reliability of the power sector through tapping alternative energy sources (in the form of a pilot windfarm) was negligible since the windfarm did not materialize. The third objective – to improve the efficiency of the electricity utility – was achieved to a modest extent. There is little evidence of improved EDD efficiency that could be attributed to the project beyond the successful impact of the pilot smart meter scheme, and the extent to which the studies and technical assistance led to enhanced efficiency is unclear. Efficiency is rated modest. Outcome is assessed as moderately unsatisfactory. Revised Objectives: Relevance of objectives is rated substantial and that of design modest. The efficacy of the first and third of the four revised objectives – increased access to the power sector and improved efficiency of the utility -- are similar or identical to the first and third original objectives, and share their ratings – substantial and modest, respectively. Achievement of the second revised objective—ensuring emergency reliability of power generation through project financing of the purchase of HFO supplies – is rated modest. While short term fiscal respite was provided, it was unrelated to any policy actions and the fiscal relief is of doubtful sustainability. Efficacy of the fourth revised objective – to reduce the negative effects of drought on water pumping in both rural and urban areas through the creation of HFO and DFO security stocks – is also rated modest. Due to the infeasibility of earmarking centrally generated electricity, the activity could not be carried out in urban areas, which are supplied from the grid. In off-grid rural areas, no experience of deployment under drought conditions has yet been reported. As with the second revised objective, there are sustainability concerns. With efficiency rated modest, outcome is assessed as moderately unsatisfactory. Overall Outcome: Since the outcome of both original and revised objectives is rated moderately unsatisfactory, a weighting exercise is unnecessary, and overall outcome is assessed as moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating The risk associated with the continued serviceability of the project's physical investment components is low. The grid rehabilitation and expansion projects used standard equipment that requires only basic and well-known maintenance procedures. EDD technical staff has demonstrated the ability to maintain this infrastructure. Under the smart metering component, EDD Commercial and IT divisions have become fully familiar with the new meters and commercial management system, and EDD is currently expanding the program. The continued maintenance of the 360 day supply of DFO financed under AF2 represents a substantial risk, as the funding for the cost of maintaining it is high, and the need for such a large stock is low. The ICRR does not discuss EDD's ability to continue paying for the necessary storage facilities. Nor does it discuss whether the Government will allow EDD to gradually reduce this stock to a level more appropriate for an emergency stock of DFO. According to information subsequently supplied by the project team, a strategic fuel stocks law was adopted on June 12, 2014, authorizing the opening of a dedicated bank account where fuel distributors may transfer a tax amounting 2 FDJ per liter, to cross- subsidize the fuel supply for water pumping in rural areas. However, it is not clear whether this would be sufficient to meet the cost of maintaining the stock and the related facilities. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry Expansion of electricity services to the urban poor in Balbala was identified as a key priority by the Government.　It was thoroughly prepared and well-designed. Nevertheless, there were three moderate shortcomings in Quality at Entry. First, while the risk to the viability of the pilot windfarm presented by downward movement of the international oil price was considered at appraisal (PAD, page 10), the risk that actually materialized was not: "unexpected supply constraints in the world market for small-scale wind power equipment leading to price hikes, and a concomitant steep decline in suppliers' interest for a market like Djibouti’s" (ICR, page 12). As a result, the first tender produced no bids, and the second tender only one bid at a proposed price which would have rendered the windfarm uneconomic and beyond available project resources. Although this had significant repercussions for the project, it is considered a moderate shortcoming since developments in the suppliers’ market were difficult to foresee at the preparation stage. Second, there were some weaknesses in the design of the institutional arrangements. The ICR (page 24) states that during the first phase of implementation, "the [Project Management Unit] PMU was hosted in a structure not fully recognized by EDD and the Ministry of Energy." Although allocated office space within EDD, the PMU reported to a Steering Committee chaired by the Director of the Prime Minister’s Office, which lacked technical capacity in energy sector issues. Also, in accordance with design, EDD assigned only five part-time staff to the implementation of the project, leading to delays. EDD soon addressed this problem by assigning five full-time staff to the project, while the PMU became hierarchically as well as physically located in EDD.　 Third, as noted in Sections 3b and 4 above, the implcations of the joint electricity-water commercial management study were not fully thought through, with the result that the operation ended up supporting a study which, while probably useful to the water entity, had lost its relevance to the project. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision There were some positive aspects to the Quality of Supervision. The team supported the reorganization of the PMU within EDD, which greatly improved its effectiveness Three restructurings (including two Additional Financings) were implemented at the Government’s request, and attempted to refocus the project in response to changing project specific and country economic circumstances. Reflecting the project’s relative importance in IDA’s Djibouti portfolio, particular attention was given to the overlap of different Task Team Leaders (TTLs). Although there were four TTLs and one co-TTL during implementation, the impact on continuity was reportedly minimized. Technical advice of the Bank team was generally considered valuable, both by EDD and the PMU. There was compliance with all covenants and with Bank safeguards and fiduciary policies (see Section 11 below). However, there were also a number of weaknesses in Supervision, which, taken together, constitute significant shortcomings. First, although the dropping of the pilot wind farm was appropriate, it is unclear why it took over three years after Board Approval to accomplish it. It might have been useful for the Bank to determine why there were no responsive bids to the first tender before undertaking a second one. Second, while the original design was reasonably articulate (consisting of two independent components --a pilot windmill and power distribution and three major studies to enhance efficiency and performance of the power sector), coherence and complementarity were eroded as new activities were added and supported by the Additional Financings. In its final form, the project incorporated a miscellany of very distinct activities which were not well connected and which did not lend themselves to a tight and easily measurable set of development objectives. As the ICR (page 23) suggests, it may have been easier and more appropriate to have designed a separate fuel supply project, or policy-related budget support to the Government, and to have reallocated the funds for the wind farm component to expanding the relatively successful grid-access activities. Third, the activities initiated under the second AF were not well thought-out. It was not until the consultant retained for the purpose was attempting to develop the regulations for the usage of the purchased HFO, that it became apparent that electricity is a fungible commodity that cannot be allocated to a specific set of consumers, because it can be used by anyone with a connection to the grid. This made the new objective as articulated unachievable. Quality of Supervision Rating Moderately Unsatisfactory Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance According to the ICR (page 24), the project did not enjoy full government support in its initial phase, when implementation arrangements did not fully accord with the wishes of the Ministry of Energy and of EDD. A stronger level of ownership and commitment evolved after the 2009 restructuring, when the PMU was transferred to EDD and alternative uses were developed for the funds originally to have been devoted to the pilot windfarm. Although the Government was unwilling to accept the necessity of reducing excess employment in EDD or to increase tariffs to the level necessary to make EDD financially viable, it did provide EDD with US$ 28 million in 2008 to ensure that it could pay its fuel import bills. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance Once EDD had allocated five full-time staff to the PMU, these remained at their position throughout implementation. The ICR (page 25) reports that the project management skills staff acquired have allowed them to manage projects from other funding sources. For example, EDD and the Djibouti Social Development Agency used US$1.2 million provided by other donors to connect 3,000 additional people and add 61 additional street lights to the distribution infrastructure financed by the project. EDD invested an additional US$ 2.1 million to build 22.5 km of additional underground transmission line in the areas covered by the project and has purchased 35,000 additional smart meters to provide this updated technology to all its customers. With a total contribution of US$ 3.28 million in kind for the grids rehabilitation and expansion component, EDD exceeded appraisal estimates. Other than some early procurement-related difficulties, administration of fiduciary and safeguards compliance was adequate. Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design M&E was designed to rely on project-specific data for its performance indicators. The PMU was responsible for collecting the data for these indicators and drafting regular M&E reports. Two of the PDO indicators were directly related to project inputs and outputs. They were increases in the number of electricity connections in poor targeted area of Balbala, and the decline in the number of annual outages in the Arta line corridor. Other PDO indicators, including the indicator of the number of people provided with access to electricity by household connections, and the total electricity technical and commercial losses as a percentage of total billed, were too broad in that they encompassed contributing factors that were external to the project. b. M&E Implementation M&E was performed by the PMU, on a regular basis, or for update of indicators during preparation of AFs and Project Restructurings. The ICR states that the PDO and Intermediate Outcome Indicators were difficult to monitor, but does not provide information that would shed light on why this was so. Nor does the ICR discuss the impact of the Level 1 restructuring and the two AFs on M&E implementation. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) The information in the “PDO indicator” table in the ICR Data Sheet is confusing has significant inconsistencies. Four examples illustrate this. First, the 2008 baseline for total technical and commercial losses as a percentage of electricity billed was 23%. The target was to be identified in the loss reduction study. However, the PDO indicator table shows that the chosen target was a very low 3%. The actual achievement was 12%, showing that the target had not been met. However, a 50% reduction from the base level would normally be a very acceptable achievement, one which could, at least in part, have been attributable to the upgrading of distribution lines and the introduction of smart meters. Second, the Data Sheet states that the target for annual outages in the Arta transmission corridor was 58 in 2010, while the actual baseline number in 2009 was only 10, and that zero outages were reported in 2014. Third, the number of direct project beneficiaries is stated to be both 4% of the country’s population (which would be 3,460 people), and 468,000 people, which is 54% of the national population. Fourth, for the increased number of electricity connections in poor targeted area of Balbala (baseline zero), the target was initially set at an extremely low 500. The PDO indicator table shows that the target was revised to 2,462 connections at project closing (12/31/2014), but that the outcome, also on 12/31/2014 was 3,828 or about 50% higher than the target chosen at the same date. c. M&E Utilization The ICR states that the M&E was utilized by the task team to follow the progress on the implementation of the different components. The PMU used the M&E to follow project progress, and to report to EDD’s management and to the Ministry of Energy. M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was classified as a Category B project (i.e. minimal, site-specific, and manageable impacts). At appraisal only the Environmental Assessment Safeguard (OP 4.01) was triggered. The environmental assessment for the Arta Wind Farm did not trigger any social safeguards because it was to be located public land. An Environmental and Social Management Plan was prepared. As a precaution, a Resettlement Policy Framework was also prepared for the Expansion of Electric Distribution in Balbala and the replacement of the distributions line from Djibouti to Arta. The work on these lines did not, in the event, entail the removal of people or economic activities, and the works were carried out on public land. Although safeguard compliance was rated as Moderately Satisfactory at times during project execution, the ICR (pages 15-16) states that the environmental and social impacts were easily managed in the field, were financially affordable, and did not delay the project implementation significantly. b. Fiduciary Compliance The Financial Management (FM) risk was rated high at appraisal and an FM plan was established as one of the conditions of effectiveness. The ICR reports that financial management covered adequately the project’s accounting and reporting arrangements, internal control procedures, planning and budgeting, counterpart funding, funds flow arrangements, external auditing arrangements, and project accounting staff issues. Staffing remained adequate and proper books of accounts and supporting documents were kept in respect of all expenditures. Supervision reports rated FM either Moderately Satisfactory or Satisfactory throughout the project’s life. Most of the financial statement audits were submitted on time, and were unqualified. Of the 40 interim financial reports submitted, one has been submitted within 30 days after the deadline and two were submitted within 90 days after the deadline. The quality of those reports improved throughout project implementation. Close supervision resulted in all legal covenants being complied with, and the latest audit reports being up to date and deemed satisfactory. Procurement was rated Moderately Satisfactory or Satisfactory in supervision reports during almost all the project period, and most goods and services purchased under the project were cost-efficient in comparison with budget estimates. As the ICR reports (page 16), “EDD achieved significant savings (and avoided costs) by including in the rehabilitation of distribution grids the assessment of used electrical equipment for re-deployment of viable material in the provinces (meters, poles). The project also implemented innovative procurement practices for EDD, dividing in lots the different distribution circuits to be rehabilitated and expanded; this allowed a smoother and more progressive improvement and integration for EDD (load increase, loss reduction), and a bearable payment profile for the suppliers – in Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) particular the little local firms.” The PMU easily managed to handle the additional work due to this procurement method. c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment For both the original and revised objectives, the efficacy of only one objective is rated substantial Outcome Satisfactory Moderately Unsatisfactory with the other modest or negligible. Efficiency is rated modest The risk of not maintaining the Risk to Development Outcome Modest Substantial emergency supply of DFO is substantial. There were significant shortcomings in Quality of Bank Performance Moderately Satisfactory Moderately Unsatisfactory Supervision (see Section 8b above) There were moderate shortcomings in Government performance including the initial Borrower Performance Satisfactory Moderately Satisfactory arrangements for managing the project and unwillingness to take measures to improve EDD’s financial situation Quality of ICR Modest --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Most of the lessons listed in the ICR are specific to the electricity sub-sector. One lesson of more general interest concerns international bidding for small projects in small countries. The limited size of pilot projects in such a country as Djibouti is frequently unattractive for international bidders. In this case, both the windfarm and the smart metering were affected. The former had to be abandoned after two unsuccessful tenders. In the case of the latter, bidders were obliged to assume that they would be automatically selected in case of project generalization, so that they could recover a part of their costs later on. But the project funding was limited to the pilot only, which was not clearly stated to the candidates. Experience in several African countries with smart meter pilot shows that there is also a danger of dumping with the pilot phase. A second lesson drawn by IEG relates to the coherence of objectives. Once a project becomes a miscellany of distinct activities, the connection between which is tenuous at best, it becomes difficult both to monitor and to evaluate the achievement of intended outcomes. In this case, while the original project was reasonably coherent, this was eroded when activities supported under the restructuring and by the Additional Financings were added or substituted. A third lesson concerns the importance of thorough research, consultation and coordination with all important stakeholders at the preparation Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DJ-Power Access and Diversification Proj(P086379) stage (including the preparation of new activities to be financed with Additional Financing). In this case, this might have avoided or mitigated the problems and consequent delays associated with the pilot windfarm, the tariff study, the joint electricity-water commercial management study, and the desire to earmark electricity generated with the purchased HFO. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICRR did a good job of describing and discussing the outcomes of the project. However, there is little analysis of the reasons for the project’s shortcomings, including why the Bank and Borrower thought that a second bidding round for the windfarm would result in a better outcome than that of the first round; why the Bank accepted a revised management study that included only the water authority, irrelevant to any of the project’s development objectives in energy ; and, while describing the problems with the implementation of the second Additional Financing, not examining why it took a consultant to point out that the project design was impossible to implement, and not analyzing the problems consequent upon the unanticipated outcome. This lack of critical analysis reduces the usefulness of the ICR as a guide for what should be done in the future. Moreover, the reasons for the dropping of the tariff study and its support by AfDB had to be clarified subsequently by the project team. There is some confusion in the assessment of Borrower performance. Specifically, Government performance should include performance of all of the Government ministries and departments, and the performance of the implementing agency should include that of the institution for which the work is being carried out, and is, generally, for responsible for carrying out the work (EDD is, in this case, the Implementing Agency). The work of the Project Management Unit should be considered as an operating department of the implementing agency, responsible primarily for contracting and reporting. More care could have been taken in ensuring that the data presented is in a consistent manner. For instance there are some unexplained discrepancies in the project cost and financing data presented in Annex 1. Also, both Additional Financings are described as Grants on the cover of the ICR and as IDA Credits on page 8. a. Quality of ICR Rating Modest