ICRR 14379 Report Number : ICRR14379 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 07/29/2014 Country : Afghanistan Project ID : P106654 Appraisal Actual Project Name : Artf - US$M ): Project Costs (US$M): 57.00 56.95 Kabul-aybak/mazar-e- sharif Power Project L/C Number : Loan/ US$M): Loan /Credit (US$M): 57.00 56.95 Sector Board : Energy and Mining Cofinancing (US$M): US$M ): 1.50 Cofinanciers : Board Approval Date : 10/04/2007 Closing Date : 03/31/2013 Sector (s): Power (92%); Central government administration (8%) Theme (s): Infrastructure services for private sector development (67% - P); Conflict prevention and post-conflict reconstruction (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Dileep M. Wagle George T. K. Pitman Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The Project Development Objective, as stated in the Grant Agreement (p.5), was: "to help provide reliable and quality power to consumers in the target areas of Kabul, Aybak and Mazar -e-Sharif" The Project Proposal (p.1) that was approved in lieu of a Project Document provides an identical statement of project objectives. This review uses the objectives stated in the Grant Agreement . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: 1: Distribution System Rehabilitation for Kabul (at appraisal: US$17.0 million; at completion: US$20.66 million). This component aimed to rehabilitate and extend the distribution system to connect medium voltage lines to customers in Kabul, and to reorganize the existing distribution system to optimize loading and improving power supply voltage to customers there . 2: Establishment of a 220 kV Substation at Aybak and Interconnection with Distribution System (at appraisal: US$6.00 million; at completion: US$9.86 million). This component aimed to set up a new 220/20kV, 16 MVA substation at Aybak, which would then be interconnected with the existing Aybak distribution system . 3: Power System Rehabilitation for Mazar -e-Sharif (at appraisal: US$25.00 million; at completion: US$23.6 million). This component aimed to rehabilitate the distribution system in Mazar -e-Sharif through the upgrading of the existing MV distribution network from 6 kV to 20 kV, connecting the distribution network to the 220/20kV substation, rehabilitating and replacing the low voltage distribution network, plus upgrading the Mazar -e-Sharif substation capacity to meet the requirements of the distribution network . 4: Institutional Capacity Building (at appraisal: US$4.00 million; at completion US$2.83 million). This component aimed to provide capacity building and implementation support to the Ministry of Energy & Water (MEW) and its entity responsible for power supply, Da Afghanistan Breshna Moassesa (DABM), for (i) operation & maintenance of the distribution systems, (ii) implementation of infrastructure metering for energy supplied in Kabul, Aybak and Mazar -e-Sharif, (iii) establishment, operation and hand -over to DABM of customer care centers, and (iv) other capacity building needs, as identified during implementation . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : The final project cost was US$56.95 million, marginally lower than the estimated cost at appraisal (US$58.50 million). There was some variability between individual components – two of which (Components 1 and 2) ended up costing somewhat more than originally anticipated and the other two costing slightly (though not proportionately) less; the difference being made up by a draw -down of the allocation for contingencies . Financing : The sources of funding for this project consisted of the Afghanistan Reconstruction Trust Fund (ARTF), which financed a grant of US$57.00 million under TF-91120, supplemented by US$1.50 million of IDA resources from existing Credit 3933. US$55.42 million were actually disbursed and the balance of US$ 1.58 million was cancelled. Borrower : No counterpart funding was provided . Dates: Dates The project’s closing date was extended three times, for a total of 39 months, as follows: a) by 21 months, from November 2009 to September 30, 2011; b) by 9 months, to June 30, 2012; and c) by a further 9 months, to March 31, 2013. As regards the first extension, it should be mentioned that since the parent ARTF Grant Agreement (TF-50576) had a closing date of June 30, 2010, the original closing date of TF -91120 was set as December 31, 2009, with the understanding that once the parent TF Agreement was extended, the closing date of the project TF would be extended to enable project completion . Subsequent extensions took place to provide additional time for project completion necessitated by delays on account of a variety of factors, some reflecting poor implementation by MEW and others on account of factors beyond its control (including the tenuous security situation in the country). 3. Relevance of Objectives & Design: a. Relevance of Objectives: High The project was relevant to the country ’s developmental objectives, which were being severely constrained by the poor condition of the power sector . At the end of 2005, only 6% of Afghans had access to public power, one of the lowest rates in the world. One third of energy consumers, connected to the public grid, were in Kabul . There was no significant provincial or rural electrification . At the time of approval the prevailing service consisted of only a few hours of supply a day to small percentage of the population which had access to the power grid, leaving a huge demand -supply gap, especially in Kabul . Hence, improving the reliability and quality of the electricity services could be considered critical to the country ’s developmental goals of poverty alleviation and economic growth. The Afghanistan National Developmental Strategy (ANDS) had energy supply among the country ’s top economic priorities. Sustainable energy and water supplies formed the eighth pillar of the ANDS . This was echoed in the World Bank’s FY12-14 Interim Strategy Note, which recognized the need to continue to address the energy needs of the businesses and people . In light of the rapidly evolving demand for electricity in Kabul and the importance of Mazar-e-Sharif as a commercial and industrial center (and Aybak as a provincial city ), the objective clearly was and remained relevant . b. Relevance of Design: Modest Project design was relatively straightforward, and built on the design standardized by studies conducted under the preceding Emergency Power Rehabilitation Project . The original design envisaged capacity building of the distribution utility (DABM) for planning, operation & maintenance of the low voltage distribution network, by hands-on/on-site training and physical rehabilitation as both being essential . In practice, however, the design of the project focused heavily on the physical rehabilitation of infrastructure, with a relatively small share of resources being allocated to capacity building . It appears that the capacity of the implementing agency, MEW – particularly for project management – remained weak, and contributed to the significant delays experienced by the project, especially in the misspecification of the concrete poles used for medium and low voltage overhead lines, interrupting power for several months and causing delays in subprojects. Low capacity in other parts of the government, especially the Ministry of Finance (MoF) also led to delays. The failure to anticipate and to mitigate capacity shortfalls at various levels through the project design greatly undermined the performance of the project . For this reason relevance of design is rated modest . 4. Achievement of Objectives (Efficacy): Objective: "To help provide reliable and quality power to consumers in the target areas of Kabul, Aybak and Mazar -e-Sharif ." 1. To help provide reliable power : Modest Outcomes : The goal of increasing power supplies to project areas in Kabul is seen to have resulted in a 253% increase over the baseline target of 482 GWh (actual achievement of 1,219 GWh); The goal of increasing power supplies to customers in Mazar -e-Sharif from 92.2 GWh to 113 GWh was similarly overachieved by 174% (to an actual level of 197 GWh). The target of providing grid power to at least 25% of the people of Aybak was fully realized . The borrower’s evaluation report, however, raises some questions on the attribution of results to the project ’s interventions. The report argues that power consumption in Kabul was already significant at 649 GWh in 2007, before the project started, rising to 978 GWh by end-2009, before the project work in Kabul was even completed . Hence, the improvements in power availability there would have been largely on account of work carried out under the earlier Emergency Power Rehabilitation Project (EPRP), which undertook extensive rehabilitation of the distribution networks in Kabul (and/or supported by other donors, as suggested by the Closing Implementation Status and Results Report ). The report also casts doubt on the results achieved in Aybak, arguing that the end-project target of connecting 25% of households to the grid was based on baseline data that significantly underestimated the percentage of population already connected . (The indicator for Mazar-e-Sharif was however considered to be valid and to have been legitimately achieved by the project ). Outputs : The intermediate goal of strengthening the low voltage distribution network in Kabul and Mazar -e-Sharif, as measured by the number of transformers installed and commissioned, was reportedly fully achieved, albeit with a delay of 11 months in each case. The goal of constructing a 220/20kV substation and associated 20kV lines at Aybak to provide grid power to residents there was also fully achieved, though with a delay of 18 months. The achievement of physical targets may mask to some extent concerns about the sustainability of the distribution system in Kabul. The closing Bank supervision drew attention to the failure of a number of transformers that had been installed, on account of overloading and poor maintenance . The failure of the project to ensure the build-up of adequate capacity within MEW (later Da Afghanistan Breshna Sherkat [DABS]), mentioned earlier, may also have contributed to this problem . 2. To help provide quality power : Modest As regards improvements in the quality of power delivered, while reliable data are not easily available, for Mazar-e-Sharif at least, there is limited evidence to suggest that average delivered voltage, which was no higher than 125V from the previous 6KV network, had reached the proper 220V line voltage threshold under the new 20kV network set up under the project . 5. Efficiency: Economic and Financial Efficiency : No economic and financial analysis was conducted as part of the original Project Proposal, at appraisal; hence no economic rates of return (ERRs) or estimates of net present value (NPV) are available for comparison with estimates at closure. Estimation of economic efficiency for the project is complicated by the fact that the Kabul component, which financed the low voltage systems and meters, was closely intertwined with the earlier -initiated Emergency Power Rehabilitation Project (EPRP), which had financed much of the cost of transmission and distribution rehabilitation. Economic returns from that component in the EPRP had been very high (of the order of 81%), and the ICR for the current project excluded the Kabul component entirely from the ERR analysis, as it was difficult to separate out the respective costs and benefits . On the basis of only the Aybak and Mazar subprojects, which accounted for only 56% of total project financing, the net present value of economic benefits were estimated at US$6.71 million for Aybak and US$13,12 million for Mazar, with estimated ERRs (based on the avoided costs of the next-best alternative) of 25% and 0.15% respectively. Though ERRs estimated at appraisal were unavailable, the ICR argues that the efficiency of the Mazar -el-Sharif sub-component clearly fell well below an acceptable threshold, whereas for Aybak it may possibly have been acceptable . It should be noted that the arguments in the ICR for excluding the (relatively large) Kabul sub-component from the economic efficiency analysis are not entirely self -evident. While much of the benefits arising may have been on account of the rehabilitation of transmission lines to Kabul under the EPRP project, from an economic perspective this does not invalidate the role of the distribution rehabilitation in Kabul . In so far as it completed “the last mile� at relatively low cost, it would have generated relatively high returns, which in turn might have indicated positive economic efficiency . In the absence of an estimate in the ICR for the specific work undertaken for the current project however it becomes difficult to make a meaningful judgment, other than to suggest that depending on how benefits were allocated between the two projects, efficiency may well have been positive for this sub-component, and hence for this project as a whole . Administrative and institutional efficiency : From the ICR, it would appear that project management was handled in a less -than-optimal manner. which led to major delays in implementation, poor maintenance and equipment failure . As a result, implementation took twice as long as originally anticipated, eventually undermining project sustainability . This was on account of failure to provide adequately for capacity -building, especially for project management, and this led to both technical and institutional shortcomings : Technical : Quality control was poor, resulting in some defective equipment being installed . Particular issues were observed to have arisen with the concrete poles and distribution transformers installed in Kabul and Mazar-e-Sharif, due to incorrect specification of orders placed by MEW . Installation work by the contractors was of variable quality and not consistently acceptable, and corrective action was not always taken even when they were issued with corrective action reports . Institutional : O&M training during the project life was often inadequate . Though contractors were required to provide such training as part of their scope of work, no standard of quality or competence was specified, which resulted in training being often delivered to the wrong local staff or in an ineffective manner . Installation contracts included provision for equipment and spare parts, but the lists were often inadequate and quantities inadequate, thereby increasing the likelihood that assets installed will fall into disrepair faster than necessary. Taking account of these factors, efficiency is rated as Modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project’s objectives are highly relevant . However, relevance of design is modest . Efficacy is rated modest, for both reliability and quality objectives . Efficiency is also rated modest . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Serious questions remain regarding the sustainability of the project . As mentioned in Section 5, serious shortcoming were observed at both technical and institutional levels . Systems installed, especially for distribution systems, were not implemented with any focus on quality, longevity or sustainability . Secondly, the security situation in the country continues to be volatile, and political risks continue to be high . Overall, considering this state of affairs, the risk to achieving a sustainable outcome in the medium term is rated High. High . a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: a. Quality at entry: The project components were in line with country priorities and appear to have been designed as complementary to the existing EPRP project, with a view to filling in specific gaps that remained unfinanced after that project’s restructuring. While no Project Appraisal Document (PAD) was prepared, a detailed Project Proposal that was approved could be considered to be the de facto PAD, on the basis of which to assess the quality of the project at entry . The Bank's Quality Assurance Group included this project in its Quality Assessment Lending Portfolio (QALP) in 2010 and found the design and implementing arrangements to be satisfactory. The Minutes of the project concept review meeting included a recommendation that the project be fast-tracked for implementation. The issue of sustainability, and risks thereof, were raised during that meeting, and the team was strongly advised to ensure that capacity building for the utility be included in the project. However, these risks were eventually not adequately mitigated in design and such capacity building as took place happened at MEW, which is not the institution that would be responsible for future investments or O&M. at-Entry Rating : Quality -at- Unsatisfactory b. Quality of supervision: The Bank team conducted regular supervision missions, often jointly reviewing the EPRP project, and project issues are fairly clearly documented in the Supervision reports . The project was extended several times on account of various delays in implementation . As the ICR recognizes, there is a disconnect between the various implementation problems highlighted in the supervision reports and the Satisfactory /Moderately Satisfactory ratings that were accorded by the Bank team to the project ’s performance throughout its life, right up to the final supervision which made a sharp reversal and downgraded the rating to Unsatisfactory . The same was true of the various restructurings /requests for extension, which recognized the problems resulting from delays and weak capacity, but continued to express confidence that the development objectives remained attainable. To some extent, supervision teams may have been making allowance for the deteriorating security situation in the country, which clearly impacted implementation performance, but it would also suggest a lack of rigorous monitoring, especially in light of the weaknesses later identified in baseline data for the indicators and the problems in attributing results . The ICR suggests this may have been on account of an over -reliance on the Project Management Firm, whose quality of work had begun to deteriorate after 2010. Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: Coordination issues presented the most significant challenge to the implementation of the project . Lack of coordination and a long-standing dispute between ministries and DABS (a department under MEW, created in 2009 to play the role of the national electric utility ) was reflected in such actions as the suspension of pole delivery by the Ministry of Energy and Water (MEW) due to non-compliance of design standard, non-issuance of customs exemption certificates by the Ministry of Finance, which also delayed the processing of letters of credit for contractors, all of which were avoidable and which led to major delays in the implementation of the project. This no doubt reflects the fragile (post-conflict) nature of Afghanistan’s public administrative structure; in light of which the decision to rely on country systems for disbursement and financial management may also have contributed to problems encountered . Government Performance Rating Unsatisfactory b. Implementing Agency Performance: Lack of capacity at MEW, reflected in weak contract management and inability to address technical issues, significantly impacted the project ’s performance. All project components experienced delays in implementation, including delays in executing contracts (which took an average period of 125 days between MEW receiving bids to signing the contracts ) and performing the work. Of all of the contracts implemented, the Borrower’s Evaluation Report records that the most on -time project – the meter supply contract – still took 320 days longer than expected, while the worst -delayed project, the Mazar distribution rehabilitation, took 850 days longer than originally planned . Adequate attention was given to safeguard issues, and there were some financial management issues, including lack of internal control mechanisms and late submission of the external audit Implementing Agency Performance Rating : Unsatisfactory Overall Borrower Performance Rating : Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The outcome indicators (one for each component, covering rehabilitation of infrastructure, but not for institutional capacity building) were broadly relevant to the objectives of the project . Their principal weakness lay in the fact that they did not adequately take account of and differentiate between the project and the earlier EPRP project, which had already financed a significant portion of the infrastructure rehabilitation for the Kabul component, leaving the current project to finance gaps and add -ons. As a result, attribution of results was going to be a challenge from the outset. Nor did Key Performance Indicators take account of the danger that poor quality of product (e.g. of transformers, that later failed) could undermine actual achievement . The difficulties that later emerged in underestimating start-of-project baselines in the results matrix, thereby rendering the end -of-project targets largely useless for both the Kabul and Aybak components, further exacerbated these problems . b. M&E Implementation: The results matrix was regularly updated in successive supervision reports, suggesting that a regular flow of information was being maintained. The fact that the Final Supervision Report raised (for the first time) concerns regarding attribution of results is an indication of the fact that these data had not been rigorously analyzed at the time they were collected, during the implementation of the project . c. M&E Utilization: Feedback from monitoring and evaluation did not appear to have played much of a role in helping corrective action be taken on a timely basis . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project was classified Category B under OP /BP 4.01 Environmental Assessment and not expected to raise significant environmental concerns . No Integrated Safeguards Sheet was prepared and the Environmental & Social safeguards Framework (ESSF) used for the EPRP project was used without change for this project as well. Efforts to strengthen capacity were pursued during implementation through the Project Management Firm, which supported regular monitoring . Training was provided to improve the capacity of both government and contractors to implement safeguards . During implementation, safeguards issues arising included mine clearance (handled by the Mine Action Center for Afghanistan, as per the ESSF ), Rights of Way (handled in compliance with the provisions of the ESSF and the ESS, including local community consultation along the proposed routes ), and polychlorinated biphenyl testing (which took place but did not reveal their presence (ICR, p.8). The ICR notes (p.8) that while safeguards supervision was regularly documented through 2009, less information was divulged in later years. According to the ICR, this may have been because few safeguard issues were at stake by then. b. Fiduciary Compliance: Some financial management (FM) issues were reported, including lack of internal control mechanisms and late submission of the external audit for 2011-12. The Bank's Supervision Reports consistently rated FM issues in the Satisfactory/Moderately satisfactory range . Very few contracts were to be procured under the project, which did not experience procurement problems, apart from delays, as such (ICR, p. 8). However, contract management was affected by institutional capacity constraints. c. Unintended Impacts (positive or negative): d. Other: 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Unsatisfactory Moderately There were significant shortcomings in Unsatisfactory the projects design, efficacy and efficiency. Although the project appeared to have met or exceeded its results targets, serious concerns had arisen about their attribution and validity of data, as well as about the overall sustainability of the project . Risk to Development High High Outcome : Bank Performance : Unsatisfactory Unsatisfactory Borrower Performance : Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR provides the following key lessons : Even when a project is conceived as a follow up to an existing operation, it needs to be appraised thoroughly on a stand -alone basis, so that proper attention can be paid to design issues . Building implementation capacity is as important as investments in physical infrastructure . Infrastructure was rehabilitated under the project, but remained at risk of falling into a build -neglect-rebuild cycle on account of lack of capacity created for operations and maintenance . Results indicators need to be built on a clearly defined results chain, linking inputs to attributable outcomes . In countries with a history of conflict and poor capacity, consideration should be given to use of relative or incremental indicators (e.g. number of households connected and metered ). Ensuring adequacy of supervision . Given the importance of on site supervision, which - for this operation - was essential for quick identification (and resolution) of implementation problems, the question is how to ensure this in the midst of deteriorating security conditions in the country, which may make frequent field visits difficult in practice. The use of management agents is one option, but did not work out too well in this project . The answer may lie in simpler design that requires minimal supervision . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is adequately evidence -based in its analysis and discussion of results . It is concise and generally internally consistent in its presentation . a.Quality of ICR Rating : Satisfactory