Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) Report Number : ICRR0021329 1. Project Data Project ID Project Name P101473 ET-Urban WSS SIL FY07) Country Practice Area(Lead) Additional Financing Ethiopia Water P129174,P129174 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-42920,IDA-51150,IDA-H2870 31-Dec-2012 200,180,221.19 Bank Approval Date Closing Date (Actual) 24-Apr-2007 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 220,999,265.57 0.00 Actual 200,180,221.19 0.00 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The original Financing Agreement (FA) and the Project Appraisal Document (PAD) noted that the Project Development Objective was "increased access to water supply and sanitation services in Addis Ababa and four secondary cities" (FA, p,4 and PAD, paragraph 23). Additional Financing (FA) modified the PDO "to increase access to water supply and sanitation services in Addis Ababa and targeted secondary cities" (AF, p. 6). This review will use the modified PDO and will subdivide the presentation to the following: • to increase access to water supply services in Addis Ababa • to increase access to sanitation services in Addis Ababa Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) • to increase access to water supply services in target secondary cities • to increase access to sanitation services in the target secondary cities. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 12-Jun-2012 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components 1. Addis Ababa Water Supply and Sanitation Authority (AAWSA) (US$ 70.0 million at appraisal, with US$ 100.0 added during the AF for a total of US$ 170.0 million, and US$ 117.4 million actual) financed the construction of new water supply sources for Addis Ababa, expanded the capacity of existing sources, expand the water supply network to reach unserved and underserved residents. Global Partnership for Output Based Aid (GPOBA) resources were planned to finance subsidies for water connections for low income areas, construct public toilet facilities, and build new sewerage connections in the city. The GPOBA financing did not come through and the new sewerage connections in the city did not materialize. This component also financed the establishment of pressure zones within the AAWSA distribution network, to repair its network leaks, improve its financial management, billing and collection, and customer management systems and catchment protection measures, as well as public awareness campaigns about water conservation. This component also financed support for policy development, revising the legal and institutional of the water supply and sanitation framework by encouraging private sector participation in its governance and delivery, as well as project management and stakeholder consultation. AFD co-financed (US$ 6.25 million at appraisal, US$ 9.0 million actual) some of the activities designed to increase production and reduce Non Revenue Water (NRW, i.e., water losses). The 2012 AF financed the construction of a new wastewater treatment plant in response to increased housing developments in the city. 2. Secondary City Water Supply and Sanitation (US$ 29.0 million at appraisal, with US$ 48.9 million added during the AF for a total of US$ 77.9 million, and US$ 75.5 million actual) financed the expansion of the capacity of existing water supply networks for 10,000 households and new water connections for 25,000 more. This component also financed the establishment of pressure zones within the distribution network, repair leaks, to minimize NRW losses and improve financial management, billing and collections, and customer management systems to increase the operational efficiency of the utilities in the target cities. In addition, this component financed the project management of the targeted city utilities as well as Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) institutional reforms to foster their autonomy. Additional financing provided resources for another city, Dire Dawa to finance infrastructure investments, improve operational efficiency, and institutional reforms. 3. Program Management (US$ 1.0 million at appraisal, with US$ 1.1 million added during the AF for a total of US$ 2.1 million, and US$ 1.2 million actual) financed equipment, workshops, study visits, and technical assistance to support national and regional program coordination units (PCUs) in building capacity in billing and collection, distribution management, reducing non revenue water (NRW), monitoring and evaluation (M&E), safeguard compliance, and refinement of the Project Implementation Manual. This component also financed the operation support, training and other capacity building activities for the national and regional entities such as the Ministry of Water, Irrigation, and Electricity and the Water Resources Development Fund (WRDF), a revolving fund used to on-lend to the participating utilities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost reached US$269.0 million and disbursed US$ 209.2 million by project close. The equivalent of US$ 27.7 million was cancelled at project close since the remaining activities were transferred to the follow on project for completion - the Second Urban Water Supply and Sanitation Project (UWSSP2). US$ 10.0 million from the United Kingdom's Department for International Development (DFID) as part of the Global Partnership for Output Based Aid (GPOBA) contribution to support connection subsidies for low income households was undisbursed (see below). The project also suffered exchange rate losses over the implementation period (from SDR to US$ and US$ to Ethiopian Birr or ETB). Financing: The International Development Association (IDA) financed this Investment Project Financing instrument with Special Drawing Rights grant (SDR 23.5 million) and credits (SDR 43.7 million and AF of SDR 96.9 million) for the equivalent sum of US$ 250 million. The UK's DFID committed but did not disburse US$ 10.0 million as part of the GPOBA contribution while France's Agency for Development (AFD), committed and disbursed US$ 9.0 million. The UK's US$ 10.0 million commitment was dropped after the 2010 Mid Term Review (MTR) because the Government would be unable to meet the one to one match required under GPOBA terms for subsidizing connections of low income households although the Government noted arrangement delays in its comments on the ICR (ICR, Annex 5). Borrower Contribution: There were no Borrower contributions in the initial project document. However, cities and regional governments provided financial contributions of US$ 29.1 million in matching funds and noted in the AF project paper. Dates: Approved on April 24, 2007, the project was originally scheduled to close on December 31, 2012. The project was closed by December 31, 2017 with an effective 60 months extension. The MTR was conducted on March 30, 2010. There were five restructurings: • on November 20, 2010, a level 2 restructuring reallocated resources among disbursement categories. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) • on June 12, 2012, a level 1 restructuring provided AF and modified the PDO. In addition, the Results Framework indicators were modified to accommodate the expansion of the project and project close was first extended to December 31, 2015, extending the project by 36 months. • on July 15, 2015, a level 2 restructuring extended the loan closing date a second time to December 31, 2016 extending the project another 12 months. This extension would address the procurement delays from two large contracts for the rehabilitation and expansion of Kality waste water treatment plants and expansion of sewer lines in Addis Ababa and delays in resolving construction contract problems in Dire Dawa City (see Section 10. Other Issues, Procurement below). • on August 19, 2016, a level 2 restructuring extended the loan closing date a third time to December 31, 2017 by another 12 months and reallocated resources among disbursement categories. This extension would allow the completion of two ongoing sewerage contracts that would increase the waste water treatment capacity in Addis Ababa from 7,500 to 100,000 cubic meter a day and complete ongoing projects in Dire Dawa City. • on December 22, 2017, a level 2 restructuring revised cost allocation among project components, reallocated resources among disbursement categories and cancelled SDR 19. 5 million or the equivalent of US$ 27.7 million in undisbursed amounts from the AF at the Government's request (see Section 5. Efficiency below). 3. Relevance of Objectives Rationale The Project Development Objective (PDO) remained relevant to the country's national development plan, the Second Growth and Transformation Plan 2015-2020 (GTP II), which aimed to attain 100% national potable water supply coverage and improving sanitation access during the period covered. The PDO was relevant to three of nine pillars that supported the Government's vision to reach lower middle income status: (1) to sustain rapid and equitable economic growth and development; (4) bridge infrastructure gaps through the domestic construction industry; and (5) manage urbanization as part of the structural transformation of the economy. The PDO was also relevant to all the Focus Areas in the World Bank's Country Partnership Framework (CPF) for 2018-2022. The PDO directly supported Focus Area 2 - Building Resilience and Inclusiveness. In particular, this PDO supported CPF Objective 2.3 Increased Access to Improved Water and Sanitation. The PDO directly supported improved access to water and sanitation services, contributed positively to the urbanization process, avoided negative impact to the environment and public health. As the CPF noted, Addis Ababa was the only city so far that had some sewer system and covered only 10% of the population. The PDO directly contributed to the World Bank's intended operations to support improvements in water and sanitation. On a broader scope, the PDO was also relevant to the CPF's Focus Area 1 - to promote structural and economic transformations through increased productivity - by promoting operational efficiency and improving financial viability of the water and sanitation utilities in the six cities of the project for future access to private financing. The PDO was also relevant to Focus Area 3 - supporting institutional accountability and Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) confronting corruption - by increasing capacity, improving governance in service delivery and strengthening citizen engagement, which were addressed by the project. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective • to increase access to water supply in Addis Ababa Rationale OUTPUTS: • 144,900 cubic meters a day was added from improved sources in Addis Ababa (original target 100,000 cubic meters a day, exceeded) • 87,790 new piped household connections were completed (original target 40,000 households, exceeded) • 176,513 existing connections were rehabilitated and received an additional 25 liters a day (revised target 20,000 household connections, exceeded) • In select low income areas, 6,000 shared yard taps were constructed and transferred to households to manage (original target) • AAWSA adopted a series of measures to address NRW, such as procurement of leak detection equipment, customer water meters, pipes and fittings; GIS hydraulic modeling of the network on 10 pilot Demand Management Areas (DMAs); and procurement of test benches to begin the process of testing water meters (AAWSA status report, October 2017) • TA to support operational and financial and commercial activities were provided. By early 2013, AAWSA installed a new computerized billing and accounting system and a new customer service and financial management system to improve billing/collections and overall financial sustainability of the utility. • Implemented new computerized billing and accounting system and a new customer service and financial management system OUTCOMES: • AAWSA operational efficiency improved its financial sustainability as a result of the new billing and accounting system. At project close, the operating ratio (the ratio of operating revenue to expenses) reached 1.17 for the period 2007-2017 (original target 1.1, exceeded). However, over that same period of Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) time, AAWSA’s expenditures were rising faster than revenue, and the operating ratio was trending slightly downward. • Overall, 3,855,000 residents benefited from new connections and rehabilitated water sources in Addis Ababa (original target 4 million, revised to 4.2 million, almost achieved). Of these beneficiaries, 50% or 1,928,000 were women (original target 2 million, revised target 2.1 million, almost achieved). There were no baseline figures for these targets. • The operating ratio (ratio of operating revenues to expenses) reached 110% (original target was 110%, achieved). According to the project team (email August 6, 2018) this ratio was an average of all five cities combined. • 2,720,000 residents benefited from piped household water connections from rehabilitation works (revised target was 3 million residents, almost achieved) • 1,104,000 urban area residents received access to improved water sources (original target 1 million, revised to 1.2 million, almost achieved) Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective • to increase access to sanitation services in Addis Ababa Rationale OUTPUTS: • 15.8 km of the main sewer network lines were laid (original target, 17.2 km, almost achieved) • 101.4 km of secondary sewer lines were laid (original target 108 km, almost achieved) • completed the construction of a wastewater treatment plant but not commissioned to begin operation (target not achieved). • completed the construction of 29 public toilet facilities and 14 shared toilet facilities (no targets provided). Small and Microenterprises (SMEs) were created to operate and manage these public toilet facilities (no targets were provided because according to the project team in an email on August 6, 2018 these achievements were inputs to the project outcome). • There were no new household connections to sewer lines (revised target 73,500, not achieved) nor were there any residents who benefited from improved sewer system (revised target 360,000 residents, not achieved) OUTCOMES: Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) • 11,662 residents in low income areas benefited from improved sanitation in Addis Ababa (original target 300,000, revised target 1.3 million, not achieved). Fifty percent of these residents were women. • There were no new household connections to the sewer lines (revised target 73,500, not achieved) nor were there any residents who benefited from improved sewer system (revised target 360,000 residents, not achieved). Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 3 Objective • to increase access to water supply services in targeted cities Rationale OUTPUTS: • Studies assessed NRW, which ranged from 10 to 13.75 percent (original target 20-35%, achieved) • 76,265 m3/day of water was added to the production capacity in the five target cities from improved sources (original target 20,000 m3/day, revised target 50,000 cubic meters a day, exceeded) • a total of 83,277 new household connections were made (original target was 25,000, revised to target of 50,000 households, exceeded) • 39,000 piped household connections were rehabilitated (revised target 10,000 households, exceeded) • Implemented new M&E system in all utilities meeting target. • TA was delivered to support operational financial and commercial activities of the utilities • Implemented new or improved computerized billing and accounting systems and new customer service and financial management systems meeting target. • New branch offices were opened in selected cities (no targets provided). • TA to support the Water Resources Development Fund (WRDF) was delivered as well as capacity building activities and institutional strengthening by providing equipment and software meeting targets. OUTCOMES: • As of October 2017, the cities of Gondar, Mekelle, Jimma and Hawasa have begun repaying their utility loans to the WRDF meeting targets. • Ratio of operating revenues to expenses reached 110% (original target 110%, achieved). The project team clarified in an email on August 4, 2018 that this ratio was the average reached by all 5 cities. • 972,498 residents benefited from piped household water connections based on new connections and rehabilitated works (original target 1.0 million, almost achieved). Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) • 1,104,000 urban area residents now have access to improved water sources (original target 1 million, revised 1.2 million, almost achieved). Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective • to increase access to sanitation services in targeted cities. Rationale OUTPUTS: • 24,258 residents from low income areas in the target secondary cities benefited from improved sanitation services (original target 0.20 million, revised to 0.25 million, not achieved) • 4 public toilet facilities were constructed (no target provided) • 50 shared public toilets were constructed (no target provided) • 2 vacuum trucks were procured (no target provided) • New drying beds were constructed for sludge treatment (no target provided) OUTCOMES: • There were 24,258 residents who benefited from access to sanitation services in target secondary cities (original target 0.25 million, not achieved). Of these beneficiaries, 50% were female (original target 0.125 million, not achieved). Rating Negligible PHREVDELTBL PHOVRLEFFRATTBL Rationale The Theory of Change, premised on three activities to achieve the PDO, was sound. There was only one key outcome indicator - the number of people with improved water supply and sanitation with specific targets of 1.0 million residents benefiting from new water supply service, 3.0 million residents with improved access to water supply, and 0.5 million residents with access to new sanitation services. Six intermediate outcome indicators were monitored for Addis Ababa and the four secondary cities. The Results Framework included a formally revised target of assisting 6 water utilities (achieved). Overall the project made some improvements in increasing Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) access to sanitation services in both Addis Ababa and the target secondary cities but unclear if the infrastructure improvements would be sustainable beyond project close. However, the PDO was a relatively low aim – improved access to water and sanitation services – and did not include impact or indicators relative to improvements in health, or labor productivity, or economic activity, although these were mentioned as long-term outcomes that the project would contribute (ICR, paragraph 10). Overall Efficacy Rating Primary reason Modest Low achievement PHREVISEDTBL 5. Efficiency Economic and Financial Efficiency: At appraisal and then again for the AF, the project provided positive economic and financial estimates in Addis Ababa but not in the other cities. With the cancellation of about US$ 27.7 million, project costs were reduced, and associated expected benefits and impacts were also reduced. These reductions adversely affected the overall economic and financial efficiency of the project. At appraisal the EIRR for Addis Ababa was 20% but at closing, this was reduced to 9.5%. In the case of the secondary cities, the EIRR at appraisal was at 15% and this was reduced to 11.6% at closing with an overall 10.4% EIRR for the overall project. The FIRR for Addis Ababa registered at 15% at appraisal and reduced to 6.1% by project close while the FIRR for the secondary cities was at 3% at appraisal and -5.5% at closing, with an overall 1.5% for the overall project. The reduced FIRR was negatively affected by cost overruns, implementation delays, exchange rate losses (SDR to US$ and US$ to ETB). The water supply system commissioning was delayed due to delay in connecting to the main energy grid. There were efficiency gains cited due to the project's participatory design that ensured greater household buy-in, the step-by-step approach in assisting utilities who had to first demonstrate capacity to deliver capital projects and capacity to maintain additional assets, improvements to operational efficiency, customer management, and prioritizing service improvements to existing assets rather than new construction. Operational and Administrative Efficiency: The project was implemented over a 10 year period and was extended three times for a total of an additional 60 months. Procurement delays in association with large wastewater treatment plant construction and delays in resolving contract issues in Dire Dawa projects contributed to this implementation delay. Even with this substantial extension, the two big contracts associated with the Kality Addis Ababa sewerage system expansion and the Dire Dawa water system expansion was not completed by project close. The Government requested the cancellation of the remainder of the funds that would not be disbursed in time and transferred the projects for completion to the recently approved Second Ethiopia Urban Water Supply Project. The Government attributed the implementation delays to lengthy project preparation, lengthy procurement processes, limitations in technical, financial, and procurement and contract management capacity, and right of way issues. Delays in commissioning the wastewater treatment plant (construction delays, delays in getting households connected to the sewer network) resulted in another restructuring in 2017 and completion of the project activities were passed on to the Second Urban Water Supply and Sanitation Project (P156433). Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 62.80 Appraisal  20.00 Not Applicable 57.80 ICR Estimate  9.50 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives was high. With two objectives substantially achieved and two rated negligible, the overall Efficacy was rated Modest. The efficiency of the project was rated modest. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome • Financial and Technical Risks: The ICR did not provide information on institutional support for sustainable operations beyond the project of the investments made under this project, particularly the sanitation components. The ICR mentioned that the utilities of the secondary target cities were meeting their repayment obligations for loans provided by the WRDF, indicating that their operations were more efficient by project close. Future needs, particularly in tariff setting to meet revenues against future expenditures, for example, in improving the utilities were not addressed. The wastewater treatment plant and some of the works for Dire Dawa which were incomplete by project close were transferred to the Second Urban Water and Sanitation Project. This follow on project, the Second Urban Water and Sanitation Project was designed to mitigate the risks to the development outcome under the project. Among those risks included the lack of support for employing women or disadvantaged groups to manage public toilets in low income neighborhoods, the discontinued use of pay as you go models for these public toilets, infrastructure failures and repair needs of the Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) expanded water supply source and distribution, lack of software updates or equipment repair. In addition, the reliability of groundwater sources and sustainability of yields of water wells financed under the project were also risks that need to be addressed in the follow on project. • Operational Risks: The improvements introduced under the project in financial and operational efficiencies of the utilities may not be sustained over time unless there is continuing support for human and institutional gaps in O&M of the utilities. In addition, utilities may delay the effectiveness of cost recovery after setting tariffs to strengthen the financial standing of utilities and access private resources for future rehabilitation needs. This risk is partially mitigated by the improvements introduced under the project on operating efficiency, financial management and customer service improvements, although these operating norms may not be sustained over time unless covered under the follow on project. • Government ownership and ownership by stakeholders: There is a risk that the Government would abandon the model of a stepped approach to assisting utilities. This risk is mitigated by the design of the follow on project noted above. Interest in stakeholder participation in improving access to water and sanitation services by residents may wane, unless beneficiaries continue to observe the benefits of such participation. This risk could be mitigated by improving communication outreach and building the capacity for citizen participation, to be addressed also by the follow on project. • Operations and maintenance (O&M) needs of infrastructure investments funded under the project may not receive adequate budgetary resources. This risk is also mitigated by project design under the follow on project. 8. Assessment of Bank Performance a. Quality-at-Entry The project was relevant to the country's development plans and included appropriate activities to achieve the PDO in Addis Ababa - improve/expand water supply production/distribution, and sewage collection/treatment, improve operational efficiency of the utilities, and the governance of the Addis Ababa Water and Sanitation Authority (AAWSA); and in the targeted secondary cities - Gondar, Hawasa, Jimma, and Mekelle, with Dire Dawa added with the AF. In the case of the secondary cities, the activities included the construction of new sewer networks in those cities. Only Addis Ababa had a sewer network that served only 10% of the urban residents. During the AF, a new activity was added for Addis Ababa - the construction of a new waste water treatment plant. The project adequately assessed the capacity of the implementing entity although the numerous actors involved in implementation revealed shortcomings in procurement and financial management capacity, especially at the level of the utilities. The approach used in selecting the secondary cities was sound as the targeted cities demonstrated capacity, interest, and buy-in to the PDO. However, the indicators in the Results Framework were mostly geared toward assessing the outcome in the water component, and only access at that, rather than sustainability of service delivery, nor were there sufficient indicators to support the evidence for the project impact on the sanitation component. There were stakeholder consultations Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) on project design but there were no household surveys during preparation and the project relied on the baseline data provided by the utilities. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team provided adequate supervision at least twice a year (for a total of 22 missions over the 10 year implementation period) with candid assessments of progress. There were no TTL turnovers. Candid assessment of disbursement delays led to an early MTR but corrective measures proved inadequate to address the recurrent challenges brought by capacity issues in financial management and procurement. Safeguard issues were addressed as these occurred although lack of personal protective equipment and other site safety measures were noted during ICR preparation site visits. Even with candid assessments during supervision mission, the Bank team delayed in downgrading performance ratings. For example, despite multiple supervision mission reports (ISRs) indicating that the sewer network activities in Addis Ababa were behind schedule, progress toward achieving the PDO remained Moderately Satisfactory until the final ISR. To avoid any further extension to the project, the 2017 restructuring dropped the following activities in Addis Ababa and passed these on for completion by the follow on Second Urban Water and Sanitation Project (P156433): (i) completion of the trunk line by another 4 km; (ii) completing and commissioning the waste water treatment plant; (iii) household sewer connections. The following activities in the secondary target city of Dire Dawa were also passed on this project because of termination due to poor performance of the drilling contractor and delays in resolving a procurement complaint: (i) completion of the electromechanical equipment, pipes supply, and installation works; (ii) extension of the network to unserved areas; (iii) provision of tertiary distribution and household connections. In both Addis Ababa and the target secondary cities, the planned GPOBA subsidies for low income household connections were dropped during the MTR because the Government was not prepared to provide program requirements of one to one matching of subsidies. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) The implementing agency was the Addis Ababa Water and Sewerage Authority for the city of Addis Ababa and the Ministry of Water, Irrigation, and Electricity (then the Ministry of Water Resources) for the targeted secondary cities of Gondar, Hawassa, Jimma, Mekelle, and Dire Dawa. According to the PAD, the utilities of the five cities (Dire Dawa was added during the AF), provided the baseline data based on available records but would be regularly updated once the M&E system was in place. Household surveys in the participating cities were to be conducted at the beginning of project implementation and at project close to support project outcome. Each city utility operator would regularly collect performance information and each city water board would submit updated semi annual reports to the then Ministry of Water Resources and the Bank. The staff of the MWR and Regional Water Boards were responsible for M&E. The staff received initial M&E training and were expected to have on-the job M&E training during implementation (PAD, paragraphs 61-63). The third project component focused on strengthening M&E at various levels. However, the design framework focused on expansion in the access to water supply with little attention paid to sanitation aspects of the project or its sustainability. The PAD defined sustainability of the project to be measured by the autonomy of the utilities and performance agreements between the utility and their boards. The sanitation indicators did not fully capture the sanitation investment activities. The indicators in the Results Framework focused on infrastructure creation rather than its use. b. M&E Implementation The utilities were trained in M&E processes and provided inputs to the M&E system and institutionalized some processes of the regular data collection and M&E principles. The framework was too narrowly focused on expanding access to the water supply. A number of indicators for sanitation were not tracked. For example, the sustainability indicators noted above in design were not tracked as part of the results framework. Indicators for sanitation, particularly for those outside of Addis Ababa, did not reflect all the investments made in the sanitation sector and not adequately tracked. The results framework also included indicators which were difficult to track regularly and should have been replaced during the 2012 AF. Most of the indicators used were also focused more on creating the infrastructure rather than the use of the infrastructure services. There was no baseline survey conducted early on in the project or even at project close, although baseline data was provided by the participating utilities. There was poor planning on how the Results Framework indicators were going to be monitored in practice as evidenced by the dropping of one indicator and the struggle to measure the operation ratio outside Addis Ababa for the life of the project. Some of the estimation techniques were overly simplified and could have benefited from the 2012 AF with minor changes to increase attribution. c. M&E Utilization The project had limited use of the M&E system because of its focus mostly on water supply access factors. The project used the results framework to help inform the restructuring and overall supervision follow up priorities, but did not benefit the non-water supply (sanitation) aspects of the project. Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) M&E Quality Rating Modest 10. Other Issues a. Safeguards Environmental and Social Safeguards. The project was assigned an Environmental Category B for the purposes of OP/BP4.01, which requires a partial environmental assessment. The then Ministry of Water Resources (MWR) would oversee the implementation of the Environmental and Social Management Framework (ESMF) and the Resettlement Policy Framework (RPF). Since the potential adverse environmental and social impacts of project investments were not identified at appraisal, the ESMF as well as the Environmental and Social Management Plan and Environmental Guidelines for Contractors were to be included in the project bidding documents (PAD, paragraph 82). Environmental and Social Assessments as well as Resettlement Action Plans RAPs) were developed for all participating cities. In addition to OP/BP 4.01, Environmental Assessment, the project also triggered the following safeguards: OP/BP 4.11 Physical Cultural Resources, in case a cultural heritage site was found during implementation, OP/BP 4.12 Involuntary Resettlement, OP/BP 4.37 Safety of Dams and OP/BP 7.50 Projects on International Waterways. The project complied with all applicable environmental and social safeguards. All 89 project affected people were compensated following the RAPs. There was no cultural heritage site uncovered during project implementation. The necessary dam safety plans were created for Gondar and Addis Ababa and the Bank cleared the Terms of Reference for the independent panel of experts to be utilized. In the case of international waterways, the Bank notified the Governments of Djibouti, Egypt, Eritrea, Kenya, and Sudan and by the deadline provided, Sudan responded positive. Kenya after requesting more time, did not provide any further response. Eritrea objected to the project after the deadline but the Bank clarified that the project did not fall within the disputed boundaries. There was no further response from Eritrea afterwards. There were no responses registered from Egypt or Djibouti. b. Fiduciary Compliance Financial Management: The project complied with financial management requirements although there were reporting delays early in the project due to low capacity level at the utility levels and accounting challenges, poor budgeting, budget monitoring, long outstanding balances that could have been addressed early in the project. High staff turnover contributed to the low capacity. Issues were adequately addressed after supervision missions except for the long outstanding balances, which remained an issue throughout implementation. Procurement: A majority of the sub-projects complied with procurement requirements. However, non- compliance incidents were pervasive. Procurement capacity were low to begin with, particularly in the target secondary cities. Training did not adequately address this shortcoming. This lack of procurement and financial management capacity led to considerable implementation delays. While the Government had the Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) staff, there was high turnover. There was a lack of state user manuals. There was some confusion about the applicability of Bank procurement guidelines and the Government's own procurement guidelines. There was a procurement complaint in Dire Dawa and auditors were unsure which guidelines applied, which delayed procurement. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR presented the following lessons from the operations: • Using a revolving fund, such as WRDF to provide on-lending to city utilities after they provide matching funds may leverage resources and help ensure buy-ins from participating utilities. Evidence was provided by repayment patterns by these utilities. • Project design for self supporting utilities needs to focus on a realistic cost of connections to the infrastructure trunk, the willingness and ability to pay of those who will be connected, and access preferences at the household level. • When investments require access to energy supply, and if target communities may be outside the existing grid, alternative technologies that rely on low energy requirements may need to be adopted. 13. Assessment Recommended? Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Urban WSS SIL FY07) (P101473) No 14. Comments on Quality of ICR The ICR was concise, followed the guidelines, and was focused on results based on evidence. There was a detailed overview of the project and the narrative supported the ratings and available evidence. The theory of change was clear and convincing. The quality of evidence and candid analysis were aligned to the PDO. The lessons presented were useful and based on evidence. There were minor shortcomings in the completeness of data and information, particularly in the lack of indicators in the use of the infrastructure services in the Results Framework, or the lack of beneficiary surveys, which the ICR acknowledged. a. Quality of ICR Rating Substantial Page 16 of 16