Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Report Number: ICRR0023036 1. Project Data Project ID Project Name P112719 BUKHARA & SAMARKAND SEWERAGE PROJECT Country Practice Area(Lead) Uzbekistan Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46330,IDA-56980,IDA-56990 31-Dec-2015 132,854,892.12 Bank Approval Date Closing Date (Actual) 04-Aug-2009 30-Nov-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 55,000,000.00 0.00 Revised Commitment 143,445,185.30 0.00 Actual 132,854,892.12 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally Vibecke Dixon Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives Original PDO. The Project Development Objective (PDO) of the Bukhara and Samarkand Sewerage Project (BSSP) as articulated in the Financing Agreement (FA, page 5) was identical to the one in the Project Appraisal Document (PAD, paragraph 10) and aimed to: "mitigate the environmental impact from wastewater pollution and improve the efficiency and sustainability of wastewater management in Bukhara and Samarkand." Page 1 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Revised PDO. The PDO was revised in 2015 as part of a Level 1 Restructuring that included Additional Financing. The PDO of the BSSP as articulated in the Financing Agreement (FA, page 6) was to: "reduce wastewater pollution and improve the performance of utilities responsible for wastewater management in Bukhara and Samarkand." Parsing the PDO. 1. The original PDO will be parsed based on the following three objectives: (a) To mitigate the environmental impact from wastewater pollution in Bukhara and Samarkand (PDO 1); (b) To improve the efficiency of wastewater management in Bukhara and Samarkand (PDO 2); and (c) To improve the sustainability of wastewater management in Bukhara and Samarkand (PDO 3). 2. The revised PDO will be parsed based on the following two objectives: (a) To reduce wastewater pollution in Bukhara and Samarkand (PDO 1); and (b) To improve the performance of utilities responsible for wastewater management in Bukhara and Samarkand (PDO 2). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 23-Jun-2015 c. Will a split evaluation be undertaken? Yes d. Components The PDO was supported by the following three components: 1. Physical Investments (appraisal cost: US$62.40 million, of which IDA would finance: US$51.88 million, additional financing (AF): US$99.14 million, actual cost: US$52.74 million, AF: US$80.03 million). This component included the following two sub-components: Page 2 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) 1.1. Rehabilitation of Bukhara Sewerage System. This would include: (a) rehabilitating and replacing existing sewers; (b) rehabilitating existing wastewater pumping stations; (c) expanding (2.5km) the sewer system; (d) rehabilitating the Bukhara Wastewater Treatment Plant (WWTP), including rehabilitating and replacing equipment, financing minor civil works and enhancing the plant's energy efficiency; (e) providing operational equipment; and (f) providing engineering consultants and other rehabilitation-related technical assistance (TA). 1.2. Rehabilitation of Samarkand Sewerage System. This would include: (a) rehabilitating and replacing existing sewers; (b) rehabilitating existing wastewater pumping stations and constructing an additional one; (c) expanding (4.8km) the sewer system; (d) rehabilitating the Main Wastewater Treatment Plant and the Farhad Wastewater Treatment Plant, including rehabilitating and replacing equipment, financing minor civil works and enhancing the Plant's energy efficiency; (e) providing operational equipment; and (f) providing engineering consultants and other rehabilitation- related TA . 2. Institutional Strengthening and Capacity Building (appraisal cost:US$1.04 million, of which IDA would finance: US$0.87 million, AF: US$4.29 million, actual cost: US$0.07 million, AF: US$1.25 million). This component would focus on improving utility management and consumer orientation by training staff and introducing better communication and public awareness efforts in the two vodokanals. This component included the following two sub-components: 2.1. Institutional strengthening and capacity building of Bukhara Vodokanal (BVK) and local communities. This would be achieved through: (a) development of communications strategies and public awareness raising campaigns; (b) staff training in utility management with an emphasis on improving the consumer orientation; and (c) establishment of a Pilot Supervisory Control and Data Acquisition System (SCADA) to enhance responsiveness to emergencies and to routine maintenance. 2.2. Institutional strengthening and capacity building of Samarkand Vodokanal (SVK) and local communities. This would be achieved through: (a) development of communications strategies and public awareness raising campaigns; (b) staff training in utility management with an emphasis on improving the consumer orientation; and (c) establishment of a Pilot Supervisory Control and Data Acquisition System to enhance responsiveness to emergencies and to routine maintenance. 3. Feasibility Studies for Future Investments (appraisal cost US$1.07 million, of which IDA would finance US$ 0.89 million, actual cost: US$0.48 million). This component would finance consultant studies for future Government priority investments in the water supply/sanitation sectors. It would be managed by the central government. 4. Project Management (appraisal cost US1.64 million, of which IDA would finance US$:1.35 million, AF: US$1.5 million, actual cost: US$1.56 million, AF: US$0.69 million). This component would finance the strengthening of Project Coordination Unit (PCU), including its branches in Bukhara and Samarkand. Under this item, funds would also be provided for monitoring and evaluating (M&E) project activities including supporting: technical, environmental, social monitoring, project audits, training and financing of operating costs. Revised Components. The structure and order of the components were revised. The change included consolidating the original components 2 (Institutional Support) and 3 (Feasibility Studies) and prioritizing them as component Page 3 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) 1(Institutional Strengthening and Capacity Building), reflecting the renewed emphasis on institutional strengthening and performance beyond infrastructure, and building on the growing reform agenda. Component 1 was recategorized as component 2, and component 4 as component 3. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total cost of the project was estimated at US$66.16 million including US$2.32 million and US$3.45 million physical and price contingencies, respectively. The project received additional IDA financing in the amount of US$105.00 million, which increased the total cost to 171.16 million (see below for details). This amount was revised downwards to US$143.45 million. The actual cost according to the ICR Data Sheet (page 2) was US$136.82 million. Financing. The project was originally financed through an IDA loan of US$55.00 million (revised down to US$53.45 million). In 2015, the project received additional IDA financing of US$105.00 million (revised down to US$90.00 million). The total IDA Credit was US$160.00 million (revised amount US$143.45 million). The actual disbursed amounts were US$50.89 million and US$81.97 million for the original IDA Credit and the AF, respectively. The total disbursed IDA amount was US$132.85 million (ICR Data Sheet, page 2). The ICR (paragraph 20) stated that in 2020 the project was restructured and US$15.00 million of IDA funds were cancelled. According to the ICR (paragraph 20) the cancellation of funds was due to cost savings from competitive procurement bidding. Borrower Contribution. The borrower was expected to contribute US$8.00 million of counterpart funds. According to the ICR (footnote #10) the borrower contributed US$3.90 million of counterpart funds. The lower than expected counterpart contribution was due to challenges faced by the Government. The ICR did not elaborate on the nature of these challenges. Dates. The project was approved on August 4, 2009 and became effective eight months later on April 3, 2010. The Mid-term Review (MTR) was conducted in September 30, 2013, which was about three years and five months after effectiveness. While the PAD did not specify a date for the MTR, this Review finds that conducting the MTR 3.5 years after effectiveness was reasonable. The project closed on November 30, 2021, about six years beyond the original closing date on December 31, 2015. The six years extension was to accommodate the implementation delays and allow enough time to complete activities. These delays were also in the context of major political and sectoral changes following a change in political leadership in 2017. There was a significant push for reforms across the sector, and frequent changes to the project’s authorizing environment. These included the creation of the line ministry (Ministry of Housing and Communal Services - MHCS) and formation of regional WSS utilities, among others, as well as new internal procedures for procurement clearances. Implementation delays had been aggravated by the impacts of the COVID-19 pandemic, including restrictions on imports of goods and equipment, as well as human resource capacity issues associated with the pandemic. The project was restructured four times, the first was a Level 1 and the remaining three were all Level 2 restructurings as follows: 1. On June 23, 2015, when the amount disbursed was US$37.99 million, in order to approve an Additional IDA financing of US$105 million, revise the PDO, revise the Results Framework (RF) and targets, change Page 4 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) components and cost, extend the loan closing date by three years and six months from December 31, 2015 to June 30, 2019, change in disbursements arrangements, change in legal covenants, and change in implementation schedule. 2. On November 2, 2018, when the amount disbursed was US$69.54 million, in order to extend the AF loan closing date by 18 months, until December 31, 2020, to enable sufficient time to complete ongoing activities, revise the RF, and change the implementation schedule. 3. On March 25, 2020, when the amount disbursed was US$96.29 million, in order to change the components and cost, cancel US$15.00 million financing, and reallocate funds between disbursement categories. 4. On December 22, 2020, when the amount disbursed was US$103.61 million, in order to extend the AF loan closing date from December 31, 2020 to November 30, 2021, and revise the RF. Rationale for Changes and impact on the Theory of Change (ToC). The project costs were underestimated in feasibility studies, counterpart financing was removed from project funding sources, and the SDR experienced a significant devaluation against the US dollar. Hence, an additional financing was required to scale up the project and cover the financing gap from the parent project. The revision of the PDO made it measurable and attributable by focusing on wastewater pollution removal rather than environmental outcomes. The change also broadened the project’s ambition as the project’s objectives were changed to address utility performance more holistically including customer relations aspects. Overall, the changes were relevant and justified. While the PDO was revised, this did not impact the ToC. 3. Relevance of Objectives Rationale Context at Appraisal. The sewerage systems in Bukhara and Samarkand were more than 40 years old and needed to be replaced. Deferred maintenance needed to be addressed through a combination of crash preventive maintenance and rehabilitation. The project aimed to improve service delivery and manage the environment in a sustainable manner. Previous Bank Experience. The Bank benefited from the experience and trust developed with the central, regional, and municipal governments during the recent implementation of the Bukhara and Samarkand Water Supply Project (BSWSP). This project would also benefit from improved capacity of the Project Coordination Unit (PCU) which under BSWSP became adept at implementing Bank financed projects. The project also drew on the recommendations in the Bank’s 2007 Municipal Services Sector Review Note as well the experience under the Rural Water Supply, Sanitation and Health Project (closed in 2008). Consistency with Bank Strategies. At appraisal, the PDO was in line with the objectives of the Bank’s Country Assistance Strategy for Uzbekistan (CAS, 2008) by addressing the need to (a) improve service delivery and (b) manage the environment in a sustainable manner. Page 5 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) At completion, the PDO remained in line with the current Bank Country Partnership Framework for Uzbekistan (CPF, FY16–FY20). Focus areas under the CPF were based closely on the Uzbekistan Systematic Country Diagnostic (SCD, FY16–FY20), which identified a significant need to further develop water infrastructure and to strengthen the Water Supply and Sanitation (WSS) sector management. Specifically, under focus area (c) public service delivery, where the CPF identifies two core actions highly relevant to BSSP: 1. Increasing access and improving the quality of water supply and sanitation services and 2. Promoting energy security and efficiency and reducing the economy’s energy intensity. These were reflected in the operational programming in the CPF, including three WSS-related operations, one of which was BSSP. Consistency with Government Strategies. At appraisal, the PDO was in line with the Government's National Environmental Action Plan (1998) and National Action Plan for Environmental Hygiene of the Republic of Uzbekistan (1999), in which upgrading urban sewerage and municipal wastewater treatment facilities was a priority. At completion, the PDO remained in line with Government's strategy and plans in the WSS sector. Specifically, with the priorities of the Program for the Comprehensive Development and Modernization of Water Supply and Sewerage Systems (2017–2021) which was the key sector strategy at the time of project closing. It prioritized integrated measures such as continued reconstruction and rehabilitation of infrastructure; adoption of modern technologies, information systems, and automated billing; and the introduction of sustainability-aligned tariffs, with the view of improving public service delivery. Level of PDO The original PDO was pitched at an ambitious level to address the development outcome. The revised PDO was more realistic and was pitched at an adequate level to address the development outcome. Summary of Relevance of Objectives Assessment. The original and revised objectives were both in line with Bank's CPF as well as with Government's priorities for the WSS. The revision of the PDO made it measurable and attributable by focusing on wastewater pollution removal rather than environmental outcomes. Therefore, Relevance of Objectives is rated High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To mitigate the environmental impact from wastewater pollution in Bukhara and Samarkand. Rationale Page 6 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Theory of Change (ToC). To achieve the stated objective, the project would finance the rehabilitation and expansion of the Bukhara and Smarkand sewerage systems including pumping and treatment stations. These activities would rehabilitate and expand sewerage systems including energy efficient pumps, and wastewater treatment station with energy efficient equipment would be completed and operational. These activities were expected to increase household access to sewerage service and increase the pollution removed from wastewater through wastewater treatment plants. This would result in reducing the environmental impact from wastewater pollution. The achievement of the PDO was underpinned by the following assumptions: 1. The project assumed that households would connect to sewerage service through self-funding or support from local governments; 2. The project interventions would be sufficient to mitigate the environmental impact from wastewater despite other non-project related impacts; and 3. Public awareness campaigns would be able to effectively change customer behavior. While the stated activities were relevant and connected to the PDO, improving environmental outcomes proved to be a broad target and beyond the full scope of the project. Hence, the PDO was later revised to make it more focused, measurable and attributable to the project activities. The stated assumptions were questionable, because poor customers might not be able to connect if they do not receive support, and environmental impacts proved to be broad and beyond the full scope of the project. Outputs/Intermediate Results Indicators (IRIs) 1. 49.48 km and 34.35 km sewer lines replaced in Bukhara and Samarkand (59% and 100% of the original targets achieved). 2. 51.46 km and 91.70 km new sewer lines laid in Bukhara and Samarkand (125% and 155% of the original targets achieved). 3. 24,464 and 14,691 households connected to the public sewerage system in Bukhara and Samarkand (544% and 105% of the original targets achieved). Outcome In the original design, PDO 1 was to be measured through two indicators: 1. Pollution load on downstream communities measured by sewerage removed at WWTP (5-Day Biological Oxygen Demand (BOD5) tons/year), and 2. Collection of sewerage into the WWTP (tons/year). For the first indicator, the project achieved 1,765 BOD5 tons/year compared to an original target of 1,950 BOD5 tons/year, and a baseline of 1,590 BOD5 tons/year for Bukhara Vodokanal (BVK) (achievement rate 48%). Results were better for Samarkand Vodokana (SVK) where the project achieved 3,201 BOD5 tons/year compared to an original target of 3,050 BOD5 tons/year, and a baseline of 2,450 BOD5 tons/year (achievement rate 125%). For the second indicator, the project achieved 2,006 tons/year compared to an original target of 2,200 tons/year, and a baseline of 2,000 tons/year for BVK (achievement rate 3%), while for SVK, the project achieved 4,042 tons/year compared to an original target of 3,400 tons/year, and a baseline of 2,700 tons/year for BVK (achievement rate 192%). Summary of Efficacy Assessment of PDO1. While the project met or exceeded most targets for its IRIs, it fell short on meeting the outcome targets for BVK, but achieved or exceeded targets for SVK. However, the afore-mentioned outcome indicators were not enough to measure environmental impact from wastewater Page 7 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) (ICR, paragraph 26). The PDO was phrased at a high level, but the original RF indicated that improved environmental impact was to be achieved through better treatment, additional wastewater connections, elimination of wastewater losses through damaged sewers. There was clear evidence of increased pollution load removed and increased collection of wastewater, as well as additional beneficiaries and decreased sewer breakages. The project achieved mixed results against its outcome indicators, but given the insufficiency of adequate indicators to measure the outcome, the rating for PDO 1 is Modest. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective To reduce wastewater pollution in Bukhara and Samarkand. Revised Rationale Theory of Change (ToC). To achieve the stated objective, the project would finance the rehabilitation and expansion of the Bukhara and Smarkand sewerage systems including pumping and treatment stations. These activities would rehabilitate and expand sewerage systems including energy efficient pumps, and wastewater treatment station with energy efficient equipment would be completed and operational. These activities were expected to increase household access to sewerage service and increase the pollution removed from wastewater through wastewater treatment plants. This would result in reducing the wastewater pollution in Bukhara and Samarkand. The achievement of the PDO was underpinned by the following assumptions: 1. The project assumed that households would connect to sewerage service through self-funding or support from local governments; 2. The project interventions would be sufficient to mitigate the environmental impact from wastewater despite other non-project related impacts; and 3. Public awareness campaigns would be able to effectively change customer behavior. The stated activities were relevant and directly connected to the PDO in a plausible causal chain. The revised PDO was more focused, measurable and attributable to the project activities. However, the stated assumptions were questionable, because poor customers might not be able to connect if they do not receive support. Outputs 1. 49.48 km and 34.35 km sewer lines replaced in Bukhara and Samarkand (59% and 100% of the revised targets achieved) 2. 51.46 km and 91.70 km new sewer lines laid in Bukhara and Samarkand (125% and 155% of the revised targets achieved) 3. 24,464 and 14,691 households connected to the public sewerage system in Bukhara and Samarkand (544% and 105% of the revised targets achieved). 4. By project closing, a two-phased rehabilitation of the Samarkand WWTP (serving a population of 250,000) and the construction of an entirely new WWTP in Farkhad neighborhood (serving a population of 8,300) were Page 8 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) both completed and commissioned. Prior to the project, in Samarkand effluent from the existing sewer network was discharged with only partial compliance with national standards, while in Farkhad, collected raw sewerage was discharged into the Zarafshan River (ICR, paragraph 28). Outcome  The reduction of wastewater pollution was primarily driven by rehabilitation/construction of Wastewater Treatment Plants (WWTPs) and by increased household connections. Also, while not measured under the project, it is plausible to assume that reduced sewer blockages, breakdowns, and wastewater spills had a positive impact on reducing wastewater pollution.  The project exceeded the target of 3,100 tons/year of BOD5 removed from Samarkand WWTP with a final value of 3,148 tons/year, and an additional 53.1 tons/year from Farkhad WWTP.  14,691 households were connected to the sewerage system exceeding the revised target of 14,000. Sewerage blockages and breaks, which resulted in raw sewerage spills and wastewater pollution, decreased from 1,056 per year in 2009 to 345 in 2021, or 67%, exceeding the revised target of 700.  The project also financed a two-phased rehabilitation of the Bukhara WWTP. The first phase rehabilitated the WWTP’s capacity to handle wastewater up to 50,000 m3/day. By project completion, the plant was able to treat all wastewater collected in line with national standards (daily inflows of sewerage reached 43,960 m3/day by 2021, ICR, paragraph 29). The WWTP removed 1,765 tons BOD5/year compared to a revised target of 2,000. Prior to the project, the effluent was discharged without complying with national standards. The ICR (paragraph 29) explained that while the plant was able to treat all wastewater currently received, the target was not met due to the quantity of sewerage currently received, and the achievement of this target was not affected by the incomplete works under Phase II.  In Bukhara, sewerage coverage increased from 56 to 70%, with 24,464 households (122,320 people) gaining first-time access to sewerage services and significantly exceeding the target of 4,500 households. Sewerage blockages and breakdowns decreased from 1,404 per year in 2009 to 589 in 2021, or 58%, exceeding the target of 600 per year. Summary of Efficacy Assessment. The project met or exceeded its IRIs (except one) , and outcome targets were met or exceeded. Phase II of the Bukhara WWTP was expected to be completed with the secured Government financing. The evidence provided in the ICR point to the success of the project in reducing wastewater pollution in Bukhara and Samarkand. Therefore, the efficacy with which this objective was achieved is rated Substantial. Revised Rating Substantial OBJECTIVE 2 Objective To improve the efficiency of wastewater management in Bukhara and Samarkand. Rationale Page 9 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Theory of Change (ToC). To achieve the stated objective, the project would train utility staff on O&M, customer orientation, financial management, and corporate management. The project would support training on the implementation of supervisory control and data acquisition systems (SCADA), and run public awareness campaigns. This was expected to improve data acquisition and management, improve the capacity of utilities staff, increase awareness of customers with regards to the need of regular payments and the benefits of sewerage connections, and increase revenue as a result of new household connections. This would result in improved energy efficiency in sewerage systems and wastewater treatment plants, improved operational and financial sustainability of utilities, and reduced sewerage blockages and repairs. All this combined would result in the improvement of the efficiency of wastewater management in Bukhara and Samarkand. Anticipated long-term outcomes included improved public health and sustainable development. The achievement of the PDO was underpinned by the following assumptions: 1. The project assumed that households would connect to sewerage service through self-funding or support from local governments; 2. The project interventions would be sufficient to mitigate the environmental impact from wastewater despite other non-project related impacts; and 3. Public awareness campaigns would be able to effectively change customer behavior. The stated activities were relevant and directly connected to the PDO in a plausible causal chain. However, the stated assumptions were questionable, because poor customers might not be able to connect if they do not receive support, and public awareness campaigns does not necessarily change people’s behavior. Outputs The RF did not have any outputs indicators under this objectives.  Energy efficiency was achieved due to modernization of WWTPs with variable speed blowers and process automation (ICR, paragraph 31). However, the ICR did not provide further details on the number of speed blowers installed.  All six of Samarkand’s existing pumping stations were rehabilitated during the original project, with two additional pumping stations constructed with the AF (ICR, paragraph 31).  Out of 16 pumping stations in Bukhara city, 10 were replaced and 2 rehabilitated in the original project, and a further 4 replaced during the AF (ICR, paragraph 31). Outcome Improving the efficiency of wastewater management in Bukhara and Samarkand was driven by improving energy efficiency at the corresponding WWTPs. At the Bukhara WWTP, energy intensity decreased from 4.1 to 2.5 kWh/kg BOD5 removed, exceeding the target of 3.0 and representing an energy cost saving of 39%. At the Samarkand WWTP, energy intensity decreased from 4.0 to 1.90 kWh/kg BOD5 removed, exceeding the target of 3.0 and representing an energy cost savings of 51% (ICR, paragraph 31). The results achieved under the project represented an estimated cost saving of US$112,960 for Bukhara and US$268,884 per year for Samarkand. The ICR (paragraph 31) explained that a score less than1.5 kWh/kg BOD5 removed was considered good according to benchmarking studies, while a score higher than 5 kWh/kg BOD5 removed suggested low efficiency. Page 10 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Summary of Efficacy Assessment. The above-mentioned results demonstrated that the project significantly improved energy efficiency of wastewater management. The targets for the outcome indicator were both exceeded as noted above. Therefore, the efficacy with which this objective was achieved is rated High. Rating High OBJECTIVE 2 REVISION 1 Revised Objective To improve the performance of utilities responsible for wastewater management in Bukhara and Samarkand. Revised Rationale Theory of Change (ToC). To achieve the stated objective, the project would train utility staff on O&M, customer orientation, financial management, and corporate management. The project would support training on the implementation of supervisory control and data acquisition systems (SCADA), and run public awareness campaigns and support the development of feasibility studies for future projects. This was expected to improve data acquisition and management, improve the capacity of utilities staff, increase awareness of customers with regards to the need of regular payments and the benefits of sewerage connections, and increase revenue as a result of new household connections. This would result in improved energy efficiency in sewerage systems and wastewater treatment plants, improved operational and financial sustainability of utilities, and reduced sewerage blockages and repairs. All this combined would result in the improvement of the efficiency of wastewater management in Bukhara and Samarkand. Anticipated long-term outcomes included improved public health and sustainable development. The achievement of the PDO was underpinned by the following assumptions: 1. The project assumed that households would connect to sewerage service through self-funding or support from local governments; 2. The project interventions would be sufficient to mitigate the environmental impact from wastewater despite other non-project related impacts; and 3. Public awareness campaigns would be able to effectively change customer behavior. The stated activities were relevant and directly connected to the PDO in a plausible causal chain. However, the stated assumptions were questionable, because poor customers might not be able to connect if they do not receive support. Outputs 1. 1,102 utility staff trained (110% of the revised target achieved). 2. SCADA system operational in Samarkand WWTP and Farkhad WWTP (revised target achieved). 3. Three FS completed (revised target achieved). Outcome Page 11 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Energy efficiency and operational and financial sustainability were achieved with High and Substantial efficacy, respectively, as discussed under original PDO2 and PDO3. Improving performance would be assessed through: (a) coverage of services and (b) customer service aspects. By project completion, 545,775 people received either new access to the sewer network or are ready to connect to sewers in front of their houses, exceeding the revised target of 500,000 people. This number does not include households who already had connections before the project, and who now receive higher levels of services (ICR, paragraph 36). Improving customer service was achieved through training of utility staff and public awareness outreach. According to the ICR (paragraph 37) "training improved the professionalism of utilities as demonstrated by enhanced customer information systems and data utilization and more rapid response to incidents." A pilot survey using a mobile application in Bukhara showed that out of 134 respondents, 94% were satisfied with their level of service in 2021. However, the impact of training on the performance of utilities was not explicitly monitored (ICR, paragraph 37). With implementation delays, the focus of the project was more on the completion of large infrastructure components, and the communications and public awareness program including training became "less of a priority" (ICR, paragraph 37). A pilot survey using a mobile application in Bukhara showed that out of 134 respondents, 94%were satisfied with their level of service in 2021. Summary of Efficacy Assessment. The project exceeded its target on beneficiaries and improved energy efficiency and operational and financial sustainability. The pilot beneficiary survey demonstrated a high level of satisfaction among approximately 100 of the customers. That said, there was no evidence to demonstrate actual improved customer service and nothing to show for actual public awareness outreach. Overall, efficacy of achieving this objective is rated Modest. Revised Rating Modest OBJECTIVE 3 Objective To improve the sustainability of wastewater management in Bukhara and Samarkand. Rationale Theory of Change (ToC). To achieve the stated objective, the project would improve the capacity of utility staff by the provision of training, support the operationalization of the SCADA system, and support public awareness campaigns, support feasibility studies for future projects. These activities would improve the capacity of utility staff, increase revenues from new household connections, reduce blockage and repairs, and improve financial sustainability of utilities through increasing revenue and decreasing costs. As a result the sustainability of wastewater management in Bukhara and Samarkand would be improved. Anticipated long- term outcomes included improved public health and sustainable development. The achievement of the PDO was underpinned by the following assumptions: 1. The project assumed that households would connect to sewerage service through self-funding or support from local governments; 2. Page 12 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) The project interventions would be sufficient to mitigate the environmental impact from wastewater despite other non-project related impacts; and 3. Public awareness campaigns would be able to effectively change customer behavior. The stated activities were relevant and directly connected to the PDO in a plausible causal chain. However, it was not clear how supporting the development of feasibility studies would contribute to the achievement of the PDO. Also, the stated assumptions were questionable, because poor customers might not be able to connect if they do not receive support, and public awareness campaigns does not necessarily change people’s behavior. Outputs 1. 1,102 utility staff trained (110% of target achieved). 2. SCADA system operational in Samarkand WWTP and Farkhad WWTP (target achieved). 3. Three FS completed (target achieved). Outcome Improving sustainability of wastewater management in Bukhara and Samarkand was assessed through two main elements relating to the financial and operational sustainability of the two utilities. Financial sustainability was measured by OCC (the ratio of total operating revenues over total operating costs), while operational sustainability was measured by the number of sewer blockages. Prior to the project, both Vodokanals were receiving significant Government subsidies to pay debt obligations and cover exchange rate losses, and incurring financial losses caused by low tariffs and high energy costs, among other reasons (ICR, paragraph 33). By project closing, the Bukhara Suvtaminot had an OCC of 1.65 (exceeding the target of 1.45) and Samarkand Suvtaminot 1.36 (partially achieving the target of 1.40). The project achieved mixed results on OCC targets where it was exceeded for Bukhara Suvtaminot and partially achieved for Samarkand Suvtaminot. The ICR (paragraph 33) noted that OCC was volatile through implementation due to increased operational costs associated with large infrastructure development. Operational sustainability of the sewerage system improved as evidenced by a decrease in blockages from 1,404 per year in 2009 to 589 in 2021 in Bukhara, exceeding the target of 600 per year; 51.46 km of new sewers were laid (41 km target) and 49.48 km replaced (82.70 km target). The ICR (paragraph 34) noted that prior to the project, the number of blockages in the older parts of the Bukhara sewerage system was about 160 times as high as the blockage rate in well-managed US utilities. In Samarkand, sewerage blockages decreased from 1,056 per year in 2009 to 345 in 2021, exceeding the target of 700; 91.7 km of new sewers were laid (59.2 km target) and 34.35 km replaced (34.35 km target). Summary of Efficacy Assessment. The project achieved mixed results. While the targets for operational sustainability were exceeded, targets for financial sustainability were only partially achieved. That said, and with a 100% increase in tariffs in 2021, it is plausible to assume that going forward financial sustainability of wastewater management will experience further improvements. Therefore, the efficacy with which this objective was achieved is rated Substantial, despite some shortcomings. Page 13 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Original Objectives. The project achieved mixed results against its outcome indicators, but given the insufficiency of adequate indicators to measure the outcome, the rating for PDO 1 was Modest. The project significantly improved energy efficiency of wastewater management. The targets for the outcome indicator were both exceeded as noted above. The rating for PDO 2 was High. As for PDO3, the project achieved mixed results. While the targets for operational sustainability were exceeded, targets for financial sustainability were only partially achieved. That said, and with a 100% increase in tariffs in 2021, it is plausible to assume that going forward financial sustainability of wastewater management will experience further improvements. Therefore, the rating for PDO3 was Substantial. Overall efficacy before restructuring is rated Modest due to insufficiency of indicators to measure the outcome of the first objective. Overall Efficacy Rating Primary Reason Modest Insufficient evidence OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Revised Objectives. The evidence provided in the ICR point to the success of the project in reducing wastewater pollution in Bukhara and Samarkand. Specifically, the project exceeded the revised target of 3,100 tons/year of BOD5 removed from Samarkand WWTP with a final value of 3,148 tons/year, and an additional 53.1 tons/year from Farkhad WWTP. The project also reduced sewerage blockages and breaks, which resulted in raw sewerage spills and wastewater pollution, from 1,056 per year in 2009 to 345 in 2021, or 67%, exceeding the revised target of 700. Therefore, PDO1 is rated Substantial. The project exceeded its target on beneficiaries and improved energy efficiency and operational and financial sustainability. Also, anecdotal evidence point to a high level of satisfaction among customers. PDO2 is rated Substantial. Overall, the project resulted in improved access to sanitation, improved wastewater collection and treatment, and improved utility performance and sustainability, particularly in energy efficiency and operational performance. Therefore, overall Efficacy is rated Substantial, but barely so. There was a lack of adequate indicators to comprehensively measure the project achievements related to operational sustainability, and the original targets for financial sustainability were only partially achieved. Page 14 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Overall Efficacy Revision 1 Rating Substantial 5. Efficiency Economic and Financial Efficiency Analysis (EFA) ex ante  Applying a typical cost-benefit analysis to the project was not possible because the water supply/sewerage tariffs were tightly controlled by the central government, and there were no meters or sewerage tariffs, levied and collected independently from water supply services (PAD, Annex 9).  A cost-effectiveness assessment was considered in qualitative terms, identifying the expected benefits, including improved health of residents due to reduced pollution. The analysis concluded that interventions in sewer rehabilitation were urgent, and the project should prioritize network rehabilitation due to ageing of critical infrastructure and associated declining levels of service performance.  At the AF in 2015, a cost-benefit analysis was conducted with the quantification of costs and benefits. With the estimation of the costs of investments and incremental benefits, the cost-benefit analysis estimated the economic internal rate of return (EIRR) at 13% in Bukhara and 14% in Samarkand.  A financial analysis was conducted to investigate cost recovery levels in SVK and BVK. The analysis looked at the connection/interaction between annual tariff increases, annual net income, cash flow deficits, and the financial working ratio. However, the PAD did not include and explicit estimate for the financial analysis. ex post  The economic value of investments were reassessed at closing. This included updating estimated benefits as well as actual project costs incurred, to evaluate implementation efficiency. The analysis was conducted as an incremental analysis to the one done in 2015, where changes during implementation were compared to expected outcomes.  For Bukhara, the revised cost-benefit analysis showed a lower, EIRR at 12%, while for Samarkand, the revised analysis showed an increase in the EIRR to 16%.  Financial Analysis. Both Suvtaminots were independent regional water and sanitation companies, with all consumers within the same region charged a standardized tariff. As a result of sector reforms to aid sustainability, water and sanitation tariffs were raised by more than 100% in 2020, effective from January 2021. According to the ICR (paragraph 42) tariff increases resulted in significant increases in revenues, and both Suvtaminots’ financial statements showed improved cost recovery from fee collections.  Implementation Efficiency. The project was implemented over a 12 years period compared to an estimation of 6 years at appraisal. In total, the project was extended by 71 months. While cost savings from competitive procurement in the AF were a major source of efficiency, project delays and poor- quality feasibility studies in the parent project, which underestimated project costs, had a negative impact on implementation efficiency (ICR, paragraph 41). Implementation delays were exacerbated by the impacts of the COVID-19 pandemic, including restrictions on imports of goods and equipment, as well as human resource capacity issues associated with the pandemic (ICR, paragraph 18). Implementation Page 15 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) efficiency in Bukhara was negatively affected by spending on phase II without associated benefits, although positively affected by a higher level of new household connections than estimated, as well as due to cost savings of around 15%. Summary of Efficiency Assessment. The project experienced significant implementation delays. The under estimation of costs at appraisal necessitated additional financing to complete the project activities. The AF was significantly larger than the parent project (US$105 million versus US$55 million for the parent project). While the parent project experienced higher actual costs, the AF benefited from cost savings that amounted to 15% of the allocated funds (ICR Annex 4, paragraph 19). Despite the extended implementation period, the project did not fully complete the secondary treatment of the WWTP in Bukhara and missed the target for pollution removal. The ICR (Annex 4, paragraph 6) noted that works were ongoing to complete these works using Government funds. The EIRRs at completion were slightly lower for Bukhara (12% compared to 13% at appraisal) and slightly higher for Samarkand (16% compared to 14% at appraisal). Finally, the ICR (Annex 4) noted that the rehabilitation of existing sewers resulted in significantly fewer blockages as well as the energy efficiency at the WWTP achieved beyond the planned level. Overall, Efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original Objectives. Relevance of Objectives was rated High. Overall efficacy was rated Modest due to insufficiency of indicators to measure the outcome of the first objective. The project significantly improved energy efficiency of wastewater management. While the original targets for operational sustainability were exceeded, there were insufficient indicator as outcome level to fully measure operational sustainability, and the original targets for financial sustainability were only partially achieved. Efficiency was rated Modest. The project experienced significant implementation delays and costs were much higher than anticipated at appraisal. Based on a High rating for Relevance of Objectives and Modest rating for each of Overall Efficacy and Efficiency, Outcome is rated Moderately Unsatisfactory. Revised Objectives. Relevance of Objectives was rated High. Overall Efficacy is rated Substantial. The project succeded in reducing wastewater pollution in Bukhara and Samarkand as evidenced by exceeding targets on Page 16 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) BOD5 and significantly reducing sewerage blockages. The project also partially achieved its targets on financial sustainability, and with reduced sewerage blockages, it is plausible to assume that operational sustainability improved, however, more indicators were needed to comprehensively measure this aspect. Efficiency was rated Modest. The project experienced significant implementation delays and costs were much higher than anticipated at appraisal. Based on a High rating for Relevance of Objectives, Substantial rating for Overall Efficacy and Modest rating for Efficiency, Outcome is rated Moderately Satisfactory. Split Rating Before Restructuring After Restructuring Relevance of Objectives High High Overall Efficacy Modest Substantial (weak) PDO1 Modest PDO2 High PDO3 Substantial Revised PDO1 Substantial Revised PDO2 Modest Efficiency Modest Modest Outcome Rating Moderately Unsatisfactory Moderately Satisfactory Numerical Value of Outcome 3 4 Ratings Disbursements (US$) US$37.99 million US$94.81 million Share of disbursements (%) 28.60% 71.39% Weighted Value of Outcome 0.86 2.86 Rating Moderately Moderately Satisfactory Satisfactory Final Outcome Rating (0.86 + 2.86 = 3.72 (0.86 + 2.86 = 3.72 rounds up to 4) rounds up to 4) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR discussed two main risks that could potentially impact the development outcome: Page 17 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) 1. Technical risk related to the operationalization of wastewater infrastructure. While infrastructure works were successfully tested and commissioned, the incomplete status of the second phase of the Bukhara WWTP poses a risk to development outcome. The ICR (paragraph 80) noted that the GoU secured an estimated US$3.5 million for the completion of the remaining works, which were estimated to be completed by September 2022. The Government also retained the Project implementation Consultant (PIC) from the project to continue working on the site to enable successful completion. However, inadequate reinvestment in WSS services more generally, particularly since this has been a persistent challenge in the sector, remains a long-term risk. That said, recent tariff increases suggest a positive change and the ability of utilities to plan for preventative maintenance works (ICR, paragraph 80). 2. Institutional risk at the utility level related to operational and financial performance. There are still gaps in capacity and utilities still struggle to attract and retain skilled employees (ICR, paragraph 81). These risks were expected to be partially mitigated through support from the on-going Bank funded Water Services and Institutional Support (WASIS) project, which will moderate risks to the institutional development under BSSP. The following risk is emphasized by IEG: 3. Environmental Risk. It is important to ensure that the operation and maintenance of the rehabilitated water treatment plants continue to be supported. Lack of funding and poor maintenance could result in shortcomings in treating wastewater which will undermine the environmental achievements under the project. 8. Assessment of Bank Performance a. Quality-at-Entry  The project was developed to respond to the Government of Uzbekistan's (GoU) long-standing request to finance priority sanitation investments in Bukhara and Samarkand cities (ICR, paragraph 48). The project objectives were in line with the GOU’s priorities for the Water Supply and Sanitation sector (see section 3 for details). Also, the involvement of the Bank brought in needed external funding and technical know-how for sustainable operations and good maintenance (PAD, paragraph 4).  The project was developed as a follow-up operation to the Bukhara and Samarkand Water Supply Project (BSWSP) (P049621), which only focused on water supply. It mainly aimed to tackle the large-scale environmental pollution from limited wastewater treatment (ICR, paragraph 5).  The project design featured infrastructure rehabilitation combined with institutional reforms. To enable better financial oversight, design had a centralized financial management through the PCU. However, financial management remained challenging throughout implementation.  Implementation readiness was poor. The ICR (paragraph 51) acknowledged that the project was not ready for implementation at Board approval. Feasibility studies significantly underestimated capital expenditure costs, and detailed designs and bidding documents were not prepared. This had a significant negative effect on the project in that it needed additional financing that nearly tripled the original allocation and delayed the project with 6 years. Page 18 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719)  Technical documentation suffered from poor quality which was exacerbated by limited capacity of implementing agencies and the need to hire a Project Implementation Consultant (ICR, paragraph 51). All of which resulted in significant implementation delays  Seven risks were identified at appraisal related to economic, financial, institutional, and fiduciary risks. The overall risk rating was moderate. Despite that fiduciary risks including financial management and procurement risks (both rated Substantial) were well-identified from the predecessor project, they still posed challenges during implementation (ICR, paragraph 49).  M&E design suffered from a notable shortcoming. The Results Framework (RF) lacked indicators to realistically measure/assess the environmental impact of the original PDO (see section 9a for more details). Summary of Quality at Entry (QAE) Assessment. While the project capitalized on the predecessor project experience, implementation readiness was poor and mitigation measures were not enough to counter the substantial fiduciary risks. Also, poor feasibility studies resulted in a gross under estimation of costs, which necessitated additional financing to cover the financing gap. Finally, M&E design suffered from a notable shortcoming as the original PDO was pitched at a higher level of impact beyond the scope of the project, this was later addressed through a revision of the PDO. Based on the afore-mentioned information, QAE suffered significant shortcomings and is therefore rated Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team conducted 22 implementation support missions over the12 year life of the project. The project benefited from having task team leaders based in the country and able to provide daily hands-on support. Project restructurings were utilized to address design weaknesses and financing gaps. However, despite these efforts and the lengthy implementation duration, works were not completed at the Bukhara WWTP. Despite some shortcomings, the Bank team guided the project with a good amount of flexibility through challenging implementation conditions and managed to achieve most of the expected outcomes. Therefore, Quality of Supervision is rated Moderately Satisfactory. Summary of Bank Performance Assessment. QAE suffered significant shortcomings as noted above and supervision successfully guided the project under challenging implementation conditions. Based on the assigned ratings for both criteria and with the outcome rating in the satisfactory range, Bank Performance is rated Moderately Satisfactory. Quality of Supervision Rating Moderately Satisfactory Page 19 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design  The PAD did not include a Theory of Change (ToC) or results chain. Nevertheless, the ICR included an ex-post ToC that was constructed based on the PDO, the project activities and the results indicators as reported in the PAD. Overall, the ToC in the ICR reflected the relation between the project inputs, outputs, outcomes and long-term outcomes. However, the original PDO was pitched at a high level that was not captured by the outcome indicators in the Results Framework.  The original PDO was to be assessed through five PDO outcome indicators: 1. Pollution load on downstream communities measured by sewerage removed at WWTP; 2. Collection of sewerage into the WWTP; 3. Energy efficiency at the WWTP and wastewater pumping station; 4. Financial working ratio; and 5. Number of sewer blockages per year. While all five indicators were measurable, with reasonable targets and included baseline data, they were not sufficient to comprehensively capture the broad environmental impacts under the original PDO.  The RF included seven intermediate outcome indicators. These were linked to the project activities, measurable, and with reasonable targets.  Overall, M&E design lacked indicators that could realistically measure the environmental impact of the project. Otherwise, the RF was well designed and included relevant indicators, baseline values, and reasonable targets. The RF also included a useful description of how the indicators should be used during implementation. However, some indicators focused largely on outputs and not outcomes, for example, by recording number of staff trained without linking this to the outcome of better utilities performance (ICR, paragraph 65). b. M&E Implementation  M&E implementation suffered from several shortcomings during implementation. First, the management information system (MIS) described in the PAD was not established; Second, establishment of the SCADA systems at the Farkhad and Samarkand WWTPs was delayed (ICR, paragraph 66); Third, implementation suffered from low capacity in utilities to implement M&E activities as their focus was on addressing major procurement challenges (ICR, paragraph 67); Fourth, M&E specialists positions remained vacant despite that they were planned as part of the core implementation team; and Fifth, customer satisfaction surveys were only implemented as a pilot despite being part of M&E design to collect meaningful information about project interventions along with data on general levels of satisfaction with services (ICR, paragraph 67).  M&E functions were eventually assigned to PICs, which were increasingly able to provide data for the RF, collected through site supervision and their growing embeddedness within utilities’ operations.  The PDO was revised as part of the 2015 restructuring to be more measurable and attributable to the project activities. Also, the RF was streamlined to better capture the progress toward the project’s objectives. The indicator financial working ratio was corrected to operating cost coverage Page 20 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) (ICR, paragraph 66). The 2015 restructuring also clearly assigned M&E roles with day-to-day M&E to utilities, in line with efforts to increase their management authority in the project. c. M&E Utilization  According to the ICR (paragraph 68), "M&E utilization did partly allow to inform project management and course correction." M&E data enabled the Bank and the project management to take necessary corrective actions through project restructuring to ensure achieving the project objectives.  However, M&E utilization could have benefited from a thorough review during implementation to address any weaknesses, in terms of results and capacity to carry out M&E. Summary of M&E Quality Assessment. M&E design lacked relevant indicators to assess the original PDO. This was corrected through restructuring and revision of the PDO during implementation. M&E implementation suffered from several shortcomings as noted above, and most importantly the customer satisfaction surveys were not implemented as envisioned. Improvements made during project implementation enabled the M&E system to generate sufficient information to assess the achievement of the objectives. Finally, utilization was limited and could have benefitted from conducting the envisioned customer surveys. Overall, M&E Quality is rated Modest. M&E Quality Rating Modest 10. Other Issues a. Safeguards The Environmental risk rating of the project was Category B (Partial assessment). Four safeguard policies were triggered under the project: Environmental Assessment (OP/BP 4.01), Involuntary Resettlement (OP/BP 4.12), Physical Cultural Resources (OP 4.11) and Projects on International Waterways (OP/BP 7.50). An exception to OP/BP 7.50 was granted due to the project’s focus on rehabilitation of existing infrastructure. The project was expected to significantly improve the environment due to expanded coverage and more efficient sewage services in both cities. Also, it was expected to help lower health risks due to the poor sanitation and pollution generated by constant sewer leaks. Possible negative environmental impacts would be minor site-specific, short-term and easily identified/addressed by mitigation measures and good construction practices. Compliance with Environmental Safeguards. According to the ICR (paragraph 71) "environmental and social safeguards were a persistent challenge for the project." Safeguard compliance was negatively impacted by the absence of central safeguards oversight. Utilities had limited environmental and social management capacity and lacked qualified specialists within their institutions. The PCU did not have the position of environmental safeguards specialist to ensure proper oversight of sub-projects and report on compliance. The quality of safeguard documentation, coordination between different parties, and the speed Page 21 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) and effectiveness with which non-compliance issues were addressed were all negatively impacted by the lack of qualified staff with the implementing partners and contractors (ICR, paragraph 71). Compliance with Social Safeguards. According to the ICR (paragraph 72) "there were no large-scale resettlement activities under the project and most of the works were carried out within existing facilities or along roadways." Four Resettlement Action Plans were prepared, with adequate compensation paid to project-affected parties by project closing. However, during implementation, inconsistencies in social safeguard policies were a challenge. For example, there were instances of project-affected persons not being compensated for project impacts on time due to lack of capacity to prepare and implement the Resettlement Action Plans. The ICR did not provided an explicit statement on compliance with safeguard policies, but stated that "the project closed with a Moderately Satisfactory environmental and social safeguards rating (paragraph 72)." b. Fiduciary Compliance Financial Management (FM). FM suffered from systemic issues across much of the country's portfolio. There were challenges associated with inadequate capacity of public sector accounting; poor financial reporting and audit arrangements; weak internal controls, including an underdeveloped internal audit system; and weak legislative oversight (ICR, paragraph 74). The project experienced late submission of interim unaudited financial reports, late submission of audited project financial statements, and delayed documentation of expenditures incurred. The ICR (paragraph 74) noted the following key reasons the led to noncompliance with financial management/disbursement requirements: "(a) the PCU was understaffed and faced high staff turnover during most of the project implementation period, leading to a heavy workload on those still within the unit and (b) due to lack of experienced specialists in the market to implement international financial institution-funded projects, it took time to build PCU staff capacity." FM was rated Moderately Satisfactory at closing, with no pending actions (ICR, paragraph 74). Procurement. While procurement was implemented in accordance with the Bank’s Procurement and Consultant Guidelines and in accordance with the provisions of the Financing Agreement, it was often the central cause of major delays. The project lacked a proper procurement strategy (ICR, paragraph 75). The ICR (paragraph 75) reported that interviews (conducted as part of the ICR preparation) revealed that procurement capacity was identified as a key constraint, including quality, integrity, and efficiency issues. Procurement challenges included the absence of a single institution for procurement-related activities and weak capacity for drafting, reviewing, and managing contracts. Other challenges included poor quality packages and many cases of retendering. The Bank team provided training to address specific areas of need to improve procurement capacity. Procurement improved after additional implementing responsibility was given to utilities, supported by the project implementation consultants (PICs). Procurement was rated Moderately Satisfactory at project closing. c. Unintended impacts (Positive or Negative) --- Page 22 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) d. Other "The project was expected to benefit poorer households through access to sewers, and the overall environmental benefits of the project would have positive effects for households living downstream of the two cities. In addition, having operational sanitation systems was a necessity for the growth and development of the tourism industry in Bukhara and Samarkand, and which was not the case in 2009. A key economic driver for both cities, the development of the tourism sector will contribute to shared prosperity through jobs and increasing income (ICR, paragraph 46)." 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory M&E design and Implementation Quality of M&E Substantial Modest both suffered from significant shortcomings. Quality of ICR --- Substantial 12. Lessons The ICR included four lessons. The following three are emphasized with some adaptation of language: 1. Successful public sector reforms require a gradual approach with sectoral reforms and enough time for changes to materialize. Public sector reforms require time, trust, and flexibility, and realism in timing and the achievement of results. In a country context where the WSS sector is still maturing, additional time should be built into project implementation, recognizing that these changes are likely to have a profound effect on implementation. This also has implications for the lending instruments that best support public sector reforms. For example, a series of interventions with a number of credits/loans linked to the Government’s medium-term program toward incremental changes has proved successful in achieving a more ambitious intervention. Also, support through selective entry points can show success and can form the basis for later progress, even in difficult areas such as public service reform. Sector reform requires strong country ownership and to this end the Bank needs to tailor its support to the pace of reform. 2. Implementation readiness is a critical element of projects that if not appropriately addressed may lead to unnecessary implementation delays. Projects need to be ready for implementation to reduce delays that carry over the entire project period and hinder holistic project implementation. In BSSP, delays that originated due to lack of readiness resulted in increased pressure over time to deliver on major, but late, construction works, which had a negative impact on softer project interventions and can be argued to have lowered project ambitions in these areas. Initial implementation delays can be reduced by completing more works during project preparation Page 23 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) such as (a) preparation of request for proposal for consulting services, (b) approval of feasibility studies by the Government, (c) preparation of tender documents for the first contract packages, and (d) advance actions for the procurement of consulting services or construction contracts. 3. If project design does not reflect clear implementation roles for implementing entities, it may impede decision-making in complex projects. The involvement of too many entities in project activities, particularly in procurement decision-making, can cause major delays and reduce the authority of implementing entities to make required decisions. The BSSP's experience demonstrated that the lack of project management autonomy was a key contributor to the poor performance of the project in its early years. Also, the interference from national entities in the day- to-day project activities can be an additional impediment for progress if not planned for in design. 13. Assessment Recommended? No 14. Comments on Quality of ICR Quality of Evidence. While M&E implementation faced challenges, enough data was collected to track the progress of activities and assess the achievement of the PDOs. However, customer satisfaction surveys were only implemented as a pilot. These would have collected meaningful information about project interventions along with data on general levels of satisfaction with services. Quality of Analysis. The ICR provided clear linking between evidence and findings and used the evidence base to serve the arguments under the different sections, in particular the discussion on outcomes. Lessons. Lessons reflected the project experience and were based on evidence and analysis. Results Orientation. The ICR included a comprehensive discussion on the achievement of the PDO. The discussion was adequately balanced between reporting on the achievement of outcome indicators and what the project actually achieved on the ground. Consistency with guidelines. The ICR used the available data to justify most of the assigned ratings. Discussion of outcomes was adequate. The efficiency analysis could have benefited from including a comparison of costs of similar works under different projects. Conciseness. The ICR was well written and provided comprehensive coverage of the implementation experience and candidly reported on shortcomings. However, reporting on safeguards did not include an explicit statement on compliance and one safeguard stated in the PAD was not reported in the ICR (Physical Cultural Resources (OP 4.11). Overall, the Quality of the ICR is rated Substantial, but with minor shortcomings. Page 24 of 25 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BUKHARA & SAMARKAND SEWERAGE PROJECT (P112719) a. Quality of ICR Rating Substantial Page 25 of 25