Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) Report Number : ICRR0020959 1. Project Data Project ID Project Name P006553 BR APL SP Integrated Wtr Mgmt Country Practice Area(Lead) Additional Financing Brazil Water P125829 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-76610,IBRD-76620,IBRD- 30-Sep-2015 226,900,000.00 81490 Bank Approval Date Closing Date (Actual) 09-Jul-2009 30-Mar-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 104,000,000.00 0.00 Revised Commitment 93,532,949.54 0.00 Actual 93,532,949.54 0.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Salim J. Habayeb Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The Integrated Water Management in Metropolitan Sao Paolo Horizontal Adaptable Program Lending (APL), also known as Manancias Program, included three projects: The Government of the State of Sao Paolo (GESP) project, the State Water and Sanitation Autonomous Utility (SABESP) project, and the Municipal Government of Sao Bernardo do Campo (PMSBC) project. According to the Loan Agreement of the Government of the São Paulo (GESP) project of September 27, 2010 the project had three objectives, which were consistent with the three objectives of the program: The objectives were “i) to protect and maintain the quality and reliability of the Metropolitan Region of Sao Paolo’s (MRSP) water resources and potable water sources; ii) to improve the quality of life of the poor Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) populations residing in key targeted urban river basins in MRSP; and iii) to improve the institutional capacity and the metropolitan management and coordination in MRSP in water resources management, water pollution control, land-use policy and basic service provision.” According to the Loan Agreement of the State Water and Sanitation Autonomous Utility (SABESP) project of October 28, 2009 and the Loan Agreement of the Municipality of São Bernardo do Campo (PMSBC) project of March 29, 2012 the objectives of the program and the projects were: “i) to protect and maintain the quality and reliability of the Metropolitan Region of Sao Paolo’s (MRSP) water resources and potable water sources; and ii) to improve the quality of life of the poor populations residing in key targeted urban river basins in MRSP. These two objectives were consistent with objectives 1 and 2 of the program. The ICR provided a combined evaluation of the program as a whole, with separate evaluations of each sub- project presented as appendices. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The program included four components: 1: Institutional capacity building (appraisal: US$34.24 million total cost, out of which US$13.85 million IBRD financing; actual cost US$8.78 million, actual IBRD financing US$2.14 million): This component was to finance the strengthening of capacity of GESP and other executing agencies and promoting improved metropolitan management and coordination with regard to key metropolitan challenges of water resources management, water pollution control, land-use policy, and basic service provision. Activities included supporting the planning and implementation of sub-basin Environmental Development and Protection Plans and their respective land-use laws, creating a forum for seminars on Metropolitan Governance and Water in MRSP, carrying out different studies, and developing and monitoring environmental and water quality models. The cost variation between appraisal estimates and actual cost was due to several key activities not being implemented. These activities included: conducting the technical and economic feasibility study for the São Paulo Macro-Metropolis Water Production System, implementing the Geographic Information System. 2: Urban upgrading (appraisal:US$62.9 million total cost, out of which US$13.8 million IBRD financing; component not implemented): This component was to finance improving the standards and layouts of urban occupation in the targeted sub-basins and improving the quality of life of the residents of these sub-basins, especially in low-income communities. Activities included the urbanization of slums and irregular settlements, recuperation of high-risk and degraded areas, and the construction of housing units for involuntary resettlement, preparation of housing plans. This component was to be implemented by the Municipal Government of Sao Bernado do Campo (PMSBC) project which closed without implementing any activity in September 30, 2015 due to long delays in designing the slum upgrading interventions, increase in costs of interventions and lack of funds to finance the resettlement management plan. 3: Environmental protection and recovery (appraisal:US$19.61 million total cost, out of which Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) US$11.73 million IBRD financing; actual cost US$10.30 million, actual IBRD financing US$5.44 million): This component was to finance revegetating and reforesting, urbanizing public areas and creating green and leisure spaces for common use, establishing environmentally protected areas, rehabilitating and protecting reservoirs and water production systems, and controlling of hazardous substances in the region. After the restructuring several activities such as the second phase of the Liner Park were not implemented. Also, the cleaning and revegetation of the Taiacupeba reservoir was delayed and therefore only partially implemented. 4: Integrated water supply and sanitation (appraisal: US108.54 million total cost, out of which US$82.49 million IBRD financing; actual cost US$122.96 million, actual IBRD financing US$86.12 million): This component was to finance improving wastewater management, water supply systems, and solid waste management. Several activities such as implementing the sanitary land fill for Embu Guacu, and conducting the sewage system works for the Marsilac District were not implemented. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost: The combined total program cost was estimated at US$227.17 million. However, due to the closing of the PMSBC project, the program cost was reduced by US$41.4 million. During the project restructuring in 2015, the housing activities under the GESP project were cancelled, reducing the cost by US$30.6 million. Furthermore, after restructuring, the GESP project did not implement several activities, resulting in another cost reduction of US$13.48 million. The actual project costs were US 141.69 million. Financing: The International Bank for Reconstruction and Development (IBRD) financed the following loans: i) A US$4 million loan for the GESP project; ii) a US$100 million loan for the SABESP project; and US$20.82 million loan for the PMSBC project. Actual disbursement was US$93.54 million. Borrower Contribution: The borrower was to contribute US$102.35 million, and the actual contribution was US$48.86 million. Dates: The program was restructured on September 30, 2015 to: i) extend the closing date for the GESP and SABESP loan from September 30, 2015 to March 30, 2107 to allow the completion of activities, in particular those related to the response to the water crisis; ii) prioritize activities to help solve the ongoing drought crisis affecting the MRSP; iii) review the institutional arrangements and indicators; iv) increase the percentage of loan financing for the GESP project; and v) adapt the Results Framework to reflect these changes. 3. Relevance of Objectives & Design a. Relevance of Objectives Substantial. The program’s objectives were relevant given the Metropolitan Region of Sao Paolo’s (MRSP) strategic importance and the critical issue of water supply for the city’s economic growth and social and environmental sustainability. In 2007, MRSP accounted for 17 percent of the national Gross Domestic Product (GDP) and 10 percent of the population. However, Sao Paolo’s extremely low per capita water availability was comparable to the country’s driest areas. Also, over 1.9 million people, the majority being Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) poor, had been living in slum settlements in the Guarapiranga and Billings river basins, close to the city center, causing pollution to the reservoirs through wastewater discharge. Also, there was a need to strengthen the state’s Water Resources Management (WRM) and to strengthen the sector’s capacity. At the time of appraisal, the objectives of the program supported the government’s pluri-annual development plan (2004-2007), which outlined the challenges facing metropolitan regions and large cities in Brazil, as they deal with constraints to growth, social inclusion, environmental degradation, and the appropriate planning and management of services. The objectives of the program were also in line with Brazil’s most recent pluri- annual development plan (2016-2019), which focuses on more community participation, land management and urban transformation. The objectives of the program were consistent with the Bank’s Country Partnership Strategy (FY2008-2011), which identified water scarcity and environmental degradation as urgent obstacles for achieving sustainable economic growth. The objectives were in line with two of the four objectives of the Bank Country Partnership Strategy (FY2012-2015): promoting regional economic development through strategic investments and policies, and improving sustainable natural resources management and climate resilience. Also, the most recent Country Partnership Strategy (FY2018-2023) aims to increase urban resilience and provide more sustainable and inclusive urban services. Rating Substantial b. Relevance of Design Modest. The operation was built on several studies that identified slums, which are lacking sanitation infrastructure, as pollution source to water bodies. Choosing an APL as a lending instrument was adequate since it allowed for the implementation of several projects addressing interrelated challenges such as urban pollution, poverty and land issues in the MRSP area. Activities to protect and maintain the quality and reliability of the MRSP water resources and potable water resources included conducting studies in different areas such as evaluation of pollution loads in specific reservoirs, reviewing and updating Environmental Protection Plans (PDPAs) of sub-basins, and purchasing urban cleaning equipment for municipalities in the MRSP. Activities to improve the quality of life of the poor populations residing in key targeted urban river basins in MRSP included urbanizing slums and irregular settlements, recuperating high-risk and degraded areas and preparing housing plans. Activities to strengthen institutional capacity and improve metropolitan management and coordination in MRSP in water resources management, water pollution control, land-use policy and basic service provision included building capacity within the Government of the State of Sao Paolo (GESP) and other executing agencies through conducting different studies and strengthening capacity in project management, monitoring and evaluation and disseminating results. However, the design had several shortcomings. First, the APL’s scope was too complex. Second, there was a disconnect between the APL and the PDO: Even though the Municipal Government of Sao Paolo, which was originally included in the APL, was removed from the APL since its participation was in disagreement with Bank rules, the PDO or the Results Framework were never updated to reflect this removal. Third, there was an inconsistency between the APL’s and projects’ PDO: the implementation of each project (even if all of the planned activities were implemented) could not lead to the achievement of the PDO attributed to the operation. Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) Fourth, the program’s activities were not well specified resulting in a lack of understanding on how they would contribute to the achievement of the PDO. Fifth, the project design included a large amount of activities which were to be completely financed by other funds than the loan and not to be implemented by the main implementing agencies, but by other agencies. Finally, the underlying assumption of how certain activities were to contribute to the achievement of the PDO was not clear or not properly laid out. Furthermore, the project did not take negative externalities into account, such as the government's fiscal constraints. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Protect and maintain the quality and reliability of MRSP’s water resources and potable water sources. Rationale Outputs under the GESP project: • A study on the nonpoint source water pollution for the Alto Tiete water production system was conducted, and provided information about the land use impact on the quality of the water bodies in the headwaters of the Alto Tiete river basin. • A study to monitor the water quality of the Guarapiranga and Billing reservoirs was conducted to evaluate the pollution load in these two reservoirs. • A concept study on the Marsilac district sewerage system with engineering designs was developed. • Several Environmental Development and Protection Plans (PDPAs) were reviewed and updated. The PDPA for Guarapiranga, Billing, Alto Juquerey, Alto Tiete Cabeceiras, Cabucu, Tanque Grande, and Cotia were prepared. The first PDA version for Guaio, Alto Juquia and Jaguari were developed. The target of this indicator, 10 PDPA prepared/revised for each sub-basin, was achieved. • Technical specifications for an integrated land use supervision and control system were developed. • A hydrology study of rivers (under the influence of the Billings reservoir) in the Baixada Santista was conducted. • Urban cleaning equipment was purchased and provided to 11 municipalities in the MRSP aiming to improve solid waste collection and protecting water bodies. • The design and construction of 9 July Park were implemented. This 54 hectares park aimed to protect the reservoir while also providing leisure to approximately 20,000 people per year. • The Camino Atlantica Park was designed and constructed. Outputs under the SABESP project: Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) • The Braganca Paulista Waste Water Treatment Plant (WWTP) was constructed in order to fulfill a compensatory measure required by the Praciacaba, Capivari and Jundiai river basin committee to continue allowing the water withdrawal transportation from this basin to the Alto Tiete basin to complement the MRSP water supply needs. • The Grajau water supply pumping station was constructed to improve water supply distribution benefiting 130,000 people. • Water leakages were reduced by replacing water pipelines. The ICR does not specify how many pipelines were replaced. • Works on the Rio Pequeno to Rio Grande were conducted, benefiting 1.2 million people. • A study was developed to improve institutional capacity for water resource planning, not achieving the target of three studies. Outcomes: • The volume of Biochemical Oxygen Demand (BOD) pollution loads removed by treatment plants and sewerage systems supported under the project was 2,574 tons per year, but short of the target of 2,949 tons per year. • Improvements were made and equipment for the Boa Vista water treatment plant filtering membranes system was purchased increasing the water production from 15m3 per second to 16m3 per second, benefiting 300,000 people. • Improvements on the Rio Grande transfer to the Taiacupeba dam were made, and they increased the water production by 4m3 per second. • Water production increased from zero to 157,680 m3/year, achieving the target. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Improve the quality of life of the poor populations residing in key targeted urban river basins in MRSP. Rationale Outputs under the GESP project: • 100 hectares of degraded areas were recovered, not achieving the original target of 213 hectares. • An integrated citizenship center was implemented, achieving the target. The citizenship center provides services to approximately 350,000 low-income people living in the Guarapiranga river basin right bank. Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) Outputs under the SABESP project: • A sanitation system including connection pipeline and interceptors connecting to the main sewerage network exporting the wastewater to a treatment plant located outside the Billings river basin, where no effluent can be formally discharged, were constructed. These investments benefitted a low-income community that has gradually settled in the Billings river basin. • A study and designs on the transfer of sludge from Guarau Water Treatment Plant to Barueri WWTP was developed. • A study to interconnect the Rio Grande da Serra and Ribeirao Pires sewerage system was carried out. Outcomes (from SABESP project): • An estimated 248 tons BOD per year were abated, benefiting approximately 16,000 people in low- income communities due to works that were conducted on Mombaca and Crispim sewerage’s secondary and main pipelines, pumping stations. Also, interceptors connecting to existing WWTP were implemented. The works are ongoing and are expected to be completed in December 2017. • Works on the Cocaia-Lagoinha sewerage system were conducted, including the construction of primary and secondary networks, pumping stations, and connecting main ducts to existing WWTP. These investments also abated an estimated 248 tons BOD per year and benefit approximately 25,000 people in low-income communities. • Works on the Branca Flor sewerage interceptor system were executed, abating 88 tons BOD per year and benefitting approximately 5,600 people in low-income communities. • Works on sewerage primary and secondary pipeline connecting to TO-13 basin were conducted in Carapicuiba and Cotia municipalities,resulting in the abatement of 44 tons BOD per year and benefitting approximately 2,800 people in low-income communities. • Works on Braganca Paulista sewerage treatment plant were conducted resulting in the abatement of 2,050 tons BOD per year and benefitting approximately 130,000 people in low-income communities. In spite of some favorable results shown above, there is no direct evidence that the quality of life of the poor populations has actually improved, and there was no assessment through opinion/satisfaction surveys of the beneficiaries. Also, major related activities were not implemented such as the slum-upgrading interventions, and population resettlement (under the PMSBC project which closed without any activities being implemented). Rating Modest PHREVDELTBL PHEFFICACYTBL Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) Objective 3 Objective Improve the institutional capacity and the metropolitan management and coordination in MRSP in water resources management, water pollution control, land-use policy and basic service provision. Rationale Outputs under the GESP project: • Procurement for solid waste collection equipment concluded and equipment in place, achieving the target. • One study was developed to improve institutional capacity for water resources planning, not achieving the target of three studies. • Two seminars to discuss water resources management and planning aspects of MRSP were conducted, achieving the target. Outputs under the SABESP project: • Hydrodynamic models for the Guarapiranga, Billings, Jacarei, and Jaguari reservoirs were developed. Outcomes: No outcomes were reported under this objective. Rating Modest PHREVDELTBL PHREVISEDTBL 5. Efficiency Economic Efficiency: The PAD (p. 63) included a cost-benefit analysis for the full program amounting to a total cost of US$272.3 million. The cost-benefit analysis for the urban upgrading and improvement of water supply and sanitation interventions estimated an overall Internal Economic Rate of Return (IERR) of 23 percent and at a 12 percent discount rate a Net Present Value (NPV) of BRL 137 million. In addition, the cost-benefit analysis for preserving the existing sources to supply MRSP with potable water estimated an IERR of 22 percent and a NPV of BRL 294 million. Therefore, the results of these analyses indicated that the program was expected to have a high positive return When the project was restructured in September 2015, another cost-benefit analysis was conducted. The analysis compared the expected benefits from the newly introduced water security investments in regards to the costs of alternative options to provide water at times of scarcity. These options included water tankers at R$38.87/m3 and tariffs charged to high consumption (contingency tariffs) at R$10.36/m3. The IERR was Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) estimated at 245 percent and the NPV at US$770 million. The ICR (p. 40) presented a cost-benefit analysis which included all program costs and only economic benefits that could be quantified. These economic benefits included the environmental benefits associated with the removal of BOD and COD pollution loads, the consumer surplus associated with improved water supply services, the willingness to pay for improved sanitation services, the benefits associated with improving water security, the health effect associated with burden of diseases avoided, and the savings in travel costs due to the existence of nearer urban parks. Other benefits such as increases in property values, environmental impacts, relaxation etc. were not included. The analysis estimated an IERR of 18.5 percent including health benefits and an IERR of 17.3 percent excluding health benefits. At a discount rate of 12 percent, the present value of net benefits was US$43.6 million and the benefit-cost ratio was 1.41. At a discount rate of six percent, the present value of net benefits was US$165.4 million and the benefit-cost ratio was 2.05, all suggesting a positive economic return on the program’s investments. Operational Efficiency: Major shortcomings were reported, including non-implementation and/or extensive implementation delays due to unspecific description of project activities, lack of professional expertise in the market to assist the project, dropping out of participants, cancellation of several activities, and closing of the PMSBC project without implementing any activity. These indicate an inefficient use of project resources. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated Substantial as the operation was responsive to the country needs and the objectives were consistent with government and Bank strategies. Relevance of design is rated Modest given significant shortcomings. Achievement of the objective to protect and maintain the quality and reliability of MRSP’s water resources and potable water sources is rated Substantial. Achievement of the objective to improve the quality of life of the poor populations residing in key targeted urban river basins in MRSP is rated Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) Modest. Achievement of the objective to improve the institutional capacity and the metropolitan management and coordination in MRSP in water resources management, water pollution control, land-use policy and basic service provision is rated Modest. Efficiency is rated Modest due to significant shortcomings in the efficiency of implementation. Taken together, these ratings are indicative of major shortcomings in the project’s preparation and implementation, and therefore an Outcome rating of Unsatisfactory. a. Outcome Rating Unsatisfactory 7. Rationale for Risk to Development Outcome Rating The Municipality of Sao Paulo’s Secretary for the Green and Environment has been adequately managing parks for over ten years, and would be expected to mitigate potential risks to the sustainability of the two newly constructed parks such as encroachment from regulated settlements. Also, SABESP has carried out adequate water and sewerage maintenance in 367 municipalities with which it has concession contracts and provided quality services to approximately 28 million consumers. The project’s investments were integrated into the water and sanitation infrastructure that SABESP manages. SABESP’s operational and maintenance procedures in water supply and sanitation have been classified as adequate in Brazil. However, there are several broad uncertainties. The government faces substantial fiscal constraints, and there are frequently changing priorities. There is volatility in political support, which was described by the ICR as difficult to achieve. Institutional support appears to be fragile, as would be consistent with the fact that “the strengthening of institutional capacity and the improvement of management and coordination in the Metropolitan Region of São Paulo in water resources management, water pollution control, land-use policy, and basic service provision” was deemed to be modest at project closing. Since it is unlikely that the above uncertainties would be resolved in the near-term, the overall the risk to development outcome is rated Substantial. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The preparation of the program took seven years. Several studies to support the program’s rationale were conducted. Stakeholders were consulted during project preparation and the project’s objectives were in line with the government’s pluri-annual development plan (2004-2007). The adoption of a horizontal APL operation as a lending instrument brought along several challenges. During project preparation, several Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) borrowers were interested in participating in the program, but, by the time of appraisal, this number decreased to four participants/projects. No adjustments were made to the type of lending instrument and the scope of the program, despite the significant smaller number of participants. Also, its implementation structure was overly complex. The Bank team identified relevant risks, but a major risk that was not taken into account was the risk that intended participants would drop out of the project. The operation did not take into account fiscal constraints, and this resulted in the government allocating less funds than planned towards project activities. Also, in order to allow for more implementation flexibility, the definition of activities was kept very vague, resulting in the borrower institutions choosing activities that were aligned with their own agendas. The PDO of the program and of each project were almost the same and each project was legally bound through the loan agreement to achieve the PDO. However, the achievement of the PDO was dependent on the contribution of all different projects, but each project was legally bound to achieve the PDO through the loan agreement. Furthermore, the program design allowed borrowers to join the program after the operation had started, resulting in more uncertainty with regards of the final size of participating projects. The Results Framework was inadequate (see section 10a). Given major shortcomings in the Bank’s performance at entry, the Quality at Entry is rated Unsatisfactory. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision The Bank team conducted semi-annual supervision missions, provided guidance throughout project implementation, responded to questions from the counterparts in a timely manner, identified bottlenecks, and developed solutions to overcome them. The Bank team provided technical support especially in regards to fiduciary and safeguards aspects. The Bank team stated that Implementation Status Reports (ISRs) ratings and key issues for management attention were usually candid, but also overly optimistic in terms of expected results. The ISR ratings fluctuated from Satisfactory to Moderately Unsatisfactory in spite of implementation deficiencies. The rating was downgraded to unsatisfactory at project closing. The Bank did not take any action concerning the inconsistencies between PDO, project activities, and the Results Framework. The ICR (p. 29) stated that the Bank was more focused on implementation progress than development impact, and supervision missions were mainly about contract implementation. The operation's restructuring was late, five years into implementation. Given significant shortcomings, the quality of supervision is rated Moderately Unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. Assessment of Borrower Performance Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) a. Government Performance Under this operation, the government and implementing agencies were essentially indistinguishable. Therefore, this ICR Review provides only an overall borrower performance rating. GESP project: The government of the State of São Paulo in Brazil was represented in this project through three government entities that were also the executing agencies: the State Secretariat for Water, Sanitation, and Water Resources (SSRH), the State Secretariat for the Environment (SMA) and the State of Sao Paulo Urban Development & Housing Company (CDHU). The SSRH was responsible for the overall project and the SMA and CDHU had specific roles to perform. Overall commitment to core activities such as water quality studies to inform policies was adequate. However, according to the Bank team, the commitment to achieving the project’s objective was less clear due to several reasons such as political, economic, and financial changes of large magnitude affecting the country at the national, regional and local levels. There were changes in government investments priorities. During project implementation, significant funding challenges were faced due to the recent fiscal crisis. Since the loan proceeds financed only 25 percent of project expenditures, these fiscal constraints had a negative impact on project implementation. Due to fiscal restrictions the development of consultancy work was not a priority anymore, contributing to the low level of loan disbursement. The program management unit (UGP) was responsible for the implementation of the project. The UGP only included a few staff and was supported by a consulting company with expertise in carrying out technical, financial, and procurement activities under Bank projects. UGP’s procurement performance was inconsistent. Also, the GESP team resisted to use standardized Bank procurement documents. The procurement plan was changed several times. Procurement processes were lengthy with a negative impact on project implementation. SABESP project: The SABESP showed inconsistent commitment to the project throughout is implementation. Even though the project was embedded in SABESP’s organizational structure and given the same priority as other projects under SABESP’s investment plans, the institution did not have its own implementation unit. Bank processes could not be streamlined with SABESP’s procurement rules. According to the Bank team, SABESP prioritized activities that were clearly aligned with its own priorities. The project was implemented through different SABESP’s departments with delays. The UGP produced progress reports, including procurement and financial management reports, on a regular basis, but with delay. The environmental and resettlement safeguard were addressed adequately, except for the hiring of the dam safety panel which was delayed. PMSBC project: The government of the Municipality of Sao Bernado do Campo appeared to be committed to project implementation, as it overcame lengthy processes to get the required clearances to implement the project during preparation and appraisal. The government made efforts to identify additional funding sources once the project’s cost increased. However, by the time the project closed, these funds were still not available. The UGP was keen to identify alternative solutions to the urban upgrading investments. But UGP failed to provide basic technical support to finalize the packages for starting the bidding processes, such as identifying intervention costs, developing implementation timetable, final design, and estimating the number of people to be resettled. There was inability to speed up the process of hiring a management firm for technical support. Even though the PMSBC Housing Secretary had extensive past experience with urban upgrading activities, it lacked familiarity with Bank’s processes. The National Bank’s (CEF) cumbersome approval processes also contributed to significant delays. Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance See above. Implementing Agency Performance Rating Moderately Unsatisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The overall objective of the APL was clearly specified while the PDO indicators did not reflect any of the individual projects, but rather the desired overall outcomes, which the program aimed to achieve through the implementation of all projects’ activities. There was a disconnect between project activities, PDO and the Results Framework. The original Results Framework included 15 PDO indicators and 33 intermediate outcome indicators of which 13 PDO indicators and 23 intermediate outcome indicators were dropped during the restructuring in September 2015, five years into project implementation, due to different reasons such as indicators capturing too many externalities. Most indicators were measurable in terms of numbers, timing, and location. Intermediate outcome indicators measured each project’s outputs, and the program planned to conduct studies and surveys to contribute to overall progress monitoring. b. M&E Implementation Each project modified and simplified the Results Framework to reflect the individual project’s activities. The projects’ Result Frameworks were kept up to date, but were not used as a tool to guide implementation. It turned out that the studies and surveys to monitor progress on PDO indicators were too complex, long, expensive, and not adequate monitoring tool. Studies were therefore dropped in the September 2015 restructuring, during which the Results Framework was simplified. Several PDO and intermediate outcome indicators were dropped or modified, and new ones were added. c. M&E Utilization The Results Framework was mainly used to register the achieved targets. M&E data were not used Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) effectively to support project management and decision making. The studies and surveys that would have contributed to monitoring progress towards the achievement of overall objectives were dropped, as stated above. In conclusion, it is unlikely that M&E arrangements will be sustainable. M&E Quality Rating Modest 11. Other Issues a. Safeguards The program was classified as category A and triggered safeguards OP/BP 4.01 (Environmental Assessment), OP/BP 4.04 (Natural Habitats), OP/BP 4.11 (Physical Cultural Resources), OP/BP 4.12 (Involuntary Resettlement), and OP/BP 4.37 (Safety of Dams). Environmental Assessment: The Bank team prepared a regional environmental assessment in accordance with the requirements of a category A project. During implementation, specific environmental assessments were prepared for each project intervention. In addition, as required by the Brazilian environmental licensing procedures, specific environmental studies were conducted. All results were disseminated, discussed and disclosed. Natural Habitats: The only activities that triggered this safeguard were the slum-upgrading investments under the PMSBC project. While the preparation for the slum-upgrading fully complied with the safeguard, the actual slum-upgrading intervention was never implemented. Physical Cultural Resources: Procedures to comply with this safeguard were included in the operations manual and the construction manual. Environmental assessments, which were prepared for each activity, also included an assessment following OP 4.11 directives. Involuntary Resettlement: In order to address the resettlement of 976 families during the slum-upgrading intervention, a Resettlement Framework (RF) was developed, disseminated, discussed with stakeholders and disclosed. The RF was integrated into the operations manual to guide the implementation of the PMSBC and SABESP projects. According to the ICR (p. 13) the projects complied with the RF. Also, a Resettlement Action Plan was prepared for the PMSBC project. However, since the PMSBC project was closed without having implemented any activities, none of the action plan activities were implemented. The safeguard was triggered due to the construction of the Braganca Paulista wastewater treatment plant and a road accessing the plant under the SABESP project. However, the construction did not have any negative impact on landowners in the area. Safety of Dams: During project preparation several project activities triggered the safety of dams safeguard. None of these activities ended up being implemented. A new activity was implemented under the SABESP project, triggering this safeguard. The activity involved the recovery of the Taiacupeba Dam Hydraulic Basin Capacity, removal of vegetation that had grown in the basin, and activities to enhance and protect the green area buffer zone around the reservoir. Mitigation activities included a review of panel of experts to assess the situation. Based on the panel’s recommendation, an action plan was developed. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) b. Fiduciary Compliance Financial Management SABESP project: SABESP’s experience and internal control procedures allowed for a satisfactory financial management throughout project implementation. Even though the project experienced some delays in adapting the SABESP’s corporate system to generate Interim Financial Reports at the beginning of project implementation, necessary adjustments were made and performance improved. The project’s disbursement rates were affected by the state water shortage crisis and the devaluation of the Brazilian real. All audit reports were submitted in a timely manner, had unqualified opinions and no ineligible expenditures were found. Throughout project implementation all financial management supervision missions were rated Satisfactory. GESP project: The PIU and management firm’s experience had a positive impact on the financial management performance of the project. At the beginning of implementation, the project experienced some delays in adapting the state’s administrative system to generate consolidated interim financial reports at the program and project level. Action plans were implemented, but the devaluation of the Brazilian real in 2015 had a negative impact on the disbursement rate. Generally, audit reports were submitted in a timely manner, except for the 2014 report. All audit reports had unqualified opinions. All interim financial reports were accepted by the Bank and were received in a timely manner. No ineligible expenditures were found. Throughout project implementation, all supervision missions were rated Satisfactory or Moderately Satisfactory. PMSBC project: Only a very small fraction of this loan was disbursed and 99.54% of the loan was cancelled. All interim financial reports were received in a timely manner and were accepted by the Bank. Due to the low disbursement rate, no formal audit was conducted and the Financial Management Specialist implemented alternative procedures to assure the eligibility of expenditures, and there were no instances of ineligible expenditures. All financial management supervision missions were rated Moderately Satisfactory. Procurement SABESP project: According to the ICR (p. 16) the SABESP’s procurement performance was adequate and its technical capacity in conducting biddings and selecting consultants was adequate. At the beginning of implementation, the project experienced some delays since SAPESP had to get familiar with the Bank’s procurement rules. The discussion about new standard documents were lengthy due to SABESP’s organizational structure which included several sectors such as legal, technical, control, and procurement. However, once the bidding documents were completed, the procedures were performed satisfactorily. The project restructuring, which responded in part to the reprioritization of SABESP’s interventions, impacted the pace of procurement since several ongoing processes were suspended, canceled and eventually taken up again. GESP project: GESP’s procurement performance was inconsistent throughout project implementation. The GESP had limited experience in working with the Bank’s procurement rules. The procurement plan was changed several times. The GESP team resisted using the Bank’s standardized procurement documents, and the quality of terms of references was inadequate, resulting in long delays due to time consuming technical discussions. The procurement selection process for studies to be conducted was also very lengthy, and, by the time of the Mid-Term Review in May 2013, only one contract financed through the loan had been signed. PMSBC project: Given PMSBC’s limited experience in working with multilateral agencies, the procurement Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) team had low capacity in dealing with international consulting services and faced several technical challenges. PMSBC concluded only one procurement process which included the hiring of a consultant, and which took nine months to complete. The PMSBC’s procurement function was rated unsatisfactory throughout project implementation. c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory --- This ICR Review takes into consideration broad Risk to Development Modest Substantial uncertainties in political and Outcome institutional support, and fiscal constraints. Bank Performance Unsatisfactory Unsatisfactory --- Moderately Moderately Borrower Performance --- Unsatisfactory Unsatisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR (pp. 31-32) presented several useful lessons adapted by IEG: 1. Providing clear definitions for activities to be implemented under the project facilitates successful implementation. Under this operation, activities were kept vague without clear description in order to allow for flexibility. However, the lack of clarity had a negative impact on implementation since there was not enough information to develop a credible implementation schedule and cost estimates. The Borrower pointed out that Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR APL SP Integrated Wtr Mgmt (P006553) this put an enormous burden on the client organizations. 2. Sharing the same objectives among several projects under a horizontal APL might make it difficult to measure progress of each project towards the PDO since the PDO cannot be achieved by just one project, but only by all projects together. Under the program, each project was legally bound to achieving the PDO even though its achievement was dependent on the implementation of activities falling under the responsibility of different institutional entities, beyond each project’s institutional boundaries. 3. When a program gets modified, the revision of the objectives and Results Framework would guide the project through its amended pathway, and facilitates a pertinent evaluation. Under this program, even though several projects dropped out, the PDO and Results Framework were never modified. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR provides a good overview of the program and participant projects experience. It is appropriately candid, consistent, and results-oriented. It provides useful lessons learned from the program’s experience. The report notes the gaps in evidence, as related studies and surveys were dropped. However, the ICR lacks conciseness in its narrative. Overall, the quality of the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 17 of 17