ICRR 12815 Report Number : ICRR12815 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/21/2007 PROJ ID : P006046 Appraisal Actual Project Name : Water Sector Reform US$M ): Project Costs (US$M): 46.80 33.34 Project Country : Argentina Loan/ US$M ): Loan /Credit (US$M): 30.00 24.53 Sector Board : WS US$M): Cofinancing (US$M ): Sector (s): Water supply (39%) Sewerage (39%) Central government administration (22%) Theme (s): Regulation and competition policy (20% - P) Pollution management and environmental health (20% - P) Environmental policies and institutions (20% - P) Other urban development (20% - P) Access to urban services and housing (20% - P) L/C Number : L4484 Board Approval Date : 06/01/1999 Partners involved : Closing Date : 07/31/2002 03/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Kristin S. Little Ronald S. Parker Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The project aimed to improve the quality and efficiency of water and sanitation services in urban and rural areas in Argentina by supporting private sector participation . The original objective was to support the reform process in about five medium-size municipal and provincial utilities while addressing the pending issues of existing private utilities, regulatory frameworks, and regulatory agencies . The project aimed to achieve four specific outcomes : (i) efficiently run utilities under private management; (ii) private utilities with a clear strategy to expand services to the poor; (iii) cost-effective regulatory practices; and (iv) reliable financial and operational information on utilities . The objectives were revised in 2004: (a) to support the reform of the water and sanitation sector in Argentina through the promotion of efficiency in the operation and financing of water utilities; (b) to support the establishment of sound regulatory frameworks for the water sector; and (c) to address universal service and environmental issues pertaining to said sector. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 06/01/2004 c. Components (or Key Conditions in the case of DPLs, as appropriate): A. Investment component (US$23.5 million at appraisal; US$21.7 million after restructuring; US$23.95 at closing). Investments to restore the capacity of existing systems : rehabilitation and replacement of pipes, revamping and improvement of water treatment plants and intake facilities, upgrading substandard connections, supply and installation of water meters, repair and replacement of pumping and control equipment, supply of emergency and maintenance equipment, improvement and extension of water and sewerage services to the poor . B. Institutional Component (US$5.2 million at appraisal; US$2.8 million after restructuring; US$.34 at closing) Project management, training of regulators, technical assistance to continue advancing the reform of the sector, establishment of a sectoral information system, and development and implementation of environmental regulations . C. Environmental Component (US$1.0 million [p 10, ICR, p 35, PAD] at appraisal, [though the ICR states on p 50, that the component was new after project restructuring, and that it was US$ 0.8 million]; US$0.2 million after restructuring; US$0.0 at closing) This component consisted of two parts : (i) updating of key environmental standards, preparation of guidelines for environmental impact assessment, environmental auditing of selected subprojects, and a prototype water quality management diagnostic study for the Salidulce river basin . (ii) emergency actions to mitigate negative environmental impacts related to water infrastructure . It included, inter alia: environmental clean-up services, removal of hazardous wastes, and equipment to address environmental damage . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project was originally estimated to cost $ 47.1 million, with $17.1 coming from the borrower and $30.0 coming from the Bank. In the end, just over half of this was invested by the Borrower (51.8 percent), and 81.7 percent from the Bank. Project revision came late. The government had moved away from PSP -led reform by 2000, yet a restructuring proposal was not submitted until 2002. The amendment to the loan was signed by the Government of Argentina in July 2005--three years after the original closing date, 6 years into the 8 year loan period. 3. Relevance of Objectives & Design: Both the initial and revised project objectives were relevant to the current CAS as they focused on improving the quality and efficiency of water and sanitation services, which are important to supporting economic growth, expanding access to the services for the poor, and strengthening regulatory capacity at the provincial level . The project’s emphasis on working closely with the provinces and local utilities to improve local regulatory capacity and financial self-sufficiency at the utility level also remains in line with the approach recommended in the current CAS . However, a major focus of the initial and revised objectives (although slightly less so in the revised objectives ) is PSP, which is at odds with the more neutral approach reflected in the current CAS . Furthermore, the performance monitoring and evaluation system did not fully reflect project objectives as it was not geared to effectively consult the poor and monitor impacts on the poor . The decision to have the PMU staffed by permanent national water and sanitation agency personnel is in line with current thinking at the Bank, in order to mainstream the improved technical capacity and to protect the institutional memory of a project rather than have it rest with independent consultants . However, the strategy proved susceptible to insufficient allocation of personnel to the PMU, due to the government ’s change in strategic direction since the preparation of the project. 4. Achievement of Objectives (Efficacy): Negligible. After revision, the project was intended to promote the efficient operation and financing of water utilities in Argentina, establish sound regulatory frameworks, and address service and environmental issues, none of which was satisfactorily achieved . Objective (a), to support the reform of the water and sanitation sector in Argentina through the promotion of efficiency in the operation and financing of water utilities, was not fully achieved as EIRRs reveal that for the bulk of the investment, the targets were not met . Achievement: modest. Objective (b), to support the establishment of sound regulatory frameworks for the water sector, was not achieved, as none of the provinces received technical assistance to strengthen their regulatory frameworks . Achievement: Negligible. Objective (c), to address universal service and environmental issues pertaining to the sector, was not achieved as none of the utilities received technical assistance relating to tariff policies or to investment alternatives to better reach the poor. Further, none of the funds in the environmental component (c) were disbursed. Achievement: Negligible. Three out of five utilities were ultimately supported through project -financed investments, only two of these had undergone a reform process, which had already occurred under the previous project (Ln. 3281-AR). As a result, none of the initial project development objectives can be considered to have been met . 5. Efficiency (not applicable to DPLs): Economic analyses in the PAD and ICR were in -depth and sophisticated. However, the questions posed by each differed. From an ex-post perspective, the question posed was no longer "to reform or not to reform." The relevant question had become: "Were these investments worthwhile? " This implies that the figures below for appraisal and ICR need to be considered on their own . The economic analysis at appraisal did not demonstrate (1) why it made sense to direct funds to PSP -reformed utilities only, as opposed to a larger selection of utilities fulfilling desired efficiency criteria; or (2) whether the investment of these funds would lead to economically and financially viable benefits. Additionally, the appraisal analysis did not reflect the high internal plumbing costs that households face if they choose to connect to new sewerage disposal services . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal Yes 12% 50% ICR estimate Yes 4% 54% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Assessing the outcome against both the original and revised objectives, rating each and weighting them according to their share of disbursements, the outcome rating is Unsatisfactory . While the revised project development objectives focus less on PSP by opening up investments in public utilities, the desired results were not achieved . Investment subprojects, (component 1) suffered from slow procurement (because of inadequate evaluation of consumer demand for sewerage connections ). Service improvements for the urban poor showed almost no activity . There were, however, some positive environmental impacts from the rehabilitation and expansion of the wastewater treatment facility in Olavarria. Rating: Unsatisfactory. Institutional strengthening (component 2) saw virtually no progress as the government had shifted away from PSP . The government also frowned on using consultants, as was planned in the project originally, favoring in -house capacity (see section 9). There were no national budget allocations made to these activities, except for the national WSS information system (SPIDES), which is currently not being supported. Rating: Unsatisfactory. Funds for the environmental actions component (component 3) were never disbursed. The government never took interest in this component . Rating: Highly Unsatisfactory. a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The realization of the full benefits of the project will depend on several variables : (i) the financial sustainability and efficient operation of each utility; (ii) government ownership and commitment at the provincial and federal level to make connections to new sewerage services possible for all households for whom the connection has been made technically possible, through appropriate, well -targeted and coherent subsidy policies; and (iii) a renewed commitment from provincial and federal government entities to support institutional strengthening in the sector, especially with regard to regulatory activities and productive partnerships between the operating utilities and the overseeing authorities. Thus far, progress in these areas is weak . The investment in utilities shows weak returns, government ownership continues to lag, and a renewed commitment to support institutional strengthening is absent . Overall, the government’s ownership and commitment to devise appropriate, well -targeted and coherent subsidy policies in order to increase connection rates to newly available sewerage networks is unclear . There is an awareness of the problem, but there is no concerted federal strategy to strengthen regulatory capacity at the provincial level. In addition to the above risk considerations, appropriate measures will be necessary to ensure that the increased capacity for wastewater disposal generated under the project is accompanied by the appropriate capacity for wastewater treatment to avoid further environmental contamination . Finally, there are serious concerns about the sustainability of the WSS information system since there is no team in place to maintain the database or the website now that the project is closed, and no replacement funding has been allocated to this activity by the national agency for water and sanitation works (ENOHSA). a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Quality at entry: The risk of a changing macroeconomic environment was not considered in project design, even though the country was entering a recession at the time of approval . Likewise, the risk of a change in the government’s PSP-focused strategy or commitment to the project strategy was not fully considered, despite clear signals of mounting public opposition to PSP . Project design did not fully take into account the risk that tariffs would not be adequately adjusted to changing costs during project execution, despite the lack of experience in Argentina with tariffs closely following operational costs and the admitted infancy of the regulatory systems to be established. Quality of Supervision: The Quality of Supervision Assessment (September 2002) judged the project reporting to be weak and observed that there was no evidence that the M&E system described in the project ’s operational manual was being used by the project team, on either the Bank ’s or the borrower’s side. The report also noted that the project restructuring was conducted without an in -depth review of needs, and pointed out that the Bank should have focused more on addressing the emerging issues of PSP -oriented reform, such as social and poverty issues and regulatory issues, in addition to convincing the government of the project's objectives . Procurement processes were reportedly slow on the Bank side as well as the borrower ’s. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Initially, government ownership and commitment to achieving the project ’s objectives was high. But with the change in government, support was entirely lacking . Willingness to appoint new staff was absent . The government staffed the PMU with one person . The PMU was meant to be staffed with a coordinator backed up by team of eight consultants, however . Funds for this were provided for under the loan, but never used . It was subsequently decided that it would be better to staff it from within, but then this was never done . Following project completion, no arrangements were made to maintain the The SPIDES database . The PMU produced punctual and well -prepared semi-annual report to the Bank, but these reports did not systematically track progress on project performance indicators . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: Design: The national water and sanitation agency, through the PMU, was charged with supervising and controlling the compliance of sub-project executors with the operational and financial performance indicators detailed for the project, verifying their progress towards improved efficiency, and monitoring the progress of the poverty pilot sub-component and regulatory strengthening component . The M&E system as designed, however, was not geared to effectively consult the poor and monitor impacts on the poor, nor did the project ’s supervision budget allow sufficient resources to focus on this (Quality of Supervision Assessment [September 2002]) Implementation: Implementation of the M&E system was difficult because of serious understaffing (see section 9, Assessment of Borrower Performance ). While the PMU prepared reports to the World Bank on the status of the project, these reports did not systematically track progress on performance indicators for the project . Utilization: There was insufficient use of the monitoring and evaluation system . The Quality of Supervision Assessment (September 2002) judged the project reporting to be weak and observed that there was no evidence that the M&E system described in the project ’s operational manual was being used by the project team, on either the Bank’s or the borrower’s side. The M&E system was not used to monitor impacts on the poor, nor were resources ever made available for this purpose . The ICR noted that the project restructuring might have been an opportunity to strengthen M&E design and practice . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Originally most investments were to be rehabilitative, and the project was classified Environmental Category B . In the end, a substantial portion of the works actually financed consisted of new sewerage collection infrastructure and of the expansion of existing sewerage treatment capacity . Net environmental benefits were observed in one sub-project, and expected in another . On the fiduciary side, an unsatisfactory first round of bids in Tucuman was appropriately thrown out and the process started anew . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Unsatisfactory Unsatisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The tool that is used to achieve project objectives should not be used as the objective itself . Focusing on PSP as an objective meant that the project missed the opportunity to generate greater demand for sub -projects by including public utilities, and in turn improve achievement of the goal of enhanced efficiency and improved service quality and sustainability. Continuing open and candid dialogue with the Borrower about changing sector priorities can save a project . The dampening of government commitment toward PSP -oriented reform before loan effectiveness should have been an early indication of the need to revise the project components . Making the terms of sub-borrowing more flexible, without sacrificing efficiency objectives, could have helped to support utilities showing progress toward efficiency goals (not only privately managed utilities ) or addressing urgent needs for the poor . Evaluating the actual demand for services, and anticipated cost and affordability to consumers is important to help ensure that investments will be worthwhile . This project highlights the importance of assessing beneficiary demand and likelihood of connections to new services . The existence of a partially viable substitute for the new service offered (septic tank disposal) and high connection costs led to much lower than expected rates of connection to newly expanded sewerage networks in at least two of the three sub -projects. As a result, realized benefits are also much lower than expected. Using socio-economic assessment tools such as stakeholder consultations and surveys, combined with refined assumptions in the ex -ante economic analysis can help address this issue . 14. Assessment Recommended? Yes No Why? To learn more about the communication between the Bank and borrower, and to inform an upcoming IEG water study, which will include PSP in its analysis . 15. Comments on Quality of ICR: This is an well-written ICR that adds much value to an otherwise unsatisfactory experience . It demonstrates a keen understanding of the issues and difficulties, and lays out the experience in a clear thorough manner, closely following ICR guidelines. It candidly reports shortcomings along with insightful explanations . The economic analysis was thorough and well written. There were inconsistencies in the numbering of paragraphs making the reading a bit confusing at times, but overall it is an impressive ICR . a.Quality of ICR Rating : Satisfactory