Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) Report Number : ICRR0021300 1. Project Data Project ID Project Name P114204 ELETROBRAS Distribution Rehabilitation Country Practice Area(Lead) Brazil Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-78840 30-Jun-2015 215,102,317.36 Bank Approval Date Closing Date (Actual) 03-Jun-2010 29-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 495,000,000.00 0.00 Revised Commitment 271,812,880.00 0.00 Actual 271,812,880.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Fernando Manibog Dileep M. Wagle Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The project development objective (PDO) was “to improve the financial and operational performance and the commercial management of the six DisCos by reducing electricity losses, increasing bill collection rates, and improving quality of service.” (Loan Agreement dated February 24, 2011, Schedule 1, page 6). "DisCos" stand for distribution companies. The Project Appraisal Document (PAD, dated April 18, 2010) and the Implementation Completion and Results Report (ICR, dated June 29, 2018) both stated the same PDO. Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) This ICR Review will assess the extent to which three related but separable objectives were achieved in following six DisCos: (1) Eletrobras Distribuição Alagoas (CEAL) Company in State of Alagoas; (2) Boa Vista Energia in Roraima State; (3) CEPISA in Piauí State; (4) Eletrobras Distribuição Rondônia (CERON) in Rondônia State; (5) Eletroacre in Acre State; and (5) Amazonas Energia in the State of Amazonas. Three objectives are: PDO1: To improve operational performance PDO2: To improve commercial management PDO3: To improve financial performance A split evaluation was not conducted because the lowering of the project scope was commensurate with the lower commitment size. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project had two main components: Component 1: Service Quality improvement and Loss Reduction (Appraisal estimate, US$675.3 million; actual US$259.6 million for IBRD funds only, as disaggregated figures on counterpart funding were not available) This component was intended to improve the quality of service, reduce electricity losses, and increase collection rates in electricity distribution. Goods, equipment, works and services was to be acquired for three main areas: (a) distribution network reinforcement (US$257.4 million); (b) implementation of Advanced Metering Infrastructure (AMI) and other measures to maximize metered consumption (US$387.9 million); and (c) modernization of the DisCos’ Management Information System (MIS) (US$30 million). Component 2: Institutional Strengthening (Appraisal estimate, US$34 million; US$10.99 million for IBRD funds only, as disaggregated figures on counterpart funding were not available) ) This component was intended to strengthen the operational capacities of the six DisCos through technical assistance, training, studies, equipment and consulting services. The specific activities include: (a) performance-based management (US$13 million); (b) environmental and social impact management (US$2 million); (c) support to community outreach (US$9 million); and (d) support to selected programs at a sectoral training and research training center in the state of Acre. Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Costs At appraisal in April 2010, the total project cost was estimated at US$709.3 million. Given the cancellations in the Bank loan and the Government counterpart at the June 2015 and March 2017 restructuring, the total disbursement and actual project cost at the December 2017 closing date was US$419.7 million. Financing At appraisal in April 2010, the total project cost of US$709.3 million was to be financed by a Bank loan of US$495.0 million and the Borrower’s contribution of US$214.3 million. At the first Level 2 restructuring on June 30, 2015, US$143.81 million had been disbursed, leaving a balance of US$351.69 undisbursed. At the second restructuring on March 30, 2017 (9 months before the closing date), more than half of the Bank loan or US$271.81 million had been disbursed and US$223.19 remained undisbursed. This was also the amount disbursed when the project closed on December 31, 2017, representing 45 percent less than the originally approved amount. Borrower Contribution The Borrower’s aggregate actual contribution was US$147.9 million, or 31 percent less than the appraisal estimate of US$214.3 million, as indicated above. During the first project restructuring on June 2015, counterpart funds were cancelled. The Bank filled the financing gap by reallocating project savings that accrued due to the devaluation of the Brazilian Real and cost efficiency gains from successful international bidding. At the second restructuring on March 2017, activities that were added during the first restructuring (to take advantage of large savings due to devaluation of the Real and successful competitive bidding) were cancelled—following a decision by Electrobras shareholders not to disburse any new loan proceeds to the DisCos except for contracts signed before August 4, 2016. Dates The project was approved on June 30, 2010 and became effective on May 23, 2011. A midterm review was conducted two years later on May 28, 2013. There were two Level 2 restructurings. The first was on June 30, 2015, during which the financing plan, the implementation schedule, and the results framework were changed, and the closing date extended as a result. The second was on March 30, 2017, during which components and costs were changed, the results framework and implementation schedule were modified, both Bank and Borrower financing was cancelled while remaining funds were reallocated among disbursement categories, and the closing date was extended once again. The project, originally set to close on June 30, 2015, closed two and a half years later on December 29, 2017. Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) 3. Relevance of Objectives Rationale The relevance of the PDOs is high with respect to the current strategies of the government and the Bank. The PDOs are fully aligned with the Country Partnership Framework (CPF) for FY18 to FY23. The Eletrobras project’s objective of improving electricity service quality and reducing service interruptions—in six of Brazil’s poorest states and mostly in urban areas—directly serve the CPF’s Focus Area 3 on “inclusive and sustainable development” by providing “inclusive and sustainable urban services.” Moreover, the project’s PDOs addressed one of the risks identified in the CPF related to “the difficulties in attracting private investors”, by supporting the Government in improving the technical capacity and financial position of the six DisCos with a view to their future privatization. In this specific regard, the recent actions taken by the Government further underline the high relevance of the project’s PDOs: (a) Early in 2018, the Government disclosed the draft bill proposing to privatize Eletrobras; and (b) In March 2018, the Government formed a special congressional committee to analyze the proposed privatization bill. As recently as July 2018, Brazil’s planning minister announced the Government’s plans to auction off one of the units owned by Eletrobras in July and holding the auction of the other five units owned by Eletrobras until August. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve operational performance Rationale Theory of Change The PAD does not discuss the project’s theory of change. It only refers briefly to the higher-level objective to which the project contributes, by stating that improvements in the quality of electricity services in the six states are “a means to improve the quality of life and foster economic growth.” (PAD, paragraph 19, page 7) The PAD’s Annex 3 present’s the project’s Results Framework, which contains the year-by-year targets for Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) intermediate and final outcome indicators in the six DisCos. Actions and specific percentages that demonstrate institutional improvements are also delineated. While useful for monitoring purposes, the Results Framework does not analyze causality and attribution aspects that a credible theory of change would require. The ICR (Figure 1) does present diagrammatically the project’s theory of change (Figure 1, page 7). It makes a laudable effort (infrequently seen in recent ICRs) to delineate the critical assumptions that would facilitate the progression of activities into outputs, and outputs into outcomes. As correctly represented, many of these assumptions are institutional capacity factors and behavioral changes. However, the accompanying text (ICR, paragraph 5, page 6) only briefly reiterated the PDOs and the key outcomes lifted from the boxes in Figure 1. However, a rigorous theory of change would require an analysis of whether the project’s activities were the right ones (e.g., other important activities were not neglected), and whether they were also of an adequate scale and timing. The combined or joint effect of the diverse activities also need to be shown as linked along a causal chain from outputs to outcomes. Finally, the attribution of those outcomes to the project’s specific interventions needs to be analyzed based on a counter-factual. Reiteration of the list of PDOs, outputs and outcomes does not achieve this analysis. It is important to note that (a) the energy regulator ANEEL required some indicators to be adopted and (b) PETROBRAS added its own indicators, hence there were 26 PDO-level outcome indicators and 42 intermediate output indicators. This seems to be an excessive number of indicators, particularly the 26 indicators meant to capture higher-level outcomes. The ICR (paragraph 93, page 27) indicates that "the overly ambitious targets were not adequately adjusted because Eletrobras wanted them to be aligned with the required indicators and targets from the regulator." However, it would have been advisable to reduce the number of PDO-level indicators. (ICR, paragraph 89, page 26) The outputs and outcomes presented below for the three PDOs were based on Annex 1: Results Framework and Key Outputs of the ICR, pages 36 to 59. PDO 1: To improve operational performance Outputs The following outputs were substantially achieved or exceeded: • New medium voltage (MV) lines - 1,032 kilometers were installed compared to 1,096 kilometers planned. • Automatic reclosers – 1,465 of the planned 966 reclosers were installed, thus exceeding the target. • Voltage regulators and banks of capacitors – 761 were installed compared to 772 planned. Amazonas, CEAL and Boa Vista exceeded their respective targets. • To mainstream socio-environmental guidelines, training and workshops were conducted, and the following were produced: (a) three manuals on Works, Hazardous Wastes, and Solid Wastes; (b) two Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) informational handbooks; and (c) four executive reports. The following intermediate result was modestly achieved: • Rehabilitation of MV and low voltage (LV) lines and upgrading of circuits – 988 kilometers were rehabilitated compared to 2,066 kilometers planned. Outcomes The following key outcomes resulted from the upgrading of distribution networks in the sic DisCos (by rehabilitating LV and MV lines, and installing automated reclosers), the installation of voltage regulators and banks of capacitors, and the adoption of environmental and social guidelines: • Decrease in the frequency of service interruptions. Except for Amazonas and CEAL, the other four DisCos achieved their target reduction (CERON and CEPISA exceeded their respective targets. For CERON, the appraisal baseline for service interruptions was 43, the target was 28, and 19.5 was achieved. For CEPISA, the appraisal baseline for service interruptions was 33, the target was 21, and 14.7 was achieved.) • Decrease in the duration of service interruptions. Except for Electroacre and CEAL, the other four DisCos achieved their target reduction (CEPISA and Boa Vista exceeded their respective targets.) • Institutional improvements through the adoption of sound social & environmental management practices and improved social action and communication programs. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve commercial management Rationale Outputs The following outputs were substantially achieved: Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) • Advanced Metering Infrastructure (AMI) – 12,475 AMI meters were installed for the planned 7,060 MV consumer points, and 40,851 AMI meters were installed for the planned 103,000 LV consumer points. • Re-listing of consumers – 3,130,099 consumer points and 1,888,653 consumers were re-cadastered, based on the decision during implementation that the current Commercial Management System (CMS) did not need replacement, but the cadaster of consumers needed to be updated. • Environmental and social training and workshops --these were provided to improve commercial management, together with improving operational performance as indicated above in PDO 1. The following intermediate result was not achieved: • Full implementation of Enterprise Resource Planning (ERP) across the six DisCos – Eletrobras opted for using SAP for the entire company instead of ERP. This goal was not achieved due to costs, conflict with procurement rules and time delays. Outcomes The main outcomes were the installation and adoption of AMI systems, the re-cadastering of consumer points and consumers, and the roll-out of environmental and social guidelines and training (although Eletrobras opting instead for SAP, hence the ERP was not implemented as planned). 100 percent of MV clients were included in the AMI system, which broke the culture of fraud. This was possible because the number of MV fraudsters was small (compared to LV users), whereas detection was instantaneous and corrective response was quick, thus making investments in fraud unprofitable. In Rondonia, for example, the number of recidivists (who were caught but tried to commit fraud again) were reduced from 28 to 3. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective To improve financial performance Rationale Outputs The intermediate results discussed under PDO 2 above—related to AMI installation and re-cadastering of consumers--and were assessed as substantially achieved, also support the achievement of PDO 3. Outcomes Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) The PDO indicator of reducing total losses was not achieved. At the project’s December 2017 closing, the total losses for all the six DisCos were still significantly higher (i.e., ranging from 6 to 11 percentage points) than the lower targeted levels that were supposed to be attained after losses had been reduced. The actual levels achieved were near to the 2010 baselines, some showing only marginal improvements. At closing, Amazonas had a total loss level that was worse, i.e., higher than the appraisal baseline. The PDO indicator of increasing collection rates was also not achieved. With the exception of CEPISA, the other DisCos did not achieve their targets. At the project’s closing in 2017, CERON and CEAL remained at the original baseline level in 2010, showing no improvement. Electroacre and Boa Vista showed declines in collection rates. The ICR correctly indicates (paragraph 87, page 26) that the loss reduction and collection rate indicators mismatched the scope of the project activities: "They reflected the assumption that the package of measures to be implemented by the project was comprehensive enough to address all loss drivers, which turned out not to be true. Therefore, they were eventually too broad and thus too sensitive to exogenous factors to properly capture the contribution of the project..." Paragraph 89 also indicates that: "...the indicator measuring collection rates--already reasonably high--focused on a second order issue because their improvement would not lead to a game changer with respect to revenues." More direct measures of financial performance would have been more appropriate. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale Taking into account the substantial achievements in operational performance (PDO 1) and commercial management (PDO 2), as well as institutional improvements particularly the adoption of environmental & social management practices, the overall project efficacy rating is Substantial. PDO 3 to improve financial performance was achieved only to a modest extent, which according to the ICR (paragraph 50, page 17) was influenced significantly by a combination of the severe economic recession, huge tariff hikes that led to higher levels of payment defaults, and the migration of large industrial customers that left a larger base of poor paying customers. As a counter-factual, the ICR further indicates that as a result of implementing the project’s activities, over 725 GWh were recovered and aggregated, without which losses would have been much greater by the time the project closed. Overall Efficacy Rating Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) Substantial 5. Efficiency At appraisal, the project's economic benefits were defined as savings that accrue from the reduction of electricity generation resulting from lower technical losses and reduction on resource use due to increased efficiency. The reduction in CO2 emissions were also indicated but was not included in the calculations. Over a 10-year benefit stream under a base case scenario, and using a 10 percent discount rate, the economic internal rate of return (EIRR) was 18.5 percent and the net present value (NPV) was US$260 million. For the ex-post economic analysis, the economic value of quality improvement (i.e., the reduction in electricity supply interruptions) was calculated similarly as in the PAD, by taking the consumed energy (in MWh) as well as the prevailing tariffs upon the reduction of the indicator related to electricity supply interruption. As in the PAD, it was assumed that the monetary value billed by the DisCos from the additional and constant electricity supply reflects an equivalent benefit extracted by the consumers. Based on this conservative assumption-- which does not reflect the consumer surplus, the additional value generated by businesses, or the losses and damages that come with service interruptions--the EIRR is 13.1 percent using a 10 percent discount rate. There were shortcomings in administrative and implementation efficiency. The project's duration exceeded original plans (6.5 years instead of 5 years). The political and economic shocks during the project implementation led to the Brazilian Real’s rapid devaluation of over 50 percent. Out of 252 contracts, only eight were ICBs hence 244 contracts were to be bid in local currency. At the start of the devaluation, only US$ 21 million was disbursed but within a few months, the project doubled the currency available for project execution. At project closure, disbursements were in local currency, i.e., R$842 million out of the appraisal estimate of R$ 866 million. Moreover, with the Bank’s strong advocacy and support, the project achieved substantial price reductions through its competitive ICB processes. For example, the largest ICB contract was signed for US$94 million, instead of the estimated US$130 million. Taking these considerations into account, the project's efficiency is rated substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) Rate Point value (%) *Coverage/Scope (%) Available? 95.00 Appraisal  18.50 Not Applicable 95.00 ICR Estimate  13.10 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's current relevance to the strategies of both the government and the Bank is high. The achievement of PDO1 to improve operational performance is substantial, based on the decreases in the frequency and duration of service interruptions in most of the DisCos, and their adoption of social and environmental management practices. The achievement of PDO 2 to improve commercial management is also substantial, based mainly on the installation of Advanced Metering Infrastructure (AMI) systems and the re- cadastering of consumer points and consumers. The achievement of PDO 3 is modest, mainly because the DisCos did not achieve the targeted reduction in total losses. The overall efficacy rating is substantial. Although there were implementation delays, efficiency is substantial, taking into account the price reductions from ICB processes, and the doubling of the currency available for project execution as a result of the significant devaluation of the Brazilian Real. The overall outcome rating is moderately satisfactory (as in the ICR) in line with the Bank Guidance for ICR Preparation dated September 27, 2018 (Appendix H, page 38), which indicates a moderately satisfactory rating when “there are moderate shortcomings in the achievement in one or more of the objectives/outcomes use in the assessment of the overall Efficacy.” For this project, this applies specifically to Objective 3 (“To improve financial performance”), which has a modest rating for Efficacy. It is important to note that after the ICR’s completion, the six DisCos that have benefitted from the project have been privatized. This was a significant outcome since they were in very bad shape and considered unsuitable for privatization when Eletrobras took control. While their privatization cannot be entirely attributed to the project, it did play a role in improving their technical, financial and commercial performance thus making them more attractive to investors. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) Complex and inter-related economic, socio-political and financial factors affect the operating environment for the six DisCos, which could present moderate risks to the sustainability of project outcomes. For example, a tariff increase--such as the large ones that occurred in 2015--could trigger higher non-technical losses and lower collection rates that could directly impact the finances of the DisCos. A mitigating factor is that as of January 2018, the government has passed national legislation that established a variable tariff rate that is cheaper off-peak. Brazil's economy is also recovering from a severe recession with GDP growth rates increasing in 2019, and unemployment and inflation also projected to decrease. These trends would positively impact collection rates. The installation of smart meters (AMI) for big consumers, e.g., factories, hotels, schools, hospitals, and businesses, have resulted in maintenance improvements, better management of voltage control, and lower levels of theft. These taken together are likely to lead to additional loss reduction and improved collection rates, which would help improve the financial performance of the DisCos. Nonetheless, Brazil's uncertain medium-term prospects need to be watched carefully. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank adopted a strategically relevant approach of addressing operational, commercial and financial issues. Although already established elsewhere, the selection of the AMI system was considered innovative (by enabling distant oversight by Electrobras in Brasilia) and appropriate for Brazil as an upper- middle-income country. According to the ICR (paragraph 105, page 30), "the project pioneered the design and implementation of AMI at a large scale in developing countries." The project design's emphasis on integrating institutional improvements, including social and environmental management, was highly appropriate. The Bank team also identified risks effectively, including political and other interference risks, and proposed sufficient mitigation measures. There were, however, project design shortcomings. For four of the DisCos, the baselines were incorrectly defined initially. Also, the project design assumed that a new commercial management system was needed (CMS) but it was only during implementation when it was realized that the problems being encountered was not because of an inadequate system but resulted rather from the faulty quality of the input data. This led to the need for a full re-registration of the consumer base. There were also weaknesses in the initial procurement capacity assessment, which did not fully capture the large capacity requirements (for an actual 260 instead of the originally estimated 80 contracts) and the also did not address early enough the initial reluctance to use ICB procedures. More generally, there was inadequate emphasis on training and on advancing or completing change management processes at the project preparation stages, thus contributing implementation delays. Finally, the loss reduction targets and hence the financial performance PDO were over-ambitious and proved very difficult to achieve during implementation. In light of the significant design shortcomings and over-ambitious financial performance targets, the Bank's quality at entry is rated moderately satisfactory. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank project team conducted 14 missions to support implementation, but visited only two states on each mission due to budget constraints. After an initial period of slow disbursements and a Moderately Unsatisfactory rating for both the Development Impact (DI) and Implementation Progress (IP), a seasoned TTL moved in Brasilia and took over the project in July 2013, with support from a junior energy specialist in Washington D.C. The more intensive supervision resulted in an increase in the disbursement rate and an upgrading of both DP and IP to Moderately Satisfactory within a period of 18 months. Another evidence of proactivity during supervision was the involvement of a senior procurement specialist to provide intensive assistance to the client in its very first and highly complex ICB procurement process for the Advanced Metering Infrastructure (AMI), which was encountering many difficulties given the isolated, low-competition Amazonian market. The Bank's AMI expert also joined supervision missions, and a specialized, external consulting firm was contracted to augment the Bank's capacity in reviewing the AMI bid evaluation process. A special monitoring tool was developed to monitor the 254 contracting processes. According to the ICR (paragraph 106, page 30), "...the World Bank team went beyond usual supervision and teamed up with the central PIU team to find solutions and overcome obstacles...Before the project closed, the Bank team and PIU co-organized a series of evaluation workshops, three regional and one national to discuss difficulties and lessons learned, including the World Bank performance, which was unanimously praised by the participants." Taking into account the above positive steps, and the resulting increase in disbursements, the successful AMI procurement, and putting the project on track toward achievement of the PDOs, the quality of the Bank's supervision is rated satisfactory. In line with IEG Guidelines for Reviewing ICRs (page 56), the overall Bank Performance rating is moderately satisfactory when Bank Performance "was rated moderately satisfactory on one dimension and satisfactory or highly satisfactory on the other dimension", which is the present case given the moderately satisfactory Quality at Entry and satisfactory Quality of Supervision. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E design was highly complex, with 26 PDO-level indicators and 42 intermediate indicators, which appeared excessive, potentially overlapping, and may not all be necessary. Additional indicators were developed by Electrobras during implementation. The Brazilian regulator--Electricity Regulatory Agency (ANEEL)--required many of the indicators, which did have its advantages. ANEEL has the strong incentive to be informed to help determine tariffs and penalties; moreover, ANEEL regularly improves the methodology and informs the DisCos of best practices. However, certain indicators mismatched the scope of project activities, such as the loss reduction and collection rate indicators, which were designed under the assumption that the project's activities alone were sufficient to address all of the drivers behind losses, which was not the case. Thus, those two indicators proved to be too broad and sensitive to exogenous factors such a political and economic crises. More specific indicators to measure financial performance would have been more appropriate. The institutional improvement indicators (such as "the adoption by DisCos of sound environmental management practices, improved M&E, and effectiveness of social action and communication programs") was not well formulated and proved difficult to measure particularly since it was ambiguous and did not have a baseline. b. M&E Implementation M&E was implemented effectively, despite some moderate shortcomings. On the positive side, the fact that the energy regulator (ANEEL) itself required the indicators benefitted M&E implementation by giving the DisCos strong incentives to routinely collect and analyze performance data for subsequent verification by the regulatory agency. Eletrobras had sufficient staff and demonstrated strong commitment to monitor all the project activities through regular field visits and a close working relationship with the financial management team's inspections before payment installments were approved. The M&E process was in line with the performance-based management system under the project's Component 2. Data was entered into the MIS, analyzed, and included in semi-annual reports, which is the reporting frequency required by the Loan Agreement. According to the ICR, para 92 (page 27), the quality of the reports "was sound and data were well presented and analyzed." Some minor shortcomings include the following: (a) the six core technical indicators and four social indicators that were added during the 2-13 mid-term review were not incorporated in the Eletrobras Operational Manual or its Progress Reports, hence data for those indicators were not reported on a systematic basis; and (b) while full adoption of ANEEL's indicators had its advantages, mainly in terms of providing strong data collection and reporting incentives, it worked against proactively adjusting what was widely recognized as overly ambitious performance targets. c. M&E Utilization Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) During the monthly meetings between Eletrobras and the six DisCOs, M&E data was discussed and actions were proposed to improve implementation performance. Eletrobras management used M&E data to report on implementation progress, comply with mandatory regulatory indicators, and also to try to reduce penalties when those are levied. The regulatory indicators pre-existed the project, hence Electrobras was able to quickly detect the deterioration of the project's baselines once the exogenous shocks started to occur during the project's initial years. Eletrobras added its own indicators to the ones required by ANEEL. These indicators, which were related to performance-based management, were used extensively by the central PIU and also the PIUs of the individual DisCos. However, the M&E data could have been more actively utilized much earlier to adjust the overly ambitious target indicators. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was assigned an Environmental Category B at appraisal. The project triggered the following safeguard policies: Environmental Assessment (IOP/BP 4.01), Natural Habitats (OP/BP 4.04), Pest Management (OP/BP 4.09), Indigenous Peoples (OP/BP 4.10), physical and Cultural Resources (OP/BP 4.11), and Involuntary Resettlement (OP/BP 4.12). The ICR (paragraph 100, page 28) briefly states, without much background discussion, that: "Overall, there were no safeguards compliance issues...There were no adverse impacts with physical and/or economic displacement and any interference with indigenous people." Except for one instance in 2012, the Bank supervision's ratings for safeguards compliance was all Satisfactory. The sole rating of Moderately Satisfactory was due to the need in 2012 for the DisCos to strengthen their social and environmental team. b. Fiduciary Compliance Financial Management (FM). Fiduciary compliance during project implementation and Bank supervision had instances of Moderately Unsatisfactory (MU) ratings as well as Substantial Risk ratings. MU ratings resulted from errors and inconsistencies in the Interim Financial Reports (IFRs) which were being prepared manually, incomplete and inaccurate reports from the DisCos, and shortages in FM staff at the central and DisCo levels. Eletrobras never implementated the Bank's recommendation to use a specific system or software to consolidate the IFRs, opting instead for extra work to perform manual checks that could have been automated. All IFRs submitted during the project were considered acceptable but were generally submitted late (but within 30 days). Ineligible expenditures have not occurred. As of the ICR's preparation (June 2018), three of the five audit reports were received on time, and all the audit reports had unmodified (i.e., clean) opinions. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) Nonetheless, on a consistent basis, the Office of the Federal Controller General highlighted areas of improvement in internal controls, particularly in the DisCos. Procurement. Significant procurement delays occurred in 2013 due to exogenous shocks, leading to a Moderately Unsatisfactory rating. The delays were caused by limited procurement capacity, inability to prepare sound bidding documents, delays in attracting bidders, incompatibility of Bank and national procedures for International Competitive Bidding (ICB), and technical issues related to the complicated homologation process for the new smart meters. Moreover, Eletrobras wanted to support Brazilian companies and was reluctant to open the market to international companies, citing concerns about discontinuity of maintenance in remote regions. To surmount these bottlenecks, the Bank reinforced supervision and assigned a new, field-based TTL, which significantly facilitated procurement and implementation follow-up. A specialized engineering firm was also contracted to provide assistance with the highly complex procurement packages. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Quality at entry had shortcomings related to design flaws and over- ambitious financial Moderately performance targets. Quality Bank Performance Satisfactory Satisfactory of supervision is satisfactory. In line with IEG guidelines (see Section 8b), the overall Bank Performance rating is moderately satisfactory. Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) The Bank team conducted evaluation workshops in Brazil during the December 2017 ICR mission, wherein the main lessons presented below (with some IEG paraphrasing) were identified and discussed. The ICR (paragraphs 113 to 123, pages 33 to 35) provides a more extensive set of lessons and recommendations, with some references to future sector outlook. 1. A sound database and the regularization of customers need to be included as part of the initial project design and prioritized once implementation starts. The re-cadastering of customers is a crucial step to addressing massive fraud in a systematic way, and can bring considerable benefits in terms of improving commercial management and financial performance. However, regularization of customers is a costly and lengthy process, hence the extensive relisting needs to be part of project design and given high priority at the start-up of project implementation. 2. The capacity of distribution companies need to be strengthened so that they can adequately and systematically detect and correct massive practices of fraud. While the Advanced Metering Infrastructure (AMI) was able to break the culture of fraud among the relatively small number of large MV clients, all of whom were included in the AMI system, it is much more difficult to address the huge numbers of illegal connections on LV lines. The need to relist these led to a "tsunami" of cases that the DisCos were not prepared to regularize in the short time required. A much scaled-up effort that includes regularizing and shielding the billing of LV customers, as well as legal support to unlock interventions in invaded areas, seem to be required to accelerate the response capacity of the DisCos. 3. Close supervision and intensive support are required to facilitate large, complex and innovative procurement packages. Bank project teams need to conduct a comprehensive capacity assessment during project preparation and appraisal, so that capacity gaps can be filled early on (such as appointing a field-based TTL, a procurement expert, and/or a specialized engineering firm) and implementation delays can be minimized or avoided. New bidding documents can also be developed and tailored to combine goods and information technology services, as was done for the innovative AMI contract in this project. 4. The Results Framework and the performance indicators need to be adjusted in line with the realistic factors that affect financial and operational performance. In this project, the indicator for "total loss reduction" as a percentage of total energy was too broad. It proved to be highly sensitive to too many external factors that were beyond the project's control, and thus was not accurately indicative of the project's contribution. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ELETROBRAS Distribution Rehabilitation (P114204) The ICR is well prepared. The ICR team clearly put a major effort in the preparation of the ICR, and the extensive and quantitatively-based Annex 5: Borrower's Comments augments the ICR's high quality even more. Annex 4 on the Efficacy and Efficiency analysis was very helpful in providing detailed tabular and graphical data that could not all fit in the main text. The ICR is strong on accountability. It relies substantially on evidence that is technically grounded in the electricity sector and is well-referenced, particular in terms of documentation sources. The ICR's lessons and recommendations are solidly based on the project's own implementation experience, and more specifically on evaluation workshops that were held in Brazil during the December 2017 ICR preparation mission. The ICR's analysis is highly results-oriented and frequently uses the project's Results Framework as a reference point. It is candid, internally consistent, and in line with the OPCS guidelines for ICR preparation. However, the discussion of environmental and social safeguards seemed too brief at 6 lines. Also Annex 3 seemed to confuse project costs with sources of financing. The ICR's main text was also long at 35 pages, not counting the annexes, which is more than twice the length indicated by OPCS. With the annexes, the ICR document amounts to 193 pages and could be more concise. a. Quality of ICR Rating Substantial Page 17 of 17