Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Report Number : ICRR0021064 1. Project Data Project ID Project Name P108444 MZ-Energy Dev. & Access Project (APL-2) Country Practice Area(Lead) Mozambique Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46810 30-Jun-2015 80,000,000.00 Bank Approval Date Closing Date (Actual) 04-Feb-2010 15-Jun-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 80,000,000.00 0.00 Revised Commitment 80,000,000.00 0.00 Actual 71,671,496.32 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ihsan Kaler Hurcan Peter Nigel Freeman Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives In August 2003, the Board of the World Bank approved a two-phase adaptable loan program (APL) for Mozambique, which was called the Energy Reform and Access Program with the objective “to increase access to modern energy in peri-urban and rural areas, thereby facilitating improved quality of life of the respective communities and generating income” (PAD, p.6). This project constituted the second phase of the APL series. Before the Board approval of the second phase, the name of the program was changed to the Energy Development and Access Program (i) to align the program with the new energy policy and strategy of the Government of Mozambique (GoM); (ii) to align the second phase of the program with the GoM’s National Energy Development and Access Program (NEDAP); and (iii) to reflect the scope of the Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) International Development Association’s (IDA) new assistance strategy for the energy sector in Mozambique (PAD, p.6). According to the financing agreement (FA, p.5) and the project appraisal document (PAD, p.11) the development objective of the Energy Development and Access Project (EDAP) was: “to increase access to electricity and modern energy services in peri-urban and rural areas in a sustainable and affordable manner.” In this review, the development objective will be evaluated as consisting of two objectives: Objective 1: to increase access to electricity in peri-urban and rural areas in a sustainable and affordable manner. Objective 2: to increase access to modern energy services in peri-urban and rural areas in a sustainable and affordable manner. A split evaluation has been undertaken in this review, because the project scope was changed at the first restructuring (please see sections 2.d and 2.e below) while the project objectives remained the same. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The project had three components (FA, p.5). 1.Reinforcement of the Primary Networks and Grid Extension. (Appraisal cost: US$50.0 million; Actual cost: US$49.3 million) This component consisted of (i) the rehabilitation of existing primary lines in Maputo, Matola and Chimoio; (ii) the extension of medium and low voltage grid in peri-urban areas in Nampula and Tete provinces; and (iii) installation of a transmission efficiency pilot project at Bairro 25 de Junho. In addition, technical assistance in project implementation, procurement, supervision, and social and environmental safeguards was to be provided to the Electricidade de Moçambique (EdM), the implementing agency of Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) this component. 2. Investments on Rural and Renewable Energy. (Appraisal cost: US$18.0 million; Actual cost: US$17.7 million) This component aimed at (i) scaling up micro and small investments on renewable energy production and distribution systems, such as the installation of solar photo voltaic (PV) systems in rural school and health clinics, electrification of selected villages by renewable energy technologies and development of a market for PV products in rural areas and facilitating the participation of private sector; (ii) promoting the use of biomass energy systems, such as the distribution of improved woodfuel stoves, introduction of improved charcoal kilns, support to inter-fuel substitution for traditional biomass, and demonstration projects to accelerate market penetration of renewable energy technologies; and (iii) technical assistance in project implementation and management to the Fundo de Energia (FUNAE), the implementing agency of this component. 3 . Energy Sector Planning, Policy and Institutional Development. (Appraisal cost: US$10.2 million; Actual cost: US$4.7 million) This component included technical assistance activities aimed at (i) elaborating and disseminating the national Rural Electrification Strategy and Investment Program; (ii) strengthening institutional capacity in Mozambique for the development of new energy infrastructure projects; (iii) strengthening the institutional capacity of the National Electricity Commission (CNELEC) in conducting its advisory and regulatory function; and (iv) strengthening the capacity of Ministry of Energy (later Ministry of Mineral Resources and Energy – MIREME) to improve its performance and governance, including support for the preparation and broad consultation process for a flexible sector wide approach, and financing of operating costs. Revised Components At the time of the first restructuring in February 2015, which was four months before the original project closing date of June 2015, the components were revised as follows, because of the changes in government needs and priorities, the necessity to move funds among project activities and the decrease in available funds due to sharp appreciation of the USD against the SDR, the credit currency: Component 1: Reinforcement of the existing Mocuba and Pemba substations through the provision of four new power transformers and respective switchgear was added as a new project activity due to changing government priorities and urgent investment needs to meet the increase in power demand. Component 2: Activities related to improved charcoal kilns and inter-fuel substitution for traditional biomass were cancelled. Demonstration projects to accelerate market penetration of renewable energy technologies were also cancelled. Component 3: Support for the elaboration and dissemination of the Rural Electrification Strategy and Investment Program and for the preparation and broad consultation process for a flexible sector wide approach was cancelled. Instead, upon the request of GoM, installation of six solar water heaters in selected hospitals was included as a project activity under this component. Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost was originally estimated at US$80 million including contingencies and an unallocated amount of US$1.8 million. The actual cost at project completion was US$71.70 million. The ICR (p.6) states that a strong appreciation of the USD against the credit currency of SDR, and subsequently against the local currency of MZN, which amounted to a US$8.7 million decrease in the available funds, was the main reason for not only the difference between the estimated and actual project costs, but also the changes made in the project activities, as explained in Section 2.d. Financing: The project was to be financed by the IDA without any other co-financier. In line with the estimated project cost, the IDA funding was set at US$80 million at appraisal. However, as explained in the previous paragraph, due to the appreciation of the USD against the SDR and the MZN, the funding amount dropped by US$8.7 million. Borrower contribution: At appraisal, there was no borrower contribution foreseen for this project. However, FUNAE financed additional PV systems in the amount of US$439,279. Restructurings and Dates: There were three restructurings. • First Restructuring (February 20, 2015): The project closing date was extended for 18 months from June 30, 2015 to December 31, 2016. After the Board approval, it took 13 months for project to become effective. The project implementation was further delayed by another seven months after effectiveness because the initial deposit to project account could not be made before the outstanding funds of the phase 1 project were all refunded (ICR, p.25). Changes to the national procurement rules resulted in longer clearance times by the GoM which in return slowed down project implementation. Because of these reasons, an 18-month project closing date extension was needed to complete project activities. As explained in Section 2.d above, the project components were revised. The main reasons for this revision were the changes in the GoM’s needs and priorities, the need to move funds among project activities since some required more financing than initially estimated and some required less, and the appreciation of the USD against both the SDR and the MZN which resulted in a project funding shortfall of US$8.7 million. There were also some changes made in the results framework. Major ones are given below: • Baseline of the first PDO indicator was erroneously entered as number of connections whereas it should have read as number of people provided with access to electricity. This baseline was corrected. • The second PDO indicator measuring “the number of people provided with access to electricity under the project by household connections (Rural)” was deleted and this figure was included in the overall connections figure. • A core indicator was added to measure the number of direct project beneficiaries. • Indicators related to the generation capacity of renewable energy constructed under the project and the National Rural Electrification strategy and investment program were deleted. • A new indicator was added to measure the progress in the implementation of solar water heaters at six hospitals. Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Lastly, related to flow-of-funds, the GoM agreed to establish a designated account for EdM which was responsible for 62 percent of the project amount. The other two implementing agencies, i.e. FUNAE and MIREME, had their designated accounts and the pace of disbursements made by these two agencies was relatively fast. On the other hand, EdM did not have a designated account and was using the GoM’s public financial management system which caused delays in disbursements. (For a detailed discussion of the changes made at the first restructuring, please see Restructuring Paper - RES14050.) • Second Restructuring (June 14, 2016): MIREME had mistakenly posted a Component 3 disbursement category under Component 1. Since there were not sufficient funds left to repost the amount to Component 3, a transfer in the amount of US$0.7 million was made from Component 2 to Component 3. This amount became available because of the partial cancellation of the solar PV maintenance contract under Component 2 (Restructuring Paper - RES14050, pp.6-7). •Third Restructuring (December 16, 2016): The project closing date was extended for another 6 months from December 31, 2016 to June 30, 2017, to allow the completion of project activities under Component 1 (Restructuring Paper – RES25461). 3. Relevance of Objectives Rationale Modern energy services are defined as consisting of two main components: (i) access to electricity; and (ii) access to clean cooking facilities, such as fuels and stoves that do not cause pollution in houses (International Energy Agency - IEA). Lack of physical access and affordability issues stand out as the main barriers for the expansion of modern energy services to the poorest households which are mostly in rural areas. The project appraisal document (PAD) did not provide a clear definition of modern energy services. However, the project activities are designed to increase electricity access in the urban and semi-urban areas through grid extension and in rural areas through the application of solar PV systems. Furthermore, the second component included the introduction of woodfuel stoves and improved charcoal kilns. The way the project activities are designed broadly follows the IEA's definition of modern energy services. Mozambique is one of the poorest countries in the world. One of the goals of the Government of Mozambique (GoM) is to alleviate poverty through the expansion of modern electricity services in the country. At appraisal in 2009, the electricity access rate in Mozambique was 10.5 percent. At project closing, it had increased to 26 percent. Furthermore, only two percent of the rural households have access to electricity, whereas the remaining 98 percent rely on kerosene and wood for lighting and cooking, respectively. Therefore, the project objective “to increase access to electricity and modern energy services in peri-urban and rural areas in a sustainable and affordable manner” is still highly relevant to the country conditions in Mozambique in order to alleviate poverty in the mid and long term. Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) The first project objective to increase electricity access was also consistent with the World Bank’s Country Partnership Framework (CPF FY2017-2021) at project closing. It corresponds to “Objective 4. Expanding Access and Improved Reliability of Electricity” where electrification access will be increased through the rehabilitation of existing grid and the application of mini grids, individual home systems, and lighting as off- grid electrification solutions. (CPF 2012-15, p.27). However, the current CPF does not include the objective to expand modern energy services through the promotion of biomass energy systems use, such as improved woodfuel stoves, improved charcoal kilns, and inter-fuel substitution for traditional biomass. IEG understands that the World Bank’s focus in the provision of modern energy services in Mozambique will focus on increasing the electricity access, both on-grid and off-grid, only. Therefore, the Relevance of Objective is rated Substantial. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective to increase access to electricity in peri-urban and rural areas in a sustainable and affordable manner. Rationale Outputs • Six kilometers of transmission line was constructed. The target was 156 km. However, according to the ICR (p.43), after the preparation of the final design it was found that a 6 km extension would be sufficient to achieve the target connections number. The relevant indicator in the project documents was not revised, but this change in the target value was acknowledged in the final Implementation Status Report. • The length of the distribution lines constructed was 256 km which was the target figure. • The number of off-grid schools electrified increased to 403 from a baseline figure of 150. The target was 400. • The number of off-grid health centers electrified increased to 345 from a baseline figure of 150. The achievement of this project activity fell short of the target figure of 400. • In the target 30 villages, 523 community connections were achieved. There was not a target set for the community connections. • According to the Implementation Completion Report prepared by FUNAE, the implementing agency of the second component, 2,519 solar PV systems are installed, including schools, health centers, households, lamp kits, pump systems, vaccine fridges, streetlights, police posts, commercial refrigerators and administrative posts. The target was 3,152 PV systems. • Integrated Business Management System (IBMS) is installed at Electricidade de Moçambique (EdM), Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) the implementing agency of Component 1. • There is no indicator measuring the sub-station rehabilitation activities. Outcomes • The number of people provided with new on-grid electricity service increased to 497,538 from a baseline figure of 255,000. Target was 405,000 people. In other words, the project’s goal was to establish new on- grid connections to 150,000 people. The achievement was 242,538 people. • The PDO indicator measuring “the number of people provided with access to electricity under the project by household connections (Rural)” was deleted at the first restructuring. Neither the ICR nor the project papers provided an explanation for the deletion of this indicator which was subsumed under the overall connections indicator (ICR, p.6). Therefore, there was no indicator which measured the increase in the number of people with access to electricity in rural areas separately. However, the ICR (p.14) calculates this number by using the World Development Indicators of the World Bank and arrives at 20,200 people, whereas the number of solar systems installed at household level is reported as 744 households in FUNAE's project completion report. If lamp kits are included, the total number reaches to 1,419 households (ICR, p.73). Assuming six persons per household, the total number of rural connections becomes 8,514 people. Although it is not possible to come to a clear conclusion about the number of people connected to electricity in the rural areas through project activities, given the number of solar systems FUNAE was able to install in these areas, i.e. 2,519 PV systems compared to a target number of 3,152 PV systems, the project could not fully achieve the expected outcome in rural electrification. • More number of schools are now electrified than targeted, i.e. 253 schools against a target of 250. • The number of health clinics electrified by the project, which is 195, did not reach the target of 250. This was caused by some health clinics not being suitable for PV installation where PV panels would be exposed to potential theft. This resulted in a decrease in the number of health clinics eligible for PV installation. The theory of change of the project regarding the project objective to increase access to electricity in peri- urban and rural areas was robust. The project activities to rehabilitate and reinforce existing networks, extension and intensification of grid in peri-urban areas, and installation of solar PV systems in rural areas were to be expected to lead to increased access to electricity in the project areas. However, project's theory of change did not clearly explain how this would be achieved in a sustainable and affordable manner. The project was successful in increasing electricity access in peri-urban project areas through grid expansion. The ICR (p.13) provides a detailed explanation of the affordability of grid-based access. According to the multi-tier electricity access framework of Sustainable Energy for All (SE4ALL), if a household spends less than 5 percent of household income on electricity, then this is regarded as affordable. The ICR calculates this ratio as 2 percent for grid-based customers. However, a similar calculation for off-grid-based customers results in a rate of 5.8 percent, which is higher than the affordability Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) rate defined by SE4ALL. Sustainability of on-grid electricity access is ensured through the rehabilitation and expansion of the grid (ICR, p.13). Upgrading of the substations in the project areas must have had a positive impact on the sustainability of electricity access since new connections to the grid are not expected to overload the system. However, the ICR does not provide information about the works done at the substations nor how these contributed to the achievement of this objective. Sustainability of off-grid access depends largely on the proper operation and maintenance (O&M) of the PV systems. The implementing agency of the second component, i.e. FUNAE, will remain the owner of the PV systems installed in the rural areas and has already made arrangements to train technicians who will be in charge of O&M of the PV systems (ICR, pp.15-16). However, availability of funds to FUNAE to cover the O&M costs of PV systems stands out as a potential risk to the sustainability of off-grid electricity access, which is also directly linked to the affordability issue. Overall, despite some minor shortcomings, the project was partially successful in increasing electricity access through both grid extension and application of off-grid solar PV solutions. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective to increase access to electricity in peri-urban and rural areas in a sustainable and affordable manner Revised Rationale Note: Since the project scope regarding this objective did not change at the first restructuring, a split evaluation has not been undertaken for this objective. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective to increase access to modern energy services in peri-urban and rural areas in a sustainable and affordable manner. Rationale Outputs Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) • Please see the outputs under Objective 1. • 15 PV water pumping systems were to be installed. Although all the systems were purchased under the project, none were installed because geophysical studies and drilling activities for the installation of these systems were not carried out. However, after project completion, FUNAE completed the geophysical studies and tendered out the installation of water pumping systems. • 4,000 improved woodfuel stoves were distributed against a target of 50,000. • No improved charcoal kilns were distributed. The target was 500. Outcomes The ICR (p.10) makes reference to an earlier impact assessment conducted under the first phase of the APL series to explain the qualitative results of increase in access to modern energy services. These include refrigeration of perishable drugs and vaccines at health centers, possibility of patient care after dark, literacy programs at night for rural residents, increased safety due to street lighting and substitution of kerosene with electricity for lighting. The project monitoring and evaluation (M&E) framework did not capture the impact of rural electrification on modern energy services, nor an impact assessment or a beneficiary survey was conducted after project completion. In rural households in Mozambique, biomass, especially wood fuel, is the primary energy source for cooking which is the main source of household air-pollution, a serious public health problem in poor countries. Therefore, distribution of 50,000 improved woodfuel stoves was expected to contribute to the solution of this problem. However, due to budget constraints only 4,000 woodfuel stoves could be installed. Due to the same reason, no improved charcoal kilns were distributed. These budget constraints eventually led to the restructuring of the project components (See section 2.e). Overall, when evaluated according to the project’s theory of change before restructuring, the project was not successful in increasing access to modern energy services through the distribution of improved woodfuel stoves and charcoal kilns, and the installation of PV water pumping systems. Although the project was successful in increasing electricity access, which is captured in the first objective, it is not possible to come to a conclusion about the level of its impact on increasing modern energy services in the project areas. Therefore, IEG rates the achievement of this objective before restructuring as Negligible. Rating Negligible PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective to increase access to modern energy services in peri-urban and rural areas in a sustainable and affordable manner. Revised Rationale Outputs Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) • Please see the outputs under Objective 1. • 15 PV water pumping systems were to be installed. Although all the systems were purchased under the project, none were installed because geophysical studies and drilling activities for the installation of these systems were not carried out. However, after project completion, FUNAE has completed the geophysical studies and tendered out the installation of water pumping systems. • 4,000 improved woodfuel stoves were distributed against a target of 4,000. • 6 solar water heaters were installed at hospitals achieving the target figure. Outcomes After the first restructuring in 2015, the scope of the project related to the second objective was changed, while the project objectives remained the same. The project activity of distributing improved charcoal kilns was cancelled. The target of woodfuel stoves was lowered to 4,000 from an original 50,000, although this was not included in the results framework as an indicator. However, upon the request of the Government of Mozambique, installation of six solar water heaters for six hospitals was included as a new project activity. Since the project development objective was not revised, it is not possible to argue that installation of a small number of improved woodfuel stoves and additional six solar water heaters in hospitals would suffice to achieve the project objective to increase access to modern energy services in project areas, even though these are not negligible project outputs. In other words, these project activities, due to their limited scope, are not consistent with the achievement of this objective. On the other hand, IEG recognizes the effort of the project team to revise the theory of change in order to overcome the budget constraints and to meet the GoM’s emergency demands. IEG rates the achievement of the second objective after restructuring as Modest. Revised Rating Modest PHOVRLEFFRATTBL Rationale The project was successful in increasing access to electricity in peri-urban and rural areas with some minor shortcomings. The changes made in the project design at the first restructuring were related to the second project objective to increase access to modern energy services. The achievement of this objective was negligible before the restructuring. After the scaling down of the project scope regarding the distribution of woodfuel stoves and cancellation of improved charcoal kilns at the project restructuring, the achievement of this objective was rated Modest. Overall, IEG rates the achievement of the project objectives barely Substantial. Overall Efficacy Rating Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Substantial 5. Efficiency A “with project” or “without project” economic analysis was not conducted at appraisal. The PAD (p.113) explains the reason as increasing access to electricity and modern energy services being “a stated national policy objective and a committed government program of action”. The economic analysis at appraisal focused on the increase in electricity access through grid expansion and use of solar PV in rural areas, but excluded the improved woodfuel stove and improved charcoal kiln activities. The PAD (p.120) argued that the positive impact of improved woodfuel stoves on gender, health and energy savings had already been proven in previous projects in both Mozambique and other countries. But the dissemination of improved woodfuel stoves being not financially viable was also acknowledged in the PAD. On the other hand, there is not a specific reason given in the PAD why the improved charcoal kilns were excluded from the economic analysis. However, the PAD (p.121) stated that improved charcoal kiln activities, which were modeled on earlier academic research programs, aimed to promote an improved and sustainable management of forest resources at the village level. It was expected that these activities would eventually lead to a national strategy for improved and controlled charcoal production in Mozambique. Contrary to these goals and expectations, improved woodfuel stoves activities were scaled down from a target of 50,000 to 4,000, and the improved charcoal activities were cancelled at the first project restructuring. At appraisal, the economic analysis covered 81 percent of the total project. However, economic internal rate of return (EIRR) and net present value (NPV) were calculated separately for Component 1 activities, excluding those related to institutional development, and for solar PV activities under Component 2. These calculations yielded and EIRR of 26 percent and an NPV of US$36.5 million for Component 1 activities. The EIRR and NPV for solar PV activities were 18 percent and US$1.75 million, respectively. The discount rate used for the NPV calculations at appraisal was 12 percent, however, the time horizon for Component 1 calculations was 20 years, and for Component 2, it was 15 years. At project closing, economic analysis covered 91 percent of the total project cost. The EIRRs calculated for the components yielded 25 percent for Component 1, and 8 percent for Component 2. Because of the addition of the installation of solar water heaters as a project activity under Component 3, an EIRR was also calculated for this component which yielded 68 percent. At project closing, a project level EIRR was calculated which yielded 34 percent. At appraisal, project level EIRR was not calculated. EIRR Comparison Appraisal Project Completion Component 1 26% 25% Component 2 18% 8% Component 3 N/A 68% Greenhouse gas emission and other emission impacts were included in the economic analysis at project completion. These were not included in the appraisal economic analysis. There were some other differences in the assumptions used in the economic analyses made at appraisal and project completion as follows: Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Component 1 1. At project completion, per kWh value of PV monthly fees was used as a proxy for electricity consumer benefits. At appraisal, this was assumed to be 40 percent of the sales value of Electricidade de Moçambique (EdM), the implementing agency of this component, in 2009, and increasing to 60 percent in 2022. The ICR (p.53) states that because of the difference in the actual tariff and sales, and that it is not clear how 40 percent and 60 percent of sales were calculated, the appraisal assumptions could not be used at project completion for calculating consumer benefits. 2 . At appraisal, it was assumed that the installation of Integrated Business Management System (IBMS) at EdM would result in a 5 percent reduction in annual total operating costs. The ICR reports that there was no evidence at project completion to support this assumption. 3 . When project was to be completed, reduction in non-technical and technical loss was assumed to be 5 percent at appraisal economic analysis. At the time of project completion, since the new and rehabilitated network was recently commissioned, there was no evidence for reduction in loss. The economic analysis at project completion assumed the reduction to be 1 percent. 4 . According to EdM’s annual report in 2015, the demand for electricity in the Component 1 area increased by 5 percent. However, project completion economic analysis assumed this to be zero, because (i) demand curve could not be calculated due to data constraints; (ii) energy efficiency improvement activities were expected to strengthen; and (iii) future exogenous shocks on demand cannot be controlled by the economic analysis. At appraisal, it was assumed that electricity demand would increase by 4 percent as of 2009, by 5 percent as of 2014 and by 6 percent as of 2016 (PAD, p.114). 5 . At appraisal the tariff in 2018 was estimated to be MZN 8 per kWh. At project closing, customers connected to the EdM network through the project were all residential consumers with lifeline tariff of MZN 1.07 per kWh. Component 2 1. For the solar PV benefits, the alternative was assumed to be off-grid electricity at appraisal. At project completion, on-grid supply cost estimated in EdM Master Plan was used as alternative for rural households. 2 . Although the appraisal economic analysis did not include activities related to improved woodfuel stoves, at project completion, the benefits from woodfuel stoves, such as fuel saving and labor, were included in the economic analysis. 3 . At appraisal, a 15-year of PV life time was assumed. At project completion, it was assumed that the PVs would last for 10 years. 4 . At appraisal it was assumed that PV customers would repay the PV capital cost. However, the only payment the PV customers make is for maintenance, not for repaying the capital cost. Given above major differences in the economic analysis assumptions and the calculations based on these assumptions, the project activities under Component 1 were delivered more efficiently than those of Component 2. An EIRR of 8 percent for Component 2 is below the hurdle rate of 12 percent used at appraisal. Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Financial Analysis A detailed financial analysis of EdM, the implementing agency of Component 1 was conducted at appraisal (PAD, pp. 126-140). It was concluded that EdM would not be financially sustainable due to the reasons unrelated to the project. The PAD (p.130) noted that the challenge EdM was to face in following years was its costs growing at a faster pace than its revenues. The financial analysis conducted at project closing using 1.3 percent as the financial opportunity of cost of capital also concluded that EdM had a negative financial net present value due to its ability to recover the costs of services. Furthermore, EdM failed to meet the financial covenants of current ratio and debt service coverage ratio, which were both set as 1.1. The implementing agency of Component 2, Fundo de Energia (FUNAE) is a public institution founded in 1997 and it is responsible for increasing rural electrification and promotion of alternative energy. FUNAE does not operate as a commercially operated utility. Since the project activities of Component 2 were to be financed by grants, it was not expected that the project would have any notable financial impact on FUNAE (PAD, p.141). The financial analysis conducted at project completion using 16 percent as the cost of capital resulted in a negative financial net present value for FUNAE. Financial internal rate of return was 13 percent. (The difference between the two institutions’ weighted average cost of capital rates comes from the funding type. The funds for Component 1 were on-lent to EdM from the Ministry of Finance at a rate of 1.3 percent and there was no capital contribution from EdM. Therefore, 1.3 was taken as the weighted average cost of capital. The funds given to FUNAE were grant. Therefore, 16 percent was estimated from the treasury bill of the Central Bank of Mozambique as the weighted average cost of capital for FUNAE.) Operational and Administrative Efficiency The project effectiveness was delayed for 13 months until the effectiveness conditions were met. After the effectiveness, it took another seven months for the disbursement of the initial deposit which was blocked until the outstanding funds from the first phase of the APL series were repaid. Problems in flow of funds arrangements and procurement further delayed the project implementation. As a result, the project closing date was extended for two years. There were significant changes to project design during implementation. Most importantly, the target of improved woodfuel stoves to be installed was lowered from 50,000 to 4,000. This activity would have scaled up the distribution of improved woodfuel stoves which had begun in the first phase of the APL series and was an important part of increasing or accelerating decentralized access to modern energy services in rural areas. Similarly, the cancellation of the of project activities under Component 3, such as the elaboration of a national “Rural Electrification Strategy and Investment Program” was a major setback. (However, it should be noted that this program, which is titled National Electrification Strategy, was later successfully developed under the SE4ALL initiative.) The development of a strategy to engage with private sector and a maintenance strategy for the PV systems were cancelled, as well. These activities were later financed by the funds provided by the Norwegian government. The main reason for these major project design changes was the constraint on the project budget caused by the appreciation of the USD against SDR an MZN. Upon the request of the Government of Mozambique, remaining limited funds were partly used for the refurbishing of additional substations to meet the rapid increase in electricity demand and the installation of solar water heaters in six Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) hospitals. Although the IEG acknowledges the difficulties the project was faced with due to the exchange rate fluctuations, and notes the importance of the additional substations works to improve the reliability of electricity supplied to the grid-connected customers and the provision of hot water in hospitals for health purposes, the project efficiency suffered from changes in design and implementation delays. Overall, because of significant shortcomings in the operational administrative efficiency and the low economic rate of return achieved in rural solar PV activities, the IEG rates the efficiency of the project as Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 91.00 ICR Estimate  34.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Although the project objectives did not change, the project scope did because some project activities especially those related to Components 2 and 3 were either reduced or cancelled. The Relevance of Project Objectives is rated Substantial. Under split rating, the achievement of project objectives is rated Modest before the first restructuring when 44.9 percent of the project funds was disbursed. Since the Efficiency is also rated Modest, the Outcome before restructuring is rated Moderately Unsatisfactory. After the restructuring, the achievement of project objectives is rated Substantial. Since there was no change in other ratings, the Outcome rating after restructuring is Moderately Satisfactory. Per ICRR Guidelines, the Outcome of the project is rated as Moderately Satisfactory, but barely so. The calculation is given below. Before first After first restructuring restructuring Relevance of Objectives Substantial Efficacy Objective 1: To increase access to electricity Substantial Substantial Objective 2: To increase access to modern energy Negligible Modest services Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Overall Efficacy Modest Substantial Efficiency Modest Outcome Rating Moderately Moderately Satisfactory Unsatisfactory Outcome Rating Value 3 4 Amount Disbursed (US$ million) 32.17 39.53 Disbursement (%) 44.9% 55.1% Weight Value 1.35 2.20 Total weights 3.55 (rounds up to 4) Overall Outcome Rating Moderately Satisfactory (4.0) The ICR rated the Relevance of Objectives as High, the achievement of project objectives as Substantial and the Efficiency as Substantial. Per ICRR Guidelines, the Outcome rating in the ICR should have read Satisfactory, but this rating was erroneously recorded as Moderately Satisfactory. Therefore, the IEG downgrades the Outcome rating of this project from Satisfactory to Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The financial viability of the grid operator EdM stands out as a major risk to the development outcome. As explained in the financial analysis section under the Efficiency section, EdM’s revenues are not high enough to cover its costs. This would constrain EdM’s ability to continue with grid expansion and new customer connections, for which the utility operator relied on funds from various development partners. Technically, EdM has the capacity to operate and maintain the new lines and other transmission and distribution system upgrades, however for the sustainability of these services, the financial viability of EdM has to be improved. Affordability of off-grid electricity by rural households poses a threat for the sustainability of project outcomes. According to SE4ALL multi-tier access framework, affordability of off-grid access at the third tier is defined as household electricity consumption of 365kWh per year and less than 5 percent of household income spent on electricity. This is the level where duration of electricity is minimum eight hours during the day and four hours during the evening. In the off-grid project areas, electricity consumption per household is 173kWh per year, which is less than half of the consumption level defined in Tier 3, but the cost of this lower is 5.8 per cent of household income, which is higher than the Tier 3 criterion. Adequate arrangements are in place for the near term sustainability of PV system O&M services, but it is not certain whether these arrangements could be maintained during the lifetime of PV systems. According to the information provided in the ICR (p.17), FUNAE contracted third party entities to train technician who will be in charge of the O&M of the installed PV systems. Part of the O&M costs are covered by the fees paid by the customers. It is not clear what these fees amount to in covering the overall O&M costs. Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Continued strengthening of the local communities’ capacity in solar PV of these systems and establishment of systems standards should contribute to the sustainability of O&M services. 8. Assessment of Bank Performance a. Quality-at-Entry The project was highly relevant to the country conditions in Mozambique regarding poverty reduction, gender and social development. It was also strategically relevant to the priorities of the Government of Mozambique which aimed to increase electricity access to 15 percent by 2019. The project benefited from the experience gained during the implementation of the first phase of the APL series and lessons learned in other similar bank projects (PAD, p.14). The project was designed to achieve the development objectives through proven interventions, such as the distribution of solar PV systems for lighting in rural households where extension of grid was not economically or technically feasible. Improved woodfuel stoves were to increase access to modern energy services through the efficient use of firewood and prevention of indoor smoke. Electrification of schools and health centers in rural areas was to directly contribute to the achievement of the project objectives. The technical assistance under Component 3 was to support the government in the preparation of a national strategy which would set the framework for further expansion of rural electrification. Rehabilitation and extension of the grid network was directly linked to increasing access to electricity. Installation of integrated business management system would strengthen the institutional capacity of EdM, the electricity system operator. Therefore, the relevance of project activities to the objectives was high and there was a clear causal chain among project inputs, activities, outcomes and the achievement of project development objectives. Technical assistance was to be given to strengthen the institutional capacity of the project implementing agencies in environmental conflict identification, analysis and management. The Environmental and Social Management Framework, and Resettlement Policy Framework were disclosed in October 2009 prior to the Board approval of the project. Establishment of a Social and Environmental Safeguards Unit at EdM was included in the financing agreement as a condition for credit effectiveness. A detailed risk assessment was provided in Annex 6 of the PAD. However, there were some moderate shortcomings at appraisal. The economic analysis included some assumptions which could not be replicated at project closing. (Please see the discussion under Section 5. Efficiency.) Costs of PV systems for staff housing of the schools and health centers in rural areas and the two-year maintenance program for rural solar PV systems were not included in the economic analysis. The total cost of these items would be around US$4.75 million which caused a budget shortfall. The estimated budget shortfall raised to US$6.35 million because of the taxes and duties applicable to imported solar PV systems, which were not taken into account during appraisal. The flow of funds arrangements related to EdM resulted in implementation delays until a designated account for EdM was established at the first restructuring (ICR, p.24). The M&E design had some erroneous baseline and target figures which were later corrected at the first restructuring. The possibility of project implementation delay due to outstanding funds from Phase 1 should have been captured at the appraisal stage. Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Overall, IEG rates Quality-at-Entry as Moderately Satisfactory due to moderate shortcomings. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision There were four task team leader (TTL) changes during the project implementation period of seven years. Supervision missions were regularly held. Thirteen Implementation Status Reports (ISR) were prepared. The mid-term review was held in March 2014. This formed the basis of the project restructuring in February 2015, almost one year later than the mid-term review. The Bank’s supervision of fiduciary and safeguard aspects of the project was adequate. However, there were some moderate shortcomings in Bank’s supervision of the project. Faced with budget constraints, the project team’s focus shifted away from the development impact of the project, mostly related to the project activities of Components 2 and 3. Without a revision of the project development objective, some important project activities, such as the distribution of improved woodfuel stoves and technical assistance in the elaboration of a national rural electrification strategy, were substantially scaled down or cancelled. Project design was changed materially as a result of the changed priorities of the Government of Mozambique. Reporting requirements defined in the financing agreement were relaxed which, according to the ICR (p.35), was caused by regular interactions of the TTL stationed in Mozambique with the staff of implementing agencies. As a result, “the formal M&E and the regular project status reporting requirements were replaced with monitoring data collection and preparation of implementation status summaries per component during missions” (ICR, p.35). Although the EdM was persistently encouraged to comply with the covenants defined in the financing agreement, this was not enforced. The lack of a designated account for EdM should have been identified and addressed early on rather than at the mid-term review. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Although clearly defined, the project objective “to increase access to electricity and modern energy services in peri-urban and rural areas in a sustainable and affordable manner” were rather complex for monitoring and evaluation (E&M) purposes. First, the PAD did not include a definition of “modern energy services”. Second, the project did not include any activities to achieve “sustainability and affordability”. Although the project components were designed to achieve the project objectives to increase access to electricity and modern energy services, as defined by the International Energy Agency, the outcome of the project activities related to Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) the distribution of improved woodfuel stoves and charcoal kilns were not adequately reflected in the results framework or results monitoring (PAD, pp.40-43). The indicators were mostly specific, relevant, measurable and time-bound, but they were output oriented. They were designed to measure the achievements in electricity access increase in terms of household, school and health center connections, and construction of transmission and distribution lines. Monitoring of the project activities related to the refurbishment of substations were not included in the M&E design, nor was the achievement of sustainability and affordability of access to electricity and modern energy services. There was no indicator measuring the distribution of improved woodfuel stoves and charcoal kilns, either. Monitoring of electricity losses and average interruption frequency in the project area, which are related to efficiency and reliability of power supply, respectively, were not relevant to monitor the achievement of the project objectives. Each project implementing agency would compile M&E data separately and the Ministry of Mineral Resources and Energy (MIREME) would be responsible for integrating these M&E reports and submit a consolidated report to the Bank together with the quarterly progress reports. As explained below and in Section 8.b. Quality of Supervision, these requirements were relaxed because of the presence of the Bank’s task team leader. b. M&E Implementation The project was restructured in February 2015 and, as explained in Section 2. Project Objectives and Components, some project activities were cancelled or scaled down without a revision in the project objective. The weaknesses of the M&E design, explained in the above section, were not corrected during implementation, except the correction in the target values of the project development objective indicators, which were defined as number of household connections whereas they should have read as number of people connected to electricity. On the other hand, the target value of the indicator related to the construction of transmission line should have been corrected as 6 km, but this correction was not carried out (ICR, p.28). The M&E reporting procedures were not implemented according to the provisions in the financing agreement. Rather than MIREME consolidating and reposting the M&E data to the Bank, each implementing agency provided an update about project progress directly to the Bank during the supervision missions. The focus of M&E was solely on the increase in electricity access. c. M&E Utilization The findings of the Midterm Review were utilized to restructure the project in February 2015. Some indicators were deleted at that restructuring because of the cancellation of some project activities caused by budget constraints and changes in the priorities of the GoM, rather than the findings of M&E (Please see Section 2. Project Objectives and Components). There is no evidence in the ICR whether additional financing was considered instead of the cancellation of some important project activities, such as the distribution of improved woodfuel stoves and charcoal kilns, and technical assistance in policy design and institutional capacity strengthening. The M&E data were utilized to monitor the progress in implementation of project activities in increasing electricity access, rather than the achievement of project objectives, including sustainability and affordability. However, the project team informed that FUNAE, the implementing agency of Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) Component 2, conducted a survey regarding the willingness-to-pay level of the rural customers for electricity after which the electricity tariff for these customers were adjusted according to the findings of the survey. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Category B under OP/BP 4.01 (Environmental Assessment), and OP/BP 4.12 (Involuntary Resettlement) was triggered. Environmental Assessment OP/BP 4.01: At appraisal, potential adverse environmental and social impacts were defined as workers’ safety, loss of vegetation, disposal of used batteries of PV systems, construction waste and loss of livelihoods (PAD, p.29). The final version of the Environmental and Social Management Framework (ESMP) was disclosed on October 16, 2009 in Mozambique and the World Bank Infoshop. Establishment of an environment unit at EdM was included as a condition for effectiveness in the financing agreement. This unit supervised the implementation of ESMP. At times, the compliance of the contractors with the safeguards policies related to workers health, sanitation and safety, and environment could not be monitored by the EdM’s safeguards unit due to insufficient number of staff (ICR, p.30). With the help of the Bank, an action plan was prepared to remedy these shortcomings. The project was compliant with OP/BP 4.01. Involuntary Resettlement OP/BP 4.12: This policy was triggered due to the potential adverse social impacts related to physical displacements, land acquisition and/or damage to assets and means of production (PAD, p.145). Although such impacts were expected to be minimal, a Resettlement Policy Framework was prepared and disclosed in Mozambique and the World Bank Infoshop, because the construction of medium voltage lines were not known at appraisal. During implementation only one project activity required involuntary resettlement. Nine households were affected by this project activity. A site specific Resettlement Action Plan was prepared and the site was vacated after all project affected people were compensated. The project was found compliant with OP/BP 4.12. b. Fiduciary Compliance Financial Management At appraisal, the proposed financial management (FM) arrangements for the project were found to be satisfactory, with moderate residual FM risk rating (PAD, p.26). The financial management of the project was to be the responsibility of the implementing units overseen by Ministry of Mineral Resources and Energy. During project implementation, there were no major financial management issues. All three project Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) implementing agencies were compliant with the Bank’s policies and procedures. In each project implementing agency, qualified financial management specialists managed the project’s accounting, financial reporting and disbursement issues (ICR, p.31). Interim financial reports and annual audit reports were submitted regularly and they were unqualified. Although no Borrower contribution was planned for the financing of the project, the project implementing agency of Component 2, FUNAE, financed US$439,279 for additional PV systems (ICR, p.51). EdM failed to comply with the financial covenants stipulated in the financing agreement. Until the establishment of a separate designated account for EdM at the first project restructuring in February 2015, EdM had encountered problems with the flow-of-funds, since EdM was expected to pay its invoices first and be reimbursed later by the MIREME. This led to delays in project implementation (ICR, p.24). Procurement Project implementing agencies were compliant with the Bank’s guidelines in procurement. Key procurement activities were completed successfully. No-objections were issued by the Bank’s project team on a timely basis. However, the long time required for contract approval by the Commission for External Economic Relations, which is chaired by the prime minister and approves contracts above US$1 million, resulted in project implementation delays since some contractors could not be given their advances to start work (ICR, p.31). c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Moderately Moderately Bank Performance --- Satisfactory Satisfactory M&E had significant shortcomings in its design. Focus was on monitoring the achievement in increasing Quality of M&E Substantial Modest access to electricity and less on modern energy services. Some project activities were not monitored. M&E findings Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) were not sufficiently used to steer the direction of the project to achieve project objectives. Quality of ICR Substantial --- 12. Lessons Three lessons are derived from the ICR by IEG. Project objectives which are not clearly defined can be overlooked during project implementation. The project objective “to increase access to modern energy services” was not defined in the PAD or the ICR. The project’s focus shifted on increasing electricity access and moved away from the distribution of woodfuel stoves, which is the other component of modern energy services, according to the definition by the International Energy Agency. At the first project restructuring, while the project activities related to the woodfuel stoves were substantially decreased, the project development objective was not revised nor the M&E design was changed to monitor the achievement of project objective to increase modern energy services. A lack of understanding of governments’ large-contract clearance procedures can adversely affect project implementation. As explained in Section 10.b. Fiduciary Compliance, advance payments to some contractors could not be made on time because of the lengthy approval process of large contracts by the Commission for External Economic Relations, which is chaired by the prime minister. As the ICR (p.38) states, assessment of such clearance procedures at appraisal would help design more realistic procurement plans and implementation schedules. Establishment of designated accounts for project implementing agencies, especially for those which have state-owned economic enterprise status, can facilitate an efficient flow-of-funds and disbursement system. The use of government accounts and systems required EdM to pay the invoices first and to be reimbursed by the MIREME later. This caused delays in project implementation until a designated account was established for EdM at the first restructuring. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is candid in critically assessing the shortcomings in project implementation and achievement of the project objectives. It is consistent internally. The analysis is based on evidence. The lessons learned appropriately respond to the specific experiences and findings for the project. Economic analysis is detailed. Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Energy Dev. & Access Project (APL-2) (P108444) However, efficacy section focuses only on the increase in electricity access avoiding sufficient discussion of the achievement of the project objective to increase access to modern energy services. The section on quality at entry could have benefited from a detailed discussion. The ICR fails to apply a split-rating which was necessitated due to the material changes made in the project components at the first restructuring. The ICR also erroneously rates the outcome of the project Moderately Satisfactory, whereas it should have read Satisfactory given that the Relevance of Objectives was rated High, Efficacy and Efficiency were rated Substantial. a. Quality of ICR Rating Substantial Page 22 of 22