Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) Report Number: ICRR0023455 1. Project Data Project ID Project Name P160658 Rural Electrification Project Stage II Country Practice Area(Lead) Vanuatu Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-60720,IDA-D1930,TF-A4979,TF- 30-Jun-2022 2,021,924.21 A5406 Bank Approval Date Closing Date (Actual) 31-May-2017 30-Jun-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 14,220,000.00 10,220,000.00 Revised Commitment 14,220,000.00 1,528,550.33 Actual 2,279,386.91 1,528,550.33 Prepared by Reviewed by ICR Review Coordinator Group Joel J. Maweni Dileep M. Wagle Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objectives (PDOs), as stated in the Financing Agreement dated July 23, 2017 (Schedule 1), were to "support increased penetration of renewable energy and increased access to affordable electricity services in the dispersed off-grid areas of Vanuatu.“ Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) The statement of the PDOs in the Project Appraisal Document (PAD) dated May 9, 2017 (p.17) was identical to that in the Financing Agreement except for the inclusion, in the Financing Agreement, of the word “affordable” to describe electricity services in the dispersed off-grid areas of Vanuatu. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The Vanuatu Rural Electrification Project Stage II (VREP-II) comprised three components as follows: Component 1: Provision of Solar Home Systems (SHS) and Micro Grid Systems (Estimated cost at appraisal - U$5.37 million, Actual cost - US$0.3 million). SHS and micro grids were to be provided in off- grid rural areas of Vanuatu by private sector vendors through a demand driven approach. The component targeted 37 public institutions and about 8,400 households for a total coverage of about 42,000 people. Component 2: Construction of Mini Grid Systems (Estimated cost at appraisal - U$6.8 million, Actual cost - US$0.0 million). This component was to support provision of electricity services to about 550 households or 2,750 people, and to public institutions and businesses. Up to five mini grids were to be selected from communities meeting the assessed criteria for financial viability - i.e., those comprising at least 75 households and with more than 50% of their total load accounted for by businesses and public institutions. Component 3:Technical Assistance and Project Management (Estimated cost at appraisal - U$2.0 million, Actual cost - US$1.7 million). The technical assistance (TA) was to: (a) support vendor and product registration processes for implementation of Component 1; (b) hiring of an owner's engineer to help with the feasibility studies, design, procurement and construction supervision of the mini grids under component 2, including the preparation and monitoring of safeguards management plans; and (c) provide project implementation support to the Department of Energy (DoE) of the Ministry of Climate Change and Natural Disaster which had overall responsibility for implementing the project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Total project cost at completion was US$2.00 million (ICR, Annex 3) compared to US$14.17 million estimated at appraisal (PAD, p. 21). The financing plan at appraisal envisaged that US$14.17 million of the total cost was to be financed by an International Development Association (IDA) Credit and Grant of US$4 million, a Scaling up Renewable Energy in Low Income Countries Program (SREP) Grant of US$6.77 million and a Pacific Region Infrastructure Facility (PRIF) Grant of US$ 3.45 million. The Borrower was expected to provide a US$1.5 million “in-kind” contribution to cover Vanuatu’s project related costs such as organization of land acquisition for the project and other inter-departmental project coordination costs (PAD, p.20). Due to the slow uptake of the SHS and micro grids (Component 1) and non-implementation of the mini grids (Component 2) only US$2.28 million or about 16% of the total allocated financing was disbursed as follows: IDA Credit and Grant (US$0.68 million), SREP Grant (US$0.58 million) and PRIF Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) Grant (US$1.02 million).The amount disbursed out of the PRIF Grant was higher than that of other financing sources because an attempt was made to prioritize its usage as it initially had an earlier closing date. There is a US$0.28 million discrepancy between the total amount of disbursements as shown in the Data Sheet (portal data) and the amount shown in Annex 3 of the ICR which is primarily due to exchange rate differences between the financing currencies and the US$. Dates The project was approved on May 31, 2017, and it became effective on August 21, 2017. The project’s original closing date was June 30, 2022. However, the PRIF grant was scheduled to close earlier on June 30, 2020. The project was restructured twice. The first restructuring on October 4, 2018, was intended to give Vanuatu access to additional financing for renewable energy investments under the Sustainable Energy Financing Project (P098423) - a regional project that was initially financing renewable energy in Fiji. Delays in reaching agreement on implementation arrangements with Fiji, difficulties in concluding legal agreements with participating financial institutions and the loss of interest by key financial institutions prevented the use of this financing in Vanuatu. The second restructuring on March 20, 2020, extended the closing date of the PRIF grant by two years to June 30, 2022, to align it with that of the overall project. A mid-term review (MTR) was completed on May 18, 2021. The MTR proposed a redesign of component 1 and extension of the project closing date by two years because of problems that had been encountered with payment of ineligible subsidies under the Vanuatu Rural Electrification Project Stage I (VREP-I) (P150908) and because of slow demand. The proposed redesign concept envisaged replacement of the vendor delivery model under component 1 with a procurement model for public institutions. The vendor model was terminated by the end of 2021, however, attempts to restructure the project were abandoned due to differences between the Bank and the Government on the procurement rules to apply to the redesigned component, and particularly to the mini grids component. The project closed on June 30, 2022, as scheduled. 3. Relevance of Objectives Rationale Country context: Vanuatu is an archipelago in the South Pacific Ocean consisting of 83 relatively small, geologically newer islands of volcanic origin (65 of them inhabited). The population of about 258,000 in 2017 was almost evenly distributed among the six administrative provinces. The most recent population estimate was 319 000 in 2021. Country strategies. Recognizing the key role of energy as a driver of the country’s social and economic development the Government of Vanuatu (GoV) developed the National Energy Roadmap (NERM, 2013- 2030). This strategic document contained the following five key priorities to guide the development of the energy sector: (a) Access to secure, reliable, and affordable electricity for all citizens by 2030; (b) Petroleum supply - secure, reliable and affordable petroleum supply throughout Vanuatu; (c) Affordability through lower cost energy; (d) Energy security at all times; and (e) Climate change – mitigation of climate change through renewable energy and energy efficiency. The NERM set targets for electricity generation from renewable energy of 65% and 100% by 2020 and 2030 respectively, including 14% of electricity from Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) biofuels. A 100% target for electrification of public institutions in off‐grid areas by 2030 was also established. Sector context: Electricity was being provided by independent main grids in larger urban areas, isolated mini grids in areas with lower population concentrations and decentralized systems. Two private sector concessionaires generated and supplied electricity in four concession areas (Efate, Tanna, Maleluka and Luganville). Installed generation capacity in the grid networks of Port Vila, Luganville, and parts of Tanna and Maleluka was about 32 MW and was inadequate to meet national power supply needs. The critical sector issues which were considered in the development of the NERM were: (a) the inadequacy of the grid installed generation capacity to meet the national power needs; (b) the low level of electricity generation from renewable energy sources (29%); (c) like in most Pacific Islands Countries, the high dependency on costly imported diesel and other fossil fuels; and (d) the low level of access to electricity at 30% overall with much lower levels of access in rural areas. Lower population densities in rural areas, remoteness and large distances between customers, low consumer loads and low incomes limited the scope for building new distribution lines or extending existing ones to serve new customers. The NERM, as updated in 2016, incorporated green growth objectives for catalyzing sustainable development, including promotion of renewable energy in the country’s main economic sectors (ICR, p.13). This Project was designed to support increased penetration of renewable energy and scaling up of access to electricity by 15% (PAD, Figure 1, p.15) in Vanuatu’s off-grid areas using SHS, mini and micro-grids based on hydro, solar and biofuels, and was, thus, consistent with both the key sector issues and NERM objectives. Alignment with World Bank Strategies: The design of the project was informed by the Regional Partnership Framework (RPF) for nine Pacific Island States (2017-2021) including Vanuatu. Although the report was not formally approved by the Board at the time of project approval it was sufficiently advanced to provide a strategic basis for the project’s design. The RPF was subsequently approved in October 2017. The RPF’s areas of focus were: (a) full exploitation of available economic opportunities; (b) enhancement of employment opportunities; (c) protection of incomes and livelihoods; and (d) strengthening of the economic growth enablers of macroeconomic management, infrastructure, and closing knowledge gaps. The project supported the RPF’s focus area (d) on strengthening the economic growth enablers of …infrastructure and …” Specifically, the RPF emphasized increasing electricity supply and efficiency, and setting the stage for increasing renewable energy development. At closure on June 30, 2022, the project remained aligned with the RPF (2017-2021) which had not been replaced or updated. The project was not only consistent with the Bank’s RPF, but also supported: (a) the World Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity; (b) the EAP flagship report - One goal two paths and sustainable development goal #7 on ensuring access to reliable, affordable, and sustainable energy for all and SDG #5 on achieving gender equality and empowering all women and girls. This project was part of an overall World Bank engagement plan for supporting Vanuatu’s NERM objectives of increasing access to electricity and improving the role of renewable energy in the generation and provision of electricity services. The plan included two projects which were ongoing at project appraisal, namely; (a) the Improved Access to Electricity Project (P125604) financed by the Global Partnership for Output-based Aid (GPOBA) to support intensification of connections within the concession areas with a target of increasing the overall connection rate by 10%; and (b) the Vanuatu Rural Electrification Project Stage I (VREP-I) which was financing scaling up access to electricity for 17,500 households (an access rate increase of 30%) through “plug and play” systems and for aid posts and communities in the off-grid space. This project (VREP-II) aimed to increase access by an additional 15% using mostly renewable resources, SHS, micro and mini grids and included electrification of schools, health care centers, dispensaries, aid posts and other institutions providing services to communities. Both VREP-I and VREP-II were also Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) designed to support the Scaling up Renewable Energy Program Investment Plan prepared by the GoV in 2013 and approved by the SREP sub-committee in November 2014. Thus, VREP-II represented a logical progression and sequencing of Bank support to Vanuatu and was reasonable at the strategic level. The relevance of project objectives is rated Substantial because: (a) the project was consistent with the GoV’s energy objectives as articulated in the NERM (2013-2030) (although it did not explicitly reference renewable energy) and SREP Investment Plan (2014) and addressed key sector issues facing Vanuatu in the energy sector at appraisal; (b) remained well aligned with the RPF (2017-2021) at project closure, and with other regional and international development strategies for the energy sector; and (c) was part of a coherent World Bank strategic framework for assisting Vanuatu in realizing its energy sector objectives. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective "....to support the increased penetration of renewable energy in the dispersed off-grid areas of Vanuatu." Rationale Theory of Change The project had the following two objectives: 1. PDO 1: to support increased penetration of renewable energy in the dispersed off-grid areas of Vanuatu; and 2. PDO2: to support increased access to affordable electricity services in the dispersed off-grid areas of Vanuatu. Figure 1: Theory of Change diagram of the VREP-II on page 8 of the ICR presents a credible theory of change (TOC) for the project. The project activities were to provide electricity to off-grid households, public institutions and businesses using primarily domestic renewable energy resources (solar, hydro, biofuels, etc.). Component 1 involved the provision of SHS and micro-grids by vendors under a subsidization model intended to ensure affordability of the retail costs to consumers. Under Component 2 mini grids were to be constructed for five villages to provide electricity to households, public institutions, and businesses. Component 3 provided technical assistance to support operation of the vendor model, under component 1, and hiring of an owner’s engineer to help with preparation, design, and construction supervision of mini grids. These project activities were expected to lead to the achievement of the project’s objectives. The TOC includes several outputs and intermediate indicators and three outcome indicators. The outcome indicators for the first objective on penetration of renewable energy were identified as MW capacity of RE additions Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) (target of 4.5MW) and annual renewable energy generation (target of 2.7 GWh annually) (ICR pp. 8 and 9).The outcome indicator for the access objective was the number of people provided with new electricity services under the project (target of 44,750). In the long term the increased availability of electricity would support improvements in living standards, contribute to the Bank’s twin goals, facilitate better provision of public services, help improve profitability of businesses and contribute to the global climate change agenda. Overall, the TOC is sound with clear causal relationships between activities, outputs, and outcomes but there were some shortcomings. First, the project’s midterm review (2021) subsequently found out that only relatively well to do households could afford the size of SHS provided under component 1 and that these households already had electricity. Thus, the SHS would not help to increase access but only to improve service to those who already had it. Further, VREP-I had helped to substantially increase access such that Vanuatu had almost achieved 100% electrification with only the remotest areas remaining unelectrified. These were the areas in which the size of SHS supported by VREP-II were most unsuited due to affordability constraints. The second shortcoming of the TOC is the use of renewable energy generation capacity and annual energy generation as indicators for increased energy penetration. The ICR recognized that the share of RE in the total generation capacity and annual energy generation would have been better indicators (ICR p. 14). However, the non-comparability of SHS to grid supplied power and the lack of adequate information on non-project RE activities in the sector means that the indicators used are second best, but practical. Outputs and intermediate outcomes The same activities and outputs were expected to contribute to the achievement of both project objectives. The actual outputs compared to expectations were as follows: 1. 370 SHS/micro grids were installed compared to a target of 8,400. 2. No mini grids were constructed under the project compared to the target of 5, 3. 104 Public institutions received a SHS, or micro grid compared to a target of 37. 4. Zero Number of people connected to electricity through a mini grid constructed under the project compared to a target of 2,750. 5. Direct project beneficiaries of which 50% are female compared to a target of 49%. 6. Total households of which 11.8% are female headed households received a SHS, micro grid or received electricity through a mini grid compared to a target of 13%. 7. Zero Participants in consultations during project implementation activities compared to a target of 2,000. 8. Zero Participants in consultations during project implementation activities – female compared to a target of 980. 9. 100% Grievances registered related to project benefits resolved compared to a target of 100%. The completion and delivery of SHS, micro grids and mini grids was critical for the project to achieve its objectives. The level of SHS sales was only 370 out of 8,400 or about 4% and none of the mini grids were constructed. As shown above there was substantial underperformance on most of the intermediate indicators. The ICR evaluation is based on the three outcome indicators described above with additional evidence derived from two out of nine intermediate indicators. Data on the balance of the intermediate indicators is Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) provided in the results framework. The three outcome indicators and the two intermediate indicators are adequate for the evaluation of the project’s outcome. The vendor model for the SHS was plagued by low demand because of several factors, including the small number of vendors that showed interest, the absence of attractive payment terms combined with the lower subsidy level of 33% compared to 50% under VREP-I, and disagreements between the GoV and the Bank over the subsidy arrangements under VREP-I. In addition, the GoV expanded VREP-I to include potential customers in the concession areas who were unlikely to be connected to the grid soon which may have reduced some of the demand for VREP-II. By December 2021, a decision was made to abandon the vendor model and transition to a public procurement approach for SHS and micro grids for public institutions. The mini grids component had faced difficulties with a two-year delay in the engagement of the owner’s engineer. The GOV’s subsequent decision to use national procurement procedures for the mini grids could not be agreed to by the Bank. The proposal to restructure the project was then abandoned and the project closed on schedule on June 30, 2022. Outcomes The limited project implementation resulted in very low achievements of outcome indicator targets as follows: 1. 0.29 MW of new RE capacity was constructed compared to a target of 4.5 MW. 2. 0.32 GWh of annual renewable energy generated compared to a target of 2.7 GWh. The objective of increasing the penetration of renewable energy was not achieved because only about 6 percent of the RE capacity target was installed through SHS/micro grids. None of the 5 mini grids were installed. Thus, the corresponding annual energy generation was a meagre 0.32GWh or about 11.8 % of the targeted 2.7 GWh. Rating Negligible OBJECTIVE 2 Objective "..to support the increased access to affordable electricity services in the dispersed off-grid areas of Vanuatu." Rationale Objective 2 shares the same TOC, outputs and intermediate outcomes as described under Objective 1 above but had a separate outcome indicator – the number of people who received electricity service under the project. Outcomes Like Objective 1 the low level of project implementation meant that a small number of people received new electricity services under the project, i.e., only 3,625 people compared to the target of 44,750. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) Rating Negligible OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The implementation of activities designed to support the achievement of both PDO1 and PDO2 was negligible and, hence, the Negligible overall efficacy rating. The SHS were to be installed through a vendor model, but the uptake was extremely low due to consumer affordability and financing constraints for both vendors and consumers. The level of subsidy provided to the project retail costs of SHS (at 33% and less than 50% under VREP-I) was not adequate to improve consumer affordability and neither vendor nor microfinancing was available. The mini grids component suffered initially from delays in the engagement of an owner’s engineer who was to support feasibility studies, design, and procurement of the schemes. Before bidding could be undertaken the GoV decided to use national procurement procedures instead of the procedures agreed under the Financing Agreement (Schedule 2, C (3). Since the Bank could not agree to the GoV’s proposal the component was not implemented and, hence, no RE capacity or annual energy generation resulted from it. Overall Efficacy Rating Primary Reason Negligible Low achievement 5. Efficiency The economic analysis (levelized economic cost of energy) conducted at appraisal showed that SHS/micro grids were the least cost solution for most communities. The economic analysis was supplemented by an affordability assessment which showed that systems up to 1kWh/day (about 450-watt peak or maximum electrical capacity supplied under ideal temperature and sunlight conditions) would be affordable for most households assuming a 33% subsidy and payment of outlays over 4 years and a discount rate of 10%. The key factor driving affordability was the availability of payment plans to spread the household cost over 4 years. This was demonstrated during the MTR of VREP-I. This type of payment plan was not available under VRP-II and, hence, the ICR did not apply the appraisal stage methodology for ex post efficiency evaluation of the project. The VREP-II costs without the 4-year payment plans were assessed to be considerably more expensive and unaffordable, hence, few SHS were sold. Instead, the ICR team used a unit cost methodology to compare the VREP-II cost per Wp of installed RE capacity to that of VREP-I and the sector more generally. This showed a cost of Wp installed capacity for VREP- II of US$7.7 compared to US$8.0 for VREP-I. The VREP-I cost was assessed as reasonable by the project’s ICR. For the mini grids, the economic and financial analysis at appraisal had supported the conclusion that 100% of the capital costs needed to be fully subsidized and the tariff would need to be set at the base level of an existing Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) grid service to cover the O&M costs. As these parameters would vary with the type of service this delayed preparation of the TOR amid considerable discussions among the DoE, the regulator and service providers. Eventually the mini grids were not implemented following the disagreement between the Bank and the GoV regarding the procurement procedures to be applied. Given that this component was not implemented, the ICR had no basis for an ex post economic analysis of this component. The protracted focus of both the Bank and GoV teams on resolving the issue of ineligible subsidies under VREP-I constrained the time given to managing implementation of both VREP-I and VREP II. Covid-19 also impacted administrative efficiency through restriction of travel for Bank staff and the absence of the project manager who was stranded in his home country for more than a year during which period he was not able to provide project management support. The ICR analysis concluded that on a unit cost approach VREP-II is somewhat below the norm that would be expected in the sector and that together with administrative efficiency issues leads to a Modest rating for efficiency. IEG agrees with this rating. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s outcome is rated Unsatisfactory based on a Substantial rating for the relevance of Objectives, a Negligible rating for efficacy, and a Modest rating for Efficiency. The outcome rating is weighed down heavily by the very low level of implementation of activities and delivery of outputs and, therefore, the major shortfalls in achieving expected outcomes. a. Outcome Rating Unsatisfactory Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) 7. Risk to Development Outcome The risks to development outcomes are not significant since only a small part of the project was completed. The relevant risks pertain to inadequate operation and maintenance of the 104 SHS and micros grids for public institutions and the 370 SHS. 8. Assessment of Bank Performance a. Quality-at-Entry The were some significant shortfalls in the project’s quality at entry. VREP-II was approved about three years into the implementation of VREP-I when lessons were starting to emerge which had important relevance for the design of the new operation, especially regarding subsidies and vendor and consumer financing constraints. A better phasing of the VREP Program would have allowed enough time for lessons from VREP-I to be used to improve the design of VREP-II, particularly in the areas of project management, verification of SHS and the identification of ineligible subsidies. Despite these challenges VREP-I exceeded its target for sales of solar PV systems for homes and reached about 93% of its targeted beneficiaries. Efficacy was assigned a substantial rating which together with a substantial rating for relevance of objectives and a modest rating for efficiency resulted in an outcome rating of Moderately Satisfactory. The high achievement on sales of SHS was possible, despite the issue of ineligible subsidies, because of the higher subsidy level (50%) on relatively smaller scale systems than for VREP-II which made the systems more affordable, and the availability of some vendor financing (the rate of uptake increased substantially after one large vendor who provided financing was registered) (ICR p. 17). The economic analysis of the project suggested that the higher cost SHS would be less affordable and yet a lower subsidy (33% compared to 50% under VREP-I) was built into the project design. Although 33% was an “initial” subsidy level it was never revisited during implementation even when demand did not pick up. Increasing the subsidy was considered for public institutions solar PV systems in the context of the MTR and the restructuring of the project but was not implemented, since the restructuring did not proceed. DOE’s portfolio of Bank-supported projects was increasing, and VREP-II was to become the largest project it had ever managed. While DoE recruited more staff and the project provided technical assistance to help with management of vendor and product registration support, preparation design and construction supervision of mini grids, the critical issue of an independent verification agent for subsidies was not pursued leading to problems of ineligible subsidy expenditures during implementation and to their retrospective reassessment. The key design flaws for VREP-II were: (a) the overestimation of demand for the larger scale solar home systems that was not based on any analytical studies or market consultations; (b) the assumptions that consumer financing would be available for the non-subsidized portion of the retail cost of SHS from vendor financing or microfinancing; (c) the establishment of the level of subsidy of the retail cost of SHS Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) at 33% compared to 50% for the smaller SHS under VREP-I; and (d) failure to insist on an independent verification agent for SHS.. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision Bank supervision missions were conducted about twice a year and ISRs were updated on a regular basis to inform management on project status. The ICR reports several issues with the quality of supervision, including the taxing burden on both the Bank team and the DoE staff of dealing with the problem of ineligible subsidy expenditures, the understaffing of the Bank team at times, the excessive time taken to complete the MTR (more than 12 months) which was finally completed after a new task team had taken over. The MTR started in February 2020 and was completed in May-July 2021. The lengthy time taken to finalize it was due to several reasons, including the efforts to resolve the issues of ineligible expenditures under VREP-I, a change in World Bank Task Team Leaders, and the need to process newly received information regarding the mini grids component (ICR p. 15). In addition, the MTR was conducted during Covid-19 period when the Bank’s implementation support was being delivered primarily on a remote basis from Sydney, and when the Project manager was stranded in his home country, away from Vanuatu, due to travel restrictions. Following completion of the MTR discussions between the Bank and the DoE culminated in an agreement to restructure component 1 replacing the vendor model with a public procurement process for SHS for public institutions. However, in April 2022 the GOV decided to use national procurement procedures for the mini grids component, a procedure that the Bank did not agree to. The Bank then allowed the project to close on schedule on June 30, 2022, The ICR implies that the task team was rigid on the issue of ineligible subsidy expenditures and that the difficult relationship between the Bank and the Borrower led to the failure to restructure the project and its closure and cancelation of substantial concessional resources for the country. In its request for project restructuring and extension of closing date in November 2021 the GoV cited among reasons for the project problems: "governance" failures by Bank staff, incompetent project design and an unworkable sales and subsidy verification methodology. There were major shortfalls in the project’s quality at entry and some significant ones during implementation. The Bank's supervision performance did improve with a change in task team in the last couple of years, but much had been lost, and the project went on to close without achieving its objectives. Hence, the Bank performance at entry is rated Unsatisfactory and that during supervision is rated Moderately Unsatisfactory. Thus, the overall rating for the Bank's performance is rated Unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project was generally well designed with appropriate PDO outcome and intermediate outcome indicators, baseline targets, and arrangements for monitoring and evaluation. The main indicators for tracking the progress of the project were described in the Results Framework (PAD, p.49) and additional ones were to be included in the Project’s Operational Manual. Overall, the M&E system was adequate although given its growing portfolio of World Bank financed projects the DOE’s capacity was getting strained, especially given the responsibilities of a complex vendor subsidy mechanism. The project provided TA to support DoE in project management, including on vendor subsidy calculations and verifications. There were some shortcomings in the M&E system design as follows: (a) a misalignment in the provision of SHS and the PDO2 of increasing access to electricity services because the large SHS were not affordable to low income households which were the target group with no access; (b) no indicators were included to measure outputs of the TA which could have served as intermediate outcome indicators; and (c) the DoE did not retain an independent verification agent who, perhaps, could have helped to ease some of the burden and to facilitate resolution of the disputes between the Bank and the GoV. b. M&E Implementation The DoE had the responsibility for collection of data, reporting to the World Bank through the six-monthly progress report, Interim Financial Reports, and audited financial statements. The ICR reports that the DoE shared progress monitoring data with the Bank in the early years but stopped in 2020 and hence the limited data available in the project’s last years. The ICR evaluation is based on data available on the three PDO outcome indicators and two intermediate results indicators. Most of the other seven intermediate indicators, including those on gender and participants consultants during project implementation have zero values in the Results Framework (ICR, p. 49). DOE’s non-sharing of data with the Bank in the last years of the project was probably due to the difficult relationship with the Bank or the Department’s preoccupation with addressing subsidy controversies with the Bank relating to VREP-I. The ICR rightly notes that no adjustment was made to the expected indicator of SHS sales even when it became apparent that demand was not robust, pending the completion of a MTR which took more than 12 months to complete. c. M&E Utilization The utilization of the M&E system was constrained by the slow pace of project implementation, the non- sharing of data in the latter years of the project and the difficult relationship between the Bank and the GoV. The data on all the factors affecting the uptake of SHS (vendors and consumer financing, supply chain issues level of subsidy and consumer affordability) could have been used to implement a timely restructuring of component. Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) M&E is rated Modest because of the implementation shortfalls (non-reporting by the Borrower in the latter years of the project) and failure to use the M&E data to revise the project based on lessons learned during implementation.. M&E Quality Rating Modest 10. Other Issues a. Safeguards At appraisal VREP-II’s E&S risk was assessed as Moderate and, hence, it was classified as an EA category B project triggering the following World Bank operational policies: (a) OP 4.01 – Environmental Assessment; and (b) OP 4.12 – Involuntary Resettlement. OP4.01 was triggered because some activities like modular solar, battery, and hybrid diesel systems could result in some site specific, and time bound negative impacts. Activities were screened for environmental and social impacts and mitigation measures for health and safety, noise and waste disposal were put in place. OP4.12 was triggered because projects required land acquisition and or lease by project communities. At appraisal, an Environmental and Social Management Framework was prepared to provide a process for conducting environmental and social assessments, preparing environmental management plans, including a code of practice for safe disposal of batteries. A Resettlement Policy framework was prepared and later implemented for the DOE by the owner’s engineer. An ESIA for mini grids was completed but the mini grids were not implemented. Regulations for the disposal of solid wastes were not adopted by project closure. The last ISR rated safeguards compliance as Unsatisfactory as of August 2021. DOE had not recruited the E&S staff as required by the financing agreements and, therefore, could not monitor and assist in the implementation of safeguards. The DOE was assisted by the owner’s engineer in managing the E&S aspects of the mini grids up to the bidding stage before the project closed. b. Fiduciary Compliance Financial Management The financial management arrangements for the project met the requirements of the Bank’s OP on Investment Project Financing. However, there was a high turnover of the Borrower’s FM staff during implementation resulting in delayed preparation of financial statements and submission of IFRs, and audited reports. The problems were exacerbated by lack of proper reconciliation of financial data with the Government Financial Management System – Smartstream. The FM rating in the final ISR in 2022 was Moderately Unsatisfactory. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) Procurement At project closure contracts had been signed for only US$1.88 million and no contracts had been signed since September 2020. The procurement management problems were significant with lack of regular submission of procurement plans, huge delays in GoV procurement decisions, poor record keeping and contract management. A major project restructuring and extension of the closing date that had been discussed during the MTR and between the Bank and the Gov by December 2021 but could not be implemented because of disagreement between the Bank and the GoV on the procurement procedures to apply to the restructured project and the mini grid component. The disagreement arose from the Government’s decision in April 2022 to apply national procurement procedures to the mini grid component instead of the procedures that had been agreed in the Financing agreement (Financing Agreement, Schedule 2, (C3)). The Project manager was stranded for a long time in his home country due to Covid-19 restrictions and when his contract expired in October 2021 it took considerable time for a local replacement to be hired c. Unintended impacts (Positive or Negative) --- d. Other Gender The Project design did not have a specific gender objective or targeted component or measures to address gender issues. However, because it aimed to increase access to electricity in rural areas and primarily through renewable energy sources it was expected to reduce the burden on women of organizing alternative energy sources, and widen their employment, learning and education opportunities (PAD, p. 66). In addition, renewable energy would have positive health impacts, including improving maternal and child mortality. The project’s M&E system included indicators for the proportion of females in total direct project beneficiaries, female headed households who received SHS, micro grids or received electricity through mini grids and the % of females who participated in project consultations during implementation. The % of females in total project beneficiaries was 50% compared to a target of 49% and that of female headed households who received SHS, micro grids or electricity through mini grids was 11.8% compared to a target of 13%. The indicator on participation in project consultations during implementation was not reported and, therefore, no data was provided for the proportion of female participants. Institutional Strengthening Substantial technical assistance was provided by VREP-II to support the DOE in expanding rural electrification and renewable energy in Vanuatu. The institutional strength gained from the project (e.g., from the activities of the owner’s engineer) can potentially help the DOE in future electrification work, including under the work initiated through the Green Growth Fund in 2018. Mobilizing private sector financing Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) The project included opportunities for mobilizing private sector financing through consumer and vendor financing of SHS and through micro finance. Technical assistance was provided to promote vendor and microfinance to make SHS affordable but was largely unsuccessful. One VREP-II vendor used vendor financing to promote SHS, but the financing was less attractive than that used under VREP-I, due, perhaps, to capital or cash flow constraints. The project was restructured in 2018 to link it to the regional Sustainable Energy Financing Project to avail partial loan guarantees to participating renewable energy lenders, but the attempt was not successful. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Unsatisfactory Unsatisfactory The project design had major flaws including: (a) inadequate assessment of demand for the larger SHS than those included in VREP-I; (b) inadequate subsidy level and financing mechanisms for consumers and Moderately vendors, (c) absence of an Bank Performance Unsatisfactory Unsatisfactory independent verification agent for SHS sales and subsidy expenditures. Supervision was deficient with delayed/protracted efforts to remedy design flaws, and task team deficiencies exacerbated by the constraints due to Covid-19 Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR has included several important lessons learned from the preparation and implementation of the Vanuatu REP-II. IEG has derived the following complementary or additional lessons: 1. The Bank’s engagement in a series of operations in the same sector with same implementation agency should be carefully designed to avoid institutional capacity overload. When the VREP-I was approved the DoE had already implemented other World Bank projects, including VREP-I and the Improved Access to Electricity Project which were in progress. VREP-II was to become the largest Bank supported project the DOE had ever Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) implemented requiring expansion of human and financial resources. In such circumstances it is important to ensure that there is political commitment within the administration to permit a buildup of the required resources. Although substantial TA was provided by the Bank to assist with the resource buildup, DOE still needed to hire other staff and was unable to secure E&S staff as required by the legal agreements, suffered from frequent turnover of financial management staff, and lacked adequate procurement staff. 2. Projects with complex components or elements such as the retail cost subsidy program for the SHS require not only technical skills but a flexibility to adapt on both the Borrower and Bank side. With flexibility and a strong political commitment, the subsidy levels could have been modified based on a sense of the market requirements. The assessment from the VREP-I MTR indicated that that a critical determinant of affordability for SHS was the availability of sound payment plans for the consumer portion of the capital costs. Increasing the subsidy percentage for the public institutions was proposed and would, perhaps, have been implemented had project restructuring been undertaken. However, the vendor model was to be dropped in the restructured project and, hence, the SHS subsidy for the households was no longer relevant. 3. Appropriate phasing of repeater projects (or use of a multi-phase approach) is essential to channel lessons learned from one operation to the next and to carefully analyze the impact of differentiators between operations on outcomes. The emerging lessons on the subsidy program, the vendor, and product registration processes under VREP-I could have been analyzed before completing design of VREP-II. The decision to move to higher capacity SHS could also have benefited from a more detailed market analysis to inform the changes. The decision to have a lower subsidy for higher capacity SHS, which seems counterintuitive, could have benefited from more stakeholder inputs. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is very well written although it is lengthy is some places, due to the need felt by the team to clearly explain the difficulties that led to the failure of the project to achieve its objectives. Nonetheless, the main text of nearly 30 pages is more than twice the recommended length. Overall, the ICR is to be applauded for its candor, but the ICR is unclear whether political economy issues lay at the heart of the difficult relationship between the GoV and the Bank. In its comments on the ICR the GoV made strong adverse comments regarding the design of the project, the competence and governance practices of Bank staff during project preparation and the earlier period of project implementation. At the same time while the NERM suggests a strong GoV commitment to the development of the energy sector, the failure to adequately staff and resource the DoE and the apparent rigid position on procurement procedures to the extent of giving up substantial concessional resources is perplexing and could have benefited from a more explicit statement on the root cause of the problem. Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Rural Electrification Project Stage II (P160658) It is understandable that data on many of the output and intermediate indicators were not available towards the end of the project after the DOE stopped providing project monitoring data to the Bank in late 2020. Probably the effect is minimal since activities seemed to have scaled down by then, but the ICR could have explained the absence of data on many of the same indicators in the earlier years up to 2020. a. Quality of ICR Rating Substantial Page 17 of 17