Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Report Number: ICRR0024221 1. Project Data Project ID Project Name P165497 SEAP Country Practice Area(Lead) Somalia Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A9095,TF-A9159,TF-B3136 30-Jun-2022 6,914,869.02 Bank Approval Date Closing Date (Actual) 21-Dec-2018 30-Jun-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 7,200,000.00 7,200,000.00 Revised Commitment 6,985,536.13 6,985,536.13 Actual 6,914,869.02 6,914,869.02 Prepared by Reviewed by ICR Review Coordinator Group Ihsan Kaler Hurcan Vibecke Dixon Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Somalia Multi-Partner Fund Grant Agreements (p.5) dated January 26, 2019 (with the Federal Government of Somalia) and February 3, 2019 (with Somaliland), and the Project Appraisal Document (p.18), the project objective is “to expand access to electricity in targeted urban, peri-urban, and rural communities.” The project was to be implemented by the Ministry of Energy and Water Resources (MoEWR) of the Federal Government of Somalia (FGS) and the Ministry of Energy and Minerals (MoEM) of the Government and Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Somaliland (GoSL). Somaliland declared independence in 1991, but the international community does not recognize Somaliland’s independence. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The Somalia Electricity Access Project consisted of three components: A. Electrification of Households and Small Businesses through Standalone Solar Home Systems. (Appraisal cost: US$3.0 million; actual cost: US$3.11 million) This component was to finance off-grid activities to increase households and small enterprises’ access to electricity. These activities consisted of the following: (a) provision of results-based expansion grants and seed grants to solar distributors for investment in key solar business functions; (b) promotion of quality assurance to limit the availability of and demand for poor quality or counterfeit solar products; and (c) implementation of consumer awareness and citizen engagement campaigns to increase household understanding and use of quality off-grid solar technology. B. Analytical work for enabling electrification through Solar-Powered / Hybrid Mini-grids. (Appraisal cost: US$1.00 million; actual cost: US$0.45 million) This component was to finance analytical work for the development of the mini-grid sector. The activities consisted of the following: (a) detailed geospatial mapping of the country to identify current status of mini- grids and potential future sites; (b) revision of property rights and land issues related to energy infrastructure investment; (c) preparation of pre-feasibility studies for hybridization, operational enhancements, and densification of existing mini-grid sites; (d) preparation of pre-feasibility studies for new sites identified in geospatial mapping; (e) development of structuring options for the financing, operation, and ownership of new mini-grids; and (f) definition of legal, institutional, and financing arrangements for the development of mini-grids. C. Technical Assistance, Capacity Building, and Project Management. (Appraisal cost: US$1.75 million; revised cost at the time of additional financing: US$3.20 million; actual cost: US$3.36 million) This component was to finance activities to strengthen the capacity of the MoEWR and the MoEM for overall energy sector management, power and access planning, and implementation of future development projects. The capacity strengthening activities consisted of the preparation of energy sector studies, development of energy strategies, revision of energy policies, analytical work, improvement of internal infrastructure and systems, training of the staff including workshops and study tours, and the establishment of a project implementation unit within the ministry. This component was also to finance the implementation of a needs assessment to identify priority capacity building interventions for both ministries and training Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) needs for their staff. The scope of this component expanded at the first restructuring in June 2020 (see Revised Components below). Revised Components At the first project restructuring, the following activities were added to the third component to be implemented by the MoEWR and financed by an additional financing (ICR, p10): (a) Options analysis for the electricity sub-transmission and distribution integrated development least cost investments for major load centers in Somalia; (b) identification and prioritization of public facilities for grid and off-grid electrification; and (c) Strategic Environmental and Social Assessment for the energy sector. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project cost was originally estimated at US$5.75 million. The revised project cost at the time of the additional financing was US$7.20 million. The project closed on June 30, 2023, with an actual cost of US$6.91 million. Financing: At appraisal, the Somalia Multi-Partner Fund grant amount was estimated at US$5.75 million. The additional financing amount was US$1.45 million. The project disbursed US$6.91 million at project closing fully financing the project activities. Borrower’s contribution: At appraisal, no borrower’s contribution was estimated, and none materialized by project closing. Project Restructurings: The project was restructured twice including an additional financing.  First Project Restructuring and Additional Financing (July 6, 2020 – Level 2): An additional financing amounting to US$1.45 million as grant was approved to finance the additional activities added to the third component (See the Revised Components above). To monitor the progress of these additional activities, two intermediate results indicators were added to the results framework: (a) Completion of the options analysis for the electricity sub-transmission and distribution integrated development least cost investments; and (b) complete Strategic Environmental and Social Assessment for the energy sector.  Second Project Restructuring (June 14, 2022 – Level 2): The project closing date was extended by 12 months from June 30, 2022 to June 30, 2023 to allow time for the completion of the options analysis study added to the project scope at the time of the additional financing, which was delayed because of the failure of the negotiations with the best ranked bidder, and the disbursement of the result-based financing, which was delayed because of the late recruitment of grant managers. Supply chain challenges resulting from the global pandemic restrictions also delayed the project implementation because solar home systems could not be delivered to project sites. Dates: The project was approved on December 21, 2018 and became effective on April 1, 2019. The project team decided not to conduct a Mid-Term Review (MTR) because of the delayed start of project towards the planned MTR date. The original project closing date was June 30, 2022, but it was extended by 12 months (please see the second project restructuring above for the reasons of project closing date extension). The project closed on June 30, 2023. Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) 3. Relevance of Objectives Rationale At project closing, the project objective was highly aligned with the World Bank’s then current strategy as defined in the Country Partnership Framework for the Federal Republic of Somalia for the period FY2019- 2022, which was extended to cover the fiscal year 2023. As a pilot project, the project sought to address the development problem and a fragility driver of lack of access to electricity in the urban, semi-urban, and rural areas of Somalia including Somaliland to increase inclusion across the society with the expectation that the results of the project would lead to a scaling up of electrification in the country. The project was to achieve this objective by addressing the insufficient access to financing of the solar service providers through the provision of a limited amount of seed grant to build up their stocks and performance-based grants (managed by local banks as grant managers) for the installation of stand-alone solar systems. The implementation of a pay-as-you-go model by the service providers and increased familiarity of the local banks with this business model were expected to ensure sustainability of the market model and increased financing that would lead to a scaling up of electricity through stand-alone solar systems supported by World Bank and other donors-financed follow-on projects. The project objective corresponded to “Focus Area 2: Restoring Economic Resilience and Opportunities” and the project supported the achievement of three strategic objectives—Strategic Objective 2.1: Improve the business environment and lower barriers to entry through the provision of results-based grants and provision of electricity; Strategic Objective 2.2: Increase access to finance for inclusion and digital development through results-based financing and installation of stand-alone solar systems at households increasing inclusions; and Strategic Objective 2.3: Increasing access to renewable energy through the provision of solar power to households and businesses. The project objectives were highly relevant to the country context. As a project to be implemented in a fragility, conflict and violence (FCV) status country, the project objective was to be achieved through a pilot implementation of results-based grants for the installation of stand-alone solar systems. The project was to support the already existing solar service providers and local banks. The achievement of the project objective was to lead to the scaling up of electrification activities. Increasing access to electricity and the sustainability of the business model through the development of a market mechanism were relevant to the country context where population is widely dispersed, and the expansion of the grid was not seen as an economically viable solution in the short run. The project benefited from the experience of the World Bank-IFC Lighting Africa program and was piloted with the aim of scaling up electrification through stand-alone solar systems. However, the objective of this project to increase access to electricity was an output-oriented objective because it did not encompass the outcomes expected from the project’s intervention in addressing the barriers in the solar service providers market, such as increased availability of local finance from the banks, sustainability of the pay-as-you-go business model, and sustainability of the stand-alone solar systems installed under the project. Additionally, the project objective formulation did not encompass the outcomes expected from the implementation of the project activities under the second and third components related to “the planning for reform of the entire power sector, including scaled-up access expansion using other options like mini-grids and the national grid, needed to achieve the long-term impact” (ICR, p.10). These activities were monitored by intermediate results indicators, but their implementation did not directly contribute to the achievement of the project Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) objective to increase access to electricity although they were critical in informing the follow-on Somali Electricity Sector Recovery Project, P173088 (SESRP). Overall, while the project objective formulation did not fully encompass the outcomes expected from the project’s intervention, the objective of this pilot project as formulated was highly aligned with the World Bank strategy and relevant to the context in an FCV status country. The project objective was also of high strategic importance to the country, the achievement of which was expected to lead to follow-on projects in increasing access to electricity. Therefore, the overall relevance of the project objective is rated High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To expand access to electricity in targeted urban, peri-urban, and rural communities. Rationale Theory of Change of the Project The project was designed to implement two sets of activities. Under the first set of activities, the project was to extend performance-based grants through local financial intermediaries (as grant managers) to the solar service providers upon the verification of the installation of stand-alone solar systems (project outputs) to households and businesses under a pay-as-you-go model. This intervention was expected to address the insufficient availability of financing to solar service providers and increase the familiarity of local banks with this business model. This activity was to be supported by campaigns to increase awareness of households and businesses about high quality products and limit the penetration of low quality and counterfeit products to the market through the promotion of quality assurance. The expected outcome would be the establishment of the foundations of a sustainable solar services market with the potential for scaling up. These activities were expected to result in increasing electricity access by 113,900 people and 100 businesses through the installation of 21,600 stand-alone solar home systems. The second set of activities to be implemented were related to the development of the mini-grid sector in the country through the preparation of pre-feasibility studies and revision of sector related issues (such as detailed geospatial mapping of existing mini-grids and potential sites for mini-grid development, pre-feasibility studies for hybridization operational enhancements, and densification of existing mini-grids and for new sites identified in geospatial mapping, and revision of property rights and land issues) and capacity building of the government authorities in overall energy sector management, power and access planning, and implementation of future development projects through trainings, workshops, and study tours, and preparation of energy sector studies and strategies. The expected outcome of these outputs were increased availability of data and capacity to scale up mini-grids that had already been financed by various international donors and Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) increased capacity of the MoEWR and MoEM in power sector management and planning with the ultimate goal of expanding electricity to unserved communities. Overall, the causal links from project inputs and outputs to the expected intermediate outcome of increased access to electricity were direct and valid, and the achievement of the project objective could be attributed to the project’s intervention through the implementation of project activities under the first component. However, the project objective as formulated was output-oriented and did not capture the project’s outcome related to solar service market creation and its sustainability. Furthermore, the implementation of other activities under the second and third components did not directly support the achievement of the project objective because of the shortcomings in the formulation of the project objective. The project formulation did not include the outcomes related to the development of mini-grids and capacity building of the MoEWR and MoEM in power sector planning and management. Although the project design was robust and appropriate for a pilot project that would have been expected to lay the foundations for scaling up electrification in the country by addressing the barriers in the solar service market and strengthening of capacity through studies and training, the inadequate formulation of the project objective failed to capture these outcomes. Outputs  Grant funding disbursed to solar home system distributors (US$): The project disbursed US$0.83 million to 11 solar service providers in FGS and US$0.70 million to 12 solar service providers in Somaliland as results-based grants totaling US$1.53 million following the verification of the installation of stand-alone solar systems by an independent verification agent, achieving the target to disburse US$1.50 million.  Number of households with new stand-alone solar systems: Through the disbursement of results- based grants to solar service providers, the project facilitated the installation of 33,661 stand-alone solar systems surpassing the target of 21,500.  The number of male-owned and female-owned small businesses with new standalone solar system: A total of 1,107 small businesses gained access to electricity through the project against the target of 100. The significantly higher achievement was because of the solar service providers’ promotion of larger systems in Somaliland that were more suitable for small and medium size enterprises. The project financed the installation of these systems at 975 small businesses, whereas the number in FGS was 132, where the majority of the products solar service providers marketed were more suitable for households. However, this indicator did not provide a breakdown based on gender.  Generation capacity of energy constructed or rehabilitated (MW): The total generation capacity created because of the installation of the solar home systems is 0.36 MW, exceeding the target of 0.20 MW.  Consumer awareness campaigns completed: The project financed various campaigns to increase consumer awareness to increase households and businesses’ understanding of good quality solar systems. This indicator was monitored as a Yes or No indicator.  Number of consultations for men and women: The project held 17 such consultations against the target of eight. The topics included in these consultations were environmental and social instruments, national energy policy formulation, preparation and enactment of the electricity act, the least cost geospatial electrification plans, and consumer education and awareness events for the project and Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) during sales and marketing of solar products, including an event on women and off-grid electricity organized by grant managers (ICR, p.27). The project was gender-coded and included the implementation of a gender action plan. The results framework included gender specific indicators to capture the project activities that targeted women.  Percentage of households with new stand-alone solar systems, of which headed by female: The target was 20 percent. At project closing 24 percent of the households with new stand-alone solar systems were headed by women.  Separate consultations with women and men prior to and during implementation: The project also held separate consultations based on gender as planned.  Women-targeted in the information and knowledge campaigns delivered: The project organized specific events for women such as awareness for solar products and trainings on energy issues including off-grid electricity. The following outputs did not directly support the achievement of the project objective, but some informed the follow-on Somalia Electricity Sector Recovery Project (SESRP) to scale up electrification in the country and others contributed to capacity strengthening at the ministries.  Number of studies on mini-grid sector completed: The project financed 10 studies that were not restricted with mini-grid sector. The target was four studies on mini-grid sector. The consultants hired under the project and an Energy Sector Management Assistance Program (ESMAP) team supported the MoEWR and the MoEM in the preparation of the Indicative Least Cost Geospatial Electrification Plan to Achieve Universal Access, the Somali Electricity Supply Industry Institutional Structure Study, the Strategic Environmental and Social studies for the energy sector, the Environmental and Social Management Framework, the Labor Management Policy, the Resettlement Framework Procedure, and the Capacity Building and Training Plan for the Energy Sector.  Completion of Options Analysis for the Electricity Sub-Transmission and Distribution Integrated Development Least Cost Investments for major load centers in Somalia: The project could not complete this analysis despite a project closing time extension of one year. The ICR (p.29) notes that “the Options Analysis for the Electricity Subtransmission and Distribution Integrated Development Least Cost Investments for major load centers in Somalia was initially procured and implemented under SEAP, but its implementation went beyond project lifetime. The implementation of this study in FGS and Somaliland will be completed under the ongoing Somali Electricity Sector Recovery Project.” This indicator was added to the results framework at the first project restructuring.  Complete Strategic Environmental and Social Assessment for the energy sector: The project completed this assessment as planned but it will be reviewed and finalized under the SESRP. This indicator was added to the results framework at the first project restructuring.  Number of Federal Government of Somalia Ministry of Energy and Water Resources staff trained: The project provided training to 120 members of staff on energy related topics such as energy sector management, power and access planning, revision of energy policy, and project implementation including procurement, financial management, and environmental, social and gender issues. The target was 10.  Number of Government of Somaliland Ministry of Energy and Mineral staff trained: The project provided the same trainings listed in the previous indicator to 78 members of staff of the Ministry of Energy and Minerals of Somaliland. The target was 32. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Outcomes  Number of people provided with new or improved electricity service: The project used the SE4ALL Multi-Tier Framework for Measuring Access to Electricity (MTF) benchmarks for different sizes of solar systems to estimate the number of people who gained access to electricity through the installation of 33,661 stand-alone solar systems under the project. It is estimated that 221,513 people gained access to electricity against the target of 113,900 people (148,625 people in FGS and 72,888 in Somaliland). No targets were specified for FGS or Somaliland. This indicator does not provide a breakdown of the number of people who gained access in urban, peri-urban and rural communities nor does the results framework include an indicator with targets set for these three settlement levels. This is a shortcoming of the M&E system in capturing the impact of the project although the project objective included these settlements in its formulation. The project successfully piloted the performance-based grants for the installation of stand-alone solar systems resulting in a significantly higher number of people than the target gaining access to electricity at Tier 3 of the MTF, which corresponds to at least three hours of electricity in the evenings. Although the results framework did not include indicators to capture outcomes achieved as a result of the project’s intervention, the piloting of the performance-based grants through local banks addressed the lack of financing barrier to the development of solar services market in the country. Based on the experience gained from the implementation of the performance-based grants, the International Bank of Somalia (the grant manager in the FGS) attracted additional funding from other financiers and increased its financing of stand-alone solar systems from US$0.33 million in 2020 to US$1.90 million in 2022 (ICR, p.12). The implementation of the pay- as-you-go model was critical in increasing the affordability of solar stand-alone solar systems by low income households. Despite its low budget, the implementation of the project activities in all regions of Somalia including Somaliland resulted in increased access to electricity among nomadic communities and use of solar power in agriculture. The beneficiary survey conducted at project closing confirmed the achievement of outcomes expected as a result of access to electricity through stand-alone solar systems such as improved security, adult learning at night, improved health services (safer child delivery at night and extended pharmacy hours, and elimination of phone charging cost (ICR, p.12)). Although the project activities implemented under the second and third components of the project did not directly support the achievement of the project objective, the preparation of the power sector related studies and strategies and the delivery of training to the staff of MoEWR and MoEM increased the institutional capacity in the power sector. This manifested itself in Somalia joining the East African Power Pool on March 24, 2022. The studies completed under the project informed the design and preparation of the SESRP with a funding of US$150 million aiming to increase access to cheaper and cleaner electricity and reestablish the electricity supply industry (ICR, p.13). Overall, despite the mismatch between the formulation of the project objective and the design of the project, and the gaps in the theory of change, the project achieved its objective, reduced the barriers to the development of solar service market, increased the affordability and quality of stand-alone solar systems, and facilitated the preparation of a follow-on project that is expected to scale up electrification in the country. Therefore, the efficacy of the project in achieving the project objective is rated High. Rating Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) High OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project achieved its objective and delivered outcomes that established a sound solar service market foundation by addressing the barriers in the solar service market and increased capacity in the power sector. These achievements are expected to lead to the scaling up electrification in the country. Overall Efficacy Rating High 5. Efficiency Economic Analysis At appraisal, a cost-benefit analysis was conducted for the activities to be implemented under the first component that were expected to generate quantifiable benefits. Technical activities to be implemented under the second and third components were excluded from the analysis. The benefits of having access to electricity were quantified as the avoided costs of lower-quality electricity and lightning alternatives. This was a conservative approach because other quantifiable benefits were not included in the analysis such as the cost savings from charging mobile phones using solar energy, improved health because of the elimination of kerosene use, avoided greenhouse gas emissions, and incrementally increased economic activities because of the availability of lighting after sunset. The costs included in the analysis were the costs of the stand-alone solar systems. The calculations resulted in an Economic Internal Rate of Return (EIRR) of 14.2 percent and a Net Present Value (NPV) of US$0.67 million at a discount rate of 6 percent confirming the economic viability of the project’s intervention. When benefits from the reduction of greenhouse gas emissions are included, the EIRR increased to 29.8 percent and the NPV to US$1.15 million. The same methodology was used at project closing using actual costs. The EIRR was estimated at 14.5 percent and the NPV US$ 1.05 million. When benefits from the reduction of greenhouse gas emissions are included, the post-project EIRR increased to 39.1 percent and the NPV to US$3.63 million because of the installation of larger capacity stand-alone solar systems in businesses in Somaliland. Operational and Administrative Efficiency Initial delays in the hiring of the grant managers and E&S specialists because of the FCV context in the country delayed project implementation and adversely affected the project’s operational and administrative efficiency. The retendering of the options analysis study because of unresponsive proposals in the first bidding further delayed project implementation. These delays required a 12-month extension of the project’s closing date. This delay could have been avoided if grant managers had been hired in advance of project’s effectiveness. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Overall, while there were minor shortcomings in the operational and administrative efficiency of the project, the project’s benefits, including the reduction in greenhouse gas emissions, were higher than the estimate at appraisal. Therefore, the project’s efficiency in achieving the project objective is rated Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 52.00 Appraisal  14.20  Not Applicable 45.00 ICR Estimate  14.50  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Although the project objective was output-oriented, and its formulation did not fully encompass the outcomes expected from the project’s intervention, the project objective was highly aligned with the World Bank strategy and relevant to the context in an FCV status country and of high strategic importance. Therefore, the relevance of project objectives is rated High. The project fully achieved the project objective, and the project’s results were critical in creating a strong foundation both in terms of market creation and strengthening institutional capacity in scaling up electrification in the country. The project’s efficacy in achieving the project objective is rated High. The operational and administrative efficiency of the project had minor shortcomings in procurement that required a 12-month extension of the project closing date, but the project delivered benefits higher than those estimated at appraisal. The project’s efficiency in achieving the project objective is rated Substantial. Overall, the outcome of the project, which was implemented in an FCV context, is rated Highly Satisfactory. a. Outcome Rating Highly Satisfactory 7. Risk to Development Outcome Technical: Stand-alone solar systems for households provide electricity for a limited time after sunset with a battery. They provide electricity at Tier 3 of the Multi-Tier Electricity Access Framework at best. Productive uses of electricity supplied by such solar systems are limited. While the stand-alone solar systems’ impact on economic and social lives of unserved people is not negligible, it is not transformational. Therefore, to address dissatisfaction and ensure the sustainability of access to electricity, increased supply of electricity Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) would be needed to meet the increasing demand either through the installation of larger capacity solar systems or other means of electricity supply. Maintenance of solar panels and replacement of batteries: This stands out to be a moderate risk as long as the solar service providers continue with the pay-as-you-go business model and have sufficient maintenance capacity and battery stocks to service the customers. Financial constraints in the solar service sector: The project supported the solar service providers through performance-based grants to expand their solar panel stocks and businesses. The project also resulted in an increase in the credit lines offered by local banks to solar service providers. However, the sector is far from sustaining itself with the credit available in domestic financial markets. The sustainability of the sector depends on the support from donors in the form of foreign exchange for the solar service providers to import solar panels and provide further financing to mainstream the business model. 8. Assessment of Bank Performance a. Quality-at-Entry Increasing access to electricity in the country was of high strategic importance for the FGS and GoSL to increase inclusion and improve the people’s socio-economic welfare. The project preparation benefited from the detailed due diligence conducted under the Off-Grid Lighting Market Assessment for Somalia study, mini-grid and off-grid solar technical studies, an energy sector needs assessment, and a renewable energy study conducted by other multilateral and bilateral donors. The introduction of performance-based grants to solar service providers and the pay-as-you-go model were appropriate to address the financing barrier to the development of the solar service market and ensure the sustainability of the project outcomes while setting the foundations for scaling-up of electrification through the installation of stand-alone solar systems. Technical aspects of the project were adequate. Although the assumptions were conservative, the economic analysis was adequately conducted showing the economic viability of the project’s intervention. The project was gender-coded and the results framework included indicators to capture the impact of the project on women. However, the project objective formulation did not include the outcomes expected from the implementation of the project activities in increasing access to electricity and capacity building, although the results framework included intermediate activities to capture the project’s such impact. The risks to the implementation of the project and the achievement of the project objectives in an FCV status country were adequately identified, and the mitigation measures were in place. The MoEWR and the MoEM were to implement the project in FGS and GoSL, respectively, to increase the impact of the project. The shortcomings in the implementation capacity of the ministries were to be addressed through the hiring of grant managers and a third party verification agent. This method proved to be effective during project implementation. The project was not fully ready for implementation because the preparation of some implementation arrangements and M&E implementation plan were delayed after effectiveness considering the FCV status of the country. However, the setting up of the project implementation units and the hiring of consultants in advance could have improved project’s readiness. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Overall, while there were moderate shortcomings in the readiness of the project and the M&E design and arrangements, the World Bank’s performance in ensuring quality at entry at an FCV context was adequate to successfully implement the project. Therefore, the quality at entry is rated Satisfactory. Quality-at-Entry Rating Satisfactory b. Quality of supervision Eight supervision missions were held during the five-year project implementation period. A review of the Implementation Status and Results Reports (ISRs) and Aide Memoirs showed that the performance reporting was candid and detailed. During the first year of the project, the World Bank project team provided project implementation support to the MoEWR and MoEM, while the project implementation arrangements were being finalized. The location of the World Bank Task Team Leader in the same time zone of Somalia (although in a different country) facilitated easier and frequent contact with the authorities in FGS and GoSL in real time. The focus of the project team on the development impact was sufficient because the successful achievement of the project’s expected outcomes led to the follow-on SESRP to scale up electrification in the country. The project team processed an additional financing to implement additional studies that were to inform the design of the follow-on interventions, but the implementation of the options analysis study could not be completed by project closing date although it was extended by 12 months. The project team was aware of the issues that arose in financial management and safeguards implementation and supported the project implementing agencies, but a report that aimed at improving the quality of the environmental and social aspects of the project could be prepared only towards the end of the revised project’s closing date. The project team decided not to conduct a Mid-Term Review (MTR), because most of the project activities could start only towards the planned MTR date. The project team conducted a detailed review of project implementation one year before project closing that could have been qualified as an MTR but was not named as such. The absence of an MTR did not adversely affect the monitoring of project implementation and the achievement of project’s results in this FCV status country. Overall, the quality of supervision is rated Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project objective to increase access to electricity was output-oriented. Because of the formulation of the project objective that was restricted to increasing access to electricity, the project’s theory of change Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) had gaps in establishing direct causal chains between the implementation of the technical assistance activities and the achievement of the project objective. The indicators in the results framework were sufficient to capture the achievement of the project outputs (such as the installation of the stand-alone solar home systems) and the increase in the number of people who gained access to electricity. These indicators and the methodology to measure them were specific, measurable, achievable, relevant, and timebound. However, the results framework did not include any indicators capturing the outcomes expected from the implementation of technical assistance activities although there were intermediate results indicators related to these activities. The results framework did not include an indicator to capture the achievement of the project objective in urban, semi-urban, and rural areas, although these three settlement levels were clearly identified in the project formulation. At appraisal, the M&E system was outlined but a detailed M&E plan, along with the Project Implementation Manual, was to be prepared after the effectiveness of the project to ensure that the project implementation would move forward in this FCV context. b. M&E Implementation The project team confirmed that a detailed M&E plan was prepared after project effectiveness. The project implementation units prepared and submitted quarterly progress reports as planned. These sufficiently reported on the achievement of the indicators and the implementation issues. The independent verification agent hired under the project was effective in identifying issues related to the installation of the stand-alone solar systems in Somaliland and ensuring that the performance-based grants were distributed based on the verified data. At the time of the additional financing, two new indicators were added to the results framework to capture the achievement under the new activities added to the project scope. The shortcomings in the M&E design in capturing the outcomes related to the implementation of the technical assistance activities were not addressed during project implementation, but the M&E system was sufficient to measure the achievements of the results and the project objective in increasing access to electricity in the country. c. M&E Utilization The M&E findings were used to process an additional financing and project restructuring. M&E data were adequately used to provide evidence for the achievement of outcomes. As the ICR (p.20) notes “the independent verification reports not only served as the basis for disbursement of the RBF, but also served as training and awareness raising for the solar providers and beneficiaries.” The M&E findings and data were critical in informing the follow-on World Bank-financed SESRP. Overall, although the M&E design had moderate shortcomings in capturing the outcomes expected from the implementation of the technical assistance activities, the M&E system as designed and implemented was sufficient to assess the achievement of the project objective and test the links in the results chain related to increasing access to electricity. The M&E utilization was effective in informing the follow-on World Bank intervention. Therefore, the M&E quality is rated Substantial. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Category B under Environmental Assessment (OP/BP 4.01) and did not trigger any other safeguard policy, because the project activities consisted of the installation of stand-alone solar systems on the rooftops of houses and businesses that did not require any resettlement or acquisition of land. Environmental Assessment (OP/BP 4.01): The project was classified as Category B because the limited and reversible impact of the activities financed under the project on environment. The only potential adverse environmental impact of the project anticipated was the improper recycling and disposal of batteries at the end of their three to five years. The project prepared an Environmental and Social Management Framework incorporating the disposal of solid waste from the installation of stand-alone solar systems and development of a project-specific environmental code of practice for the collection, transport, storage, and disposal of used batteries (PAD, p.44). The ESMF was disclosed in-country and on the World Bank’s InfoShop on September 24 and 25, 2018, respectively. The MoEWR of the FGS and the MoEM of Somaliland held stakeholder consultations on May 19 and May 22, 2018. A project-level grievance redress mechanism was established for communities and individuals to submit complaints about the adverse effects of the project. During project implementation, there were no serious environmental and social (E&S) issues, but there were shortcomings in reporting by the project implementation unit, grant managers, and solar service providers on E&S performance. E&S staff’s hiring was delayed. The time dedicated by the E&S specialists hired to the implementation of the safeguards policy was limited resulting in inadequate oversight. To address these issues, a report highlighting challenges encountered, key achievements, lessons learned, and recommendations was prepared but only towards the end of the project (ICR, p.20). The project team confirmed that a project-specific environmental code for the disposal of spent batteries was developed, and its implementation was one of the eligibility criteria for the performance-based grants. b. Fiduciary Compliance Financial Management The External Assistance Fiduciary Section under the Office of the Accountant General oversaw and managed the project financial management. The shortcomings in financial management capacity and related risks were adequately identified at appraisal. The lack of internal audits to provide independent assurance and risk management to World Bank-supported projects adversely affected the project’s financial management. Unaudited interim financial reports and audit reports were submitted with occasional delays. Because of slow processing of funds, there were delays in payments to the vendors, and documentation did not fully support the payments. The ICR does not report any issues of corruption or misuse of funds associated with the project. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Procurement The procurement of two local banks as grant managers was delayed, which resulted in a slow start of the project and a project closing date extension because performance-based grants could not be fully disbursed before the original project closing date. There were also delays in the hiring of the E&S specialists. The procurement of the consultancy services for options analysis study had to be retendered because of non-responsive proposals submitted by bidders. The time required to retender the options analysis study was the other main reason for the project closing date extension. However, despite this time extension, the study could not be completed before the revised project closing date. In an FCV status country, advance procurement of some activities (such as hiring of grant managers) could increase project’s efficiency and allow the completion of the project activities within the original project closing date (ICR, p.15). c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Highly Satisfactory Highly Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons This review has drawn three lessons based on the information in the ICR. The hiring of expertise at the preparation stage can increase project readiness and avoid implementation delays in an FCV status country. The project was not fully ready for implementation when it became effective. Some arrangements were left until after effectiveness such as detailed a M&E plan and a project implementation plan. While the World Bank project team fully supported the project implementation units during the early phases of project implementation, the hiring of expertise and completion of the project implementation arrangements took almost one Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) year because of the consultants’ reluctance to work in an FCV status country. This adversely affected the project’s efficiency. The hiring of expertise at the preparation stage or before project effectiveness to support the PIUs in an FCV status country could be critical in avoiding project inefficiencies and implementation delays. The use of performance-based grants through local banks as grant managers can help increase local capacity and address the lack of financing barrier to market development critically contributing to the achievement and sustainability of project outcomes. One of the main barriers to the development of solar service market in Somalia was the lack of financing. The project addressed this barrier through the administration of performance-based expansion grants by local banks. This approach resulted in the increased familiarity of the local banks with the business model and ensured the installation of targeted number of stand-alone solar systems in the project areas. The solar service providers gained access to financing to expand their businesses, and the local banks sourced additional financing from other donors to increase their credit facilities. This successful market creation is expected to ensure the sustainability of the services of the solar companies. The true extent of project outcomes cannot be captured if the project objective is output- oriented. The project objective to increase access to electricity was output-oriented, and it was to be achieved by implementing the project activities under the first component. However, the project activities under that component had a wider impact in developing the local solar service market. In addition, the technical assistance activities implemented under the second and third components were critical in increasing the local capacity in scaling up electrification through stand-alone solar systems and mini-grids and in power sector management both in FGS and Somaliland. These outcomes were also critical in informing the follow-on World Bank-financed SESRP, but they were not captured by the project objective nor the results framework. This created gaps in the project’s theory of change. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is concise and candid. It provides a detailed overview of the project. Its narrative is evaluative with sufficient description of the project’s performance, but information is insufficient to explain some issues encountered during project implementation such as why a Mid-Term Review was not undertaken and whether a detailed M&E plan was prepared after effectiveness. The reporting is not restricted to a discussion of the achievement of the project objective but provides additional information and evidence for the achievement of other outcomes related to the development of the solar service market and how the project results and M&E findings informed a follow-on project to scale up electrification in the country. The report is results oriented. The interrogation of the evidence is sufficient; the salient points are summarized concisely; and the linking of evidence to findings is clear. The report is internally consistent. The lessons are based on the specific experiences and findings of the project and are sufficiently linked to the narrative. Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SEAP (P165497) Overall, the quality of the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 17 of 17