Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Report Number: ICRR0024284 1. Project Data Project ID Project Name P149521 Development of CCS in South Africa Country Practice Area(Lead) South Africa Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A3137 30-Dec-2021 6,760,374.83 Bank Approval Date Closing Date (Actual) 22-Mar-2017 29-Feb-2024 IBRD/IDA (USD) Grants (USD) Original Commitment 23,000,000.00 23,000,000.00 Revised Commitment 10,000,000.00 10,000,000.00 Actual 6,760,374.83 6,760,374.83 Prepared by Reviewed by ICR Review Coordinator Group Maria Shkaratan Avjeet Singh Avjeet Singh IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Original Project Development Objective (PDO) was “to assess the feasibility of, and build expert capacity for, carbon capture and storage in the territory of Recipient” (Grant Agreement, page 6). The PDO was substantively the same but phrased slightly differently in the Project Appraisal Document (PAD): “to assess the feasibility of, and build expert capacity for, carbon capture and storage in South Africa” (PAD, page ii). Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) For the purposes of this Implementation Completion and Results Report (ICR) review, the Original objective will be assessed as follows: Original PDO 1: To assess the feasibility of carbon capture and storage in South Africa, and Original PDO 2: To build expert capacity for carbon capture and storage in South Africa. The PDO was revised during the second restructuring of June 30, 2023, as follows: “to assess the feasibility of, and build expert capacity for, carbon dioxide storage in South Africa”. The part of the Original PDO on carbon capture was removed, resulting in a reduction in the Project scope. For the purposes of this Implementation Completion and Results Report (ICR) review, the Revised objective will be assessed as follows: Revised PDO 1: To assess the feasibility of carbon dioxide storage in South Africa, and Revised PDO 2: To build expert capacity for carbon dioxide storage in South Africa. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 30-Jun-2023 c. Will a split evaluation be undertaken? Yes d. Components 1. Original components Component 1 CO2 Pilot Storage Project (cost at appraisal: US$36.5 million; actual cost: US$10.99 million) was intended to finance the following: geological investigation and site selection; schematic and detailed engineering design; procurement and construction; and operations and monitoring. Upon Component completion, the South African National Energy Development Institute (SANEDI) and the Government were responsible for decommissioning the constructed facility, which was beyond the Project mandate. The selection of the Zululand Basin for geological investigation was based on prior studies, lower costs, shorter implementation time, and consultations with stakeholders. Component 2 CO2 Pilot Capture Project (cost at appraisal: US$1.5 million; actual cost: US$0 million) comprised Front-End Engineering Design (FEED) of carbon capture facilities at Eskom’s Kusile Coal Power Plant. Following Component completion and once additional funding was secured, the procurement, construction, operation, and testing of the carbon capture facilities were to be performed. After Component completion, SANEDI/Eskom and the Government were responsible for decommissioning the Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) constructed facility, which was beyond the Project mandate. The Kusile Power Plant was selected due to its design and siting, which were conducive to a retrofit with carbon dioxide capture equipment. Revised Components: The components were revised, as follows: Component 1 (carbon storage facilities) was amended twice. At Restructuring 1 of June 2021, activities to build expert capacity in carbon capture and storage (CCS) were added. At Restructuring 2 of June 2023, the procurement and construction, and operation of storage facilities (CO2 injection, storage, and monitoring) were removed because they could not be completed by Project closure. Delays were caused by the following: (i) Project effectiveness was achieved very late, 19 months after approval, (ii) decision making through the governance structure established at appraisal was slow; (iii) stakeholder consultations were delayed; and (iv) completion of environment and social impact assessments, which were prerequisites for the physical activities, was delayed. As a result, the remaining activities under Component 1 were: the investigation and characterization of a suitable site, and the engineering design. Component 2 (carbon capture facilities) was dropped during Restructuring 2 of June 30, 2023, because it could not be completed by the Project's closure. Eskom's focus on resolving power blackouts and increasing generation output at Kusile plant took priority, leading to delays with the design and construction of the carbon capture facilities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The appraisal estimate was US$38.0 million, while the actual disbursement was US$11.0 million. The difference of US$25.2 million is due to cancelled or unused funding, the scale-down of Component 1 activities, and the cancellation of Component 2 activities. Additionally, an undisbursed amount of US$1.8 million was transferred to the client’s designated account but not used and is expected to be refunded to the Bank by the end of the grace period (December 31, 2024). Project Financing: The Project was financed through a Carbon Capture and Storage Trust Fund (CCSTF) grant, with an estimated amount of US$23.0 million at appraisal. The actual disbursement at closure was US$4.97 million. Borrower/Recipient contribution: The Borrower’s contribution was estimated at US$15.0 million at appraisal, but the actual contribution amounted to US$6.06 million (in South African rands: R 96,937,490). Project Dates: The Project was approved on March 22, 2017, and became effective on November 10, 2018. There was no MTR review. The Project was restructured twice: (i) on June 28, 2021; and (ii) on June 30, 2023. The original closing date was on December 30, 2021. The Project was extended twice, for the total of two years and two months (or 26 months), to February 29, 2024, which was the date of the Project’s actual closure. Restructurings: The Project had two restructurings: Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Restructuring 1 (June 28, 2021) involved the following changes (in addition to the modification of Component 1 to add capacity building activities, as described above):  New Implementation Agency: this responsibility was transferred from SANEDI to the Council for Geoscience (CGS), based on its technical capacity and its role as a key contributor to CCS development in the country. A new Project implementation unit (PIU) was established at the CGS, with decision-making and project management responsibilities redefined.  Change in the site selected for the carbon storage facility (Component 1): the site was changed from the uMkhanyakude District Municipality in the Kwa Zulu Natal Province to the Govan Mbeki Municipality in the Mpumalanga Province. The new site was closer to the largest emission sources, thus reducing the need to transport CO2. Also, new monitoring showed seismicity at the original site.  Extension of the closing date by 18 months, from December 31, 2021 to June 30, 2023, to allow time for Project implementation.  Changes to the results framework (RF) (related to Component 1): o Introduction of a new PDO indicator to monitor the newly added capacity building activities; o Increase in the intermediate results indicator (IRI) targets: (i) for boreholes drilled: from 7 to 15; and (ii) for annual public engagement summaries planned for disclosure: from 6 to 20. Restructuring 2 (June 30, 2023) involved significant changes, effectively transforming the Project from piloting CCS operations focusing on research aimed at the geological characterization of the proposed carbon dioxide storage site. The main changes were as follows:  Revision of the Project Development Objective (PDO): the PDO was revised to remove the carbon capture development sub-objective, following the cancellation of Component 2, as detailed above. This also resulted in cancellation of US$13 million from the CCS-TF grant; and required amendments to the Project's legal documents.  Modifications of the results framework (RF): the RF was adjusted, including the removal of indicators that measure the achievement of results under the cancelled Component 2.  Reduction in the scope of the carbon storage development activities (Component 1): the investigation and design phases were kept, while the procurement and construction, as well as the operation phases were dropped. As a result, the storage development sub-objective could no longer be reached within the Project.  Extension of the closing date by eight months, from June 30, 2023 to February 29, 2024, to allow time for Project completion. Split evaluation. The Project was restructured twice, and Restructuring 2 involved a reduction in the Project’s scope, necessitating a split evaluation. 3. Relevance of Objectives Rationale Country and Sector Context. At Project appraisal, South Africa’s energy sector was dominated by coal, which was the main contributor to the country’s high emission intensity. In line with its climate change Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) mitigation strategy and energy security objectives, the government aimed to invest in low-carbon and clean energy, with CCS development being a part of the agenda. The potential for CCS had been investigated since 2004, including by the South African Center for Carbon Capture and Storage (SACCCS), a division within SANEDI. In 2012, the government endorsed a 20-year South African CCS Roadmap, prepared by SANEDI. Also, the government enacted legislation to provide incentives for investments in greenhouse gas (GHG) mitigation, including potentially for CCS. A draft Carbon Tax Bill was issued in 2015, and the Carbon Tax Act was enacted in 2017. The latter included a provision for carbon offsets, allowing companies to invest in GHG and CO2 mitigation projects to reduce their carbon tax liability. Relevance to Government Strategies during implementation and at closure. The Project was aligned with government strategies, including the South Africa’s objective to transition to a low carbon economy, a key objective outlined in the National Development Plan (NDP) 2030. At the sector level, the Project was aligned with the CCS Roadmap for South Africa, endorsed by the government in 2012 and comprised the following phases: (i) assessment of CCS potential; (ii) completion of CO2 storage atlas; (iii) CO2 test injection; (iv) CCS demonstration; and (v) commercial CCS application. By the time of Project appraisal in 2016, phases (i)-(ii) had been completed by the CGS. The Project was designed to support phases (iii)-(iv). South Africa’s latest Integrated Resource Plan (2023) stated the need for continued partnerships with international organizations and countries using cleaner coal technologies, citing the Project as an example. However, there were shortcomings in government support for the Project during implementation, given other sector priorities, as well as deficiencies in Project management and monitoring, which led to delays in implementation and the eventual removal of carbon capture activities and scaling down of carbon storge activities. Relevance to the WBG’s Assistance Strategies at closure. The Project was aligned with the WBG’s Country Partnership Framework (CPF) FY2022-26, which focused on the following strategic areas: (i) promoting increased competition and improving business environment for sustainable growth, (ii) strengthening micro, small, and medium enterprises, and supporting skills development to foster job creation, and (iii) enhancing the infrastructure investment framework and selected infrastructure services. The Project was also aligned with Objective 1.2 Greater Climate Change Resilience and Environmentally Sustainable Investments in Selected Sectors, which emphasized achieving a secure and sustainable electricity generation mix. Previous sector experience. A Multi-Donor Carbon Capture and Storage Capacity Building Trust Fund (CCS TF) was established in 2009 to support CCS capacity and knowledge building in developing countries. WBG used the CCS TF to support two phases of CCS development in South Africa. Phase 1 financed a review of the regulatory framework for CCS, techno-economic review of CCS implementation, capacity building for CCS, and the development of a national and local public engagement plan for the PCSP. Phase 2 financed two projects: a Programmatic Technical Assistance for Capacity Building for Carbon Capture and Storage (PTA, P151193); and the reviewed Project. The PTA was a 5-year program running in parallel with the reviewed Project, providing its analytical foundation, including data analysis and project planning for the PCSP and the pre-feasibility stage of the CCPP. The objectives were pitched at the correct level both at appraisal and at closure: CCS development was a key part of the government’s climate change mitigation strategy and was well aligned with the WBG country strategy; and the Project was designed to follow previous and ongoing CCSTF-financed support to South Africa. However, government support to the Project during implementation was insufficient, given other Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) sector priorities and shortcomings in Project management and monitoring. Therefore, Relevance of Objectives is rated as Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective PDO 1: To assess the feasibility of carbon capture and storage in South Africa. Rationale The theory of change (ToC) for the Project was not included in the PAD; it was prepared for the ICR (ICR, page 8). The ICR presented two ToCs: for the Original Project and for the Revised Project. Both ToCs showed causal links from Project activities to outputs/intermediate indicators, and then to outcomes/PDO indicators, and to the long-term goal of contributing to the country’s carbon mitigation strategy. The Original Project supported the following PDO1 activities: (i) for the pilot carbon storage facility: site selection, design, procurement, construction, and operation of the facility; and (ii) design of the carbon capture facility. The expected outputs of these activities were: (i) completion of seismic surveys, drilled wells, confirmed PCSP site, and disclosed public engagement summaries; (ii) completion of the CCPP FEED progress reports. The achievement of the outputs would ultimately result in the following outcomes: (i) completion of the assessment of carbon storage feasibility; (ii) completion and approval of the CCPP FEED. The Revised Project supported one PDO1 activity: site selection and design of the pilot carbon storage facility. The expected outputs of this activity were: completion of the seismic surveys, drilled wells, confirmed PCSP site, and disclosed public engagement summaries. There was one outcome: completion of the assessment of carbon storage feasibility. The ICR noted the following weaknesses of the ToC (ICR, page 9): (i) a disconnect between the PDO and the financed activities - while the PDO called for an assessment of the feasibility of both CO2 storage and capture, the funded carbon capture activities were limited to the FEED phase and did not include the construction and operation phases; (ii) the carbon capture activity outcome was set at an output level, as the Project funding covered only the design (FEED), while the pilot facility construction and operation (the intended outcome) depended on expected additional funding; and (iii) the carbon storage output indicators covered only the early stages of the activity chain and did not include the construction and operational stages which were essential to confirm the feasibility of storage. The ICR’s ToC provided a comprehensive description of the PDO1 logic, linking activities to outputs and then outcomes, and included a thorough analysis of the ToC/RF deficiencies. However, it had shortcomings. First, PDO2 was not included in the ToC. Second, critical assumptions were missing, though they were implicitly presented in the analysis of Project’s risks in other ICR sections, including Sector Context (page 7), Key Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Factors During Preparation (pages 20-22), Quality at Entry (page 24), and Lessons and Recommendations (page 25). The achievement of the Original Project’s results will be assessed using the indicators and indicator targets as specified during Restructuring 1 of June 2021, as follows: Outputs/Intermediate Outcomes: 1. “Cumulative Pilot Carbon Storage Project (PCSP) seismic surveys completed (Number)” (baseline: zero surveys, target: 2 surveys). The achievement at closure was 2 surveys; the target was reached. 2. “Cumulative PCSP boreholes logged (Number)” (baseline: zero boreholes, target: seven boreholes). The achievement at closure was 22 boreholes; the target was exceeded. 3. “PCSP site confirmed (Yes/No)”. The target was reached. 4. “Cumulative annual public engagement summaries disclosed (Number)” (baseline: zero summaries, target: six summaries). The achievement at closure was 20 summaries; the target was exceeded. 5. “Cumulative CCPP FEED progress reports completed (Number)” (baseline: zero reports, target: eight reports). The achievement at closure is unknown because this indicator was dropped at Restructuring 2. 6. “Cumulative PSCP geological profiles constructed (Number)” (baseline: zero profiles, target: five profiles). The achievement at closure was five profiles; the target was reached. PDO outcomes: 1. "Completion of the assessment of the feasibility of CO2 storage in the Govan Mbeki Local Municipality in South Africa (Yes/No)". The target was not reached. The ICR stated that at Project closure, only the site selection and design phases were completed, while the procurement, construction, and operations phases remained outstanding. The carbon storage activities under the Project did not progress far enough to establish the feasibility of carbon storage (in other words, to achieve PDO 1). However, the geological investigation of offshore basins and the site selection for a pilot storage facility were completed. (ICR, page 15) 2. “Completion and approval of the Carbon Capture Pilot Project (CCPP) FEED (Yes/No)”. The related activity (Component 2) was later dropped, and therefore this target was not reached. Rating. The Original Project barely achieved its intended Objective 1 results, and its efficacy rating is Negligible. Specifically, neither of the two PDO indicators was reached: the carbon capture activity was dropped with negligible achievement, and the progress with the carbon storage activity was insufficient to even establish the feasibility of CO2 storage development in the country and therefore to achieve Objective 1. Although the geological characterization of the basin and the storage site selection were completed, this alone was insufficient for assessing the feasibility of carbon storage. Such an assessment was necessary to achieve the PDO on CO2 storage, as outlined in the ToC. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Rating Negligible OBJECTIVE 1 REVISION 1 Revised Objective PDO 1: To assess the feasibility of carbon dioxide storage in South Africa. Revised Rationale Please see the discussion of the ToC under Objective 1, Original Project. The achievement of the Revised Project’s results will be assessed using the indicators and indicator targets as specified during Restructuring 2 of June 2023, as follows: Outputs/Intermediate Outcomes: 1. “Cumulative Pilot Carbon Storage Project (PCSP) seismic surveys completed (Number)” (baseline: zero surveys, target: 2 surveys). The achievement at closure was 2 surveys; the target was reached. 2. “Cumulative PCSP boreholes logged (Number)” (baseline: zero boreholes, revised target: 15 boreholes). The achievement at closure was 22 boreholes; the target was exceeded. 3. “PCSP site confirmed (Yes/No)”. The target was reached. 4. “Cumulative annual public engagement summaries disclosed (Number)” (baseline: zero summaries, revised target: 20 summaries). The achievement at closure was 20 summaries; the target was reached. 5. “Cumulative PSCP geological profiles constructed (Number)” (baseline: zero profiles, target: five profiles). The achievement at closure was five profiles; the target was reached. PDO outcomes: 1. "Completion of the assessment of the feasibility of CO2 storage in the Govan Mbeki Local Municipality in South Africa (Yes/No)". The target was not reached. The ICR noted that to assess the feasibility of CO2 storage (PDO1), it was necessary to bring the pilot storage facility to the stage of injection, operation, and monitoring, which was not achievable (ICR, page 16). Rating. The Revision 1 Project did not reach its PDO indicator target under Objective 1: it was unable to assess the feasibility of CO2 storage because progress towards it was insufficient. However, the first milestones were reached (geological investigation of offshore basins, and site selection for pilot storage facility), which is reflected in the achievement of the IRIs. Therefore, the efficacy rating is weak Modest. Revised Rating Modest Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) OBJECTIVE 2 Objective PDO 2: To build expert capacity for carbon capture and storage in South Africa. Rationale PDO2 was not included in the ICR’s ToC. The achievement of the Original Project’s results will be assessed using the indicators and indicator targets as specified during Restructuring 1 of June 2021, as follows: Outputs/Intermediate Outcomes: There were no output/intermediate outcome indicators under Objective 2, Original Project. PDO outcomes: 1. “Cumulative South African professionals trained and hired/contracted to work on the Pilot Carbon Storage Project (PCSP) (Number)” (baseline: ten professionals, target: 30 professionals). The achievement at closure was 64 professionals; the target was exceeded. The gender specific target (females trained and hired) was also exceeded. However, it is worth noting that this PDO indicator is output oriented. Rating. The Original Project partially achieved its intended Objective 2 results, and its efficacy rating is Modest. While the capacity for carbon storage was increased, there is no evidence of changes in the capacity for carbon capture, as it was not measured. Since all carbon capture activities under the Project were dropped at Restructuring 2 of June 2023, it is logical to conclude that carbon capture capacity building was not accomplished. Also, the PDO indicator on carbon storage under Objective 2 is output/immediate outcome oriented, which further weakens the results. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective PDO 2: To build expert capacity for carbon dioxide storage in South Africa. Revised Rationale PDO2 was not included in the ICR’s ToC. The achievement of the Revised Project’s results will be assessed using the indicators and indicator targets as specified during Restructuring 2 of June 2023, as follows: Outputs/Intermediate Outcomes: Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) There were no output/intermediate outcome indicators under Objective 2, Revised Project. PDO outcomes: 1. “Cumulative South African professionals trained and hired/contracted to work on the Pilot Carbon Storage Project (PCSP) (Number)” (baseline: ten professionals, target: 30 professionals). The achievement at closure was 64 professionals; the target was exceeded. The gender specific target (females trained and hired) was also exceeded. Rating. While the Revision 1 Project exceeded its intended Objective 2 result, the indicator measuring it (people trained or hired/contracted) was at the output/immediate outcome level. Therefore, the Objective 2 efficacy rating under Revised Project is Substantial, though marginally so. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Original Project: For the Original Project, the rating for efficacy is Negligible. The results on assessing the feasibility of carbon capture and storage (Objective 1) were barely achieved, specifically: (i) the carbon capture activities were not implemented (dropped at Restructuring 2); and (ii) the carbon storage targets were barely achieved: while specific milestones were reached, such as the geological characterization of offshore basins, and the selection of a site for the pilot storage facility, this was insufficient to reach PDO 1 on assessing the feasibility of carbon storage. The results on building capacity for carbon capture and storage (Objective 2) were partially reached: (i) there is no evidence that carbon capture capacity was increased; and (ii) while the indicator for carbon storage capacity was exceeded, it is output/immediate outcome-oriented, providing weak evidence of the achievement of the intended result. Objective 1 was rated as negligible and objective 2 as weak modest, resulting in an overall Negligible rating. Overall Efficacy Rating Primary Reason Negligible Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale For the Revision 1 Project, the rating for efficacy is Modest. The PDO results on assessing the feasibility of carbon storage (Objective 1) were not achieved, while the PDO results on capacity building for carbon storage (Objective 2) were achieved even though it was at output/intermediate result level. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Overall Efficacy Revision 1 Rating Primary Reason Modest Low achievement 5. Efficiency a. Economic Analysis: At appraisal, an assessment of GHG emissions removal from the atmosphere, due to the Project, was conducted. The assessment used metric tons of GHG emissions and emissions removal as the unit of analysis; these values were not monetized, and no costs-benefits analysis, or Project’s EIRR were produced. The removal of GHG emissions was assessed at 16,000 metric tons per year (gross) of 14,150 metric ton per year (net of emissions produced due to the operations of the Project-financed carbon storage facility). At closure, the assessment of GHG emissions removal produced at appraisal was replicated for the Project as modified at Restructuring 1 (the only change made was in the location of the storage facility site). The removal of GHG emissions was assessed at 16,000 metric tons per year (gross) or 14,527 metric ton per year (net of emissions produced due to the Project-financed carbon storage facility operations). The net value at closure was slightly higher than at appraisal because the costs of CO2 transportation were reduced due to the re-location of the storage site closer to the sources of emissions. Identical to the appraisal level assessment, the post-closure assessment did not include monetization of the GHG removals and did not produce cost-benefit analysis or EIRRs. While no estimates of Project’s economic or financial benefits were produced at appraisal or closure, the ICR noted that “the current carbon tax in South Africa at US$10.29/metric ton is substantially below what would be required to provide incentives for the private sector to invest in CO2 capture and storage facilities given the actual costs to the FEED stage (US$11 million) and the estimated costs which are likely to be more than double the actual costs to date according to CGS’s broad estimates”. (ICR, page 17) b. Administrative Efficiency: Project implementation was expected to take 57 months (or four years and nine months), but it was delayed by 26 months (or two years and two months). At the same time, the activities were significantly scaled down. The main factors contributing to the delays were as follows: (i) a long period between the Project’s approval and effectiveness, caused by external factors, such as government inter-agency communication and delayed approvals of implementation actions; (ii) the impact of COVID-19; (iii) delayed procurement for safeguard instruments for the new carbon storage facility site; and (iv) discontinuities after the change in the implementing agency from SANEDI to CGS (see more details in section 8.b. Quality of Supervision). The Project also faced significant procurement and financial management issues, which led to delays and the need to scale down carbon storage activities. The ICR mentioned that these issues could have been mitigated by better due diligence on the part of the WB team during the transition of implementation arrangements from SANEDI to CGS. In addition, Eskom’s focus on top sector priorities (blackouts and increase in generation) led to reduced attention to the carbon capture component, resulting in its cancelation. However, it is worth noting that the Project was recognized as high-risk at appraisal, as it was the first project globally supported by a multilateral development agency to pilot CO2 capture and storage in a developing country. Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Considering the absence of estimates for EIRRs or cost-benefit analysis, and also taking into account delays and cancellations in the context of reduced scope, the Project’s efficiency is rated as Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original Project: Revised Project: Relevance of objectives Substantial Efficacy Negligible Modest Efficiency Modest Outcome Unsatisfactory Moderately Unsatisfactory Outcome value 2 3 Amount disbursed, US$ million 4.97 0.0 Disbursement percentage 100.0% 0.0% Weight value 2.0 0.0 Total weight 2.00 Overall outcome rating Unsatisfactory The overall Relevance of objectives is rated as Substantial. Original Project’s efficacy is Negligible, and Revised Project’s efficacy is Modest. Efficiency is rated as Modest. Thus, the overall outcome is rated as Unsatisfactory for the Original Project and Moderately Unsatisfactory for the Revised Project. Based on the shares of the disbursed funds before and after Restructuring 2 of June 2023 (US$4.97 million or 100.0 percent and US$0.0 million or 0.0 percent), the overall Project outcome rating is Unsatisfactory* (1.0 * 2 + 0.0 * 3 = 2.0)*. * Based on a six-point scale, where: 1 = Highly Unsatisfactory, 2 = Unsatisfactory, 3 = Moderately Unsatisfactory, 4 = Moderately Satisfactory, 5 = Satisfactory, and 6 = Highly Satisfactory. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome Government ownership or commitment. The Project was negatively affected by the priorities in the energy sector, particularly the focus on increased generation to avoid electricity blackouts. There is a risk that this issue will continue to dominate energy sector in the near future, making the completion of the Project a lower priority. However, the development of CCS is a key action item in the country’s long term climate change mitigation strategy, which places it higher on the government's priority list. Financial. While substantial progress was made towards piloting the CO2 storage and preparing the FEED for the capture facility, the risk exists that concessional funding may not be available to finance the remaining steps of both pilots, and that government financing may not be provided considering other priorities. At the same time, the importance of CCS development in the country’s climate change mitigation strategy increases the likelihood that budgetary resources will be allocated for these tasks. Institutional. The Project was negatively affected by constrained institutional capacity for its implementation and sustainability. While the project attempted to mitigate this risk through capacity-building activities, it remains a concern. Technical (innovative technologies). The Project was designed to pilot innovative technologies, a task which involves a significant level of risk. While the initial phases of the carbon storage pilot implementation were accomplished, the task remains to be completed after Project closure. Although substantial learning took place during Project implementation, providing a basis for mitigating these risks, international expertise will be required to complete the pilot carbon storage project. 8. Assessment of Bank Performance a. Quality-at-Entry The Project’s design was generally adequate and accounted for multiple risks associated with this innovative Project, including the following. To mitigate the risk of introducing innovative technologies, international experts were hired, and expert assessments of results at each phase of implementation were used to approve the next phase. To mitigate the risk of limited experience with implementing WB projects, capacity building support was provided. Environmental risks, such as CO2 leakage and induced seismicity, were mitigated by keeping the pilot small, using rigorous site selection, and adhering to best practices in monitoring and operation. Stakeholder risks were minimized through early and continuous engagement with stakeholders (ICR, pages 20-21, 24). However, design deficiencies were noted, specifically, the weak alignment between the PDO, the components, and the RF, as described in section 9.a. M&E Design. Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Project implementation documents, including aide memoires, were issue oriented and substantive, with several ISRs filed during implementation (ICR, page 24). After Restructuring 2 of June 2023, the Project scope was significantly reduced. The carbon capture component was dropped. Under the carbon storage component, only the investigation and characterization of a suitable site, along with the schematic and detailed engineering design for the carbon storage facility, remained. These activities were fully implemented, while the procurement, construction, and operation phases were removed. As a result, the storage development sub-objective could no longer be reached within the Project; however, it was not revised. Implementation was negatively affected by both external and internal factors, leading to a 26-month delay and a significant reduction in the Project scope (with carbon storage activities scaled down and carbon capture activities dropped). External factors included: (i) Eskom’s focus on its top priority of resolving power blackouts and increased generation output at Kusile plant, which led to delays and eventual cancelation of the CCPP activities; (ii) challenges SANEDI faced in securing attention from the Department of Mineral Resources and Energy (DMRE), resulting in the transfer of the implementing agency function from SANEDI to CGS; (iii) difficulties during the transfer process, including discontinuities in Project management and the use of important international experts and advisory panels; (iv) the impact of COVID-19; and (v) procurement delays in hiring consultants to prepare new safeguard instruments following the relocation of the carbon storage site, which led to a scale-down of carbon storage activities. The internal factors included: (i) insufficient due diligence by the Bank team in supporting the transition of implementation arrangements from SANEDI to CGS, leading to CGS’s unfamiliarity with Bank financial management and procurement requirements, delayed procurement, and a scale-down of the carbon storage activities; (ii) delayed Bank procurement decisions/responses (as indicated by CGS), affecting the competitiveness of some bids; and (iii) rigid and time-intensive Bank administrative procedures (as perceived by CGS), especially during Project restructuring and in relation to safeguard instruments. (ICR, pages 21-22, 24). Since the Bank Performance at Entry is rated as Moderately Satisfactory, and the Bank performance at Supervision as Moderately Unsatisfactory, the overall Bank performance is rated as Moderately Unsatisfactory. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) a. M&E Design The RF reflected the logic of the Project's interventions as planned at appraisal and was logical and manageable. All RF indicators were time-bound and attributable to the Project, and most of them were quantitative. However, the RF had deficiencies, including weak alignment between the PDO, the components, and the RF, as follows: (i) while the PDO included a sub-objective to assess the feasibility of carbon capture, the financed activities – the engineering design of carbon capture facilities - could not support such an assessment, as this would require the construction and pilot operation of the facilities, which was the next phase of the overall Project; (ii) the RF was skewed towards measuring carbon storage results, while carbon capture results were measured by only one (PDO) indicator for Objective 1 and none for Objective 2; and (iii) the output/intermediate outcome indicators for the feasibility of carbon storage only covered the geological characterization and site selection phases, neglecting subsequent phases of design, construction, and operation of the storage facility. (ICR, pages 20-21) At Project appraisal, SANEDI was responsible for Project’s M&E, including conducting weekly team meetings, monthly meetings with contractors, quarterly steering committee meetings, and producing annual progress reports. The PCSP and the SACCCS steering committees were tasked with overseeing the PCSP and CCPP FEED components respectively and reviewing implementation on a monthly or quarterly basis. Several subcommittees and advisory committees were also established to provide additional M&E support. (ICR, page 22) b. M&E Implementation The ICR reported that following transfer of the Project to CGS, M&E was not fully formalized between the Bank and CGS, and progress reports on Project implementation were not provided regularly. The Steering committees and advisory panels were replaced by the CGS Project team, with reporting oriented towards CGS Board requirements and experts being brought in as needed. CGS’s internal reporting became less visible to the Bank, as were the decision criteria for transiting from one Project phase to the next. (ICR, pages 22-23) c. M&E Utilization The ICR noted that monitoring and evaluation reports provided the basis for key decisions taken by the CGS management and Board. Although some monitoring reports were provided to the Bank, they were irregular. The gaps in reporting were filled by frequent discussions with the Bank team, as reflected in mission aide memoires and Implementation Status and Results Reports (ISR). (ICR, page 22) Considering the shortcomings in RF design, and the gaps in the M&E reporting following the transfer of implementation responsibilities to CGS, the M&E Quality is rated Modest. M&E Quality Rating Modest 10. Other Issues Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) a. Safeguards Environmental and Social Safeguards. At appraisal, the Project was classified as Environmental Category A (Full Assessment). The following safeguards policies were triggered: Environmental Assessment OP/BP 4.01 and Natural Habitats OP/BP 4.04. The following Safeguards instruments were prepared: preliminary Environmental and Social Impact Assessment (ESIA) and Environmental and Social Management Plan (ESMP) for the original PCSP site, and preliminary ESIA with ESMP for CCPP (the latter was prepared by Eskom). The ESIAs and ESMPs were disclosed by the Bank in 2016. After the change in the PCSP site at Restructuring 1 in June 2021, new ESIA and ESMP had to be prepared. The contracts for the PCSP ESIA were awarded in December 2022, and environmental authorization was received by September 2023. The potential impacts and mitigation measures included: (i) to mitigate groundwater contamination, boreholes were drilled around the site to assist with early detection; (ii) to protect the gravesites, culturally appropriate measures were to be taken; (iii) air, water, and soil quality was to be monitored by CGS. Safeguards were rated Moderately Satisfactory in the final ISR (December 5, 2023) because the ESIAs and ESMPs had been completed and authorized. (ICR, page 25) b. Fiduciary Compliance Financial management (FM). The ICR noted that the appraisal-level financial management assessment of SANEDI concluded that it had an adequate governance structure to comply with the country’s financial management requirements and its own financial policies; and to reduce the risk of fraud and corruption. SANEDI was required to submit quarterly interim financial reports (IFRs) to the Bank, along with audited financial statements, within six months of the end of each financial year. However, CGS faced delays and difficulties in preparing the IFRs due to insufficient familiarity with the Bank’s requirements - issues that should have been addressed during Restructuring 1. The Project’s FM rating was Moderately Satisfactory in the final ISR archived on December 5, 2023, with no outstanding audits or IFRs at Project closure. (ICR, page 24) Procurement. The ICR highlighted several risks identified during the project appraisal, including delays in procurement processing delays and poor quality of bidding documents, which ultimately affected the quality of the outputs. To mitigate the risks, procurement training was conducted during Project preparation. However, both SANEDI and CGS experienced deficiencies in procurement management. As of December 2022, only three contracts had been awarded across seven activities. An example of delayed procurement was the selection of consultants for the Revised Project’s PCSP’s ESIA, which was completed 18 months after Restructuring 1. The key factors included CGS ‘s unfamiliarity with project management and the WBG procurement procedures, unclear roles and accountability, and low thresholds for Board approval of contracts. As a result of these delays and the limited number of contracts awarded and implemented, procurement was rated as Unsatisfactory in the latest ISR, dated December 5, 2023. (ICR, page 23) c. Unintended impacts (Positive or Negative) --- Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Both the Original and Revised Projects barely achieved Moderately Outcome Unsatisfactory Objective 1 targets on carbon Unsatisfactory storage and did not achieve the targets on carbon capture. Moderately Moderately Mainly implementation Bank Performance Satisfactory Unsatisfactory shortcomings. Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The following lessons were derived from the ICR (ICR, pages 25-26): 1. Engaging and supporting clients in piloting new technologies involves significant risks, but such activities bring considerable benefits in generating knowledge and developing capacity, thus building a foundation for technology adoption. The Project did not achieve a full implementation of the pilot carbon storage facility component due to delays and the subsequent cancellation of the construction and operation phases. However, the completion of its first phase - comprising geotechnical investigations, site selection, and CO2 storage facility design -has provided insights into the potential of the country’s geological formations to store CO2 emissions, forming a strong foundation for the next phases: construction, operation, and monitoring of the storage facility. 2. Changing project implementation arrangements mid-stream requires careful consideration to ensure that the project management systems are either retained or replaced by adequate systems, and that the new implementing agency is brought up the learning curve and provided with resources. In the Project, the risk of limited institutional capacity for Project implementation and sustainability was mitigated at appraisal, including by using international advisory committees and implementation consultants. However, the transfer of the Project to CGS did not include sufficient capacity building. As a result, CGS used its own internal processes for approvals. Shortcomings in procurement processes and financial management - including difficulties in preparing interim financial reports (IFRs) and awarding contracts - led to significant delays and the scale-down of Component 1. 3. The design of CCS projects would benefit from considering a range of sectors, including industry, rather than focusing on coal and the public sector. The management of CO2 capture and storage has increasingly shifted to industry globally. Hence, in the future, South Africa could benefit from consultations with industrial emitters when selecting sites for CO2 injection. Additionally, Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Development of CCS in South Africa (P149521) the country could explore the conditions and incentives that would increase the uptake of CO2 management activities by a range of sectors. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides a wealth of technical details to understand the value-added of the Project; the context of implementation, the factors of its performance; main outcomes; and various aspects of Project implementation, including efficacy, administrative efficiency, M&E quality, and safeguard and fiduciary compliance. The ICR is analytical and internally consistent, and the lessons are thoughtful and applicable to similar innovative pilot projects across client countries. However, there is a minor shortcoming: the information on the counterpart contribution and on Project costs by financing source at closure is inconsistent. On balance, considering the high-quality technical information and analysis provided, the rating is Substantial. a. Quality of ICR Rating Substantial Page 18 of 18