Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) Report Number: ICRR0023407 1. Project Data Project ID Project Name P130013 KAZAKHSTAN ENERGY EFFICIENCY PROJECT Country Practice Area(Lead) Kazakhstan Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-14185 30-Jun-2017 21,740,760.58 Bank Approval Date Closing Date (Actual) 22-May-2013 30-Jun-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 21,763,000.00 21,763,000.00 Revised Commitment 21,763,000.00 21,740,760.58 Actual 21,740,760.58 21,740,760.58 Prepared by Reviewed by ICR Review Coordinator Group Maria Shkaratan Joel J. Maweni Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) is to improve: (a) energy efficiency in public and social facilities; and (b) the enabling environment for sustainable energy financing (Grant Agreement, page 7). The PDO was stated identically in the PAD. The PDO was not revised. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) For the purposes of this Implementation Completion Report (ICR) review, the objective will be assessed as follows: PDO1: To improve energy efficiency (EE) in public and social facilities. PDO2: To create enabling environment for sustainable EE financing. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-Sep-2020 c. Will a split evaluation be undertaken? Yes d. Components 1. Original components Part 1 Development and Implementation of Demonstration Subprojects in Public and Social Facilities (cost at appraisal: US$18.0 million; actual cost: US$15.08 million) was to invest in demonstration EE subprojects for: (i) public and social facilities such as schools, kindergartens, and healthcare facilities; and (ii) street lighting. Subproject types. The selection of the subprojects would be competitive and subject to the following: (i) proof of structural soundness; (ii) no plans for closure, downsizing, or privatization; (iii) a payback period of up to eight years; (iv) an energy audit conducted within the previous three years; (v) allocation of financing per oblast not to exceed US$1.5 million; and (vi) subprojects to cost no less than US$50,000 and no more than US$150,000 for schools/kindergartens, US$250,000 for hospitals, and US$500,000 for street lighting. Technical EE measures would include building envelope measures such as insulation, repair, or replacement of doors and windows; heating and cooling systems upgrade and replacement; fuel switching; reflective surfacing behind radiators; control systems; pipe insulation; chiller or A/C replacement; and such energy-using systems as heat pumps, lighting, pump fans, and solar water heaters. Various business models, contracting models, and financing models would be evaluated to identify viable approaches for the future scale-up, including through private sector involvement. Part 2 Technical Assistance (cost at appraisal: US$3.8 million; actual cost: US$ 6.7 million) aimed to provide: (i) project implementation support; (ii) technical studies (EE market assessments, energy audits, and pilot oblast EE Master Plans for replication in the other regions, and other, identified during implementation); (iii) awareness, outreach, and information campaigns; (iv) legal, institutional, and Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) regulatory reviews and workshops, with a focus on financing options, ESCOs, and constraints to meeting the Government EE targets; and (v) design of a viable financing mechanism (such as an EE Fund). 2. Changes in components during implementation The Project’s components remained unchanged during implementation. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The appraisal estimate was US$21.76 million, and the actual Project cost was US$21.74 million. The undisbursed amount of US$22,240 was canceled from the Grant Account at closure (World Bank letter dated April 6, 2023). Project Financing: The Project was financed by a trust fund (TF) grant from the Government of Switzerland, Swiss Agency for Development and Cooperation (SDC), channeled through a Trust Fund set up between SDC and the World Bank (Free-standing TF for Europe and Central Asia (ECA)). Borrower/Recipient contribution: The Borrower’s contribution was to be in-kind and was valued at US$1.3 million at appraisal. Because the contribution was not tracked, it was not included in actual component costs (ICR pages. 9 and 10) or actual total Project costs (ICR data sheet), but was included in total costs in Annex 3 (ICR, page 46). The team informed IEG that this number is very reasonable, especially considering the increase in the number of sub-projects for which the beneficiary contribution was used to prepare and supervise the subproject. Project Dates: The Project was approved on May 22, 2013 and became effective on June 1, 2015. The mid- term review was held on October 27, 2014. The original closing date was June 30, 2017. The Project was extended four times for a cumulative period of five years (or 60 months) to June 30, 2022. Project Restructurings: The Project underwent five restructurings, which involved the following changes in indicators’ targets: i. Reduced targets for three indicators: o For the PDO indicator “Quantified Energy Savings, GWh”: from the original 825 GWh to 690 GWh (2017 restructuring), and to 620 GWh (2019 restructuring). The explanation in the ICR was that the revision was based on the results of the completed investments, which yielded energy savings lower than expected at design. o For the related intermediary indicator “CO2 emission reductions in retrofitted facilities through EE investments, metric tons”: from 400,000 tons to 310,000 tons (2017 restructuring) and further to199,000-ton CO2 (2019 restructuring) based on both the reduced target of the energy (as noted above) and an updated (lowered) by the International Energy Agency (IEA) emission factor. o A scale-down of the PDO indicator, from “Development of a sustainable energy financing mechanism, New financing mechanism launched” to “Program design to scale-up EE investment in public sector is submitted to the Government for approval” (2019 restructuring). This was because the term “financial mechanisms” was used improperly and did not reflect Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) the Project’s expected outputs and outcomes. This error in formulating the RF PDO indicator was corrected during the September 2020 restructuring. ii. Savings due to currency devaluation and efficient contract management resulted in: o A decreased target of the intermediary indicator “Cumulative investments in public and social facilities”: from US$19,000,000 to US$18,500,000 (during the 2017 restructuring); to US$16,450,000 (during the 2019 restructuring); and to US$15,950,000 (during the 2021 restructuring). o An increased target of the intermediary indicator “Number of subprojects commissioned in public and social facilities”: from the original 75 sub-projects to 85 sub-projects (during the 2019 restructuring). This led to an increase in the number of Project’s beneficiaries. o Reallocation of financing from Part 1 to Part 2. (US$0.75 million). 3. Relevance of Objectives Rationale Country and Sector Context: Kazakhstan’s fast economic growth in the years after the 2008-2009 global financial crisis was linked to a sharp increase in energy and electricity consumption and a tightening of the energy demand-supply gap. Energy demand was projected to further increase by at least 50 percent by 2035 (IEA). Kazakhstan was among the top 10 most energy-intensive and carbon intensive economies globally, and the economy was dominated by energy intensive industries. At the time of Project approval, it was recognized that high energy intensity negatively affects the country’s competitiveness globally and creates barriers to economic development. The country’s energy savings potential was estimated at US$1.3 billion per year, and there was a large potential for energy efficiency improvements in all economic sectors. Public and residential sectors were important because they accounted for 55 percent of the country’s heat consumption and 20 percent of electricity consumption. Relevance to Government Strategies at approval. By Project appraisal, the Government had recognized that EE development was critical to prevent the growth-slowing energy shortages, improve industrial competitiveness and environmental performance, and address the social impact of the increase in domestic energy prices. In March 2010, the Government targeted a 10-percent decrease in energy intensity of GDP by 2015 and 25 percent by 2020. In January 2012, a newly approved Energy Efficiency Law defined an effective legal, regulatory, and institutional framework for EE, and an action plan: Comprehensive Program for Energy Efficiency (CPEE) followed. The CPEE focused on the sectors with the lowest energy efficiency, including industry and the municipal/residential sector. Mechanisms in the CPEE included fiscal incentives, standards and codes, awareness raising, state budget allocations with private sector leverage, and the creation of a National Energy Savings Fund. About 70 percent of the public and residential buildings would require retrofitting to comply with the new thermal efficiency standards. The CPEE also called for mandatory energy audit of all public buildings and the application of EE performance criteria to major building renovations. The Project supported the new EE policies by providing advisory policy and regulatory support; capacity building related to the design, implementation, and financing of EE initiatives; and demonstration investments in EE. Relevance to Government Strategies at closure. Kazakhstan’s Strategic Development Plan to 2025 incorporated the Green Economy Concept, including the goal to reduce energy intensity by at least 25 percent from the 2008 levels by 2025, and 50 percent by 2050. Improved energy efficiency, as in the Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) Project, was integral to the achievement of this result, and the Strategic Development Plan to 2025 targeted a 15 percent decrease in energy consumption in the public sector (including housing and communal services). The country had also enacted comprehensive legislation (the Law on Energy Saving and Energy Efficiency, 2012) to help reduce energy intensity of GDP. Relevance to the World Bank Group's (WBG’s) Assistance Strategies at approval. The Project was aligned with the WBG’s Country Partnership Strategy (CPS) FY12–FY17, specifically the third engagement area focused on environmental sustainability of the economy, for which the key outcome was “Cumulative energy savings in targeted public facilities with an increase from 0 to 825 GWh by 2017”. Relevance to the World Bank Group's (WBG’s) Assistance Strategies at closure. The Project was aligned with the Country Partnership Framework (CPF) FY20–FY25, which supported energy efficiency as a key objective under focus area 3 aimed at securing sustainable, resilient, and low-carbon growth. The Kazakhstan Systematic Country Diagnostics (2018) found addressing environmental sustainability critical for competitiveness and, thus, supported investment in EE. The relevance of objectives is rated High because they were aligned with the energy efficiency goals and targets of the Government’s Strategic Development Plan to 2025 and the Green Economy Concept, as well as with the related legal framework. In addition, supporting investments in EE was a key objective of the WBG’s CPF FY20-FY25, while energy savings in public facilities was a key outcome of the CPS FY12-17. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve EE in public and social facilities. Rationale The theory of change (ToC) for the Project was developed for the ICR. It showed a direct, logical causal link from inputs to outputs, to intermediate outcomes, and to PDO outcomes of this Project. The Project-financed EE subprojects were intended to demonstrate energy and GHG emissions savings (intermediary outcomes); while policy and institutional development would result in development of a sustainable EE financing mechanism and legal and regulatory support to future EE investments (intermediate outcomes). Thus, improvements in energy efficiency would be demonstrated and enabling environment for EE improved (PDO outcomes). Both the PAD and the ICR stated that PDO1 was to improve energy efficiency in public and social facilities and the PDO1 outcome indicator was quantified energy savings (GWh). However, according to the logic of this Project, implicit in the PAD, quantified energy savings (from demonstration subprojects) were one of Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) several intermediate indicators rather than a direct measure of energy efficiency improvements. Another deficiency of PDO1, as formulated in the PAD, was that it was pitched at a level above the Project’s actual scope and ambition since the subprojects, which were to be selected based on bidding and therefore not known in advance, could only demonstrate the potential for EE benefits rather than improve EE in public and social facilities. The TOC, as outlined in the ICR (page 8), did not specify a PDO1 outcome indicator and target but instead simply added “demonstrated” in parenthesis to the PDO1 outcome of improving energy efficiency in public and social facilities without stating how these would be measured. The TOC, thus, implies that PDO1 was to demonstrate the benefits of EE subprojects in public and social facilities which differs from achieving “improvements in EE in such facilities.” The TOC, nevertheless, correctly identified energy savings and GHG emission reductions through subprojects as intermediate indicators, amongst others. IEG’s review of the PAD and ICR, and discussions with the task team confirmed that the real intended PDO1 was to demonstrate the potential benefits of EE projects in public and social facilities rather than to improve energy efficiency in such facilities. IEG’s evaluation was carried on the basis of PDO1 as explicitly stated in the Grant Agreement and PAD and as inferred from the ICR TOC, and from discussions with the task team. Both approaches yielded the same efficacy conclusions. Another TOC deficiency is related to PDO2: the ToC does not fully reflect the implicit ToC in the PAD. The latter lists the measures to create enabling environment for sustainable EE financing (as reflected in the ICR’s ToC), but also states that the Project would support the “design of a sustainable energy financing mechanism (e.g., EE Fund) with…implementation plan” (project description, Part 2: PAD, page 7), which was unrealistic considering below cost recovery energy prices, low level of electricity metering, disincentives for commercial banks to invest in EE, and other barriers. By omitting this expected outcome, the ICR’s ToC effectively reflected the Revised Project under PDO2. Outputs and intermediate indicators: - At closure, total investments in subprojects amounted to US$16,060,000, below the target US$19,000,000. The reasons were: (i) efficient project management and (ii) currency devaluation. - By closure, the Project had implemented 96 EE subprojects in public and social facilities, against the original target of 75 subprojects. The target was exceeded. It was possible to invest in a larger number of subprojects due to: (i) efficient project management and (ii) currency devaluation. This was the most critical indicator under PDO1, directly supporting the objective of improving EE in public and social facilities through demonstration sub-projects. At IEG’s request, the Project team confirmed that the size of the implemented subprojects was on par with the expectations at design, so this result shows that the total volume of work was above the expectations. - By closure, 53,625 people benefited from the completion of the EE subprojects, against the original target of 45,000 people. The target was exceeded. The beneficiaries were defined as the staff and service recipients of the beneficiary facilities. This indicator is linked to the indicator on the number of delivered sub-projects. - By closure, energy savings from the sub-projects (a PDO indicator) amounted to 719 GWh, as compared to the original target of 825 GWh. At IEG’s request, the team provided the following information to clarify how the original target was estimated. This original indicator’s target was based on a priori estimates, prior to the experience of actual sub-project implementation. By design, the subprojects were to be selected based on Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) competitive bidding. Therefore, it could not be known beforehand what kind of EE improvements would be implemented and, therefore, what level of energy savings would be achieved. Once the sub-projects were delivered, the actual data for savings became available and was used to adjust the targets for this indicator at restructurings. IEG notes that based on the above information, this indicator was effectively designed as adjustable and should be evaluated together with the indicator of the number of sub-projects implemented. Further, it was an RF design error to use this indicator at the PDO level: according to the ToC (constructed for the ICR) and to the design of Project activities as reflected in the PAD, this is an intermediate outcome indicator. - By closure, avoided GHG emissions from the EE subprojects amounted to 306,105 metric tons, against the target of 400,000 metric tons. The target was not achieved. This indicator is linked to the energy savings indicator. Overall, under PDO1, as the review of the efficacy of intermediate indicators/outputs indicates, the Project achieved substantial progress towards improved energy efficiency. This is reflected in the progress on energy savings - the closest indirect measure of improved energy efficiency – and in the achievement of the most important expected result for demonstrating potential benefits of EE subprojects – the number of implemented EE sub-projects. While the target of expected energy savings was not fully reached, it should be noted that this indicator was effectively designed as adjustable target and should be treated as such and evaluated together with the indicator of the number of sub-projects implemented. Therefore, this non-achievement is considered a minor shortcoming. Rating. The Project achieved its original PDO1 objective, with a minor shortcoming, and therefore its efficacy is rated Substantial. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective To improve EE in public and social facilities. Revised Rationale Please see the discussion of the TOC under objective 1, Original Project. Outputs and intermediate indicators: - The Project delivered 96 EE subprojects in public and social facilities, against the revised target of 85 subprojects. The target was exceeded, as in the Original Project. - Total investments in subprojects amounted to US$16,060,000, against the revised target of US$15,950,000. The target was exceeded, as opposed to the Original Project. Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) - The number of beneficiaries was 53,625 people, against the revised target of 48,620 people. The target was exceeded. - By closure, energy savings (a PDO indicator) amounted to 719 GWh, as compared to the original target of 620 GWh. The target was exceeded. - By closure, avoided GHG emissions from the EE subprojects amounted to 306,105 metric tons, against the target of 199,550 metric tons. The target was exceeded. Overall, under PDO1, the Revised Project exceeded all its expected outputs and intermediate outcomes. The PAD outcome indicator of energy savings was exceeded and so was the number of subprojects which was an implied outcome indicator for the TOC implied PDO outcome of demonstrating the potential for EE improvements in public and social facilities. Rating. Although the revised Project exceeded all its intermediate indicators PDO1 is rated Substantial for efficacy because of the indirect link of the indicators to the stated PDO outcome. Revised Rating Substantial OBJECTIVE 2 Objective To create enabling environment for sustainable EE financing. Rationale Please see the discussion of the TOC under objective 1, Original Project. Outcomes: - At approval, the Project aimed at addressing multiple regulatory, policy, and institutional barriers to creating a functional and sustainable financing mechanism for EE investments. The activities included “EE market assessments, energy audits, and pilot oblast EE Master Plans for replication in other regions”, “legal, institutional and regulatory reviews and workshops, with a focus on sustainable financing options”, and “design of a sustainable energy financing mechanism (e.g., EE Fund) with…implementation plan” (project description, Part 2: PAD, page 7). The Project made significant progress with developing a framework that would form a basis for EE development and for replicating the experience with the sub-projects in the industrial sector (which has economic incentives to seek funding for demand-side EE) (source: discussion with the task team leader (TTL)). The ICR reported (ICR, page 15) that due to the Project, a new government framework was being developed for EE in the industrial and building sector. Additionally, the World Bank has broadened the scope of engagement with the Government on electricity tariff reform and on greener development. The Project was also a pioneer in engaging with local and provincial governments on the EE agenda. However, with below cost recovery energy prices, limited capacity to conduct energy audits, and low incentives of commercial banks to invest in EE (among other barriers), there was no realistic expectation that Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) a sustainable national-level financing mechanism for EE could be fully developed and ready for launching based on the implementation of project activities as described in the PAD. Thus, the expected outcome reflects a weakness of the Project design (i.e., a deficiency of the implicit ToC in the PAD), mirrored in the following PDO-level indicator in the PAD’s RF: “Development of sustainable energy financing mechanisms” (target: “new financing mechanism launched”) (PAD, page 17). This design deficiency was corrected during the September 2020 restructuring when the indicator and target were revised to: “financial model developed to evaluate investment and structure financing for EE in public sector”, and “program design to scale-up EE investment in public sector is submitted to the Government for approval”, respectively (September 2020 Restructuring Paper, page 6). Thus, although significant progress was made towards an environment for sustainable energy efficiency financing, the results chain was not adequate to deliver a launch of a financing mechanism and hence Objective 2 was only partially met by the time the PDO indicator and associated targets were revised under the September 2020 restructuring. Hence, Objective 2 in the Original Project is rated Modest for efficacy. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective To create enabling environment for sustainable EE financing. Revised Rationale Please see the discussion of the TOC under objective 1, Original Project. Outcomes: - The revised outcome target for PDO2 was achieved: the target was “Program design to scale-up EE investment in public sector is submitted to the Government for approval”, and, in fact, the package with the Program (an EE Scale-up Concept) and the developed financial model were submitted to the Government. In November 2022 (about 5 months after Project closing date), the Government confirmed its support to these proposals, and the package was submitted to the Parliament for discussion (ICR, page 21). Rating. The Project achieved its revised PDO2 target – the package with recommended measures and the financial model was delivered - and made significant progress towards creating enabling environment for EE financing overall. However, the RF evidence showing the achievement of the main PDO2 indicator, as stated in the ICR – “Development of legal, institutional and regulatory basis for setting up EE financing mechanisms” - is limited to the submission of the package, which is not sufficient. Therefore, the efficacy for Objective 2 in the Revised Project is rated Substantial. Revised Rating Substantial Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Original Project: For the Original Project, the efficacy is Substantial. Under Objective 1, substantial progress was achieved towards improved EE (PDO1), and the Project achieved its most important targets, with a minor shortcoming of not reaching the intermediate indicator’s target of subprojects’ energy savings. Under PDO2, the Original Project had strong achievements towards creating enabling environment for sustainable EE financing, however, the results chain was not adequate to deliver a launch of a financing mechanism and hence PDO 2 was only partially met. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale For the Revised Project, the efficacy is Substantial. The Project achieved all its PDO1 intended outcomes, despite a minor deficiency of having an indirect link between the RF indicators and the stated PDO outcome. Under PDO2, the Project achieved its revised objectives and made significant progress towards creating enabling environment for EE financing, with the shortcoming of an insufficient RF evidence for a full achievement of the outcomes. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency I. Economic and Financial Analysis. Economic and financial analysis was conducted for Part 1 (investments in subprojects), both at appraisal and closure. The methodology was the same at appraisal and at closure, with two exceptions: (i) the former relied on energy audits in four sample installations and the latter on actual energy savings from the sub-projects financed by the Project and (ii) the externalities (GHG reduction) were included in the calculations at closure but not at appraisal. Both at appraisal and at closure, benefit streams consisted of the value of the heat and electricity savings based on tariffs for public buildings. For the financial internal rate of return (FIRR), nominal tariffs were used, while for the economic internal rate of return (EIRR), the benefits were estimated at the long-run marginal cost (LRMC) of electricity supply. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) 1. At appraisal. a. Financial analysis. The FIRR on the sub-project investments was estimated in the range of 12 percent to 56 percent depending on the sub-project assessed, which indicates that the financial return of the Project was expected to be positive. b. Economic analysis. The economic internal rate of return (EIRR) was in the range of 17-77 percent depending on the sub-project assessed, which confirms that the economic return of the Project was expected to be positive. 2. At closure. The analysis relied on energy savings data from the actual 96 sub-projects. a. Financial analysis. All subprojects had positive financial return on investment, but lower than estimated at approval: the average FIRR for the Project was equal to 4 percent. The rate improved from the first three stages of sub-project selection, when the FIRR was only 2 percent, to the last two stages, when the FIRR increased to 10-14 percent. b. Economic analysis. The EIRR at closure was 4.6 percent, below the level expected at appraisal. The EIRR increased from the earlier stages of the Project to the later stages. Accounting for the environmental externalities (GHG reduction), the EIRR amounts to 6.6 percent, still below the appraisal estimate. II. Administrative efficiency. The Project was able to achieve its objectives within the original financing envelope. However, it was extended four times, for the total of 60 months (full five years). Two of these five years are accounted for by the delays in reaching effectiveness, due to the Government’s rules for the approval of projects financed through a trust fund. The remaining delays can be partially attributed to the learning process involved, as the Project was innovative (ICR, page 31). The Project also encountered significant implementation delays related to the COVID-19 pandemic, which severely impacted Kazakhstan. This required additional Project restructuring to transfer financing from Part 1 to Part 2 to allow for a continued functioning of the PMU. However, the Project’s supervision was very efficient. Once the Project became effective, it took only 3.5 months to select the first 19 sub-projects. The EE scale-up concept was also prepared in a short time of one year and eight months, from launch to completion of the EE scale-up concept and the implementation of some of the recommendations. During the nine years of Project implementation, 15 supervision missions were conducted and 15 ISRs issued, despite the COVID-19 pandemic. The reporting was diligent and candid. Significant savings were generated through efficient and astute contract management; they were utilized to increase the much- needed technical assistance for policy and legal environment for public investments in EE and innovatively used to support EE development in the industry. (ICR, page 35). The Project’s efficiency is rated as Modest, mainly due to the EIRR estimated at a substantially lower rate at closure than at appraisal and due to the significant implementation delays. Efficiency Rating Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 82.70 Appraisal  17.00  Not Applicable 69.40 ICR Estimate  4.60  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Revised Project (against the 3rd Evaluative criteria: Original project restructuring targets) Relevance of PDO High High Efficacy Efficacy: PDO1 Substantial Substantial Efficacy: PDO2 Modest Substantial Efficacy: Overall Substantial Substantial Efficiency Modest Modest OUTCOME Moderately Satisfactory Moderately Satisfactory Outcome ratings, numerical 4 4 value Disbursements before and after the 3rd restructuring, 17.08 4.66 US$ million Percent disbursed 78.5 21.5 Weighted value of outcome 3.14 0.86 rating Final outcome rating 4.0 With both the Original and Revised Projects being rated as Moderately Satisfactory, the Overall Project outcome rating is Moderately Satisfactory. For information, the shares of the disbursed funds before and after restructuring were as follows: US$17.08 million or 78.5 percent of the Project costs at closure were disbursed before the restructuring in September 2020 and US$4.66 or 21.5 percent were disbursed after that restructuring. The rating is due to minor deficiencies in Project’s efficacy - results chain deficiency, the indirect link between the PDO1 indicator and the stated PDO1 outcome, and insufficient RF evidence for the PDO2 achievement - Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) and the implementation inefficiencies resulting in the EIRR at closure being below the EIRR at appraisal and in a significantly extended Project closure date. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Institutional capacity for the scale-up. This risk could arise if the Government has insufficient capacity to manage the replication of the demonstration sub-projects supported by the Project. To mitigate this risk, the Project provided technical assistance and training to the implementation agency and the PIU. Policy. This risk could arise if the policy or legal barriers to EE investment or disincentives for the government to invest in EE prevent the scale-up. However, considering that: (i) the EE scale-up concept prepared under the Project was formally supported by the government, and a related package was submitted to the Parliament for discussion; as well as (ii) a significant progress made by the Project with developing a framework for EE development in the industrial sector, important mitigation measures were applied. It should be mentioned however, that policy and legal barriers to EE in the country are still high. Social (mitigation of negative social impacts). During the May 2017 mission, significant occupational health, and safety hazards, inconsistent with safety standards, were identified at sub-project sites. The ICR reports that “good practice recommendations were made and most likely complied with for the rest of the duration of Project implementation,” however, the risk remained Substantial (ICR, page 36). Environmental (negative impacts). This risk was rated Moderate at approval. It was increased to Substantial in January 2018, because of the environmental safeguards mission in May 2017, when waste management practices inconsistent with international standards were identified. The ICR states that relevant recommendations were provided and “most likely complied with”. (ICR, page 36) 8. Assessment of Bank Performance a. Quality-at-Entry The Project’s design responded to the need to incentivize EE investment in the country through demonstration sub-projects and policy and institutional support. The selected entry point – EE in public and social buildings – was also chosen wisely, providing direct social benefits to the population, fiscal benefits to local governments, and raising awareness among a wide range of the beneficiaries. The sub- projects were selected competitively, increasing the potential benefits to the stakeholders. The institutional arrangements for implementation were also well designed and successful. The risks were analyzed and mitigated during the preparation stage. However, as reported in the ICR, there was some underestimation of the gaps in the institutional, policy, and regulatory framework that was needed to Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) support EE development (ICR, page 34), which required adjustments in Part 2 during implementation. In addition, the design of the RF for the Project had significant deficiencies, as described in sections 4 (Achievement of efficacy) and 9 (M & E Design, Implementation & Utilization) of this Review. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The ICR reported that the Project’s effectiveness was delayed by two years due to specific rules for the approval of projects financed through a trust fund. The Project’s supervision was efficient, diligent, and innovative. The first round of 19 demonstration sub-projects was selected within 3.5 months from effectiveness. The study on design of the EE financing was completed and its recommendations implemented within one year and eight months from the launch of the study in September 2015. During the nine years of Project implementation, 15 supervision missions were conducted and 15 ISRs issued, despite the COVID-19 pandemic. The reporting was diligent and candid. Significant savings were generated through efficient and astute contract management; they were utilized to increase the much-needed technical assistance for policy and legal environment for public investments in EE and innovatively used to support EE development in the industry. (ICR, page 35) The work funded by the Project supported legislative changes that were then linked to the World Bank’s Development Policy Loans. As a result, the Parliament approved related laws. Also, the Project supported the analysis underpinning the country’s Country Climate and Development Report (CCDR). (ICR, page 30) However, the deficiencies of Project design could have been addressed earlier during implementation and to a larger extent than was done. This includes a review of the RF to include project-level outcome indicators relevant to the Project’s scope and a timely adjustment of the indicator related to creating a financing mechanism. Instead, the RF had only intermediate outcome and output indicators until Project closure (no adjustment to include project-level indicators was made); and it took seven years to revise the PDO2 indicator “Development of Sustainable Energy Financing Mechanisms” and its outcome target with a realistic alternative. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Overall, the Project's RF mostly reflected the logic of Project interventions in the PAD and was sufficiently linked to the PDO. Most of the RF indicators were quantitative, and all of them were time-bound, had Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) baselines and targets, and were attributable to the Project. There was a gender-disaggregated indicator. The RF measured the intermediary outcomes (such as energy and emission savings and the number of people benefitting from subprojects) and the outputs (such as number of sub-projects implemented). While there were no Project-level outcome indicators in the RF, the intermediate and output indicators allowed for the evaluation of the Project’s efficacy. However, the RF had deficiencies. First, the PDO2 indicator in the PAD had a target of launching a financial mechanism for EE by Project closure, which was unrealistic considering the stage of the energy sector reform in the country, including the below cost recovery energy prices and the regulatory environment. The error in formulating this indicator (as well as the related inadequacy of the implicit results chain in the PAD) was corrected during the September 2020 restructuring when it was reworded as “financial model developed to evaluate investment and structure financing for EE in public sector”, with the target of “program design to scale-up EE investment in public sector is submitted to the Government for approval” (September 2020 Restructuring Paper, page 6). Second, the PDO1 indicator (energy saved from the sub-projects) is, in fact, an intermediate indicator reflecting the energy saving benefits from the implemented demonstration subprojects. Instead, it was labeled in the RF as the PDO1 indicator, while PDO1 was formulated as “To improve EE in public and social facilities” and the related PDO outcome in the ICR’s ToC, was stated as “Improvements in EE in public and social facilities (demonstrated)”. Clearly, while the PDO was stated at a level above the Project’s scope and ambition, the PDO1 indicator was formulated below the level of the expected Project-level PDO outcome of the demonstrated (for a scale-up) EE in public and social facilities. In addition, this indicator was effectively designed as adjustable because the selection of demonstration subprojects was based on bidding, making an a priori estimate of energy savings impossible. For these reasons, it was an RF design error to use this indicator at the PDO level. Due to the deficiencies discussed above, the Project’s M&E design is rated as Modest. b. M&E Implementation The ICR reported that the PIU diligently monitored each subproject from the time of the energy audit and through the contracting and construction. Data-gathering forms were designed and circulated to the facilities concerned as well as to the contractors and others. The data were collected and compiled at various points in the subproject cycle and tabulated and reported periodically as agreed with the World Bank. The 15 ISRs were based on these data and thereby provided clear evidence of a well-implemented monitoring process. (ICR, page 32) M&E implementation is rated Substantial. c. M&E Utilization The ICR reported that the monitoring data generated by the PIU had been crucial to learning the characteristics of the EE sub-projects, whereby needed adjustments could be made to the various parameters of the Project. The monitoring process, thus, strongly supported the proactivity displayed by the World Bank supervision team. In obtaining approval for five restructurings, as well as adjusting the targets for the RF indicators, the monitoring data allowed the World Bank supervision team and the PIU to support the achievement of Project objectives and create conditions for a scale up. (ICR, page 32) Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) M&E Quality is rated Substantial based on a modest rating for ME design due to weaknesses described above and strong substantial ratings for both implementation and utilization. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Environmental Safeguards. The Project was classified as Environmental Category B, with potential impacts such as air pollution, dust, noise, construction wastes, and asbestos. The Bank’s safeguard policy of Environmental Assessment (OP/BP 4.01) was triggered. An Environmental Management Framework (EMF) including a simple Environmental Management Plan (EMP) was prepared and disclosed. It was discussed with the counterpart before appraisal (on November 15, 2012), and the final EMF was posted on the KazEE website (www.kazee.kz) and submitted to the World Bank. The EMF was to be integrated into the Project Operation Manual, while the sub-projects’ EMP would be integrated into the construction contracts for individual sites, and the filled tabular EMP checklist would be attached as an integral part of the works contract. The ICR reports that an inspection of the sub-project tender documents or the requests for expressions of interest did not include any EMP checklists. The actual contract documents were not inspected. Therefore, environmental compliance for the sub-projects remains unclear. The ICR reported no accidents, or environmental issues, or affected population during implementation. (ICR, page 33) Social Safeguards. No social safeguard policies were triggered at approval. The ICR reported that important occupational health and safety risks were identified during a World Bank safeguards mission in May 2017. As a result of the findings of this mission, the environmental and social risk of the Project was increased from Moderate to Substantial and remained at that level at closing. In early 2019, the grievance redress mechanism (GRM) established for the Project was reviewed, and the World Bank team requested the Energy Efficiency Development Institute (EEDI) to design and maintain an adequate Project’s GRM that can ensure transparency and accountability in World Bank-financed operations. As part of the restructuring in March 2019, the GRM was made a condition for extension of the closing date to December 29, 2020. EEDI launched a new website (www.eeq.kz) dedicated to the Project, which provided contact details of the World Bank country office and the PIU, as well as a feedback form through which grievances can be filed. The World Bank team also requested that information boards reflecting GRM information be made available at the Project sites. Although the World Bank team had made the beneficiaries aware of the GRM at the early stages of the Project, at Project closing, the level of awareness of the GRM among beneficiaries and stakeholders and its use could not be clearly established. The safeguards rating in the last ISR was “Substantial”. The ICR reported no outstanding social safeguard issues at closure. (ICR, page 33) b. Fiduciary Compliance The ICR reported that the financial management arrangements were assessed by the Bank at the time of appraisal in 2012 and found to be satisfactory. However, given prior assessments, a timebound action plan was agreed upon to bring the organization into fully satisfactory status, which included: (a) updating the Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) accounting software to meet the Project’s reporting and accounting requirements; and (b) recruiting a dedicated financial management specialist, both within 30 days of effectiveness. Additionally, the Project would rely on the counterpart’s budgeting and (partially) internal control systems but would be otherwise using the Bank’s fiduciary systems. The ICR further reported that a financial management supervision report, dated September 2020, found the financial management arrangements established by the PMU of the EEDI for the implementation of the Project to be adequate and acceptable by the Bank standards. The report also found the PMU to be in full compliance with financial covenants under the Swiss Grant Agreement, including timely submission of the acceptable interim unaudited financial reports and annual audited financial statements. Furthermore, the auditor’s report of June 2022 found full compliance with the requirements of the World Bank and with all goods and works, and services having been procured according to the Guidelines of the World Bank. (ICR, page 33-34) There were no overdue audit reports at the time of Project closure. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Bank performance is rated Moderately Satisfactory due to deficiencies in project design, including: (i) the pitching of PDO1 at a higher level than justified by the planned activities and outputs; and (ii) the Bank Performance Satisfactory Moderately Satisfactory underestimation of gaps in the enabling environment for EE and and the resulting overstatement of what was achievable under PDO2. The supervision effort did not identify the problem for PDO1 and was slow to identify and correct it for PDO2. The Project’s M&E is assessed as Substantial because of Quality of M&E High Substantial inconsistencies between the expected outcome indicators Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) and the planned activities and outputs.(Section 4 and 9). Quality of ICR --- Substantial 12. Lessons The following lessons were derived from the ICR (ICR, pages 37-38): 1. Considering existing regulations (specifically, building standards and codes) was critical for the financial sustainability of the sub-projects. The sub-projects that were implemented later during the Project’s life demonstrated higher financial viability, because of “learning by doing”: it turned out that EE in improvements affecting structural items, such as walls and roofs, trigger building codes and therefore increase costs, making sub-projects financially unsustainable and creating barriers for a scale-up. At the same time, limiting EE heating-related investments to non-structural items, such as windows and doors, and combining them with investments in electricity savings, was yielding positive returns. As a result of this learning experience, the Project was able to adjust sub-projects’ design and achieve positive financial returns at closure. The alternative would be subsidization of the EE sub-projects through either budgetary support or carbon pricing. 2. Health and safety issues need to be an important focus area when implementing EE projects in public and social facilities. The Project’s experience was that shutting down the operations of social facilities to implement the sub-projects was rarely possible. Health and safety safeguards were therefore important, especially when the sub-projects were implemented in schools and kindergartens and children were in the same building during construction. Waste management and mitigation of health and safety hazards were critical to avoid accidents. In future projects, it would be important to consider how the works should be implemented to guarantee the safety of the public and staff while minimizing the disruption to the functioning of the facilities. 3. In countries like Kazakhstan, where EE reforms are at their initial stage and energy sector reforms are in the process of being implemented, incentives for the private sector to invest in EE projects in public and social facilities are low; therefore, sustainability of such projects depends on the availability of public funding, specifically, from local governments. Budget allocation process in many cases needs to be adjusted accordingly, and related policies and regulations introduced. This issue, although discussed briefly in the Project’s PAD and included in some of the Project’s technical assistance outputs, could have been a more integral part of the Project design, which is an important lesson for future EE projects in public and social facilities. 13. Assessment Recommended? No 14. Comments on Quality of ICR Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review KAZAKHSTAN ENERGY EFFICIENCY PROJECT (P130013) The ICR provides sufficient technical details to understand the value-added of the activities and the outcomes of the Project; a good justification of the PDO relevance; comprehensive and robust evidence on most aspects of Project’s evaluation; and a clear linking of evidence to findings. The ICR is technical, analytical, and has internal consistency. The lessons learned are linked to the narrative and the ratings and are useful for future lending operations. At the same time, the ICR has the following minor weaknesses: 1. The ICR could have provided a better description of the Project’s results chain in the ToC as it relates to (i) the inadequacy of the intention of delivering a launch of the financing mechanism by Project closure and (ii) actual expected Project-level outcomes under PDO1. This would support the efficacy analysis in the ICR. Also, the ICR could have been more critical of the RF design, while better explaining Project outcomes outside of the RF under PDO2 in the Efficacy section. 2. The presentation of the Project costs in the ICR was confusing, with some typos and missed details. Annex 3 was incomplete and inconsistent with the data sheet. There were also some typos such as incorrect amount of the grant on the ICR cover page (US$21,763,000 million) and inconsistent figures for completed sub-projects a. Quality of ICR Rating Substantial Page 19 of 19