Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005697 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF019255 ON A GRANT IN THE AMOUNT OF EUR 11,593,021 (US$ 13,542,967 EQUIVALENT) TO THE REPUBLIC OF TURKEY FOR THE EU/IPA ENERGY SECTOR TECHNICAL ASSISTANCE PROGRAM – PHASE 2 November 20, 2021 Energy & Extractives Global Practice Europe And Central Asia Region Official Use CURRENCY EQUIVALENTS (Exchange Rate Effective October 5, 2021) Currency Unit= TRY 10.27= EUR 1 TRY 8.86= US$ 1 EUR 1= US$ 1.16 FISCAL YEAR July 1 - June 30 Regional Vice President: Anna M. Bjerde Country Director: Auguste Tano Kouame Regional Director: Charles Joseph Cormier Practice Manager: Sudeshna Ghosh Banerjee Task Team Leader(s): Ayse Yasemin Orucu ICR Main Contributor: Nick Frydas, Mehmet Huseyin Yilmaz Official Use ABBREVIATIONS AND ACRONYMS AA Administrative Agreement BOTAŞ Petroleum Pipeline Corporation of Turkey CFCU Central Financing and Contracting Unit CO2 Carbon Dioxide CPS Country Partnership Strategy CPF Country Partnership Framework CS Consulting Services DGEA Directorate General for Energy Affairs EBRD European Bank for Reconstruction and Development EC European Commission EE Energy Efficiency EMRA Energy Market Regulatory Authority EPIAS Energy Exchange Istanbul EU European Union EUAŞ Electricity Generation Company EUD European Union Delegation in Turkey EST Energy System for Turkey FM Financial Management GA Grant Agreement GDFR General Directorate of Foreign Affairs GDP Gross Domestic Product HPP Hydro Power Plant ICR Implementation Completion and Results Report IPA Instrument of Pre-accession Assistance ISR Implementation Status and Results Report M&E Monitoring and Evaluation MENR Ministry of Energy and Natural Resources MEU Ministry for EU Affairs MIPD Multiannual Indicative Planning Document PAD Project Appraisal Document PDO Project Development Objective PIU Project Implementation Unit RE Renewable Energy REOI Request for Expression of Interest RES Renewable Energy Source RETF Recipient-Executed Trust Fund RF Results Framework S&I Supply and Installation SCADA Supervisory Control and Data Acquisition TA Technical Assistance TEIAŞ Turkey Electricity Transmission Company ToR Terms of Reference TPP Thermal Power Plant Official Use TSO Transmission System Operator TTL Task Team Leader WB World Bank Official Use The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ..................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ..............................................................13 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDOs ............................................................................................................13 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................15 C. EFFICIENCY ...........................................................................................................................22 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................23 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................23 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 24 A. KEY FACTORS DURING PREPARATION ...................................................................................24 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................26 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 29 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................29 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................30 C. BANK PERFORMANCE ...........................................................................................................30 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................32 V. LESSONS AND RECOMMENDATIONS ............................................................................. 33 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 35 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 52 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 54 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 55 ANNEX 5. BORROWER COMMENTS ...................................................................................... 56 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 76 Page 1 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name EU/IPA Energy Sector Technical Assistance Program - P151934 Phase 2 Project Country Financing Instrument Turkey Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Turkey Ministry of Energy and Natural Resources Project Development Objective (PDO) Original PDO Project Development Objective is to develop the administrative and technical capacity of relevant institutions to achieve an increasing alignment with the EU Acquis and the Europe 2020 targets in the areas of (1) energy efficiency, (2) internal energy market, and (3) long-term energy planning and modelling. Page 2 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 13,542,967 13,542,967 12,954,909 TF-19255 Total 13,542,967 13,542,967 12,954,909 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Borrowing Agency 503,685 0 0 Total 503,685 0 0 Total Project Cost 14,046,652 13,542,967 12,954,909 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 19-Jun-2015 24-Nov-2015 09-Jan-2019 31-May-2020 28-May-2021 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 26-Apr-2020 10.23 Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Highly Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 25-Jan-2016 Satisfactory Satisfactory 0 Page 3 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) 02 19-Jul-2016 Satisfactory Moderately Satisfactory 0 03 06-Feb-2017 Satisfactory Moderately Satisfactory 2.26 04 02-Oct-2017 Moderately Satisfactory Moderately Satisfactory 2.26 05 06-Jun-2018 Moderately Satisfactory Moderately Unsatisfactory 2.72 06 17-Dec-2018 Moderately Satisfactory Moderately Satisfactory 2.72 07 27-Jun-2019 Moderately Satisfactory Satisfactory 5.10 08 27-Dec-2019 Satisfactory Satisfactory 7.52 09 14-Aug-2020 Satisfactory Satisfactory 10.98 SECTORS AND THEMES Sectors Major Sector/Sector (%) Energy and Extractives 100 Other Energy and Extractives 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 0 Enterprise Development 80 MSME Development 80 Environment and Natural Resource Management 0 Climate change 20 Mitigation 20 ADM STAFF Role At Approval At ICR Regional Vice President: Laura Tuck Anna M. Bjerde Country Director: Martin Raiser Auguste Tano Kouame Director: Laura Tuck Charles Joseph Cormier Page 4 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Practice Manager: Ranjit J. Lamech Sudeshna Ghosh Banerjee Kari J. Nyman, Ayse Yasemin Task Team Leader(s): Ayse Yasemin Orucu Orucu ICR Contributing Author: Nickolas F. Frydas Page 5 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL 1. Country context. Turkey is a large, upper-middle-income country with a strong record of inclusive growth, but recent shocks are risking the economic and social gains made since the early 2000s. Turkey achieved rapid economic and social development in the 2000s, with poverty incidence more than halving and real Gross Domestic Product (GDP) increasing by 50 percent by 2008. Since the Global Financial Crisis, rapid growth continued but was increasingly associated with stagnant productivity, a rising current account deficit and growing foreign exchange-denominated debt stock. 2. Economy kept growing at about 3.5-4 percent annually. After a strong rebound from the global financial crisis in 2010-2011, growth slowed to 4.2 percent in 2013, and realized as 2.9 percent in 2014. Annual inflation accelerated to 8.2 percent in December 2014, up from 7.4 percent in December 2013, significantly above the Central Bank’s 5 percent target. With the increase in external financing costs and expected gradual tightening of global liquidity, Turkey`s main challenge was to sustain economic growth without deteriorating both the external and internal imbalances. 3. Turkey has been a candidate country for European Union (EU) membership. Accession negotiations started in October 2005. Since then, the EU has provisionally closed one chapter (Science and Research) and opened negotiations on 12 other chapters. Although the energy chapter has not been opened for negotiations, energy was a topic of key interest in EU-Turkey relations and enhanced energy cooperation was an integral part of the EU-Turkey agenda. The Government viewed the accession process as Turkey’s own, fundamental “modernization project”. After the June 2011 elections, the Government upgraded the institution overseeing EU accession matters into a new Ministry for EU Affairs (MEU). The new Government which came into office following the Presidential elections in August 2014 has adopted a three-stage EU action plan to accelerate the accession process. 4. Sector context. Turkey’s energy sector was in a transition phase to ensure self-reliance, robustness, diversification, supply security, competitiveness and low-carbon energy. The sector has been driven by two major characteristics: growing energy demand and import dependency, both of which are impacted by steady economic development with an average 5% annual growth coupled with sectoral leaps in energy. 5. Turkey's energy reforms achieved energy security in a high economic growth environment with rapidly increasing energy needs through a variety of inter-linked measures including electricity, gas, renewable energy (RE) and energy efficiency (EE) legislation; the establishment of an energy sector regulatory authority; energy price reform; the creation of a functional electricity market and large-scale introduction of natural gas; restructuring of state-owned energy enterprises; and large-scale private sector participation through privatization and new investment. Developing countries and emerging economics across the world have taken similar measures but few have achieved similar results, which include developing an electricity market with over 800 market participants, attracting over 30,000 MW market-based private sector power generation into this market in 2001-2014 and privatizing the entire power distribution system. Page 6 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) 6. Securing an adequate and reliable energy supply for Turkey’s growing economy, efficiently and in an environmentally sustainable manner, was a major challenge. The Government projected 3.3 percent economic growth for 2014 and continued solid growth was projected at 3.5-4 percent over the medium-term. After a temporary slowdown, the demand for energy was again growing rapidly, especially electricity at about 4-5 percent/annum. There was no immediate threat of supply inadequacy, but continuing demand growth could lead to supply shortages before the end of the decade if generation investments fall behind load growth as happened in the middle of the previous decade. Gas import was another potential risk and issue, in terms of both volume and price. Ensuring the most competitive prices was critical, as energy imports (mostly oil and gas) accounted for over 20 percent of Turkey’s imports and about 50 percent of the current account deficit. 7. The role of the Government and state-owned energy companies was shifting from investing to setting energy policy and strategy and proving network infrastructure and centralized trading platforms for the private sector. Most investment decisions in the power sector have moved from the public sector to private investors and their financiers. The main public sector electricity investor was Turkish Electricity Transmission Company (TEIAS), to provide a national grid into which private generators feed and from which private suppliers and large consumers draw their electricity supplies. In the gas sector public investments were primarily for the gas transmission network and gas storage facilities by the Petroleum Pipeline Corporation (BOTAS). Gas distribution investments have been carried out by private companies. The role of BOTAS in gas imports has shifted from a monopoly importer to a dominant importer. At the wholesale level, about 30 percent of all electricity was sold through a centralized trading platform and prices at that platform inform prices in bilateral contracts between generators, suppliers and large consumers. BOTAS prices continued to strongly influence wholesale gas prices but price formation was expected to progressively move to the forthcoming centralized gas trading platform to be managed by the new Energy Market Operations Company (EPIAS). 8. Government’s main objectives were: (a) to provide adequate information to market participants for their investment decisions to promote sustainable energy sector development; and (b) to use the forecasts to better inform its own policy decisions so as to spur private investment which meets broad long-term strategic objectives (energy security, climate, diversification, etc.). 9. Turkey’s 10th Development Plan for 2014–2018 set out the longer-term structural reforms to be pursued to address Turkey’s development challenges, such as improving productivity and tackling energy supply security to ensure sustained growth and building the skills of its population to reap the benefits of greater global integration. Turkey was moving decisively to develop an increasingly reliable and efficient energy supply, while also mitigating climate change through EE and RE development. 10. Enhanced energy cooperation was an integral part of the EU-Turkey agenda. Five cooperation areas have been agreed by the European Commission (EC) and the Government of Turkey: (i) long-term perspectives on energy scenarios and energy mix; (ii) electricity and gas market integration and development of infrastructures of common interest; (iii) global and regional energy cooperation; (iv) promotion of RE, EE and clean energy technologies; and (v) nuclear safety and radiation protection. The EU Instrument for Pre-Accession (IPA) Energy Sector Technical Assistance (TA) Program supported three of these five cooperation areas, namely points (i), (ii) and (iv). The Project was the second phase in a multi- year TA program in support of the EU/Turkey energy agenda. Page 7 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) 11. The energy sector was highlighted for EU support in the 2011-2013 Multi-Annual Indicative Planning Document (MIPD) adopted by the European Commission (EC) in June 2011. As indicated in the MIPD: “By completing the energy sector liberalization process Turkey hopes to establish a well- functioning, free and competitive energy market. This requires sustainable private sector investments and a well-functioning and regulated energy market, while limiting environmental damage, reducing greenhouse gas emissions, and increasing EE”. 12. Rationale for World Bank’s involvement. The World Bank (WB) was and remains a key partner for policy and system reforms in the Turkish energy sector. This project contributed to the realization of the objectives of the Turkey Country Partnership Strategy (CPS) for the financial years 2012-2015 (FY12- 15) period approved by the WB’s Executive Board on March 27, 2012. The CPS had three main strategic objectives and pillars: Strategic Objective 1 - enhanced competitiveness and employment; Strategic Objective 2 - improved equity and public services; and Strategic Objective 3 - deepened sustainable development. Activities planned under the objective “Deepened Sustainable Development” included policy advice and financing to address energy, environmental and climate change challenges in an integrated manner. 13. Higher-level objectives that the project contributes. The project contributed to the realization of Turkey’s energy sector development objectives and the alignment of the energy sector with the energy priorities and strategies of the EU. The project supported the implementation of: (a) Turkey’s 10th Development Plan (2014-2018); (b) the EC’s MIPD; and (c) the Turkey CPS for the FY12-15 period. 14. Theory of change (Results Chain). The project’s theory of change is illustrated in Table 1 below. The project included consultancy services, non-consultancy services, goods and supply and installation (S&I) services under four different components. The main deliverables were reports and training events, aiming to strengthen the stakeholders’ institutional capacity in three main outcome areas -EE, internal energy market and long-term energy planning and modelling- to enable them to achieve an increasing alignment with the EU Acquis and the Europe 2020 targets. Page 8 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Table 1: Results Chain of the Project Activities Outputs Outcomes Condition assessment studies (performance tests and energy audits) EE projects for implementation in electricity generation on the mechanical and electrical equipment were performed and identified. cost-effective EE improvements were assessed in 24 Hydro Power EE Analysis of Electricity Plants (HPPs) and 2 thermal power plant units. Generation Company’s EUAS enabled to make EE investment plans with the provided (EUAS) Hydro and Thermal Feasibility reports including cost effectiveness, itemized cost information on average turbine efficiency improvement Power Plants associated with recommended interventions, average of all estimation, prioritized options, and defining the scope of annual energy benefits and average capital costs associated rehabilitation works for optimizing operation and efficiency of with performance & condition improvements. selected HPPs delivered. 10 tasks including analysis of storage technologies, analysis of Turkish transmission system in terms of congestion management, needs assessment for energy storage systems in Turkey, market Institutional capacity built to assess energy storage research and feasibility study for the deployment of battery storage, applications and frequency regulation systems through Analysis of the Energy identification of the technical specifications and battery prototype workshops conducted to analyze the systems in the areas of Storage Systems pre-feasibility study along with 2 workshops, 5 trainings and a site security of supply and system stability. visit conducted. Primary reserve identified as the most profitable service for the Turkish power system through organizing regular tenders. Performance assessment through benchmarking, developing future scenarios for network assessment, technology optimization for EE projects for implementation in electricity transmission transmission elements, preparation of investment plan for an identified. optimal asset replacement schedule, recommending improvements EE Analysis of Electricity to the transmission planning methodology by comparing the current TEIAS’s network planning capacity enhanced with the analysis Transmission System TEIAS state-of-the-art and the European Network of Electricity of current EE of the electricity transmission network and operated by TEIAS Transmission System Operators (ENTSO-E) best practices, case study provided recommendations for reinforcements and applications for insulation coordination study and distance operations allowing an overall savings of around EUR 225,5 determination of transmission lines and capacity building through million. workshops, working group meetings, trainings and site visits performed. Needs assessment regarding EE of the natural gas EE Analysis of Natural Gas EE projects for implementation in gas transmission identified. transmission system completed. Transmission System operated by BOTAS Required institutional capacity for optimized network Energy savings potential around 15-30% calculated. The overall gas Page 9 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) savings potential as 2 million Sm3 of natural gas; carbon emissions operation was built through hands-on training sessions. reduction potential by nearly 3 ktons per year identified. BOTAS enabled to do medium- and long-term load projections via Synergi Pipeline Simulator (SPS) model to add up to energy savings potential at a significant level. Legal, structural and administrative basis for properly functioning energy exchange operations of Energy Markets Operation Company (EPIAS) achieved and EPIAS is functional in operating electricity and gas markets. Current situation assessed, gaps and improvement areas determined and the possible measures and solutions to solve the issues based on Ministry of Energy and Natural Resources (MENR), EPIAS, international experience with similar problems recommended on TEIAS, BOTAS and Energy Market Regulatory Authority Electricity and Natural Gas Markets Development; Transparency and (EMRA) became capable of further developing electricity and Assessments, Market Surveillance; Balancing Power Market Operations; Market gas market operations to align with the respective EU rules. Recommendations and Coupling; Guarantees of Origin; Demand Side Response and Capacity Capacity Building for Remuneration Scheme. The foundations laid for the operations of futures products in Electricity and Gas Market both electricity and gas markets through EPIAS. Operations 37 reports and 22 capacity building activities/events on the Electricity and gas market operations of EPIAS improved and the assessments and recommendations delivered. transparency platform established. 19 trainings conducted for capacity building in electricity and gas Network operation and monitoring capacity of BOTAS market development. improved by developing Supervisory Control and Data Acquisition (SCADA) system, Gas Management Tools, hardware and software according to the needs of liberalized natural gas sector. Long-term energy scenarios available for energy policy and strategy decision-making. The Energy System for Turkey (EST) model developed and delivered to MENR. MENR enabled to build energy scenarios to feed long Capacity Building for Long- term policy making with the developed institutional and Term Energy Planning and The results of the 8 different scenarios run by the EST model human resources capacity. Scenario Development presented. Long Term Energy Planning Report covering alternative 11 sets of trainings (more than 300 hours in total) organized. energy policy scenarios for the period until 2050 or 2070 to be prepared via EST model. Energy data available from the new Energy Data Center. Survey Information Management System (SIMS) software developed Establishment of an and come online. Infrastructure for the energy data center equipped with Energy Data Center the required software and hardware. Data center hardware including virtualization servers, disk arrays, Page 10 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) monitoring screens, desktop computers and notebooks required for MENR enabled to gather energy data in a more reliable and the proper functioning of both EST model and SIMS software robust way via online data management module developed purchased, installed and in operation. through the project. 495 personnel trained in the operation of the energy model and data SIMS gathered the industrial facilities’ energy consumption center. values for the 2019 Energy Balance Table. A booklet for school children on EE was completed and 1 million Public awareness for EE and RE increased by reaching more copies disseminated by the Ministry. than 35 million users (65% of 54 million active social media The visibility and awareness campaign run via four social media users in the country) with social media awareness campaign. Visibility Campaigns on accounts on Facebook, Instagram, Twitter and YouTube set up under Project Outputs and Public the name “Aklınla Verimli Yaşa”, including a separate website with Visibility and dissemination of project results provided to all Awareness regarding RE approximately 230 social media posts in total on different platforms. stakeholders with forums, seminars and workshops organized. and EE The Communication Plan for MENR’s future EE communication activities prepared and published in MENR’s website. 467 people trained through site visits, symposiums and workshops on various aspects of EE in electricity generation and transmission. Page 11 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Project Development Objectives (PDOs) 15. The PDO was to develop the administrative and technical capacity of relevant institutions to achieve an increasing alignment with the EU Acquis and the Europe 2020 targets in the areas of (1) EE, (2) energy market, and (3) long-term energy planning and modelling. Key Expected Outcomes and Outcome Indicators 16. The realization of the PDO was planned to be measured through the achievement of the following key outcome indicators: (a) Energy efficiency. EE potential in electricity transmission (TEIAS), generation (EUAS hydro) and in gas transmission (BOTAS) is assessed and projects for implementation are identified. (b) Internal Energy Market. Legal, structural and administrative basis for properly functioning energy exchange operations of EPIAS is achieved and EPIAS is functional, operating electricity and gas markets. (c) Long-term energy planning and modelling. Long-term energy scenarios are available for policy and strategy decision-making. Energy data is available from the new Energy Data Center. Components 17. The following components were supported by the project: 1. Energy Efficiency (EUR 5.07 million). Sub-components: (i) analysis of current EE conditions of EUAS’ energy generation facilities; (ii) assessment of needs for energy storage and frequency regulation systems; (iii) analysis of current EE of the electricity transmission network of TEIAS; and (iv) assessment of EE of the natural gas transmission system. 2. Electricity and Gas Market Development (EUR 1.58 million). Sub-components: (i) identification of technical infrastructure needs for a properly functioning energy exchange; (ii) feasibility studies, roadmap, and analysis to enhance the regulation, administration, and organization of an energy market; (iii) trainings for capacity building in electricity and gas market development; and (iv) design of SCADA system according to the needs of the liberalized natural gas sector. 3. Long-Term Energy Scenarios, Capacity Building and Establishment of an Energy Electronic Data Center (EUR 4.2 million). Sub-components: (i) identification of gaps in the energy data collection system and preparation of roadmap and draft legislation for improvement; (ii) purchase of software and hardware for energy data center; and (iii) trainings in effective operation of the data center. 4. Visibility and Public Awareness (EUR 0.61 million). Sub-components: (i) visibility and public awareness campaigns; (ii) fora, seminars and workshops to disseminate results of the project; and (iii) trainings on various aspects of EE in electricity generation and transmission. Page 12 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDOs and Outcome Targets 18. The PDO and outcome targets were not revised throughout the implementation. However, the end target dates of the outcomes were revised in line with the extended closing date. Revised PDO Indicators Not applicable. Revised Components Not applicable. Other Changes 19. The project’s closing date was extended by one year to May 28, 2021 as per the level 2 restructuring that was approved on May 28, 2020. Rationale for Changes and Their Implication on the Original Theory of Change. 20. The global pandemic, coronavirus disease of 2019 (Covid-19), disrupted the timely implementation and completion of the activities which required mobility of people and/or devices such as turbine efficiency measurement systems to be installed for EE analysis of HPPs. As the Government had taken several measures and restrictions related to the Covid-19 pandemic starting from March 2020, some of the project activities had to be postponed for an indefinite period and some had to be cancelled. Within this context, 10 out of 11 contracts ongoing at that time had to be extended due to the incomplete activities requiring mobility and travel of people and devices. 21. Upon MENR’s request and the agreements reached with the EU Delegation to Turkey (EUD), the implementation period set under the Grant Agreement (GA) was extended by one year to May 28, 2021 and the project activities were rescheduled accordingly. 22. The Original Theory of Change was not affected. II. OUTCOME A. RELEVANCE OF PDOs Rating: High 23. Relevance of PDO is High. The PDO was highly aligned with the country’s national development agenda and the WB’s CPS (FY12-15) during appraisal and the Country Partnership Framework (CPF) for FY18-23 period at closure. As for the relevance with the EU strategies, the PDO was designed to achieve an increasing alignment with the EU Acquis and the Europe 2020 targets in the areas of energy efficiency, energy market development, and long-term energy planning and modelling. 24. Relevance of PDO with WB strategies. This project was closely connected with the objectives of the Turkey CPS for the FY12-15 period approved by the World Bank’s Executive Board on March 27, 2012. The CPS had three main strategic objectives and pillars: Strategic Objective 1 - enhanced competitiveness and employment; Strategic Objective 2 - improved equity and public services; and Strategic Objective 3 - deepened sustainable development. Activities implemented under the Project were addressed under Page 13 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Objective 3 - Deepened Sustainable Development which includes policy advice and financing to address energy, environmental and climate change challenges in an integrated manner. 25. The project’s objective was also highly relevant with the objectives set under Focus Area 3 - Sustainability of the CPF (FY18-FY23), revised through Performance and Learning Review conducted in February 2020. Specifically, the project supported Objective 7 of the CPF, which aimed to help Turkey improve reliability of energy supply and generation of green energy; and Objective 9, which aimed to strengthen results under climate action agenda through supporting the capacity of the MENR and other energy institutions to prioritize and select more sustainable and green energy with further IBRD investments. 26. Relevance of PDO with Government strategies. In the 10th Development Plan (2014-2018), approved by the Turkish Parliament in 2013, the basic objective set for the energy sector was “to ensure uninterruptable, high-quality, sustainable, reliable and affordable energy supply.” Under the specific policies and measures described, EE, energy markets development and long-term energy supply and demand planning were highlighted. Similarly, 11th Development Plan for 2019-2023 sets out the longer- term structural reforms to be pursued in order to address Turkey’s development challenges, such as (i) increasing the share of renewables in power generation; (ii) development of measures to reduce carbon emissions, with additional measures such as EE gains; (iii) increasing electricity production from RE sources and necessary planning for integration of RE generation into the grid; (iv) incorporating energy storage systems which will be installed in order to eliminate the constraints of increasing RE on the grid; and (vi) incentivizing EE in existing buildings. Among the key objectives in MENR’s 2015-2019 Strategic Plan the following were included: (i) maintenance, reparation, rehabilitation and modernization studies are necessary in public power plants to improve their efficiency; (ii) public awareness shall be raised on EE and saving; (iii) energy markets shall be made more transparent, reliable and monitorable; and (iv) the Ministry shall use Information and Data Technologies efficiently. MENR’s Strategic Plan 2019-2023 defines 7 objectives and 31 goals accompanied by 113 performance indicators linked to these goals. Four objectives, relevant to the PDO, include Objective 1. Ensure sustainable security of supply; Objective 2. Prioritize and boost energy efficiency; Objective 3. Strengthen the organizational and sectoral capacity; and Objective 6. Enhance predictability in the energy markets. 27. Relevance of PDO with EU strategies. EU through the MIPD (2011-2013) and the subsequent Indicative Strategy Paper for Turkey (2014-2020) had set up the priorities for the energy sector in Turkey. The two main objectives of the MIPD were: (i) increased capacity and better alignment in the EE field; and (ii) further alignment with and implementation of the acquis on internal gas and electricity market. As for the Indicative Strategy Paper for Turkey, the following were identified as key priorities for the energy sector: strengthened institutional and regulatory capacity in relevant authorities; integration with the EU internal electricity and gas markets; promotion of a more efficient use of energy; and achieving the EU 2020 strategy targets. 28. The PDO was consistent with the Bank, EU and Government’s past and current strategies for the energy sector in Turkey. Therefore, relevance of PDO is rated as High. Page 14 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) B. ACHIEVEMENT OF PDOs (EFFICACY) Rating: High 29. The progress towards the achievement of the PDO was not assessed through specific quantifiable indicators since the Project only included provisions for TA. Instead, the PDO outcome indicators were set on a ‘qualitative’ basis. Therefore, the evaluation of the project’s efficacy was not done only by relying on the stated results indicators and their achievement but also by examining how outcomes contributed to the achievement of the broader project objective. In the following section, the project’s efficacy is assessed for each individual outcome area. Outcome 1: EE potential identified Rating: High 30. This outcome area was supported by Component 1 which focused on the following areas: (i) EE potential assessment in EUAS’ generation facilities (24 hydro power plants and 1 thermal power plant), (ii) assessment of needs for energy storage and frequency regulation systems and workshop; (iii) EE assessments in the electricity transmission network; and (iv) EE assessments in the natural gas transmission network. From this task, 188 reports were produced, and 26 events were organized. The main stakeholders were EUAS, TEIAS, EMRA, BOTAS and MENR. The end target for this outcome was defined as “EE potential in electricity transmission (TEIAS) and generation (EUAS hydro) and in gas transmission (BOTAS) assessed and projects identified”. 31. Several energy audits in power plants -mainly hydro- were conducted and demonstrated a significant EE improvement potential. Three Consulting Services (CS) contracts served the EE in power generation subcomponent. In total 24 HPPs and two thermal power plants were assessed. The final deliverables included feasibility reports including cost effectiveness, itemized cost estimation, prioritized options, and defining the scope of rehabilitation works for optimizing operation and efficiency of selected HPPs. A varying EE potential from 0.3% to 6.5% in different HPPs was estimated. It was calculated that performance interventions costing USD 120 million would generate USD 20 million benefits per year. 32. As a result of the EE studies conducted for hydro and thermal power plants, EUAS established a dedicated unit in charge of monitoring the power plants’ performances and planning for EE investments. A project team was set within EUAS comprising both staff from the newly established EE unit and staff from the department responsible for HPP operations. This team was extensively trained in various aspects of performance monitoring such as field performance testing, performance testing fundamentals and theory, Computational Fluid Dynamics (CFD) analysis and HydroVintage application etc. The EUAS project team also participated in field works conducted by the Consultant and was trained in a hands-on manner. This extensive training program has led to building strong institutional capacity in EUAS in terms of EE monitoring of the HPPs. Furthermore, through the organization of a symposium on EE in Electricity Generation and Transmission a wide audience was trained on topics related to EE in power generation which contributed to the capacity building efforts to a great extent. 33. An overview of the potential energy storage roadmap for Turkey was elaborated. Assessments for incorporating energy storage systems into the grid to improve integration of RE sources were developed and delivered. Different topics relevant to energy storage and frequency regulation were studied and 11 relevant reports -excluding the inception and final reports- were prepared and delivered. Moreover, eight knowledge transfer events were held: workshops, trainings, site visits, results Page 15 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) presentation that were attended by up to 60 experts from TEIAS, MENR, EMRA, the WB and ELDER which is the association of electricity distribution system operators. 34. In conjunction with project activities, EMRA started relevant legislative procedures to establish a legal framework for energy storage applications in Turkey and, upon the delivery of project outputs, published a by-law on energy storage in line with project recommendations opening up the market. 35. The main activities of the EE in electricity transmission system sub-component included a technical assessment of the Turkish power system and its components as well as an evaluation of the economic efficiency potential of the system’s operations. In total, 14 reports were prepared covering various topics analyzed. The key findings of the study showed that the Turkish transmission grid has network losses of around 2.23% of the total consumption. Based on assumption and outcomes of the cost-benefit analysis carried out in the assessment, the roll-out of the suggested reinforcements, as well as other operating recommendations may allow overall savings of around EUR 225.5 million. This will prompt TEIAS to incorporate the project recommendations in its investment plans. 36. The EE in electricity transmission system sub-component also comprised an extensive dissemination program to share the outcomes of the analysis, as well as to transfer the know-how deriving from the recommendations provided. To provide an effective knowledge transfer and the most fruitful exchange of experience and open-discussion on the power system-related topics, periodic working groups and wide-participation workshops were held at the end of each main topic. On top of the above task-related events, two main initiatives were organized to provide added value to the knowledge exchange activity: a training on network assessment and a site visit to the Italian TSO’s (Terna) National Control Centre and SAPEI HVDC link converter station in Borgo Sabotino (Rome). 37. Under the EE in gas transmission sub-component, several analyses were performed, assessing options for improving energy and operations efficiency and providing directions on where and how to tap into the demonstrated savings potential. The assessments made under the EE in gas transmission sub-component showed the potential of a significant annual reduction in gas consumption and CO2 emissions as well. The recommendations for the actions to be taken were provided to BOTAS. One of the strong results is the improved institutional capacity for optimized network operation made possible through hands-on trainings sessions. At project closing, BOTAS had been trained to do medium- and long- term load projections via dedicated models to add up to energy savings potential at a significant level, which was not there before the Project. 38. Through Visibility and Public Awareness activities, a symposium, workshops and a nation-wide campaign during the EE Week were organized. All these events contributed to the overall strengthening of the participants -mainly project stakeholders- technical capacity on topics related to EE. In addition to the aforementioned activities, a booklet on EE for school children was prepared and 1 million copies were disseminated by the MENR as part of the Visibility and Public Awareness component. 39. The overall achievement of the Outcome 1 is rated as High. The EE potential in electricity transmission (TEIAS) and generation (EUAS hydro) and in gas transmission (BOTAS) was assessed, projects were identified and provided to the stakeholders for implementation. The importance of EE was disseminated through awareness-raising and capacity building events. Page 16 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Outcome 2: Internal energy market improved Rating: Substantial 40. This outcome area is mainly supported by Component 2 ‘Electricity and Gas Market Development’ which consists of various sub-components: (i) identification of technical infrastructure needs for a properly functioning energy exchange; (ii) design of SCADA system according to the needs of the liberalized natural gas sector; (iii) feasibility studies, roadmap, and analysis to enhance the regulation, administration, and organization of an energy market; and (iv) trainings for capacity building in electricity and gas market development. Under this component thirty-seven reports were produced, and twenty- two events were organized. The main stakeholders were EPIAS, BOTAS, TEIAS, MENR and EMRA. The end target for this outcome was defined as “legal, structural and administrative basis for properly functioning energy exchange operations of EPIAS is achieved and EPIAS is functional, operating electricity and gas markets” and this target was achieved by the completion of all project activities. 41. On the energy exchange sub-component, as EPIAS had already developed its infrastructure, no input was provided in terms of EPIAS’s information technologies (IT) structure. The focus was on EPIAS’s energy exchange operations, which were analyzed and technical recommendations for a better functioning energy exchange were provided. Under this scope three reports were produced: (i) Electricity Exchange Assessment Report; (ii) Electricity Exchange Recommendation Report; and (iii) Draft Legislation Elements on Natural Gas Exchange and Implementation Report. EPIAS revised and adapted some of its exchange operations in line with the recommendations provided in the project deliverables. EPIAS was supported for the improvement of electricity exchange operations and gas trading platform as well as establishment of transparency platform. 42. A SCADA system, various Gas Management Tools, hardware and related software were developed to the needs of a liberalized natural gas sector. Under this sub-component, three technical reports were produced: (i) Needs Assessment Report; (ii) SCADA System Design Report; and (iii) Technical Specifications and Market Research Report. These reports provide an overall understanding of the importance of SCADA system, the missing functions, systems, equipment and design for an effective SCADA system and its components. The Consultant also provided recommendations to BOTAS for efficient transmission system operations and market operations of a transmission system operator (TSO) focusing on Corporate Strategy and Structure, Operating Model Definition, Organizational Alignment and Business Process Design. The outputs of this sub-component were utilized by BOTAS to procure the needed modules of a modern and full-fledged SCADA system under another EU/IPA funded project. 43. With capacity built in BOTAS’ transmission operations supported by the Project, BOTAS has been able to smoothly operate Turkey’s fast growing natural gas network. Between the appraisal and completion of the project, the length of the pipelines operated by BOTAS was expanded by 50% to 18,592 km; the number of entry points raised from 9 to 15; the maximum daily entry capacity increased to 383 million sm3 from around 250 million sm3; and BOTAS served for 50 shipper companies operating at natural gas transmission network in 2020 whereas this number was 30 in 2015. In such a significantly transitioned market, it would be challenging for BOTAS to ensure a reliable network operation without technical assistance provided by the Project. 44. EPIAS and TEIAS was supported in both electricity and gas exchange operations and Balancing Power Market operations. MENR and EMRA was supported for the improvements in legal and regulatory framework to better align with EU energy markets. A wide range of assessments and Page 17 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) recommendation reports on multiple Energy Markets topics were generated to support this work. The topics covered include: Electricity Market, Natural Gas Market and Continuous Trading Platform, Electricity Market Coupling, Balancing Power Market, Transparency, Market Surveillance and Eligibility Customer Management System, Guarantees of Origin System, Demand Side Response and Capacity Remuneration Scheme. In these reports, the present state of the Turkish energy market was evaluated and recommendations for its further development and alignment with the EU acquis were provided. The key findings of these reports were widely disseminated and discussed in the government. 45. A high priority issue was identified as increasing the price predictability in both natural gas and electricity markets. Detailed analyses were conducted, and actions recommended to lay the foundations for the operations of futures products in both electricity and gas markets operated by EPIAS. Balancing Power Market operations and ancillary service procurement of TEIAS was thoroughly analyzed and recommendations were provided for the improvement. Those recommendations were extensively discussed with TEIAS operators. For the natural gas market, it was explicitly underlined that a well- functioning futures products market is among the most important elements of a liberal and competitive market structure and a futures products market that is responsive to the needs of market participants needs to be launched. Within this context, EPIAS has matured its efforts to initiate the operations for futures products and commissioned futures markets for both electricity and gas on June 1, 2021 and October 1, 2021 respectively. 46. Capacity building for electricity and gas market development was intensively supported through the organization of a series of events for knowledge transfer to the stakeholders’ personnel. 7 workshops covering the findings and recommendations on a variety of topics studied, as described previously, were organized and short reports including the workshop material and the discussions held were generated. Furthermore, 6 seminars were organized to disseminate knowledge on market structures and operations, as well as electricity and gas market participation. These seminars also enjoyed participation from the private sector. 47. 3 study visits in European countries (Spain, Italy) were conducted involving participants from various stakeholders. The entities visited included ENTSO-E and European Network of Gas Transmission System Operators (ENTSO-G), national regulatory agencies, electricity and natural gas TSOs and one energy exchange enabling peer learning and state of the art discussion with counterparts. The initially targeted number of training sessions (3) was overachieved, since nineteen training events eventually took place. The capacity building activities had a wide coverage both in terms of content and participants. Most relevant issues were addressed and almost all stakeholders were involved in the project activities. 48. Overall achievement of Outcome 2 is Substantial. Legal, structural and administrative basis for properly functioning energy exchange operations of EPIAS was achieved and EPIAS is functional, operating electricity and gas markets. Electricity exchange and gas trading platform operations of EPIAS were supported and improved through recommendations and on the job trainings. The possible improvement areas in Balancing Power Market operations of TEIAS were determined and discussed with relevant stakeholders. The reports supported the preparations for the regulations related to demand side response and revising the capacity remuneration scheme. An extensive training program helped the strengthening technical capacity of all stakeholders in the market through several workshops and seminars. Page 18 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Outcome 3: Long-term energy planning and modelling developed Rating: High 49. This outcome area was supported by Component 3, which comprises of four sub-components: (i) improved long-term energy scenarios available for energy policy and strategy decision-making; (ii) identification of gaps in the energy data collection system and preparation of roadmap and draft legislation for improvement; (iii) purchase of software and hardware for energy data center; and (iv) trainings in effective operation of the data center. 50. Long-term Energy Scenarios contract developed and delivered the EST model. The EST model projections include structure of energy demand by sector and by energy form, power generation mix by technology, investments per energy sub-sector, CO2 emissions, explicit calculation of electricity prices and overall energy system costs. In addition, results for baseline scenario based on current energy system trends and implementation of already adopted policies and seven alternative scenarios that span a wide range of energy policies from emissions allowances, carbon taxes, support schemes to RE technologies, blending rules for biofuels, other support schemes specifically designed for industrial processes to achieve EE targets in the horizon to 2070 were obtained and delivered to MENR. The EST model was accompanied with a Data Set Report and its User Manual, which are considered the most important deliverables by MENR. 51. This sub-component developed 8 different scenarios with 8 typical days and 4 seasons run by the EST model. The reference scenario is a combination of the business as usual case and government policies; the renewables scenario is built upon the significant increase in penetration of RE sources into the system; the efficiency scenario covers both the EE measures (waste heat recovery, etc.) and emission reduction policies; the decarbonization scenario introduces the emissions trading system applied for the power sector, industry and passenger air transport and taxes for households and services sectors; the combo scenario is the combination of the renewables, efficiency and decarbonization scenarios; the combo transport scenario is a variant of combo scenario further decarbonizing the transport sector; the combo electrification scenario involves the radical electrification of all demand sectors and higher carbon prices; and the deep decarbonization scenario exploits all the decarbonization technologies already integrated in the model in order to achieve deep decarbonization/carbon neutrality by 2050-2070 by assuming high economic growth. The model also considered the inclusion of new fuels (e.g., hydrogen and synthetic ammonia) expected to play a significant role in the future’s fuel mix. 52. The component is rated highly satisfactory in terms of the expected results which already come into sight with the publication of the biennial Report on Turkey’s Electrical Energy Demand Projection for the years 2020-2040 prepared via the EST model. The modelling capacity built in MENR also contributed to the decision-making process for the ratification of the Paris Agreement by facilitating the assessment of long-term impacts of decarbonization and carbon pricing policies to the energy sector and the whole economic ecosystem. ‘Long Term Energy Planning Report’ covering alternative energy policy scenarios for the period until 2050 or 2070 is also planned to be prepared via EST model and published on MENR’s website. 53. The gap analysis for the data collection system and draft legislation for improvements were delivered under the IPA 2012 Energy Sector Program Phase I Project. However, there was not any new legislation needed for the data collection in the energy sector. Regarding data collection and usage of the model developed, Data Set Report was prepared and approved. Page 19 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) 54. The hardware components required for the Energy Data Center were purchased, installed, commissioned and accepted. These included workstations, uninterruptible power supplies, storage units, monitors, presentation tools and video conference systems. Data center hardware including virtualization servers, disk arrays, monitoring screens, desktop computers and notebooks required for the proper functioning of both EST model and Survey Information Management System (SIMS) software were purchased, installed and commissioned. SIMS Software was also developed, tested, accepted and came online by mid-June 2020. SIMS came online and gathered the industrial facilities’ energy consumption values for the 2019 Energy Balance Table. 55. Particular focus was given to institutional capacity building regarding both the EST Model and the Energy Data Center. Fifteen training sessions out of eighteen in total were organized mostly in IT to support the smooth management of the Energy Data Center and EST model. The remaining 3 training sessions had to be cancelled due to the Covid-19 pandemic restrictions. Trainings to facilities which will enter data into the survey system were conducted via online video recordings. The targeted number regarding personnel trained in effective operations of the Data Center was overachieved as more than 400 people participated in training events. 56. In addition to the above, a coherent capacity building program was designed for the increase of the institutional capacity of MENR in modelling of the energy sector and performing projections and planning studies. Tailor-made hands-on training courses (11 sets of trainings with more than 300 hours in total) were prepared to facilitate knowledge transfer while two study visits in three EU countries (Spain, Belgium and Netherlands) were conducted to provide insight to MENR experts on the implementation of planning activities at both the national and EU level. Further, the organization of technical meetings with the modelling team established under the Directorate General of Energy Affairs (DGEA) of MENR which now uses the modelling tools and collects and analyzes the required data to run the models, which has proven to be very useful for the team members. 57. This well-equipped modelling team existing within the structure of MENR, which is one of the main counterparts of the WB in Turkey, helped the Bank to develop complementary and follow-up support for the long-term energy planning studies in Turkey. The WB’s recent interventions on Turkey’s long-term low carbon development options; assessment of pathways for energy sector decarbonization and Turkey’s Country Climate and Development Report involve modelling studies for which MENR became capable of providing significant support. 58. Achievement of this outcome is rated as High. The long-term energy planning and scenarios sub- component was completed with high satisfaction, providing MENR with a very comprehensive and elaborated modelling tool and extensive training to MENR’s modelling team on its use. A new Energy Data Center was successfully established, and a large number of personnel was trained in its operations. Long- term energy scenarios are available for energy policy and strategy decision making whereas the energy data is available from the new Energy Data Center. 59. Activities under Component 4 “Visibility and Public Awareness” contributed to developing a media campaign to raise households’ awareness for EE and RE. Four social media accounts on Facebook, Instagram, Twitter and YouTube were set up under the name “Aklınla Verimli Yaşa”, including a separate website. The campaign kicked-off on October 31, 2019 with a post on World Savings Day. A social media awareness campaign has been run and reached more than 35 million users with approximately 230 social Page 20 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) media posts in total on different platforms. A Communication Plan for MENR’s future EE communication activities in all sectors was prepared. Also, a TV film and a public service advertisement on EE were produced and broadcasted. During “EE Week” from January 11 to 17, 2021, a campaign using hashtags and visuals developed through the project in the billboards was run in collaboration with TV channels broadcasting interviews with the Minister and other relevant authorities. The educational materials (videos, digital booklet) on EE for teachers and 15,000 EE coloring books for kids were designed and delivered. A cartoon character named “Akillim” having adventures promoting EE to reach out to children was designed and 15 animated Akillim videos and four 3D animations on EE were released through the social media accounts. Additionally, a booklet on EE for children, titled “Ali’s Energy Adventure – 2” was prepared. It was a follow-up activity from the Phase I Project, in which the first Ali’s Energy Adventure booklet was prepared, of which 1 million copies were distributed by the MENR. 60. Finally, four separate events for the dissemination of results of IPA 2013 projects were organized. The first event, Symposium on EE in Electricity Generation and Transmission was successfully organized on February 24, 2020, just before the Covid-19 outbreak in Turkey. However, considering the pandemic restrictions, the remaining 2 workshops on Battery Storage; Data Management and Digital Technologies in Energy Generation, Modelling and Projections; and a forum on IPA 2013 EE Projects were held virtually. There were over 1,000 sector participants in these events. Justification of Overall Efficacy Rating 61. All the project outputs and respective outcomes listed above clearly demonstrate the transitional impacts of the Project on the legislative framework and institutional setting in the Turkish energy sector. Hence, the overall efficacy rating is assessed as High according to the aforementioned ratings of each outcome area. Table 2: Overall Efficacy Rating Outcome Area Rating Justification Energy efficiency High The EE potential in the targeted sectors was identified and recommendations for implementation projects were provided to the stakeholders. Institutional capacities were strengthened through extensive training programs and new dedicated units were established in order to ensure sustainability of project outcomes. Legislative framework for and opening of energy storage market was supported. The importance of EE was disseminated through awareness-raising events. Internal Energy Market Substantial The current state of the Turkish energy markets and exchange has been evaluated and recommendations for improvement of energy exchange, balancing power market operations and ancillary service procurement were provided, discussed with relevant stakeholders. Recommendations for their further development involving the introduction of futures products in electricity and gas markets and alignment with EU’s strategies have been elaborated and provided to the responsible institutions. Long-term energy High The Long-term Energy Planning and Scenarios sub-component was planning and modelling completed with high satisfaction, providing MENR with a comprehensive and elaborated modelling tool and extensive training to MENR’s modelling team. Turkey’s electricity demand Page 21 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Outcome Area Rating Justification projection for the years 2020-2040 and the decision-making process for the ratification of Paris Agreement were facilitated by the model developed. Energy Data Center was successfully established, and a large number of personnel was trained in its operations. Overall Efficacy High C. EFFICIENCY Rating: Substantial 62. Economic and financial analyses were not carried out for this project upon design stage as its scope was financing TA and the development of an energy data center. Therefore, it was not possible to fully quantify the impact of the TA on the financial or economic benefit and it was hard to define an internal rate of return (IRR) or economic rate of return for the operation. For the assessment of project efficiency, a qualitative approach was followed. Specifically, the expected economic value of certain project outputs and the extent to which available resources used efficiently were evaluated. Whether the costs involved in achieving project objectives were reasonable in comparison with both the benefits and with recognized norms was also investigated. 63. The work for the EE component included economic and financial analyses of project proposals to improve the efficiency of electricity transmission and generation and gas transmission. Assessing the primary results, listed below, it is possible to assert that the costs involved for the contracts in the EE component were reasonable in comparison with the potential economic benefits identified. a. EE in power generation (component cost: USD 3.88 million): Economic analysis shows that average annual energy benefit per HPP is USD 1.02 million, while the average turbine efficiency improvement associated with recommended interventions was identified as 2.3%, and at a maximum of 6.5%, for EUAS’ HPPs audited in this study. b. EE in power grid (component cost: USD 0.42 million): The results of the cost- benefit analysis carried out in the assessment, suggested a possible overall savings potential of around USD 261.5 million. c. EE in natural gas transmission (component cost: USD 0.40 million): The overall gas savings potential is estimated as 2 million Sm3/yr, whereas Carbon emissions could be reduced by nearly 3 ktons per year, resulting to economic benefits of several hundreds of thousands USD per year. 64. It is considered that the available resources were used efficiently for the following reasons: a. The project led to a large number of high-quality outputs and outcomes which, in general, were delivered on time and within budget. Many of the events were able to reach more participants than originally estimated. Within the scope of Project, more than 170 reports were drafted, more than 2000 staff trained, nearly 150 events organized and 10 study visits to several EU member countries with approximately 100 participants from the government delivered. Page 22 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) b. Competitive bidding processes resulted in tender savings of about EUR 1.9 million. c. MENR successfully used almost all of the tender savings (EUR 1.79 million) by amending several contracts under all components to include additional work. Thus, for the same price more work was completed than expected upon design. d. The project has almost fully disbursed the allocated IPA grants (98.7%). 65. Taking into account the achieved results, the substantial efficiency in use of available resources both on the client’s and WB’s side, overall project efficiency is assessed as Substantial. D. JUSTIFICATION OF OVERALL OUTCOME RATING 66. The project’s relevance is High, efficacy is High, and efficiency is Substantial, thus, the overall outcome rating is Highly Satisfactory. In general, the project managed to achieve its purpose and no shortcomings were recorded. Table 3: Overall Outcome Ratings Relevance of PDO High Efficacy High Efficiency Substantial Overall Outcome Rating Highly Satisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) 67. Gender. Οn the initial review of the four components of the Project, the adoption of a gender lens did not appear to add value to the project design and therefore no gender sensitive analysis or monitoring and evaluation was conducted. However, in hindsight, it must be admitted that a set of results indicators favoring the participation of women to the project activities both as consultants and beneficiaries could be identified in the results framework. 68. Institutional strengthening was the main pillar of the project’s PDO. Remarkable institutional capacity was built at the Project Implementation Unit (PIU) operated under MENR General Directorate of Foreign Relations (GDFR). The department’s capacity at appraisal was seen as a substantial risk to the project. However, strong commitment by MENR and the extended WB support and the joint experience gained through the implementation of both Phases I and II of the IPA program, the department was assessed to be capable of assuming the management of additional large-scale interinstitutional operations, namely the IPA 2018 Phase III and IPA 2019 Phase IV Projects. 69. MENR, EMRA, BOTAS, TEIAS, EUAS and EPIAS as the project’s main beneficiaries were supported extensively through the provision of consulting services and participation in dedicated training events organized under the project. Their technical and administrative capacity has been significantly strengthened, allowing them to further develop their activities in alignment with EU provisions. 70. With this support, EUAS has established a dedicated unit in charge of monitoring the power plants’ performances and planning for EE investments; whereas DGEA of MENR has formed a modelling Page 23 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) team which is a pool of extensively trained experts responsible for developing long term energy scenarios. 71. Mobilizing private sector financing. This TA project did not directly mobilize private sector financing. However, it provided legal, administrative and technical support for properly functioning energy exchange operations in electricity and gas markets and improved MENR’s long-term energy planning capacity which would provide a healthy environment for new investment decisions in the energy sector. Additionally, all market players were provided capacity building support through publicly open workshops and trainings. 72. Poverty reduction and shared prosperity. The project did not have any direct impact on poverty reduction or shared prosperity as the activities were only consultancy services. However, increased functioning of energy markets and cost savings to be gained through EE investments obviously will serve for energy security and have a positive impact on shared prosperity in Turkey. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 73. Experience from Phase I project design phase contributed greatly to the project’s sound design and preparation. 74. Project design. The project was designed as a stand-alone, single donor, Recipient-Executed Trust Fund (RETF) to be managed by the WB and executed by the MENR. The project was the second “installment” in a multi-year TA program in support of Turkey’s energy strategy and the EU/Turkey energy agenda. The overall objective of the program was the achievement of a secure, liberal and transparent Turkish energy market in line with the EU acquis and Europe 2020 energy sustainability targets. The proposed project was financed under the EU’s IPA 2013 program for Turkey. Funds were channeled to Turkey through a Bank-administered trust fund. 75. Project design was complicated as it aimed to address several issues at the same time in Turkey’s portfolio, namely (a) alignment of the EU and the WB business; (b) flow of EU funds into the Government institution through WB management; (c) mobilizing massive funds for the MENR’s self- utilization under consultancy services; and (d) inclusion of a wide range of public sector stakeholders as key beneficiaries. Particularly, the key decision which made the project design complicated, related to points (b) and (c), was the selection of the RETF instrument despite no strict EU obligation requiring the direct use of funds by the MENR. However, the WB decided to proceed with RETF in the anticipation this would allow the Turkish government to ‘own’ the outputs of the project. Additionally, it was considered that RETF modality would allow the MENR to approve the outputs of TA as their own programs and plans. 76. The Project’s PDO presented realistic and clearly defined objectives. The ultimate objective was highly relevant to the overall IPA program scope and aligned also with the Turkish Government priorities in the energy sector. 77. Results Framework (RF) design. The RF design was challenging. Due to the nature of the overall project scope, it was difficult to set the objectives on a quantifiable basis, making results evaluation and progress tracking difficult. However, by adopting several intermediate indicators for each outcome area, it was made possible to develop a RF that would remain focused on all components and subcomponents Page 24 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) of the PDO. The team’s cautious approach at the project’s design stage allowed for realistic objectives that served the ultimate project goals. 78. Plan for M&E. It was agreed that progress monitoring would be conducted by the WB, MEU and EUD on a quarterly basis. It was agreed that MENR would prepare and submit quarterly project monitoring and progress reports (PMRs). A Steering Committee established under the Phase 1 project would also monitor the implementation of the Phase 2 project, achievement of results, and would agree on corrective actions as appropriate. The Committee was chaired by MENR and it was set to meet on a quarterly basis. Other members were WB, MEU, Ministry of Treasury and Finance (MOTF), and EUD. European Bank of Reconstruction and Development (EBRD), related state bodies, European Investment Bank (EIB) and International Finance Corporation (IFC) participated as observers. Other stakeholders were invited to participate when relevant. The Bank was set to supervise the project on a quarterly basis. Several Bank team members were based in the WB Ankara office, which also facilitated close interaction in-between the formal quarterly supervisions. 79. Stakeholder selection. MENR was set to be the lead implementing agency. Within MENR, GDFR was placed as the coordinating and executing body of the project. Within the GDFR, the Department for Foreign Investment Coordination was in charge of project implementation, procurement and financial management. Two other general directorates were closely involved: DGEA (energy policy, electricity and gas, RE) and Department of Energy Efficiency and Environment (DEEE)(energy efficiency). TEIAS, EPIAS, BOTAS, EUAS, and EMRA were set to participate in the implementation of the EE and internal energy market components as were among the main project beneficiaries. MENR and participating agencies have strong technical expertise in the areas of the TA program and thus the stakeholder selection was accurate. 80. Risk identification. Overall risk identification upon project appraisal was relatively accurate. Risk was identified in both the implementation risk and the implementation agency capacity risk which were evaluated as substantial. At the time, MENR did not yet have strong experience in the implementation of a TA programs of this scale, because MENR was still working on procurement activities of Phase 1 project and institutional capacities were not fully developed. Furthermore, the turnover of trained MENR staff posed an additional challenge. Although, most risk categories were rated low or moderate, in view of the substantial capacity risk, the overall implementation risk rating was also set to Substantial. Of course, the global pandemic was not foreseen. 81. Risk mitigation. In the Project Appraisal Document (PAD) a thorough Operation Risk Assessment Framework was elaborated, analyzing key risks and mitigation measures. The Bank was committed to supervise the project closely throughout implementation and promoted the strict implementation its guidelines for the use of consulting services by MENR. EBRD by executing its component of EU’s 2013 IPA program would ease the workload of MENR as it could focus on the implementation of WB-administered IPA 2012 and 2013 activities. 82. MENR team was strengthened by hiring additional staff. Procurement and financial management (FM) training of MENR staff was provided by the Bank. For procurement and contracting, the work under the IPA program was divided to manageable number of contracts so as to minimize the procurement and implementation coordination work falling on the MENR. 83. Readiness for implementation. The overall commitment of key energy sector stakeholders, Page 25 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) appetite of the MENR PIU, and the WB’s highly committed full-fledged team were the strong aspects of the readiness. On the other hand, weaknesses in PIU staffing at the beginning and the parallel implementation of Phase I Project were the main challenges. B. KEY FACTORS DURING IMPLEMENTATION 84. The outbreak of the Covid-19 pandemic, internal political challenges, macroeconomic crisis, insufficient institutional capacity, legislative challenges and policy barriers had notable impact on the project’s implementation. As the two main key factors affecting implementation were exogenous, the Implementation Completion and Results Report (ICR) will first discuss the factors outside the control of the Government or implementing agencies. Factors Outside the Control of Government and/or Implementing Entities 85. From the very beginning of the project, Turkey was experiencing political challenges, that remained prevalent during the implementation period. Two consecutive Parliamentary Elections in 2015 -in June and November- followed by the Constitutional Referendum in 2017 and the Presidential Election in 2018 significantly changed the political landscape of the country. From the signing of the GA to date five different governments have been in charge. 86. Throughout this period, four different Ministers for Energy and Natural Resources have been appointed, resulting in reshuffle of managers in MENR and affiliated institutions, causing a slowdown in project activities and procurement. Major changes in the policy environment and inter-ministerial structures, including MENR, occurred after the Presidential Election in July 2018. General Directorate for Foreign Relations and EU (under which PIU is assigned) was combined with another department and renamed as General Directorate for Foreign Relations and International Projects and then split up again and renamed as GDFR. General Directorate of Renewable Energy which was responsible for RE and EE affairs was integrated into DGEA; while EE and environment related tasks were transferred to the newly established DEEE later on. 87. These institutional restructurings slowed down the decision-making processes regarding both procurement and project implementation activities. Most of the bureaucrats who were in a decision- making position delayed the decisions to conclude the tenders, sign the contracts, assign the staff as members of selection committees, etc. 88. Nevertheless, these changes have not affected PIU’s staffing and assignments. To circumvent risks posed by the changes in high level positions, PIU ramped up internal reporting mechanisms to track progress of individual contracts. Additionally, the new management begun requiring monthly project progress risk assessments from each project being implemented in MENR. The political instability in the country caused a hold in organization of steering committee meetings for several months, negatively affecting the implementation rhythms of the project. 89. Finally, the outbreak of the Covid-19 pandemic has had a severe effect on the project implementation, causing a delay of 1 year in the project’s closure. The Government took a number of measures and restrictions related to the Covid-19 pandemic starting from January 2020. As a result, on- site visits, workshops and conferences had to be postponed indefinitely or cancelled, causing delays in contracts completion. The project closing date set by the GA was extended for a year to May 28,2021, Page 26 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) since 10 contracts at the time of the outbreak had to be extended to allow for completion of activities that require mobility and/or travel of people and/or devices. Factors Subject to Government and/or Implementing Entities Control 90. Project coordination was satisfactory. All project implementation responsibilities as well as intra- government coordination were assigned to the PIU formed under the GDFR and the head of the concerned department was appointed as the PIU Head. GDFR was dedicated in its role and demonstrated remarkable ownership throughout all project phases. 91. Communication between MENR and project beneficiaries proved to be challenging at certain points of the implementation phase. In line with the legal agreements, MENR established a Steering Committee composed of relevant stakeholders. The PIU also established working groups from other beneficiaries to provide technical support to the project. Different groups of beneficiaries were brought together in working groups for each consulting contract. However, as the contracts progressed, the number of relevant stakeholders increased, and it became logistically unfeasible to ensure participation from all of them at each meeting. Coupled with slow progress in some contracts, Steering Committee meetings were not held for a year, undermining project implementation progress. Upon PIU’s proposal, Steering Committee meetings restarted for specific contracts to focus on the technical details and progressed individually since May 2018. 92. Stakeholders’ staffing capacities were one of the main challenges of the project. MENR’s team capacity was recognized early on as a significant risk to implementation. During implementation period, Head of PIU changed while the lack of a FM specialist for quite a while led to additional workload for other team members. The limited human resources capacity of the beneficiary institutions inhibited effective engagement in the project. Most project beneficiaries assigned project related tasks as additional workload to their staff, resulting in response delays. To address this issue, some beneficiaries established further internal working groups. 93. The involvement of project stakeholders was challenging. Lack of ownership from some stakeholders caused significant delays, particularly in procurement processes. This could be attributed also to the fact that the responsibilities of stakeholders were not clearly elaborated and described upon project design, leading to confusion. 94. During procurement phase, some delays were recorded, and the initial procurement plan was updated many times to reflect the actual date of contract awarding and contract amounts. 18 months into the project, almost two-thirds of the procurements were still in the pre-tendering or tendering phase, raising concerns over the project’s timely completion. At that time, PIU was still working on procurement activities of Phase 1 project and sufficient institutional capacity were not fully developed. Furthermore, the turnover of trained MENR staff posed an additional challenge. 95. However, with the completion of Phase I project, increased number of experienced staff in the PIU and extensive support from the WB task team; procurement momentum picked up and all contracts were put back on track to be closed by original implementation deadline of the grant. The initial procurement process resulted in contract savings of EUR 1.9 million, the beneficiary used almost all of it before the closing date. Page 27 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) 96. The large number of contracts -19 in total- have been difficult to manage and convene all project stakeholders for each Steering Committee meeting. The management challenge posed by the large number of contracts within the project was resolved by distributing individual contracts among the seven PIU staff. Thus, responsibility of each staff was concentrated in specific contracts, which alleviated everyone’s workload and facilitated tracking procurement and implementation progress, strengthened the staff ownership, and reinforced the PIU’s capacity. 97. MENR successfully used almost all the tender savings by amending ten contracts mostly in EE assessments, energy modelling, market development and capacity building to include additional work. Several amendments for time extensions of contracts were recorded but were not connected to MENR PIU. 98. Legislative challenges. Although the GA had been effective since November 24, 2015, MENR needed an additional Official Gazette publication authorizing the opening of the Designated Account. This process was only completed in May 2016. 99. Policy Barriers. EMRA issued a public consultation process for a new Regulation on Energy Storage, which allowed TEIAS to own energy storage assets (TEIAS is particularly interested on batteries). This was in contradiction to EU market rules and raised concerns of conflicts of interest and distortions of the balancing & ancillary services competitive market. The WB discussed these concerns with a large EMRA team as well as TEIAS, to work together for viable options for Turkey and the regulation was published in May 2021 in line with the recommendations provided by the Project. 100. EMRA issued also a draft regulation on Demand Side Response for public consultations. The demand-side participation service presented in EMRA’s draft on amending electricity market ancillary service regulation presents design flaws that may result in: (i) inefficient activation of demand-side participation offers; (ii) wealth transfers to demand-side participation suppliers unrelated to services they deliver to the system. The Consultant brought these concerns to the attention of EMRA as well as MENR, both institutions agreed to work together for viable options for Turkey. Factors Subject to World Bank’s Control 101. Implementation support and supervision by the Bank was exceptional. Since the signing, the WB deployed a full-fledged task team with strong technical and operational capability providing extensive technical support to PIU including the following: review and draft TORs, review critical reports/outputs and provide comments, review event agendas and worked to strengthen them, help identify participants, train PIU staff on procurement and fiduciary aspects. The WB also hired specialists for some tasks, an extended-term and a short-term consultant in the Ankara office to follow up on the implementation on a regular basis. 102. The WB’s reporting was consistent throughout the implementation. Ten Implementation Status and Results Reports (ISRs) were reported. Key issues have been reported on time to WB management and MENR top management through 8 Aide Memoires. The WB was also required to report the progress to the EU as set out in the Administrative Agreement (AA). In line with the AA requirements, the WB submitted two interim and a final narrative progress reports to the EU. 103. The Bank supported the PIU to mitigate risks resulting from the shifting policy environment. Page 28 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) Changes in MENR’s inter-ministerial structures, the purge of public officials from other relevant beneficiary institutions and limited capacity of new hires, shifting priorities, and lack of ownership have significantly slowed procurements. The Bank management met with the Deputy Minister of MENR to increase ownership of project activities by key stakeholders, including EUAS, TEIAS and DGEA. Following this high-level meeting, the Bank team has also held several technical meetings with the PIU and representatives of EUAS and DGEA to ensure timely implementation. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) Rating: Satisfactory 104. M&E design. Overall, the project’s theory of change was clear with intermediate indicators linked to components, which underpinned their contribution to achieving the development objective. However, the results framework was not as effective in tracking project achievements at the outcome level, given the nature of most of the TA activities. 105. In the RF the PDO and the vast majority of intermediate results indicators are set on a qualitative basis. The expected progress towards the achievement of the objectives is elaborated for the five-year duration of the project, specifying the desired progress on an annual basis. Several indicators for each project outcome area were included, ensuring that all components and subcomponents would be monitored. 106. Due to the qualitative nature of results indicators and the fact that deliverables were mostly consulting reports and training activities in such a typical TA project, the M&E strategy was designed to focus on contract completion rather than the immediate and long-term outcomes which substantially depend on follow-up actions to be taken by the beneficiary independently of project activities. 107. M&E implementation. The M&E strategy included the following elements: (a) quarterly progress reports, (b) two interim and a final narrative progress reports to the EU, (c) WB team supervision visits to sites and corresponding aide memoires, (c) ISRs, (d) annual financial audits, and (e) an ICR conducted by the PMU. MENR was in charge of data collection and sharing with other stakeholders. Apart from a period of time, when Steering Committee meetings were not held, the M&E implementation process was running satisfactorily; the results indicators were monitored, and progress based on each indicator was timely reported. 108. M&E utilization. The task team’s intensive implementation support, MENR's comprehensive monitoring capacity combined with its strong dedication and ownership contributed to the ultimate satisfactory completion of the project. Close tracking of work progress triggered a timely adjustment of the project’s closing deadline to cope with major setbacks caused by the Covid-19 pandemic. Through the utilization of the monitoring data collected, potential activities to be financed by the contingency funds (tender savings) under existing contracts were identified. Justification of Overall Rating of Quality of M&E Rating: Substantial Page 29 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) 109. Given the fact that this project was the second installment of the IPA program, some of the repeated shortcomings in the M&E design could have been avoided. Despite some difficulties faced, the quality of M&E implementation and utilization was satisfactory. Overall, M&E rating is assessed as Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 110. Compliance with social and environmental safeguards. The project was classified as a Category B for social and environmental scrutiny, requiring a partial assessment and triggered OP 4.01 Environmental Assessment policy safeguard. The project successfully complied with the requirements of this policy. During the project implementation, the average rating of overall safeguard compliance was satisfactory. 111. The Bank triggered its operational policy on ‘Projects on International Waterways’ (OP/BP 7.50) under ‘CS-01c CS for EE in Power Generation’ contract since the first and second amendments to the original contract included EE assessments on cross-border rivers or their tributaries. OP/BP 7.50 is applicable to the project since the feasibility reports and EE assessment activities were carried out in HPPs located on the Euphrates and Tigris Rivers. However, it was determined that Project's EE assessment activities did not concern investments that would lead to adverse changes in the quality or quantity of water flows to other riparian countries, or that would be adversely affected by other riparian’s water use, and therefore fell within the exception under paragraph 7(b) of the Policy. Therefore, an exception to the notification requirement under paragraph 7(b) of the Policy was requested and approved so that riparian notifications were not required. Ultimately, work conducted under this component complied with the Bank’s OP/BP 7.50 operational policy requirements. 112. Fiduciary Compliance. The GA required MENR to (i) maintain an adequate system for the project accounts, (ii) prepare interim un-audited financial statements on a quarterly basis and (iii) have the project accounts audited on an annual basis. The lack of an FM specialist within the PIU at early stages of the project caused disruption in implementation in the following ways: delays in payments were recorded; the PIU could not submit the Interim Un-Audited Financial Reports of the project for the quarter ending June 30, 2016; issues about compliance with the reporting requirements of the GA were noted from the Treasury Controller. MENR appointed a dedicated FM Specialist to the project to resolve the issues. As a result, interim financial report templates were improved, and the reports were submitted on a timely basis. The MENR managed, eventually, to obtain clean audit reports for the project from the Treasury controllers. The FM rating ranged between Moderately Satisfactory and Satisfactory throughout project implementation. C. BANK PERFORMANCE Rating: Satisfactory Quality at Entry 113. The WB performance at entry is assessed as Satisfactory, as detailed in the following paragraphs. The Bank ensured a thorough and comprehensive project design and only a couple of shortcomings are noteworthy. 114. Main strengths: Page 30 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) a. The WB task team had persistent and in-depth communication and coordination with the EU headquarters at Brussels, EUD in Ankara, MENR, and other stakeholders including EUAS, BOTAS, TEIAS, EMRA, and EPIAS to create strong client ownership for each project components. b. The WB mobilized its extensive expertise in various fields of energy sector and provided high quality support in response to wide coverage of project components. c. PDO was highly relevant with country, EU and WB strategic priorities. d. The separate design of components while ensuring contract-component match was logical and pragmatic for ensuring straightforward implementation. e. The implementation agency capacity risk was identified, and mitigation measures elaborated in the Operational Risk Assessment Framework were subject to Bank’s control. f. Safeguard and fiduciary policy compliance and policy institutional arrangements were strong. 115. Shortcomings: a. A clear matrix of responsibilities among the key stakeholders could have been prepared at the design stage and annexed to the PAD. b. The number of contracts was large, leading to delays in the implementation phase and raising significant challenges in progress monitoring. 116. The quite significant risks raised due to the Covid-19 pandemic, which posed also the major factor affecting project implementation were impossible to be foreseen by the WB. Therefore, this issue is not included in the assessment of Bank’s performance at entry. Quality of Supervision 117. Quality of supervision has been assessed as Satisfactory, as detailed in the following paragraphs. 118. Main strengths: a. The WB team provided intensive implementation support to MENR PIU throughout implementation to improve several aspects of the project, such as procurement, financial management and ownership among others. Through this support, the MENR PIU’s capacity was immensely strengthened. b. The WB had meetings with responsible stakeholders to resolve issues raised and ensure the smooth and timely procurement of the contracts. c. Identified the need for triggering the OP/BP 7.50 operational policy. Page 31 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) d. The Bank’s supervision was complemented by the issuance of 10 ISRs, 8 Aide Memoires and two intermediate and a final narrative progress reports to the donor. e. The WB’s timely action to extend the closing date, while processing the extension request both for GA and AA, enabled satisfactory completion of all activities with massively increased disbursement during the extended period. f. The Bank’s supervision quality remained at a very high level, despite the major difficulties the Covid-19 pandemic provoked. The missions and meetings were held virtually. The task team also proposed and requested alternative options to be considered for remaining work not possible to be conducted as planned due to Covid-19 related restrictions. 119. Shortcoming: a. The Bank should have prompted the PIU to figure a solution for the over-populated Steering Committee meetings more quickly. Justification of Overall Rating of Bank Performance 120. The minor shortcomings were of secondary importance in comparison with the many and great strengths of WB performance both at Entry and Supervision stages of the project. Thus, the Bank’s performance is rated as Satisfactory. D. RISK TO DEVELOPMENT OUTCOME Rating: Negligible to Low 121. Administrative and technical institutional capacity built under the different outcome areas of the project will most likely be maintained. Before its completion the project had already started to have outcome impacts on both policy making and institutional structuring. Therefore, the risk to development outcome is rated as low. Some aspect may be delayed if there are extensive staffing changes in the key institutions, or significant budget reductions, but the capacity building efforts have helped ensure many of the analyses and recommendations were broadly institutionalized. An analytical approach for each outcome area is elaborated followingly. 122. Outcome Area 1: Energy Efficiency. The EE studies conducted for its hydro and thermal power plants prompted EUAS to establish a dedicated unit in charge of monitoring the power plants’ performances and planning for EE investments. The analyses conducted for the electricity and natural gas transmission systems provided the TSOs with specific recommendations regarding the improvement of their grids as well as significant knowledge, transferred through a series of training events. Technical and economic feasibility studies on the battery storage technologies considerably contributed to the legislative work conducted by EMRA and the first draft of the regulation on battery storage had been opened to the public consultation just after the related tasks in the project were completed and the regulation was revised and published in line with the recommendations provided by the Project. The impact of this area’s outcome is considered to be sustainable for the coming years. 123. Outcome Area 2: Internal Energy Market. The detailed analyses and actions recommended under Energy Market Development project lay the foundations for the futures operations in both electricity and Page 32 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) gas markets through extensive support provided to EPIAS and TEIAS. Both futures markets got into operation just after the successful completion of the project. During the implementation period, EMRA issued two draft regulations which were not aligned with the EU acquis and the recommendations set out by the consultants. Concerns raised over this were brought to EMRA’s management’s attention, and they agreed to work together with other institutions to achieve a viable solution for Turkey, showing thus its willingness for alignment with the EU acquis sustaining the capacity developed under the project. 124. Outcome Area 3: Long-term Energy Planning and Modelling. The sustainability of the Long-term Energy Modelling component results is of utmost importance for MENR which has already expressed its desire for guidance and support to ensure the sustainability of the project in the coming years, mainly due to possible staff turnovers and need for developing the modeling work further. The capacity built within MENR should be maintained in the future to ensure long term gains. Accordingly, the Bank agreed that all justifications to support the project for any additional work existed and team would seek additional funds to support the MENR in additional energy planning and modeling work provided that staff who have been trained by the project are retained. The Bank also recommended the PIU to seek additional funding opportunities to support the project within the EU’s upcoming financial assistance programs. As for the Energy Data Center, the determinant factor for the sustainability of the project’s outcome is the personnel trained in effective operations of the data center. Since the number of people trained was quite high and exceeded the target set at design, the risk is determined to be low. V. LESSONS AND RECOMMENDATIONS 125. The design of the Phase 1 and Phase 2 projects reflects the experiences and lessons learned by the Government, the EC and the Bank in carrying out TA projects. Turkey has benefited from a series of IPA-financed TAs in several sectors in the past. Normally, the Central Financing and Contracting Unit (CFCU) in the MOTF handles the contracting for Turkey’s IPA TA funds. In some cases, the MOTF and MEU have preferred using the WB trust fund route as an alternative to CFCU’s implementation modality and complementary approach to WB’s other interventions in the sector, not only for this project but also for projects in other sectors, on a highly selective basis. This would enable a blending mechanism leveraging the impacts of TA funds through investment projects funded by Bank financing. In Phase 1 and Phase 2 projects, the Government experienced and acknowledged the Bank’s added value and copied this project implementation modality to other sectors in order to sustain the TA results through implementing recommended investments with the financing support of the Bank. The EC shares this interest as well, with a view to pursue opportunities for EC/WB synergies in working with the Turkish authorities. 126. The stakeholders’ ownership has been the greatest factor affecting project success. In both Phase I and II the MENR PIU staff and head showed strong commitment and dedication for the ultimate achievement of the project’s goals. Without these levels of client ownership, it is doubtful whether certain challenges would have been overcome to allow for such levels of project success. However, lack of ownership from other stakeholders was an issue faced throughout project implementation. The WB needed to act in more than one occasion to urge beneficiary institutions to increase their ownership and promote their further inclusion. Bank’s active role in increasing ownership and motivation of the end beneficiary institutions resulted in formation of dedicated units (EE department in EUAS) and task teams (modelling team in DGEA) to ensure the sustainability of work conducted under the Project. 127. For multi-stakeholder TAs, a clear and sound coordination and responsibility matrix would be Page 33 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934) helpful to include in project design and included in the PAD. MENR would be asked to engage with other stakeholders in the design phase and ensure their ownership at the very beginning. Each party would be well informed from the project beginning of its role and responsibilities and confusion would be avoided. In this way, it is also possible to pre-determine a wider inclusion of project stakeholders, increasing the ownership levels, which as mentioned before, can contribute greatly to project success. 128. For the successful management of an operation of this scale and technical depth, strong technical capacity and expertise both on the WB and the client side is required. Designing of TORs, selection of consultants, and supervision of the consultancy firms require previous experience and deep technical knowledge. Without a doubt, the experience from Phase I and the extensive help from the WB task team helped MENR PIU to successfully perform these actions. However, due to limited staff capacity it was quite challenging for the PIU to deal with the large number of contracts and parallel implementation of the Phase I project. In addition to this, the absence of a FM specialist for some while caused problems in the FM and fiduciary compliance. Hence, the need for adequate and appropriate staffing of the PIU is underlined as a major pre-requisite. The same applies for the WB task team, which often had to take actions and provide support to various parties. It is certain that without the WB support, the project could not have reached these levels of success. 129. The sustainability of the project results is of utmost importance for the Client. Possible staff turnovers and need for developing the conducted work further would require a robust planning for capacity building at the client side and potential additional support by the Bank. The capacity built within MENR and other stakeholders should be maintained in the future to ensure long term gains. Accordingly, the Bank underlined that staff who have been trained by the project particularly in long-term energy modelling, effective operations of the data center and performance monitoring of energy generation facilities should be retained. The Bank also recommended the PIU to seek additional funding opportunities both for further TA and investment projects resulting from project activities. Page 34 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Energy efficiency potential identified Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Energy efficiency potential in Text Energy efficiency Energy efficiency Achieved, energy TEIAŞ, EUAŞ, and BOTAŞ potential has not been potential identified efficiency potential in assessed and projects identified. electricity identified for transmission (TEIAŞ) implementation. and generation (EUAŞ hydro) and in gas transmission (BOTAŞ) assessed and projects identified. 25-Jul-2014 28-May-2021 05-Oct-2021 Comments (achievements against targets): The EE potential assessment in hydropower (EUAS) has resulted in a lending operation (P177630). Objective/Outcome: Internal Energy Market Improved Page 35 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Legal, structural, and Text Critical elements Legal, structural and Achieved, EPIAS was administrative functioning of required for the administrative basis supported in both EPIAŞ improved to align with operation of EPIAŞ not for properly electricity and gas EU energy markets. available functioning energy markets in terms of exchange operations legal, structural, and of EPIAŞ is achieved administrative and EPIAŞ is functions which were functional, operating improved to align with electricity and gas EU energy markets, markets. with high ownership from EPIAS since September 2018. Assessment of efficiency and quality, and recommendations needed to improve energy exchange operations of EPİAŞ has been completed. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Objective/Outcome: Long-term energy planning and modelling developed Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 36 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Target Completion Improved long-term energy Text Long term energy Long-term energy Achieved, the contract scenarios available for energy scenarios not available scenarios available for regarding Energy policy and strategy decision- energy policy and Planning and making. strategy decision- Modeling has been making. Energy data awarded in October available from the 2018 and completed new Energy Data with 8 alternative Center. scenarios prepared. Report on Turkish Electricity Demand Projection for the years 2020-2040 was prepared via the EST model. Data center comprising of hardware and software is established. Survey Information Management System software has come online and gathered the industrial facilities’ energy consumption values for the 2019 Energy Balance Table. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Page 37 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) A.2 Intermediate Results Indicators Component: Energy Efficiency Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Current energy efficiency Text No analysis of current Analysis of the current Achieved, overall 24 conditions of EUAŞ hydro energy efficiency of energy efficiency of HPPs were assessed plants analyzed and EUAŞ hydro plants. the hydro energy for EE under the feasibility study completed. generation facilities of component. For pilot EUAŞ, including needs studies (2 HPPs), analysis and feasibility Performance Test study completed. Report and Final report were completed by June 2019. Larger contract for EE in 22 HPPs was completed by May 2021.Turbine efficiency improvement potential identified reaches 6.3%. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Page 38 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Current energy efficiency of Text Analysis of energy Analysis of current AAchieved, analysis of the electricity transmission efficiency potential of energy efficiency of energy efficiency network of TEİAŞ analyzed TEİAŞ not available. the electricity potential of TEİAŞ was transmission network completed. Based on of TEİAŞ completed. assumption and outcomes of the cost- benefit analysis carried out in the assessment, the roll- out of the suggested reinforcements, as well as other operating recommendations may allow an overall savings of around EUR 225,5 million. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 39 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Energy efficiency of the Text No assessment of Needs assessment Achieved, project was natural gas transmission energy efficiency of regarding energy completed in January system assessed the natural gas efficiency of the 2021. The overall gas transmission system natural gas savings potential is conducted. transmission system identified about 2 completed. million m3s of natural gas per year. Energy savings potential around 10-30% is calculated. The recommendations for the actions to be taken were provided; required institutional capacity for optimized network operation was built through hands-on training sessions. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Component: Electricity and Gas Market Development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of trainings Number 0.00 3.00 19.00 Page 40 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) conducted for capacity 25-Jul-2014 28-May-2021 07-Oct-2021 building in electricity and gas market development Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Technical infrastructure Text Technical Technical Achieved, as EPIAS needs for a properly infrastructure needs infrastructure needs developed its functioning energy exchange not identified to ensure a properly infrastructure in the identified functioning energy meantime. The exchange identified. Consultant reviewed EPIAS's exchange operations and provided and discussed with the Bank team the technical recommendations for a better functioning energy exchange, including on gas market operations and market opening. 25-Jul-2014 28-May-2021 07-Oct-2021 Page 41 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion SCADA system designed Text Existing SCADA system SCADA system, Gas Achieved, needs according to the needs of the not consistent with Management Tools, assessment, design liberalized natural gas sector the needs of the hardware and and technical liberalized natural gas software developed to specifications needed sector the needs of for the proper liberalized natural gas functioning of SCADA sector. system prepared. Additionally procurement for defined SCADA modules has been completed under another IPA Project by BOTAS. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Feasibility studies, roadmap, Text Feasibility studies, Regulation, Achieved, and analysis to enhance the roadmap, and analysis administration, and recommendation Page 42 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) regulation, administration, to enhance the organization of an reports covering and organization of an functioning of an energy market feasibility studies, energy market completed energy market not analyzed and results roadmaps and available of studies analyses on electricity disseminated in a and gas markets have Workshop. been submitted and widely discussed in the government. Several meetings, seminars and workshops were organized to discuss the findings and recommendations. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Needs assessment of energy Text Needs assessment not Needs assessment of Achieved, needs storage and frequency conducted. energy storage and assessment report on regulation systems frequency regulation energy storage and completed and workshop systems completed frequency regulation organized to analyze the and workshop system available for systems. conducted to analyze implementation. The the systems in the analysis suggested area of security of that batteries are Page 43 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) supply and system most cost efficient at stability. providing primary reserve service, half of the demand could efficiently be provided with batteries by 2030 (150 MW out of 275). 2 workshops and 5 trainings were organized to analyze the systems with TEIAS staff. The component was finalized on January 20, 2020. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Component: Long-term Energy Scenarios, Capacity Building and Establishment of an Energy Data Center Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Gaps in the energy data Text Review of current Needs analysis and Achieved, gap analysis collection system identified energy data collection roadmap for for the data collection and roadmap and draft system and legislation elimination of gaps in system and draft legislation for improvement to implement energy data collection legislation for improvement not completed and draft improvement has Page 44 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) prepared done legislation proposed been delivered under to implement the IPA 2012 Energy roadmap. Sector Program Phase- 1 Project. There has not been any new legislation needed for the data collection in the energy sector. Regarding data collection and usage for the software, Data Set Report has been prepared and approved. by the government. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of people trained in Text 0.00 300 495.00 effective operation of the data center 31-May-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Page 45 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Software and hardware for Text No software and Infrastructure for the Achieved, hardware energy data center hardware for energy energy data center and software for purchased data center available equipped with the energy data center required software and was installed and in hardware. operation. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Component: Visibility and Public Awareness Building Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Project Phase II visibility and Text No visibility and public Visibility and public A booklet for school public awareness campaigns awareness campaign awareness campaigns children on energy carried out has been conducted prepared and efficiency was disseminated. completed and 1 million copies disseminated by the Ministry. Social media campaign has reached 35 million users with approximately 230 social media posts in Page 46 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) total on different platforms. A symposium on EE in Electricity Generation and Transmission was organized physically while 3 events disseminating project results on battery storage, data management and energy efficiency were held virtually. 25-Jul-2014 28-May-2021 07-Oct-2021 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of fora, seminars Number 0.00 8.00 24.00 and workshops carried out to disseminate results of the 25-Jul-2014 28-May-2021 07-Oct-2021 project Comments (achievements against targets): Page 47 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of people trained Number 0.00 150.00 595.00 through site visits, symposiums and workshops 25-Jul-2014 28-May-2021 07-Oct-2021 on various aspects of energy efficiency in electricity generation and transmission Comments (achievements against targets): Page 48 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) B. KEY OUTPUTS BY COMPONENT Component 1: Energy Efficiency Outcome Indicators EE potential in TEIAS, EUAS, and BOTAS assessed, and projects identified for implementation Intermediate Results Indicators (i) Current EE conditions of EUAS HPPs analyzed and feasibility study completed (ii) Needs assessment of energy storage and frequency regulation systems completed and workshop organized to analyze the systems (iii) Current EE of the electricity transmission network of TEIAS analyzed (iv) EE of the natural gas transmission system assessed Key Outputs by Component Current EE conditions of 2 thermal power plant units and 24 HPPs were assessed, and remedial actions recommended. Energy storage roadmap for Turkey elaborated EE potential in electricity transmission system has been assessed; identified projects and component activities completed. EE potential studies in gas transmission conducted; saving potential identified; recommendations for the actions to be taken provided. 21 trainings and workshops 4 site visits for capacity building Symposium on EE in Electricity Generation & Transmission Battery Storage Workshop Data Management and Digital Technologies in Energy Generation, Modelling and Projections Workshop Component 2: Electricity and Gas Market Development Outcome Indicators Legal, structural, and administrative functioning of EPIAS improved to align with EU energy markets Intermediate Results Indicators (i) Number of trainings conducted for capacity building in electricity and gas market development (ii) Technical infrastructure needs for a properly functioning energy exchange identified (iii) Feasibility studies, roadmap, and analysis to enhance the regulation, administration, and organization of the energy market completed. (iv) SCADA system designed according to the needs of the liberalized natural gas sector Page 49 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Key Outputs by Component Several aspects of the internal energy market were assessed and recommendations were provided for the improvements in energy exchange operations, transparency, market surveillance, balancing power market, ancillary service procurement and for future developments related with demand side response, market integration and capacity remuneration. SCADA system for Natural Gas System was designed; technical specifications were specified, and a market research was conducted. 3 study visits to ENTSO-E, ENTSO-G, national regulatory agencies, electricity and natural gas TSOs and one energy exchange. 19 trainings for capacity building in electricity and gas market development Component 3: Long-Term Energy Scenarios, Capacity Building and Establishment of an Energy Electronic Data Center Outcome Indicators Improved long-term energy scenarios available for energy policy and strategy decision-making. Intermediate Results Indicators (i) Gaps in the energy data collection system identified and roadmap and draft legislation for improvement prepared (ii) Number of people trained in effective operation of the data center (iii) Software and hardware for energy data center purchased Key Outputs by Component EST model developed User Manual for EST and Data Set Report delivered Energy Data Center installed and commissioned SIMS software installed and come online 495 people trained in effective operations of the data center Trainings in information technologies to support the smooth management of EST model and Energy Data Center Dedicated hands-on trainings for the EST model 2 study visits to Spain, Belgium and Netherlands Component 4: Visibility and Public Awareness Intermediate Results Indicators (i) Project Phase II visibility and public awareness campaigns carried out (ii) Number of fora, seminars and workshops carried out to disseminate results of the project (iii) Number of people trained through site visits, symposiums and workshops on various aspects of EE in electricity generation and transmission Key Outputs by Component Informative booklet for children, Ali’s Energy Adventure-2, prepared and distributed to primary school students. Project web site and social media accounts are active. All project deliverables and events are Page 50 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) available to the public through the web site (www.enerjiprojeleri.eu). Social media campaign has reached 35 million users with approximately 230 social media posts in total on different platforms. 24 events held to disseminate project results 595 people trained through site visits, symposiums and workshops on various aspects of EE in electricity generation and transmission Page 51 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Ayse Yasemin Orucu Task Team Leader(s) Salih Bugra Erdurmus Procurement Specialist(s) Burak Taskin Procurement Specialist(s) Zeynep Lalik Financial Management Specialist Arzu Uraz Yavas Social Safeguards Specialist Esra Arikan Environmental Safeguards Specialist Aleksandre Bibileishvili Team Member Almudena Mateos Merino Team Member David Reinstein Team Member Eyup Mermer Team Member Hiwote Tadesse Team Member Jasna Mestnik Team Member Katsuyuki Fukui Team Member Koji Nishida Team Member Mustafa Ugur Alver Team Member Nickolas F. Frydas Team Member Sanjay Agarwal Social Specialist Selcuk Ruscuklu Procurement Team Tunya Celasin Aydinalp Team Member Budak Dilli Consultant Mehmet Huseyin Yilmaz Consultant Page 52 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY15 10.155 91,393.65 Total 10.16 91,393.65 Supervision/ICR FY16 13.705 103,824.10 FY17 22.672 97,772.76 FY18 34.210 153,591.09 FY19 27.176 221,137.97 FY20 24.554 166,480.15 Total 122.32 742,806.07 Page 53 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) ANNEX 3. PROJECT COST BY COMPONENT Amount at Actual at Project Percentage of Components Approval (EUR, Closing (EUR, Approval (%) millions) millions) Component 1. Energy Efficiency 4.817 5.066 105% Component 2. Electricity and Gas Market Development 1.534 1.583 103% Component 3. Long-Term Energy Scenarios, Capacity Building and Establishment of an Energy Electronic Data Center 4.771 4.210 88% Component 4. Visibility and Public Awareness 0.471 0.613 130% Trust fund administration, program management, and supervision 0.450 Total 12.043 100 Page 54 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) ANNEX 4. EFFICIENCY ANALYSIS Not applicable Page 55 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) ANNEX 5. BORROWER COMMENTS 1) Project Success IPA 2013 Phase-2 Project was designed as the continuation of the Phase-1 Project, the first ever energy sector IPA Project jointly managed with the WB. Similar to IPA 2012 Project, IPA 2013 Project was designed to increase institutional capacity in line with EU acquis. The overall objective of the project was set as “to develop the administrative and technical capacity of the Recipient’s relevant institutions to achieve an increasing alignment with the EU acquis and the Europe 2020 targets in the areas of (i) energy efficiency, (ii) internal energy market, and (iii) long term energy planning and modelling. In line with this objective, scope of work of contracts were developed to include recommendations and roadmaps tailored to MENR’s needs and extensive training events for knowledge transfer. The fact that same implementation modality was used under IPA 2012 and the PIU had gained extensive hands on experience during the implementation of IPA 2012 had a positive impact on the implementation of IPA 2013 especially in the area of efficiency. Thanks to this experience and PIU’s high ownership, PIU successfully tendered and implemented 19 contracts with a total disbursement of almost EUR 11.5 million. Contracts were designed considering the lessons learnt from IPA 2012 and large contracts with multiple stakeholders were avoided as much as possible. The needs were clearly identified and this translated into high stakeholder ownership in contracts which were highly relevant to the current needs of the beneficiaries. Examples of such contracts and respective stakeholders include CS01d storage contract and TEIAS and CS02a Energy Market Development contract and EPIAS. Furthermore, ownership and sustainability of IPA 2013 was further enhanced through design and successful implementation of various training events under each contract. The positive effect of these knowledge transfer events on the sustainability is most evident in the CS03 Long Term Energy Scenarios and Capacity Building contract under which a model tailored to MENR’s needs was designed together with MENR team that will continue to run the model with the scenarios they will develop based on their needs. In short, lessons learnt during contract design and implementation of IPA 2012 were largely taken into account during the implementation of IPA 2013 which was a major contributing factor to its overall success. Another contributing factor was the close communication between the PIU and the WB which had a positive impact on the successful and timely implementation of a large number of contracts. This close cooperation was also instrumental in the allocation of tender savings to ongoing contracts in the form of extensions/amendments. The result was disbursement of 98.6% of all funds despite challenges posed by the pandemic outbreak. As a result, IPA 2013 Project is evaluated as highly successful in terms of relevance, efficiency, efficacy and impact/sustainability aspects. In the below sections, achievements of IPA 2013 Project is assessed in detail. Sections are arranged according to the four components as described under the Grant Agreement. These components are further divided into “intervention areas” which were identified in line with the outcomes they served to accomplish. Non-consulting Services contracts which contribute to the implementation of the Consultancy contracts were not elaborated under the intervention areas. Under each intervention area, one or more Page 56 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) contracts corresponding to the same area are evaluated in detail. In the last section, some recommendations to further enhance success of future projects are outlined. Component 1: Energy Efficiency Intervention Area Corresponding Contracts GA Sub articles Energy Efficiency in Power CS01a, CS01b, CS01c 1.a.i, 1.a.ii, 1.a.iii, 1.a.iv, 1.a.v Generation Relevance: High. In order to be able to measure the efficiency gains from the EE investments in EUAS power plants it is necessary to begin by analyzing the current state of the power plants vis-a-vis energy efficiency. In this sense, analyzing conditional assessment in hydraulic and thermal power plants, measuring turbine performance, assessment of potential efficiency measures including EE audits and finally knowhow transfer for implementing the efficiency measures are essential for a sustainable roadmap for EE in the power generation sector. Therefore, EUAS, it was the best course of action for EUAS to work with and learn from an experienced counterpart with international background on how to carry out such efficiency related measurements and assessmens.t. Considering this, the outputs under this intervention area are highly relevant to solving the problems identified in the assessment of the EE potential in the electricity generation sector on the public side. Furthermore, they are highly consistent with Turkey’s Energy Efficiency Strategy Paper’s strategic goals including decreasing energy losses in electricity generation sector, decreasing carbon emissions in electricity generation, efficient use of energy in the public sector, strengthening of institutional structures, employing advanced technologies and increasing awareness raising activities. Improvement of efficiency in generation is also articulated in the National Energy Efficiency Action Plan and hence activities carried under this intervention area highly in line with action E.9 of the Plan. Finally, the activities are consistent with the MENR’s 2015-2019 Strategic Plan which targeted, “… (iv) rehabilitation studies on the public generation power plants (maintenance management system, power plant performance monitoring system etc.), (v) efficient use of generation capacity of public power plants and increasing availability and capacity usage factors, … , (vii) transformation of existing domestic coal resources into electricity generation investments, (viii) improvement in primary energy intensity for the enterprises and entities and institutions, (ix) rehabilitation and modernization projects boosting generation in existing coal-fired electricity generation power plants”. Efficiency: Satisfactory. With a relatively great portion of the allocated funds (EUR 3.91 million), very significant leverage effect for mobilization of further investments for EUAS have been secured. The results of the techno-economic analysis for hydraulic power plants on the basis of cost and benefit carried out in the assessment, suggested a possible overall annual savings potential of around EUR 20.5 million by only a runner replacement investment and around EUR 805 million by the capital investments to keep them operating for EUAS. Also, the results of a technical analysis for coal-based thermal power plant are presented to EUAS. The contract for Pilot Project (2 EÜAŞ HPPs) (CS01a) is extended for two times, the Page 57 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) first was to extend the workload and the second was a time extension related with the equipment and the first extension works. The contract for coal-fired thermal power plant (CS01b) is extended one time to provide additional equipment technical support. CS01c (22 EÜAŞ HPPs) was extended seven times, first two due to broadening of the project scope, third, fourth and fifth amendments due to time extension caused by pandemic. Further to these five amendments, as a result of the Covid-19 travel restrictions, part of the activities could not be implemented and the contract was amended for sixth and seventh times to reduce the scope. Overall, all tasks were completed timely with only three time extensions signed during the first phases of the pandemic and efficiently within the budgetary framework as planned despite a relatively low workload reduction during the pandemic with the sixth and seventh extensions. Moreover, as the contract was signed for a much smaller budget than anticipated prior to tender, the remaining budget was allocated to other areas and hence financial efficiency was always maintained. Efficacy: High. Intermediate results indicator for this intervention area is “current EE of the electricity generation of EUAS hydro power plants”. In relation to this indicator, the target has been achieved as the analysis of EE potential of EUAS has been completed. The main outcomes included a technical and economical assessment of the electricity generation by EUAS and its components as well as an evaluation of the economic efficiency potential of the HPPs in a sustainable environment. The key findings of the consultant’s study demonstrated that EUAS has a potential of possible overall annual savings potential of around EUR 20.5 million by only a runner replacement investment and around EUR 805 million by the capital investments to keep them. In total, reports for 22 HPPs were prepared covering various topics analyzed. Finally contracts under this area also comprised extensive dissemination activities to disseminate the outcomes of the analysis as well as to transfer the knowhow deriving from the recommendations provided. An effective knowhow transfer was the most fruitful outcome alongside trainings and programs with technical expertise held. Sustainability and Impact: High. The analyses for electricity generation provided EUAS with specific recommendations on efficiency improvement as well as important information conveyed through a series of training events. The fact that external assistance was received from an experienced counterpart with international track record, increased the effectiveness of the knowhow transfer & capacity building, expanded EUAS’s desire to establish professional ties and cooperation with different entities beyond the project schedule and strengthened the prospects for sustainability of the impacts of the assessments/investment recommendations via the established capacity at EUAS thanks to the project activities. Also, the reinforcement recommendations with overall annual savings potential of around EUR 20.5 million by only a runner replacement investment and around EUR 805 million by the capital investments to keep them Page 58 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) will pave the way for further investments in increasing the efficiency of the power generation. The impact of this area’s outcome is considered to be sustainable for the coming years. Intervention Area Corresponding Contracts GA Sub articles Energy Efficiency in Electricity CS01e 1.a.v, 1.a.vi, 1.a.viii, 1.a.xi, Transmission 1.a.x Relevance: High. Capacity building, knowhow transfer and assessment of potential are crucial in creating a sustainable pathway for EE across all sectors. Particularly for electricity transmission grids operated by TEIAS, this intervention areas directly relates to the need for technical capacity improvements for carrying out such assessment. In this respect, activities under CS01e contract are highly relevant to the problems identified in assessment of EE potential on power networks. The contract is highly coherent with the Energy Efficiency Strategy Paper’s strategic goals including reducing energy intensity by 20% per GDP until 2023; and energy losses in industry and service sectors, decreasing energy demand and carbon emissions of buildings, efficient use of energy in the public sector, strengthening of institutional structures, capacities and cooperation, employing advanced technologies and increasing awareness raising activities. Also, the activities are coherent with the MENR’s 2015-2019 Strategic Plan which targeted (i) maintenance, reparation, rehabilitation and modernization studies for publicly owned and operated energy assets to improve their efficiency; (ii) increasing public awareness on EE and saving; and with the Multi-Annual Indicative Planning Document (MIPD) (2011-2013) for IPA which aimed at “increased capacity and better alignment in the EE field” and “promoted EE and renewable energies” alongside the EU’s “20% increase in EE by 2020” target. Efficiency: Satisfactory. With a relatively small portion of the allocated funds (EUR 0.36 million), very significant leverage effect for mobilization of further investments has been secured. The results of the cost- benefit analysis carried out in the assessment, suggested a possible overall annual savings potential of around EUR 225.5 million for TEIAS. The contract was extended twice, first due to time extension and expert change needs, the second for extension of the services to make use of overall contractual savings. Overall, all tasks were completed timely with only one time extension and efficiently within the budgetary framework as planned. Efficacy: High. Intermediate results indicator for this intervention area is “Current EE of the electricity transmission network of TEIAS analyzed”. In relation to this indicator, the target has been achieved as the analysis of EE potential of TEIAS has been completed. The main outcomes of this contract included a technical assessment of the Turkish electricity transmission grid operated by TEIAS and its components as well as an evaluation of the economic efficiency potential of the system’s operations. The key findings of the consultant’s study demonstrated that the Turkish transmission grid has network losses at around 2.23 % of the total consumption. Based on assumption and outcomes of the cost-benefit analysis carried out in the assessment, the roll-out of the suggested reinforcements, as well as other operating Page 59 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) recommendations may allow for overall annual savings of around EUR 225.5 million. In total, 14 reports were prepared covering various topics analyzed. The CS01e contract also comprised extensive dissemination activities to disseminate the outcomes of the analysis as well as to transfer the knowhow deriving from the recommendations provided. To provide an effective knowhow transfer and the most fruitful exchange of experience alongside open-discussion on the power system-related topics, periodic working groups and wide-participation workshops were held at the end of each main topic. On top of the above task-related events, two main activities were organized to provide added value to the knowledge exchange activity: a training on network assessment and a site visit to the Italian TSO’s (TERNA) National Control Centre and SAPEI HVDC link converter station in Borgo Sabotino (Rome). Sustainability and Impact: High. The analyses conducted for the electricity transmission system provided TEIAS with specific recommendations regarding the improvement of its grids as well as significant knowledge, transferred through a series of training events. The fact that external assistance was received from an experienced counterpart with international track record, elevated the effectiveness of the knowhow transfer & capacity building, expanded TEIAS’s desire to establish professional ties and cooperation with the consulting firm beyond the project schedule and strengthened the prospects for sustainability of the impacts of the assessments/investment recommendations via the established capacity at TEIAS thanks to the project activities. Also, the reinforcement recommendations with overall annual potential energy savings of around 225.5 million will pave the way for further investments in increasing the efficiency of the power system. The impact of this area’s outcome is considered to be sustainable for the coming years. Intervention Area Corresponding Contracts GA Sub articles Energy Efficiency in Natural CS01f 1.b.i, 1.b.ii Gas Transmission Relevance: High. EE is crucial for a natural gas transmission system as large as the one operated by BOTAS. In order to quantify the gains from EE investments, BOTAS needed an experienced counterpart to undertake efficiency assessments, which are provided by the activities under this intervention area. The activities regarding EE of the natural gas transmission system under the CS01f contract such as energy consumption optimization for operation of natural gas pipelines and system optimization for operation of natural gas pipelines are highly relevant to the problems identified. In this regard, CS01f is highly coherent with the Energy Efficiency Strategy Paper’s strategic goals including reducing energy intensity by 20% per GDP until 2023; and energy losses in industry and service sectors, efficient use of energy in the public sector, strengthening of institutional structures, capacities and cooperation, employing advanced technologies and increasing awareness raising activities. It is also coherent with the MENR’s 2015-2019 Strategic Plan which targeted (i) maintenance, reparation, rehabilitation and modernization studies for publicly owned and operated energy assets to improve their efficiency; (ii) increasing public awareness on EE and saving; and with the Multi-Annual Indicative Planning Document (MIPD) (2011-2013) for IPA which aimed at Page 60 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) “increased capacity and better alignment in the EE field” and “promoted EE and renewable energies” alongside the EU’s “20% increase in EE by 2020” target. Efficiency: High. Through the capacity building activities and know-how transfer regarding the network operations, significant leverage effect has been secured by a small fund (EUR 0.35 million) without any additional investments. As a result of the TA studies, it was identified that BOTAS has a potential for energy savings around 15-30% per year which puts forth the economic benefits of project. The contract was amended 4 times, first of which is due to a scope and budget extension without a time extension. Second and third amendments were time extensions due to the Covid-19 restrictions regarding the access to the Consultants’ offices. To solve this problem, the Consultant employed teleworking and virtual meetings were held between the Consultant and BOTAS experts. The last amendment covered both scope and time extensions. Overall, all tasks were completed timely with only two time extension and efficiently within the budgetary framework as planned. Efficacy: High. Intermediate results indicator for this intervention area is “EE of the natural gas transmission system assessed”. The target related to this indicator was achieved as the needs assessment regarding EE of the natural gas transmission system has been completed. The main outcome of the activities conducted under CS01f is increased institutional capacity for optimized network operations. The potential for energy savings corresponding to around 15-30% (i.e. 2 million Sm3 natural gas per year) and carbon emissions reduction corresponding to nearly 3 ktons per year are the key findings of the consultant’s study. Through the project activities, BOTAS has gained the capacity to develop further medium and long-term load projections via SPS model, which will facilitate BOTAS to identify additional energy savings potentials. In total, 5 reports were prepared covering the analysis of various aspects of gas transmission. The contract also included trainings to provide an effective knowhow transfer and exchange of experience. On the other hand, old and inefficient simulation software (Synergi Gas) was replaced with SPS model. Through these activities BOTAS’s modelling capacity was increased, enabling BOTAS to operate transmission system more effectively and efficiently. Sustainability and Impact: High. In addition to specific recommendations for improvement of the natural gas transmission system, BOTAS has achieved significant gains from the trainings provided in terms of knowledge transfer. The SPS modelling software developed for BOTAS is an advanced and modern simulation software, and the modelling capacity of BOTAS staff was increased through trainings. Thanks to this knowhow transfer, BOTAS has become able to do undertake the modelling activities on its own. There is an obvious prospect Page 61 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) for impact in terms of the identification of further savings potentials, since BOTAS has gained the capacity to develop further medium and long-term load projections. Intervention Area Corresponding Contracts GA Sub articles Use of Electricity Storage to CS01d 1.a.vii Increase Efficiency in Electricity Transmission Relevance: Satisfactory. Batteries and energy storage are becoming more important with the increase of renewable deployment as they serve to mitigate the as they mitigate the negative externalities of intermittency of renewables particularly on the transmission grid level. In this respect, assessment of needs for energy storage and frequency regulation systems thereof remain key as a first step to develop policies to make use of batteries at the grid level. Therefore, CS01d is relevant to figure out the level of need for energy storage at grid services and corresponds largely to the need for TEIAS to familiarize with the benefits and uses of this ever evolving technology. CS01d is also coherent with Turkey’s Energy Efficiency Strategy Paper which aimed at “providing 30% of total electricity production from renewables”, and with the EU’s the Multi-Annual Indicative Planning Document (MIPD) (2011-2013) for IPA which aimed at “promoted EE and renewable energies” and the EU’s “20% of energy from renewables by 2020 target” alongside the 11th National Development Plan target 491.3. Efficiency: Moderate. The contract for CS01d was signed on April 11, 2018 and the project activities commenced on April 30, 2018. It was first planned to end before January 25, 2019, but the amendment signed on January 24, 2019 extended the deadline of the project to become July 24, 2019. The project at first consisted of 6 tasks and was extended by the amendment signed on July 24, 2019 to January 20, 2020 reaching 10 tasks mostly to include the trainings suggested by the Consultant and needed by TEIAS. Overall, all tasks were completed timely with only one time extension and efficiently within the budgetary framework as planned. Efficacy: Satisfactory. Intermediate results indicator for this contract is “Needs assessment of energy storage and frequency regulation systems completed and workshop organized to analyze the systems”. In relation to this indicator, the target has been achieved as the needs assessment report on energy storage and frequency regulation system is available for implementation. An overview of the potential energy storage roadmap for Turkey was elaborated. Assessments for incorporating energy storage systems into the grid to improve integration of RES were delivered. The analysis suggested that batteries are most cost efficient at providing primary reserve service, half of the demand could efficiently be provided with batteries by 2030 (150 MW out of 275). Different topics relevant to energy storage and frequency regulation were studied and 11 relevant reports -excluding the inception and final reports- were prepared and delivered. 2 workshops and 5 trainings were organized to analyze the systems with TEIAS staff. Moreover, 8 Page 62 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) knowledge transfer events were held: workshops, trainings, site visits, results presentation that were attended by up to 60 experts from TEIAS, MENR, EMRA, the World Bank and ELDER, which corresponds to 500 days received by the stakeholders. Sustainability and Impact: Satisfactory. Technical and economic feasibility studies on the battery storage technologies considerably contributed to the legislative work being done by EMRA and the first draft of the regulation on battery storage had been opened to the public consultation just after the related tasks in the project were completed. Eventually, the by-law on storage activities in the electricity market was issued in the Official Gazette on May, 9th 2021. The required institutional capacity at TEIAS has been established by the capacity building activities under CS01d contract through the know-how transfer activities which will contribute to a possible repeat of the assessments. The impact of this area’s outcome is considered to be sustainable for the coming years. Component 2: Electricity and Gas Market Development Intervention Area Corresponding Contracts GA Sub articles Electricity and Gas Market CS02a 2.a, 2.b, 2.c, 2.d, 2e Development Relevance: High. Turkey has been experiencing rapid demand growth in all segments of energy sector over the last few decades, posing a challenge for decision makers to ensure on the one hand the security of supply and on the other hand a well-functioning energy market. The market operator EPIAS was established in 2015 to operate electricity and gas markets; and only some markets were operational by the time of implementation. In this respect, there was a need for assessment and recommendation studies regarding the market structure and transactions. Moreover, EPIAS’s capacity had to be improved in line with the best available practices in the EU. This project provided international experience to meet these needs. Therefore, the activities under the CS02a contract are relevant to figure out the key findings for the energy markets and provide recommendations and training activities. The activities are also coherent with MENR’s Strategic Plan (2015 – 2019) which aimed at “transparent, reliable and monitorable energy markets through establishment of EPIAS”, “seeking the opportunities of market coupling” and “establishment of a market-based demand side participation mechanism”. The activities of the project are also in line with MENR’s current Strategic Plan (2019 – 2023) which aimed at “increasing the effectiveness and scope of energy markets” and “increasing the transparency and financial predictability”. Besides, the activities are coherent with the EU’s the Multi-Annual Indicative Planning Document (MIPD) (2011-2013) for IPA which aimed at “further alignment with and implementation of acquis on the internal gas and electricity market.” The activities are also coherent with 10th Development Plan, which aims at Page 63 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) “effective market surveillance in energy sector” and “development of the energy exchange to provide reference price signals to the investors”. Efficiency: Moderate. CS02a was originally designed to be completed within 13 months and it was a complicated project in terms of the width of the scope, diversity of the types of deliverables and the complexity of beneficiary and stakeholder structure of the project. With regard to the implementation, the project was originally composed of Inception, Assessment and Recommendation phases. Given the complexities of the project, delays occurred starting from the Inception Phase, mainly due to the differences regarding the understanding of the project by different parties. These differences were even more visible when the first drafts of the outputs under the Assessment Phase were delivered. The Consultant mainly remained on identification of gaps, while the understanding of the Beneficiaries was that the Assessment Phase should also include analyses on the reasons and potential/real outcomes of these gaps. As a result, further delays occurred. However, with the close collaboration and timely intervention of PIU and the WB, the discrepancies in the approach to the tasks were overcome and the implementation of the project accelerated. Under this project, six amendments were made, three of which were time extensions without any additional scope, total of which correspond to eleven months. First five out of this eleven months was due to the differences in understanding of the project by the parties, which occurred as a result of the aforementioned complexity of the project. The remaining six months’ extension was due to the Covid-19 travel restrictions, where the consultants could not travel to Turkey for field work and the events could not be organized. As a mitigating measure, the PIU and the Consultant decided to hold the events online and held ad hoc meetings with beneficiaries to avoid any further delays. Other than those, one out of six amendments was only for amending the payment schedule in order to streamline the disbursements with the actual progress. One amendment was related to the broadening of scope together with a two months’ time extension, while the last one was for narrowing down the scope since it was impossible to organize the closing conference due to Covid-19 restrictions. To sum up, the project was completed in 26 months, doubling the original timeline. Although the six months’ delay can be considered acceptable given the Covid-19 travel restrictions, and the two months’ time extension is sufficient to complete the additional tasks, the five months’ delay during the Inception and Assessment phases stemmed from the inefficiencies in the work by the Consultant and the delays in the commenting rounds by the beneficiaries. Therefore, the project is considered moderately efficient. Efficacy: High. The outcome indicator corresponding to this Intervention Area is “Legal, structural, and administrative functioning of EPIAS improved to align with EU energy markets”, while the intermediate results indicators are (i) “Number of trainings conducted for capacity building in electricity and gas market development”, (ii) “Technical infrastructure needs for a properly functioning energy exchange identified”, and (iii) “Feasibility studies, roadmap, and analysis to enhance the regulation, administration, and organization of an energy market completed”. Page 64 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) The project covered a broad area of intervention including conduction of electricity and gas market assessment studies, determination of shortcomings in the electricity and gas exchange operations, improvement of capacity regarding transparency and market surveillance, analysis of balancing and ancillary services operations and exploration of market coupling and splitting opportunities in electricity market, provision of TA on guarantees of origin system and recommendations for future primary and secondary legislation in gas market. Together with the extension of scope in 2020, additional tasks were integrated in the project, namely analysis on capacity remuneration schemes and assessment and recommendations regarding demand side response and other flexibility opportunities. In terms of the deliverables, the project included assessment and recommendation reports, workshops, seminars, study visits and trainings. The outcome indicator for the project is considered to be achieved since the EPIAS was supported in both electricity and gas markets in terms of legal, structural, and administrative functions which were improved to align with EU energy markets. The first intermediate indicator is considered overachieved, since 6 trainings, 7 workshops and 6 seminars were organized, while the target was set as 3. The second indicator is considered achieved, as EPIAS developed its infrastructure in the meantime and EPIAS's exchange operations were improved through recommendation reports, workshops and study visits. The last indicator is also considered achieved, given that feasibility studies were conducted and roadmaps and recommendations were developed to enhance the functioning of an energy market. Therefore, the efficacy of CS02a is considered high. Sustainability and Impact: High. In terms of sustainability, the project helped developing staff capacity of the beneficiaries and energy market players, which will ensure the continuity of the outcomes. A variety of topics were discussed in the project events, to which a broad range of beneficiaries and stakeholders attended. Improvement of the staff capacity led to new assignments in the institutions, leading to a new perspective on the level of decision making. The project also has important achievements in terms of the realized and forecasted impacts. The market assessments and recommendations, and the study visits contributed to the development of electricity and gas futures markets. The activities regarding guarantees of origin helped EPIAS to establish Turkish guarantees of origin scheme, YEK-G. The government plans a restructuring in the gas market, which shows the prospect for further impact. Lastly, the activities regarding demand side response and other opportunities for flexibility created an impetus in EMRA, which is expected to translate into future projects in this area. Intervention Area Corresponding Contracts GA Sub articles Improvement of Gas IT CS02b 2.f Infrastructure Relevance: High. SCADA system is one of the most important assets for liberalized gas markets due to its ability to allow for better monitoring and transparency. BOTAS’s SCADA System had shortcomings in terms of Page 65 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) meeting the needs of a liberalized gas market. CS02b aims at determining the role and importance of SCADA systems in TSO operations and market operations, studying on existing SCADA facilities of BOTAS to determine main gaps and missing functions, and developing the design of a well-functioning SCADA system. In this respect, the activities under this intervention area are highly relevant to the problems identified. CS02b is coherent with the EU’s the Multi-Annual Indicative Planning Document (MIPD) (2011- 2013) for IPA which aimed at “further alignment with and implementation of acquis on the internal gas and electricity market.” Efficiency: Satisfactory. Within the project, tender documents for the supply of SCADA system were developed, which were aimed to be in line with PRAG rules for a possible future IPA procurement. Therefore, the tender documents were submitted to CFCU for their comments with respect to PRAG rules. Due to delays by the CFCU, the contract was extended 2 times for a total of 3 months. Therefore, given that the 3 months’ extension is not considered substantial, the project is considered efficient on a satisfactory level. Efficacy: High. The main outcomes of CS02b are the need assessment and design of the SCADA system. Intermediate results indicator for this contract is “SCADA system designed according to the needs of the liberalized natural gas sector”. The target related to this indicator was achieved, as the needs assessment and design regarding SCADA system has been completed. In total, 4 reports were prepared. The consultant also made some recommendations about “Corporate Strategy and Structure”, “Operating Model Definition”, “Organizational Alignment” and “Business Process Design”. Sustainability and Impact: High. The activities under this contract enabled BOTAS to invest in the right SCADA system to monitor and operate the natural gas network. The SCADA system was procured by BOTAS in line with the design provided under this project. Moreover, “Data Center”, “Geographic Information System (“GIS”) Tool” and “Demand Forecasting Tool” designed within this contract were procured under IPA 2015 “Supply of Equipment for Improvement of Gas Network Infrastructures” project. Relevant trainings for successful Page 66 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) operation of the SCADA were also provided within a separate project under IPA 2015 titled “TA for Improvement of Gas Network Infrastructures”. Component 3: Long Term Energy Scenarios and Capacity Building and Establishment of Data Center Intervention Area Corresponding Contracts GA Sub articles Long Term Energy Scenarios CS03, CS04a, G-SI-01 3.a, 3.b, 3.c, 3.d, 3.e and Capacity Building and Establishment of Data Center Relevance: High. Analyzing the risks and opportunities posed by developments in the energy sector, their impact on the market, and their interaction with climate and other policies will play a decisive role in whether they will be successful in the long term. In this respect, this intervention area aims at increasing the institutional capacity of MENR and its attached and related institutions in modelling of energy sector and performing projection and planning studies. On the other hand, successful data collection and management are required to achieve an enhanced modelling and projection capacity and conduct the projection and scenario studies successfully. Through implementation of the activities under this area, MENR experts were guided to gather the needed data in the appropriate form for further use in the projection and scenario studies. In addition to the modelling work carried out under CS03 contract, Energy Electronic Data Center (i.e. G-SI-01 contract) was installed, enabling MENR to manage data successfully. In addition, the data to be used in the model were standardized and the Survey Information Management System (SIMS) designed under the CS04a contract started to collect data digitally. MENR increased their institutional capacity and had the opportunity to predict the long-term effects of technological changes, climate policies and decisions on resources in the process of determining long-term strategies and policies. The activities under this intervention area are highly coherent with MENR’s aim to enhance security of supply through long term projections as articulated under Article 5.1 of Electricity Energy Market and Security of Supply Strategy Paper. Furthermore, all three contracts are directly coherent with the Target (6.3) "Supply plans will be made with medium- and long-term demand projections" under the Objective 6 “Increasing predictability in the market” in the MENR’s Strategic Plan. Efficiency: High. The contract for CS03 was signed on 8th October, 2018. Throughout the implementation period, the contract was amended five times. Under the first amendment, a study visit to Norway was included in the scope. The second amendment covers both time extension and budget increase due to an extension of scope regarding the enhancement of the model and to provide additional trainings for the additional scope. Under the third and fourth amendments, the scope of the work didn't change, but time extensions were made in order to carry out the travels and trainings that could not be realized due to the travel restrictions of Covid-19. However, since the Covid-19 restrictions continued, with the fifth amendment the study visit was excluded from the scope of work, replacing it with further enhancement of the model with additional scenarios and respective trainings. To sum up, all outputs were achieved with effective utilization of resource and budget and the budget unused due to Covid-19 restrictions were allocated to Page 67 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) other areas of need. The data center contract was signed 10th May 2019. Purchasing and installation of the hardware was completed timely. There was one time extension without any budgetary increase and it was made to meet the arising needs of the beneficiary which were already within the scope of the contract. As a result, although the data center contract time was extended without any other changes, the project is highly efficient due to the fact that the additional requests of the end beneficiary, which are in the coverage of the scope, due to the needs occurred during the implementation period were completely provided. Efficacy: High. Intermediate results indicators for this intervention area are “developing long-term energy scenarios in order to make decisions about energy policy and strategies”, “training 495 personnel for the operations of the energy model”, “providing energy data from the new Energy Data Center” and “establishment of infrastructure for the energy data center with the required software and hardware”. In relation to modelling of demand and supply and design of scenarios, "The Energy System for Turkey (EST)" was developed. EST is a fully-fledged energy demand and supply model, designed as a single-country tool for detailed energy system projections, energy demand forecasting, power sector planning, as well as for impact assessment of national climate and energy policy decisions with a horizon up to 2070 With the EST model, 8 different scenarios were run and the results of these scenarios were presented to MENR. Seven training modules were realized over the course of 11 weeks (more than 300 hours and 50 workdays in total) with a staged teaching approach where the model is developed in stages hand to hand with the trainees, therefore it can be stated that MENR experts are in a position to fully exploit the EST model capabilities. Additionally, two Study Visits in three (3) EU countries were organized in order to discuss planning related issues MENR faces. Moreover, Survey Information Management System (SIMS) software was developed and data center hardware including virtualization servers, disk arrays, monitoring screens, desktop computers and notebooks required for the proper functioning of both EST model and SIMS software was purchased, installed and put in operation. In the light of this information, the target has been achieved at the level of not only output but also outcome through enhanced capacity of MENR in modelling and scenario design. Sustainability and Impact: High. The activities under this intervention enabled MENR to assess electricity supply, investments, CO2 emissions, costs on the basis of resources for each long-term supply scenario, as well as detailed demand projections for each sector and each energy type, and to gather energy data in a more reliable and robust way via online data management module. Thus, MENR gained the capacity to collect data in a more reliable and robust way and to manage them successfully, and to predict the long-term effects of technological changes, climate policies and decisions on resources for determining long-term strategies and policies. With this capability, MENR collected the industrial facilities’ energy consumption values for the 2019 Energy Balance Table via SIMS and by using this data, prepared a report on Turkey Electrical Energy Demand Projection for the years 2020-2040 via the EST model and published on MENR’s website. Moreover, it is planned to prepare Long Term Energy Planning Report covering alternative energy policy scenarios for the period until 2050 or 2070 via EST model. In addition to the modelling work undertaken during the project, MENR staff acquired the necessary knowledge to design their own scenarios and run the ETS model to obtain the results of these scenarios. Therefore, improvement in the capacity of MENR staff to design scenarios and analyze their impact will translate into sustainable use of the ETS model by Page 68 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) the MENR staff. Using the ETS model will lead to more informed decision making as the policy makers can use the results of the model to estimate the impacts of planned policy changes. Component 4: Visibility and Public Awareness Intervention Area Corresponding Contracts GA Sub articles Organization of Media CS05, NCS02a 4.b Awareness Campaign Relevance: High. EE is one of the main pillars of Turkey’s energy policy and national strategic goals in terms of ensuring energy security supply and climate change. However, although several awareness raising campaigns were organized in the past years, there was still room for improvement, particularly towards households. Therefore, as the main objective of the intervention area is to create a campaign to raise awareness among households on RE and energy efficiency, the “Aklınla Verimli Yaşa” campaign was created specifically for this project to inform households on the potential benefits of simple everyday EE measures. Furthermore, design and dissemination of children’s booklet was implemented under a separate contract to increase awareness regarding EE in school children. Therefore, these two projects are highly relevant in terms of raising awareness in households with the ultimate of aim of encouraging them to implement EE measures in their everyday lives. Activities under these two contracts are also coherent with the National Energy Efficiency Action Plan (NEEAP), under which EE targets have been defined and raising public awareness in RE usage and EE has become even more crucial in order to reach those targets. On the other hand, the Energy Efficiency Strategy Paper published in the Official Gazette on February 25, 2012 highlights increasing awareness raising activities regarding EE in Turkey as a strategic goal. Efficiency: Satisfactory. The contract for the awareness campaign was amended 6 times in total, 4 of which were realized in order to broaden the scope of the contract and 2 were realized for time extension. As the “Aklınla Verimli Yaşa” (Act Smart, Live Efficiently) campaign has been very successful and has reached many people especially on social media, 4 amendments were realized to broaden the scope to include additional activities. Moreover, in order to yield further benefits from the success of this project, as the site visits, face-to-face trainings etc. could not been conducted under other contracts due to Covid- 19 measures, these savings were allocated to awareness raising activities and this allowed MENR to broadcast more public service ads, social media videos and even to create a cartoon character named “Akıllım” (Smarty) which mainly aims to raise awareness among children to be used specifically within the “Aklınla Verimli Yaşa” (Act Smart, Live Efficiently) campaign. The budget of the contract was almost Page 69 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) doubled and the newly added activities have kicked-off the awareness campaign and the number of the people the campaign has reached has substantially increased. On the other hand, Covid-19 measures adversely affected the implementation of some of the activities under this contract such as broadcasting public service ads and videos and also interviews with representatives of the MENR on TV channels as the relevant experts and technical staff could not come together. Therefore, the contract was extended two times, corresponding to a total of one year. Regardless of this, thanks to the great efforts of the PIU, all activities were timely completed after these two extensions. Finally, regarding the contract for design and dissemination of Ali’nin Enerji Serüveni-2, the contract was amended once to allow more room for the design of the booklet. 100,000 copies were produced and distributed to selected schools around Turkey under the contract. The contract was implemented successfully and timely with a relatively small budget. Efficacy: High. Intermediate results indicator of this intervention area is “Visibility and public awareness carried out”. In line with this indicator the target has been achieved since the main outcome of these two contracts is to run a successful awareness raising campaign which will reach out to a wide range of people through social media and other platforms. Under the awareness campaign contract, social media accounts on Facebook, Twitter, Instagram and YouTube were created under the name of “Aklınla Verimli Yaşa” together with a separate website and all the outputs were shared through these accounts and website. The campaign has reached more than 35 million users through these accounts. Moreover, in order to facilitate the awareness raising activities in EE Communication plan was prepared for future awareness raising activities. Under the awareness campaign two TVC films have been prepared and broadcasted and TV channels broadcasted interviews with the Minister and other representatives of the Ministry on EE week. During EE week, billboards all over Turkey were covered with the hashtags and visuals of the campaign. In addition, videos, digital booklets have been prepared for teachers in order to be used in classes. Lastly the cartoon character “Akıllım” (Smarty) was designed specifically for the campaign to reach out the children. Even though the project was adversely affected by Covid-19 restrictions, the campaign has reached more people than expected especially through social media accounts on which the outputs were being posted regularly. Under the children’s booklet contract, 100,000 booklets were distributed. MENR printed and distributed another 900,000 copies and published the online version of the booklet on its website. Sustainability: High. Since all of the accounts created under this awareness raising campaign are handed over to the MENR and the campaign continues to draw attention of targeted groups, the sustainability of the action is considered high. MENR keeps posting videos and other visuals through these accounts on EE and RE Page 70 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) regularly which enables MENR to enhance awareness raising activities and inform the public about the recent developments in the relevant sectors. Moreover, as the intellectual property rights of the cartoon character “Akıllım” (Smarty) is reserved to the MENR, the cartoon character will feature in the social media posts, videos, and TV ads within the scope of the future awareness raising activities of MENR. On the other hand, in order to boost the awareness raising activities in EE and RE, MENR programmed further IPA interventions which includes activities of preparing booklets, videos, TV ads etc. under the same Aklınla Verimli Yaşa (Act Smart, Live Efficiently) campaign. Lastly, the Communication Plan contributed to enhance the capacity of MENR in terms of awareness raising activities, as future awareness raising activities of MENR were assessed and the separate activities were defined for each targeted group. Design of the children’s booklet was also handed over to MENR and the online version of the booklet is accessible through MENR’s website. MENR can use the character Ali in its own children related awareness activities in the future. As both issues of the booklet (the first issue was designed under IPA 2012), children can continue to access it in the future as well. Intervention Area Corresponding Contracts GA Sub articles Fora, Seminars, Workshops to NCS02b 4.a, 4.c, 4.d Disseminate Results of the Project Relevance: High. Emerging areas in energy such as efficiency in transmission and generation, storage, digitalization and data management are at top of the agenda for public and private sector parties alike. MENR in its capacity as the policy maker in the sector encourages private sector to focus on these global trends. As the IPA 2013 Project also touches upon most of these global trends, synergies could be formed between IPA 2013 projects and those investments already undertaken by the private sector. Therefore, events under this intervention area were designed and organized with a view to disseminating the outputs of the IPA 2013 Project and inform the interested parties of the outstanding results of the IPA 2013 contracts. The project is consistent with MENR’s aim to increase EE in the industrial sector as targeted in the NEEAP under Action No S.3. Efficiency: High. Initially the dissemination events under this contract were intended to be realized as face-to-face events. As this was no longer possible after March 2020, the contract was amended four times (two-time extensions to allow for more time to plan and execute the online events and twice to reflect the budgetary changes arising from these events being held online), As a result, the planned events except for the symposium on EE in generation and transmission events were held online and hence reached a greater audience. Therefore, with a relatively small portion of the allocated funds, very significant leverage effect for visibility and awareness has been secured. Additionally, funds that were unused due to the fact that Page 71 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) events could not be held physically were allocated to the organization of these events through an online platform. The video recordings of these events are available online. Efficacy: High. Indicator for this activity is “eight fora, seminars and workshops to be carried out to disseminate results of the project”. This target has been overachieved as 21 fora, seminars and workshops carried out in total. The main outcome of this activity which is set as increasing the level of knowledge and awareness by reaching a target group in energy sector of approximately 500 people in the total was also surpassed as the online events reached a much larger audience in total. Sustainability: High. As mentioned in the efficiency subsection, an online platform was established to organize the events which could not be held physically under this contract. The video recordings of these events are available online and MENR aims to publish the links on the project website (www.enerjiprojeleri.eu). This website holds the outputs of IPA 2012 and 2013 projects and will continue to be operated by MENR. Therefore, it is safe to say that the outputs of these events and all outputs of IPA 2012 and 2013 projects in general will be accessible to those who are interested. As a result, the results achieved in the implementation of IPA projects will continue to be benefited by the public and private parties. Future Recommendations In overall terms, the IPA 2013 Project is evaluated as highly successful by the PIU. The two main contributing factors to the success of IPA 2013 were the dedication and hard work of the PIU and the WB team and the fact that the contracts touched upon some areas that were highly relevant to the beneficiary institutions/units. Storage was very a hot topic for MENR hence ownership of TEIAS under CS01d was high. EPIAS was highly interested to learn about EU experience and use it its market design hence their ownership in IPA CS02a was one of the main reasons behind the successful implementation of the contract. Moreover, long term energy scenario development was a very crucial topic for shaping the future of Turkish energy market, in terms of liberalization and security of supply, which made DGEA of MENR very much involved in the implementation. It is believed that experience gained during the implementation of IPA 2012 and IPA 2013 by both the PIU and the WB team will translate into better design and more efficient implementation of future projects. Despite these accomplishments, some recommendations for future cooperation are as below: • As beneficiary ownership continued to be a soft spot in the implementation of the projects, ownership should be secured since the beginning of the programming phase. Therefore, the beneficiaries should be involved beginning from the design of the projects rather than being asked to submit comments on a draft TOR prepared by WB and/or PIU. • Under IPA 2012, larger contracts involving a larger number of beneficiaries with longer implementation periods were designed. This was avoided under IPA 2013 as it was found that these large contracts were difficult to manage and stakeholder ownership posed a problem. The result was more close cooperation between beneficiaries and the PIU during the implementation of these smaller contracts. One big market related contract, namely CS02a, was implemented under IPA 2013 and despite the diligent work by the PIU, the Beneficiaries and the WB, Page 72 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) coordination of different stakeholders was indeed challenging at times. It is therefore recommended to continue to design contracts with a smaller scope and which cater to the needs of a small set of beneficiary groups. • IPA 2013 wide Steering Committees were difficult to organize and largely inefficient due to the high number of contracts and beneficiary institutions/units. It is recommended to limit the program level steering committees to WB and PIU staff and continue to monitor activities through contract based steering committees and ad hoc meetings if necessary. • At times it was difficult for the WB team to review a large number of reports as well and this sometimes resulted in delays in the review and approval of reports. It is recommended for WB assign technical experts to review reports for each contract and inform the PIU of these experts. These experts should ideally be involved since the design of the scope of work. • Even though WB provided hands-on support, financial management and payments continued to be an administrative burden for the PIU. WB should seek ways to provide a dedicated financial manager to the PIU. • In order to increase the level of achievement of indicators and to design indicators that are more relevant and easier to measure, the WB and the PIU should focus more on the design of PAD outcomes, outputs and indicators during the negotiation phase. 2) Administrative Performance and Lessons Learned for PIU and WB PIU: Staff capacity for design, tendering and implementation The staff working in the implementation of the IPA 2012 Project were involved in the IPA 2013 Project since the programming phase. The same staff undertook tendering, contract management including payments and financial management. The administrative capacity of the PIU was mainly due to hands on experience gained during the Phase-1 Project. Despite personnel turnover and the pandemic outbreak, PIU staff managed to complete the implementation of all contracts, contracted the unused funds via amendments and disbursed 98.6% of the allocated funds. Administrative Measures and Continuity During the contracting and implementation, however, staff turnover due to unpaid and maternal leaves, and the adverse impact of the pandemic, posed challenges in retaining the staff capacity at a certain quality. Nevertheless, the PIU managed to recruit and train the new staff in line with the requirements of the operational procedures. Effective hand-over to new staff was ensured through personal efforts, close teamwork and digital media despite the pandemic conditions and teleworking were employed to minimize the effects of the pandemic on the timely implementation of the project. Delays in tendering and implementation periods The pandemic outbreak of 2021 resulted in delays in most of the ongoing contracts. The negative effects of these delays were mitigated by the hard work of the PIU and the beneficiaries. Nevertheless, time extensions were inevitable. The PIU staff and beneficiaries employed teleworking which resulted in a Page 73 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) somewhat longer response time to outputs. However, all activities under all contracts were still completed within the implementation deadline. Monitoring and Review of Outputs Monitoring and review of outputs were challenging at times for the PIU staff who had to manage multiple contracts at the same time. In some contracts beneficiary ownership problems resulted in some delays as well. However, thanks to the high dedication and ownership of PIU and close cooperation with the beneficiaries, significant delays that would affect the success of the contracts were avoided. Financial Management and Payments Financial management and payments were implemented by the PIU who found it time-wise challenging to handle contract management and financial issues at the same time. However, thanks to the hands-on experience gained during IPA 2012 Projects, payment processes were much faster and smoother compared to IPA 2012. WB: Guidance and assistance to PIU staff for design, tendering and implementation As opposed to IPA 2012, most contracts under IPA 2013 were post review contracts which meant WB did not have to review the contracts in advance. WB team was however always willing to advise the PIU on procurement related issues. WB team was also involved in TOR preparation stage and provided valuable technical input that was also instrumental in the design of scope of work that was highly relevant for the Turkish energy sector. Enforcement for Administrative Measures and Continuity A dedicated WB team was instrumental in ensuring timely guidance. Both formal correspondence and ad hoc communication between the WB and PIU ensured that delays were not significant to the successful implementation of the project. Monitoring and Review of Outputs WB review of outputs were highly welcome and WB team provided valuable input to project reports. However, at times delays in review of reports by the WB posed challenges for the effective implementation of the contracts. It is recommended for WB assign technical experts to review reports for each contract and inform the PIU of these experts. These experts should ideally be involved since the design of the scope of work. Financial Management and Payments Under IPA 2012 ICR, PIU recommended the WB to assign a financial manager to work at MENR with the PIU. This did not materialize under IPA 2013 as well. WB, however, provided hands-on support to PIU in financial matters and payments. Even though the PIU is much more efficient in processing payments Page 74 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) compared to IPA 2012, it is still at times an administrative burden for the team. WB should seek ways to provide a dedicated financial manager to the PIU. Page 75 of 76 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P151934 ) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) Page 76 of 76