Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004416 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF16532 ON A GRANT IN THE AMOUNT OF EUR 10.625 MILLION (US$14.6 MILLION EQUIVALENT) TO THE REPUBLIC OF TURKEY FOR THE EU/IPA ENERGY SECTOR TECHNICAL ASSISTANCE PROJECT ( P131921 ) March 26, 2018 Energy and Extractives Global Practice Europe And Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective March 25, 2018) TRY 4.85 = EUR 1 TRY 3.94 = US$1 FISCAL YEAR July 1 – June 30 Regional Vice President: Cyril E Muller Country Director: Johannes C.M. Zutt Senior Global Practice Director: Ranjit J. Lamech Practice Manager: Sameer Shukla Task Team Leader(s): Ayse Yasemin Orucu ABBREVIATIONS AND ACRONYMS AA Administrative Agreement bcm Billion cubic meters BETF Bank-executed Trust Fund CFCU Central Financing and Contracting Unit CPS Country Partnership Strategy CPF Country Partnership Framework EBB Electronic Bulletin Board EC European Commission EE Energy Efficiency EMRA Energy Market Regulatory Authority EPC Energy Performance Contracting EPIAS Energy Exchange Istanbul ESCO Energy Service Company ESP Energy Service Provider EU European Union EUD European Union Delegation in Turkey FM Financial Management FRiT Funding for Refugees in Turkey GA Grant Agreement GDEA General Directorate of Energy Affairs GDFREU General Directorate of Foreign Affairs and EU GDP Gross Domestic Product GDRE General Directorate of Renewable Energy GTP Gas Trading Platform ICR Implementation Completion and Results Report IPA Instrument of Pre-accession Assistance IPF Investment Project Financing IRR Internal Rate of Return ISR Implementation Status and Results Report M&E Monitoring and Evaluation MENR Ministry of Energy and Natural Resources MEU Ministry for EU Affairs MIPD Multiannual Indicative Planning Document MoD Ministry of Development Mtoe million tonnes of oil equivalent NGO Nongovernmental Organization PAD Project Appraisal Document PDO Project Development Objective PIU Project Implementation Unit PV Photovoltaic RE Renewable Energy REOI Request for Expression of Interest RES Renewable Energy Source RETF Recipient-executed Trust Fund RF Results Framework SOE State-owned Enterprise SMEs Small and Medium Enterprises TA Technical Assistance TEDAS Turkey Electricity Distribution Company TEIAŞ Turkey Electricity Transmission Company ToR Terms of Reference TTL Task Team Leader The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ..................................................... 6 A. CONTEXT AT APPRAISAL .........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 10 II. OUTCOME .................................................................................................................... 11 A. RELEVANCE OF PDOs ............................................................................................................ 11 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 12 C. EFFICIENCY ........................................................................................................................... 21 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 24 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 24 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 26 A. KEY FACTORS DURING PREPARATION ................................................................................... 26 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 27 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 30 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 30 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 31 C. BANK PERFORMANCE ........................................................................................................... 31 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 33 V. LESSONS AND RECOMMENDATIONS ............................................................................. 34 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 37 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 51 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 53 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 54 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 55 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 68 ANNEX 7. LIST OF ENERGY EFFICIENCY AUDITS CONVERTED INTO INVESTMENT SO FAR ....... 69 Page 1 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P131921 EU/IPA Energy Sector Technical Assistance Project Country Financing Instrument Turkey Technical Assistance Loan Original EA Category Revised EA Category Partial Assessment (B) Organizations Borrower Implementing Agency Republic of Turkey Ministry of Energy and Natural Resources Project Development Objective (PDO) Original PDO The development objective is the enhancement of the Turkish energy sector in line with the EU Energy priorities and strategies in energy efficiency, renewable energy, and the natural gas market. Page 2 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 7,247,613 12,707,872 11,874,098 TF-16532 Total 7,247,613 12,707,872 11,874,098 Non-World Bank Financing Borrower 552,792 0 0 Total 552,792 0 0 Total Project Cost 7,800,404 12,707,872 11,874,098 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 30-May-2014 30-May-2014 04-Apr-2016 31-Mar-2017 28-Sep-2017 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-Feb-2017 5.79 Change in Results Framework Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 09-Jan-2015 Moderately Satisfactory Moderately Satisfactory 0 02 09-Jul-2015 Moderately Satisfactory Moderately Satisfactory 2.22 Page 3 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 03 30-Jan-2016 Satisfactory Satisfactory 2.22 04 14-Jul-2016 Satisfactory Moderately Satisfactory 3.86 05 09-Feb-2017 Moderately Satisfactory Moderately Satisfactory 5.79 06 12-Jul-2017 Moderately Satisfactory Moderately Unsatisfactory 5.79 07 29-Sep-2017 Moderately Satisfactory Moderately Satisfactory 9.89 SECTORS AND THEMES Sectors Major Sector/Sector (%) Energy and Extractives 100 Renewable Energy Biomass 5 Oil and Gas 23 Renewable Energy Geothermal 5 Public Administration - Energy and Extractives 9 Renewable Energy Solar 5 Renewable Energy Wind 5 Other Energy and Extractives 48 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 46 Enterprise Development 46 MSME Development 46 Finance 27 Financial Infrastructure and Access 27 MSME Finance 27 Public Sector Management 23 Public Administration 23 State-owned Enterprise Reform and 23 Privatization Page 4 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Environment and Natural Resource Management 5 Climate change 5 Mitigation 5 ADM STAFF Role At Approval At ICR Regional Vice President: Laura Tuck Cyril E Muller Country Director: Martin Raiser Johannes C.M. Zutt Senior Global Practice Director: Laszlo Lovei Ranjit J. Lamech Practice Manager: Ranjit J. Lamech Sameer Shukla Kari J. Nyman, Ayse Yasemin Task Team Leader(s): Ayse Yasemin Orucu Orucu ICR Contributing Author: Mustafa Ugur Alver Page 5 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL 1. Country context. During the pre-appraisal period, Turkey rebounded strongly from the global crisis with gross domestic product (GDP) growth reaching 9.2 percent and 8.8 percent in 2010 and 2011, respectively. At the time of appraisal, in 2012, full year GDP growth eased to 2.2 percent and was followed by a staged recovery in 2013 with real GDP growth reaching 4.3 percent. Economic growth was projected to slow down significantly in 2014 with the existing account deficit improving on the back of stronger exports. At the end of appraisal, the growth expectation was not high; thus, it was foreseen in the Project Appraisal Document (PAD) dated May 29, 2014, that the growth would range from 3 to 4 percent for 2015–2017. The real growth based on constant prices in 2015 and 2016 was realized as 4.0 percent and 2.9 percent, respectively, which also verified the projections in average. 2. At appraisal stage, Turkey’s accession to the European Union (EU) was on top of the political agenda. Accession negotiations started in October 2005. Turkey viewed the accession process as its own, fundamental ‘modernization project’, and after the June 2011 elections, the Government upgraded the institution overseeing EU accession matters into a new Ministry for EU Affairs (MEU). At the time, the EU had provisionally closed 1 chapter (Science and Research) and opened negotiations on 12 other chapters. 3. Sector context. By the time of appraisal, although the energy chapter had not been opened for EU Acquis negotiations, energy was a topic of key interest in EU-Turkey relations. Securing an adequate and reliable energy supply for Turkey’s growing economy, efficiently and in an environmentally sustainable manner, was a major challenge. After a temporary slowdown during the global financial crisis between 2008 and 2009, the demand for energy, especially for electricity, had grown rapidly. The continuing demand growth could have led to supply shortages if generation investments would have fallen behind load growth as happened in 2006 and 2007.1 Gas import was another potential bottleneck, in terms of both volume and price. Ensuring the most competitive prices was also critical, as energy imports (mostly oil and gas) accounted for over 20 percent of Turkey’s imports and about 50 percent of the current account deficit. 4. National priorities. To achieve Turkey’s development goals and realize sustainable growth, the Government was pursuing a wide range of economic policies and structural reforms, which were elaborated in various strategy documents including the Electricity Market and Supply Security Strategy (2009), the Energy Efficiency Strategy (2012), National Renewable Energy Action Plan (2014), the National Climate Change Strategy (2010), and the Action Plan (2011). Turkey’s 10th Development Plan (2014– 2018), approved by the Parliament in July 2013, builds on these strategies and successful fundamental energy sector reforms. 1 The relatively good performance of the Turkish economy between 2002 and 2012 has been a major driver of energy demand and investment in the Turkish energy market. Total primary energy supply reached 129.2 million tonnes of oil equivalent (Mtoe) from 80.6 Mtoe with an average annual growth rate of 3.2 percent between 2000 and 2015. Over 10 years (2007–2016), electricity demand rose almost 46 percent and reached 278.4 TWh with an annual growth rate of 4.3 percent. The gas demand grew even faster, rising from 14 billion cubic meters (bcm) in 2000 to 49 bcm in 2015. Although economic growth slowed down in 2013, energy demand continued to grow more than the world average. All these figures emphasize the critical importance of energy security for Turkey. Page 6 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Table 1. Energy Import, Total Import, and Current Account Deficit Share of Current Account Energy Import Total Import Years Energy in Total Deficit (US$, (US$, billion) (US$, billion) Import (%) billion) 2005 21.3 116.8 18.2 21.0 2006 28.9 139.6 20.7 31.2 2007 33.9 170.1 19.9 37.0 2008 48.3 202.0 23.9 39.4 2009 29.9 140.9 21.2 11.4 2010 38.5 185.5 20.8 44.6 2011 54.1 240.8 22.5 74.4 2012 60.1 236.6 25.4 48.0 2013 55.9 251.7 22.2 63.6 2014 54.9 242.2 22.7 43.6 2015 37.8 207.2 18.3 32.1 2016 27.2 198.6 13.7 32.6 5. The energy sector was highlighted for EU support in the 2011 –2013 Multiannual Indicative Planning Document (MIPD) adopted by the European Commission (EC) in June 2011. As indicated in the MIPD: “By completing the energy sector liberalization process Turkey hopes to establish a well- functioning, free and competitive energy market. This requires sustainable private sector investments and a well-functioning and regulated energy market, while limiting environmental damage, reducing greenhouse gas emissions, and increasing energy efficiency.” The EC, the Ministry of Energy and Natural Resources (MENR), and the MEU collaborated closely on the preparation of the envisioned multiyear technical assistance (TA) program for the enhancement of Turkey's energy sector in line with the EU energy priorities and strategies. 6. Rationale for World Bank’s involvement. The World Bank was and remains a key partner for policy and system reforms in the Turkey energy sector. At the design stage, the project was consistent with the Turkey Country Partnership Strategy (CPS) (FY12–FY15). The CPS Strategic Objective 3: ‘Deepened Sustainable Development’ included policy advice and financing to address energy, environmental, and climate change challenges in an integrated manner. The project objectives were aligned with CPS Thematic Area 6: ‘Improved supply of reliable and efficient energy, increased use of renewable energy sources and climate actions under implementation’. Two of the specific CPS outcomes under Thematic Area 6, namely Outcome 2: ‘Renewable electricity generation as a percentage of total generation increased’ and Outcome 4: ‘Cumulative energy savings of 4,372 GWh or 1.5% total annual demand by 2016 to be achieved’, were supported by this operation. 7. Higher-level Objectives that the project contributes. The project contributes to the realization of Turkey’s energy sector development objectives and the alignment of the energy sector with the energy priorities and strategies of the EU. It also contributes to the implementation of (a) Turkey’s Electricity Market and Supply Security Strategy, Energy Efficiency Strategy, National Renewable Energy Action Plan, and National Climate Change Strategy and Action Plan; (b) the EC’s MIPD; and (c) the Turkey CPS. Page 7 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 8. Through the CPS (FY12–15), the World Bank provided multiple funds to Turkey for enhancing electricity and gas supply security, improving competitive energy markets, and promoting renewable energy and energy efficiency projects. They included Development Policy Financing and Investment Project Financing (IPF), as well as additional financing to existing operations. The IBRD energy sector lending to Turkey reached US$2.7 billion during the CPS period.2 The World Bank provided extensive TA and analytical/advisory support in Turkey’s energy sector (see Table 2). To scale up the World Bank’s TA support in the energy sector, a joint pilot operation with the EU was planned, with the potential to expand to other sectors. Therefore, the first installment of a large multiyear EU/IPA-financed programmatic TA program for the energy sector was endorsed by the EU-wide IPA committee in 2012. Table 2. IBRD Energy Sector Lending and Advisory Services and Analytics Deliveries to Turkey - FY12–17 9. Theory of change (Results Chain). The project has been designed as a free-standing, single donor, hybrid Trust Fund (Bank-executed Trust Fund [BETF] and Recipient-executed Trust Fund [RETF]) to be managed by the World Bank and executed by the MENR The project included consultancy services under five different components. The long-term outcomes were envisaged as the enhanced alignment with the EU Acquis in the energy sector. Please see Figure 1 for details. Project Development Objectives (PDOs) 10. The PDO is the enhancement of the Turkish energy sector in line with the EU energy priorities and strategies in energy efficiency, renewable energy, and the natural gas market. 2 FY17 deliveries are also included in this calculation as the new Country Partnership Framework (CPF) covers FY18 –21. Page 8 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Key Expected Outcomes and Outcome Indicators 11. The realization of the PDO was planned to be measured through the achievement of the following key outcome indicators: (a) MENR Energy Sector Alignment. The MENR approves a program for the enhancement of the Turkish energy sector in line with the EU energy priorities and strategies, based on a review and regulatory impact assessment of the alignment of Turkey’s legal and institutional framework for electricity, natural gas, energy efficiency, and renewable energy with the EU’s Energy Acquis. (b) Natural Gas Market Development. BOTAŞ and Energy Market Regulatory Authority (EMRA) approve a design and regulations, respectively, for the wholesale trading of natural gas. (c) Renewable Energy Development. The MENR approves a plan for the acceleration of renewable energy development including the improvement of Turkey Electricity Transmission Company (TEIAŞ) power system flexibility to integrate a larger share of renewable electricity. (d) Energy Efficiency Promotion. Turkish financial institutions receive and review financing applications for energy efficiency projects for small- and medium-scale industries and buildings. Figure 1. Results Chain of the Project Page 9 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Components 12. The following components are supported by the project: 1. MENR (EUR 0.925 million). Sub-components; (i) institutional review and capacity building of the MENR and (ii) review of alignment of Turkey’s legal and institutional framework for electricity, natural gas, energy efficiency and renewable energy with the EU’s energy acquis 2. Natural Gas Market Development (EUR 2.5 million)3. Sub-components; (i) support for the unbundling of the national gas supply and transmission company BOTAŞ; and (ii) design and regulations for a gas trading platform and a plan for effective gas market opening 3. Renewable Energy Development (EUR 2.2 million). Sub-components; (i) a review of current situation, barriers and road map; (ii) renewable energy grid integration; and (iii) business development services to small and medium-sized industries 4. Energy Efficiency (EUR 4.5 million). Sub-components; (i) market development including the provision of business development services to small and medium-sized industries and (ii) capacity building for commercial energy efficiency lending market 5. Visibility and Public Awareness (EUR 0.5 million). B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 13. The PDO and outcome targets were not revised throughout the implementation. However, the end target dates of the outcomes were revised in line with the extended closing date. Revised PDO Indicators Not applicable. Revised Components Not applicable. Other Changes 14. The project’s closing date was extended by 6 months to ensure satisfactory completion of the planned activities. The closing date extension allowed the project to complete all the activities with the total amount of the grant almost fully disbursed. 3 In the officially approved Project Appraisal Document and Legal documents, component 2 was Natural Gas Market Development and component 3 was Renewable Energy Development. However, in the SAP, Procurements plans and other implementation documents, Component #2 and #3 are switched. Nevertheless, this ICR refers to the PAD and Legal Documents in numbering the components. Page 10 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Rationale for Changes and Their Implication on the Original Theory of Change Note applicable. II. OUTCOME A. RELEVANCE OF PDOs 15. Relevance of PDOs is High. The project objectives are highly relevant to the country’s development agenda and the World Bank’s CPF at closing. The PDOs in the EU project fiche were set as “the enhancement of Turkey’s energy sector in line with the EU energy strategies and priorities” for the overall IPA program. However, the IPA 2012 project (Phase I) introduced a more targeted PDO by inserting the specific areas of “renewable energy, energy efficiency and the natural gas market.” 16. World Bank strategies. The project objectives remained as the key priorities for the energy sector cooperation between the Government of Turkey and World Bank as confirmed by the CPF (FY18–FY21). The energy sector strategies framed under two objectives in the new CPF, namely Objective 7 (Improved Reliability of Energy Supply and Generation of Green Energy) and Objective 9 (Increased Sustainability of Infrastructure Assets and Natural Capital). 17. Objective 7 of the new CPF aims to help Turkey reduce its energy dependence, support the energy reform agenda, and diversify its energy generation (among other things, to include more renewables). Harmonization with EU standards is also identified as a key objective in the new CPF. The CPF program encompasses (a) increasing the percentage of renewable electricity generation and improving its integration into the grid through the ongoing Renewable Energy Integration (P144534), the Private Sector Renewable Energy and Energy Efficiency (P124898), the Geothermal Development (P151739), and the EU/IPA Energy Sector TA projects (P131924 – P151934); and (b) developing energy trading and restructuring of BOTAŞ through TAs and Development Policy Loans. 18. Objective 9 aims to improve the efficiency of energy consumption which is critical for Turkey’s competitiveness and sustainable economic growth. Therefore, not only did the EU Acquis agenda remain as a key anchor for the Turkish energy sector but the focus areas identified in the PAD also remained key priorities in Turkey’s energy reform program as the new CPF is endorsed. 19. Government strategies. The 10th Development Plan of Turkey (2014–2018) was approved by the Turkish Parliament in 2013 before the Board approval of the project. Two specified energy programs were defined in the plan as Domestic Resource-based Energy Production Program (including renewables) and Energy Efficiency Improvement Program, which were well aligned with the PDOs related to renewable energy and energy efficiency. The plan also emphasized the importance of enhancing competition in energy markets, which also aligned with the development of a GTP. 20. EU strategies. The Framework Agreement between Turkey and the EC regarding the second period (2014–2020) of IPA II was signed in February 2015. The relevant priority for the energy sector in the Framework Agreement are promoting renewable energy and energy efficiency, improving electricity market via new performance-based tariff calculation methodology, and improving gas markets through the enabling of sustainability of the gas networking system. The PDOs thoroughly align with current EU Page 11 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) policies described in the strategy paper as the EU was committed to the issues of increasing the use of renewable sources, improving energy efficiency, ensuring supply security through the means of establishing fully liberalized energy markets, enhancing market integration, and developing necessary infrastructures. 21. The PDOs are still highly consistent with the Government, World Bank, and EU’s current strategies for the energy sector in Turkey. Therefore, relevance of PDOs is rated as High. Table 3. The Relational Matrix of Project Components and Basic Policy Documents Electricity MENR National 9th Five Market Energy 10th Five MENR 2010– Renewabl Year and Supply Efficiency Year 2015–2019 2014 e Energy Developm Security Strategy Developm Strategic Strategic Action ent Plan Strategy Paper ent Plan Plan Plan Plan (2006) Paper (2012) (2013) (2015) (2010) (2013) (2006) MENR (Energy Comp. #1 Sector x – x x x x x Alignment) Natural Gas Comp. #2 Market – – – – x – x Development Renewable Comp. #3 Energy x x x x x x x Development Energy Efficiency Comp. #4 x x x x x x x Promotion B. ACHIEVEMENT OF PDOs (EFFICACY) 22. The objective of the project is “enhancement of the Turkish energy sector in line with EU energy priorities and strategies in energy efficiency, renewable energy, and the natural gas market.” However, the PDO outcome indicators did not provide a definite target for level of enhancement. This indicates that the project’s aim was to progress toward EU energy strategies and policies rather than committing on a specific quantifiable outcome. 23. The PDO indicators were set on a ‘qualitative’ basis. The targets were framed as ‘Program, Plans, or Financing Applications’ and focused more on describing outputs and process rather than on specifying outcomes. For example, approving a program (as seen in the first three PDO indicators) may be considered ‘attributable’, but its impact on the enhancement outcome is immeasurable. In Turkey, there are many regulations and programs, which were approved in the past but not implemented fully or successfully, such as Energy Efficiency Strategy Paper and Natural Gas Market Law. 24. In light of the above, it is possible to assert that the current Results Framework (RF) does not allow to evaluate the overall efficacy of the project just by relying on the stated indicators. Nevertheless, this Implementation Completion and Results Report (ICR) evaluates whether the PAD-defined outcome indicators (the plan/program/feasibility study) are delivered or not; then, it discusses to what extent the Page 12 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) output contributed to achieve the expected broader outcome. In addition to the indicators specified in the RF, Table 4 summarizes the project’s efficacy. Table 4. PDO and Outcome Areas Outcome Areas Results PDO. Enhancement of the Turkish energy sector in line with the EU energy priorities and strategies in energy efficiency, renewable energy, and the natural gas market Outcome Area 1: Enhancement of the • 260 energy audits were realized. Turkish energy sector in line with the EU • 79 subprojects, developed by the audits, were converted into energy priorities and strategies in energy investments. efficiency • 17 regional forums were held in different regions with various sectors. • An informatory booklet on energy efficiency targeting children was published and disseminated to 100,000 students in primary schools. The MENR printed another 900,000 copies for further dissemination. • The market for EE in small and medium enterprises (SMEs) was triggered among various sectors. • Commercial banks are involved in financing of SME EE projects and contributed to the World Bank’s SME EE project (P122178). • The capacity of the banking sector for evaluating and financing EE projects belonging to SMEs improved because of the project developed tools. Outcome Area 2: Enhancement of the • A total of 155 small-scale renewable energy projects were Turkish energy sector in line with the EU developed with feasibility studies and bankability reports. energy priorities and strategies in • To date, 69 out of 155 feasibility studies have turned into renewable energy investment. • Social and environmental safeguards step-by-step guide according to RE technology outlining how the environmental and social issues can be addressed efficiently when planning a new facility and when going through the licensing process was developed. • A small-scale RE road map has been endorsed. • Non-hydro renewable capacity reached 10,940 MW with a share of 12.8 percent; the entire capacity is connected to the grid. Outcome Area 3: Enhancement of the • The Gas Trading Platform (GTP) design is completed. Turkish energy sector in line with the EU • EMRA issued a regulation for the constitution of the GTP. energy priorities and strategies in the • In line with this regulation, EPIAS announced to initiate market natural gas market operations by May 2018. • The Natural Gas Market Legislative and Implementation Report presenting the unbundling options for BOTAŞ was produced. Cross-Cutting Outcome Area: • The General Directorate of Foreign Affairs (GDFREU) built Enhancement of the Turkish energy remarkable capacity with solid experience in managing large- sector in line with the EU energy scale TA, strong capacity to review terms of reference (ToRs) priorities and strategies and prepare tendering documents aligned with the World Bank and EU procedures. • Regulatory Impact Assessment Report coupled with a training Page 13 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Outcome Areas Results was developed. • The website for the EU/IPA TA project was created and utilized for dissemination of products (http://www.enerjiprojeleri.eu/). • In total, 3,000 participants attended trainings, workshops, site visits, and so on. Outcome Area 1: Enhancement of the Turkish energy sector in line with the EU energy priorities and strategies in energy efficiency Rating: Substantial 25. This outcome area is mainly supported by Component 4 which comprises two fundamental tasks: (a) market development, including business development services to SMEs, and (b) capacity building for the commercial EE lending market. This task had 28 deliverables with the MENR, GDRE, TEIAŞ, Financial Institutions, energy service companies (ESCOs), SMEs, and nongovernmental organizations (NGOs) being the main stakeholders. 26. Under the market development subcomponent, several useful reports were produced. These are the EE Market Assessment Report, Horizontal Technologies Report, Energy Saving Measures by Process Report, Policy Support to Ensure Effective Implementation of the EE Strategy Report, Benchmarking Guideline, ESCO Development Roadmap, and ENEVEL Tool Brand Report. The reports covering diverse aspects of energy efficiency supported each other to improve the market, as well as the market players in an efficient manner. Among those, the instruments such as ESCO Model Contract Template, Guaranteed Savings Energy Performance Contracting (EPC) Template, Lease Contract Template, Shared Savings EPC Template, Verified Savings EPC Template, and Benchmarking Tool are assessed as fundamentally useful references for further EE market development in Turkey. The outputs were widely disseminated in the regional forums and in the trainings which helped enhance the EE sector. The tools are also publicly disseminated through the project website (http://ipaee2012.enerjiprojeleri.eu/en/18/documents). However, the General Directorate of Renewable Energy (GDRE) is still working to publicize all those produced material through its website, as the most effective way to disseminate. In case the GDRE fails to disseminate those outputs in its website, it will be an important missed opportunity in further enhancing the EE sector. 27. The most critical output contributing to the outcome area was the energy audits. A total of 260 audits4 were carried out in industrial SMEs, service sector buildings (hospitals, hotels, and so on), and municipal units. Before the audits were initiated, Standardized Energy Audit (for buildings and SMEs) and Bankability Report Templates were developed. Bankability Report Templates are the first of their kind in Turkey and were designed to help facilitate the banking sector’s understanding of these technical energy audit reports. The audit reports along with credit assessments were elaborated. In this regard, communication with the banks was facilitated on behalf of the audited plants. The contractor and subcontractors were guided to consider the possibility of solar applications and cogeneration/tri- generation alternatives as well. As a result, of 260 energy audit reports (for a total of 848 subprojects) produced, 37 of them were implemented, 42 of them are in the planning phase, and 120 of them are being considered for investment (see annex 7). The total investment size for the implemented projects 4The original target in the Administrative Agreement, ‘supporting at least 15 SMEs in developing EE Projects’ has been surpassed. Page 14 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) has reached US$1.68 million so far. The total investment cost needed to realize all the subprojects created through the audits was estimated at US$297 million, creating an annual cost savings of US$87 million and CO2 savings of 0.2 million tons. 28. Through the awareness-raising activities, a comprehensive list of seminars, workshops, and trainings were undertaken for several stakeholders. Seventeen different forums have been held with various stakeholders in different regions. The stakeholders included Chambers of Industry, organized industrial zones, sectoral associations, and energy service providers (ESPs) (so-called ESCOs). In that respect, the project reached the maximum number of stakeholders for EE promotion compared to any other project in Turkey so far. 29. Under the capacity building for the commercial EE lending market subcomponent, a comprehensive report on New Banking Products and Financing Modalities was produced. Training programs were also developed and seven different training sessions with differing topics have been implemented for several commercial banks in different regions. Guidelines for Banks, Guidelines for ESCOs, and Guidelines for Leasing Companies provided a good basis for this task as well. Report for Energy Efficiency Toolbox and Investment Appraisal Tool, including nine spreadsheets dedicated for different aspects, are utilized by the commercial banks which would like to evaluate the EE projects. 30. The project also contributed to the Turkey SME Energy Efficiency Project (P122178) financed by the World Bank. Through public banks, a total of US$201 million was envisaged to be channeled to SMEs for EE projects. The awareness raised through the IPA TA contributed to SME EE projects’ implementation as some of the funded projects were developed through the audits conducted in this project. Table 5. PDO and Intermediate Result Indicators Measuring Outcome 1 PDO/Intermediate No. Target Actual % Indicator 4 Energy Efficiency Turkish financial institutions receive and 260 SME EE audits No Promotion review financing applications for energy completed; 37 numeric efficiency projects for small and medium- implemented already target scale industries and buildings available 11 Formulation of at least 25 energy 260 audits completed; 848 100% efficiency projects for small and medium- subprojects developed sized industries and buildings 12 Consultant Engagement: Yes Consultant Engagement: 100% Yes 31. Overall significant progress was observed in this outcome area. Turkey had already been struggling to find appropriate ways to improve EE investments in SMEs and buildings at the time the project was designed. Despite several earlier attempts, the ESCO model could not be developed; moreover, local banks mostly avoided financing such projects. Through the project activities, effective tools, which will ultimately be useful when the legal barriers are eliminated, were developed. The Government has recently issued legislation regarding EPCs, allowing public buildings management to enter into 15-year contracts. As a follow up, the Ministry recently requested the World Bank’s guidance to prepare a secondary legislation regarding EPCs. If the GDRE publicizes all tools and reports developed under the project through its website, as expected, the impacts might be more visible. The undertaken energy audits, on the other hand, provided a solid base for empowering ESPs and enhancing the capacity Page 15 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) of FIs. The project’s contribution, which focused on SMEs energy efficiency instead of large industry, was considered a solid step in enhancing energy efficiency in Turkey. As a result, this outcome area is assessed to reach its intended objectives. Thus, the outcome rating is Substantial. Outcome Area 2: Enhancement of the Turkish energy sector in line with the EU energy priorities and strategies in renewable energy Rating: Substantial 32. This outcome area is supported by Component 3, which originally comprised three subcomponents: (a) Review of current situation, barriers, and road map; (b) Renewable energy grid integration to distribution and transmission systems; and (c) The provision of business development services to small and medium-sized industries. Component 3 included 15 deliverables with the MENR, DGRE, TEIAŞ, Turkey Electricity Distribution Company (TEDAS), EMRA, NGOs (GUNDER, TUREB, JESDER, ELDER), and Electricity Distribution Companies being the main stakeholders. The achievements regarding the outcome area are summarized in the following paragraphs. 33. A comprehensive plan to ensure the acceleration of renewable energy development in Turkey integration of a larger share of renewable electricity have been prepared under the project. Recommendations included regulatory issues, network planning, network operation, connection procedures. Workshops and trainings on different aspects of RE such as sustainability of renewable energy policy, renewable energy sources (RESs) integration, economics of RE heating and cooling, offshore wind technology and market, global solar energy practices, and financing mechanisms for FIs were successfully carried out with inclusions of wider stakeholders. Focusing on the issues with RES integration challenges faced in the transmission system, field visits to Italy (energy storage and converter station) and Germany (onshore and offshore wind grid connection) were also realized. Guidelines for Investors, based on the existing processes, have been prepared on geothermal power plant, wind, solar, and biomass. The Government has recently initiated work to assess technical potential for offshore wind and the Energy Minister announced a forthcoming auction for a large-scale offshore wind farm. 34. A financing and support Table 6. Realization of Investments for the RE Feasibility Studies mechanism has been proposed for Project Status Count small-scale RE development. A total Credit disbursed/construction about to start 42 of 155 small-scale renewable energy Credit disbursed/under construction 13 projects5 were developed with Under construction with equity/waiting for credit approval 5 feasibility studies and bankability reports. Table 6 summarizes the Under construction with equity/no credit application made 4 progress level of 69 projects, with Project returned to investment 5 progress to date. Overall, the GRAND TOTAL 69 estimated investment amount for these projects reached US$130 million, annually generating around 183,447 MWh of electricity and saving 93,142 tCO2 equivalent of emissions. A simple calculation tool for website users was designed to assess the feasibility of small-scale wind, solar, heat pump, biogas, and landfill gas technologies. These tools were disseminated to the relevant audience through a workshop. 5 The original target in the AA, ‘supporting at least 10 SMEs in developing RE Projects’ has been surpassed. Page 16 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 35. One of the critical issues which was highlighted during project preparation was the ‘environmental and social impacts’ of the renewable energy projects. Increased number of small- and medium-scaled hydropower plants and growth in wind power and geothermal sector ignited public concerns regarding the ecological impacts of these developments. This resulted in some scrutiny in legislation and was therefore identified as one of the barriers in the energy sector. To respond to this issue, the TA project involved site selection guidance notes (geothermal, wind, solar photovoltaic [PV], small hydro, biomass power projects) to developers as a part of the deliverable on ‘(c) Renewable Energy Development - (i) review of current situation, barriers and road map’. These guidance notes provided valuable information regarding (a) licensing process for new power facilities; (b) environmental legal framework relevant to mentioned renewable energy projects; (c) major environmental and social issues which have to be considered when designing, locating, building, and operating renewable energy projects; (d) environmental and social standards used by international financial institutions in their decision making on financing of such projects; and (e) a step-by-step guide outlining how these environmental and social issues can be addressed efficiently when planning a new facility and when going through the licensing process. Guidance notes also provide information to developers on the benefits of a meaningful and continued stakeholder engagement and a strong grievance redress mechanism. 36. The outcome area targeted to support TEIAŞ in improving the system flexibility to integrate a larger share of renewable electricity. The development of on-grid renewable installed capacity, particularly the small RE power plants, increased significantly in recent years. The build-up and increasing share of wind generation and emerging solar generation pose major integration and technical challenges to TEIAŞ, which in turn require sophisticated responses from TEIAŞ. At the end of 2017, the non-hydro renewable capacity reached 10,940 MW with a share of 12.8 percent. The total small-scale (<1 MW) RE capacity reached 3,444 MW from a level of 0 in 2013, just before the project started. Table 7. PDO and Intermediate Result Indicators Measuring Outcome 2 PDO/Intermediate No. Target Actual % Indicator 3 Renewable Energy The MENR approves a plan for The MENR approved a plan for the No Development the acceleration of renewable acceleration of (small scale) numeric energy deveelopment including renewable energy development target the improvement of TEIAŞ including proposal for improvement available power system flexibility to of TEIAŞ power system flexibility to integrate a larger share of integrate a larger share of renewable renewable electricity. electricity. 9 A review of current situation, A review of current situation, 100% barriers and road map for barriers and road map for renewable renewable energy, and a plan energy, and a plan for renewable renewable energy grid energy grid integration to integration to distribution and distribution and TEIAŞ transmission TEIAŞ transmission systems. systems have been completed. Formulation of at least 5 A total number of 155 renewable renewable energy projects for energy prefeasibility and bankable small- and medium-sized reports for small- and medium-sized industries and buildings. industries and buildings have been completed as of the Implementation Page 17 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) PDO/Intermediate No. Target Actual % Indicator Status and Results Report (ISR) date. 10 Consultant Engagement: Yes Consultant Engagement: Yes 100% Figure 2. Distribution of RE Installed Capacity in Turkey 40000 50 35000 40 Installed Capacity (MW) 30000 25000 30 20000 15000 20 10000 10 5000 0 0 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 HYDRO GEOTHERMAL WIND SOLAR BIOMASS + W.HEAT Renewable Share (%) Source: TEIAS 37. The enhancement of the energy sector in renewable energy, both in terms of installed capacity and regulatory framework, is achieved. The activities carried out through the project, particularly the knowledge transfer from European Transmission System Operators (TSOs) played a role in strengthening TEIAŞ’s response to grid integration of RE investments. Some of the training materials produced through this component were publicly disseminated in the project website, to enhance the project’s impacts (http://yeipa2012.enerjiprojeleri.eu/en/25/training-materials). Overall, the efficacy of this outcome area is assessed as Substantial. Outcome Area 3: Enhancement of the Turkish energy sector in line with the EU energy priorities and strategies in natural gas sector Rating: Substantial 38. This outcome area is supported by Component 2, which comprises two subcomponents: (a) Support for the unbundling of the national gas supply and transmission company BOTAŞ, and (b) Design and regulations for a gas trading platform and a plan for effective gas market opening. 39. On the GTP side, the achievements were high. The review of the existing Electronic Bulletin Board (EBB) of BOTAŞ and feasibility studies for a new EBB, functional and structural design of a GTP,6 and related components (including support on elements of necessary regulations) considering its integration with 6GTP is a fully electronic, cleared organized market offering natural gas trading in continuous spot and long-term financial or physically delivered contracts for the delivery of natural gas in Turkey. The GTP consists of three subplatforms: (a) Balancing Gas Platform, (b) Spot GTP, and (c) Futures and Derivatives GTP. Page 18 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) BOTAŞ’ SCADA and simulation systems were designed to contribute this outcome. Review of Existing EBB and Report for Functional Design and Specifications for the New EBB were produced. In terms of structural and functional design of GTP, six main reports were produced: (a) Assessment of European and Turkish Gas Markets, (b) Analysis and Review for Turkish Market Model Report, (c) Analysis of Algorithm Requirements Report, (d) High Level Requirements for Turkish Market Platform Report, (e) Technical and Functional Design Report, and (f) Final Design Specifications Report. A market model was proposed, which consisted of three main components: GTP; Collateral, Clearing, and Settlement Mechanism; and Capacity Allocation and Transparency Platform. 40. Afterwards, additional complementary tasks were added and delivered based on the MENR’s top management’s request: namely, assessment and recommendations for the establishment of Turkish natural gas trading hub; assessment and recommendations for the limitations of BOTAŞ stemming from the Decree Law on SOE No. 233;7 trainings on energy trading, risk management, pricing methodologies; ENTSOG 10-Year Network Development Plan; tariff structures; and congestion management for BOTAŞ. The trainings for each were for three days and were delivered with high attendance level. 41. The design of the GTP was completed and modified following in-depth consultations with EMRA, EPIAS, and BOTAŞ. In addition, trainings through five site visits to Italy, Germany, Austria, Belgium, and the United Kingdom were successfully organized with the participation of key gas sector stakeholders. After systematic endeavors exerted through the project, a certain breakthrough was achieved. The Government of Turkey decided to establish the GTP, and EPIAS announced the interval steps for operating the new system based on the work carried out under this outcome by mid-2018. The necessary legislation, `the organized wholesale gas trading regulation`, is issued in 31/03/2017 and promulgated in Official Gazette numbered 30024. Therefore, based on the tangible and visible progress, the target was met and expected capacity was created. Table 8. PDO and Intermediate Result Indicators Measuring Outcome 3 PDO/Intermediate No. Target Actual % Indicator 2 Natural Gas BOTAŞ and EMRA approve a GTP design has been prepared, No Market draft design and regulations, widely discussed, and delivery numeric Development respectively, for a mechanism approved by the MENR, EMRA, and target for the wholesale trading of EPIAS. EMRA and EPIAS have available natural gas commenced work to integrate GTP in EPIAS. Elements of necessary regulations have been drafted. 7 Design and draft regulations for Design and draft regulations for a 100% a mechanism for the wholesale mechanism for the wholesale trading trading of natural gas of natural gas has been achieved. 8 Consultant Engagement: Yes Consultant Engagement: Yes 100% 7The Decree Law on state-owned enterprises (SOEs) No. 233 was put into effect in 1984. It sets the rules and procedures that SOEs should comply with for productive, profitable, and efficient operation. As one of the options is to create a new SOE that is only responsible for gas transmission, the restricting and/or regulatory articles of the mentioned Decree Law had to be elaborated. Page 19 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 42. Gas unbundling (the mandatory requirement of EU Acquis) was linked to this outcome area although it has been bundled under contract 1. Under this subcomponent, the Natural Gas Market Assessment Report, assessing Turkish gas market for functionality and current legislation, was delivered. The Natural Gas Market Legislative and Implementation Report presenting the unbundling options for BOTAŞ was also produced. These BOTAS unbundling issue were widely discussed by stakeholders. Even though those comprehensive reports provided useful information for portraying the road map and capacity building, unbundling of BOTAŞ requires strong governmental decisiveness beyond the project objectives. The endeavors successfully leveled the playing field through building capacity in the sector to discuss the unbundling of BOTAŞ. Such discussion did not exist before the project. It can also be asserted that the efforts exerted in the GTP subcomponent have supported this task. Yet, the concrete results of this task were and are subject to the high-level decisions of the Government. 43. The BOTAŞ unbundling issue is not only a technical decision but also a strategic decision for the Government of Turkey. The project contributed strongly to inform key stakeholders on the technical aspects of the unbundling issue and initiated the ‘first discussion’ among key Government stakeholders. As the project never committed to complete BOTAŞ unbundling, the current situation is not assessed as a failure for the project objectives. Thus, the outcome area is rated Substantial considering strong achievement in GTP and acceptable level of achievement in unbundling. Cross-cutting Outcome Area: Enhancement of the Turkish energy sector in line with the EU energy priorities and strategies 44. The objectives also included a cross-cutting outcome which envisaged the general enhancement of the Turkish energy sector in line with the EU Acquis. This was broadly supported by the activities clustered under the first and fifth component: (a) institutional review and capacity building of MENR; (b) review of alignment of Turkey’s legal and institutional framework for electricity, natural gas energy efficiency, and renewable energy with the EU’s Energy Acquis; and (c) visibility and public awareness activities.8 45. Reports on developing the expertise capacity of the MENR staff on energy statistics, energy planning, energy modeling, and demand forecasting were of crucial importance. Energy Demand Forecast Planning Report, Energy Statistics and Accounts Report, and so on were delivered and served for capacity enhancement at the MENR. The Energy Demand Forecast Report also provided a baseline for the Long-Term Energy Planning Scenarios contract in the follow-up operation (Phase-II). 46. Extensive EU Acquis alignment-related works, including a screening and gap analysis, workshops, and trainings, were undertaken. A Regulatory Impact Assessment Report, coupled with a training, was developed to improve decision making. A large team from the MENR actively joined the studies, which inevitably enhanced the capacity to determine the gaps in an updated manner. In total, more than 3,000 participants have attended the trainings, workshops, and forums. Support for preparing EU Acquis screening tables was provided and booklets were produced. The MENR and MEU officials expressed that the deliverables were provided on time as inputs to the EU-Turkey energy chapter discussions and formed the baseline. 8 Component 5 contributed to achieve all outcome areas in a cross-cutting manner. Page 20 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 47. Increased visibility of the MENR and its related projects and increased public and media awareness on energy efficiency, renewable energy, EU Acquis alignment process, and EU energy policies were completed through the Visibility and Public Awareness Component. Through this task, organizational support to project meetings, workshops, and regional forums were carried out, informative videos were produced, a spot film was screened on TV channels, and an informative booklet on energy efficiency targeting children was published and disseminated to 100,000 students in primary schools. The MENR printed an additional 900,000 copies of the booklet for further dissemination. The website for the EU/IPA TA project was created and utilized for dissemination of products (http://www.enerjiprojeleri.eu/). 48. The activities under this outcome area helped meet the PDO, the enhancement of the Turkish energy sector in line with EU energy priorities and strategies, which also required capacity enhancement of the MENR. As a result, the MENR is now aware of existing gaps in EU Acquis alignment and is ready to focus on the missing points when the energy chapter is opened for discussion. The efforts helped improve the capacity in several institutions operating under the MENR. The main counterpart, the Project Implementation Unit (PIU) of the GDFREU of the MENR, which is also responsible for EU alignment studies, only had limited capacity at the beginning of the project. Certain and visible competencies were developed throughout implementation so that the PIU could undertake similar projects/programs with high confidence. The GDFREU is now assessed as a fully competent unit with solid experience in managing large-scale TA and strong capacity to review ToRs and prepare tendering documents aligned with World Bank and EU procedures. Justification of Overall Efficacy Rating 49. The overall efficacy rating is assessed as Substantial given all three outcome areas achieved Substantial results. Table 9. Overall Efficacy Rating Outcome Area Rating Outcome Area 1: Enhancement of the Turkish energy sector in line with the EU energy Substantial priorities and strategies in energy efficiency Outcome Area 2: Enhancement of the Turkish energy sector in line with the EU energy Substantial priorities and strategies in renewable energy Outcome Area 3: Enhancement of the Turkish energy sector in line with the EU energy Substantial priorities and strategies in natural gas market Overall Efficacy Substantial C. EFFICIENCY 50. The project did not include a financial and economic analysis at the design stage. Therefore, it is not possible to fully quantify the impact of the TA on the financial or economic benefit of the actual investment.; it is hard to define an internal rate of return (IRR) or economic rate of return for the operation. Nevertheless, this ICR will still assess the efficiency of the operation in terms of: (a) what the expected monetized value of the project’s benefits was and the degree to which these targets could be met; and (b) whether the resources could be harnessed efficiently at the design and implementation stages. Page 21 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 51. Components 3 and 4 could have been suitable to conduct ex post analysis with some degree of quantification. However, no quantitative investment targets/estimations were made at design stage. For Components 1, 2, and 5, defining targets for monetized value of the project’s benefits was neither meaningful nor easy. 52. Considering the actual results observed both in Component 3 and Component 4, the efficiency is very promising. As mentioned in the efficacy section, 69 renewable energy projects with an investment cost of US$75.5 million were either constructed or under construction. Considering the total cost of Component 3 (EUR 2.2 million), this is a significant success. Regarding EE, as mentioned earlier, a total of 260 audits (848 subprojects) were conducted, 37 of them were implemented, 42 of them are in planning phase, and 120 of them are being considered for investment. The total investment cost of 79 projects implemented/under construction is US$1.58 million and total annual cost saving is US$0.54 million. It is important to note that savings is calculated on an annual basis and the cumulative savings will be at least 10 to 15 times bigger. 53. The efficiency in ‘use of resources’ for the overall project implementation was Substantial in line with the following factors: (a) The project led to a large number of high quality outputs and outcomes which were delivered on time (many project feasibilities, several of which materialized into investments). (b) Actual component costs remained more or less the same as the estimated component costs at appraisal. (c) The total administrative costs incurred for implementation support did not exceed the EU funds provided through the BETF to cover supervision costs). (d) A dedicated core team remained the same and maintained continuity throughout the implementation period. The local-office-based co-TTL remained the same during the entire project implementation together with key task team members such as procurement, financial management (FM) specialists, and technical staff from Energy GP. (e) The disbursements reached 98 percent at the end of the grace period. (f) Although the average time for a regular IPF operation is five years, this operation achieved full completion within a three-year implementation period. (g) The proactive decision of the PIU and the task team in issuing the expressions of interest for contracts before the signing of the Grant Agreement helped save considerable time in the implementation process. 54. On the client’s side, the resource efficiency was Substantial. (a) The MENR received all planned outputs with high quality on time through an operation executed in house. This solid success has been achieved despite MENR having no prior Page 22 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) experience in implementing large-scale consultancy service operations involving several energy sector stakeholders. (b) The project has almost fully disbursed the allocated IPA grants, which is considered as solid success given that overall EU/IPA grant disbursements made available to Turkey through decentralized system, particularly in the energy sector, still has backlogs in Turkey. (c) Considering the broader-level stakeholder engagement, at times the implementation efficiency was uneven among various stakeholders mainly due to high staff turnover in some stakeholders. However, institutions interviewed mostly expressed their satisfaction on the implementation quality. 55. There were some issues in the use of resources, however, these were not related to the project’s efficiency. They were rather related to the nature of implementing RETF instruments for a TA operation of this scale. The TA included many consultants and a large number of technical outputs that required extensive support from the Bank’s task team. (a) The World Bank had to allocate considerable staff time in this operation in both the preparation and implementation period although the project amount was of smaller scale compared to other energy sector investment projects in Turkey. In total, the preparation and supervision costs reached US$ 900,000 between FY12-FY17. (b) To ensure the highest quality, there were many iterations in finalizing the tendering documents and technical reports while bringing considerable workload on the task team at various levels. In addition, task team had to comment at all stages of procurement, work on the ToRs, and review deliverables. The World Bank liaised with the MENR, key participating institutions, European Union Delegation in Turkey (EUD), European Bank for Reconstruction and Development, and consultant firms on implementation and progress-related issues. This yielded an overloaded supervision system resulting in implementation delays. As a result, actual disbursements were skewed toward the end of the project closing and remained below 50 percent up to six months before the closing. Assessment of Efficiency Rating 56. The project has been the main instrument for the World Bank’s energy sector policy intervention, with an aim to deepen reforms in the power and gas markets, support energy-efficient market development, and promote renewable energy policy. Although the overall preparation and implementation has been intensive, the project has created good ownership at the beneficiary level and resulted in progress in the mentioned areas. In line with the above explanations, the efficiency of the project was; (a) Substantial in terms of financial return; (b) Substantial on the use of resources for the client side because of satisfactory implementation and remarkable capacity built; and Page 23 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) (c) Substantial on the use of resources for the World Bank because of the factors described in paragraph 53. 57. Considering the achieved results, built capacity, and completion of activities in the envisaged time frame and administrative costs, the ‘Opportunity Cost’ of this operation is assessed to be paid off. The efficiency for the large-scale TA operations through RETF instrument has a certain trade-off on the World Bank side due to the intensive efforts to design and supervise the operation. The World Bank had to provide considerable resources for an operation under a three-year implementation. The ‘Opportunity Cost’ for World Bank staff time is high, but for this project case, it is assessed that the Opportunity cost is fully paid off. 58. Therefore, the efficiency rating takes this reality into account and assesses this project’s efficiency within its own characteristics. Overall, the project efficiency, is assessed as Substantial. D. JUSTIFICATION OF OVERALL OUTCOME RATING 59. The project had no shortcomings in its relevance and only had minor shortcomings in its efficacy mostly related to the slower than anticipated process in some EU Acquis-related policies. The efficiency was solid at the project level and on the client side, yet, required considerable resources from the World Bank side. The project’s relevance is High, efficacy is Substantial, and efficiency is Substantial. Thus, the overall outcome rating is Satisfactory. Table 10. Overall Outcome Ratings Relevance of PDO High Efficacy Substantial Efficicency Substantial Overall Outcome Rating Satisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) 60. Gender. At the appraisal stage, the adoption of a gender lens was not assessed to add value and therefore no gender-sensitive analysis or monitoring and evaluation (M&E) was planned. However, attempts to measure the impact of the TA and subsequent projects under the energy efficiency component of the project were highlighted as an area that a gender lens would bring valuable insights. 61. At the implementation stage, a survey was conducted as part of Component 4 to test the global evidences that energy use, energy costs, and energy efficiency potentials of SMEs change according to whether there are women among the partners or directors of firms. In line with this hypothesis, among the 260 audits carried out within the scope of the project, a small questionnaire on whether there are women among the partners or directors of SMEs was undertaken. The survey indicated there were female partners or directors in 67 of the enterprises in which those audits were carried out but there were no female partners or directors in 95 enterprises. Information about the remaining 98 enterprises could not be obtained. Of the 13 firms that have implemented the proposed EE measures among the interviewed firms, 6 enterprises have female partners or directors. Therefore, the ratio of firms with female partners or directors (6/67) have higher ratio of implementing the EE measures compared to firms without female partners and directors (7/95). Page 24 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 62. Institutional strengthening. One of the strongest results of the project is the institutional capacity built at the MENR GDFREU. At appraisal, the department was established with limited capacity. Despite the challenges faced throughout the implementation process, the MENR GDFREU built remarkable capacity, including the management of such a large-scale interinstitutional consultancy operation with solving technical, administrative, financial, and procurement-related issues. The capacity built at the MENR PIU has strong potential to assume the management of operations of a similar scale, as proven in Phase II of the EU/IPA TA, as well as other IPA 2018–2020 projects. 63. Another major institutional strengthening was seen in the interinstitutional coordination. The design of the operation enabled various energy sector public institutions to sit together in the Steering Committee (at strategical level) as well as in the Evaluation Committee (at operational level) for assessment of tendering documents and finalization of the firm selection. In the contract management stage, the DGRE, General Directorate of Energy Affairs (GDEA), BOTAŞ, TEIAŞ, and EMRA have provided their comments for the finalization of outputs and participated in working groups established by the PIU to discuss key project deliverables. Therefore, the strong coordination built among the different public- sector stakeholders of the energy sector at both strategical and operational scale is assessed as an important long-term gain for Turkey. 64. Mobilizing private sector financing. The project made a solid contribution to the mobilization of private sector financing in renewable energy and energy efficiency investments. A total of 155 small-scale RE investments’ bankability and technical feasibility assessments were made. By the project closing date in end-September 2017, 69 out of the 155 feasibility studies returned into real investment either through credits or equity. 65. A similar contribution can also be seen in EE investments in SMEs and buildings, which is the other stream for mobilization of private sector financing in the project design. Through Component 4, the energy audit studies were carried out following the capacity development actions. As explained in detail in Section II above, by project closing date, a total of 37 projects were completed and 42 of them were planned for implementation. 66. In addition to the directly attributable outcomes, the broader range of impacts of the ‘techno- economic and policy assessment and strategy proposal for promotion and grid integration of renewable energy sources’ provided additional leverage to the RE investments , particularly for the small-scale investors. The World Bank Private Sector Renewable Energy and Energy Efficiency lending operation to Turkey Industrial Development Bank (TSKB) and Turkey Development Bank (TKB) also helped mobilize financing to private sector financing. However, this project targeted the least developed and highest potential segment of private sector, small-scale investors through business development services. 67. Poverty reduction and shared prosperity. The project did not have any direct impact on poverty reduction or shared prosperity as the activities were only consultancy services. However, increased energy sector alignment and cost savings through improved energy efficiency obviously have overall impact on shared prosperity in Turkey. Page 25 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 68. Project design. The TA, as described in the PAD, was the ‘first installment’ to utilize EU IPA funds directly by the Government institutions under the World Bank’s administration. Therefore, an RETF model, was planned. The design was complicated due to the newly tested modality. The RETF instrument selection for such a large scale and comprehensive TA was highly ambitious. The project implementation arrangements aimed to address several issues at the same time in Turkey’s portfolio, namely (a) alignment of the EU and the World Bank business; (b) flow of EU funds into the Government institution through World Bank management; (c) mobilizing of massive funds for the MENR’s self-utilization under consultancy services; and (d) inclusion of a wide range of public sector stakeholders as key beneficiaries. Particularly, the key decision which made the project design complicated, related to aim (b) and (c), was the selection of the RETF instrument despite no strict EU obligation requiring the direct use of funds by the MENR. However, the World Bank decided to proceed with RETF in the anticipation this would allow the Turkish government to ‘own’ the outputs of the project. Additionally, it was considered that RETF modality would allow the MENR to approve the outputs of TA as their own programs and plans. 69. The tripartite legal framework to ensure the flow of funds. The EU and Government of Turkey signed the Financing Agreement, dated March 27, 2013. Second, an Administrative Agreement between the EU and the World Bank was signed on November 6, 2013, to enable the World Bank’s administration of the funds on behalf of the EU. However, the last legal piece, the Grant Agreement (GA) between the Government of Turkey and World Bank could not be signed on time due to a loophole identified late in Turkish legislation.9 The MENR did not have an authorization to sign a GA with the World Bank. The Undersecretariat of Treasury informed the World Bank that EU grants are excluded from their authority.10 This limbo cost almost 15 months’ time in project approval until the MENR lawyers discovered a dated law (Law 244) in Turkey allowing Cabinet to authorize a line ministry to sign international agreements on behalf of the Government of Turkey. Afterwards, the MENR was authorized to sign the GA for the EU/IPA TA through a Cabinet Resolution. A second Cabinet Resolution was required to declare the effectiveness of the project. Following the approval of two Cabinet Resolutions, the project became effective on May 30, 2014. 70. RF. The RF design was challenging. The design-related issues are mentioned in detail in the efficacy section. Clearly, the team preferred to set objectives at the right level of ambition for an operation of this scale on energy policy being implemented for the first time. Therefore, the key expected outcomes faced a certain trade-off between staying on the ‘attribution-realism’ or ‘specific-measurable’ side. The task team’s cautious approach in setting the objectives therefore helped avoid ‘unrealistic’ objectives at the design stage, but this compromised the quality of the RF to measure the actual impact of the operation. 9 The Turkish Debt Law mandates Turkish Treasury to sign all foreign borrowing and grant receives on behalf of the central government budget institutions, including the MENR. 10 The Turkish Debt Law excluded EU grants from the Treasury’s regular grant receival authority in line with the EU’s Indirect Management Modality. Therefore, all EU grants—under the decentralized modality—had to be processed through the EU- accredited CFCU. Page 26 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 71. Plan for M&E. It was agreed that the monitoring of the results would be conducted jointly by the World Bank, MEU, and EUD on a quarterly basis. A Steering Committee, including the MENR, MEU, Treasury, EUD, and the World Bank, was established to coordinate the project and to facilitate stakeholder participation. 72. Stakeholder selection. The selection of stakeholders was accurate by placing the MENR GDFREU as the coordinating and executing body of the project. The GDEA and GDRE were informed and involved during the design stage. The energy regulator EMRA, supply and transmission system operator BOTAŞ, and transmission system and electricity market operator TEIAŞ also participated acquired in the design. However, the Ministry of Development (MoD) should have been involved in the Steering Committee. 73. Risk identification. Overall risk identification was accurate but some critical risks were underestimated. The PAD accurately identified the client capacity related risk ‘Substantial’ as the MENR. However, the design risk, which was rated as Moderate at concept and reduced to Low at appraisal, materialized even before GA signing and continued throughout the implementation. The implementation faced considerable delay due to the required time for the learning curve of the MENR to work with the World Bank operational policies. Moreover, the institutional cooperation among the MENR and other stakeholders had not been rated as a substantial-risk item, although institutional coordination—even under the same ministry—is always a substantial risk in the Turkey portfolio. 74. Mitigation measures. Despite the perceived high-capacity challenges, the mitigation measures were insufficient at the design stage. The Project Concept Note accurately pointed out the shortage of staff at the MENR GDFREU. The risk mitigation measures were mainly relying on the Government’s expected actions to strengthen the capacity. Those actions were ultimately not under the World Bank’s control and turned out to be an unmitigated risk during the implementation. 75. Readiness for implementation. The overall commitment of key energy sector stakeholders, appetite of the MENR PIU, and the World Bank’s highly committed full-fledged team were the strong aspects of the readiness. Furthermore, the proactive action of the MENR PIU to process requests for expression of interest (REOIs) and start drafting of ToRs improved the readiness to a certain extent. B. KEY FACTORS DURING IMPLEMENTATION 76. The implementation period of the EU/IPA Phase1 operation coincided with a period when Turkey faced several political and security challenges. In broader terms, the implementation progress was affected significantly by factors outside the World Bank’s and MENR’s control. For the sake of clarifying the exogenous factors affecting the implementation the ICR will first discuss the factors outside the control of the Government or implementing agencies. Factors Outside the Control of Government and/or Implementing Entities 77. Since early 2014, Turkey experienced a series of political challenges. This included a long election cycle, starting with local elections in March 2014, Presidential Elections in August 2014, and Parliamentary elections in both June and November 2015 and continued with Constitutional Referendum in April 2017. Between 2014 and 2017, five different governments11 were in charge. Then, Turkey experienced several 11 Sixty-First Government (July 2011–August 2014), 62nd Government (August 2014–August 2015), 63rd Government (August Page 27 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) terrorist attacks and a failed coup attempt between 2015 and 2017. A countrywide state of emergency was declared. 78. During this period, three different Ministers for Energy and Natural Resources were appointed, which also resulted in a change in high- and mid-level positions in the MENR and other energy sector public institutions. The Undersecretary of the MENR, the General Directors of TEIAŞ and BOTAŞ, Board Members of EMRA, and a reasonable number of mid-level management of MENR and other energy sector public stakeholders experienced reshuffling through this period. Security measures following the coup attempt included dismissal of number of public servants from the institutions. Despite all the changes at the most senior levels, the project was still implemented well. 79. Furthermore, one of the internationally contracted advisory firms expressed some concerns to visit Turkey due to security concerns for a certain period. This also had a negative impact on the implementation speed of the operation. Factors Subject to Government and/or Implementing Entities Control 80. Coordination among the key stakeholders was satisfactory. All project implementation responsibilities as well as intra-government coordination were assigned to the PIU formed under the GDFREU and the head of the concerned department was appointed as the PIU Head. The coordination role included drafting of the ToRs, selection of consultants, and managing contracted firms during the preparation of the reports. At all these stages, the GDFREU kept all relevant parties included and connected to ensure smooth implementation. All interviewed key stakeholders expressed their satisfaction with the coordination conducted by the GFDA. 81. A more clear-cut responsibility sharing among the stakeholder institutions would have been beneficial. The GDFREU was identified as the executing authority, that is, all approvals, transactions, and disbursements are under the GDFREU’s jurisdiction. The other stakeholders were not legally responsible for the output approval. The interviewed public officials in those institutions expressed that they always limited their role to providing comments and perhaps to the extent of providing ‘no objection’, but not ‘approval’. However, the GDFREU set a strong precedent of not approving any report without receiving the final comments from the related stakeholders. Yet, an officially approved, clear-cut matrix of responsibilities defining who clears, concurs, advises, and approves was not in place. 82. The GDFREU demonstrated remarkable ownership and leadership throughout the implementation period. The dedicated ownership and leadership of the PIU, particularly the Head of PIU who also did not avoid taking proactive steps for the sake of implementation, enabled the project to be implemented, and the project rebounded in the second half of implementation. These proactive actions included preparation of REOIs before signing of the GA to save time for implementation. Also, continuous follow-up with the stakeholder public institutions and contracted firms enabled timely completion of the project. 2015–November 2015), 64th Government (November 2015–May 2016), 65th Government (May 2016–present). Page 28 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 83. The Achilles heel of the implementation was mainly the human resources capacity at the stakeholders. The project suffered considerably because of high PIU staff turnover in the first half of the implementation. At the design stage, due to the lack of previous experience of the MENR in working with consultancy operations under World Bank funding, it was planned that experienced staff from TEIAŞ, BOTAŞ, and TEDAS were to be placed at the PIU. However, this did not materialize. Consequently, the PIU staff found themselves with a huge technical and operational workload to implement the project. Consequently, this returned as a huge operational burden on the task team. 84. During implementation, the PIU adopted a responsible approach on fiduciary issues. However, the FM ratings varied between Satisfactory and Moderately Unsatisfactory due to issues during implementation about compliance with the reporting requirements of the GA. These were mainly due to high staff turnover in the PIU and lack of qualified FM staff for long periods, creating temporary but additional workload for PIU staff covering the FM tasks. Despite these deficiencies, the MENR managed to obtain clean audit reports for the project from the Treasury controllers. The MENR has a stable FM staffing level since 2017 and the current FM arrangements are satisfactory. 85. Over the life of the project, the procurement activities were handled in compliance with the requirements of the GA, Procurement Plan, and procurement/consultant guidelines applicable for the project. In line with the original implementation setup, procurements were carried out by the MENR PIU established under the GDFREU. The MENR PIU collaborated with public stakeholders in procurement, and contract management stages. Procurement staffing capacity was established by the MENR PIU through on-the-job training and intensive support was provided by the World Bank at all stages of procurement processes with a short duration of external support financed from the MENR’s Global Environment Facility grant. Capacity constraints and unfamiliarity with the World Bank procurement procedures combined with the duration required for the learning curve resulted in delays in the signing of the consulting services contracts; however, the contracts were completed by the implementation deadline of the grant. 86. The MENR PIU managed the contracts with extensive technical support from the World Bank team. Design of the project limiting the number of contracts to be financed, with the World Bank’s support going beyond the regular fiduciary functions, helped reduce the transaction burden on the MENR. Contracts signed with consultants had several amendments for time extensions. These were mostly due to delays not directly connected with the MENR PIU but with other departments or agencies involved with the implementation of the contracts. 87. Social and environmental safeguards. Since this is a TA project and does not finance any investments which may have adverse impacts on the environment, a stand-alone safeguards document was not prepared. However, in view of the fact that under the RE and EE components of the project, the TA covers consultancy services for preparation of RE and EE projects for SMEs and buildings, it was required that the ToR of consultants refers to the World Bank’s safeguard requirements. The ToRs were prepared accordingly before hiring of the consultants. Factors Subject to World Bank Control 88. The IPA Phase 1 project enjoyed extensive implementation support and supervision, probably beyond expectations. Since the signing, the World Bank deployed a full-fledged task team with strong technical and operational capability. The World Bank hired an extended-term consultant in the Ankara Page 29 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) office to follow up on the implementation daily. The team included a seasoned TTL based in the headquarters and later a locally based TTL with continuous supervision. 89. The World Bank’s reporting was robust throughout the implementation. Seven ISRs were reported; all of them were on time. Key issues have been reported on time to World Bank management and MENR top management through the Aide Memoires. The World Bank also reported the progress to the EU as set out in the AA. In line with the AA requirements, the World Bank submitted two progress reports to the EU. 90. The MENR PIU capacity was not built overnight. Therefore, the World Bank task team had to deliver frequent, daily, and in-depth implementation support to build the client’s capacity. Although the project has been designed as recipient-executed, while all contracts were prior review, the World Bank’s procurement and technical staff had to spend considerable time to review the PIU-prepared documents before the ‘no objection’. The World Bank’s Energy Global Practice team provided technical comments to ensure improving the quality of the draft reports by going the ‘extra mile’. The interviewed stakeholders, particularly TEIAŞ, expressed their appreciation of the technical value added provide by the World Bank task team throughout the implementation process. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) 91. M&E design. The project theory of change was solid because of clear identification of required activities and components to reach the intended project objectives. The selected outcome indicators, however, were mostly qualitative. Thus, most indicators at the PDO level were difficult to measure and not specific to prove the progress toward the PDO. For this reason, despite the strong theory of change, the M&E design was mostly weak, particularly for the MENR Sector Alignment and Energy Efficiency outcomes. The intermediate result indicators mostly focused on delivery of the reports regardless of the quality. Consultant engagement has been tagged as an intermediate result indicator although it should be assessed as an input of the project. In some cases, quantitative and qualitative targets were put under one results indicator, which made it hard to justify whether it has been achieved or not. Thus, M&E design was weak. 92. M&E implementation. The project established a Steering Committee including the MENR, EU, World Bank, other energy sector public stakeholders, MEU, and Treasury to monitor the progress on a quarterly basis. The Committee’s high-level dedication on satisfactory implementation of the project helped achieve the results. The M&E data was collected on time by the PIU and reported to the World Bank, EU, and the Steering Committee on time. However, due to M&E design weaknesses, the collected data were limited to the RF requirements. 93. M&E utilization. As most of the M&E data were either input based or qualitative, the impact of the M&E data was negligible in decision making. The task team’s intensive implementation support, the PIU’s strong dedication and ownership, and the Steering Committee’s persistence in following up helped the project deliver almost all envisaged outputs by meeting most of the project outcome targets. However, it is hard to claim that M&E utilization had reasonable impact on the success. Page 30 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Justification of Overall Rating of Quality of M&E 94. For a first-time-implemented, large-scale consultancy operation as this, it is challenging to design a robust M&E. However, the M&E of the operation could have been designed better. There were shortcomings in the design of the M&E system, making it difficult to assess the achievement of the stated objectives and test the links in the results chain. Therefore, overall M&E rating is assessed as Modest. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 95. Compliance with social and environmental safeguards. Since this is a TA project and does not finance any investments which may have adverse impacts on the environment, a stand-alone safeguards document was not prepared. Therefore, there was no issue relating to ‘compliance’ with environment and social safeguards. All four contracts were subject to prior review. All contracts received World Bank ‘no objection’ and all of them followed fiduciary rules of the World Bank. 96. No procurement complaints were received during the implementation of the project and there was no case of misprocurement for the project. Close follow-up and assistance from the World Bank team have been provided to address procurement issues on time and the project was regularly supervised by the World Bank’s procurement specialist. The consulting services contracts financed under the grant were completed as budgeted, with least cost implications from time extensions. By the time of the grant closing, there was no case of arbitration or dispute resettlement reported for the financed contracts. Procurement compliance has been Moderately Satisfactory throughout the project. C. BANK PERFORMANCE Quality at Entry 97. The World Bank performance at entry is assessed as Moderately Satisfactory, as detailed in the following paragraphs. This operation was ‘boldly’ designed as a pathfinder to initiate the potential cooperation with the EU. Therefore, the World Bank’s performance at entry was already under strain due to several design factors, such as PIU staffing, interinstitutional coordination mechanism, EU/Bank/MENR coordination, legal procedures for signing and effectiveness, technical design of a comprehensive TA at this scale, and assessment of all relevant risks. 98. Main strengths: (a) The World Bank task team had persistent and in-depth communication and coordination with the EU headquarters at Brussels, EU delegation in Ankara, MENR, and other stakeholders including BOTAŞ, TEIAŞ, DGRE, EMRA, and so on. (b) The project strategic relevance with Turkey, EU, and World Bank’s strategies was high. (c) The separate design of components while ensuring contract-component match was logical and pragmatic for ensuring straightforward implementation. (d) The decision on announcing the REOIs and preparing the ToRs before the GA signing to save time was an accurate one. Page 31 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) (e) Gender lens was applied where it was seen fit. (f) The contracts were designed to be subject to prior review, which was appropriate given the inexperienced PIU at the beginning of the implementation. 99. Shortcomings: (a) A minor deficiency in the design was clustering four different tasks under contract 1, which also included visibility. (b) The World Bank’s risk assessment was skewed toward the MENR’s insufficient capacity, but mitigation measures were out of the World Bank’s control. (c) The World Bank should have designed a mechanism inside the BETF component to include some support for the PIU, particularly to hire procurement and FM specialists experienced with World Bank operations. (d) A clear matrix of responsibilities among the key stakeholders could have been prepared at the design stage and annexed to the PAD. (e) Because this was the first large scale RETF TA, the task team maintained a simple and mostly non-measurable RF. Despite the strong achievements made by this operation, the design stage RF did not allow to fully capture the impact of the operation. 100. The design stage was rather long due to the 15-month delay in signing the GA because of the loophole discovered in Turkish domestic legislation for EU-originated grants. However, this is not assessed as a design stage shortcoming for the World Bank’s performance as it would be impossible to foresee. Indeed, the task team’s efforts to overcome the issue via cooperation with MENR lawyers helped resolve the case. The solution then paved the way for the remainder of the EU-funded RETFs, including Funding for Refugees in Turkey (FRiT) operations, which reached over a US$200 million business line in Turkey. Quality of Supervision 101. Quality of supervision has been assessed as Satisfactory, as detailed in the following paragraphs. 102. Main strengths: (a) The World Bank provided highly intensive support throughout implementation. The PIU’s capacity has been supplemented through this support. Although the alignment of tendering documents with the World Bank’s procurement rules was the PIU’s responsibility, the task team had to engage frequently due the fiduciary capacity shortage of the PIU. (b) The World Bank’s Energy Global Practice team provided technical support to refine the ToR content. (c) After the contract signing with the winning bidders, the World Bank’s Energy Global Practice Page 32 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) team stayed on board to provide technical comments on the draft consultant reports. (d) The fiduciary support was even more frequent compared to the technical support. (e) Throughout the implementation, the World Bank’s performance reporting was candid. In total, seven ISRs were filed. DO ratings were Satisfactory for two and Moderately Satisfactory for the remaining five. The rating was Moderately Satisfactory at closing. The number of missions was sufficient. (f) In line with the AA requirements, the World Bank submitted two progress reports. (g) The World Bank’s timely action to extend the closing date, while processing the extension request both for GA and AA, enabled satisfactory completion of all activities with massively increased disbursement during the extended period. 103. Shortcomings. There were two shortcomings in the implementation: (a) the operational inefficiency faced throughout the implementation and (b) no revision made in the RF although it was weak. 104. Although the implementation progress was uneven and a big chunk of disbursements was mostly skewed at the end of the implementation, most of the factors causing the delay were exogenous. The full completion of activities without any fiduciary, safeguard, or any other problems for a first-time- implemented, large-scale RETF operation is considered a notable success. However, not resolving the PIU staffing issues through a potential short-term consultant support to help the PIU team daily on FM and procurement issues is considered a minor shortcoming. Justification of Overall Rating of Bank Performance 105. The World Bank’s overall performance rating is assessed as Moderately Satisfactory given the moderate shortcomings at the preparation stage and minor shortcomings at the supervision stage. However, the World Bank’s overall performance deserves recognition for satisfactorily designing, implementing, and completing such an innovative and complicated operation. D. RISK TO DEVELOPMENT OUTCOME Rating: Negligible to Low 106. The development objectives achieved by the project will most likely be maintained. However, since the project was designed as an umbrella which covers very different aspects of energy field, it would be better to analyze the possible continuity of the outcomes separately. 107. Outcome Area 1 (Promoting Energy Efficiency) consists of intertwined tasks which reinforce each other. The first task targeting the market development, particularly in SMEs, enabled the companies to develop EE projects and to submit them to financial institutions with bankable reports. The second task mainly focused on capacity building for the commercial EE lending market. These achievements under this outcome area mean the following: Page 33 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) (a) SMEs from different manufacturing sectors became aware of the importance of EE, which could result in continual improvement. (b) Local financial institutions gained confidence in and acquired capabilities for EE financing. 108. Even though some legislative measures and robust mechanisms are needed to extend the impacts of the implementation, the mentioned achievements would trigger and inspire new actions. Briefly, it can be assumed that the risks to maintaining the achievements are low. 109. Outcome Area 2 (Development of Renewable Energy Sector) is less varied in comparison with the first one. The promising results obtained through financing the small-scale RE projects guarantees that the risks to maintaining the achievements are low. Similarly, all the produced materials and tools are likely to be used in market operations by different stakeholders, including local banks. 110. Outcome Area 3 (Natural Gas Market Development). As EMRA, the regulatory authority, issued a regulation for the constitution of the gas platform and EPIAS, the energy exchange market operator, announced initiation of market operations by April 2018, the achievement was secured to be maintained. No serious risk is foreseen for the maintenance of the achievement. Further, the IPA Phase II Project will continue to support the Government on further steps for a liberalized market. The unbundling support for BOTAŞ and gas market opening, as discussed earlier, requires strong governmental decisiveness. The project provided a strong basis for initiating the necessary discussions; however, it cannot be guaranteed that those efforts will result in the unbundling of BOTAŞ. 111. Cross-cutting outcome area. One of the strongest and probably the longest-term outcomes of the project can be considered the built capacity at the MENR GDFREU, including the management of such a large-scale interinstitutional consultancy operation with solving technical-, administrative-, financial-, and procurement-related issues. The capacity built has strong potential to assume the management of operations of a similar scale, as proven in Phase II of the EU/IPA TA, as well as other IPA 2018–2020 projects. V. LESSONS AND RECOMMENDATIONS 112. The project proved the high potential of maximizing finance for development (MFD) through large scale technical assistance services in Middle Income Countries. Eventually, the produced tools, conducted audits, drafted feasibilities ended up mobilizing considerable private sector financing for RE and EE. Furthermore, the observed outcomes during the implementation period is just the initial steps of this achievement and further expansion is expected. On the other hand, the regulations developed thanks to the project outputs, helped to level the playing field in energy sector; particularly the natural gas sector. The GTP is expected to further expedite the private sector financing into this business in Turkey. 113. Although the alignment of World Bank and EU procedures took time at the preparation stage, the development impact of this operation proved the high value added of EU/World Bank joint operations, which was followed by several others in the Turkey portfolio. The EU-World Bank cooperation enabled creation of a double anchoring mechanism for the project objectives to align strongly with Turkey’s development priorities. This partnership enabled development of a brand-new business line to accommodate the MENR’s capacity-building demand where loan financing was not the preferred Page 34 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) option. This new business line continued with Phase II of the EU/IPA TA, which is under implementation. However, beyond that, it paved the way for the scaled-up interventions in other sectors, including Urban Development, Social Protection, and Transport through either the EU or IPA of EU FRiT funds. 114. For the EU/World Bank joint operations, the design of implementation arrangements, particularly the procedural and operational applicability, deserves utmost attention. The recipient- executed model has a higher potential to enhance capacity at the client level. However, the design of the instrument should consider Government, EU, and World Bank policies/procedures. In the tripartite legally binding operations, procedural hurdles of any party can result in a domino effect in the preparation and implementation period. Therefore, in the design stage, the procedural risks must be identified in detail, including the mitigation measures. 115. The biggest determinant of success for a TA operation at this scale is the ‘client ownership’, regardless of the instrument type. Among all stakeholders, the biggest credit for the achievement of the objectives and satisfactory completion of the operation belongs to the MENR PIU staff and PIU head because of their strong commitment and endless dedication. Without the complete client ownership, the project could not have been implemented with this level of success. The RETF modality also contributed considerably to enhance the local capacity at the MENR. Probably, if the project were run through BETF, the reports and produced materials might have not utilized due to lack of ownership by the client. Having said that, without the client ownership, an RETF operation would have greater risk of failing. 116. The RF must be designed properly to accurately assess the impact of the RETF TA at this scale. It is a challenging task to design an attributable, measurable, and realistic RF for large-scale TA operations with a list of consultancy services. Setting attributable and measurable outcomes are essential for capturing the real development impact. Therefore, an experienced M&E specialist should be included in the task teams at the design stage to avoid any vagueness in the RF design. The weak RF design caused weak M&E utilization and difficulty in assessing the overall performance of this operation. In addition, during the implementation process, the RF should have been revised to better capture the project impact. 117. A large-scale TA operation of this kind requires strong technical capacity and expertise both on the World Bank and the client side. Designing of ToRs, selection of consultants, and supervision of the consultancy firms require previous experience and deep technical knowledge. The World Bank’s supervision and support helped create strong capacity at the MENR, particularly at the PIU, to prepare high-quality ToRs and conduct a solid selection process. Stakeholders have also expressed that the World Bank’s extended task team’s strong technical knowledge and intensive review of the draft reports helped improve the products. Therefore, the level of success for similar large-scale RETF consultancy operations will depend on the World Bank’s experience in the sector and the country. Thus, the World Bank should be selective and limit its engagement in large-scale RETF TA to the sectors where the World Bank has strong comparative advantage and in-house capacity. 118. A clear-cut coordination and responsibility matrix needs to be in place for multi-stakeholder TAs executed by the recipient. Based on the feedback provided by the partner agencies, a more effective way could have been considered for the proper circulation of the reports and outputs produced by the contractors. It is strongly recommended to develop a matrix and annexed to the PAD at the beginning of the project and identify the position of each agency for each task in detail, which would likely prevent the confusion in roles, responsibilities, and expectations. Page 35 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 119. Consolidation of activities under component-level contracts is required with an inexperienced PIU, but the follow-up operations should bypass this rule by seeking clear-cut contracts. The number of tasks assigned to one single company or consortium under one contract caused problems during the selection and implementation process, especially in Component 1, as this consisted of four different and mostly unrelated (or partially related) tasks. This is because one awarded company having specific expertise in one task could be the wrong option for another task. Therefore, it is highly recommended to divide and separate the tasks if they need different knowledge and expertise from different fields. Additionally, cross-cutting activities need a stand-alone contract. It is strongly recommended to form either an independent contract for awareness-raising activities cross-cutting and serving all the other components or define several subtasks regarding awareness and visibility separately for each component. . . Page 36 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion MENR (Energy Sector Text Turkey's energy sector The MENR approves a The MENR approves a The MENR approved a Alignment) can be enhanced in program for the program for the report on line with the EU enhancement of the enhancement of the programming for the energy priorities and Turkish energy sector Turkish energy sector enhancement of the strategies. in line with the EU in line with the EU Turkish energy sector energy priorities and energy priorities and in line with the EU strategies, based on a strategies, based on a energy priorities and review and regulatory review and regulatory strategies, based on a impact assessment of impact assessment of review and regulatory alignment of Turkey's alignment of Turkey's impact assessment of legal and institutional legal and institutional alignment of Turkey's framework for framework for legal and institutional electricity, natural gas, electricity, naturalgas, framework for energy efficiency and energy efficiency and electricity, gas, energy renewable energy renewable energy efficiency and with the EU's energy with the EU's energy renewable energy acquis. acquis. with the EU's energy acquis. Page 37 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 18-Jul-2012 31-Mar-2017 28-Sep-2017 25-Sep-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Natural Gas Market Text Design and regulations BOTAŞ and EMRA BOTAŞ and EMRA Gas Trading Platform Development not available. approve a draft design approve a draft design (GTP) design have and regulations, and regulations, been prepared, widely respectively, for a respectively, for a discussed and delivery mechanism for the mechanism for the approved by the wholesale trading of wholesale trading of MENR, EMRA and natural gas natural gas. EPIAS. EMRA and EPIAS have commenced work to integrate GTP in EPIAS. Elements of necessary regulations have been drafted. 18-Jul-2012 31-Mar-2017 28-Sep-2017 25-Sep-2017 Comments (achievements against targets): Page 38 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Renewable Energy Text Turkey's energy The MENR approves a The MENR approves a The MENR approved a Development security and climate plan for the plan for the plan for the change strategies acceleration of acceleration of acceleration of (small include acceleration of renewable energy renewable energy scale) renewable renewable energy deveelopment development energy development development. including the including the including proposal for improvement of TEIAŞ improvement of TEIAŞ improvement of TEIAŞ power system power system power system flexibility to integrate flexibility to integrate flexibility to integrate a larger share of a larger share of a larger share of renewable electricity. renewable electricity. renewable electricity. 18-Jul-2012 31-Mar-2017 28-Sep-2017 25-Sep-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Energy Efficiency Promotion Text Turkey's energy Turkish financial Turkish financial To date about 205 security and climate institutions receive institutions receive SME EE audits change strategies and review financing and review financing completed.Turkish include acceleration of applications for applications for financial institutions deployment of energy energy efficiency energy efficiency have started to efficiency measures projects for small and projects for small and receive and review and technologies. medium-scale medium-scale financing applications industries and industries and for energy efficiency Page 39 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) buildings. buildings. projects for small and medium-scale industries and buildings (30 financed already and 83 in the application for financing), exact number of IPA Phase I Project related financing applications will be reported following the Project closing date. 18-Jul-2012 31-Mar-2017 28-Sep-2017 25-Sep-2017 Comments (achievements against targets): A.2 Intermediate Results Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion MENR (Energy Sector Text Regulatory impact Regulatory impact Regulatory impact The initial reports on Alignment) assessment not assessment of assessment of gap analysis in available. alignment of Turkey's alignment of Turkey's electricity, natural gas, legal and instittutional legal and institutional energy efficiency and framework for framework for renewable energy electricity, natural gas, electricity, natural gas, legislation have been Page 40 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) energy efficiency and energy efficiency and prepared and renewable energy renewable energy approved. MENR and with the EU's energy with the EU's energy EU are using these acquis. acquis. documents as a basis for EU accession negotiations. Regulatory impact assessment work has been completed. Further, a training on 'Regulatory impact assessment of alignment of Turkey's legal and institutional framework for electricity, natural gas, energy efficiency and renewable energy with the EU's energy acquis' has been conducted for MENR staff. 18-Jul-2012 31-Dec-2015 31-Mar-2016 25-Sep-2017 Consultant Engagement Yes/No N Y Y Y 18-Jul-2012 31-Dec-2015 31-Mar-2016 25-Sep-2017 Comments (achievements against targets): Page 41 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Natural Gas Market Text Design and regulations Design and draft Design and draft Design and draft Development not available. regulations for a regulations for a regulations for a mechanism for the mechanism for the mechanism for the wholesale trading of wholesale trading of wholesale trading of natural gas. natural gas. natural gas has been achieved. Further activities include, (i) Assessment and Recommendations Report for the Turkish Natural Gas Trading Hub; (ii) Assessment and Recommendations Report for the Limitations of BOTAŞ for Unbundling; (iii) Assessment, Recommendations and Roadmap Report for BOTAŞ. 18-Jul-2012 31-Dec-2015 31-Mar-2016 25-Sep-2017 Consultant Engagement. Yes/No N Y Y Y 18-Jul-2012 31-Dec-2015 31-Mar-2016 25-Sep-2017 Page 42 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Renewable Energy Text Review, road map and A review of current A review of current A review of current Development plan not available. situation, barriers and situation, barriers and situation, barriers and road map for road map for road map for renewable energy, renewable energy, renewable energy, and a plan renewable and a plan renewable and a plan renewable energy grid energy grid energy grid integration to integration to integration to distribution and TEIAŞ distribution and TEIAŞ distribution and TEIAŞ transmission systems. transmission systems. transmission systems Formulation of at least Formulation of at least have been completed. five renewable energy five renewable energy A total number of 133 projects for small and projects for small and renewable energy medium-sized medium-sized prefeasibility and industries and industries and bankable reports for buildings. buildings. small and medium- sized industries and buildings have been completed as of ISR date. 18-Jul-2012 31-Dec-2015 28-Sep-2017 25-Sep-2017 Consultant Engagement Yes/No N Y Y Y Page 43 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 18-Jul-2012 31-Dec-2015 28-Sep-2017 25-Sep-2017 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Energy Efficiency Promotion Text Insufficent capacity, Formulation of at least Formulation of at least As of ISR date the shortage of energy twenty-five energy twenty-five energy Consultant completed efficiency projects. efficiency projects for efficiency projects for about 205 SME EE small and medium- small and medium- audits and bankability sized industries and sized industries and reports. Additionally, buildings. buildings. several awareness raising activities and trainings completed with regards to ESCO market development and EE project preparation. A report on EE banking products was finalized and approved. An EE policy review is being finalized. 18-Jul-2012 31-Dec-2016 28-Sep-2017 25-Sep-2017 Consultant Engagement Yes/No N Y Y Y Page 44 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 18-Jul-2012 31-Dec-2016 28-Sep-2017 25-Sep-2017 Comments (achievements against targets): Page 45 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) B. KEY OUTPUTS BY COMPONENT Given the project outcome/component/output design is very complex, the standard ICR Annex 1.B did not fit for the purpose. Please see attached the detailed outcome/component/output list below: Component Contract Reports 4 CS04 EE Policy Support Report 4 CS04 EE Market Assessment Report 4 CS04 Benchmarking Functions Report 4 CS04 ENEVEL ESCO Model Report 4 CS04 ESCO Models in EU Report 4 CS04 Current Turkish ESCO Market Report 4 CS04 Mitigated Financial Risk ESCO Model (an EPC Model) Report 4 CS04 Power Purchase Agreement (PPA) Model Reports Outcome Area 1: Enhancement 4 CS04 Roadmap for the Development of the ESCO Market in Turkey Report of the Turkish energy sector in 4 CS04 Energy Audit Report Template line with the EU energy priorities and strategies in 4 CS04 Feasibility and Bankable Report Templates energy efficiency 4 CS04 Energy Efficiency Toolbox 4 CS04 Guidelines for Banks and ESCOs 4 CS04 New Banking Products and Financial Modalities Report Trainings 5 CS01 Regional Forum: Ankara Sanayi Odası (EE) 5 CS01 Regional Forum: Eskişehir Sanayi Odası (EE) 5 CS01 Regional Forum: Adana Sanayi Odası (EE) 5 CS01 Regional Forum: Ege Bölgesi Sanayi Odası (EE) 5 CS01 Regional Forum: İstanbul Sanayi Odası (EE) 5 CS01 Regional Forum: Kocaeli Sanayi Odası (EE) Page 46 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 5 CS01 Regional Forum: Konya Sanayi Odası (EE) 5 CS01 Regional Forum: Gaziantep Sanayi Odası (EE) 5 CS01 Regional Forum: Denizli Sanayi Odası (EE) 5 CS01 Regional Forum: Antalya-Kemer Turistik Otelciler ve İşletmeler Birliği (EE) 5 CS01 Regional Forum: Türkiye Süt, Et, Gıda San.ve Üreticileri Birliği (EE) 5 CS01 Regional Forum: Taşıt Araçları Yan Sanayicileri Derneği (TAYSAD) (EE) 5 CS01 Regional Forum: OSTİM OSB (EE) 5 CS01 Regional Forum: İkitelli OSB (EE) 5 CS01 Regional Forum: Mersin Tarsus OSB (EE) 5 CS01 Regional Forum: İzmir Atatürk OSB (EE) 5 CS01 Regional Forum: Veliköy OSB (EE) 4 CS04 Training of Trainers 4 CS04 Training on Energy Management (for ESCOs) 5 CS01 Training on EU Acquis EE 5 CS01 Workshop for the EU’s Energy Objectives and Strategies on EE 4 CS04 Training on Investment Appraisal (for ESCOs) 4 CS04 Training on Energy Auditing (for ESCOs) 4 CS04 Training on Process Technologies (for ESCOs) 4 CS04 Training on ESCO Project Development and Contracting 4 CS04 Training on Energy Efficiency Financing (for Commercial Banks) Site Visits 5 CS01 Press Trip to Belgium and Netherlands Component Contract Reports Outcome Area 2: Enhancement 3 CS02 Assessment of Current Situation and Barriers Report of the Turkish energy sector in line with the EU energy Small Scale RE Financing and Support Mechanisms Assessment and 3 CS02 Recommendations Report Page 47 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) priorities and strategies in 3 CS02 Recommendations Final Report (Guidelines for RE Investors) renewable energy 3 CS02 RE Project Selection Criteria and Implementation Procedure Report 3 CS02 RE Project Survey Results Reports Monitoring, Forecasting and Control System for Solar PPs Feasibility and Technical 3 CS02 Specifications Report 3 CS02 Design and Specifications of the Calculation Tool Report 3 CS02 RE Roadmap - Strategy and Action Plan 3 CS02 RE Projects Project Appraisal Reports (Feasibility and Bankability) 3 CS02 RE Environmental and Social Considerations Recommendations Report Trainings 3 CS02 Training on Offshore Wind Market 3 CS02 Training on Energy Storage 3 CS02 Training on Sustainability of Renewable Energy Policy 3 CS02 Training on Offshore Wind 3 CS02 Training on RES Integration 3 CS02 Training on RE Heating & Cooling 5 CS01 Training on EU Acquis RE 5 CS01 Workshop for EU’s Energy Objectives and Strategies on RE 3 CS02 Training on Global Solar Energy Practices 3 CS02 Training for Banks on RE Technologies 3 CS02 Workshops for RE Assessment and Recommendations 3 CS02 Workshop for RE Action Plan Workshops for Small Scale RE Financing and Support Mechanism Assessment, 3 CS02 Recommendations and Action Plan 3 CS02 Workshop for Monitoring Solar Plants Site Visits Page 48 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) 5 CS01 Press Trip to Denmark (RE) 3 CS02 Field Visit to Italy and Germany (for storage, on- and off-shore wind) Component Contract Reports 2 CS03 Functional and Technical Overview of EBB Report 2 CS03 Functional Requirements of the New EBB Report 2 CS03 Functional Design and Specifications for New EBB Report 2 CS03 Survey and Evaluation Report 2 CS03 Market Model Analysis Reports (1-2-3) 1 CS01 Natural Gas Market Assessment Report 1 CS01 Natural Gas Market Legislative and Implementation Report Outcome Area 3: Enhancement 2 CS03 Field Visits Assessment Report of the Turkish energy sector in 2 CS03 Final Survey, Analysis and Evaluation (European Markets) Report line with the EU energy 2 CS03 Technical and Functional Design of GTP Report priorities and strategies in the 2 CS03 Final Design Specifications Report natural gas market Assessment and Recommendations Reports for the Turkish Natural Gas Trading Hub and Limitations of BOTAS Stemming from the Decree Law on State Economic 2 CS03 Enterprises No.233 Trainings 5 CS01 Training on Policy of Natural Gas Market 5 CS01 Training on EU Acquis Natural Gas Markets 5 CS01 Workshop for EU’s Energy Objectives and Strategies on Natural Gas Site Visits 2 CS03 Field Visit to Italy, Germany, Austria, Belgium, UK Cross Cutting Outcome Area: Component Contract Reports Enhancement of the Turkish 1 CS01 Institutional Review of MENR Report energy sector in line with the 1 CS01 Energy Planning Report Page 49 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) EU energy priorities and 1 CS01 Energy Statistics Report strategies 1 CS01 Action Plan for Energy Planning and Energy Information System 1 CS01 Acquis Alignment Screening Reports (Electricity, N. Gas, RE, EE, Energy Statistics) 1 CS01 Regulatory Impact Assessment Report 5 CS01 Awareness Action Plan Report Trainings 5 CS01 Training on Energy Statistics 5 CS01 Training on EU Practices on Energy Statistics 5 CS01 Energy Planning Seminars 5 CS01 Training on Policy of Electricity Market 5 CS01 Training on EU Acquis Electricity Markets 5 CS01 Training on EU Acquis Energy Statistics 5 CS01 Workshop for Regulatory Impact Assessment 5 CS01 Workshop for EU’s Energy Objectives and Strategies on Electricity 5 CS01 Soft Skills Training Site Visits 5 CS01 Site Visit to France (for energy planning and statistics) Page 50 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Ayse Yasemin Orucu Task Team Leader(s) Salih Bugra Erdurmus Procurement Specialist(s) Zeynep Lalik Financial Management Specialist Frances M. Allen Team Member Jasneet Singh Team Member Lisa Lui Counsel Fatma Yesim Akcollu Oguz Team Member Salih Kemal Kalyoncu Team Member David Reinstein Team Member Pierre Audinet Team Member Almudena Mateos Merino Team Member Arzu Uraz Yavas Social Safeguards Specialist Esra Arikan Environmental Safeguards Specialist Debabrata Chattopadhyay Team Member Selcuk Ruscuklu Team Member Koji Nishida Team Member Page 51 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY12 0 856.45 FY13 28.410 244,310.75 FY14 5.100 28,382.84 Total 33.51 273,550.04 Supervision/ICR FY13 0 0.00 FY14 .225 2,079.45 FY15 36.215 143,694.20 FY16 42.527 224,824.62 FY17 6.714 74,244.57 FY18 14.780 164,848.75 Total 100.46 609,691.59 Page 52 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 3. PROJECT COST BY COMPONENT Amount at Actual at Project Percentage of Components Approval (EUR, Closing (EUR, Approval (%) millions) millions) (1) MENR - (a) institutional review and 0.925 0.925 100 capacity building of MENR and (b) review of alignment of Turkey's legal and institutional fr amework with the EU's energy acquis (2) Natural Gas - (a) support for the 2.500 2.500 100 unbundling of the national gas company BOTAŞ and (b) design and regulations for a gas trading platform and a plan for effective gas market opening (3) Renewable Energy - (a) review of 2.200 2.200 100 current situation, barriers and road map, (b) renewable energy grid integration and (c) business development services to small and medium-sized industries (4) Energy Efficiency - (a) market 4.500 4.500 100 development including business development services to small and medium-sized industries and (b) capacity building for commercial energy efficiency lending market (5) Visibility and Public Awareness 0.500 0.500 100 Trust fund administration, program 1.150 1.150 100 management, and supervision Total 11.775 11.775 100 Page 53 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 4. EFFICIENCY ANALYSIS Not applicable. Page 54 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS BORROWER’S ICR 1) Project Success Phase-I Project: IPA 2012 Energy Sector Programme Phase-1 Project was the first IPA Project directly implemented by MENR (and not by one of its institutions) and also the first one jointly managed with the World Bank. The objective of the project was the “Enhancement of the Turkish Energy Sector in line with the EU Energy Priorities and Strategies in energy efficiency, renewable energy, and the natural gas and electricity markets”. In order to achieve this objective, the project was split into 4 parts and implemented through 4 consultancy service contracts covering all subsectors of energy. In terms of Phase-1 Project’s relevance to Turkey’s strategies, a clear link can be established between national documents and the project components, However, although project outputs contributed to (i) awareness in acquis and (ii) an increase in the technical expertise in natural gas, energy efficiency and renewable energy sectors and (iii) encouraging investments in renewable energy and energy efficiency sectors, it is not very easy to measure how much these achievements will translate into development of the Turkish energy sector in line with EU priorities. Regarding efficiency, implementation of 4 large contracts was at times challenging. A higher number of contracts with more specific focuses and shorter implementation periods could have been easier to efficiently manage and it would also be more successful in terms of keeping stakeholder interest alive. However, this approach would probably lead to a higher level of administrative burden (as is the case in the Phase-2 Project). In terms of effectiveness, clear links were established between the project objectives, results and OVIs. These were taken as given and further detailed in the contracts. Although most of the results were attained it should be noted that some results referred to legislative changes or changes in implementation arrangements and the attainment of these results remains to be seen. In terms of sustainability, studies conducted offered a wide range of recommendations such as specifications for the gas trading platform (GTP), ways to improve demand planning and energy data centre and measures to further encourage development of the renewable energy and energy efficiency sectors. Studies towards trial operation of GTP and improvement data collection and energy demand and supply modelling will be realized under Phase-II Project. Sustainability in the renewable energy and energy efficiency assistance was monitored by the consultants who reported on project implementation and loan applications of the beneficiaries who received assistance in project preparation and documentation for their prospective investments. Page 55 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Component 1: Acquis Alignment & Institutional Capacity of MENR, Unbundling Support for BOTAS and Visibility & Public Awareness Relevance of the Project Objectives to Turkey’s Energy Sector Needs and Strategies: The objectives of the project are related to increasing MENR’s capacity to lead and monitor the energy reform process with particular focus on energy planning and statistics and natural gas market liberalization and increasing awareness on energy efficiency and the EU policies. The objectives regarding MENR’s capacity are in close alignment with Turkey’s objectives to establish a competitive market and investment friendly environment articulated in the 9th and 10th Development Plans, MENR’s 2015-2019 Strategic Plan Electricity Market & Security of Supply Strategy Efficiency: The contract was composed of 4 separate tasks. In order to ensure the outputs were reviewed by the relevant personnel, separate lists of personnel who will review the outputs were agreed upon in the inception phase. The sequential reporting within a task was successful in terms of ensuring that each report was thoroughly reviewed to make sure they catered to MENR’s needs. However, it should be noted that in some cases where the approval process took longer than expected, sequential submission of reports led to delays in the submission and approval of consecutive reports. In total, 9 reports under 4 tasks were submitted and approved. As this was mainly a capacity building project, reports were designed to go hand in hand with trainings. These trainings were attended by a total of around 1100 people (calculated as the total of man*days for each event). In the project fiche and the terms of reference, it was aimed to train only 50 people under Task 1 and another 150 people under Task 2. In terms of budgetary disbursement, the expenses regarding both trainings and public awareness activities were designed to be reimbursed through pro-rata payments. Around 92% of the budget allocated to these expenses was disbursed. Hence, it can be said that the budget allocated to these activities was sufficient. Efficacy: Objectives of the project were designed to be met through delivery of reports, organization of trainings, stakeholder meetings and public awareness activities. Although it is hard to measure capacity building, which was designed to be attained through reports and trainings, the wide attendance to meetings and trainings indicate interest in the project activities. Furthermore, analysis of the training participation surveys also demonstrated a high level of satisfaction. Two main achievements of this component in terms of capacity building that were not specified in the Terms of Reference include hands-on-support to MENR during the preparation of acquis comparison tables requested by the European Commission during its studies of the Energy Chapter and preparation of an Impact Assessment Guideline and provision of a training in order to train the MENR staff on how to conduct an impact assessment which is required by the Regulation on Legislation Preparation Procedures and Principles. Page 56 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) With regard to natural gas market activities, although the activities conducted were successful in terms of raising awareness of the BOTAŞ staff about liberalization and unbundling, further steps to achieve liberalization and unbundling are yet to be taken. Finally, with regard to public awareness activities outputs of the task included informative videos, press trips, promotional materials and a children’s booklet. Although these outputs were relevant to the projects in hand, lack of a holistic approach to awareness and the inclusion of awareness activities under a technical contract resulted in fragmented planning and as a result it is hard to measure if the activities conducted have proven to be successful in terms of raising public awareness. Finally, especially with regard to the first two tasks, many of MENR’s internal units and institutions were identified as stakeholders. As none of these stakeholders were the main stakeholder, it was hard at times to coordinate and attract the interest of all stakeholders. Additionally, lack of a definite set of stakeholders in the awareness task led to an inefficient level of input to the design and implementation of this task. Also as this was not a technical task, at times, it was not perceived as a priority. Component 2: Development of Renewable Energy Sector Relevance of the project objectives to Turkey’s energy sector strategies: In the Electricity Market & Security of Supply Strategy, approved by High Planning Council in 18.05.2009 (No. 2009/11), the main target is identified as providing 30% of total electricity production from renewable energy. In relation to that, the detailed targets of renewable energy utilization are given within the Institutional Strategic Plan for the MENR (April 2010). Also, in Climate Change Strategy (approved by the High Planning Council Decision dated May 03, 2010 and numbered 2010/8) and Climate Change Action Plan (approved on 2 May 2011 and numbered 2011/1 by Climate Change Coordination Council) the targets envisage increasing the share of renewable energy in electricity production, development of capacity by 2015 so as to increase utilization of renewable energy sources, and ensuring technological development by 2020 for energy production from renewable energy sources. Efficiency: The design of this component was mainly built upon three pillars: (i) techno-economic & policy assessment and strategy proposal for promotion and grid integration of increasing levels of renewable energy sources, (ii) financial, techno-economic & policy assessment and strategy proposal as well as pilot bankability support for small-scale renewable energy investments, (iii) feasibility study and procurement documents for monitoring, forecasting and control system for solar power plants. Throughout the implementation, under each pillar, trainings and study visits to EU examples were organized. Trainings and study visits were helpful in supporting the policy and technical knowledge of the MENR and TEIAS staff in the sense that they have provided theoretical, regulatory and policy background for sub-sectors and practical examples of implementation as well as an opportunity for comparing and contrasting between Turkey and EU Member States. Then, for each output (feasibility study, roadmap, assessment report, strategy and action plan), multi-stakeholder workshops were organized to collect comments from stakeholders and finalize the reports. Workshops were useful in bringing in the critical ideas and unvoiced propositions from altering layers of the market participants and institutions. Page 57 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Although use of trainings and workshops was useful and an effective way of achieving the project objectives, the design of the Terms of Reference and scope of work referred the venue and organizational expenditures to Component 1 which was inefficient due to hardships in coordination between two different Contractors. Also, bankability support for only 10 SMEs was foreseen in the initial design of the project in the project fiche as opposed to 155 small scale projects supported in reality for bankability and technical feasibility assessment. Hence, it is noteworthy to recognize that the budget foreseen for the SME support services to be procured was overestimated and the number of willing SMEs to participate to the project was underestimated in the initial design. A similar criticism can be made for the number of workshops, trainings and trainees foreseen in the initial design in the project fiche. Comparably, 8 trainings for totally 234 trainees were organized against the initially designed 3 trainings, 7 workshops with totally 329 participants were organized against the initially designed– 1 workshop for 50 participants and 12 days for 2 site visits with totally 15 visitors were organized against the initially designed 5 days for a single site visit for 5 visitors. Nevertheless, the outcome of these activities proved efficient thanks to the intervention made by the PIU at the ToR and contract negotiation phase. Efficacy: General remarks: Competitive tenders regarding 1000 MW mega projects for each solar PV and wind energy generation plants and equipment manufacturing were successfully organized with the contribution of the assessment and recommendations of the Consultant under the Recommendations Report and Roadmap - Strategy and Action Plan that were being prepared concurrently with the tenders. Also, as well as the above-mentioned reports and plans, the Consultant also prepared the Recommendations Report and Roadmap Plan for financing and support for small-scale RE investments. A calculation tool for each RE technology type was created to foresee LCOE and IRR of the RE investments. Both the reports and the Tool have already been valuable inputs for the DGRE’s current studies for designing the post-2020 RE support mechanism. On the other hand, feasibility study and procurement documents for a monitoring, forecasting and control system for solar power plants have been prepared by the Consultant. This system is considered to contribute to DGRE’s fraud detection and monitoring functions for increasing LV and MV connected solar PV plants while assisting TEIAS to predict production. Indicators of these objectives are as follows: • Indicator 1: 5 personnel of MENR trained in promotion and monitoring of renewable energy development by 2014. Achievement: 8 trainings for totally 234 trainees were organized by September 2017.However, due to delays in administrative procedures regarding conclusion of a Grant Agreement between Republic of Turkey and the Bank, the project started implementation only at the end of 2015. • Indicator 2: Review of issues and options in financing and support mechanisms for small-scale renewable energy projects. Page 58 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Achievement: Financing and support mechanisms for small-scale RE investments were assessed and a Recommendations Report and Roadmap Plan was prepared. • Indicator 3: Study of the grid’s technical capacity for the integration of electricity produced from renewables enhanced by 2015. Achievement: Bottlenecks in grid integration of RES in Turkey were analyzed and a Recommendations Report and a Roadmap - Strategy and Action Plan were prepared. Reasons for delayed achievement is explained under Indicator 1. • Indicator 4: Supporting at least 10 SMEs on renewable energy through business development services Achievement: Bankability support for only 10 SMEs were foreseen in the initial design of the project in the project fiche. Then, during negotiations with the Contractor, the number was increased to a maximum of 186 with varying sizes and characteristics of investments. Accordingly, the whole budget allocated for this activity was fully utilized and 155 small-scale renewable energy investments have been supported for bankability and technical feasibility assessment. As of September 28th, 2017, 69 of them have either received credit disbursements or went ahead with construction with equity. It is known that credit was disbursed for 42 of them, which are waiting for their construction to begin. 13 projects are under construction with credit disbursements, while 5 projects started their construction while they are waiting for credit disbursement. 4 projects have gone ahead and started their constructions with equity, and 5 projects have returned to investments after receiving credit disbursements. Of the other 86 projects, some of them were already at idea level, and as a result they received feasibility studies, and the remainder are still in the application process for credits. Component 3: Design and Implementation Plan for Gas Trading Platform: Relevance of the Project Objectives to Turkey’s Energy Sector Needs and Strategies: The natural gas market in Turkey is regulated, monitored and supervised within the framework of the Natural Gas Market Law which was adopted in 2001. Although the Law envisages a fully liberalized market structure, there are shortcomings with regard to implementation. Accordingly, the Strategic Plan of the MENR articulates specific targets aiming at encouraging investments through the development of competitive and transparent energy markets and improvement of infrastructure investment procedures in the energy sector. Within this context, the project regarding ‘Design and Implementation Plan for Gas Trading Platform (GTP)’ mainly aimed to support BOTAŞ as the system operator and EPİAŞ as the market operator for their respective future roles in a more competitive and transparent gas market. The alignment between project Page 59 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) objectives such as enhancement of the competitiveness of the Turkish gas market; development of market rules, operational requirements for domestic and international exchanges and pricing methodologies to be employed in a competitive market environment and Turkey’s sector strategies regarding the development of competitive and transparent energy markets is obvious and there is not any inconsistency experienced at the ‘objective’ level. Efficiency: The contract regarding Design and Implementation Plan for GTP Project was signed for a period of 12 months and it was extended to 25 months with an extension of services and the budget. The original scope of the project was executed under 2 main tasks: 1. Review of the existing EBB and feasibility studies for new EBB: Overall, 17 functional requirements for transparency, 94 functional requirements for primary and secondary capacity tradings were developed for the new EBB. These functional requirements were analyzed in-depth and led to 21 specifications. 2. Functional and structural design of a GTP and related components considering its integration with BOTAŞ’ SCADA and simulation systems: Recommendations for most appropriate market model for Turkey and the related market transformation road map by taking into account the market development, physical and institutional capacity of Turkey and the needs for the measurement infrastructure for effective operation of the balancing markets were delivered. The Consultant, in total, has delivered 10 separate reports on the market analysis, business requirements, functional and technical design of GTP; successfully organized and completed 4 site visits to Italy, Germany, Austria-Belgium and the United Kingdom, respectively; and the 11-day-long training sessions with the attendance of an average 45 participants for each session from different institutions (BOTAŞ, MENR, EMRA, EPİAŞ, World Bank, European Delegation, etc.). The additional scope of the project identified within the extension of the contract was executed under three main tasks: • Assessment and Recommendations for the Establishment of Turkish Natural Gas Trading Hub • Assessment and Recommendations for the Limitations of BOTAŞ Stemming from the Decree Law on SOE No. 233 • Trainings on energy trading, risk management, pricing methodologies, ENTSOG 10-Year Network Development Plan, tariff structures and congestion management. The Consultant, within the extension of the services, has delivered 3 separate reports and the 9-day-long training sessions with the attendance of an average 75 participants for each session from different institutions (BOTAŞ, MENR, EMRA, EPİAŞ, PETFORM, etc.). The original contract of the project was concluded with a budget amounting the 65% of the budget allocated for the Component 3 in the design phase. The amended contract with the extension of the Page 60 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) services was concluded with a budget amounting the 15% of the design budget and so 80% of the budget estimation in the design phase was disbursed in total. Hereby two conclusions could be drawn: either the budget allocated in the design phase is overestimated or the contractor was too ambitious to be awarded even at the expense of making a loss taking into account that its financial proposal was under the average of the all financial proposals. Efficacy: The following outcomes were expected through the implementation of the project: Confirmation of the compatibility of the new EBB with the requirements of an OTC in a gas market platform and recommendations for changes to its technical and functional specifications; preparation of the functional design for the establishment a GTP and related components to enable commercial balancing and/or wholesale spot gas trading; providing the necessary and sufficient conditions for the establishment of the Turkish natural gas hub; training of personnel through site visits, trainings and workshops. All expected outcomes of the project were met by conducting following main activities: Review of existing EBB and new EBB development; assessment of European and Turkish gas markets; gas trading platform and market design; preparation of the necessary and sufficient conditions for Turkish Natural Gas Hub; training and capacity building. Only a few months following the project completion, EMRA adopted the regulation regarding the organized natural gas wholesale market, which aims to let the market players trade natural gas anonymously in an organized liberal market operating by continuous trade principles, besides letting the transmission system operator balance the system by entering the continuous trade platform when needed. In addition, EPİAŞ revealed that it plans to initiate the market operations by the April 2018. In the light of these developments it could be concluded that the outcomes of the project like design of the GTP are certainly relevant with the objectives of both this project and the gas sector. However, as it was concluded at the Final Report of the project, the design of the project activities did not directly address the sector’s current and urgent needs such as prevention of potential distorting effect of BOTAŞ’ dominant market position on the fair competition environment. GTP serves as the key tool, as a “roof”, to reach a well-functioning, liquid gas trading hub which will pave the way towards sustainable/competitive prices. Nevertheless, without the prerequisites, the “building blocks”, Gas Trading Platform alone will not lead towards a liquid gas trading hub. Therefore, the outcomes of the project are relevant with the objectives but will not likely to have an impact in the liberalization of the gas sector in the short term. This may give an indication for the weakness of the design of the project activities. Component 4: Energy Efficiency Relevance of the Project Objectives to Turkey’s Energy Sector Needs and Strategies: With regard to energy efficiency policies, priorities set in the 9th Development Plan for 2007–2013. To build the required legal framework in the field of energy efficiency in line with the 9th Development Plan, Energy Efficiency Law and its secondary legislation were put into force at 2007 and 2008 and an Energy Efficiency Strategy was approved in February 2012. The Energy Efficiency Strategy Paper calls for energy intensity reduction of at least 10% for each subsector within 10 years, based on 2011 consumption levels. Page 61 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Furthermore, Section 1.14 of the 10th Development Plan (2014–2018), which was approved in 2013 includes a section for the Energy Efficiency Development Program. In line with these policies and incentives, it is clear that it is vital to overcome the various market barriers in Turkey in order encourage energy efficiency investments. Informational (pilots, audits, project formulation, awareness building) activities, incentive and policy measures, implementing market-based mechanisms (banking products, ESCOs, leasing) and capacity building efforts are the tools which can help to overcome these barriers. In line with the need to overcome the barriers, the Energy Efficiency component is designed to meet the following objectives: • Ensuring market development and scale-up for EE in the industrial and building sectors; • Supporting for commercial lending for EE in these sectors; • Developing an ESCO industry in Turkey, to help package, finance and implement EE projects; and • Enhancing the policy and institutional framework in line with the EU acquis; Finally, as the main focus of the component was to facilitate various market actors to overcome barriers to investment, this project has been implemented in collaboration with other relevant institutions, such as Industrial Organization Zones, Industrial and Commercial Chambers, Turkish banks which finance or are interested in financing EE investments and international financial institutions (IFIs) and donors that support them with their lines of credit and other TA. Efficiency: The contract regarding Energy Efficiency Project was signed for a period of 18 months and it was extended to 24 months with an extension of services and the budget. In the scope of the Project, policy support was provided to help effective implementation of the Energy Efficiency Strategy. An energy efficiency market study was conducted and a market assessment report on the current situation of the EE market in Turkey was developed accordingly. There were several organizations such as seminars and workshops implemented in various stakeholders including Chambers of Industry. Two major training programs were also developed and implemented for ESCOs and commercial banks. A number of ESCO model contracts were developed and associated reports were prepared along with a roadmap report to develop an ESCO industry in Turkey. Close to 300 energy audits were also conducted to raise awareness in SMEs as part of the Project. The outcomes of these audits were analyzed to help SMEs in project financing with the help of bankable reports developed for each audit. ESIA brief note and a gender sensitive survey were also developed based on the energy audits. Energy Efficiency financing mechanisms including new banking products and financial modalities that can be applied in Turkey were presented. Guidelines for stakeholders such as banks and ESCOs were also generated. Page 62 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) To sum up, a wide range of activities (reports, audits, training and awareness raising events) were conducted and various stakeholders such as MENR, SMEs and banks received support necessary to address the needs of the energy efficiency sector. In terms of budget, it was set at a sufficient and reasonable level so that all the financial proposals of prospective consultants were close to each other and also close to the planned budget of € 4.5 million. Efficacy: The following results were expected through realization of the Energy Efficiency Component: • Supporting and enhancing the effective implementation of the measures stated in Energy Efficiency Strategy Paper. • Improving project preparation capacity in energy efficiency and developing capacity for Commercial Energy Efficiency Lending Market. In order to attain those results, four output level objectives were set. Main achievements of the Project in relation to these four objectives are summarized below: Ensuring market development and scale-up for EE in the industrial and building sectors The Consultant developed a comprehensive Report on the assessment of Energy Efficiency market in Turkey. Furthermore, the Consultant developed a benchmarking Report and a benchmarking Tool with the aim of comparing Energy Efficiency indicators in Turkey against BAT/BREFs. A number of seminars/workshops were organized for several stakeholders including Chambers of Industry (CoI), Organized Industrial Zones (OIZs) and sectorial associations. Detailed technical training programs were developed and implemented for the EVDs in Turkey which aimed to help develop a sustainable Energy Efficiency market. 262 Energy Audits were conducted for SMEs which is one of the most important achievements of the Project. Also, standardized templates were developed for Energy Audit Reports and Bankable Reports. Bankable Reports especially is a key element to link the investors to the financial institutions which would eventually help overcome the biggest barrier in front of the Energy Efficiency market; “access to Energy Efficiency financing”. For 8 municipalities and 253 SMEs Energy audits were conducted, bankability reports were produced and communication with the banks was facilitated on behalf of these beneficiaries. Preparation of Guidelines for Stakeholders, guidelines were developed for ESCOs as well as banks and leasing companies. Supporting for commercial lending for EE in these sectors Training programs were developed and implemented for several commercial banks, a comprehensive study was conducted to identify Energy Efficiency financing mechanisms that are applicable in Turkey, all SMEs where Energy Audits were conducted were contacted after the Energy Audit Reports were developed and shared with them and consultancy was provided as to how measures included in the Report can be implemented and what financing options can be considered in Turkey to finance these measures. Page 63 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Developing an ESCO industry in Turkey, to help package, finance and implement EE projects Several ESCO Model Contract templates that can be used in Turkey were developed and ESCOs were trained. Enhancing the policy and institutional framework in line with the EU acquis An analysis of the Articles of the new EU Energy Efficiency Directive (2012/27/EU) was conducted Complex articles of the EED were also explained in detail a roadmap was developed for transposing and implementing the EED. In relation to awareness raising and training activities, 370 people from various Chambers of Industry and Commerce attended awareness raising events and 52 staff from GDRE, 300 people from various ESCOs and 335 participants from various commercial banks attended the trainings. To sum up, this component was executed in a way that ensures all stakeholders were targeted and more importantly in a way that ensures these stakeholders benefit from the outputs of the component. This is especially true for capacity building activities where stakeholders were informed about both the benefits of energy efficiency and the main results of the reports submitted to MENR. Other indications of successful implementation are the wide geographical coverage of these events and the high level of interest and attendance by the stakeholders and also 37 projects are started/conducted within 253 audits. Future Recommendations • The design phase of the project was conducted between 2011 and 2012. The Financial Agreement between Republic of Turkey and European Commission was signed and adopted in March 2013 and the Grant Agreement between Republic of Turkey and the Bank was signed in May 2014. First contract under the Phase-I Project was signed in March 2015. As it is seen, there are more than three years between the design and implementation phases. Therefore, the scope of the projects needs to be designed in a more flexible way to allow for the adjustments to serve for the current gaps and needs of the beneficiary institutions. • Identifying the end-beneficiary interface responsible for each task in beneficiary institutions other than the PIU and strengthening its role is beneficial in order to smooth the flow of exchanges and increase efficiency. • A more extended use of dedicated meetings between the Consultant experts and the end- beneficiary Contact Groups and other Stakeholders should be foreseen to present, discuss, and consolidate the various Reports/Deliverables. • The language issue should be taken into account when organizing the tender with the necessary provisions in terms of skills to be requested from the consultant and the client/beneficiary and in terms of appropriate budget allocation for Translation - Interpretation. • Planning of ToR should consider a more specific organization of the work packages including Task leaders and Team Leader and be more precise in the scope of work and requested performances. Page 64 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) • Incidental expenditures for venues, lunch and organizational matters must be included in each of the Component’s budget instead of assigning them to a separate Contract that makes it difficult to coordinate between two different Contractors. • Consultant experts must be held liable for spending more time in the field than home for liaison, data collection, and reporting in the ToR. • When study/site visits are to be programmed, internal assignment procedures should be taken into consideration and the number of both visits and the participants should be kept at a reasonable level. 2) Administrative Performance and Lessons Learned for PIU and WB PIU Staff capacity for design, tendering and implementation Number of PIU staff was inadequate to fulfil its obligations especially in the tendering phase, however in the implementation phase, the PIU was supported with additional staff and so all contracts were completed successfully. PIU staff were qualified and able to learn the rules of the Bank for tendering and contract management. However, as it was first of its kind, it took time for settling the joint management experience in MENR. Administrative Measures and Continuity During tendering, there were some irregularities in the number of personnel dedicated to PIU tasks and change of positions. However, with the IPA II period and accreditation of the MENR as the Lead Institution in the Energy sector, a more stable and closely monitored personnel organization have been introduced which also indirectly enhanced the regularity in the PIU. Delays in tendering and implementation periods Delays in starting the implementation of the projects mainly emanated from the delays in conclusion of the Grant Agreement which took approximately 15 months, later than initially anticipated. PIU introduced a proposal to overcome the legal ambiguities which involved giving authorization to the Minister of Energy and Natural Resources to sign International Agreements as per Law no: 244. This paved the way to the signing of the Grant Agreement. Monitoring and Review of Outputs Review of project outputs was an issue partly due to lack of dedicated personnel on part of some of the end-beneficiary institutions, and partly due to occasional weak enforcement in PIU due to staff changes. Review periods were prolonged for some outputs due to these reasons. Financial Management and Payments VAT exemption, stamp tax and payment procedures emerged as the main problematic areas for payments. PIU had to obtain VAT exemption certificate from relevant public bodies to exempt these Page 65 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) payments from VAT. This procedure took some time (approximately 12 months) and paved the way for IPA 2013 Energy Sector Program Phase-II project. Also, the VAT exemption could not be utilized by the sub-contractors of the Contractor because of the administrative setting defined in the Administration and Grant Agreements. It should be addressed in the future arrangements. In addition, IPA projects are legally exempt from any taxes however the Contract between the PIU and the Consultants are subject to stamp tax. Stamp tax must be included in the Contracts during negotiations and Contractors must be notified in the Contract about their liability to pay the stamp tax. WB Guidance and assistance to PIU staff for design, tendering and implementation The Bank did not have a training and orientation plan for MENR PIU staff which obliged the personnel to adapt to the newly introduced contract management modality in a hands-on manner. The Bank Guidelines referred to as the source of information needed in the tendering phase do not always steer the PIU in an effective and efficient way. This increased the prospects for mistakes or delays in the adaptation process. The results were successful at the end of the implementation. However, the Bank is suggested to plan trainings on contract management and Bank Guidelines for procurement for similar future collaborations with governmental institutions where beneficiary institutions will be required to function and supervise according to WB guidelines. The Bank, in general, was reluctant to give prompt and written feedbacks for the clarification requests made by the PIU and preferred to deliver its comments in the review meetings. This method was both time-consuming and reduced the quality of feedbacks. The Bank is suggested to give its comments in a timelier manner and in writing as much as possible. Enforcement for Administrative Measures and Continuity The Bank put forth some positive efforts to bring up the issue regarding the irregularities in the number of personnel dedicated to PIU tasks and change of positions at the high-level meetings with the Ministry. Delays in tendering and implementation periods Reviews by the Bank of the Evaluation Reports of the PIU on the tenders were done by different Bank personnel which occasionally resulted in contradicting opinions and requests from the Bank regarding tender evaluation. The procedures regarding the receipt of Bank’s ‘No Objection Letter’ for all phases in the tendering process caused delays in each stage until the signing of the contract mainly because of late responses by the Bank. Monitoring and Review of Outputs The Bank experts’ participation in review of outputs was helpful in guiding the Consultants for producing more effective reports. However, changes in reviewing personnel on Bank’s side sometimes resulted in contradicting comments by the Bank on the same reports. Continuity in review experts proved to be significant for sustainability of the review of outputs. Also, the Bank experts have provided remarkable contribution to the outputs with their international sectoral experience. However, some cases suggested Page 66 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) that the Bank review experts must be more flexible in finding common ways to adapt to Turkey’s specific circumstances in certain sub-sectors. Knowledge of the Turkish legal, regulatory and policy practices must be highlighted on the Bank’s side. Financial Management and Payments Although the Bank has provided some ad hoc assistance to the PIU for financial issues, it should have provided much more support regarding the financial reporting, realization of the payments and accounting of the payments when PIU’s incapacity for financial management is considered. The Bank is suggested to provide a full-time assistance to the PIU for payments and accounting with a dedicated expert. Page 67 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) Final Procurement Plan Bank's Contract Estimated Contract Type Reference Procurement Review No Description Cost(EURO) Amounts(EURO) Grant Method Method Total (#a) Total (#a) (Prior/Post) CONSULTANTS' SERVICES Consultant Services for Institutional Review and Capacity Building of MENR, Acquis EU IPA CS-01 Alignment in the Energy Sector, BOTAŞ CS 2,525,000.00 2,520,000.00 QCBS Prior 12 Unbundling Support and Effective Gas Market Opening, Visibility and Public Awareness EU IPA Consultant Services for Development of CS-02 CS 2,200,000.00 2,194,804.00 QCBS Prior 12 Renewable Energy Consultant Services for Design and EU IPA CS-03 Implementation Plan for a Gas Trading CS 1,400,000.00 1,110,000.00 QCBS Prior 12 Platform EU IPA CS-04 Consultant Services for Energy Efficiency CS 4,500,000.00 4,549,500.00 QCBS Prior 12 Total Consultants' Services 10,625,000.00 10,374,304.00 Contingency 250,696.00 GRAND TOTAL 10,625,000.00 Note: QCBS = Quality- and Cost-based Selection; (#a) = The total costs inclusive of all physical and price contingencies but excluding taxes incurred in Turkey. Page 68 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) ANNEX 7. LIST OF ENERGY EFFICIENCY AUDITS CONVERTED INTO INVESTMENT SO FAR Investment Size No. Subsector Province Status (TRY) 1 Shopping Mall Van 2,200,000.00 Implemented 2 Plastic Aydin 7,845.00 Implemented 3 Plastic Aydin 44,913.00 Implemented 4 Plastic Aydin 1,290,451.00 Implemented 5 Plastic Aydin 268,578.00 Implemented 6 Hotel Adana 3,000.00 Implemented 7 Hotel Adana Not shared Implemented 8 Hotel Adana 3,000.00 Implemented 9 Iron-Steel Industry Zonguldak 2,100.00 Implemented 10 Iron-Steel Industry Adana 5,000.00 Implemented 11 Textile Adana 21,750.00 Implemented 12 Textile Adana 2,200.00 Implemented 13 Paper Mersin 25,000.00 Implemented 14 Paper Mersin 107,330.00 Under construction 15 Textile Denizli 250,000.00 Implemented 16 Textile Denizli 10,791.10 Implemented 17 Textile Denizli 17,721.00 Implemented 18 Chemical Izmir 8,495.00 Under construction 19 Iron-Steel Industry Istanbul 3,500.00 Implemented 20 Iron-Steel Industry Istanbul 1,600.00 Implemented 21 Iron-Steel Industry Istanbul 3,000.00 Under construction 22 Iron-Steel Industry Istanbul 30,000.00 Under construction 23 Iron-Steel Industry Istanbul 58,000.00 Implemented 24 Iron-Steel Industry Istanbul 640,000.00 Implemented 25 Iron-Steel Industry Istanbul 232,327.80 Under construction 26 Chemical Tekirdag 1,500.00 Implemented 27 Hotel Istanbul 173,584.00 Implemented 28 Textile Tekirdag 15,000.00 Implemented 29 Textile Tekirdag 130,000.00 Implemented 30 Textile Istanbul 5,000.00 Under construction 31 Textile Istanbul 25,000.00 Under construction 32 Textile Istanbul 7,500.00 Under construction 33 Iron-Steel Industry Konya 5,000.00 Implemented 34 Iron-Steel Industry Konya 5,000.00 Implemented 35 Textile Bursa 404,093.00 Implemented TOTAL 6,008,278.90 Page 69 of 70 The World Bank EU/IPA Energy Sector Technical Assistance Project ( P131921 ) Page 70 of 70