Report No: ICR00388 IMPLEMENTATION COMPLETION AND RESULTS REPORT ON A CREDIT IN THE AMOUNT OF EUR 20,700,000 MILLION (US$25 MILLION EQUIVALENT) TO THE Republic of Kosovo FOR Kosovo Digital Economy (KODE) April 29, 2025 Digital Development Europe And Central Asia Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT CURRENCY EQUIVALENTS (Exchange Rate Effective March 03, 2025) Currency Unit = EUR EUR1 = US$1,038 US$1 = EUR 0,963 FISCAL YEAR 2025 January 1 - December 31 Regional Vice President: Antonella Bassani Country Director: Xiaoqing Yu Regional Director: Michel Kerf Practice Manager: Michel Rogy Task Team Leader (s): Charles Hurpy, Natalija Gelvanovska-Garcia, Rhedon Begolli ICR Main Contributor: Nicola Bressan, Samuel Chicheportiche Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ABBREVIATIONS AND ACRONYMS ARKEP Regulatory Agency for Electronic Communications and Postal Services BFD Budget and Finance Department (of the Ministry of Economic Development) CAGR Compound Annual Growth Rate CETEP Creating Employment through Export Promotion (of German Agency for Internation Cooperation) CPF Country Partnership Framework COB Cost-benefit analysis CQS Selection based on consultants’ qualifications DE Digital Economy DPO Development Policy Operation ECA Europe and Central Asia ESMF Environmental And Safeguards management framework ESMP Environmental and Social Management Plan ERP Economic Reform Program ERR Economic rate of return EU European Union EYE Enhancing Youth Employment project (of Swiss Agency for Development and Cooperation) FDI Foreign direct investment FM Financial management FY Fiscal year GDP Gross domestic product GoK Government of Kosovo GM Grant Manual GRM Grievance Redress Service (of the WBG) HEIs Higher educational institutions IBRD International Bank for Reconstruction and Development (of the WBG) ICK Innovation Centre Kosovo ICT Information and communication technologies IDA International Development Association (of the WBG) IFC International Finance Corporation IFR Interim Financial Report IRR Internal Rate of Return M&E Monitoring and Evaluation MED Ministry of Economic Development MFD Maximizing Finance for Development (World Bank approach) MoF Ministry of Finance MNOs Mobile Network Operators MTEF Medium Term Expenditures Framework NDS National Development Strategy 2016-2021 (Kosovo) NPV Net Present Value NSMS National Spectrum Monitoring System Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT NREN National Research and Educational Network OECD Organization for Economic Co-operation and Development OP/BP Operational Policies / Bank Procedures PAD project Appraisal Document PAs Protected area PCC Project and Component Coordinator PDO Project Development Objective PPP Public Private Partnership PFS Project Financial Statements PFM Public Financial Management POM Project Operations Manual PPSD Project Procurement Strategy for Development QCBS Quality-and cost-based selection R&D Research and Development RENs Research and Educational networks SCD Systematic Country Diagnostic SEO Search Engine Optimization SMEs Small and Medium Enterprises SOE Statement of Expenditures SPDs The World Bank’s standard procurement documentation STEP Systematic Tracking of Exchange in Procurement TA Technical assistance TTL Task Team Leader USAID US Agency for International Development VPN Virtual private network WeBa Western Balkans WB The World Bank Group Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT TABLE OF CONTENTS DATA SHEET ................................................................................................................................................. i I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ......................................................................... 1 A. CONTEXT AT APPRAISAL ................................................................................................................................................. 1 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION...................................................................................................... 3 II. OUTCOME ......................................................................................................................................... 4 A. RELEVANCE OF PDO ........................................................................................................................................................ 4 B. ACHIEVEMENT OF PDOs (EFFICACY) ............................................................................................................................... 5 C. EFFICIENCY ...................................................................................................................................................................... 9 D. JUSTIFICATION OF OVERALL OUTCOME RATING.......................................................................................................... 10 E. OTHER OUTCOMES AND IMPACTS (IF ANY) .................................................................................................................. 10 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME................................................... 12 A. KEY FACTORS DURING PREPARATION .......................................................................................................................... 12 B. KEY FACTORS DURING IMPLEMENTATION ................................................................................................................... 12 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME ..................... 13 A. QUALITY OF MONITORING AND EVALUATION (M&E) .................................................................................................. 13 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .......................................................................................... 15 C. BANK PERFORMANCE ................................................................................................................................................... 15 D. RISK TO DEVELOPMENT OUTCOME .............................................................................................................................. 16 V. LESSONS AND RECOMMENDATIONS ................................................................................................ 16 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................................................ 18 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ....................................................... 25 ANNEX 3. PROJECT COST BY COMPONENT ......................................................................................................... 27 ANNEX 4. ECONOMIC ANALYSIS ........................................................................................................................ 28 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ................................. 37 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ................................................................................................... 38 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT DATA SHEET @#&OPS~Doctype~OPS^dynamics@icrbasicdata#doctemplate BASIC DATA Product Information Operation ID Operation Name P164188 Kosovo Digital Economy (KODE) Product Operation Short Name Investment Project Financing (IPF) Kosovo Digital Economy (KODE) Operation Status Approval Fiscal Year Closed 2019 Original EA Category Current EA Category Partial Assessment (B) (Restructuring Data Sheet - 23 Dec Partial Assessment (B) (Approval package - 05 Jul 2018) 2024) CLIENTS Borrower/Recipient Implementing Agency Ministry of Finance, Republic of Kosovo Ministry of Economy DEVELOPMENT OBJECTIVE Original Development Objective (Approved as part of Approval Package on 05-Jul-2018) The Project Development Objective (PDO) for the Kosovo Digital Economy (KODE) Project is to improve access to better quality and high-speed broadband services in project areas and to online knowledge sources, services and labor markets among citizens, and public and academic institutions. s s s @#&OPS~Doctype~OPS^dynamics@icrfinancing#doctemplate FINANCING i Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Financing Source Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 25,000,000.00 20,127,999.71 19,167,769.12 IDA-62850 25,000,000.00 20,127,999.71 19,167,769.12 Total 25,000,000.00 20,127,999.71 19,167,769.12 RESTRUCTURING AND/OR ADDITIONAL FINANCING Amount Disbursed Date(s) Type Key Revisions (US$M) • Results • Disbursement Estimates 16-May-2023 Portal 12.42 • Loan Closing Date Extension • Reallocations • Results 17-Jul-2024 Portal 17.92 • Loan Closing Date Extension • Implementation Schedule • Disbursement Estimates 23-Dec-2024 Portal 19.17 • Loan Cancellations • Reallocations @#&OPS~Doctype~OPS^dynamics@icrkeydates#doctemplate KEY DATES Key Events Planned Date Actual Date Concept Review 17-Oct-2017 17-Oct-2017 Decision Review 12-Mar-2018 12-Mar-2018 Authorize Negotiations 21-May-2018 14-May-2018 Approval 05-Jul-2018 05-Jul-2018 Signing 20-Jul-2018 20-Jul-2018 Effectiveness 16-Nov-2018 16-Nov-2018 Restructuring Sequence.01 Not Applicable 16-May-2023 Restructuring Sequence.02 Not Applicable 17-Jul-2024 Restructuring Sequence.03 Not Applicable 23-Dec-2024 Mid-Term Review No. 01 19-Apr-2021 19-Apr-2021 Operation Closing/Cancellation 31-Dec-2024 31-Dec-2024 ii Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ICR/NCO 30-Jun-2025 -- @#&OPS~Doctype~OPS^dynamics@icrratings#doctemplate RATINGS SUMMARY Outcome Bank Performance M&E Quality Satisfactory Satisfactory Modest ISR RATINGS Actual Disbursements No. Date ISR Archived DO Rating IP Rating (US$M) 01 17-Feb-2019 Satisfactory Satisfactory 0.00 02 20-Oct-2019 Satisfactory Satisfactory 0.00 03 29-Mar-2020 Satisfactory Satisfactory 1.70 04 19-Nov-2020 Satisfactory Satisfactory 1.70 05 09-Jun-2021 Satisfactory Satisfactory 3.56 06 20-Feb-2022 Satisfactory Satisfactory 8.99 07 02-Sep-2022 Satisfactory Satisfactory 9.23 08 17-Mar-2023 Satisfactory Satisfactory 12.10 09 19-Sep-2023 Satisfactory Satisfactory 13.81 10 25-Jan-2024 Satisfactory Satisfactory 14.57 11 24-Jul-2024 Satisfactory Satisfactory 17.92 12 20-Dec-2024 Satisfactory Satisfactory 17.92 @#&OPS~Doctype~OPS^dynamics@icrsectortheme#doctemplate SECTORS AND THEMES Sectors Adaptation Mitigation Major Sector Sector % Co-benefits Co-benefits (%) (%) FY17 - ICT Infrastructure 72 0 0 iii Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT FY17 - Other Information and 8 0 0 FY17 - Information and Communications Technologies Communications FY17 - Public Administration - Technologies Information and Communications 20 0 0 Technologies Themes Major Theme Theme (Level 2) Theme (Level 3) % FY17 - Human FY17 - Gender 12 Development and FY17 - Labor Market Policy and FY17 - Skills Gender 26 Programs Development FY17 - ICT Policies 20 FY17 - ICT FY17 - Private Sector FY17 - ICT Solutions 84 Development FY17 - Youth FY17 - Jobs 12 Employment ADM STAFF Role At Approval At ICR Practice Manager Jane Lesley Treadwell Michel Rogy Regional Director / Zhenwei Qiang Global Director Jose Luis Irigoyen Zhenwei Qiang Practice Group Vice President / Sangbu Kim Country Director Linda Van Gelder Xiaoqing Yu Regional Vice President Cyril E Muller Antonella Bassani ADM Responsible Team Leader Natalija Gelvanovska‐Garcia Charles Hurpy Co-Team Leader(s) Rhedon Begolli Rhedon Begolli ICR Main Contributor Nicola Bressan, Samuel Chicheportiche iv Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL 1. Kosovo had made significant efforts to promote access to and use of digital technologies, but at appraisal high- quality connectivity was unevenly distributed (urban/rural) and usage was mostly unproductive. The discrepancy between rural and urban areas both in terms of education programs and access to school was critical. Only 1 percent of households had access to fiber, and many public institutions relied on low-speed connectivity or were unconnected. 2. The education system in Kosovo had not succeeded in equipping students with the technical skills to thrive in the changing labor market and suffered from poor digitalization. Public education spending was low relative to Western Balkan (WeBa) peers, and quality of education remained poor and inequitable. Kosovo’s higher educational institutions (HEIs) often overpaid for broadband services that did not meet their quality needs.1 Unlike other European Union (EU) countries, Kosovo had not yet established a research and education network. 3. The Ministry of Economic Development (MED) guided and coordinated information and communication technology (ICT) sector development as well as all major digital economy activities . Regulatory Authority of Electronic and Postal Communications (ARKEP), Kosovo’s telecommunications sector regulator, supports competition in the market and oversees quality of services and consumer protection. Both these institutions reported having difficulties retaining qualified staff, resulting in nimble yet overstretched teams. The Ministry of Education had formal authority over digital education matters, but little capacity to implement its objectives related to higher and basic education. 4. Acknowledging the limitations in human capacity and financial resources in pursuing its goals on digital matters, the government requested World Bank support, building on an existing positive relationship and common understanding of priorities. The project required World Bank expertise and guidance to navigate the diverse activities. It was built on an ongoing cooperation between MED and the World Bank, with its design heavily influenced by international experience pooled by the World Bank, World Bank-financed technical assistances, and World Bank-financed pilot projects. At appraisal, there was broad consensus on the necessity to crowd-in fixed connectivity investments in unprofitable areas with public co-financing as well as for other components such as the Spectrum Monitoring System. Conversations were advanced enough for Kosovo Digital Economy (KODE) to be mentioned in Kosovo’s National Development Strategy (NDS) 2016–2021. Theory of Change (Results Chain) 5. The theory of Change was not a requirement of the PAD at appraisal and was developed for this ICR. It represents components and subsequent outputs at appraisal, prior to the modifications brought forward during implementation. FIGURE 1: RESULTS CHAIN AND THEORY OF CHANGE Development Project Outputs Outcome Long‐term Challenges Components Outcomes Villages connected to Unevenly distributed high‐speed broadband high‐speed broadband E pansion of digital connec vity in Increasing Kosovo s access Ins tu ons connected to underserved areas and connec ons to the high‐speed broadband Improve access to global economy improving the enabling Lack of ins tu onal internet be er quality and environment for access to high‐quality ireless broadband high speed broadband Establishment of a broadband connec vity services services in project Na onal Spectrum Management System areas Increasing incomes Poorly interconnected sustainably higher educa on train People skilled and ins tu ons youth in online skills employed a er training and Increasing access programs Lack of decent job to kno ledge, Enhance condi ons for opportuni es, especially informa on, and Improve access to accelerated private in rural areas services Establishment of a online kno ledge, sector gro th and Na onal Research and services labor employment Unproduc ve use of the Educa on Net ork and markets for ci ens, internet connec on to GEANT public sector, academia support Improve access to Limited human and Support Kosovo s e orts project management, online labor market nancial in European Union implementa on and informa on, promote implementa on integra on ci en engagement a areness and civil capacity of authori es ac vi es society par cipa on 1 19 universities and colleges (out of 29, in total) pay, on average, €3.45 per Mbps per month for internet. Official Use Only Page 1 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT 6. Under the theory of change developed, the project delivered on all its key outputs at appraisal, effectively contributing to the PDOs and achieving impactful additional activities compared to the scope at appraisal. Detailed further in the outcome section and in annex 6, the project’s main achievements as in the PAD include, but are not limited to (a) providing fixed broadband connectivity to 4,475 rural households and 103 schools and hospitals with awareness campaigns; (b) establishing a functioning National Spectrum Monitoring System (NSMS); (c) training 2,000 beneficiaries in high-demand ICT skills; and (d) connecting all HEIs to GÉANT2 and setting up a National Research and Educational Network (NREN), providing them with dedicated high-quality connectivity. Under the same envelope of funding, the project delivered on additional activities in line with PDOs. The main achievements are (a) fiberization of 73 rural cellular towers and (b) provision of wiring works and dedicated connectivity to the internet and the NREN for 243 primary and secondary schools. Follow-up activities directly linked to the PAD’s original scope were also conducted, most notably through (a) the training of 200 additional beneficiaries in ICT skills and (b) provision of additional data hosting and cloud services via the NREN. These outputs are widely recognized by national authorities and EU partners to have been achieved with effective use of resources, notably by pooling in private capital, and to have tangibly contributed to Kosovo ’s development, bolstering the country’s digital infrastructure for citizens and institutions as well as its human capital and overall digital readiness. Project Development Objectives (PDOs) 7. The Project Development Objectives of KODE were v b q y h h‐ broadband services in project areas and to online knowledge sources, services and labor markets among citizens, and public and academic institutions. Although they were formulated jointly at appraisal, in this document “PDO 1” refers to “improve access to better quality and high-speed broadband services in project areas” and “PDO 2” refers to “improve access to online knowledge sources, services and labor markets among citizens, as well as public and academic institutions”. This decision is to simplify the ICR’s assessments and align with the structuring of activities, which followed separate objectives. Key Expected Outcomes and Outcome Indicators 8. The achievement of the PDO was measured by progress on four outcome indicators. The key monitoring indicators as defined in the PAD and after the restructurings are (a) people provided with access to the internet (number); (b) public institutions with access to high-speed internet through the project (percentage); (c) beneficiaries of improved HEIs access to high-speed broadband through the project, of which female (percentage); and (d) beneficiaries of activities in Component 2 who report being employed (percentage). Components 9. The KO v Af f f €20.7 , that is, US$21.7 million equivalent (at 2018 rates) through three components: • Component 1 - Digital inclusion (€15.3 million, that is, US$16.1 million equivalent) supported (a) the expansion of digital connectivity through co-financing the deployment of high-speed broadband connectivity in areas that have been identified as unserved or underserved and (b) improvement of the enabling environment for wireless broadband services, through the deployment of the NSMS. 2 GÉANT is a pan-European data net ork dedicated to the research and education community. It interconnects Europe’s National Research and Educ ation Networks (NRENs), providing a high-bandwidth, high-speed, and highly resilient backbone that supports over 50 million users across more than 10,000 institutions. The network is essential to Europe’s e-infrastructure strategy, supporting open science with an exascale-ready infrastructure and advanced networking services for trusted access. It offers the highest levels of capacity and security users need, here and hen they need it. GÉANT’s infrastructure is built on a dual-layer network comprising transmission and packet layers. The transmission layer utilizes a dark fiber core, illuminated by advanced optical transmission equipment like the Infinera DTN-X, capable of delivering 500G super-channels. The packet layer supports both Layer 2 and Layer 3 services, offering Ethernet connections and IP services through carrier- grade routers: https://network.geant.org/european/ Official Use Only Page 2 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT • Component 2 - Digital work and empowerment (€4.5 million, that is, US$4.7 million equivalent) supported (a) the Youth Online and Upward (YOU) program and increased access to knowledge, information, and online services and (b) set up of KREN and connection to GÉANT and support of online labor market information and services in project areas. • Component 3 - Project implementation support (€740 000, that is, US$776,250 equivalent) facilitated (a) the Project Implementation Unit (PIU)-KODE, including monitoring and evaluation (M&E) activities, training for PIU staff, fiduciary and safeguards, and project management functions and (b) conducting extensive household surveys to monitor progress and estimate development impact of project activities, including tracking the impact of the project investments on beneficiaries’ incomes, employment status, and poverty levels. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Date of Description of changes restructuring By March 2023, the project had already achieved significant milestones such as the completion of foreseen subcomponent 1.1. activities with significant savings, but under Subcomponent 2.1 only ~400 YOU beneficiaries had completed trainings, corresponding to half the interim target that was set for June 2023 (810 young people in total by the end of June 2023). Under Subcomponent 2.2, the KREN was already operational in March 2023 and offering cloud-based digital services to HEIs. The restructuring therefore included: • Reallocation of undisbursed funds, for a total of €10.5 million i.e. US$10.9 million equivalent March 23, (see Annex 6) to a new set activities aligned with the PDO: (i) the provision of dedicated high- 2023: speed internet access to 220 primary and secondary schools nationwide and installation of Restructuring internal network to the schools, (ii) the fiberization of cell towers with high-speed optical cable #1 internet, (iii) the scaling-up of the KREN and its services to schools, iv) the training of 200 additional beneficiaries under the YOU program. • Reallocation of funds: €5 million i.e. US$5.2 million equivalent from Category (2) ‘Grants under part 1.1 of the project’ to Category (1)‘Goods, orks, non-consulting services, training and incremental operating e penses for the project’ to finance the activities above. • Eleven-month extension of the project closing date from end of June 30, 2023, to end of May 31, 2024. • Adjustment of baselines and targets to reflect new closing dates and the reallocation of funds. The project introduced: • A seven month extension of the closing date from May 31, 2024 to December 31, 2024. The March 31, extension was made necessary by the delays encountered by trainings and during The second 2024: phase of the provision of dedicated high-speed internet to 120 out of 220 primary schools. Restructuring • A change in the outcome targets, introducing a new PDO-level indicator was added under #2 this second restructuring to reflect school connectivity activities not foreseen at PAD and introduced under the first restructuring December, 15, Partial cancellation of project funds: Acknowledging US$1.4 million equivalent was not disbursed as 2024: of December 15, 2024 (6.5 percent of total funds), the third restructuring requested the cancellation Restructuring of this amount from Category (1) of the project. #3 Revised PDOs and Outcome Targets Official Use Only Page 3 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT 10. The PDO remained unchanged throughout the project. Outcome targets were revised as mentioned above and in annex 6. Revised Components 11. As detailed earlier, in the dedicated outcome sections and in annex 6, the project reallocated funds to new activities for a total of around €7 million, around US$7 million equivalent. The two key new activities involved the fiberization of cellular towers and connectivity works in primary and secondary schools, complemented by other activities within the scope of the PAD, such as equipment acquisition for training centers and extension of existing trainings. Other Changes 12. Changes in intermediate results indicators by components were made (see section IV. A and annex 6). Rationale for Changes and Their Implication on the Original Theory of Change 13. Rationale for changes in indicators is explained in section IV. A and changes in activities are explained in outcome sections. As no fully-fledged theory of change was realized by appraisal, changes did not impact it. II. OUTCOME 14. Although the project underwent two restructurings and associated target revisions, a split rating is not deemed necessary as the project’s ambition and scope were sizably increased compared to the PAD. As articulated in section II, the PAD’s original scope of activities was maintained during implementation, with the addition of various key activities and corollary investments that expanded its reach mostly by using the resources liberated by the sizable savings in the fixed connectivity subsidy subcomponent. While the Results Framework did not adequately reflect the project’s expansion, the ICR quantifies this increase and assesses its outcome throughout this document. The ICR also argues that the scope and objectives of the original PAD activities were effectively achieved with no decrease in ambition, mobilizing own calculations and assessments beyond the Results Framework. A split rating is therefore not justified despite the downward revision of two key indicator targets (for Subcomponent 1.1 and Component 2) during the first restructuring, which did not stem from a reduction in the project’s ambition but rather from a readjustment of errors in the baseline and targets of the Results Framework. A. RELEVANCE OF PDO Assessment of Relevance of PDOs and Rating 15. Relevance of PDOs is rated as High. 16. The PDOs remain fully aligned with the ongoing Country Partnership Framework (CPF) for FY2023-20273, after alignment by design with the 2017-2021 Kosovo CPF. KODE’s development objectives were critical in tackling some of Kosovo’s key development challenges—poverty and social exclusion especially for women and youth, lack of decent jobs, and non-inclusive internet coverage and usage. In the 2023–2027 CPF, KODE’s achievements are either directly mentioned as a comparative strength of the country (notably PDO 1’s efforts on connectivity) or reiterated as a current objective, showcasing their ongoing relevance at exit of the project (the improvement of public service delivery most notably). ‘Good digital connectivity’ is hailed as one of the key strengths of the country to be leveraged for growth, reflecting both the project’s success in improving the country’s fundamentals and its recognized development impact. This 2023–2027 CPF acknowledges the achievements from KODE and elaborates on the rationale behind connectivity as an enabler of growth, requiring public interventions when economic spillovers far outweigh commercial returns. The project’s focus on demand- oriented IT skills training and awareness programs remains pertinent in the 2023–2027 CPF, as highlighted in objective 2.2 3Country Partnership Framework for Kosovo 2023-2027: https://documents.banquemondiale.org/fr/publication/documents- reports/documentdetail/099630307172332478/idu140cb495813f591474d1b5571a370b92b193f Official Use Only Page 4 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ‘Remove barriers to the economic empowerment of women and youth. The project also contributed to objective 2.2, the High-Level Outcome of ‘Increased environmental resilience’, and the underlying objective of EU accession. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 17. The overarching development objective of KODE is highlighted in section I.A. PDO 1: Improving Access to Better Quality and High-Speed Broadband Services in project Areas 18. The project was very effective at achieving PDO 1, by surpassing several targets both in filling coverage gaps as v f h y’ v . The project has met the targets of all PDO-level indicators and all intermediate indicators related to PDO 1. The project achieved its key outcome of coverage with high-speed broadband of the entirety of the country’s hite ones bet een 2018 and 2023 . A second key achievement was the successful creation of the NSMS, ensuring the monitoring of frequency bands and bolstering network security. Given the overweighing of the costs in the connectivity activity, once the white zone objective was achieved the project redirected funds to follow-up activities in line with the development objective, notably fiberizing 73 cellular towers and connecting 243 schools to reliable broadband. 19. The pilots led by MED in 2017 and 2018 (KO-Pilots) built upon analytical work led between 2014 and 20174 to fy 266 z ‘ h z ’, so called project areas (see figure 2). These white zones, as defined by the EU, are areas where no fixed ultrafast network (at least 100 Mbps) is present or credibly planned in the relevant time horizon. In Kosovo and for the project, the white areas had no fixed last mile connectivity, with only spotty wireless connections. Completed by 2018, four connectivity pilots, composed in a total of 14 different tendering lots, succeeded in providing broadband access to 43 villages/cadastral zones qualifying as white zones, for a total of ~3,014 households at the time of application (See Annex 6).i A total of 76 schools and 27 health institutions were also connected and provided with free connectivity for 5 years. Given the success of these pilots, the tendering process and contractual responsibilities of the scheme, detailed below, were replicated for the main project. 20. The project followed up by completing the connection of white zones of the country to at least 100 Mbps connectivity (under Subcomponent 1.1), covering 4,474 households and 164 cadastral zones, achieving the spirit of the objective set out in the PAD, while falling short of the previously calculated target of connecting 266 cadastral zones and 10,544 households directly via the project.5 The difference of 6,070 households between the stated objective at appraisal and the final amount nominally connected via the grants is mostly due to (a) the non-project-financed pilots having covered about 27 percent of total households estimated at appraisal (3,014 based on grant application data), but also (b) the number of total households having decreased by the time all grant applications were executed (estimate of around 657 households in the 164 zones covered by the project); and (c) around 2,546 households in some of the more accessible and economically profitable white areas having been connected independently by providers before subsidies were extended to their cadastral area, which could be attributed to a success of the MED pilots in mobilizing Internet Service Providers (ISPs) to solve rural connectivity. This difference for (a) is to be considered a reporting and goal setting oversight and not a failure of efficacy, for (b) it could be argued that population flight should have been taken into account in the ERR/NPV calculations, yet the inclusion objective towards emigration-prone areas as inherent in the project’s PDOs, while for (c) it is to be considered a success of the successive lot design of the project, in the form of capital mobilization. The exact list of areas was also adapted following field visits, with 49 cadastral zones for 572 households by the 2011 census identified to have no inhabitants by the time of grant application starting in 2019, and 22 additional white 4notably the Innovative and Green Growth for Rural Kosovo Technical Assistance (TA), and Kosovo Digital Economy TA, 5 The results frame ork’s PDO-level indicator, as detailed in Anne 6, is not deemed appropriate to measure the project’s impact as it relates to overall fixed internet users in the country, without a clearly attributable objective. The PAD on the other hand details a clear objective of 10,544 households which is deemed valid for efficacy considerations, as it stems from census data at the time of appraisal and corresponds to the inhabitants in the targeted white zones. Official Use Only Page 5 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT areas being identified between appraisal and implementation. The data are detailed in annex 6. The connection of the last white area village to broadband was inaugurated in March 2023, eradicating connectivity deserts in the country as shown in the figure below. FIGURE 2: EVOLUTION OF WHITE AREAS BETWEEN 2018 AND 2022 21. The project managed to successfully attract private investments by using a private design-build-operate (DBO) subsidy model, achieved through 16 rounds of tenders, 59 grant agreements, and 21 service providers. The project’s approach of using competitive matching grants to attract private investment in the ‘ hite ones’ involved selecting ISPs based on least-subsidy reverse auction modality to build, manage, and commercially exploit publicly co-financed broadband networks, aiming for speeds of at least 100 Mbps, upgradable to 1 Gbps in public institutions. The project’s design was technology neutral in its procurement process yet given its requirements the infrastructure rolled out was entirely linear fiber infrastructure, which ensured that the PDO’s focus on quality beyond speed as met fiber’s reliability, redundant and stable physical connections can support high data throughput with low latency and minimal service interruptions. For the first 11 calls the matching public grant-private investment ratio was of 1 to 1, with a further readjustment to 4 to 1 grant-private capital for the remaining 5 calls, as the project reached more remote areas requiring more support to reach. The project’s approach involved a arding contracts in lots, each designed to group several cadastral zones. The significant number of lots and tender calls allowed for more granularity, a progressive mobilization of competition and adjustments in the list of final areas to target. Operators were required to meet several obligations aligned ith the EU’s approach to regulating publicly co-funded broadband projects, such as open access to infrastructure. 22. Subcomponent 1.1. also achieved the connection to broadband of public institutions in project areas, totaling 76 schools and 27 health institutions. To maximize the project’s social impact and ensure equitable access to digital technologies, the MED required ISPs via the tender processes to provide free high-speed internet access to schools and healthcare facilities for five years. The project also effectively realized technical capacity building efforts for ARKEP and MED, contributing to strategy setting efforts and being followed up by additional demands for market analyses for ARKEP. 23. The project successfully provided access to quality fixed broadband in project areas, contributing to increased uptake, but without addressing broader usage objectives. The improvement in the quality of access is entirely attributable to the project, allowing for future-proof, stable participation in the digital economy and society compared to non-fixed, low-speed and unreliable connectivity present, only in some contexts, prior to the project. As of 2024, ISPs are reporting amounts of fixed connectivity subscriptions in these areas equivalent to around 95% of total households, Official Use Only Page 6 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT suggesting uptake rates just short of the country (fixed internet penetration rates per households standing at 129 percent by the end of 2023), whereas virtually no subscription was present pre-project (see annex 6).6 24. Under Subcomponent 1.2, capacity building and technical assistance actions have been carried out to improve the regulatory and legislative environment. On top of such efforts was the successful creation of the NSMS—which includes four fixed monitoring stations, mobile monitoring station, and relevant equipment for ARKEP—ensuring the monitoring of frequency bands and bolstering network security. The NSMS supports various critical sectors, including national security, public order, emergency services, border protection, civil aviation, railway safety, and audio-visual media. This main station is also backed with other fixed and mobile ones. The NSMS enhances the efficiency and security of spectrum usage, ensuring that frequencies are used optimally and protected from unauthorized interference. This contributes to the overall stability and reliability of communication networks. 25. Following the first restructuring, several additional activities were introduced toward the achievement of PDO 1 and conducted successfully. One key initiative was fiberizing 73 mobile phone towers to high-speed optical fiber (previously connected via microwave) to enhance mobile broadband services in rural areas, complementing the fixed connectivity efforts by improving bandwidth, lowering latency, and supporting the demands of 5G networks. The economic benefits directly spill over onto mobile subscribers in targeted areas, and the grant scheme was executed with conditions and awarding mechanism similar to the fixed connectivity efforts. The other significant activity involved providing internal wiring works and high-speed connectivity to 243 schools in around 30 different municipalities, extending access to KREN and its cloud-based services. Many of these lacked internal networks or had makeshift ad hoc networks. Stemming from discussions with the Ministry of Education to support implementation of its Education Strategy, the schools ere selected using the ministry’s database, conversations ith municipalities, and field visits. While the administration uses those services actively and various examples of teachers already using these tools to shoulder classes have been reported, the work established the necessary foundation for at-scale digitalization of education and teaching of digital skills, to be followed up with digital content, teacher training, and dedicated curricula. PDO 2. Improving access to online knowledge, services and labor markets for citizens, public sector, academia 27. The project was substantially successful in working towards PDO 2, meeting most and in several cases exceeding targets. While the project effectively helped the country’s educational sector overcome a long-standing infrastructure and service gap and set up a blueprint for upskilling and job insertion programs, it showcased two key shortcomings related to the inclusion objectives of the YOU activity and the failure in ensuring post-project sustainability plans for KREN. 26. Under Subcomponent 2.1, the project was successful in the roll-out of the YOU program, training 200 students more than the foreseen 2,000 in high-demand digital skills. Adapted from the successful Women in online Work (WoW) pilots, the trainings provided extensive focus on ICT and web-oriented skills, with a practically-oriented teaching methodology. After conducting scoping efforts with the private sector to identify the skills most in shortage in the labor market, 16 different providers were selected to deliver 22 packages of in-person trainings, with 16 learning modules7 in seven different centers in the country. Training locations, by design, did not overlap with the geographical locations of fixed broadband connectivity schemes. Complemented by 35 hours of soft skills training, the training programs necessitated progress reports and final examinations for obtaining a completion certificate. More than 17,000 applications ere received across the different packages of trainings, revealing the success in outreach and the public’s high interest for digital skills training and the project. 27. The project achieved high levels of satisfaction and indicates promising labor market results. Two post-training surveys were conducted on top of qualitative interviews and focus groups with participants, the first of which being an 6 As of 2024, ISPs active in project areas reported an amount of subscriptions totaling 4239. These concern fixed subscriptions specifically. 7 Going from 100 hours for QA manual testing to 265 hours for the cybersecurity and network administration modules. Official Use Only Page 7 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT exit survey reporting satisfaction rates of 90% and a positive evaluation of the skills gained by 94% of participants.8 The second survey, initiated as part of the ICR process and focusing on labor outcomes as per its later realization, corroborated these findings by reporting a satisfaction rate of 92 percent (see annex 6 for further details).9 More importantly, this second survey highlighted encouraging outcomes regarding to labor results, with a post-project employment rate of 38 percent (compared to 26% pre-training), 48 percent of participants stating that the training has improved their employability, 17 percent of participants stating they found employment at least in part thanks to the training and 90 percent of those employed reporting being so by a company with active operations in Kosovo10. These results matching initial targets related to satisfaction and provide encouragement for YOU’s impact on labor supply, especially given that a sizeable proportion of the graduates then proceeded to return to further schooling (around 31% of participants reported being students post-training). A crucial further element to highlight is the geographical distribution of the students, as the training focused on injecting ICT skills in areas outside of the capital city, currently concentrating the vast majority of ICT jobs and skills. Further details on this final survey‘s results are provided in anne 6. 28. h ’ , h b h h YOU related to youth targeting as well as the age and gender of participants, and, to a minor degree, the level of precariousness of participants. Rather than a failure in efficacy, this was due to deliberate decisions by the PIU and the Government to avoid positive discrimination in the selection of candidates. For instance, it was decided not to conduct specific actions to promote the YOU program to women, gender participation quotas were not put in place, and access to the program was not restricted to the 18–35 age range and specific educational attainment levels. The rationale pursued by the Government was to maintain a meritocratic selection process, basing the selection exclusively on motivation interviews, curriculum, and an entry test. This choice resulted in the gender objective not being met, with 44 percent of women graduating opposed to the initial objective of 50 percent. Similarly, 36 percent of graduates were beyond the 18– 35 age range, and only 31 percent reported themselves as unemployed (38 percent as employed and 31 percent as bachelor students). 29. Under Subcomponent 2.2, KREN was created and equipped with infrastructure and services beyond the perimeter foreseen at appraisal. KREN improved the reliability, speed, and cost of HEIs’ connectivity, allo ing closer cooperation among national and regional institutions and preparing for data-intensive research usage as well as providing access to corollary services. All Kosovo’s HEIs, over 31 campuses, ere successfully connected to a dedicated 7,000 Gbps internet connection, KREN, and the pan-European net ork. By increasing the band idth of HEIs’ internet access and providing access to a dedicated network (within Kosovo and outside, via access to GÉANT) independent of external demand, KREN facilitates research collaboration and knowledge exchange within Kosovo and internationally, making it faster and capable of withstanding some of the more consuming real-time collaborations. Additionally, KREN also provides key data hosting services for staff and students in digital delivery of education, having created a dedicated data center and provided Microsoft 365 licenses, Moodle, and similar cooperation tools. Through GÉANT, HEIs also benefit from reduced costs on IT services via collective bargaining and access to leading academic journals and databases, thereby enhancing research quality at a low cost.11 30. Th x f KR N’ v y y h y y h h v y ff achieved in Subcomponent 1.1, providing essential tools to institutions and fulfilling a crucial step toward a more 8Based on results of 307 self-completed exit interviews of YOU trainees who successfully completed trainings during November 2022-May 2023. 9 Calculations were made out of the latest post training survey on employment status and opportunities by PIU. The satisfaction rate was calculated based on the number of people that responded “slightly effective” to “e tremely effective” (358 people) at the question “To hat e tent do you feel the training has improved your employability?” out of total number of people surveyed (385 people). 11 At the time of writing, 16 HEIs are actively using KREN data center resources, with 5 of them relying almost entirely on KREN’s platforms, eb, and data hosting services. A total of 1.06 petabyte of data were transmitted in 2024, indicating substantial reliance and usage on connectivity services. Active usage rate of licenses appears to be limited, with MS365 licenses having only around 750 active users out of the 41,000 total potential licensed users. This implies follow-up efforts to integrate these services in education and research delivery remain necessary. Official Use Only Page 8 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT holistic digitalization of educational delivery. While little data is available on beneficiary schools’ present utili ation rate of KREN’s suite of services (from connectivity to Moodle), they act as a prerequisite for highly transformative practices such as advanced communication platforms, collaboration and e-learning software, and access to modern documentation. Follow-up work—including connecting the remaining schools to the internet, training personnel, and creating digitally adapted curricula—would allow to fully capitalize on the investment. 31. A shortcoming in the project’ ff v , despite the success in delivering a NREN matching an evolving demand from the client, is the failure in establishing and rolling out a post-project sustainability plan for KREN, leaving the achievements of the project at risk of subsiding. Despite concerns being raised repeatedly, KREN and its staff remained under the project’s payroll, ithout a solution regarding legal status and recurring funding sources being chosen or implemented. At the time of writing, a tentative legal status and governance model has been identified, yet its rollout still has to go through. The risks mainly concern the departure of hard-to-replace staff, who are essential for the daily functioning of KREN’s data center and resources, as ell as covering other recurring costs. 32. The last activity foreseen under PDO 2, the provision of a digital awareness campaign, was executed successfully, as were the additional activities added to the appraisal perimeter, notably a cyber awareness campaign in schools and the provision of hardware to three YOU training centers. Justification of Overall Efficacy Rating 33. The efficacy is rated as Substantial. 34. The project achieved the totality of its outcome indicators and intermediate ‘N b f h h v v y’ f O1 ‘N b ff 18–35 v h h YOU ’ (See annex 6). While, as detailed in section IV.A, several mistakes in the monitoring system were observed, the efficacy of the project in pursuing the underlying objectives of its activities has been argued for in the dedicated sections. 35. Going beyond indicators, the project achieved significant advancements for Kosovo that would likely not have occurred otherwise, overcoming key market failures and compensating a long-standing infrastructure gap compared to regional and European peers. The project increased the quality and coverage in target areas of fixed and mobile connectivity, improved the governance of mobile connectivity, increased supply of high-demand skills, and bolstered the infrastructure and digital services of public institutions in target areas as well as the entire higher education system. The shortcomings regarding efficiency, which relate to inclusion outcomes of trainings and timeliness in ensuring sustainability plans of KREN, are minor compared to the project’s achievements as outlined and beyond the PAD. C. EFFICIENCY Assessment of Efficiency and Rating 36. The overall efficiency is rated as High. The project maintained reasonable costs throughout its activities, both compared to appraisal and regional benchmarks. It adhered to deadlines with minor delays, while adjusting to procurement guidelines and adapting to unforeseen delays. 37. A h f , h v v N V f €3.8 , h , U $3.9 million equivalent and an ERR of 35 percent. With a significantly higher ERR than the social discount rate (SDR) of 7.3 percent, the project’s perimeter at appraisal delivers more value to society than the opportunity cost of resources spent. 38. The project achieved all activities foreseen at appraisal, completed additional activities with the same funds, and achieved €1.3 million, that is, US$1.4 million equivalent in final savings, implying it surpassed the originally estimated ERR. Most activities were completed within expected costs, with minor deviations. The savings under Subcomponent 1.1 are attributed to both erroneous target calculations and the project’s efficiency in attracting private investments beyond the nominal amount set in the grant agreements. Cost increases totaling €738,000, that is, US$767,420 equivalent ere largely offset by other savings beyond Subcomponent 1.1, totaling €546,000, that is, Official Use Only Page 9 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT US$567,000 equivalent. This demonstrates success in efficiency, especially considering project extensions totaling 18 months, causing additional implementation and running costs and contractual penalties linked to unavoidable delays. Further detail on the costs is provided in Annex 6 and the a reviewed economic analysis is provided in Annex 4. 39. Narrowing down on the most expensive activity at appraisal, the fixed connectivity grant scheme maintained reasonable costs despite the de facto narrowing of the scope to smaller, more mountainous regions (see annex 6). At PAD, the project as estimated to have a cost per connected cadastral one of €44,436 i.e. US$ 46,207 equivalent and a cost per household of €1,121 i.e. US$ 1,165 equivalent. The final project’s outputs sho cased significantly lo er average costs, €19,442 i.e. US$20,391 and a cost per citizen of €166 i.e. US$174, despite a sizeable decrease in the population living in the cadastral zones compared to 2011 census data and a greater subsidy-to-private investment nominal ration in the grants. This is notably due to the competitive dynamic created by the broad participation of service providers. Compared to 2018 pilots, the project maintained acceptable cost ratios, even in sparsely populated areas, incurring higher costs per household. Additionally, 2,546 households by 2011 census data in various cadastral areas were connected independently by ISPs without direct intervention but attributable to crowd-in from public investments, further lowering the cost ratios. 40. Th ff y v b y h A , €7 million, that is, US$7.2 million equivalent, while remaining within reasonable costs and benchmarks . The two key additional activities included the fiberi ation of 73 cellular to ers (€3.3 million, that is, US$3.4 million equivalent) and dedicated iring and connectivity orks for primary and secondary schools (€2.9 million, that is, US$3 million equivalent). The cost per fiberi ed to er as €45,622, that is, US$47,440 equivalent, and per connected school as €11,999, that is, US$12,477 equivalent, both of which are considered efficient given regional averages and the challenging terrain. 41. Realistic planning and efficient project management from the PIU helped maintain expected timelines for key activities, except for YOU trainings, and mitigated delays in follow-up activities. Operating nimbly and drawing from previous pilots and technical assistances, the project achieved most of its intended outputs well within the lifespan of the project, requiring time extensions only due to COVID-19 delays affecting YOU trainings and the progressive increase in ambition of follow-up activities, utilizing freed-up funds from Subcomponent 1.1 In the face of the challenges posed by COVID-19, the project successfully adapted by conducting activities parallelly in a reduced time frame, for instance, by implementing remote work protocols and leveraging digital tools to maintain project momentum and ensure timely completion. In the additional activities, the project quickly identified areas of intervention and appropriate design of measures, encountering only minor delays related to procurement in a context of low availability of qualified vendors. Furthermore, the project adhered to rigorous procurement practices while ensuring reasonable use of supervisory resources. D. JUSTIFICATION OF OVERALL OUTCOME RATING 42. The overall outcome is rated Satisfactory. The relevance of PDO is rated as High, the efficacy is rated as Substantial and the efficiency is rated as High. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 43. Classified as gender informed, the project considered gender in the applicable subcomponents and set gender objectives in its design, but chose not to prioritize them over other principles in the key gender-informed activity. The project built the skills development programs under YOU specifically targeting women to improve employability and bridge the gender gap in access to and use of digital technologies, integrating a gender representation objective in its selection of students. Yet, the project did not achieve said objectives as the PIU and the Government deliberately chose to select participants among applicants in a more merit-based fashion, not integrating gender among its metrics. No dedicated efforts were put in place for gender-oriented outreach under YOU, nor were satisfaction and labor outcomes disaggregated by gender in the monitoring efforts. No gender considerations were put forward in other activities prone Official Use Only Page 10 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT to have them, such as the awareness campaigns and the cyber education campaign conducted in schools. The other activities financed via the project are gender-agnostic in their design, comprising connectivity infrastructure, governance reform, or connection of educational institutions, but have direct and concrete positive gender spillovers. Finally, the project has put in place a gender impact monitoring mechanism, with a gender-disaggregated analysis of one of the four PDO indicators and one of the 10 intermediate results. Institutional Strengthening 44. The project supported and achieved institutional strengthening at the regional, national, and targeted local levels. At the national and local levels, the project contributed to bolstering institutional development directly and indirectly. Regionally, the project has strengthened institutional development at the regional level notably by connecting HEIs to GÉANT, allowing for international cooperation among institutions. Further, the running of the project allowed MED to acquire new expertise in subsidy models, training program setup, and World Bank-aligned procurement processes. Mobilizing Private Sector Financing 45. The project was successful in mobilizing private capital mainly via Subcomponent 1.1. The project achieved a total nominal amount of private investment of €2.2 million, that is, US$2.3 million equivalent via the tender agreements. To this, a sizable amount estimated by the PIU at €10 million is to be added in the form of follow-up investments beyond the subsidy’s contractual agreements. This amount comprises (a) follow-up last mile connections to subscribing households, (b) independently connected areas for 3,014 households with KO-Pilots (see annex 6), (c) free subscriptions to public institutions. More generally, the project’s achievements, namely those on spectrum governance reform and connectivity, have the potential to act as enablers for private sector investment in the economy. Some of that potential is already materializing, with various positive reports of connectivity enabling the growth of tourism in rural areas. Poverty Reduction and Shared Prosperity 46. The project contributed directly to the twin goals of ending extreme poverty and boosting shared prosperity, most clearly by providing trainings to disadvantaged youth and increasing access to high-quality connectivity in disadvantaged areas. The project closed access gaps, helping more rural communities connect to high-quality fiber networks, notably in more remote communities that are among the most disadvantaged in the country. Research suggests a strong relationship between access to broadband services and economic growth. 12 The project also acted to train internet users in those areas for productive usage of the internet. The YOU program, by targeting untrained and unemployed youth, is expected to have a direct positive impact on poverty, enhancing occupational outcomes of these target groups and contributing more broadly to prosperity by improving human capital and supply of trained workforce to the local economy. Initial results of ex post surveys suggest encouraging results for these assumptions. Other Unintended Outcomes and Impacts 47. The project contributed to increasing resilience and adaptation capacity to current and anticipated vulnerabilities posed by climate change. This was notably achieved by eradicating white areas of connectivity and increasing societal digital skills, which allow for greater responsiveness to exogenous climate shocks and long-term implementation of digitally enabled adaptation instruments. Other subcomponents of the project, such as institutional capacity building and infrastructure development, also contributed indirectly to enable climate change adaptation capabilities. 12For example, Röller and Waverman (2001) established a clear causal relationship between broadband penetration and gross domestic product (GDP) growth, estimating that a 10 percent increase in broadband access could result in a 2.8 percent increase in GDP. Official Use Only Page 11 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 48. The project design benefited from pilots, technical assistance work and regional best practices, targeting identified priorities and appropriately engaging stakeholders. Consultations conducted for both components, involving vendors, employers, public institutions and citizens in project areas served to tailor activities and their design to the context, as well as to mobilize participants in tenders and beneficiary awareness. This effort facilitated the fluid leveraging of private capital and expertise and ensured the timely execution of administrative processes, in a project that by design aimed to shift responsibilities to private actors. 49. The project benefited from strong commitment and support of MED and Kosovo’s Government. Such political backing and borrower ownership contributed to the project’s stability and resilience, with highly motivated staff. MED, specifically the ICT department under which the PIU was established, played a central role in the strategic direction and technical supervision of the project. MED demonstrated commitment to the objectives by allocating budgetary resources for piloting and preparing before World Bank financing (see section V). 50. The Project design could have further integrated considerations around access obstacles beyond broadband and productive digital usage and should have included a more robust monitoring system. For Subcomponent 1.1, the preparation phase notably ignored pilots in its cost assessments, a major factor behind the eventual redistribution of funds from the subsidy scheme to other activities not foreseen at design. Other factors explaining this divergence could have been variably anticipated, from levels of competition in the tendering process to progressive decrease in population. Access to devices and affordability of subscriptions, key obstacles to internet access in rural areas, were not tackled by the project design. Digital literacy was accounted for in the project, with a minor digital awareness campaign being integrated into the design, but preliminary data indicate that as a major obstacle to productive use of the internet by citizens and institutions, and therefore to solving the inclusion objectives inherent in the PDOs. The monitoring system was flawed, as detailed in section IV.A, which hindered the capacity to fine-tune activities’ implementation modalities. B. KEY FACTORS DURING IMPLEMENTATION 51. Project management was effective, quickly developing capacity and following best principles of transparency and procurement practices. The PIU took advantage of the dedicated capacity-building efforts in the project, adopting World Bank guidelines and publishing orderly, well-documented tenders. It conducted diversified activities, showcasing adaptability. It systematically uploaded documentation online, consulted with citizens, published notices in project areas, and animated the grievance redress mechanism (GRM). The PIU maintained stable communication and collaboration with ministerial counterparts, ensuring political buy-in throughout the project. It coordinated and consulted various vendors for infrastructure projects and consulting services, local authorities, and heads of HEIs. 52. The project demonstrated a strong capacity to adapt and reassess throughout its implementation, particularly in response to the challenges posed by the COVID-19 pandemic and the change in minister. The pandemic necessitated significant adjustments, including the postponement of the YOU program’s launch (Subcomponent 2.1) and the shift to online tools for citizen engagement (CE) and training activities. Despite these challenges, the project maintained progress, particularly in the establishment and operationalization of KREN and the connectivity subsidy scheme. 53. The project promptly reinvested funds liberated from Subcomponent 1.1, quickly identifying activities and putting them into execution. This reveals significant technical competence from the PIU and an effective cooperation with ministerial counterparts regarding governmental priorities. The PIU successfully and progressively implemented ~€7 million (that is, US$7.3 million) of follow-up activities after conducting efficient preliminary analytics on expected costs and needs. The project efficiently communicated and coordinated with existing and new stakeholders (that is, primary schools and the Ministry of Education) to ensure prompt delivery and prepare usage of investments, also compensating for lack of capacity in the Ministry of Education. Official Use Only Page 12 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 54. The project showcased several flaws in the design of the M&E framework, with several outcome indicators being either ill-calculated, failing to track improvements attributable to the project, or not being an accurate metric for progress toward PDO. The Results Framework is further detailed in annex 6. Beyond the Results Framework and based on guidelines submitted by the World Bank, the PIU financed various surveys among beneficiaries under Subcomponent 1.1, a beneficiary survey and focus groups among beneficiaries of the YOU program, and a series of surveys and focus groups with universities and colleges under Subcomponent 2. Their quality and effectiveness is deemed mediocre, as discussed below, and the PIU relied mostly on field visits, direct consultations with ISPs, and beneficiaries to verify the project’s effectiveness. 55. Despite revisions, the Results Framework could not enable a relevant monitoring of progress toward the outcome targets of PDO 1, both for activities foreseen before and during appraisal . The key outcome indicator, ‘to improve access to better quality and high-speed broadband services in project areas’, as revised but at no point accurately reflected the project’s impact, given that it as calculated on overall subscriptions to fi ed broadband in the country as opposed to an actual estimate of new subscriptions in project areas. Cadastral zones connected were never integrated in the targets and nor was quality verification. One outcome ‘public institutions with access to high-speed internet through the project’ and t o intermediate indicators (‘healthcare/educational institutions with access to high- speed broadband internet in project areas’) ould have been tracked more precisely in a numerical format. Follo -up activities were also poorly monitored, with connection of cellular towers never added to the Results Framework and the ‘number of schools connected ith improved digital connectivity’ only included during the second restructuring. 56. The Results Framework also had shortcomings in evaluating progress toward the outcome targets of PDO 2. For PDO 2, hich aims to ‘improve access to online kno ledge, services, and labor markets for citi ens, the public sector, and academia’, the first outcome target, ‘beneficiaries of improved HEI access to high-speed broadband through the project’ was adjusted from 80 percent to 85 percent in the second restructuring. However, since all HEIs have access to high-speed broadband connectivity through the project, and all students and teachers of the HEIs have access to KREN and its services, the target was ultimately surpassed (100 percent). This suggests that the indicator was not well defined. Similarly, the sub- indicator ‘of hich female’ as initially set at 80 percent but as revised to 60 percent in the first restructuring paper to reflect the actual number of female university students (total number of university students is 95,335, of which 55,783 are female). However, this indicator was also ill-defined, as 100 percent of women in HEIs have access to the internet, leading to a mismatch in the definition and revision of this target, which hindered effective monitoring. The only relevant indicator for monitoring progress as the last one for PDO 2, ‘percentage of beneficiaries of activities in Component 2 ho report being employed’, hich as adjusted up ard after each restructuring to reflect progress in that activity. Intermediate indicators are deemed to be appropriate. 57. The intermediate results indicators were globally well designed to monitor progress by component , barring the ‘ f h h h- b b ’( x 6). For instance, the target for health care institutions with access to high-speed broadband internet was set at 100 percent, as opposed to being a numerical target, with 38 institutions in the PAD and 43 in the last Aide Memoire. The same applies to the target for education institutions, with 129 in the PAD and 105 in the last Aide Memoire. M&E Implementation 58. Despite the three restructurings, the project did not proceed to correct the mistakes integrated into the M&E design and did not promptly integrate follow-up activities outside the scope foreseen at appraisal. Only one indicator was added to the results framework, noting the "number of schools connected with improved digital connectivity", but Official Use Only Page 13 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT having been added only during the second restructuring far after the activity had been initiated. Other activities, such as the connection of cellular towers or the acquisition of equipment to institutions, were never tracked. 59. The methodology for data collection and analysis was lackluster. Taking Component 1 as an example, Subcomponent 1.1, an annual survey was conducted by KANTAR, an international market research company. However, KANTAR's work focused on an outdated list of cadastral zones and calculated a baseline at a time when several subsidies had already been rolled out. Furthermore, it focused on satisfaction regarding internet connectivity rather than assessing the connectivity per technology types, failing to distinguish between fixed and mobile internet in particular. The project’s main benefits, centered around the improvement in quality of connectivity subscriptions, was not adequately monitored in its efficacy nor eventual impact. Similarly, the usage of connectivity by connected public institutions was not followed directly by Kantar. Consequently, the reports lacked key insights for the project. For Component 2, initial surveys failed at providing a comprehensive picture of efficacy, with a need for more precision on the type of contract and sector of activity. 60. For Subcomponent 2.1, the ex post training survey assessed participant satisfaction with the courses but failed to reflect the impact of the training on job opportunities. Without collecting structured labor information at entry and not following up with ex post data collection on employment, the attributability of job opportunities to the specific training received could not be estimated. Other analyses, such as the breakdown of the location of hiring companies and the effectiveness by training center, could also not be calculated until follow-up surveys conducted under this ICR. Similarly, for Subcomponent 2.2, highlighting the internet availability in public institutions is necessary but not enough to understand the impact of the project. A more meaningful approach would have been to assess the number of actual connections relative to the total population and to disaggregate this data based on the effective utilization of the internet, rather than relying on theoretical metrics. M&E Utilization 61. The utilization of M&E by the project, through the data collected, has only partially informed decision-making and ensured data quality. The M&E system could have been more effective in alerting project managers to discrepancies between forecasts and actual outcomes as well as in proposing corrective measures and orienting follow-up activities. Feedback collected was still used to inform activities, for instance, with data collected on applications to specific YOU trainings, which allowed for further iterations of the most in-demand trainings. Yet, awareness-building activities in Subcomponent 1.1 areas were not reiterated nor strengthened, despite being realized well below the budget foreseen at PAD and lackluster data showcasing improvement in access to online services in project areas.13 Justification of Overall Rating of Quality of M&E 62. The overall rating of quality of M&E is Modest. 63. Several activities of the project were ill-tracked in the monitoring framework design and loosely followed by h ’ h , h h h ’ v h v complications. While key objectives were delivered and many of them were followed outside of the Results Framework, notably with field verifications of connectivity and consultations, the flaws in the design could have hampered readjustments in cases of ineffectiveness. The project did not correct these mistakes and did not use the Results Framework to orient its decision-making and follow-up activities.14 Follow-up activities to encourage and facilitate access to devices and subscriptions could have been put in place in Subcomponent 1.1 project areas. Lack of rigor in training surveys diminishes the available inputs for further activities in the future. 13 Similarly, information on gaps to internet use relating to minority groups were not targeted directly by the project, despite being highlighted by KANTAR’s ork. The PIU used sources outside of the formal M&E and Results Framework roll out adaptations. 14 For instance, use of software tools delivered to HEIs could have been encouraged with trainings or communication activities had it been better monitored. Official Use Only Page 14 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 64. Th ’ f y Satisfactory, with high compliance observed in procurement and financial management (FM). The project’s FM risk as considered substantial at appraisal, mostly due to limited professional capacity and gaps in implementation. Despite having no experience with World Bank procedures and policies, with the exception of the Procurement Specialist, the PIU quickly adapted its processes and rules to the new obligations and aligned itself with best practices on reporting, disbursement, budget adequacy, and internal control effectiveness. Audit reports were submitted on time with unmodified opinions, and the budget planning process was effective. The project’s procurement performance as consistently rated as highly satisfactory, adhering to the World Bank’s Procurement Guidelines and utilizing the STEP15 system for tracking procurement activities. Despite the high fiduciary risk, mitigation measures and careful procurement planning led to successful processes and generated savings. 65. The project implemented environmental and social safeguards satisfactorily, classified as Category B due to civil works for broadband infrastructure installation. An Environmental and Social Management Framework (ESMF) was developed to ensure compliance, guiding the assessment and management of environmental and social issues. The project triggered the Environmental Assessment (OP/BP 4.01) but not the Natural Habitats Policy (OP/BP 4.04) or Physical Cultural Resources Policy (OP/BP 4.11). The ESMF allowed only Category B activities, excluding Category A projects and those in critical habitats. Public consultations were held, and the ESMF and Environmental and Social Management Plan (ESMP) checklists and site-specific ESMPs were disclosed for public comments. The project complied with all applicable safeguard policies, ith no aivers from World Bank’s safeguards or environmental and social/fiduciary policies and procedures. C. BANK PERFORMANCE Quality at entry 66. The World Bank helped prepare a project that was technically sound and tackled long-standing national sectoral priorities, consistent with the World Bank and regional development objectives . The project’s design incorporated lessons learned from international engagements, previous technical assistances and pilots led in the country, and regional best practices. When necessary, international consultants and dedicated capacity-building efforts were provided to further orient the allocation of resources and advise on technical standards. The World Bank effectively prioritized quick and efficient design, while aiming to shift risks and costs toward non-public actors. The project ensured real government ownership. 67. Fiduciary, implementation arrangements, and institutional aspects were assessed thoroughly and tackled preemptively, contributing to readiness for implementation upon approval. Needs for institutional FM, audit, and supervision staff functions were adequately developed and accounted for in the economic arrangement as well as the regional and country context at design. Risk assessment and mitigation measures were effectively identified. Institutional capacity was estimated carefully and addressed via dedicated capacity building, hiring, and awareness efforts. Cooperation among stakeholders was also facilitated by the World Bank. 68. As discussed earlier, the World Bank failed in helping counterparts design a sound M&E system, and cost estimates for Subcomponent 1.1 appeared off the mark retrospectively. Quality of Supervision 69. The project benefited from regular and timely supervision support from the World Bank, shouldering an autonomous PIU. Implementation support missions and key task team members, including colleagues in the Country Management Unit, monitored progress and provided e tensive support to the PIU. The quality of the World Bank’s implementation support was highly appreciated, with agencies highlighting responsiveness of the team to the challenges that arose during the implementation. Through the change of one Task Team Leader (TTL), the team ensured successful 15 Systematic Tracking of Exchange in Procurement. Official Use Only Page 15 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT onboarding of the new manager and assisted in keeping informed the new ministerial counterpart, who took the helm of MED in 2021. 70. The project team also provided sizable capacity building in procurement, FM, and environmental and social framework to the implementing agencies. The World Bank mobilized a wide range of technical expertise to support the clients and find practical solutions to complex technical challenges. The task team also provided extensive information on the progress related to gender, environmental, and social safeguards through its supervisory role during implementation. Justification of Overall Rating of Bank Performance 71. The rating of overall World Bank performance is Satisfactory. While some fla s ere noted in the World Bank’s M&E efforts as well as in the definition of targets and costs at design, the World Bank fulfilled its duties appropriately, steering successful preparation and providing crucial guidance throughout the project. The World Bank’s e pertise, processes, and collaboration with the PIU ensured effective delivery. D. RISK TO DEVELOPMENT OUTCOME 72. The risk to development outcome is Medium. The project’s achievements are mostly immune to future risks because connectivity investments under Subcomponent 1.1 are owned and operated by private operators, and there are low costs and strong governmental buy-in for upkeeping developments such as the NSMS. Yet some of the project’s impact risks being eroded due to changes in demographics, effective infrastructure utilization, and interministerial coordination. For instance, progressive exodus from rural and mountainous regions risks diminishing the development impact of fixed and mobile connectivity developments. KREN’s path ay to transition to post-project funding and legal status has not been secured yet at the time of completion, implying concrete risks of losing hard-to-replace staff. Without follow-up investments and efforts to roll out digital education, the dedicated connectivity works installed would yield little results and may risk not being renewed at the end of the five-year contractual clause for internet provision. V. LESSONS AND RECOMMENDATIONS 73. Nurtured through preliminary pilots and dedicated technical assistances, strong political buy-in at entry of the project was a key factor in its successful delivery, yet lost some momentum mid-way through the project. The technical assistances, pilots, and consultations mobilized in the lead-up and the strong integration of KODE in national strategies ensured political backing and legitimacy for the PIU’s actions. Strong client demand for activities, from the NSMS to white areas connectivity, ensured swift implementation. Alongside a very effective cooperation with Bank teams, this allowed for very effective scoping and implementation of follow-up activities with the liberated budget from the fixed connectivity subsidy schemes. In future projects, task teams could aim to address client objectives and priorities with more bite-sized pilots, field-oriented analytics and data collection work, ensuring appraisal starts from an advantageous political position for the project. Attention needs to be given in maintaining this momentum in later stages of the project, by communicating about the project’s importance and results to e tra-ministerial political stakeholders and newcomers. 74. K v ’ x, h v -out of least-subsidy reverse auction schemes proved to be a winning design for addressing market failures in both fixed and mobile connectivity, all the while mobilizing private capital. Simple in its design and backed by vendor consultations, the model pursued for the connectivity of white zones and cellular towers enabled timely and resource-efficient delivery. In a context with various small, localized providers with expertise but little resources, this design proved to be effective in crowding-in investments. Future projects should continue, when environment is suitable, to adopt principles of open access and non-discriminatory pricing for last-mile infrastructure investments, to enhance efficiency and foster competition in overcoming market failures. 75. Capacity building efforts were paramount h ’ h h v b f h of M&E. The extensive capacity building efforts integrated into the project allowed for successful delivery of the project both from a technical and an administrative perspective. Beyond the project, MED acquired significant expertise in Official Use Only Page 16 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT procurement of complex parallel subsidy schemes, intervening in fields in which it had little experience. This was achieved because of extensive consultations and mapping of needs during and before appraisal, which did not flag shortcomings related to monitoring and evaluation. Moving forward, task teams should consider further the need to sharpen PIU’s competence and awareness of the importance of M&E. 76. While succeeding in increasing quality of connectivity, h ’ ff could have included more efforts on usage objectives. In white areas, an inclusion gap persists and little information is present on productive internet usage rates. Beyond the relied-upon connectivity provided to HEIs via KREN, software services to HEIs and internet and software services offered to schools currently have low usage rates and integration in service delivery practices. To fully capture the potential of these investments, future initiatives should include comprehensive strategies to enhance digital skills, target access obstacles relating to affordability and devices, and develop digitalized service delivery. 77. When backed with a proactive PIU and clear targets, restructurings can be an opportunity to adapt project activities to just-in-time initiatives complementing and strengthening h ’ reach. The project’s success in obtaining savings allowed for impactful adaptations to emerging client needs. A clear example is the scale up of KREN’s services and reach, decided at the first restructuring, resulting directly from MED’s ill to provide KREN services to schools and ensure data hosting capacity for HEIs. Other follow-up activities, which had not been scoped at appraisal, were executed successfully and complemented KODE’s long term impact. 78. The establishment of a comprehensive Citizen Engagement framework, the consultations with vendors, municipalities and the wider private sector both prior and during implementation enabled quick delivery, responsive management and cost-effectiveness. The CE framework included preliminary consultations, youth dialogues, public hearings, a grievance redress mechanism (GRM), and a beneficiary satisfaction survey. Consultations with beneficiaries, private sector partners and private sector representation groups were maintained throughout the lifecycle, forming an advisory board to the project, A key recommendation is to maintain and expand these engagement activities, ensuring that consultations at national and local levels (workshops, round tables, focus groups) are integrated in project preparation. 79. The trainings conducted are showcasing excellent satisfaction and promising results in boosting employability, notably due to the adherence to best practices such as effective preliminary targeting work. More than 90% of participants declared themselves satisfied with the training and applications were nine times greater than available slots, proving high appetite and timeliness. The improvement in labor outcomes as per current ex-post survey results, with a 7% decrease in the unemployment rate and 17% of participants reporting having found a job at least in part thanks to the training, is promising and lends itself to being repeated at scale. This is especially relevant given the high distribution of graduates outside of the capital (59% as per the survey) and having returned to further studies (30%). At present, emigration among graduates appears to be limited and most participants having found employment have done so either in Kosovar companies or in companies operating in Kosovo, implying a materialization of positive spillovers for the country. This success is the result of trainings incorporating some renowned best practices in their design and implementation, the key one being the preliminary consultations with the Kosovar private sector and the collection of feedback between rounds, allowing for pre-emptive scoping of labor market needs and for more iterations of the most effective and high- demand training programs, such as ERP and Network Administration. The provision of a diploma following an exit test and the complementary training on soft skills also contributed to the program’s success providing a blueprint for future iterations and applications in other contexts. Among the potential improvements noted to achieve more sustainable employment outcomes a) a tighter linkage with local and international employers within the training program, b) more long-term systemization of such trainings via the appropriate public institutions, c) the design of a more structured ex- post diagnostic integrating gender-disaggregated metrics and closer follo ing of the training’s impact on incomes and employment. Official Use Only Page 17 The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS @#&OPS~Doctype~OPS^dynamics@icrresultframework#doctemplate A. RESULTS FRAMEWORK PDO Indicators by Outcomes To improve access to better quality and high-speed broadband services in project areas Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year 274,156.00 Dec/2018 400,000.00 May/2024 400,000.00 May/2024 380,640 Dec/2024 People provided with access to the Comments on achieving targets Target achieved, but indicator deemed invalid. Data is of total fixed internet subscriptions in the country, as Internet (Number) reported by ARKEP for Q4 2023. Public institutions with access to 0.00 Dec/2018 100.00 May/2024 100.00 May/2024 100.00 Dec/2024 high-speed internet through the Comments on achieving targets Target achieved. Project (Percentage) Number of schools connected with 0 Jan/2024 220 Dec/2024 243 Dec/2024 improved digital connectivity Comments on achieving targets Target achieved (Number) Improve access to online knowledge, services & labor markets for citizens, public sector, academia Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year Beneficiaries of improved HEI access 0.00 Mar/2020 85.00 May/2024 85.00 Dec/2024 100.00 Dec/2024 to high-speed broadband through the Comments on achieving targets Target achieved. All HEI have access to high-speed broadband through the project, and all Project (Percentage) students and teachers of the HEI have access. 0.00 Dec/2018 85.00 May/2024 85.00 May/2024 100.00 Dec/2024 Of which female Comments on achieving targets Target achieved. (Percentage) 20.00 Dec/2018 50.00 May/2024 50.00 Dec/2024 75 Dec/2024 Page 18 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Percentage of beneficiaries of Comments on achieving targets Target achieved, as per an exit survey conducted among participants in late 2024. Out of 358 respondents, 148 activities in Component 2 who report responded being either employed or freelance (41%) and 267 responded being either employed, freelance or in being employed (Percentage) further studies (74%). In this sample, the baseline was respectively of 102 and 240. Intermediate Indicators by Components Digital Inclusion Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year Private capital attracted into digital 0.00 Dec/2018 33.00 May/2024 33.00 May/2024 38.87 Dec/2024 connectivity as co-investments under Comments on achieving targets Target achieved. Based on data from the signed grant agreements. More private capital was mobilized than Subcomponent 1.1 (Percentage) what was nominally inserted in the awarded bids, yet this is not captured by the indicator above. Healthcare institutions with access to 0.00 Dec/2018 100.00 May/2024 100.00 May/2024 100.00 Oct/2024 high-speed broadband internet in Comments on achieving targets Target achieved. Project areas (Percentage) Educational institutions with access 0.00 Dec/2018 100.00 May/2024 100.00 May/2024 100.00 Dec/2024 to high-speed broadband internet in Comments on achieving targets Target achieved. Project areas (Percentage) No Dec/2018 Yes May/2024 Yes May/2024 Yes Dec/2024 National Spectrum Monitoring System implemented (Yes/No) Comments on achieving targets Target achieved. 0.00 Dec/2018 75.00 May/2024 75.00 May/2024 81.00 Dec/2024 Beneficiaries satisfied with access to Comments on achieving targets Target achieved. The result of 81% is an extrapolation of the results of the last round of interviews finalized in and quality of broadband services August 2023 in the cadastral zones foreseen at PAD under subcomponent 1.1. 81% refers more specifically to (Percentage) the average of the people in such areas that declared themselves satisfied of a) the quality and b) the speed of connection. No further interviews were planned after the 2023 round and the final list of cadastral zones under the project was different from the one assessed by the survey. Digital Work and Empowerment Indicator Name Baseline Closing Period (Original) Closing Period (Current) Actual Achieved at Completion Result Month/Year Result Month/Year Result Month/Year Result Month/Year 0.00 Dec/2018 2,000.00 May/2024 2,200.00 Dec/2024 2,200.00 Dec/2024 Page 19 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Comments on achieving targets Target has been achieved. It should be noted that only around 75.3% of beneficiaries of trainings were between Youth (aged 18-35) provided with 18 and 35 years old, with the remainder being between 16 and 18 or older than 35. The indicator above is digital skills training under the YOU therefore inclusive of these beneficiaries that were technically above or below the age range desired during Program (Number) preparation. 0.00 Dec/2018 1,000.00 May/2024 1,000.00 May/2024 881.00 Dec/2024 of which female (Number) Comments on achieving targets Target achieved. 0.00 Dec/2018 80.00 May/2024 80.00 Dec/2024 90.00 Dec/2024 YOU program participants expressing satisfaction that the Program Comments on achieving targets Target achieved. Positive ans ers to ‘“Did the training meet your e pectations?’. Based on results of 156 self - responds to their specific needs completed post training interviews of YOU beneficiaries. This survey is currently ongoing. Other more recent (Percentage) surveys showcase similar rates of reported satisfaction with the program (56% reporting the program was 'Extremely effective' or 'Very effective' at improving their employability). No Dec/2018 Yes May/2024 Yes May/2024 Yes Dec/2024 Establishment of Kosovo NREN (Yes/No) Comments on achieving targets Target achieved Higher educational institutions with 0.00 Dec/2018 80.00 May/2024 80.00 Dec/2024 80.00 Dec/2024 access to the GEANT network Comments on achieving targets Target achieved. The agreement with GEANT is signed. Out of 20 HEIs in Kosovo, 16 were connected to GEANT (Percentage) via KREN. Further HEIs could access GEANT through KREN. Households in Project areas covered 0.00 Dec/2018 100.00 May/2024 100.00 May/2024 100.00 Oct/2024 by Digital Awareness Program Comments on achieving targets Target achieved, referring to a leaflet campaign conducted in areas covered by subcomponent 1.1's fixed (Percentage) broadband subsidy scheme. Page 20 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT B. KEY OUTPUTS Page 21 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT To improve access to better quality and high-speed broadband services in project areas 1. People provided with access to the Internet PDO Indicators 2. Public institutions with access to high-speed internet through the project 3. Number of schools connected with improved digital connectivity 1.White zones covered with access to internet Key Outputs by Component 2. NSMS installed (linked to the achievement of the PDO Outcome) 3. Schools and Hospitals in white areas covered with internet Improve access to online knowledge, services & labor markets for citizens, public sector, academia 1. Beneficiaries of improved HEI access to high-speed broadband through the project PDO Indicators 2. Percentage of beneficiaries of activities in Component 2 who report being employed Key Outputs 1. HEI with access to high-speed broadband (linked to the achievement of the PDO Outcome) 2. Employment of beneficiaries of activities of Component 2 Page 22 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT project Implementation Support Intermediate Results Indicators / Key Outputs / (linked to the achievement of the Component) Digital Work and Empowerment 1. Youth (aged 18-35) provided with digital skills training under the YOU Program 2. YOU program participants expressing satisfaction that the Program Intermediate Results Indicators responds to their specific needs 3. Establishment of Kosovo NREN 4. Higher educational institutions with access to the GEANT network 5. Households in project areas covered by Digital Awareness Program 1. 2,100 2. 90% Key Outputs 3.YES (linked to the achievement of the Component) 4.80% 5. 100% Digital Inclusion 1. Private capital attracted into digital connectivity as co-investments under Subcomponent 1.1 2. Healthcare institutions with access to high-speed broadband internet in project areas Intermediate Results Indicators 3. Educational institutions with access to high-speed broadband internet in project areas 4. National Spectrum Monitoring System implemented 5. Beneficiaries satisfied with access to and quality of broadband services Page 23 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT 1.33.10 Key Outputs 2. 100% (linked to the achievement of the Component) 3. 100% 4. 81% Page 24 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Charles Pierre Marie Hurpy Team Leader Natalija Gelvanovska-Garcia Team Leader Rhedon Begolli Team Leader Jonida Myftiu Financial Management Specialist Kushtrim Plakolli Procurement Specialist Carlos Lago Bouza Procurement Specialist Shpresa Kastrati Environmental Specialist Bekim Imeri Social Specialist Ruxandra Costache Counsel Keong Min Yoon Counsel Rahmoune Essalhi Procurement Team Ivana Bojic Team Member @#&OPS~Doctype~OPS^dynamics@icrannexstafftime#doctemplate B. STAFF TIME & COST Staff Time & Cost Stage of Project Cycle No. of Staff Weeks US$ (including travel and consultant costs) Preparation FY18 32.520 214,673.43 FY19 0.000 579.40 FY20 0.375 2,617.71 FY23 0.000 22.07 Total 32.90 217,892.61 Page 25 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Supervision/ICR FY19 17.084 87,082.01 FY20 18.029 94,329.29 FY21 13.266 78,195.30 FY22 12.833 101,948.80 FY23 13.971 95,151.35 FY24 17.347 98,503.26 FY25 13.087 56,964.95 Total 105.62 612,174.96 Page 26 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ANNEX 3. PROJECT COST BY COMPONENT Component Amount at Approval (US$M) Actual at project Closing (US$M) Digital Inclusion 18.6 12.9 Digital Work and Empowerment 5.5 5.4 project Implementation Support 0.9 1 Page 27 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ANNEX 4. ECONOMIC ANALYSIS This economic analysis completes the efficiency section of this ICR, by updating the economic model calculated at appraisal with the available actuals from the project. This updated analysis inserted the actual costs disbursed (as reported in annex 6) into the original cost‑benefit spreadsheets for each sub‑component foreseen at PAD, and when available added data on key activity outputs (i.e. households connected, public institutions connected, etc.). The ICR deliberately preserved all ex‑ante assumptions for which there was no actual data available (productivity gains, discount‑rates, operators revenues, etc.). No specific efforts were put in place to collect data specifically for the economic analysis, nor to re‑engineer the foundation of the model. A further element to take into account is that the project’s activities that were not foreseen at appraisal (for a total of around 7 million USD) could not be captured in this updated model. While a more in-depth analysis would be necessary for a more precise calculation of the actual economic impact of the project on the country, this updated model is sufficient to confirm the project’s positive impact. Under this updated model, the total NPV went from 3.8 to 4.1 million EUR, providing further proof of the project’s positive impact and efficacy. Subcomponent Ex ante Ex ante Ex ante cost Ex post NPV Ex post IRR Ex post cost N V (€, internal rate envelope (€, (€, ) (%) v (€, millions) of return (IRR) millions) millions) (%) 1.1. (as foreseen in PAD) 1.4 21% 11.9 1.1 78% 3.1 1.1. extra-PAD activity: Could not be Could not be fiberization of cell calculated for calculated for towers N/A N/A N/A this ICR this ICR 3.3 1.2 (as foreseen in PAD) 0.8 37% 3.3 1.3 52% 3.2 1.2. extra-PAD activity: dedicated Could not be Could not be connectivity calculated for calculated for works for schools N/A N/A N/A this ICR this ICR 2.9 2.1 (as foreseen No IRR has No IRR has in PAD, with 200 been been additional calculated for calculated for beneficiaries) subcomponent subcomponent 2.1. as no 2.1. as no neutral state is neutral state is attained via attained via the execution the execution of trainings, of trainings, therefore IRR therefore IRR is not is not 1.4 applicable. 1.6 1 applicable. 1.9 Page 28 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT 2.1. Extra-PAD activity: purchase Could not be Could not be of IT and office calculated for calculated for equipment N/A N/A N/A this ICR this ICR 0.2 2.2 (as foreseen in PAD) 0.2 23% 2.8 0.7 9% 3.1 2.2. Extra-PAD activity: acquisition of Could not be Could not be additional calculated for calculated for licenses for KREN N/A N/A N/A this ICR this ICR 0.4 Core Assumptions from the Model TABLE: COST-BENEFIT (COB) ANALYSIS Assumptions Social discount rate 7.09%-7.30% β (pure time preference rate) 1.00 ε (elasticity of marginal utility of consumption) 1.50 σ (e pected gro th rate per capita of consumption) 4.06-4.2 Financial discount rate 19.43%-17.52% Rf (Risk-free rate) 4.03% Rm-Rf (equity risk premium) 11.94% β (market beta for the telecommunications industry) 1.29 Average of social and financial discount rates 13.26% Discount rate calculations Equity risk premium - Albania 13.32% Equity risk premium - Bulgaria 9.23% Equity risk premium – Bosnia and Herzegovina 16.46% Equity risk premium - Croatia 10.97% Equity risk premium - Romania 9.71% Market beta - telecommunications equipment industry 1.29 Risk-free rate - Kosovo 4.03% Expected growth in consumption per capita 4.06% The ex-ante cost-benefit (CoB) analysis employed both social and financial discount rates to evaluate the present value of project benefits and costs. Social Discount Rate (SDR). The SDR is the rate used in the literature to evaluate the profitability of public investments from a social perspective, typically ranging from 3 to 10% (IPCC, 2022). A lower SDR increases the present value of long- Page 29 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT term benefits, making investments in infrastructure and human capital appear more attractive. Small changes in SDR can significantly affect Net Present Value (NPV), especially for long-horizon projects. Set at 7.09% for subcomponents 1.1 and 1.2, and at 7.30% for subcomponents 2.1 and 2.2. SDR is based on the Ramsey Formula r = β + ε × σ, here • β the pure time preference rate (1.00), reflects society’s inherent preference for current consumption versus future consumption. • ε the elasticity of marginal utility of consumption (1.5), representing aversion to inequality over time. • σ the e pected gro th rate of consumption per capita (estimated at 4.06%–4.2%). Financial Discount Rate (FDR). The FDR shows whether a project is attractive to private capital. Public-private partnerships, as seen in KODE’s broadband rollout, must address this high required return through subsidies or cost-sharing. Calculated using the Capital Asset Pricing Model (CAPM) and set at 19.43% for subcomponents 1.1, 1.2 and 2.2, and 17.52% for subcomponent 2.1. This reflects the expected return a private investor would require, incorporating market risks. Inputs include: • A synthetic risk-free rate (4.03%) for Kosovo • Equity risk premiums (11.9%–13.3%) based on regional comparators • Industry beta coefficients (1.13–1.29), capturing risk profiles of the telecom and software sectors Net Present Value (NPV). The NPV is the difference between the present value of benefits and the present value of costs over a project’s life cycle. It incorporates the time value of money, meaning that future benefits are orth less than immediate ones. Formally: Where Bt are the benefits, Ct the costs, r the discount rate, and t the time period. A positive NPV means the project is economically viable; a negative NPV suggests reconsideration unless justified by social or strategic reasons. The remaining assumptions specific to each subcomponent are listed in the sections below. Subcomponent 1.1: Financing Digital Connectivity The ex-ante CoB analysis for Subcomponent 1.1 projected high economic returns by expanding broadband in 266 underserved cadastral zones to connect 10 544 households, 38 healthcare institutions and 129 schools using a DBO (Design-Build-Operate) co-financing model. In this model, ISPs were selected via reverse auctions, with a public-private matching grant of 1:1 in early tenders. The model assumed improved household access to digital services, higher institutional connectivity, and broad-based productivity gains. The core ex-ante model used for reporting purposes indicated an IRR of 21% over a five-year horizon under a 50:50 public-private co-financing structure. Net Present Value (NPV) was calculated using a social discount rate of 7.09% and a financial discount rate of 19.43%, reflecting the cost of capital for public and private actors respectively. In the “ ith the project” scenario, the IRR is higher to the social discount rate, making the subcomponent profitable The ex-ante model is considering and quantifying direct benefits: consumer surplus was estimated for 10 544 households, cost savings by 129 schools, and cost savings by 38 healthcare institutions. The ex-post model is 4 475 households, 105 schools and 43 health institutions. The difference in NPV and IRR between ex-ante and ex-post COB depends on the number of households, schools and health institutions impacted as well as the cost of the project during the 5 years. The PAD Page 30 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT anticipated a steady cost of the project during the 5 years while the ICR revealed that the 266 white zones were covered between 2019 and 2021 impacting substantially the IRR and the NPV. The ex-post COB has not considered the additional tasks of subcomponent 1.1 and have not calculated an ex-post social discount rate and financial discount rate. FIGURE 3: COB ANALYSIS (EX-ANTE) FOR SUBCOMPONENT 1.1 (IN EUR) Despite lower household impacts, institutional benefits were notable. NPV. Based on the updated ex-post benefit streams derived from observed increases in institutional connectivity —the recalculated Net Present Value (NPV) using a 7.09% social discount rate is EUR 1.1 million. IRR. A recalculated Internal Rate of Return based on actual implementation data reaches 78%, indicating a rapid payback and a high rate of return relative to the discounted investment stream. Page 31 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT FIGURE 4: COB ANALYSIS (EX-POST) FOR SUBCOMPONENT 1.1 16 Subcomponent 1.2: Improving the enabling environment for digital connectivity The ex-ante CoB analysis for Subcomponent 1.2 focused on providing ARKEP with a network of spectrum monitoring stations. Benefits were expected from increased spectrum efficiency, and better precision in measuring the spectrum usage. The five-year NPV under the social discount rate of 7.09% was estimated at EUR 0.804 million. The calculated IRR stood at 37%. Sensitivity analysis showed strong dependency on the percentage of spectrum utilization improvements. FIGURE 5: COB ANALYSIS (EX-ANTE) FOR SUBCOMPONENT 1.2 Ex-post, the planned number of fixed monitoring sites was implemented as expected, though delays occurred in procurement and system integration. The additional government income from spectrum use was not as high as anticipated, and enforcement of illicit spectrum use proved less impactful than assumed. As no granular enforcement 16 In yellow is what the ICR ended up modifying: total project costs without new activities added after restructuring (the same applies for each subcomponent) Page 32 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT dataset is publicly accessible, validating the full economic impact ould require ARKEP’s internal records or independent analysis of frequency allocation logs. The ex-post NPV and IRR for Subcomponent 1.2 are higher to the ex-ante estimates. The calculated Net Present Value (NPV) rose at approximately EUR 1.4 million using a social discount rate of 7.09%, and the Internal Rate of Return (IRR) rose at 52%, reflecting the projected gains from improved spectrum enforcement and institutional efficiency. The difference in NPV and IRR between ex-ante and ex-post COB depends on the cost of the project. However, it is important to note that additional connectivity tasks for primary and secondary schools (premium connectivity of 243 schools)—were not initially part of the Subcomponent 1.2. These interventions, while consistent with the overall PDO and broader digital inclusion objectives, primarily contributed to Subcomponent 1.1’s e pansion and are not directly attributable to spectrum monitoring efforts. As such, they were excluded from the calculation of benefits and costs for this subcomponent to preserve analytical clarity and avoid overestimating NSMS-specific returns. FIGURE 6: COB ANALYSIS (EX-POST) FOR SUBCOMPONENT 1.2 Subcomponent 2.1: Youth Online and Upward (YOU) program The YOU Program was expected to train 2,000 young people in front-end development and online freelancing skills. Ex- ante models estimated substantial economic returns based on income differentials before and after the training and compared this to a self-study counterfactual. At a 33% success rate in online employment, the NPV using a 7.3% SDR was estimated at EUR 1.38 million , with sensitivity analysis showing sharp performance drops below a 25% success rate. Page 33 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Ex-post, 2200 young individuals (18-35) benefited from YOU program. The restructuring of March 2023 that reallocated resources and aimed to enroll another 200 participants has not been taken into account in this ex-post COB analysis. Survey data and platform monitoring showed a lower-than-expected online employment success rate with 35% of youth that remained unemployed after the training. The ex-post NPV is EUR 1 million, below the ex-ante NPV (1.38) due to higher subcomponent cost. There is no IRR neither in the ex-ante model nor in the ex-post because IRR indicates because IRR signifies the rate of return attained when the NPV of the project reaches a neutral state, precisely at the point where NPV breaks even. For subcomponent 2.1, NPV remains positive for 5 years therefore IRR calculation doesn’t apply. Subcomponent 2.2: Research and Education Network (KREN) Page 34 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT The ex-ante model projected benefits from higher productivity in research and education institutions through access to GEANT (pan-European research network). Estimated productivity boosts of 0.65% to 1.3% were applied to 13,000 staff and student users. Ex-ante NPV at a 7.3% SDR was EUR 206,502, and IRR was 23%. The model assumed a phased expansion of universities connected to the network and increasing productivity impacts. FIGURE 7: COB ANALYSIS (EX-ANTE) FOR SUBCOMPONENT 2.2 KREN was established and functional by 2023. However, the number of connected institutions remained below targets, and some delays were experienced in onboarding secondary schools and ensuring end-user uptake. While connectivity was extended to many higher education institutions, effective use of the network and awareness among faculty lagged. GEANT- induced productivity effects were likely below the initial projections. Recalculating CoB based on ex-post cost of the project, IRR is 9%, with NPV at EUR 0.70 million. The revised returns are still positive, though dependent on long-term institutional uptake. The ex-post COB is taking into account the set up of NREN and connection to GEANT, the support for online labour market information and services in project areas as well as follow up communication ork but not “A ure, Microsoft 265 and other licenses for KREN that was not foreseen in PAD. Page 35 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT FIGURE 8 COB ANALYSIS (EX-POST) FOR SUBCOMPONENT 2.2 Page 36 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS No borrower comments on the draft ICR and any comments of co-financiers and other partners/stakeholders. Page 37 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) Section I. PDO indicators, units, baseline, targets TABLE 1: BASELINE AND OUTCOME TARGETS PDO indicator Unit Baseline End Revised End Revised End Value at Detailed (PAD) Target at Target after Target after closing comments by Approval Restructuring Restructuring (October the ICR team (PAD) #1 #2 2024) To Improve access to better quality and high-speed broadband services in project areas People provided Number 274,156 544,681 400,000 400,000 380,640 This value is with access to the not linked to internet the project areas (266 cadastral zones) but represents the total number of people provided with internet at the country level, as reported by ARKEP. Additionally, the first restructuring should have revised the target to correspond to the number of people in the white zones. However, the figure of 400,000 still does not match the actual number of people in the white zones Page 38 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT with internet access (see Table 4). This indicator should therefore be considered invalid. Public Percentage 0 100 100 85 100 This value institutions with does not access to high- specify the speed internet actual through the number of project public institutions in the 266 cadastral zones that have internet access, rendering the impact harder to measure tangibly. Additionally, the second restructuring lowered the target without providing any explanation other than "to fit with the deadline of the project." Number of Number / / / 220 243 This indicator schools got added in connected with the second improved digital restructuring connectivity so as to track a key follow- up activity, even though Page 39 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT the activity had been inserted as of the first restructuring. Improve access to online knowledge, services and labor markets for citizens, public sector, academia Beneficiaries of Percentage 0 80 85 85 100 This indicator improved HEI was poorly access to high- defined in the speed broadband PAD. If the through the indicator project measures the percentage of beneficiaries of an institution provided with internet, the result will automatically be 100%. Surprisingly, the target was set at 80%, with no rationale provided for this choice of target. Of which female Percentage 0 80 59.86 59.86 100 Same as above Percentage of Percentage 20 30 50 50 62 This indicator beneficiaries of could have activities in been better Component 2 calculated, who report being given the employed quality of the exit survey that was devised to calculate it. It is crucial to capture information Page 40 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT about beneficiaries who were not employed before attending the YOU program, which was not efficiently done. Additionally, the indicator should specify whether the beneficiary is employed within the country and in the ICT sector. The primary aim of this indicator is to assess the direct causal relationship between the YOU program and its impact on employment, rather than merely identifying a correlation. Note1: project’s closing occurred on December 31, 2024, but last information available are from October 2024. Page 41 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT TABLE 2: REVISED INTERMEDIATE RESULTS INDICATORS BY COMPONENTS Original PDO indicator Unit Baseli End Revised End Value Detailed (PAD) ne Target at Target at comments by the (PAD) approval (restructuring #1) closing ICR team (PAD) (Dece mber 2024) Component 1: Digital Inclusion Private Capital attracted Percentag 0 50 33 33.1 This metric only into digital connectivity e takes into as co-investments under account nominal Subcomponent 1.1 investments from the private sector from the contractual agreements. Healthcare institutions Percentag 0 100 100 100 As previously with access to high- e discussed, this speed broadband indicator would internet in project areas have been a better indicator if calculate numerically as opposed to in percentage. Educational institutions Percentag 0 100 100 100 As above. with access to high- e speed broadband internet in project areas National Spectrum Yes/No No Yes Yes Yes Monitoring System implemented Page 42 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Beneficiaries satisfied Percentag 0 75 75 81 This indicator is with access to and e not well-defined, quality of broadband as it does not services accurately reflect the project's goals. People in the project areas might be satisfied with low-quality internet, which does not align with the objectives of the KODE project. The aim of the KODE project is to establish conditions for high-quality internet access (over 100 Mbps). Therefore, measuring the satisfaction levels of individuals with low internet usage is not a relevant metric for assessing the project's impact, as opposed to median speed of connection, increase in subscription uptake. Page 43 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Component 2: Digital Work and Empowerment Youth (aged 18-35) Number 0 2,000 2,000 2,2 The final number reported provided with digital 00 also includes around 36% skills training under the of beneficiaries not within YOU Program the 18 to 35 age range. -of which female? Number 0 1,000 1,000 881 YOU program Percentag 0 75 80 90 Employment and labor participants expressing e market outcomes would satisfaction that the have been a good Program responds to complementary metric their specific needs beyond mere satisfaction. Establishment of Kosovo Yes/No No Yes Yes Yes NREN Higher educational Percentag 0 50 80 80 No metrics on usage of institutions with access e subscriptions, on to the GEANT network improvement on speed of internet access were inserted. Households in project Percentag 0 100 100 100 No other metrics areas covered by Digital e calculating internet uptake Awareness Program and productive usage in target populations were inserted in the results framework of awareness activities. Page 44 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT TABLE 3: DIFFERENCE BETWEEN FORESEEN COSTS APPRAISAL AND EFFECTIVELY DISBURSED FUNDS PER ACTIVITY Subcomp Activity Amount foreseen Amount Difference between amount in onent in PAD disbursed as foreseen in PAD and expended in In 2018 EUR per ICR ICR calculation In 2018 EUR In 2018 EUR ICR > PAD (PAD>ICR) 1.1. Connectivity of fixed broadband € € white areas (grant scheme) 11,820,000.00 3,188,567.00 Note: the # of households and settlements connected directly € (8,631,433.00) via the project’s funds was lower than at PAD, but the objective of eradicating broadband white areas was achieved. 1.1. Fiberization of cell towers Not foreseen at € Not foreseen at PAD PAD 3,330,415.89 1.1 Technical assistance to €170,000.00 € €96,209.00 ARKEP/MED 266,209.00 Total of subcomponent 1.1 € € € (5,204,808.11) 11,990,000.00 6,785,191.89 1.2. National Spectrum Monitoring € € € (187,533.00) System 3,370,000.00 3,182,467.00 1.2. Provision of technical assistance € 20,000.00 € for Kosovo NSMS 36,855.00 €16,855.00 1.2. Premium connectivity of 243 Not foreseen at € Not foreseen at PAD schools PAD 2,915,732.64 € € Total of subcomponent 1.2 € 2,745,054.64 3,390,000.00 6,135,054.64 € € Component 1 € (2,459,753.47) 15,380,000.00 12,920,246.53 2.1. YOU program (2000 people) € € 1,650,000.00 1,971,995.92 € 321,995.92 2.1. YOU program (purchase of IT Not foreseen at € and office equipment) PAD 182,575.00 Not foreseen at PAD 2.1. YOU program (200 additional Not foreseen at € Not foreseen at PAD people) PAD 230,751.00 Total of subcomponent 2.1 € € € 735,321.92 1,650,000.00 2,385,321.92 2.2. Set up of NREN and connection € € to GEANT 2,480,000.00 2,413,457.16 € (66,542.84) Page 45 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT 2.2. Azure, Microsoft 365 and other Not foreseen at € Not foreseen at PAD licenses for KREN PAD 376,027.00 2.2. Support for online labour €410,000.00 € market information and 118,015.00 services in project areas € (291,985.00) 2.2. Follow up communication Not foreseen at € work (cyber awareness PAD 218,540.50 Not foreseen at PAD program, video editing, etc) Total of subcomponent 2.2 € € € 236,039.66 2,890000.00 3,126,039.66 Component 2 € € € 916,723.94 4,540,000.00 5,456,723.94 3. Project Implementation support €740,000.00 1,043,202.60 € 303,202.60 Component 3 €740,000.00 € € 303,202.60 1,043,202.60 TOTAL € € € (1,239,826.93) 20,660,000.00 19,420,173.07 TABLE 4: CADASTRAL ZONES, PEOPLE AND HOUSEHOLDS IN PROJECT AREAS Covered by As defined in Cadastral zones Coverage conducted KO-Pilots PAD (Project Covered by KODE registered without independently by ISPs in 2018 led Appraisal project inhabitants as per citizen after the pilots, by the Document) auto-reporting without subsidies MED Cadastral zones concerned 266 43 164 64 32 (white zones) Households in cadastral zones 10544 3296 N/A 572 2546 concerned (as per 2011 census) Households in cadastral zones concerned (as per the updated N/A 3014 4475 N/A N/A grant agreements data) Public 167 50 103 N/A N/A institutions in Page 46 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT cadastral zones concerned Planned/used € 11,820,000 €1,189,000 € 3,188,567 N/A N/A project financing Cost per household (2011 €1,121 N/A N/A N/A N/A census data) Cost per household (grant N/A € 394 € 713 N/A N/A agreement data) Cost per public € 70,778 € 23,780 € 30,957 N/A N/A institution TABLE 5: KEY DATA FROM KANTAR’S REPORTS FROM 2020 TO 2023 – APPLICABLE TO PROJECT AREAS OF SUBCOMPONENT 1.1. AT PAD (266 CADASTRAL ZONES AT PAD AS OPPOSED TO THE FINAL 164 CADASTRAL ZONES ON WHICH THE PROJECT INTERVENED) Access Don’t Main % of to have Average Income Institutions Public and population HH Internet access to Confidence Computer Net Source with academic belonging Survey internet among internet in ICT Skills Access at Monthly (Private internet institutions to lower Year access Serbian “because for Job Home (%) Salary Sector connection in project socio- (%) HH (%) it is not Market (%) (€) Wages (%) (i) areas (#) economic useful” %) class (%) 2020 84% 53% 37% 420 21% 61% 85 11% 4% 40% 2021 81% 48% 37% 359 25% 96% 53 40% 14% 33% 2022 83% 33% 28% 355 30% 91% 44 9% 16% 39% 2023 80% 31% 29% 396 27% 90% 39 27% 20% 40% (i) % of people in institutions located in project areas that declared having functioning internet connection in their institution. Subcomponent 1.1: While the surveys conducted by Kantar indicate a relative decline in internet uptake and stagnant internet usage (from 84 percent to 80 percent 2020-2023, 44 percent both in 2020 and in 2023, respectively), the baseline calculated in 2020 cannot be considered reliable, after the roll-out of all pilots and of a portion of project-financed subsidy schemes (detailed further in section IV). Furthermore, the cadastral zones under the study were not adapted to cover the significant change in final white zones as detailed above. Taking into account business subscriptions and touristic summer houses with year-round subscriptions, lowering the real number of used subscriptions, as well as the affordability of internet subscriptions in the country, the bundling of internet subscriptions ith TV coverage and the project’s a areness campaigns which can explain such a rapid rise in subscriptions, the effective internet uptake in project areas can be estimated to have been impacted positively by the project. The impact is therefore not limited to the manifest increase in quality of connectivity. An access gap appears to remain present, as Kantar’s poll suggests, ith lack of skills (29 percent) and low interest in the internet (20 percent) cited as reasons for not subscribing. Similarly, access seems to be mostly limited to mobile devices, with only 27 percent of households having access to a laptop at home. Little data is available on the impact on the productive usage of the internet, specifically for labor matters. YOU Program: More importantly, this second survey highlighted encouraging outcomes when it comes to labor results, with a post-project employment rate of 38 percent (compared to 26% pre-training), 48 percent of participants stating that the training has improved their employability, 17 percent of participants stating they found employment at least in part Page 47 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT thanks to the training and 90% of those employed reporting being so by a company with active operations in Kosovo 17. A further element to highlight is the geographical distribution of the students, as the training focused on injecting ICT skills in areas outside of the capital city, currently concentrating the vast majority of ICT jobs and skills. Further details on this final survey‘s results are provided in anne 6. Section II. TABLE 6: ANALYSIS OF YOU TRAINING’S LAST EX-POST TRAINING SURVEY (385 RESPONDENTS TO AN ONLINE SURVEY AS OF THE FIRST QUARTER OF 2025) Indicator Analysis Perceived Effectiveness Perception of training effectiveness is a critical indicator of program success. Approximately 78% of participants rated of Training on the training as either "Very Effective" or "Extremely Effective", indicating a general consensus that the program Employability positively contributed to employability. Only 18% of participants rated it as ‘Slightly effective’ (12%) or ‘Not effective’ (6%). The results are very encouraging on training quality and effectiveness, suggesting that the content and topics themselves were deemed pertinent regardless of labor outcomes. Perceived Increase in A fundamental measure of training effectiveness is whether participants noticed an increase in job opportunities and Job Opportunities interview invitations post-training. 48% of participants report having experienced an increase in job opportunities, while 52% did not. 10% of respondents directly state that they have not found a job yet and the training did not help them, 43% state that while not having found a job the training is helping their chances, 17% report having found a job at least in part thanks to the training, the remaining 30% either went into further studies, are looking or have found jobs unrelated to the training. Employment outcomes Unemployment decreased from 37.4% of participants to 30.6%. Employment moved from 26.4% to 38.4% when also taking into account freelance workers, from 22.6% to 35.3% only considering regular employees. The effect on employment varies significantly based on the course attended, suggesting diverse level of inherent demand for different skills provided. While the dataset has limited entries and its suggestions should be taken with prudence, the courses related to Python, Digital Marketing and Network Administration appear to have had the most positive effect on employment (57%, 30% and 11%), but a more systematic study needs to be conducted so as for more tangible inferences to be drawn. Regional Employment Employment opportunities vary based on geographic location, with urban centers typically offering more job Impact prospects. The analysis of employment outcomes by region found that Prishtina accounted for 49% of participants who reported increased job opportunities (compared to 32% of total participants), making it the most favorable location for employment. In contrast, participants from smaller municipalities such as Rahovec, Viti, and Drenas faced greater challenges, with over 70% of respondents in these areas reporting no increase in job opportunities. These findings emphasize the importance of regional economic development initiatives and of the hardships in labor supply in these areas. Employment in Local vs. A notable dimension of post-training employment outcomes is the type of employer participants secured jobs with. International Approximately 91% of employed participants work for companies with a presence in Kosovo and 58% work for the Companies Kosovar/regional operations of an international firm. This suggests that 33% got employed by Kosovar companies directly, as opposed to being employed in outsourcing roles for foreign firms not operating in the country, implying the concretization of positive spillovers for the labor shortages present in the country. Qualitative feedback The distribution of qualitative responses highlights the dominant themes among participants regarding the submitted by effectiveness and limitations of the training programs. Practical training emerged as the most frequently cited area for participants improvement, accounting for 55.2% of all qualitative responses. A significant portion of participants expressed concerns that the training was too theoretical and lacked real-world applicability. Many emphasized the importance of incorporating hands-on exercises, industry projects, and simulations to bridge the gap between theoretical knowledge and the demands of the job market. The recurrent nature of this feedback suggests that participants feel underprepared for practical work environments and would benefit from a curriculum that integrates real-world problem-solving, project-based learning, and direct exposure to industry tools and technologies. The second most prevalent theme, comprising 19.5% of responses, was the need for stronger job placement support and industry connections. Many participants indicated that while the training provided them with valuable knowledge, it did not necessarily translate into job opportunities. They highlighted the absence of structured job placement 17Calculations were made out of the latest post training survey on employment status and opportunities by PIU. The projected employability opportunities ratio was based on the number of people that declared to be unemployed before the training (283 people) and the number of people that ans ered “yes” to the question “Since completing the training, have you noticed an increase in job opportunities or interview invitations?” (133 people) Page 48 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT services, networking events, and employer engagement initiatives. Some suggested that training programs should actively collaborate with companies to facilitate interviews, internships, or direct recruitment opportunities. This sentiment underscores a critical gap between skills development and employment accessibility, reinforcing the idea that training programs alone are insufficient without clear pathways into the workforce. A substantial portion of respondents, representing 14.9% of the feedback, emphasized the importance of career- oriented training. These participants stressed the need for structured learning experiences that align with job market demands and industry-recognized certifications. Some noted that while the courses were informative, they lacked a clear progression toward employment, making it difficult to transition from training to securing a job. The feedback suggests that participants would benefit from a curriculum that is designed with direct input from industry professionals, incorporating the latest market trends, in-demand skills, and certification opportunities that increase employability. A smaller but still notable percentage of responses, 6.9%, pointed to the issue of training frequency and accessibility. Several participants expressed concerns that the training was offered as a one-time opportunity with limited options for continued learning. Some suggested that courses should be repeated regularly or supplemented with advanced modules that allow participants to build on their foundational knowledge. Others recommended that training programs be made more accessible to a broader audience, including those who may not have been able to participate due to scheduling conflicts or financial constraints. This feedback suggests that while the training programs were beneficial, they may not have been sufficient for long-term skill development, and additional learning opportunities could help participants remain competitive in the job market. Lastly, 3.4% of responses addressed concerns related to course structure and monitoring. Some participants reported inconsistencies in the quality of instruction, noting that certain aspects of the training could be better organized. Others mentioned that the difficulty level of assessments was too low and suggested that standardized testing or periodic evaluations could enhance the learning experience. This feedback indicates that while most participants appreciated the training, there is room for improvement in terms of course delivery, evaluation methods, and quality control. The distribution of qualitative feedback provides clear insights into the areas where training programs could be refined to maximize their impact on employability. The dominant themes suggest that participants are seeking a more practical, employment-oriented learning experience, with stronger connections to the job market and ongoing opportunities for skill enhancement. Addressing these concerns through curriculum improvements, employer partnerships, and structured career support services could significantly enhance the effectiveness of these training initiatives. FIGURE 1: STATISTICS ON THE YOU TRAINING’S LAST EX-POST TRAINING SURVEY Employment Status Before vs. After Training Page 49 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT Perceived Effectiveness Of Training On Employability Number of respondents based on municipality of residence prior to training (in blue) vs. ex post (in orange) 160 140 120 100 80 60 40 20 0 Page 50 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT TABLE 7: STATISTICS ON EMPLOYMENT PERFORMANCE PER COURSE ATTENDED (BASED ON THE RESULTS OF THE MAIN EX-POST SURVEY, 358 RESPONDENTS) SURVEY EMPLOYMENT EMPLOYMENT EMPLOYMENT RATE % CHANGE IN RESPONDENTS PRE TRAINING AFTER TRAINING AFTER TRAINING EMPLOYMENT RATE PER COURSE (FREELANCE (FREELANCE EXCLUDED) EXCLUDED) 3D PRINTING TECHNOLOGY - 7 2 3 43% 14% PRIZREN ANDROID MOBILE 9 2 3 33% 11% DEVELOPMENT – PRISHTINA AWS & LINUX – PRISTINA 4 0 0 0% 0% CYBERSECURITY – GJILAN 15 4 4 27% 0% Page 51 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT DIGITAL MARKETING – FERIZAJ 8 2 4 50% 25% DIGITAL MARKETING – GJAKOVA 11 2 3 27% 9% DIGITAL MARKETING – 18 4 12 67% 44% PRISHTINA ERP BUSINESS SOLUTION – SAP 58 38 42 72% 7% FULL STACK DEVELOPER – 10 3 5 50% 20% PRIZREN IOS MOBILE DEVELOPMENT – 5 0 2 40% 40% PRISHTINA IT ESSENTIALS – FERIZAJ 13 2 5 38% 23% IT ESSENTIALS – MITROVICA 13 0 1 8% 8% MERN FULL STACK – FERIZAJ 27 0 2 7% 7% MERN FULL STACK – GJAKOVA 9 1 1 11% 0% NETWORK ADMINISTRATOR – 49 2 5 10% 6% GJILAN NETWORK ADMINISTRATOR – 24 6 8 33% 8% PEJE NETWORK ADMINISTRATOR – 10 2 6 60% 40% PRISHTINA PYTHON – PRISHTINA 7 1 5 71% 57% QA MANUAL TESTER – 4 1 0 0% -25% MITROVICA REACT JAVA SCRIPT – 12 4 4 33% 0% MALISHEVA REACT JAVA SCRIPT – PRISTINA 33 6 12 36% 18% VIDEO EDITING AND AFTER 8 1 7 88% 75% EFFECTS – PRISTINA VIDEO EDITING BEGINNER TO 13 1 0 0% -8% PRO – PEJA VIDEO EDITING – PRIZREN 9 3 1 11% -22% WORDPRESS – MITROVICA 9 0 1 11% 11% TABLE 8: STATISTICS ON APPLICATIONS AND BENEFICIARIES PER MODULE OF THE YOU TRAINING (PARTIAL RESULTS WITH 2148 PARTICIPANTS, AS OPPOSED TO THE FINAL 2200) NUMBER OF TRAINING PROGRAM APPLICATIONS RECEIVED #CERTIFIED BENEFICIARIES NETWORK ADMINISTRATOR 3164 337 DIGITAL MARKETING 2790 205 MOBILE DEVELOPMENT 1839 152 REACT JS 1380 223 PYTHON 1370 80 VIDEO EDITING 1174 193 Page 52 Official Use Only The World Bank Kosovo Digital Economy (KODE) (P164188) ICR DOCUMENT CYBERSECURITY 848 116 IT ESSENTIALS 816 171 ERP BUSINESS SOLUTIONS 961 113 FULL STACK 605 71 MERN FULLSTACK 557 179 MICROSOFT AZURE 498 39 AWS 511 100 WORDPRESS 313 74 QA MANUAL TESTER 150 64 3D PRINTING 127 31 iThe PAD relied on prior assessments and pre-feasibility studies conducted by MED and Deloitte in 2011, identifying 266 cadastral zones (out of 1,351) as the project area, encompassing an estimated 10,544 households and 61,156 people. However, discussions with the Project Implementation Unit (PIU) revealed several critical shortcomings in the project's initial scope. Firstly, the reliance on 2011 pre-feasibility studies in a country like Kosovo, characterized by significant brain drain and rural exodus, proved inadequate. Several potential causes such as rapid demographic changes, COVID-19, rural exodus between 2011 and 2019 rendered the initial estimates inaccurate as current estimates shows a potential impact on 3,932 households (KANTAR report) instead of 10,544 households as stated in the PAD, and approximately 15,531 people (KANTAR report) instead of 61,156 households. The PAD planned to impact 61,156 people, but KANTAR's final 2023 report showed only 11,182 people in the sample areas that corresponds to 72% of project areas, equating to 15,531 people in 100% of project areas. Moreover, the PAD estimated the number of households to be 10,544. Based on KANTAR's sample, the estimated number of households should have been 7,588 (72% of 10,544), but it was estimated at 3,371 households, indicating a reduction in scope (100% would have been approximately 4,683 households). Field analysis conducted by KANTAR in the last report from 2023 identified only 2,831 households in the sample project areas, resulting in a projection of 3,932 households in the entire project area, highlighting a substantial reduction in the project's scope: 62.7% reduction of potentially impacted households; 74.6% reduction in potentially impacted population. According to KANTAR's insights, it is possible to deduce a significant reduction in the scope of work: a 62.7% reduction in the household scope of work and a 74.6% reduction in the people scope of work as mentioned above. Additionally, the actual number of cadastral zones covered by the KODE project was 164 instead of 266, as reported by the PIU. Many of the other villages either had no inhabitants (54 cadastral zones), were covered by MED's 2018 pilots (43 cadastral zones), or were already covered with coaxial cables (27 cadastral zones). It might also be prudent to note that the KODE project basis estimations rely on 2011 preliminary studies of MED, DELOITTE, and the 2011 population census hich doesn’t reflect the changes and evolutions that could have occurred between 2019 and 2023. Page 53 Official Use Only