Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005563 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF0A5324 ON A SMALL GRANT IN THE AMOUNT OF EUR 2.64 MILLION (US$ 3.05 million equivalent) TO THE Government of Ukraine FOR Strengthening Public Resource Management (P161586) November 24, 2021 Governance Global Practice Europe And Central Asia Region Regional Vice President: Anna M. Bjerde Country Director: Arup Banerji Regional Director: Lalita M. Moorty Practice Manager: Daniel J. Boyce Task Team Leader(s): Shiho Nagaki, Iryna Shcherbyna ICR Main Contributor: Adrienne Hathaway-Nuton, Iryna Kuzmina ABBREVIATIONS AND ACRONYMS BETF Bank-Executed Trust Fund COT Commercial-off-the-shelf CPF Country Partnership Framework GDP Gross Domestic Product EC European Commission ECA Europa and Central Asia EU European Union EU4PFM EU’s Public Finance Management Support Programme for Ukraine EURoPAF EU Programme for the Reform of Public Administration and Finances FM Financial Management HR Human Resources HRMIS Human Resource Management Information System ICR Implementation Completion and Results Report ICT Information and Communication Technology IFR Interim Financial Report IPSAS International Public Sector Accounting Standards ISR Implementation Status and Results Report IT Information Technology M&E Monitoring and Evaluation MOD Ministry of Defense MOF Ministry of Finance MOIA Ministry of Internal Affairs NACS National Agency of Ukraine for Civil Service NPSAS National Public Sector Accounting Standards PAR Public Administration Reform PDO Project Development Objective PEFA Public Expenditure and Financial Accountability PFM Public Financial Management PSA Public Sector Accounting RETF Recipient-Executed Trust Fund SCD Systematic Country Diagnostic SFS State Fiscal Service SSSCIPU State Service for Special Communications and Information Protection of Ukraine SU Spending Unit TF Trust Fund TTL Task Team Leader TSA Treasury Single Account UAH Ukrainian Hryvnia TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME .................................................................................................................... 10 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 17 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 23 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 28 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 30 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 37 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 38 ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 39 The World Bank Strengthening Public Resource Management (P161586) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P161586 Strengthening Public Resource Management Country Financing Instrument Ukraine Investment Project Financing Original EA Category Revised EA Category Organizations Borrower Implementing Agency Ministry of Finance of Ukraine, National Agency of Civil Ministry of Economic Development and Trade Service Project Development Objective (PDO) Original PDO Improving Transparency and Efficiency in Public Resources Management by: (i) Automating government personnel and payroll information; and (ii) Enhancing MOF's capacity in management of public financial information PDO as stated in Legal Agreement (if different from Project Paper) To increase capacity of Ukraine in effective and transparent management of its resources Page 1 of 40 The World Bank Strengthening Public Resource Management (P161586) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A5324 3,573,907 3,573,907 3,049,603 Total 3,573,907 3,573,907 3,049,603 Total Project Cost 3,573,907 3,573,907 3,049,603 KEY DATES Approval Effectiveness Original Closing Actual Closing 23-Jun-2017 28-Aug-2017 30-Jun-2019 31-Mar-2021 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 08-Aug-2018 0.30 Change in Loan Closing Date(s) Change in Implementation Schedule 24-Jul-2020 1.79 Change in Loan Closing Date(s) Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 21-Jul-2017 Satisfactory Satisfactory 0.00 02 14-Dec-2018 Satisfactory Moderately Satisfactory 0.34 03 09-Jan-2020 Moderately Satisfactory Moderately Satisfactory 1.29 Page 2 of 40 The World Bank Strengthening Public Resource Management (P161586) 04 27-Jul-2020 Moderately Satisfactory Moderately Satisfactory 1.79 ADM STAFF Role At Approval At ICR Regional Vice President: Cyril E Muller Anna M. Bjerde Country Director: Satu Kristiina Jyrintytar Kahkonen Arup Banerji Director: Edward Olowo-Okere Lalita M. Moorty Practice Manager: Adrian Fozzard Daniel J. Boyce Svetlana I. Proskurovska, Iryna Task Team Leader(s): Shiho Nagaki, Iryna Shcherbyna Shcherbyna Adrienne Elizabeth Hathaway- ICR Contributing Author: Nuton Page 3 of 40 The World Bank Strengthening Public Resource Management (P161586) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context Country Context 1. In the lead-up to project design, Ukraine faced important challenges. Political and economic disruption following the 2013-2014 Maidan uprising had led to a severe contraction of industrial production and a drastic fall in revenue. Real GDP fell by 6.8 percent in 2014, followed by a further 10 percent decline in 2015. Depreciation of the Hryvnia continued through 2014-2016, causing an acute deterioration in living conditions, and increases in the unemployment and poverty rates. Collective action had generated hopes and revealed demand for more transparency, justice, and accountable institutions to better respond to citizens’ needs and combat corruption. 2. At the time of design, the Government had taken steps toward stabilizing the economy and laying the foundation for stable democratic and economic development, adjusting expenditure to match the reduced revenue; implementing safety net measures; maintaining the level of performance of public sector institutions to bring normalcy to peoples’ lives; and confirming its long-term political goal of Euro-Atlantic integration. These measures helped to halt further economic deterioration and support growth of 2.2 percent in 2016 and 2.5 percent in 2017.1 However, unfinished structural reforms posed a constraint to future growth. Public sector institutions in Ukraine were not performing well. The World Bank Worldwide Governance Indicators (WGI, 2015) ranked Government Effectiveness in Ukraine at the 40.4 percentile, Regulatory Quality at 28.8, the Rule of Law at 23.1, and Control of Corruption at 14.9, with a declining trend in all areas. The World Economic Forum’s 2016 report ranked Ukraine in the bottom quintile of 140 countries on Institutions and Macroeconomic Environment, with Institutions being assessed at 3.1 of 7 points (130th out of 140 countries) and the Macroeconomic Environment receiving a score of 3.1 out of 7 points (134th rank of 140 countries). Public institutions were assessed to be particularly weak, with a rating of 2.9 out of 7. Sectoral and Institutional Context 3. The Government, formed in April 2016, had expressed its commitment to strengthening public institutions by approving a Public Administration Reform (PAR) Strategy and Public Financial Management (PFM) Strategy, to be led by the Deputy Prime Minister responsible for European and Euro-Atlantic integration. Improvements in public administration and public service delivery, as well as PFM reforms and macro-economic stabilization were also key priorities of the Government Action Plan for 2017. 4. While PFM had made some important advances by the time of project preparation, there were also significant challenges. The 2015 Public Expenditure and Financial Accountability assessment (PEFA)2 showed some progress in the area of budget execution and control, and highlighted Ukraine’s relatively strong Treasury function with an adequate control structure, access to budget execution information and improvements in investment management assisted by the amended Budget Code. However, it also noted that the budget function could be improved through alignment of policies with expenditure, expanding the budget planning horizon to the medium term, and strengthening the system 1World Bank. World Development Indicators. Databank. 2Ukraine PEFA 2015, International Bank for Reconstruction and Development: Kyiv, 2016. Accessible at: https://www.pefa.org/sites/pefa/files/2020-10/UA-May16-PFMPR-Public%20with%20PEFA%20Check-EN.pdf Page 4 of 40 The World Bank Strengthening Public Resource Management (P161586) of internal and external audit to facilitate improved operational performance and accountability. The assessment also recommended the introduction of non-financial performance information into the budget to improve accountability and clarity for citizens, and noted limited information exchange among the key fiscal management institutions and a burdensome and compliance-focused control environment to be significant constraints to effective financial management. 5. At the same time, PFM information and communication technology (ICT) infrastructure was inadequate to support modern PFM processes, with outdated budget planning software limiting the quality and timeliness of budget preparation. During the years of political and budgetary turmoil, all non-urgent expenditures, including information technology (IT) development, were terminated (including the cancellation of the portion of an International Bank for Reconstruction and Development (IBRD)-financed project3 aimed at supporting the design and installation of an integrated financial management information system (IFMIS) after two unsuccessful procurement attempts). This resulted in the deterioration of the ICT infrastructure which supported the management of the Ministry of Finance’s (MOF’s) core business processes. The MOF’s data storage capacity was reaching its limit, and backup servers in MOF were no longer operational. While the ICT systems in the State Treasury had been maintained and upgraded, declining remuneration levels had resulted in the outflow of qualified IT staff across all institutions, including the MOF and the State Treasury. At the same time, the ICT governance and service framework was fragmented. The MOF was not receiving adequate ICT support from its special purpose enterprise – GolovTechInform – and was considering a shared ICT services model to meet ICT needs for PFM in a cost-effective way. 6. There was also room to strengthen the management of the public sector wage bill, which represented a significant part of government expenditure (the overall public sector wage bill was more than 25 percent of government expenditures in 2017; 10.7 percent of GDP)4, but was managed without adequate efficiency or controls. There was no central personnel registry system, and an estimated 30 percent of state authorities had implemented various standalone ICT solutions for managing human resources information, hindering central monitoring and control of state employment and its fiscal costs. In the absence of a single automated registry, the National Agency of Ukraine for Civil Service (NACS) reconciled the number of civil servants and public employees manually through quarterly and annual consolidation of personnel data from some 130 state institutions at the central, oblast and rayon levels. These employment statistics did not form an adequate basis for effective oversight, enforcement or amendment of personnel policies or budget planning. To authorize monthly salary payments, the MOF was, in practice, reviewing personnel payment requests from Spending Units (SU) on a monthly basis and approving a lump sum payment to every SU for processing through the banking system. The whole procedure involved manual review by MOF staff and was time- consuming, inefficient and offered inadequate controls. 7. Both MOF and NACS had expressed the need for an automated registry of civil servants with an automated payroll system linked to a human resource management information system (HRMIS) for monitoring civil service employment, effectively managing civil servants, and planning and controlling personnel expenditure on state employees. This system would be critical for effective enforcement of the new Civil Service Law.5 The Law assumed that the NACS had accurate and timely information on the number, qualifications, rank, careers, performance and remuneration of civil servants. Implementation of an HRMIS and payroll system was also essential for the Government 3Public Finance Modernization Project, P090389. Original amount: $50 million; final amount: US$3.96 million 4World Bank. Ukraine Public Finance Review 2: Towards a More Effective and Fiscally Sustainable Public Sector. November 2018. < https://documents1.worldbank.org/curated/en/186591544547101839/pdf/PFR2-Ukraine-Report-1Nov2018.pdf> 5 The new Law, which became effective on May 1, 2016, depoliticized the civil service by clarifying the delineation between political and administrative functions within the public administration, and establishing merit-based principles for competitive appointment of civil servants. Page 5 of 40 The World Bank Strengthening Public Resource Management (P161586) to design a new, affordable and equitable salary structure that was calibrated so as to support the revitalization and renewal of the public administration with qualified professionals, while supporting the implementation of the national objective to cut state employment by 10 percent, and ensuring that the implementation of a recent policy allowing state bodies to trim their staffing levels and use personnel expenditure savings for salary top-ups did not lead to inequities in remuneration across the state administration. Rationale for Bank Support 8. The 2017 World Bank Systematic Country Diagnostic (SCD) for Ukraine (Report No. 114826-UA, April 28, 2017) identified three key requirements for stabilization and future growth and prosperity: macroeconomic stability; improvements in productivity and job creation; and more effective service provision to the population. Intensification of anti-corruption and governance reforms to curtail the influence of vested interests and strengthen institutions were considered prerequisites for the achievement of results in the three broad priorities identified by SCD. The Project was well-aligned with the cross-cutting focus area on governance in the Ukraine’s Country Partnership Framework (CPF) for FY2017-FY2021 (Report No. 114516-UA, June 20, 2017), within which the first objective was to promote transparency and strengthen PFM, and the Project was explicitly mentioned as an instrument of intervention. Jointly with other partners, the Bank supported the Government of Ukraine in preparing the new 2017-2020 PFM Reform Strategy, informed by the 2015 PEFA Assessment. 9. In 2016, the European Union (EU), which had supported the Government’s preparation of the 2016-2020 PAR Strategy and provided financing for its implementation, expressed interest in the Bank’s provision of technical assistance to Ukraine within the framework of the European Commission (EC)-World Bank Partnership Program for Europe and Central Asia (ECA) Programmatic Single-Donor Trust Fund (TF) (TF072780) to support Ukraine in increasing its capacity in effective and transparent management of its resources in specific directions of the PAR and PFM Reform Strategies. The technical assistance to Ukraine under this TF was to be provided as Bank and Recipient-executed activities, with the latter being contemplated under the Project. Given the strategic importance of the PFM and PAR Strategies in Ukraine, the Bank’s relevant expertise, and partnership with EU, the rationale underpinning Bank’s engagement was strong. Theory of Change (Results Chain) 10. The original Project Paper (Report No. PP2198, May 26, 2017) did not delineate an explicit theory of change for the Project, as this was not required by Bank procedures at the time of preparation. However, the logic behind the operation can be summarized as below. 11. Automation of personnel and payroll information in public administration institutions as well as the strengthening of PFM functions and systems would increase Ukraine’s capacity for effective and transparent management of public resources. The automation of personnel and payroll information was to be achieved through the procurement of an HRMIS solution, its customization and implementation in pilot institutions. Strengthening PFM functions and systems was to be achieved through a set of targeted activities supporting the 2017-2020 PFM Reform Strategy’s implementation and the preparation of a new ICT PFM Strategy. Page 6 of 40 The World Bank Strengthening Public Resource Management (P161586) Figure 1. Theory of Change Project Development Objective (PDO): 6 Increase capacity of Ukraine in effective and transparent management of its resources By: Automating government personnel and payroll information and enhancing MOF’s capacity in management of public financial information Outcomes Transparency of state employment and payroll. ICT Strategy for key PFM entities. (PDO Indicators)78 Measured as: Percentage of state entities at Measured as: ICT Strategy in PFM area defines a future the central administration whose personnel vision and implementation plans are developed and payroll data are available through HRMIS. Activity focus - Adoption of Concept on HRMIS - Carrying out an ICT Audit and preparation of an ICT PFM areas9 implementation; Strategy; - Market analysis of HRMIS/payroll solutions; - Carrying out an analysis of the current IT solution for budget - Procurement of HRMIS/payroll system; planning and developing functional requirements for - Development, testing, and implementation of upgrading the current or development a new budget planning HRMIS/payroll software; module; -Support to rollout of the HRMIS/payroll system - Upgrading current ICT equipment for business continuity in to selected entities (stakeholder analysis, MOF; normative acts, manuals, awareness raising, - Strengthening ICT systems’ capabilities to ensure timely monitoring/reporting on reform information exchange for cash management and forecasting. implementation, support group/help desk); - User training. Components Strengthening Human Resource Management in Support to PFM Strategy Implementation Public Administration Institutions Project Development Objectives (PDOs) 12. The PDO, per the Grant Agreement countersigned on June 30, 2017, was: “to increase capacity of Ukraine in effective and transparent management of its resources.� The PDO detailed in the Project Paper was slightly different, reading as follows: “improving transparency and efficiency in public resource management by: i) automating government personnel and payroll information; and ii) enhancing MOF’s capacity in management of public financial information.� For purposes of the evaluation of achievement of Project outcomes in this ICR, the PDO as defined in the Grant Agreement (as the legally binding document) is used, with the formulation from the Project Paper serving to guide the understanding of how this outcome was expected to be achieved. The PDO remained unchanged throughout the Project. 6 As set forth in the Grant Agreement, countersigned on June 30, 2017. The PDO as defined in the Project Paper (Report No. PP2198, May 26, 2017) was somewhat different (“Improving Transparency and Efficiency in Public Resources Management by: (i)Automating g government personnel and payroll information; and (ii) Enhancing MOF’s capacity in management of public financial information.� The second part of this statement is depicted in the “by� statement in the next row of the figure. 7 As set forth in the Results Framework in the Project Paper (Report No. PP2198), dated May 26, 2017. 8 As depicted here, both of the PDO indicators are viewed to reflect impacts that effect both of the Project’s subobjectives. However, in the Results Framework section of this ICR, they are each mapped to a single subobjective to avoid duplication of the indicators in the table. 9 As set forth in the Grant Agreement, countersigned on June 30, 2017. Page 7 of 40 The World Bank Strengthening Public Resource Management (P161586) Key Expected Outcomes and Outcome Indicators 13. The key expected outcomes and their indicators, as reflected in the Results Framework matrix within the Project Paper, appear in Table 1 below. (As explained below, there were slight differences in wording in the PDO indicators in the main text of the Project Paper and the Results Framework matrix.) The PDO indicators remained unchanged for the duration of the Project. Table 1. PDO Indicators (Project Paper Results Framework, Report No. PP2198, May 26, 2017) Indicator Definition and target Transparency of state Percentage of state entities at the central administration whose personnel employment and payroll. and payroll data are available through HRMIS10 (Target: 50 percent) ICT Strategy for key PFM ICT Strategy in PFM area defines a future vision and implementation plans are entities developed11 (Target: Yes) Components 14. The Project had two substantive components, as set forward in the Grant Agreement. Components were not revised during implementation. 15. Component 1: Strengthening Human Resource Management in Public Administration Institutions (EUR 1.92 million). This component aimed to support the design and implementation of the HRMIS, and was to be implemented by the NACS, and future system users at the national and subnational levels.12 Activities included: (i) support to the preparation of a software application concept design to specify functional requirements of the system, its architecture, expected system output, interface between the HRMIS/Payroll system with information systems in the MOF and other national registers; (ii) carrying out analysis to select and implement cost-effective solutions for hosting, operating and maintaining the HRMIS/Payroll system as part of system design and preparation of procurement documents; (iii) support the development of procurement documents for HRMIS/Payroll system documents; (iv) development (or adaptation, involving customization and parametrization), testing and implementation of HRMIS/Payroll software application across the targeted users, carrying out training of would-be users and data migration of basic personnel records, final testing, approval and acceptance of the system; (v) support NCSA [otherwise referred to as NACS in this ICR] to roll out HRMIS/Payroll system to selected civil service entities (according to an agreed plan) through (a) undertaking stakeholder analysis to identify change management interventions, (b) developing normative act(s) regulating the use of the HRMIS/Payroll system, including the alignment with personal data protection clauses in the 10 Outcome phrasing is slightly different, although functionally equivalent in the main text of the Project Paper: “50% coverage of the central state administration by HRMIS.� 11 Outcome phrasing is slightly different and reflects a further stage of implementation in the main text of the Project Paper: “ICT Strategy developed and approved for PFM area.� The formulation in the Results Framework matrix is considered the official in dicator for purposes of this ICR. 12 Project Paper (Report No. PP2198), dated May 26, 2017 Page 8 of 40 The World Bank Strengthening Public Resource Management (P161586) current legislation, (c) developing relevant manuals/instructions, (d) raising awareness of reforms through online platforms and other communication activities, (e) establishing arrangements for monitoring and reporting on implementation of reforms, and (f) establishing a support group and help desk to respond to user enquiries; and (vi) support training of trainers and delivery of pilot training of HRMIS/Payroll system users, as part of the system development process. 16. Component 2: Support to PFM Strategy Implementation (EUR 1.11 million). This component aimed to support the strengthening of the ICT function in the MOF, State Fiscal Service and State Treasury.13 Activities included: (i) support the implementation of the PFM Reform Strategy for 2017-2020 and identify future investments in ICT for PFM, through carrying out an ICT audit, including: (a) an inventory of all ICT applications, hardware and infrastructure, (b) an assessment of the ICT systems’ maintenance and service arrangements, (c) an assessment of alignment of ICT capacity with current and planned business requirements, and (d) an assessment of data integrity and data protection arrangements; (ii) support to: (a) carry out an analysis of existing IT solutions for budget planning from functional and information technology aspects and recommendations on improvements, and (b) develop functional requirements for upgrading the current or developing a new budget planning module with web access for budget executing entities that would enable automated consolidation of budget submission and support medium-term budgeting; (iii) support to upgrading existing ICT equipment for business continuity at MOF; and iv) strengthening ICT systems’ capabilities to ensure timely information exchange needed for cash management and forecasting, provided that MOF has determined information flow requirements in collaboration with the United States Treasury technical assistance team. Significant Changes During Implementation 17. The Project was restructured twice, in 2018 (Restructuring Paper, Report No. RES33090) and 2020 (Restructuring Paper No. RES42780), to extend the Project Closing Date. 18. 1st Extension. On August 9, 2018, following the Recipient’s request, the Project’s closing date was extended by 13 months (from June 30, 2019 through July 31, 2020) primarily to ensure sufficient time to complete HRMIS Phase 1 (Component 1). (By the time of this restructuring, it had been decided (through the adoption of a high-level Concept for HRMIS implementation on December 1, 2017) that NACS would implement the HRMIS in two phases: Phase 1, which would contemplate implementation of the HRMIS/Payroll system for core functions in up to 22 central executive bodies and be covered through the Project; and Phase 2, which would entail the development of several additional HRMIS modules and the country-wide rollout of the system, with extension options to further cover Phase 2 subject to the availability of additional funds and successful completion of Phase 1.) Specifically, the restructuring paper noted that the Project’s no-cost extension, made possible by the extension of the closing date of the EC-World Bank Partnership Program for Europe and Central Asia Programmatic Single Donor Trust Fund (TF072780) until July 31, 2020, would ensure the additional time needed to complete procurement and implementation of planned work on the HRMIS as well as allow for additional time for the procurement of consulting services to develop the functional and technical requirements for the budget planning module upgrade under Component 2. 19. 2nd Extension. On July 15, 2020, the Bank received an official request from the Recipient to extend the Project through March 31, 2021 to allow for completion of Project activities under Components 1 and 2 that had been affected by optimistic planning, institutional changes, and COVID-19 pandemic restrictions, and to support achievement of the PDO. Specifically, the restructuring paper noted the importance of the extension for providing NACS with the time to 13 Project Paper (Report No. PP2198), dated May 26, 2017 Page 9 of 40 The World Bank Strengthening Public Resource Management (P161586) finalize HRMIS deployment in the Phase 1 institutions, and MOF’s finalization of the ICT PFM strategy, completion of the development of technical requirements for upgrading the budget planning module and activities related to public sector accounting, and the launching of new activities - which, in turn, were important for the achievement of the two PDO indicators and advancing the PAR and PFM reforms. The Bank issued the extension notification on July 27, 2020. 20. Other. As mentioned above, following the Government’s 2017 adoption of the high-level Concept on HRMIS Implementation, it was agreed that the scope of the Project would support the implementation of Phase 1 of the HRMIS, and the Administration Agreement between the European Commission and the World Bank for the parent TF (TF No. TF072780)) amended to clarify that the Recipient-executed deliverable associated with the implementation of the HRMIS and payroll system contemplated the “first phase of HRMIS/payroll system for core functions in central executive bodies.�14 In practice, this led to a recalibration and reinterpretation of the original associated PDO indicator and the scope of project activities, although no restructuring was processed. Phase 2 of the work - covering the development of three remaining functional modules and country-wide implementation of the system - is planned to be implemented under the new EC-World Bank TF (TF073423) – Public Finance Management Support Programme for Ukraine Project (P172442). II. OUTCOME Assessment of Achievement of Each Objective/Outcome EFFICACY Achievement of Objective 1: Increase capacity of Ukraine in effective management of its resources Rating: Substantial 21. Advancements under the Project supporting the automation of government personnel and payroll information, and MOF’s capacity to manage public financial information are assessed to have had a substantial impact upon increasing the capacity of Ukraine in the effective management of its resources, due to substantial achievement of the objective vis-à-vis the automation of government personnel and payroll information and modest achievements towards enhancing MOF’s capacity in management of public financial information. 22. Automation of government personnel and payroll information. The Project’s objective of increasing capacity for effective management of Ukraine’s resources through the automation of government personnel and payroll information was substantially achieved. Under Phase 1 of work on the HRMIS, all core functional modules (15) grouped under the subsystems: Organizational Structure and Staff Orders, Payroll and Timesheets, Self-Service Cabinet, and Public Portal were developed, serving as the foundation for increasing effective resource use by automating previously manual processes and supporting the effective enforcement of the new Civil Service Law by enabling the production of timely and accurate HR information. These HRMIS modules had been developed, tested, and deployed in 19 Phase 1 institutions and their territorial bodies in 17 regions, covering a total of 76 entities with 20,973 end-users. This 14Amendment No. 1 to the Administration Agreement between the European Commission and the International Bank for Reconstruction and Development concerning the Parallel EC-World Bank Partnership Program for Europe and Central Asia Programmatic Single-Donor Trust Fund (No. TF072780), EU Program for the Reform of Public Administration and Finances (EURoPAF) (Donor Reference No. 385-741), countersigned on May 14, 2018, Page 10 of 40 The World Bank Strengthening Public Resource Management (P161586) represents approximately 54 percent of the users planned to be covered during Phase 1 (approximately 12 percent of total filled civil service positions), with the difference between planned versus actual Phase 1 coverage being due to the pending incorporation of the Secretariat of the Cabinet of Ministers (664) and the State Treasury Service (13,307) into the HRMIS as well as fewer than anticipated end users in the 19 institutions. Human resources (HR) related functionalities (e.g. all aspects connected with organizational structure, staff orders, personnel management, recruitment etc.) were in use in all 19 institutions, and the Payroll module in 10 (partial achievement of PDO #1).15 While at this time, many institutions are continuing to use their legacy systems for end payments, importantly, in addition to using the payroll module for payroll calculation, as was foreseen by the PAR Strategy and Project, the NACS has also been making payroll payments using the HRMIS since December 2020. Just as importantly, these technological advances have been institutionalized through NACS’s development of much of the legal and regulatory frameworks needed for HRMIS implementation, including the high-level HRMIS Development Concept, Regulation on HRMIS in government institutions, and NACS’s Order on HRMIS implementation in government institutions, which facilitated the HRMIS deployment and use in Phase 1 institutions and will support the HRMIS’s further country-wide roll-out. The legal framework to make the use of the HRMIS mandatory is still pending. 23. The automation of relevant human resources (HR) and payroll processes resulted in a number of notable benefits for managers, HR and payroll specialists, and civil servants in the institutions where the HRMIS has been implemented thus far. The creation of a centralized information database for data managed within the HRMIS (the foundation for what will become the single HR information database for the public sector once all institutions have migrated to the HRMIS); introduction of standard approaches for HR management; unification of document templates; elimination of the duplication of data entry functions both within institutions and across those which joined the HRMIS have made HR management within participating institutions more effective and efficient, thereby contributing to a more effective use of both human and financial resources. As a result of these changes, the duration of HR and payroll processes and procedures is being reduced in institutions using the system. HR related information available in the system is immediately available and searchable, and online coordination of documents requires much less time on the part of institutions using the system and reduces their use of paper. For example, within these institutions, employee profiles no longer need to be created (a process that on average takes 2.5 hours) each time an employee moves institutions as profiles are accessible through the HRMIS, facilitating their transfer between entities. Similarly, within institutions using the HRMIS, the automated preparation of a timesheet saves up to 3.5 hours and the processing of an employee application for vacation created through the self-service cabinet saves up to 3 hours. The use of paper for core HR documents has also been reduced in these institutions. This in turn is enabling their quicker and better-informed decision making; simpler monitoring; and better control over documents/task implementation, and means that the institutions using the system no longer need to separately procure and update individual HR solutions. 24. MOF’s capacity to manage public financial information. The Project’s objective of increasing capacity for effective management of Ukraine’s resources by enhancing MOF’s capacity to manage public financial information is viewed to have been more modestly achieved due to the impacts of the PFM ICT Strategy only being expected to materialize in the future and the relatively less ambitious scope of other activities completed under Component 2. 15During implementation, for this indicator, deployment of the HR and payroll functionalities were tracked within the smaller subset of Phase 1 institutions (resulting in the latest value for this indicator reported in ISR#4, archived on July 27, 2020, being 44 percent (vis-à-vis a target of 50 percent). However, as the formulation of this indicator was never officially changed to refer to Phase 1 institutions instead of the entirety of the central administration, in this ICR, achievement is assessed against the original formulation. Using such, the final value of the indicator is 12 percent (10 entities using both HR and payroll functionalities divided by 83 central administration entities. Were the (unofficially) modified formulation of the indicator to be used, the indicator could be considered almost achieved, with a final value of 48 percent (10 entities using both HR and payroll functionalities divided by 21 Phase 1 institutions). Page 11 of 40 The World Bank Strengthening Public Resource Management (P161586) 25. Under the Project, a new PFM ICT Strategy, the “Strategy for Digital Development, Digital Transformation, and Digitalization in PFM System until 2025,� which aims to build a modern and efficient information technologies management system in the PFM area, was developed and internally endorsed by the PFM institutions (MOF, State Treasury Service, State Tax Service, State Customs Service) (achievement of PDO indicator #2). The strategy and the corresponding action plan are expected to receive Government approval shortly. The strategy, which was built upon the foundation provided by the PFM Reform Strategy through 2025 (which’s development was informed by three PEFA assessments carried out by the Bank under the Bank-Executed Trust Fund (BETF) in 201916 as well as the 2021 PEFA Assessment of gender responsive public finance management (pioneered supplementary module to the National PEFA Assessment) and World Bank advisory support provided under the BETF) and informed by findings of the ICT audit conducted under the Project in 2018-2019, sets the stage for a coherent approach to ICT development in all PFM institutions. It focuses on the centralization of ICT services, interoperability between the PFM ICT systems and other systems, and cyber security issues. When implemented, it is expected to support the Government’s more effective use of its technological resources, creating a supportive enabling environment for more effective public financial management (and consequently, more effective resource use in general). It is aligned with the overall Concept of e- Governance Development in Ukraine dated September 20, 2017 and the new PFM Reform Strategy for 2021-2025 which is also expected to be approved shortly. 26. The MOF has also launched activities to strengthen its ICT capacity under the Project. The package of documents for creation of a comprehensive data protection system to enhance the cyber security of an integrated information and telecommunication system for the PFM system was developed and submitted for examination and endorsement (currently still pending) by the State Service for Special Communication of Protection of Information of Ukraine. The Project also helped to upgrade the MOF’s ICT infrastructure and supported more stable and efficient operation of IT systems through the procurement 240 workstations, which together with 600 workstations procured with the MOF’s own budget funds, allowed for the complete renewal of MOF’s computer complex. The MOF’s capacity for business continuity, staff training, and communication activities, specifically under the remote working arrangements made necessary by the COVID-19 pandemic, have been increased by procurement of modern video-conferencing equipment. Like the PFM ICT Strategy, these activities represent advances which strengthen the enabling environment for more effective PFM and consequently capacity for effective resource management. However, one of the Project’s major activities - the development of technical specifications and bidding documents to upgrade its Budget Planning IT module - was not able to be completed (only the first analytical reports were delivered) due to significant procurement delays associated in large part with the COVID-19 circumstances, with implications for disbursements under Component 2. 27. Under the Project, capacity building activities were also carried out within MOF that contribute to increasing its capacity to effectively manage resources. First, the MOF’s revenue and macroeconomic forecasting capacities were strengthened through the procurement of EViews licenses, and the delivery of training by the Bank on modelling revenue forecasting. These models have since been used to prepare inputs for the Budget Declaration for 2022-2024 that was approved by the Government and supported by the Parliament on July 15, 2021, as well as the 2021 State Budget. Second, capacity building on cash management and forecasting resulted in the development of a methodology for treasury single account (TSA) cash flow forecasting as well as a concept of liquidity management for 2020-2023 which was approved by Cabinet of Ministers of Ukraine Decree No.1375-P (November 4, 2020). Since October 2020, MOF prepares daily forecasts of the TSA cash flow for the current and following two months, and other analyses and estimations required for better cash management and forecasting function are also being carried out. Third, the Project also supported the adaptation of national public sector accounting standards (NPSAS) to international public sector 16 One at the national level and two at the subnational levels (Oblast and Rayon) Page 12 of 40 The World Bank Strengthening Public Resource Management (P161586) accounting standards (IPSAS), capacity building in public sector accounting by developing training materials on recent changes for upcoming country-wide trainings, and further improvement of the regulatory base for public sector accounting. Achievement of Objective 2: Increase capacity of Ukraine in transparent management of its resources Rating: Modest 28. Advancements under the Project supporting the automation of government personnel and payroll information, and MOF’s capacity to manage public financial information are assessed to have had a modest impact upon increasing the capacity of Ukraine in the transparent management of its resources, due to substantial achievement of the objective vis-à-vis the automation of government personnel and payroll information and modest achievements towards enhancing MOF’s capacity in management of public financial information. 29. Automation of government personnel and payroll information. The Project’s objective of increasing capacity for transparent management of Ukraine’s resources through the automation of government personnel and payroll information was substantially achieved. Under the Project, the HRMIS Public Portal was developed and launched in December 2019. The portal, which has received 12,592 visits since January 2020, provides online access to the general public on information on civil service institutions, including their composition and staffing by structural units with salary level indications; the HRMIS legal framework, user guidance etc. It also allows the public to track the HRMIS deployment process, thereby enabling the Government to more transparently manage its human resources. 30. Within each Government institution using the HRMIS, the new system also facilitates the internal transparency of HR and payroll data by providing their management, HR and accounting departments with immediate access to the system’s data, including breakdowns at the employee level and institutional summaries. It has also provided central entities with easier access to HR metrics, eliminating the need for them to issue special data requests to the participating departments. 31. MOF’s capacity to manage public financial information. The Project’s objective of increasing capacity for transparent management of Ukraine’s resources by enhancing MOF’s capacity to manage public financial information was modestly achieved, with direct benefits of the work on capacity for transparent management only expected to materialize in the future. Looking ahead, the implementation of the new PFM Strategy 2021-2025 (supported under the BETF, as mentioned above) is expected to substantially increase Ukraine’s capacity to transparently manage its resources by contributing to the transparency of its information on public finances. MOF’s capacity to execute this strategy will depend, in turn, in part on the effective implementation of the PFM ICT Strategy developed through the Project. The new PFM Strategy aims for Ukraine to score at least 65 points in the Open Budget Index in 2023, and plans for the carrying out of fiscal transparency assessments by the International Monetary Fund (IMF). Key planned actions under the Strategy that are expected to increase capacity for transparent management include: (i) the introduction of independent external assessment of the state budget revenues forecast by a Council of Independent Experts and publication of associated documents and methodologies; (ii) mandatory publication of the State Expertise Conclusions of public investment projects by key spending units; (iii) publication of state budget program passports and reports on their execution in machine-readable format by key spending units; Page 13 of 40 The World Bank Strengthening Public Resource Management (P161586) (iv) ensuring access to information on budget indicators of local budgets through the State Web Portal -the Citizens Budget; and (v) publication in open data format detailed information of local budgets, including budget forecasts and requests, budget program passports and implementation reports etc. 32. The alignment of NPSAS to IPSAS also represents a step that will make Ukraine’s public financial information more easily understandable to a broader audience, thereby contributing to increasing its transparency. However, as mentioned above, one of the Project’s major activities, which would have had an important impact on increasing capacity for transparent management of resources - the development of technical specifications and bidding documents to upgrade its Budget Planning IT module – could not be completed, with implications for disbursement under Component 2. Justification of Overall Efficacy Rating 33. The Project’s overall efficacy in achieving its objectives is assessed to be substantial, having substantially achieved its objective of increasing capacity for effective management of resources and made important but more modest advances towards achievement of its objective of increasing capacity for transparent management of resources. The HRMIS has been implemented within the scope of planned Phase 1 work, as described above, contributing to important increases in the Government’s effective and transparent management of resources. Advances related to MOF’s capacity to manage public financial information have more modestly increased capacity for effective management of resources, and future actions planned under the PFM Strategy for 2021-2025 that will be supported by the PFM ICT Strategy and the adaptation of NPSAS to IPSAS are expected to contribute to further increasing capacity for the transparent management of resources in the future, but direct impacts on capacity for their transparent management are more modest. Analysis of progress vis-à-vis the PDO indicators points to partial achievement of PDO indicator #1, and full achievement of PDO indicator #2. However, due to issues with the results framework discussed below, the additional information discussed above is viewed to provide important, complementary information on achievements that is critical to informing the overall assessment of efficacy. RELEVANCE Rating: High 34. The relevance of the PDO is assessed to be high. The CPF for Ukraine for FY2017-FY2021, which was in effect at the time of design was still in place at the time of closing, and activities as implemented supported the achievement of the CPF’s objectives under the cross-cutting focus area aimed at Better Governance, Anticorruption, and Citizen Engagement, under which the first objective was to promote transparency and strengthen PFM. The CPF itself was aligned with the objectives of Ukraine’s development strategies, which identified good governance as one of key priorities (as outlined in the Government Program and Action Plan adopted in April 2017 and sector-related Strategies, such as the PAR Strategy in place at design and extended through 2021 in 2018). At project closing, these areas remain within the Government’s reform priorities as reflected in the Government’s Action Plan 2021 and ongoing PAR and PFM Reform Strategies. Looking ahead, efforts to improve Ukraine’s effective and transparent management o f its resources are expected to remain in focus, with HRMIS rollout being a core priority of the recently adopted PAR Strategy through 2025, and the new PFM Strategy through 2025 (expected to be approved later this year), targeting further PFM system strengthening. Page 14 of 40 The World Bank Strengthening Public Resource Management (P161586) EFFICIENCY Rating: Substantial 35. The efficiency of the Project is assessed to be substantial. 36. A quantitative economic analysis was not undertaken at design, due to difficulties in assessing the costs and benefits of project activities prior to final definition of the parameters of the systems and implementation modalities, and a qualitative discussion of benefits included instead. As such, a quantitative assessment is not carried out in this ICR. 37. In regard to the Project’s internal efficiency, actual costs by project components were in line with their planned budgets and substantial achievement of its development objective achieved within a period of less than four years (including extensions) – a reasonable duration for a project working on the development of a complex system like the HRMIS. Overall, 87 percent17 of project funds were disbursed, corresponding to 98 percent of resources under Component 1 and 69 percent of resources under Component 2. The lower disbursement under Component 2 is the result of procurement related savings as well as the revision of a few activities, a reduction in the scope of work on the budget planning module update, and cancellation of a contract for goods that were not delivered by the project closing date due to COVID-19 related constraints. Staff training and planned study visits were also not feasible in view of world-wide COVID-19 conditions. Savings resulting from the procurement of the HRMIS allowed for additional work to be carried out under Component 1 (including the procurement of additional equipment for the HRMIS Data Center to address its storage capacity gap and ensure the reliable operation of the HRMIS for Phase 1 users) and the maintenance of its Project Implementation Unit (PIU) team for the 21-month extension of the Project implementation period. Similarly, savings under Component 2 associated with the cost of the awarded contract for developing technical specifications for budget planning module upgrading (and the cancellation of the activity on developing functional requirements to automate cash management processes as well as some reallocations from planned costs for trainings and financial management (FM) audits), supported the purchase of additional hardware (videoconferencing equipment) and EViews licenses; engagement of additional experts for the ICT PFM Strategy and on public sector accounting issues; and the maintenance of its PIU during the extension period. 38. In regard to other benefits resulting from the implementation of project activities, it is important to underline the time savings associated with the new HRMIS, which government employees in institutions using the system can now put to more productive use. Additionally, the NACS estimates that potential savings only on the cost of paper as a result of the transition to electronic preparation of documents may have been as high as Hryvnia (UAH) 13,800 between October 2020 and March 2021 due to savings on e-orders in 76 government institutions (Phase 1 institutions and their territorial bodies). Further savings of up to UAH 2 million (approximately US$74,400) per year could be realized in the future from paper savings on staff orders across all government bodies, while annual paper savings for other HR documents could reach up to UAH 1 million (approximately US$37,200). However, it is important to note that while legacy systems continue to operate in parallel to the HRMIS, total associated costs will be higher. Looking ahead, according to analysis carried out by the World Bank in 2018 under the BETF, in the future, the implementation of a combination of measures including the evidence-based payroll monitoring and control mechanisms enabled by the HRMIS (together with the rightsizing of staff positions, moderation of further remuneration increases, restructuring public administration compensation, and strengthening of institutional weaknesses within the public administration system) could allow the Government to keep the civil servant wage bill below 1 percent of GDP by 2025 while retaining and attracting qualified 17Calculated based on information denominated in the holding currency (euros), as reflected in Client Connection, which is slightly different than the US$ denominated information appearing in the ICR datasheet. Page 15 of 40 The World Bank Strengthening Public Resource Management (P161586) staff.18 Overall Outcome Rating Rating: Moderately Satisfactory 39. The overall outcome rating for the Project is assessed to be moderately satisfactory. As presented above, the Project substantially achieved its development objective of increasing the capacity of Ukraine in effective and transparent management of its resources. The subobjective on effective management of its resources was substantially achieved due to substantial achievements related to the HRMIS and more modest achievements in MOF’s capacity to manage public information, while the subobjective on transparent management of its resources was modestly achieved in light of substantial impacts of the HRMIS related work but only modest, indirect impacts associated with activities on MOF’s capacity to manage public financial information. The Project’s objectives remain of high relevance for both the Government and the Bank, with the Project being highly-aligned with the first objective of the CPF’s cross cutting governance focus area. It is considered to have demonstrated substantial efficiency despite implementation taking 21 months longer than expected, taking into account its substantial achievements using 87 percent of project funds within a period of less than four years and savings that the HRMIS system is already starting to generate in terms of staff time and paper use. Table 2. Calculation of the Overall Project Outcome Rating Outcome Component Rating Relevance High Overall Project Efficacy Substantial Objective 1 Increase capacity of Ukraine in effective management of its resources Substantial Objective 2 Increase capacity of Ukraine in transparent management of its resources Modest Efficiency Substantial Overall Outcome Rating Moderately Satisfactory 18 World Bank. Ukraine Public Finance Review 2. November 2018. Page 16 of 40 The World Bank Strengthening Public Resource Management (P161586) Other Outcomes and Impacts 40. The Project has made important contributions to the institutional strengthening of the NACS, the government bodies who joined the HRMIS, and the MOF. 41. Through the process of project implementation, NACS staff acquired skills (technical and administrative) in implementing and maintaining complex IT solutions within civil service bodies, which it is now leveraging to provide support to system users, and sharing its experience in developing and using HRMIS with other countries in the region. Additionally, as mentioned above, the legal and regulatory structures for human resources management within the public sector were strengthened through NAC’s development of much of the legal and regulatory framework needed for HRMIS implementation, including the high-level HRMIS Development Concept, Regulation on HRMIS in government institutions, and NACS’s Order on HRMIS implementation in government institutions, which facilitated the HRMIS deployment and use in Phase 1 institutions and will support the HRMIS’s further country-wide roll-out. As mentioned above, the legal framework to make the use of the HRMIS mandatory is still pending. 42. Capacity to effectively use the new HRMIS was also strengthened through the delivery of special trainings on working with HR (726 people) and Payroll (357 people) functionalities of the HRMIS (based on user manuals that are available on the HRMIS public portal) to the administrative users of all Phase 1 institutions at the central and territorial levels, in a training-of-the-trainers approach. The trained administrative users now instruct staff (HRMIS end-users) within their institutions on the use of the HRMIS. 43. MOF’s capacities in revenue and macroeconomic forecasting, public sector accounting, and cash management and forecasting were also strengthened through the Project, including support to the upgrading the status of liquidity management unit within the MOF’s organizational hierarchy. 44. The Project also supported the creation of capacity within the NACS and MOF in implementing Bank-funded operations. This will be especially beneficial for the NACS in view of the upcoming follow-up EC-World Bank Project covering Phase 2 of HRMIS implementation. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME KEY FACTORS DURING PREPARATION 45. Strong alignment with sustained Government and donor priorities, and provision of associated technical support. The Project, which was designed as a RETF accompanied by a BETF, under the EC-World Bank Partnership Program for Europe and Central Asia Programmatic Single-Donor Trust Fund – EU Programme for the Reform of Public Administration and Finances (EURoPAF), benefitted from strong alignment with Government and donor priorities. As mentioned above, it was developed to support the Government's implementation of its 2016-2020 PAR Strategy (dated June 24, 2016) through HRMIS development and implementation, and the 2017-2020 PFM Strategy (dated February 8, 2017) that MOF had prepared with the support of development partners, including the EU and World Bank. As a result, project objectives were highly consistent with long-term Government priorities and the World Bank’s and the EU’s prior engagement in the country. This alignment and the analytical support provided by the accompanying BETF helped to ensure a strong Page 17 of 40 The World Bank Strengthening Public Resource Management (P161586) analytical base underlaid planned activities, and the maintenance of sustained support for the work over the subsequent implementation period. 46. Project preparation also benefited from the provision of advisory and consultative support by the Bank to NACS before the signing of the corresponding Administrative Agreement between the EC and the Bank. This support was important for the generation of knowledge within the NACS team on similar systems, helping to share good international experiences; facilitating engagement with experts developing HRMIS in other countries, including the EU; and enabling the preparation of an Options Paper by the Bank on HRMIS which informed the Government’s decision on HRMIS implementation options. It also supported the early establishment of interagency working groups composed of HR specialists, accountants and IT specialists to decide on requirements for the future HRMIS, and helped the NACS to gain familiarity with the rules and procedures applicable to Bank-funded projects before the Project’s official launching. As a result, by the time of Project effectiveness, the NACS had a good understanding of its expectations for the upcoming system, potential options for its implementation, and draft high-level requirements for the HRMIS. 47. Insufficient Readiness for Implementation of the HRMIS. At the time of design, there was not yet a final decision on the option for the HRMIS solution. This situation was complicated by the fact that available funding was not sufficient for full-scale design and implementation of the HRMIS, as well as implementation timing issues (mentioned below). As a result, it was decided to use a phased, incremental approach for the HRMIS’s functional development and institutional roll-out, and during the course of the project preparation, four potential options for HRMIS system design and implementation were considered to identify the most appropriate approach given the circumstances (cost, implementation period, maintenance costs etc.). To facilitate the Government’s decision, NACS carried out market research on available HRMIS solutions to better understand potential costs, overall implementation timing, and the level of required adaptation in the event of the selection of a commercial-off-the-shelf (COTs) solution, and developed detailed technical specifications for the system. This cost information helped to indicate the potential amount of additional financing expected to be needed to cover the complete functional development of the HRMIS and its country- wide roll-out. 48. Unrealistic Implementation Period. The original project implementation period as set at design (two years) was limited by the duration of the EC-WB TF (TF072780), which meant that RETF activities had to be completed by June 30, 2019. While for PFM related activities the timing was quite appropriate, the timing for the whole cycle of HRMIS design and country-wide implementation was very ambitious. Reflecting this, as discussed above, it was decided to proceed with a phased approach under which only core module development and implementation in selected central government entities would be undertaken through the Project and the original closing date extended by 13 months. Nevertheless, even after doing so, optimism persisted in regard to the implementation timeline vis-à-vis the Phase’s more limited scope, as underlined by the need for a second extension of the closing date for an additional nine months. 49. Good Mitigation of Identified Risks, But Some Unidentified Risks. Potential risks and relevant mitigation measures related to stakeholder resistance and implementation capacity were identified during preparation and subsequently monitored. Significant risks related to the potential resistance of line institutions to the centralization of information on staffing and payroll and standardizing relevant procedures were proactively addressed through a variety of measures. A strong policy dialogue between the Secretariat of the Cabinet of Ministers, the Cabinet’s Reform Office, the Bank and the EU Delegation was established in order to ensure high-level commitment to implementation, thereby helping to reduce resistance from entities operating their own individual systems. Additional actions to provide high- level support include the establishment of the Coordination Council on the Issues of Public Administration Reform (established by the Cabinet of Ministers of Ukraine’s Resolution dated May 18, 2016, # 335), chaired by the Minister of Page 18 of 40 The World Bank Strengthening Public Resource Management (P161586) the Cabinet of Ministers of Ukraine, which considered HRMIS – related issues at its meetings and provided high-level guidance and endorsement to the approach, and NACS’s establishment of a Steering Committee of senior representatives, including the Secretariat of the Cabinet of Ministers and stakeholder institutions, to oversee the HRMIS design and implementation and endorse HRMIS related decisions. At the operational level, NACS set up inter-agency working groups to work on functional and technical requirements of the HRMIS to ensure that they met the needs of user organizations and facilitate testing and further implementation. 50. The implementation capacity of the implementing entities was considered another area of high risk at the time of design, specifically in view of the Project’s focus on IT-related aspects and weak IT capacities within the public administration due to the below market level salaries for IT specialists in the government. To help to mitigate this risk, in addition to filling vacant positions and hiring project-funded consultants, project implementation team specialists from reform support teams supported through an EU-funded Reform Support Program implemented by the Secretary of the Cabinet of Ministers of Ukraine were also engaged in the initial stages of preparation and implementation, in line with recommendations from the Bank and the EU Delegation. 51. However, design did not adequately identify all of the risks related to the Project’s technical design. While the Project Paper noted the difficulties that the pending definition of the system’s design parameters posed for costing, resistance to change, and potential data quality issues, it did not flag risks related to the original scoping of the Project (covering the entirety of HRMIS implementation) within a short time period (a non-negotiable at preparation). These shortcomings, reflected in technical readiness for implementation and the unrealistic implementation period, required extensions of the Project closing date. 52. The COVID-19 pandemic, which started in 2020, also ended up posing unexpected risks, with the accompanying lockdown and remote work arrangements requiring additional time to restart Government operations and project implementation. However, given the unprecedented nature of the pandemic, such risks could not have been reasonably anticipated during preparation. KEY FACTORS DURING IMPLEMENTATION Factors that Facilitated PDO Achievement 53. Phased Implementation approach to HRMIS. As mentioned above, the final decision to proceed with a COTS solution for the HRMIS under a two phased approach was concurred by the HRMIS and PAR Steering Committees following the initiation of implementation. The COTS approach was justified by a number of factors, including functional coverage that was generally standard and the results of NACS’s 2017/2018 HRMIS market analysis which showed that existing HRMIS COTS met about 80 percent of government requirements for the HRMIS; the possibility of previous testing; established project management procedures; the diversity of vendors in the international and local markets; local presence; warranty and support arrangements; and documentation. 54. The Project could cover only partial HRMIS development and implementation (Phase 1) in terms of the timeline and cost. The Government decided that Phase 1 would cover the development of all core HR and payroll functionalities as well as the Self-Service Cabinet and Public Portal, and their implementation at the ministerial level and in the NACS, E-Gov Agency, and the Secretariat of the Cabinet of Ministers of Ukraine. As a result of savings on the cost of Phase 1, it was possible to procure additional storage equipment for the HRMIS Data Center. Page 19 of 40 The World Bank Strengthening Public Resource Management (P161586) 55. NACS’s Proactivity. In order to closely follow implementation progress and quickly detect potential issues, the NACS established daily monitoring arrangements with the HRMIS supplier on progress towards each aspect of HRMIS contract implementation and system deployment in the Phase 1 institutions. This allowed NACS’s project management team to expeditiously take action to resolve issues; adjust the HRMIS deployment schedule as needed; and follow up with stakeholders on any issues they encountered and arrangements for their resolution. 56. During implementation, as anticipated in the Project’s original risk assessment, resistance to the introduction of the new centralized HRMIS was encountered from government bodies who were already using separate automated solutions for their HR and/or payroll functions. To address this, the NACS, with advisory support from the Bank, engaged in extensive communication with Phase 1 institutions and issued important regulations needed to institutionalize the system’s use. The communication aimed to demonstrate NACS’s own progress in using the system, including for salary payment, and raise awareness of HRMIS certification for comprehensive data protection and the adoption of HRMIS normative acts regulating the system’s use in public institutions. The issuance of the Cabinet of Minister’s Resolution “On approval of Regulation on HRMIS in government bodies� #1343 (December 28, 2020), which established the official status of the HRMIS in government bodies, and NACS’s follow-up Order “On implementation of HRMIS in government bodies� #6-21 (January 16, 2021), which approved several templates associated with HRMIS implementation (i.e. bilateral agreement between NACS and a government entity to join the HRMIS; entities concurrence for personal data processing, and entity’s application for organization of its access to the HRMIS) represented a critical point in addressing resistance to change, with most institutions subsequently accepting its implementation. However, in spite of these efforts, some pockets of resistance to change persisted, and two of the institutions planned for Phase 1 (the Secretariat of the Cabinet of Ministers and the State Treasury Service) had not joined the HRMIS as of Project closing. 57. Agile, Inclusive Approach to HRMIS Procurement. The arrangements for preparation of the bidding documents for the HRMIS – with support from a group of three individual consultants (on HR, payroll and IT systems) instead of a company - turned to be an effective approach, both cost and time-wise that also helped to ensure Government ownership of the planned system by drawing upon deep involvement by NACS and other stakeholders. Development of the bidding documents with technical requirements for the system took about four months, including coordination of the technical requirements within the inter-ministerial working groups set up by the NACS, and revisions following the Bank’s comments. To ensure the transparency of the bid evaluation and contract award process, as well as to strengthen its capacity, NACS included representatives from key entities, including the MOF and the former Agency for E-Governance of Ukraine (e-Gov Agency; now the Ministry of Digital Transformation) in the bid evaluation team, and representatives of the Cabinet’s Reform Office and a top manager of the E-Gov Agency participated in NACS’s bidding committee’s meeting to endorse the contract award decision. These and other measures contributed to a successful procurement process, in a country and area (HR) that are known for having particular procurement challenges. 58. Strong Bank advisory support. In light of the Project’s short implementation period, the Bank team’s provision of significant advisory and implementation support, including outputs produced under the BETF, played an important role in supporting implementation. Through the BETF, the Bank team delivered important analytical inputs and capacity building support to the Government throughout implementation. These inputs included, among others: an Options Paper (mentioned above), which informed the Government’s decision to design the HRMIS and implementation arrangements; a guidance note on revenue forecasting; and recommendations on the scope, structure, and details of the ICT Strategy based on international experience and the new realities that emerged from the COVID-19 pandemic. This support was particularly important for the successful implementation of the HRMIS, as NACS’s capacity, experience, and knowledge to implement such complex IT systems and manage Bank-funded operations was weak. Page 20 of 40 The World Bank Strengthening Public Resource Management (P161586) Factors that Presented Challenges to Smooth Project Implementation 59. Ongoing reform of civil service and continuous reorganization of governmental ministries. The reorganization of government bodies launched in the second half of 2019 resulted in changes in both the number of ministries to be covered by Phase 1 and the number of staff within these ministries. At the time that the HRMIS contract was launched, there were 18 ministries in Ukraine. By the time of project closure, there were 20 ministries, 9 of which had been established over the implementation period either as a result of the merging or separation of existing ministries and the creation of new ones, including two which were established in early 2021. Several ministries underwent both merging and then separation within the course of a year and a half period. These reorganization processes resulted in the continuous adjustment of the HRMIS’s list of institutions and impacted the time that reorganized ministries had available for HRMIS deployment procedures. 60. Additionally, changes within the PFM institutions – specifically, the merging of relevant ICT functions of the MOF, the State Treasury Service, the State Tax Service and the State Customs Service under implementation support of one state-owned enterprise – affected the timing of the preparation of ICT PFM Strategy, as the MOF had to first prepare the Concept on ICT consolidation in PFM area (approved by the Cabinet of Ministries in July 2019) and then apply relevant organizational aspects towards that. The Government’s decision to establish the Agency on State Debt Management (not yet operational) also affected the planned activities on cash management. As a result, the Project focused on capacity building in cash management and forecasting; the development of functional requirements for automation of cash management processes was removed from the activities’ plan, with the MOF planning to resume work in that area once the Agency is operational. The decision to split the State Fiscal Service into the State Tax Service and State Customs Service in December 2018 also affected the implementation of the key activities such as conclusion of the ICT audit and the launch of the budget planning module. 61. HRMIS coverage of institutions falling outside of the Civil Service Law. During project implementation, in addition to the institutional organization process discussed above, some changes were made to the HRMIS’s institutional coverage due to the evolution of the interpretation of legislation associated with Ukraine’s different payment/legal regimes. 62. The HRMIS was designed to cover (i) all public institutions subject to the Law on Civil Service at the national and subnational levels; and (ii) both civil service positions and non-civil service positions in those institutions, including political appointments, state secretaries, patronage, support services etc. that are subject to Labor Code of Ukraine, but not positions that are subject to specific legislation with different payment/legal regimes (e.g. by Law on Police, Law on Military Duty and Service, Laws on Justice, etc.). As such, while the original list of ministries to be covered by Phase 1 included, among others, the Ministry of Internal Affairs (MOIA) and the Ministry of Defense (MOD), where most staff positions are subject to other legislation and have secret status (and in the case of the MOD, the entirety of its structure is classified as a secret) (thus requiring a higher level of data protection), coverage of these ministries for NACS’s monitoring and analytical purposes was planned only to include civil servants via an interoperability solution with these ministries’ HR systems. 63. In view of the EU Delegation’s request for the extension of the HRMIS’s coverage to all staff positions outside of the civil service (including the National Police, Border Control Service, justice sector, and others), the NACS/EU Delegation/Bank engaged in a consultation process during 2019. This resulted in the definition of the current approach to the HRMIS institutional coverage during system roll-out, including for Phase 2 as described in the EC-Bank’s Page 21 of 40 The World Bank Strengthening Public Resource Management (P161586) Administration Agreement for TF073423.19 To ensure that the HRMIS is used as a valuable management tool, it was determined that it should cover all staff positions in institutions independent of the legal regime that their employment falls under. As such, it was agreed that first, the HRMIS would be rolled out to institutions with 100 percent coverage of staff positions (Phase 1 and Phase 2a). After that, HRMIS coverage could be extended to institutions with mixed staffing, including staff not regulated by the Civil Service Law and/or Labor Code, such as MOIA or MOD (Phase 2b). Due to exclusion of the MoD and MoIA from the Phase 1 list and considering requests received from several institutions from Phase 2, three other central executive bodies (both headquarters and their territorial branches) were moved to Phase 1 – the State Property Fund, the State Service of Ukraine on Medicine and Drugs Control, and the State Treasury Service. 64. Technical challenges. Technical challenges, including the need for more time for adaptation of the procured solution than expected in view of changes in the HR/payroll related legislation, non-unified processes across government institutions, as well as a number of bugs that emerged during the HRMIS’s trial operation and gaps in the software developer’s capacity, contributed to implementation delays. Additionally, the data migration process was complicated by data quality issues in a number of institutions, and required substantial efforts and more time than planned within the COVID-related working environment, as did the need to provide a more substantial system of training to human resources staff on the use of the HRMIS, complicating rollout. COVID related work restrictions also led to institutions’ delays in updating the full set of HR related information in the HRMIS, and prevented the timely launch of the payroll module’s trial operation. As a result, by project closing, the payroll module was operational in only 10 institutions, covering 3,469 users. 65. Planned activities to support the upgrading of the budget planning module to enable automated consolidation of budget submission and medium-term budgeting through development of detailed functional requirements for the new system were only able to be partially completed, due in large part to several changes in MOF’s management (and related project management) and COVID-19 circumstances contributing to procurement and implementation delays which shortened the implementation period, and some limitations in the selected provider’s capacity to fully complete the work during the remaining time as well as delays in approval by the Parliament of amendments to the Budget Code on the introduction of medium-term budget planning (occurred in December 2018), without which it was not expedient to launch the work on the budget planning module. The associated business process review which was carried out is now being used by the MOF as an input to continue planned work under EU4PFM Program support. 66. Changes in leadership. Several changes within the NACS’s top management (the head of the NACS changed three times as changes to the Cabinet were made, and IT Department management also changed) and the project team posed challenges to HRMIS implementation and affected the timing. However, the appointment of the latest NACS’s Chair at the end of June 2020 resulted in a notable acceleration of the HRMIS deployment process in Phase 1 institutions, due to improvements in communication between NACS and other government institutions, adoption of needed HRMIS related acts, and the commitment of the NACS team to complete the Phase 1 as planned. Changes in top management (MOF’s project coordinator, appointed at the level of deputy minister, also changed three times) and the project implementation team also contributed to delays in Component 2’s implementation due to the time required by the new team to understand project implementation procedures, lengthy decision making, and project management challenges between end 2018 and early 2020. However, starting in mid-2020, implementation picked up and savings accumulated to date were used on a number of new activities related to the ICT PFM Strategy’s preparation and implementation ; 19Administration Agreement between the European Commission, on behalf of the European Union, and the International Bank for Reconstruction and Development and the International Development Association concerning the European Commission-World Bank Partnership Programme Part III for Europe and Central Asia Programmatic Single-Donor Trust Fund (TF073423)/Public Finance Management Support Programme for Ukraine (EU4PFM)- Component 2 Activities dated October 16, 2020. Page 22 of 40 The World Bank Strengthening Public Resource Management (P161586) approximation of Ukrainian PSA to international standards; and the strengthening of MOF’s macroeconomic forecasting capacities and videoconferencing capacity. 67. MOF Procurement Challenges. Activities under Component 2 experienced some delays due to longer than expected procurement procedures, with particular delays observed at the activity decision making, evaluation and contract signing stages, and the COVID-19 lockdown which required MOF staff to work remotely. As a result, one selection process took 13 months (leaving only 5 months for implementation until the second extension of the Project’s closing date). Additionally, contract signing procedures tended to be lengthy, with some taking several months following the Bidding Committee’s decision. Strengthening of MOF’s implementation team in mid-2020, including the appointment of a new Project Coordinator, subsequently helped to improve and speed up internal procedures and twelve procurement processes were conducted between October 2020 and February 2021. 68. COVID-19. The COVID-19 pandemic had a substantial impact on the timely completion of project activities, resulting in the second project closing date extension. Though both the NACS and MOF adjusted their implementation arrangements to the new realities posed by the pandemic, activities such as planned study visits and some face-to-face trainings needed to be cancelled. Furthermore, as a result of the pandemic, the whole HRMIS deployment process – including user training within Phase 1 institutions had been carried out remotely. This adjustment required additional efforts and created challenges for the NACS, the HRMIS supplier, and Phase 1 institutions (a number of which were not ready for the deployment during the quarantine period). Within Component 2, remote-mode arrangements facilitated the delivery of additional support to the implementation of ICT PFM Strategy and public sector accounting (PSA) reform, MOF’s macroeconomic forecasting capacity and video-conferencing capacities, as well as FM support to the Project. One contract, however – for data storage hardware – needed to be cancelled by the MOF due to non-delivery by the project closing date that was caused by COVID circumstances, with the equipment stuck in an airport in transit to Ukraine. 69. Unexpected complications beyond the control of the MOF. The auditing of key PFM institutions’ ICT systems encountered complications during implementation due to concerns raised by the State Service for Special Communications and Information Protection of Ukraine (SSSCIPU) and the Security Service of Ukraine regarding the disclosure of security aspects of government bodies’ IT systems to external consultants. This put the implementation of the ICT Audit on hold for six months until an acceptable compromise solution (in which the audit of infrastructure sustainability and security review of PFM ICT systems were taken out of the scope of the firm’s contract; SSSCIPU itself audited these elements and provided results directly to the MOF, SFS, and the State Treasury Service) was able to be reached between the MOF, SSSCIPU and the firm. This unexpected delay contributed to the shifting of the timing of the preparation of the ICT PFM Strategy. 70. Direct Payment Challenges. Additionally, the temporary lack of the ability to make direct payments in Hryvnia (UAH) under Euro-denominated trust fund at the beginning of the project implementation (which became feasible in July 2019), delayed the MOF’s payment to a supplier for the delivered equipment (workstations) for about one year. As the MOF had to proceed via its Designated Account (which required a budget line for payment instead of post-budget recording of direct payments), this resulted in a lengthy procedure of making amendments to the 2019 State Budget to foresee relevant funds for this purpose. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME QUALITY OF MONITORING AND EVALUATION Page 23 of 40 The World Bank Strengthening Public Resource Management (P161586) M&E Design Rating: Modest 71. As mentioned above, the Project did not have an explicit theory of change, as one was not required at the time of preparation. The Project’s results framework, as reflected in the Project Paper, built from the Indicative Results Framework indicators within the EC-World Bank’s Administrative Agreement of the parent trust fund. The Project’s indicators focused on the “by� part of the PDO as specified in the Project Paper – e.g. “by automating government personnel and payroll information and enhancing MOF’s capacity in management of public financial information� – and did not clearly capture either the first part of the PDO as specified in the Project Paper (“improving transparency and efficiency in public resource management�) nor the PDO as specified in the Grant Agreement (“to increase capacity of Ukraine in effective and transparent management of its resources."). As mentioned above, given this challenge, this ICR assesses project outcomes against the PDO as specified in the Grant Agreement as the overarching legal document, using the PDO and the PDO Indicators as specified in the Project Paper to inform how these changes in capacity for more effective and transparent management of resources were expected to be achieved. 72. Both the PDO and intermediate-level indicators as specified in the Project Paper were specific, measurable (although heavily reliant on yes/no measures that do not facilitate a more nuanced tracking of progress), and relevant, with easily established baseline and annual targets. Generally, the indicators had achievable targets. It is also important to note, however, that HRMIS implementation targets were not fully consistent in wording and timing among different strategic documents, including the PAR Strategy, CPF, HRMIS Implementation Concept, EU Budget Support for PAR with HRMIS coverage as one of conditionality for macro-financial support, which created some complexities. M&E Implementation Rating: Substantial 73. During implementation, the implementing entities compiled information on the progress of results indicators based on their own regular monitoring of relevant contracts’ implementation, which was shared with the Bank and the Donor through bi-weekly/monthly progress monitoring updates and semi-annual progress reports, and implementation support meetings and missions. The Bank team monitored and assessed implementation using this information, engaged in continuous dialogue with the Recipient’s management and implementation teams, and reported on such through annual Implementation Status and Results Reports (ISRs) and semi-annual progress reporting to the Donor. The implementing agencies also reported on progress towards achievement of project objectives during Project Steering Committee meetings. 74. One challenge associated with M&E implementation arose following the focusing of the Project on a phased implementation approach. As designed, PDO indicator #1 was structured so as to measure progress against country- wide implementation of the HRMIS (instead of implementation within a subset of those – e.g. Phase 1 institutions identified following the decision to follow a phased implementation approach). Reflecting the phased approach, during project implementation, progress on PDO indicator #1 was measured vis-à-vis Phase 1 institutions. However, the PDO indicator was never officially adjusted during the Project’s restructurings to reflect this change. While the lack of restructuring prevents the formal achievement of the PDO indicator, given that the indicator tracked served the purpose of monitoring the HRMIS’s implementation progress within the set of institutions agreed upon by the Government and the Bank, the implications from a monitoring perspective are limited. Page 24 of 40 The World Bank Strengthening Public Resource Management (P161586) M&E Utilization Rating: Substantial 75. M&E data on project performance and results were used to follow implementation progress, detect issues, and advise the implementing agencies on actions to take to improve progress. The introduction of bi-weekly/monthly monitoring updates allowed the Bank and the Donor to monitor the project implementation in real time and react in a timely manner. This reporting served as a valuable tool for the implementing agencies, by highlighting outstanding issues and supporting the identification of solutions for their resolution. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 76. The Project complied with all applicable Bank policies and was consistent with the relevant Operational Policy Guidelines, and did not trigger any safeguard policies at design. No environmental or social issues were subsequently identified during implementation. 77. FM performance was consistently rated as satisfactory throughout project implementation for NACS, and as satisfactory or moderately satisfactory for MOF in view of some delays in FM reporting in 2019. All disbursements were made in line with the provisions of the World Bank Disbursement Guidelines for Investment Project Financing dated February 2017 and the project disbursement letters dated June 26, 2017 and January 24, 2019. The Recipient’s implementing agencies prepared semi-annual interim financial reports (IFRs) and organized annual audits of Project financial statements. The NACS submitted IFRs and audit reports in a timely manner throughout the Project. The MOF experienced substantial delays with IFRs and Designated Account reports in 2019, which was associated with changes in the MOF’s project team. Annual audit reports for 2017, 2018 and 2019 were also submitted with few-day delays. During implementation, the Bank team strongly encouraged MOF’s management to improve the Project’s financial reporting. In May 2020, the MOF addressed delays and resumed timely reporting, and hired an external FM consultant to support the substantial increase in project contracts under implementation in the final months before closing. Project closing activities, final payments, refunding of unused advances, and final audits were carried out smoothly by both implementing agencies. 78. Procurement processes were carried out in line with the provisions of the Grant Agreement and Project Procurement Plans as cleared by the World Bank. No specific procurement compliance issues were raised during project implementation. As mentioned above, in several cases, lengthy internal decision-making processes in MOF resulted in extremely long procurement processes, with implications for the timing of implementation. Implementing agencies followed the Bank’s recommendations on needed improvements to procurement processes and contract administration informed by the Bank’s annual procurement post reviews. BANK PERFORMANCE Quality at Entry 79. Collaboration with Government counterparts and relevant support from the Bank’s side was launched immediately after the EC-World Bank agreement on the then potential project in second half of 2016. The Bank team had the adequate skills mix needed to support design, including expertise in public financial management, public sector human resources management (and specifically HRMIS), and digital government approaches. Early cooperation and support helped the Bank team to establish a solid rapport with Government counterparts. This enabled the team to Page 25 of 40 The World Bank Strengthening Public Resource Management (P161586) identify: (i) risks, (ii) specific preparatory activities important to be ready for implementation to the extent possible, and (iii) implementation approaches that were feasible and that would also strengthen implementing entities’ capacity and meet fiduciary requirements for Bank-funded operations. 80. The Bank’s extensive preparatory work and support to the Project through the parallel BETF, including the organization of knowledge sharing activities for counterparts, provided important support for the implementing agencies’ work on project preparation, and helped them to be ready for effectiveness and a quick launching of implementation. This was particularly important for the NACS, which did not have any experience in implementing either Bank-funded operations or complex and comprehensive management information systems, such as the envisaged HRMIS. The Banks’ 2017 analytical report on HRMIS implementation options (delivered under the BETF),20 reflecting international best practices and country-level circumstances, helped to inform the Government’s decision on its HRMIS approach. However, this was not sufficient to address issues of a very short and unrealistic implementation period vis- à-vis an absence of a decision on how to approach the HRMIS’s roll-out and lack of clarity on the scope/coverage of entities under the Project from the beginning that together contributed to significant delays, challenges which should have been better anticipated given the Bank’s experience in implementing similar projects in other countries. 81. That being said, the Bank brought important lessons learned on institutional coordination, change management, and internal implementation arrangements from earlier operations and projects21, which were incorporated into design - for example, through setting up the Project Steering Committee and making HRMIS progress subject to key decisions and implementation support by the Government’s PAR Steering Committee in order to cover high-level policy and strategic management among key stakeholders, external coordination, and accountability. 82. The Bank closely coordinated with the Secretariat of the Cabinet of Ministers of Ukraine and its Reform Office on all project preparation aspects to ensure the Government’s top management’s attention and support to project implementation via specific Government decisions that might be required. 83. Design could have benefited from closer attention to ensuring consistency in the phrasing of the PDO across project documents (e.g. Grant Agreement and Project Paper), as well as further consideration of the indicators needed to demonstrate its achievement. Quality of Implementation Support 84. The Bank provided intensive implementation support and conducted frequent and regular implementation support missions, and as mentioned above, delivered a number of important analytical inputs and capacity building support activities through the parallel BETF. Implementation support missions were carried out three times per year during 2017-2019, and twice during the final year of project implementation (including one remote mission due to COVID-19). These missions were complemented by more focused technical missions on specific issues, like HRMIS or PSA, and follow-up meetings to discuss progress on agreed-upon steps and specific implementation issues. Starting in September 2020, the Bank organized bi-weekly virtual meetings with the NACS team to monitor progress with HRMIS deployment and provide advice on issues related to HRMIS contract implementation. Timely procurement support was delivered through implementation support missions, regular annual procurement post-reviews, and prior review of two packages, and comprehensive financial management implementation support was conducted on a regular basis, with regular follow-up facilitated by the local presence of the responsible FM specialist. 20 World Bank. Ukraine: Implementing a Human Resource Management Information Systems (HRMIS) in the Public Administration. Options Paper. July 7, 2017. 21 ICR for Public Finance Modernization Project (P090389). Page 26 of 40 The World Bank Strengthening Public Resource Management (P161586) 85. While the consistent presence of a country-based co-Task Team Leader (TTL) and a local team helped to ensure continuity in the Project’s implementation support, frequent changes in the main TTLs contributed to instability in the project management and relationship with both EU and the Government (the Project had four TTLs over 3.5 years). As continuity in leadership has been recognized in the evaluations of many other Bank projects as essential for the effectiveness and efficiency of implementation support, the Bank’s management could have more carefully managed the appointment of main TTLs, taking into account their career perspectives. However, the maintenance of a local team with previous experience managing Bank’s lending operations to handhold the Recipient’s implementing agencies on project implementation issues and arrangements on a daily basis throughout the project implementation period played an important role in supporting implementation. This approach helped to enhance the Recipient implementing teams’ knowledge of Bank rules and procedures, strengthened Bank implementation support by providing the team with more detailed insight to the realities of project implementation, and created an easily accessible resource for implementation teams to use for minor operational issues and questions. 86. During implementation, the Bank team generally maintained a good working relationship with the EU Delegation team, keeping them up to date on project developments, sharing draft terms of reference and outputs, and coordinating on new activities and issues as they arose. Regular meetings with the EU Delegation were organized to provide briefings on progress and implementation issues, and the Bank team also engaged the Donor for meetings with the Recipient’s implementing agencies. As part of the Project Governance arrangements for the overall EURoPAF TF, the Bank and Donor also held regular Project Steering Committee meetings which were attended by the representatives of the Recipient and its implementing agencies. These meetings, which were arranged to align with implementation support missions, provided a higher-level platform for project implementation support with the Donor, and were important for agreeing on work programs, reviewing progress, providing strategic guidance, and endorsing funding priorities. Initially, these meetings were planned to take place on a quarterly basis, but after an initial meeting in 2017 and one in 2018, during 2019-2020, they were held semi-annually. This close collaboration between the Bank, the EU Delegation and the Government helped the stakeholders to find solutions to issues as they arose, including the phased HRMIS implementation approach (and arrangements for Phase 2 funding) and to address MOF’s requests for additional actions under the PFM Component. Justification of Overall Rating of Bank Performance Rating: Moderately Satisfactory 87. In light of the extensive preparatory work, as well as the proactive and responsive support provided by the Bank team during design and implementation, but design challenges related to the original unrealistic implementation period and lack of clarity on the scope/coverage of entities covered by the HRMIS which could have been better anticipated and the frequent changes in the main TTLs during implementation, overall Bank support is assessed to be moderately satisfactory. RISK TO DEVELOPMENT OUTCOME Rating: Moderate 88. Risks to the development outcome are assessed to be moderate. Strategic documents and legal frameworks - including the 2022-2025 update to the Government’s PAR Strategy (adopted by the Cabinet of Ministers on July 21, 2021) which identifies HRMIS implementation as one of the PAR’s core objectives; HRMIS-related legal frameworks Page 27 of 40 The World Bank Strengthening Public Resource Management (P161586) established during the Phase 1 which mandate government bodies that are subject to Civil Service Law to implement HRMIS; the new PFM Reform Strategy until 2025 and ICT PFM Strategy until 2025 expected to be adopted shortly – point to the Government’s continuing prioritization of the Project’s areas of work. The expansion of the HRMIS’s modules and upcoming country-wide implementation of the HRMIS (foreseen under the new project to be financed under EC- Bank EU4PFM TF (TF073423)) 22 will support further advancement of the HRMIS reform agenda and contribute to the sustainability of results achieved to date. While awaiting the new project, the NACS is continuing with activities for HRMIS roll out to Phase 2 institutions. However, NACS’s limited IT capacity and lack of budget funding to support Phase 2 as well as limitations in the future application of the knowledge base from Phase 1 implementation due to staff turnover, do pose some risks for full and timely achievement of country-wide implementation. The NACS’s IT and HRMIS data center capacity are also critical for further maintenance of the HRMIS. Looking ahead, the proper functioning of the system will be critical for generating support for its countrywide roll-out and the sustainability of its benefits. V. LESSONS LEARNED AND RECOMMENDATIONS 89. A phased and realistic timeline and approach to implementing HRMIS – which is complex by itself - can help to mitigate implementation risks. Development and implementation of a comprehensive, integrated HRMIS and payroll system that covers all HR processes and activities takes most organizations many years to fully implement. The phased approach, recognized as necessary during project implementation, with a clearly defined scope for each phase helped to distribute costs over time; align tasks with internal resource (people) requirements; and mitigate risks by focusing on a manageable set of activities. It also supported the timely identification and resolution of emerging transition issues and allowed the Government to make sure that fundamental functions were properly automated and ready to serve as the basis for more advanced functions before proceeding to the next phase. 90. Projects with complex IT solutions require sufficient technical assistance and IT capacity strengthening to ensure technical readiness. In situations where the implementing entity’s capacity is constrained, it is important to bring good technical experts on board as early as possible. In the case of the Project, the Bank’s internal expertise in combination with such from external specialists working on similar solutions locally and internationally was an effective way of providing timely and sustained technical assistance. At the same time, it is also important for implementing entities to leverage resources from a variety of sources, including budget support and project funds, to strengthen their human and infrastructure capacity to ensure the smooth development, implementation, and maintenance of IT solutions. In the case of the Project, this was carried out through the filling of staff positions within NACS’s IT department and consultations with other government agencies (for example, the e-government agency) on HRMIS requirements (Component 1), and the hiring of IT specialists (using Project funds) and close coordination with MOF’s other IT related projects through an IT consultant (not project funded) (Component 2). 91. The active engagement of stakeholders can play an important role in generating buy-in for sensitive reforms. The active engagement of stakeholders by NACS in developing requirements for future systems and self-testing them helped to build the credibility of the system and generate buy-in from other government entities to adopt the system, facilitating implementation. This was critical for generating support for the HRMIS’s implementation and mitigating resistance to change from institutions giving up their own HR management systems for the use of the centralized HRMIS. 22This new project was expected to launch within the framework of Phase 1 to ensure continuous roll-out, with the Administration Agreement between the EC and the Bank being signed in October 2020. However, the launching of RE activities under the Public Finance Management Support Programme for Ukraine; P172442 has been delayed due to administrative complications with the preparation and signing of the Grant Agreement. Page 28 of 40 The World Bank Strengthening Public Resource Management (P161586) 92. Parallel BETFs can serve a vital role in supporting the analytical foundation of activities executed under RETF projects and increasing government capacity to implement associated actions. In the case of the Project, the BETF delivered over 20 analytical and policy notes and extensive advisory support, contributing to enhancing the capacity of the Government in a number of parallel and complementary areas including: (i) development of the HRMIS and its roll- out, (ii) development of new reform strategies such as the ICT strategy for PFM and new PFM strategy with a gender focus, and (iii) driving PFM reforms to enhance the efficiency and transparency of public expenditure (i.e., improved budget control, cash management, accounting system) and revenue management (improved tax expenditure . framework, and forecasting) with the international standards. Page 29 of 40 The World Bank Strengthening Public Resource Management (P161586) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increase capacity of Ukraine in effective management of its resources Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Transparency of State Percentage 0.00 50.00 50.00 12.00 Employment and Payroll 31-May-2017 30-Jun-2019 31-Jul-2020 31-Mar-2021 Comments (achievements against targets): Definition, as per the Results Framework in the original Project Paper: Percentage of state entities at the central administration whose personnel and payroll data are available through HRMIS. Partially achieved. The HRMIS has been introduced in 19 central governmental bodies (with payroll functionalities operational in 10 institutions). During implementation, for this indicator, deployment of the HR and payroll functionalities were tracked within the smaller subset of Phase 1 institutions (resulting in the latest value for this indicator reported in ISR#4, archived on July 27, 2020, being 44 percent. However, as the formulation of this indicator was never officially changed to refer to Phase 1 institutions instead of the entirety of the central administration, in this ICR, achievement is assessed against the original formulation. Using such, the final value of the indicator is 12 percent (10 entities using both HR and payroll functionalities divided by 83 central administration entities. Were the (unofficially) modified formulation of the indicator to be used, the indicator could be considered almost achieved, with a final value of 48 percent (10 entities using both HR and payroll functionalities divided by 21 Phase 1 institutions). Page 30 of 40 The World Bank Strengthening Public Resource Management (P161586) Objective/Outcome: Increase capacity of Ukraine in transparent management of its resources Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion ICT Strategy for Key PFM Yes/No N Y Y Y Entities 31-May-2017 30-Jun-2019 31-Jul-2020 31-Mar-2021 Comments (achievements against targets): Definition, as per the Results Framework in the original Project Paper: ICT Strategy in PFM area defines a future vision and implementation plans are developed Achieved. The ICT Strategy for the PFM System including the associated action plan (Strategy for Digital Development, Digital Transformation and Digitalization of the PFM System till 2025) has been developed by the MOF. As of the Project Closing Date, the strategy had been cleared by the PFM institutions (MOF, State Treasury Service, State Customs Service, State Tax Service) and submitted for broader, inter-agency endorsement. On July 13, 2021, the MOF submitted the strategy concurred by all relevant ministries to the Cabinet of Ministers of Ukraine; it is expected to receive final approval shortly. A.2 Intermediate Results Indicators Component: Strengthening Human Resource Management in Public Administration Institutions Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion HRMIS system implementation Yes/No N Y Y Y and testing 31-May-2017 30-Jun-2019 31-Jul-2020 31-Mar-2021 Page 31 of 40 The World Bank Strengthening Public Resource Management (P161586) Comments (achievements against targets): Definition, as per the Results Framework in the original Project Paper: HRMIS system developed and tested. Achieved. The HRMIS system has been developed in its core HR and Payroll functionalities (covering 15 modules grouped into the following subsystems: Organizational structure and Staff; Payroll and Timesheet; Self-Service Cabinet; and Public Portal) planned for Phase 1 and tested. Component: Support to PFM Strategy Implementation Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion ICT Audit of PFM Subsystems Yes/No N Y Y Y 31-May-2017 30-Jun-2019 31-Jul-2020 31-Mar-2021 Comments (achievements against targets): Definition, as per the Results Framework in the original Project Paper: An audit of ICT platform and subsystems is performed to inform an ICT strategy. Achieved. The ICT Audit of PFM systems covering the Ministry of Finance, the State Fiscal Service, and the State Treasury Service of Ukraine was completed in April 2019. The findings of the ICT PFM Audit and recommendations provided by the auditor informed the ICT PFM Strategy prepared by the MOF. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion ICT infrastruture in the MoF is Yes/No N Y Y Y upgraded 31-May-2017 30-Jun-2019 31-Jul-2020 31-Mar-2021 Page 32 of 40 The World Bank Strengthening Public Resource Management (P161586) Comments (achievements against targets): Definition, as per the Results Framework in the original Project Paper: Immediate hardware and software needs addressed. Achieved. The MOF procured 240 work stations under the Project in 2018. This procurement complemented the MOF’s wider procurement of wo rk stations using budget funds. As the result, all obsolete and outdated ICT equipment has been replaced in MOF’s headquarters, ensuring more stable work of existing IT systems and business continuity. Additionally, in June 2018, the MOF procured 10 software licenses for EViews to increase MOF’s capacity in revenue forecasting. Subsequently, in 2021, the MOF procured three additional EViews software licenses to support its macroeconomic forecasting and video-conferencing equipment to strengthen its capacity for online training and communication activities critical in the remote working environment necessitated by COVID-19 . Page 33 of 40 The World Bank Strengthening Public Resource Management (P161586) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 - Increase capacity of Ukraine in effective and transparent management of its resources23 1. Transparency of State Employment and Payroll Outcome Indicators 2. ICT Strategy for Key PFM Entities 1. HRMIS System Implementation and Testing Intermediate Results Indicators 2. ICT audit of PFM subsystems 3. ICT infrastructure in MOF is upgraded 1. Concept on HRMIS Implementation with Action Plan approved by the Cabinet of Ministers’ Resolution # 844 dated December 1, 2017, and amended on May 20,2020. 2. Bidding documents for procurement of HRMIS supply, installation, operational acceptance, and technical support used for the international bidding process. The bidding resulted in selecting Key Outputs by Component HRMIS Supplier and contract signing (Feb 5, 2019). (linked to the achievement of the Objective/Outcome 1) 3. HRMIS with core HR and payroll functionalities (15 modules grouped into 4 sub-systems: Organizational Structure and Staff; Payroll and timesheet; Self-Service Cabinet, and Public Portal) has been developed; tested, and launched in 19 government bodies of 21 Phase 1 institutions. The number of end-users entered into system – 21 570. 23Due to difficulties in dividing the indicators and activities between the two subobjectives of the Project (as set forth in the Grant Agreement), both subobjectives are jointly presented here. Page 34 of 40 The World Bank Strengthening Public Resource Management (P161586) 4. Administrative users of HRMIS in all Phase 1 institutions (HQ and territorial bodies) are trained: 726 admin users on HR functionalities and 357 admin users on Payroll functionalities. 5. HRMIS user guidelines are developed. 6. Normative and regulatory acts for the HRMIS implementation in public institutions (Cabinet’s Decree On Regulation on the HRMIS in Government Institutions #1343 dd Dec.28, 2020, and NACS’s Order On Implementation of HRMIS in Government Institutions #6-21 dd January 16,2021). 7. Hardware to increase capacity of HRMIS Datacenter has been procured. 8. HRMIS Public Portal providing available information online is operational. 9. Report on ICT Audit of PFM Systems with findings and recommendations. 10. 240 working stations to upgrade MOF’s ICT infrastructure have been procured. 11. 10 EViews licenses for revenue forecasting and 3 EViews licenses for macroeconomic analysis and forecasting have been procured. 12. Report on Review of business processes for developing functional requirements and technical specifications for budget planning information system (includes description of “as is� and “to be� processes). 13. Videoconferencing equipment for MOF. 14. Transposition tables for set of IPSAS for harmonization of relevant NPSAS. 15. Training materials on changes in PSA for training of public sector accountants. 16. Methodology for Treasury Single Account cash flow forecasting; Concept for Development of Liquidity Management approved by CMU’s Decree #1375-p dd Nov.4, 2020. Page 35 of 40 The World Bank Strengthening Public Resource Management (P161586) 17. ICT Strategy for PFM Systems until 2025 has been prepared. 18. Detailed technical design for common PFM solutions architecture; functional requirements for cloud-based technologies for MOF’s ICT System; package of documents for creating of comprehensive information protection system for MOF’s Information and telecommunication system (ITS). Page 36 of 40 The World Bank Strengthening Public Resource Management (P161586) . ANNEX 2. PROJECT COST BY COMPONENT24 Amount at Approval Actual at Project Percentage of Approval Components (EUR M) Closing (EUR M) (EUR M) 1. Strengthening of HRMS in Public Administration 1.92 1.88 97.9 Institutions 2.Support to PFM Strategy 1.1 0.76 68.6 Implementation Total 3.03 2.64 87.2 24 Calculated based on information denominated in the holding currency (euros), which is line with data reported in Client Connection. Page 37 of 40 The World Bank Strengthening Public Resource Management (P161586) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Comments received from the NACS and MoF have been reflected in the text. The EU Delegation provided comments on the parallel parent Trust Fund Completion report which was also prepared by the Bank to report on achievements under the BETF and this RETF, and applicable comments on that report have been incorporated directly into the text of the ICR or are summarized below. The EU Delegation considers the HRMIS to have had a limited impact on effective and transparent management of HR and payroll. In particular, the Delegation notes that HRMIS modules have been developed (but are still being/in need of being adjusted, as entities have concerns over functionality and data protection), and that while a number have connected and uploaded some staff data, so far the HRMIS is not used or is used in a limited way in few pilot institutions. Regarding the payroll module, it understands that only NACS is using it. The EU Delegation also raised questions on issues that are planned to be assessed within an independent review to be carried out during Phase 2 work, including: • Which institutions are using standard HR approaches and templates; • What real time pay-related information is currently available to central institutions from the pilot institutions; • Turnover of trained NACS staff, the dedicated number of NACS staff working on the HRMIS and supporting users, and its correspondence with demand. Page 38 of 40 The World Bank Strengthening Public Resource Management (P161586) ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) Cabinet of Ministers. Resolution #335. May 18, 2016. Cabinet of Ministers. Resolution #1343. “On approval of Regulation on HRMIS in government bodies�. December 28, 2020 European Commission (EC)-World Bank Partnership Program for Europe and Central Asia Programmatic Single- Donor Trust Fund Ukraine – European Union Programme for the Reform of Public Administration and Finances (EURoPAF). Minutes of the Trust Fund Steering Committee. 2017-2020. Government of Ukraine. 2016-2020 Public Administration Reform Strategy. June 24, 2016. Government of Ukraine. 2017-2020 Public Financial Management Reform Strategy. February 8, 2017. Government of Ukraine. Government Program and Action Plan. April 2017. Government of Ukraine. Government Action Plan 2021. Government of Ukraine. 2022-2025 Update to the Public Administration Reform Strategy. Adopted by the Cabinet of Ministers on July 21, 2021. International Bank for Reconstruction and Development. Ukraine PEFA 2015,: Kyiv, 2016. Accessible at: https://www.pefa.org/sites/pefa/files/2020-10/UA-May16-PFMPR-Public%20with%20PEFA%20Check-EN.pdf Ministry of Finance. Progress Reports. 2017-2020. NACS. Progress Reports. 2017-2020. NACS. Order #6-21. “On implementation of HRMIS in government bodies,� January 16, 2021. World Bank. Implementation Completion and Results Report on a Loan in the Amount of US$50 million to Ukraine for the Public Finance Modernization Project. Report No. ICR00003431. June 29, 2015. World Bank Group. Country Partnership Framework for Ukraine for the Period FY17-FY21. Report No. 114516- UA. June 20, 2017. World Bank. Grant Agreement, Ukraine: EURoPAF TF Grant No. TF0A5324 Strengthening Public Resource Management Project. Countersigned on June 30, 2017. World Bank. Project Paper on a Proposed Grant in the Amount of US$ 3.39 Equivalent to the Government of Ukraine for a Strengthening Public Resource Management Project. Report No. PP2198. May 26, 2017. World Bank. Ukraine: Implementing a Human Resource Management Information Systems (HRMIS) in the Page 39 of 40 The World Bank Strengthening Public Resource Management (P161586) Public Administration. Options Paper. July 7, 2017. World Bank. Extension of Closing Date, EURoPAF TF Grant No. TF0A5324 Strengthening Public Resource Management Project. August 9, 2018. Reg. nr. 2020/07/24-01. World Bank. Restructuring Paper on a Proposed Project Restructuring of Strengthening Public Resource Management Approved on June 23, 2017 to Ministry of Economic Development and Trade. Report No. RES33090. 2018 World Bank. Amendment No. 1 to the Administration Agreement between the European Commission and the International Bank for Reconstruction and Development concerning the Parallel EC-World Bank Partnership Program for Europe and Central Asia Programmatic Single-Donor Trust Fund (No. TF072780), EU Program for the Reform of Public Administration and Finances (EURoPAF) (Donor Reference No. 385-741). Countersigned on May 14, 2018. World Bank. Ukraine Public Finance Review 2: Towards a More Effective and Fiscally Sustainable Public Sector. November 2018. World Bank. Grant No. TF0A5324 (Strengthening Public Resource Management Project. Amendment to Additional Instructions: Disbursement Letter. January 24, 2019. World Bank. Restructuring Paper on a Proposed Project Restructuring of Strengthening Public Resource Management Approved on June 23, 2017 to Ministry of Economic Development and Trade. Report No. RES42780. 2020 World Bank. Administration Agreement between the European Commission, on behalf of the European Union, and the International Bank for Reconstruction and Development and the International Development Association concerning the European Commission-World Bank Partnership Programme Part III for Europe and Central Asia Programmatic Single-Donor Trust Fund (TF073423)/Public Finance Management Support Programme for Ukraine (EU4PFM)- Component 2 Activities Donor Reference No ENI/2020/418-207. Countersigned on October 12, 2020. World Bank. Extension of Closing Date, EURoPAF TF Grant No. TF0A5324 Strengthening Public Resource Management Project. 2018/08/09-1. July 27, 2020. World Bank. Implementation Status and Results Reports. Strengthening Public Resource Management Project. (P161586). No. 1-4 World Bank. World Development Indicators. Databank. Page 40 of 40