Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00006104 IMPLEMENTATION COMPLETION AND RESULTS REPORT IBRD 8539-AM ON A LOAN IN THE AMOUNT OF US$21 MILLION TO THE REPUBLIC OF ARMENIA FOR THE THIRD PUBLIC SECTOR MODERNIZATION PROJECT August 31, 2023 Governance Global Practice Europe and Central Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective March 31, 2023) Currency Unit = Armenia Dram (AMD) US$1 = AMD 390.60 SDR 1 = US$ 1.35 FISCAL YEAR January 1 – December 31 Regional Vice President: Antonella Bassani Country Director: Rolande Simone Pryce Regional Director: Asad Alam Practice Manager: Adrian Fozzard Task Team Leader(s): Arman Vatyan ICR Main Contributor: Migara Oshadi De Silva ABBREVIATIONS AND ACRONYMS AAPA Armenian Academy of Public Administration ADS Annual Development Strategy APSAS Armenian Public Sector Accounting Standards BPR Business Process Review CPC Competition Protection Commission CPF Country Partnership Framework CPS Country Partnership Strategy CSC Civil Service Council DPM Deputy Prime Minister DO Development Outcome EKENG e-Governance Infrastructure Implementation Unit FFPMC Foreign Financing Project Management Center FM Financial Management GDP Gross Domestic Product GFMIS Government Financial Management Information System HRMIS Human Resource Management Information System ICR Implementation Completion and Results Report ICT Information and Communication Technology IFMIS Integrated Financial Management System IRI Intermediate Results Indicator IRR Internal Rate of Return ISR Implementation Status and Results Report IT Information Technology M&E Monitoring and Evaluation MoF Ministry of Finance MoFA Ministry of Foreign Affairs MTR Mid-Term Review NPV Net Present Value PDO Project Development Objective PGO Prosecutor General’s Office PAD Project Appraisal Document PEFA Public Expenditure and Financial Accountability PFM Public Financial Management PMG Project Management Group PSMP Public Sector Management Program PSRC Public Sector Reform Commission SMEs Small & Medium Enterprises TA Technical Assistance TAMP Tax Administration Modernization Project ToR Terms of Reference ToT Training of Trainers TTL Task Team Leader TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 6 A. CONTEXT AT APPRAISAL ........................................................................................................6 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .....................................10 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDOs ............................................................................................................13 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................13 C. EFFICIENCY ...........................................................................................................................19 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................20 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................21 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 22 A. KEY FACTORS DURING PREPARATION ...................................................................................22 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................24 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 25 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................25 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................27 C. BANK PERFORMANCE ...........................................................................................................27 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................29 V. LESSONS AND RECOMMENDATIONS ............................................................................. 30 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 31 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 39 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 42 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 43 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 44 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 45 ANNEX 7. LIST OF 120 SERVICES DIGITIZED UNDER PSMP3 ................................................... 46 The World Bank Third Public Sector Modernization Project (P149913) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P149913 Third Public Sector Modernization Project Country Financing Instrument Armenia Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Republic of Armenia Office of the Government Project Development Objective (PDO) Original PDO The project development objective (PDO) is to improve quality of the public financial reporting and to improve access to selected enhanced e-government services. Revised PDO The project development objective (PDO) is to improve access to selected enhanced e-government services. Page 1 of 50 The World Bank Third Public Sector Modernization Project (P149913) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 21,000,000 20,286,058 20,286,058 IBRD-85390 Total 21,000,000 20,286,058 20,286,058 Non-World Bank Financing 0 0 0 Borrower/Recipient 15,700,000 0 0 Total 15,700,000 0 0 Total Project Cost 36,700,000 20,286,058 20,286,058 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 30-Sep-2015 06-May-2016 20-May-2019 31-Dec-2020 30-Nov-2022 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 17-May-2019 3.22 12-Jul-2019 3.22 Change in Results Framework Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Institutional Arrangements Change in Implementation Schedule 27-Oct-2020 8.85 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories Page 2 of 50 The World Bank Third Public Sector Modernization Project (P149913) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 22-Jan-2016 Satisfactory Satisfactory 0 02 16-May-2016 Satisfactory Satisfactory 0 03 17-Nov-2016 Moderately Satisfactory Moderately Satisfactory 0 04 27-Jun-2017 Moderately Satisfactory Moderately Satisfactory .30 05 15-Jan-2018 Moderately Satisfactory Moderately Satisfactory 1.00 06 06-Jul-2018 Moderately Satisfactory Moderately Satisfactory 1.44 Moderately Moderately 07 14-Jan-2019 2.70 Unsatisfactory Unsatisfactory 08 24-Jul-2019 Moderately Satisfactory Moderately Satisfactory 4.02 09 25-Feb-2020 Moderately Satisfactory Satisfactory 6.32 10 27-Jun-2020 Moderately Satisfactory Satisfactory 8.55 11 26-Feb-2021 Satisfactory Satisfactory 9.94 12 20-Dec-2021 Satisfactory Satisfactory 14.73 13 06-Jul-2022 Satisfactory Satisfactory 18.41 14 05-Apr-2023 Satisfactory Satisfactory 20.23 Page 3 of 50 The World Bank Third Public Sector Modernization Project (P149913) SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 70 Central Government (Central Agencies) 70 Information and Communications Technologies 30 Public Administration - Information and 30 Communications Technologies Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Public Sector Management 0 Public Finance Management 19 Public Expenditure Management 19 Public Administration 82 Transparency, Accountability and Good 19 Governance E-Government, incl. e-services 63 ADM STAFF Role At Approval At ICR Regional Vice President: Cyril E Muller Antonella Bassani Country Director: Eavan O'Halloran Sebastian-A Molineus Director: Mario Marcel Cullell Lalita M. Moorty Practice Manager: Adrian Fozzard Daniel J. Boyce Page 4 of 50 The World Bank Third Public Sector Modernization Project (P149913) Migara Oshadi De Silva, Davit Task Team Leader(s): Arman Vatyan Melikyan ICR Contributing Author: Migara Oshadi De Silva Page 5 of 50 The World Bank Third Public Sector Modernization Project (P149913) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Country Context 1. Since independence in 1991, Armenia has achieved average gross domestic product (GDP) growth rates of 3 percent, outpacing most lower-middle income countries, but high levels of poverty have persisted. The economic crisis of 2008 led to an increase in poverty incidence from 27.6 percent in 2008 to 35.8 percent in 2010. The economic recovery brought only modest improvement with poverty incidence at 32 percent in 2013 and left the income distribution slightly more unequal than in 2007. At the time of appraisal, the Armenian Development Strategy (ADS) for 2014-2015 1 articulated a path to achieve middle-income status by 2025 (actually achieved in 2019). The strategy is built around four pillars: creating jobs; developing human capital; strengthening the social protection system; and modernizing public administration and governance. Sectoral and Institutional Context 2. The Government of Armenia requested World Bank assistance with three areas of public sector reform: public financial management; e-Governance; and civil service capacity building. The Government had articulated reform strategies in each of these areas. 3. The Public Financial Management (PFM) Reform Strategy for 2010–2020 sought to drive reforms by implementing a comprehensive government financial management information system (GFMIS). The Public Expenditure and Financial Accountability (PEFA) Assessment of Armenia, published in May 2014, identified weaknesses in budgeting, whole-of-government financial reporting, and limitations in internal controls. The PEFA Assessment identified the GFMIS as a critical investment to support reforms, supporting the introduction of program budgeting and linking budgeting, accounting and financial reporting and thereby put in place a rigorous framework of internal controls. The GFMIS would help the government: improve the budget system and outcomes; ensure timely access to financial information across government agencies throughout the budget cycle; integrate the PFM cycle from the program budgeting to accounting and financial reporting and establishing a rigorous financial control framework; and enhance transparency and accountability of the government. 4. The Government’s Open Government Action Plan for 2014–2016, prepared as part of the Open Government Partnership, and an April 2014 e-Governance Strategy laid out a roadmap on the use of information and communication technology to support public sector reform and broader development objectives. The e-Governance Strategy sought to transform the way in which the public sector operated, bringing transparency and responsiveness through intelligent investments in information technology and its widespread use in public agencies. The Strategy acknowledged the need for substantial investments in ICT infrastructure, equipment, software, and capacity development. The Government had established state-owned e-Governance Infrastructure Implementation Unit (EKENG) as a joint-stock company responsible for the technical aspects of the e-Governance strategy. Further work was needed in the development of an enabling regulatory environment, appropriate national standards and the institutional, 1Armenia Development Strategy for 2014-2015, adopted on March 27, 2014 by Government Decree N442, available here: https://www.gov.am/files/docs/1322.pdf Page 6 of 50 The World Bank Third Public Sector Modernization Project (P149913) organizational, and procedural reforms in public agencies to enable reforms and translate e-governance into improvements in public sector performance, transparency and accountability. 5. Parliament passed the amendments to the Law of Civil Service in 2014 which enhanced flexibility in recruitment, promotion of professional staff, and provided the basis for a modern performance evaluation system. These reforms sought to address the problems of low pay relative to the private sector and stringent entrance requirements hindered recruitment and retention of qualified staff from professions and technical fields. Alongside regulatory measures, the Government sought to put in place an effective staff training and professional development program that would help strengthen performance, create opportunities for the most competent staff and underline the Government’s commitment to merit-based promotion. World Bank Rationale 6. PSMP 3 was aligned with the Country Partnership Strategy (CPS) for FY2014–17 (Report Number 81647) and built on a track record of successful public sector reform projects. PSMP3 was aligned with CPF pillar supporting competitiveness and job creation and a cross cutting dimension of Governance and Anti-corruption, which supported improvements in the efficiency and targeting of social services, strengthening the anti-corruption framework, improving efficiency of the public administration, and deepening PFM reforms. PSMP 3 built on earlier Bank engagement in public sector reforms in Armenia. The PSMP 1 (2004-2011) and PSMP 2 (2010-2017) projects supported policy formulation, business process simplification, and automation of solutions for local government, civil service, public procurement, internal and external audit, and e-Governance reforms. These projects successfully delivered key institutional reforms, capacity building and delivered core administrative e-Government applications including the Electronic Data Management System (EDMS), Human Resource Management Information System [HRMIS], and e-Tendering . Theory of Change (Results Chain) 7. The Project Appraisal Document (PAD) did not delineate the theory of change for PSMP 3 as this was not required by Bank procedures when the project was prepared. However, the logic behind the operation is summarized in figure 1 below. Page 7 of 50 The World Bank Third Public Sector Modernization Project (P149913) Figure 1. Theory of Change at Appraisal Component Challenges Intermediate Outcome Outcome Impact Implementation of ongoing PFM A. Improved efficiency, Weak financial reform strategy Public Financial effectiveness, and reporting Management Capacity building for advancing comprehensiveness of framework Information reform implementation in the the government's FM Systems MoF and accounting Strengthened policy environment necessary for introduction of a deeper e- Higher Level: Fragmented e- Governance reforms Robust e- Selected reforms of the governance Introduction of basic platforms Governance B. GoA’s e-Governance architecture, and infrastructure for framework e-Governance strategy and priority e- gaps in access supporting e-Governance leading to Solutions for Government services to public agenda efficient and Improved implemented and services Establishment of feedback effective Service Delivery operational mechanisms to facilitate public service citizen and business access to delivery government services, address which enhances grievances, and monitor client transparency and satisfaction accountability. Strengthened capacity of the CSC and AAPA for modernizing C. and raising the quality of the Strengthened technical Capacity Limited civil training system for public and institutional Building and service servants capacity of CSC and Small Capacity capacity Coordination and AAPA in line with the Building implementation of the public ADS Interventions sector reforms through supporting the PSRC D. Project Management Note: CSC = Civil Service Council; AAPA = Armenian Academy of Public Administration PSRC = Public Sector Reform Commission. Project Development Objectives (PDOs) 8. The PDO in the original Loan Agreement was to (a) improve the quality of the public financial reporting; and (b) improve access to selected enhanced e-government services. The PDO was revised during the third project restructuring on October 27, 2020. Key Expected Outcomes and Outcome Indicators Table 1. Key Expected PDO Outcomes and Outcome Indicators (in the PAD) Outcome Indicators Outcome Targets Improved quality of annual financial reporting Financial reporting for two selected pilot sectors prepared annually, comparable with approved budget, Page 8 of 50 The World Bank Third Public Sector Modernization Project (P149913) Outcome Indicators Outcome Targets include information on revenue, expenditure, financial assets and liabilities, guarantees and long-term obligations (original) Reduced time for processing a passport extension by 48 hours the Consular Department of the Ministry of Foreign Affairs Increased access to e-consular services for issuing 10,000 certificates and statements issued citizenship, civil status certificates, and statements on any existing criminal records (data disaggregated by gender) Components 9. PSMP 3 had three substantive components, supported by a project management component. The World Bank contribution to the project constituted 57 percent (US$21 million) of the overall project cost at appraisal; 20 percent of the project cost was to be financed by Government; and 23 percent by a Russian grant. 10. Component 1: Public Financial Management Information System (component cost US$19 million; of which IBRD US$7 million, US$8.2 million Russian funding and US$3.8 million is government co- financing). This component had two subcomponents. Subcomponent 1.1 sought to design, develop, and implement a GFMIS through: development of software, IT modules, and toolkits related to, among other things, budget planning, budget execution, public procurement, debt management, accounting, budget reporting, and the general ledger; acquisition of hardware, goods, equipment, and ICT licenses; provision of related training; and maintenance provision. Subcomponent 1.2 sought to support the borrower’s implementation of PFM reforms and change management processes related to public expenditure and budget preparation and execution, through: advisory services and technical assistance (TA); capacity- building activities preparing upstream and downstream aspects of PFM (budgeting, public accounting, and audit); and capacity-building activities for the rollout of the Armenian Public Sector Accounting Standards (APSAS). 11. Component 2: E-governance Solutions for Improved Service Delivery (US$14 million; IBRD US$11 million). This component supported implementation of selected government priorities identified in the Government’s e-Governance Strategy. Sub-component 2.1 supported e-Governance policy, institutions, and capacity, through: review and assessment of existing ICT policies, legal and regulatory frameworks, and technical standards; strengthening of existing, or development of new, policies, strategies, regulatory frameworks, and technical standards; development of an institutional setup and business model for e- Governance platforms and e-services; and capacity-building, training, and awareness raising activities related to e-Governance platforms and e-services including the development of a unified e-services portal incorporating an integrated feedback collection mechanism. Sub-component 2.2 supported the development of an interoperability platform; upgrade of the Human Resource Management and Electronic Document Management Systems; strengthening datacenter storage capacity; and investments in selected e-services. The initial list of e-services comprised solutions for: pre-trial case management system; income and asset declarations; drafting and publication of legal acts; transport modules for route competition and an interactive route map; and issuing licenses and license registry. Sub-component 2.3 supported the establishment of feedback mechanisms to facilitate citizen and business access to government services, address grievances, and monitor client satisfaction. Activities included development Page 9 of 50 The World Bank Third Public Sector Modernization Project (P149913) of a public e-services and information hotline and a unified e-services portal with an integrated feedback module to gather information on access to and the quality of services provided. 12. Component 3: Capacity Building and Small Capacity Building Interventions (US$1.85 million; IBRD US$1.474 million). This component supported public sector modernization and governance reform agenda laid out in the ADS. Sub-component 3.1 sought to strengthen the institutional capacity of the Civil Service Council (CSC) and Armenian Academy of Public Administration (AAPA) through: transfer of skills on the management of IT, evidence-based decision-making, and consultation and citizen engagement; revision and systemization of CSC's existing civil service training system; improvement of AAPA’s curriculum to meet long-term training needs and for continuous training; training of trainers (ToT) and secondment activities for faculty members; development of CSC’s capacity to conduct training needs assessments based on the performance assessment results and government strategies; and improvement of AAPA’s capacity to deliver high-quality continuous training to public servants. Sub-component 3.2 supported the Public Sector Reform Commission (PSRC) coordination role and implementation of the public sector modernization and governance reforms, by providing: just-in-time technical assistance; change management for e-Governance reforms across key institutions; capacity-building interventions for effective project implementation; and a gender review of public employment. 13. Component 4: Project Management (US$1.85; IBRD US1.474 million). This component provided administrative support and TA to the Project Management Group (PMG) and the Foreign Financing Project Management Center (FFPMC) for project management; procurement; FM including financial audits; monitoring and evaluation (M&E) of the project, including periodic project monitoring reports; public awareness/communication campaigns and gender-sensitive consultations; training for the PMG and FFPMC staff; and the development of the technical requirements/specifications based on inputs provided by the project’s stakeholders. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) 14. PSMP 3 was restructured three times. The first restructuring in May 2019 changed the implementing agency responsible for fiduciary functions under the project within the MoF, as part of the MoF reorganization. The second restructuring in July 2019, followed the project mid-term review conducted in May 2019, to change implementing arrangements and revise the results framework without changing the PDO. The third restructuring in October 2020 changed the project scope and revised the PDO. Revised PDOs, Outcome Targets and PDO Indicators 15. The PDO, as revised during the third restructuring, was “to improve access to selected enhanced e-government services”. The third restructuring cancelled activities related to the implementation of the GFMIS under the Component 1 and the first part of the original PDO (“to improve the quality of the public financial reporting”). 16. The Outcome Targets related to PDO element “to improve the quality of the public financial reporting” were adjusted in the second restructuring and then cancelled in the third restructuring. The target value for PDO-level indicator “Improved quality of annual financial reporting” was revised during the second restructuring from “Financial reports for two selected pilot sectors prepared annually and are comparable with the approved budget, and include information on revenue, expenditure, financial assets and liabilities, guarantees and long-term obligations” to “Three selected public administration entities that are in charge of sectors amounting to 10 percent of the state budget expenditures, prepare substantially APSAS compliant financial statements”. This revision was aligned with the government’s Page 10 of 50 The World Bank Third Public Sector Modernization Project (P149913) decision to phase in the of application of accounting standards. The Outcome Target “Improved quality of annual financial reporting” was subsequently dropped during the third restructuring together the related intermediate results indicator (IRI) “Establish preconditions required for consolidation of the financial statements.” Table 2. Revised PDO Outcomes and Outcome Indicators (after Third Project Restructuring) Outcome Indicators Outcome Targets Reduced time for processing a passport extension by the 48 hours Consular Department of the Ministry of Foreign Affairs Increased access to e-consular services for issuing citizenship, 10,000 certificates and statements issued civil status certificates, and statements on any existing criminal records (data disaggregated by gender) Revised Components 17. Component 1 was scaled back during the third restructuring with the cancellation of activities related to the implementation of the Public Financial Management Information System. The restructuring dropped Subcomponent Part 1.1 on the design, development, and implementation of the GFMIS. Subcomponent 1.2 activity (c) related to “the carrying out of capacity-building activities for the rollout of the APSAS” was also dropped because this was dependent on GFMIS implementation. MoF planned to carry out those activities using the parallel Russian grant. At the time of ICR preparation consolidated APSAS financial statements for the government have not been prepared because this was dependent on GFMIS roll out. Nor had the project made progress on the implementation of financial management reforms and capacity-building activities for preparation of upstream and downstream aspects of public financial management (budgeting, public accounting, and audit). 18. The scope of activities under Component 2 was significantly expanded during the third restructuring to include new e-Government services. The new services were: an online platform for around 30-40 digitalized public services including joint electronic platform for notifications and permits through establishment of electronic platform providing all the state licenses and permits; an e-platform for documents verification to establish a unified system to issue and validate all state documents (related to education, civil status, notary, property rights); a statistical dashboard to collect statistics from state databases and provide real-time information to decision-makers; and computers and servers to effectively manage the information system and support the rollout of the new e-Government systems. Other Changes 19. The first project restructuring changed the implementing agency responsible for fiduciary functions under the project as part of a reorganization of MoF departmental responsibilities. MoF reassigned project fiduciary functions from the Foreign Financing Project Management Center (FFPMC) to the newly created Project Implementation Department (PID) within MoF. This restructuring updated the institutional arrangements under the project to be in line with the government’s systemwide approach to project management of foreign-assisted projects. 20. The second project restructuring made adjustments to the Results Framework; extended the loan closing date by 23 months to November 30, 2022; reallocated financing between disbursement categories; and changes in institutional arrangements. The PDO-level and Intermediate Results Indicators relating to Component 1 were amended to extend the timeframe for implementation given the delays the Project was facing. The Project Management Group (PMG) assumed the fiduciary functions for Page 11 of 50 The World Bank Third Public Sector Modernization Project (P149913) the project under the office of the Deputy Prime Minister (DPM). As of June 2019, the DPM became the Head of the Project Steering Committee. The reference to the CSC in the original Loan Agreement was replaced in the revised Loan Agreement with the Civil Service Office due to changes in the legal framework for the entity. Some adjustments were made to the activities under Component 2.2, notably the e- Transport was dropped. The Project implementation schedule, the fiduciary risk was raised from moderate to substantial, and disbursement estimates were updated. 21. The third project restructuring included the change of PDO, several changes to the Results Framework to be in line with the revised PDO, and partial cancellation and reallocation of costs across components. The PDO indicator and intermediate result indicators related to financial reporting and the roll out of accounting standards were dropped. Component costs were restructured to finance activities under Component 2 that were not foreseen in the original project design. Bank financing for Component 1 was reduced by US$6.7 million (from US$7 million to US$0.3 million). US$6 million was reallocated to Component 2 (increased from US$14 million to US$20 million). US$0.7 million of Bank financing for Component 1 was canceled. Rationale for Changes and Their Implication on the Theory of Change 22. The rationale for changes in the first project restructuring was in response to the request made by the MoF of the Republic of Armenia, dated April 1, 2019, to reassign the fiduciary functions to the newly created Project Implementation Department (PID). This amendment did not entail any other changes to the project, the PDO, results framework and activities remained unchanged. 23. The second project restructuring responded to a government request dated May 29, 2019, to extend and restructure the project allowing to align with the new Government Program. 2 After elections held in December of 2018, new Government Program adopted by Government Decision N65-A, on February 8, 2019, laid out a digital strategy to improve efficiency, quality of service delivery, and enhance transparency, accountability and data availability in public administration. The PDO remained unchanged. The project closing date was extended from December 31, 2020, to November 30, 2022, to allow completion of activities critical for achievement of PDO and accommodate the Government’s new priorities for service delivery and anti-corruption. The Results Framework was revised to better align final outcomes reflecting realistically achievable indicators given the delays in implementation of Component 1. Changes to the project’s institutional and fiduciary arrangements responded to government organizational reforms. The project adopted a single disbursement category to allow for flexibility in financing project activities. 24. The third project restructuring responded to the MoF’s request of July 17, 2020, to partially cancel Bank financing for GFMIS under Component 1 and reallocate the balance of funds to support digital government initiatives under Component 2. Preparation of the bidding package for GFMIS implementation started at effectiveness. However, following changes in leadership in 2016, MoF questioned the need for the GFMIS, preferring instead to upgrade the existing Treasury system. Consultations with the counterparts, culminating in a mission to focus on GFMIS issues in mid-2019 immediately after the project’s MTR, sought to understand MoF’s continuing hesitation and facilitate implementation of the GFMIS leading to the preparation of an Action Plan for the launch of the GFMIS 2 Government Program of 2019 is available here: https://www.gov.am/files/docs/3133.pdf Page 12 of 50 The World Bank Third Public Sector Modernization Project (P149913) tender in January 2020. In December 2019, MoF informed the Bank that it would use the Russian grant for the GFMIS implementation and would postpone application of Bank financing until after the grant was fully disbursed. By mid-2020 it was clear that there was little chance that the Bank financing would be used for GFMIS activities and the GFMIS could not be implemented by the revised project closing date of November 30, 2022. MoF decided to cancel activities related to the GFMIS, reduce Bank financing for Component 1 from US$7 million to US$0.3 million, cancel US$0.7 million so that funds could be used for a health project and reallocate US$6 million to Component 2. The Government requested that the US$6 million funds be used to support six prioritized activities, selected on the basis of similar criteria as those selected at appraisal, namely: an integrated security system; e-gov system with 30-40 services; an e- platform for notifications and permits; an e-Verify system; a statistics dashboard for decision makers; and hardware for civil servants. 25. The theory of change presented in Figure 1 was adjusted to reflect the cancellation of activities related to the GFMIS but was not adjusted to reflect the scaling up of digital government activities. The PDO level and intermediate indicators relating to Component 1 were deleted. The intermediate indicator under Component 3 was revised to connect it with capacity building in civil service instead of ASPAS. At the time, the team considered the existing PDO indicators adequate to capture the scaling up of digital government activities. II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating Rating: Substantial 26. The PDO was highly relevant at appraisal and remains so. Improvement of public service efficiency through digitalization were priorities of the PFM Strategy and the e-Governance Strategy. The PDO is aligned with Armenia’s current development strategy (the Government Program approved by Government Decree on August 18, 2021). The PDO was aligned with Bank’s Country Partnership Strategy (CPS) for FY2014–17 and is aligned with the Country Partnership Framework (CPF) for FY2019-2023 (Report No. 123902) as modified by the CPF Program Learning Review completed in May 2022 (Report No. 168425-AM). The CPF and Program Learning Review identified the PSMP 3 as a Bank instrument for engagement in delivering on CPF Objective 4: Improved public spending efficiency and increased private financing for development. However, the relevance of the original PDO became modest when the expected results related to the GFMIS were dropped. The revised PDO and its follow-on project (PSMP 4) are closely aligned with the revised approach for CPF Objective 4 which addresses the challenges of limited fiscal space and corruption by improving spending efficiency in public administration and supporting the government in the creation of a digitalized public service. Accordingly, the relevance of the revised PDO is high. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 27. Tables 3 and 4 provide the ratings applied to the PDO indicators by outcome before and after the third project restructuring. Table 3. Results achieved before the October 2020 restructuring. Page 13 of 50 The World Bank Third Public Sector Modernization Project (P149913) Indicators in Results Framework Achievement Against end-Target PDO Indicators Improved quality of financial reporting. Baseline: Financial Not achieved. Work was underway to develop a reports for two selected pilot sectors are prepared annually methodology for the preparation of but do not include all information on revenue, expenditure, consolidated financial statements in early 2020 financial assets and liabilities, guarantees, and long-term when procurement of TA for piloting the new obligations financial reporting framework was interrupted. Target: Financial reporting for two selected pilot sectors No progress was made after the second project prepared annually and are comparable with the approved restructuring. budget and include information on revenue, expenditure, financial assets and liabilities, guarantees, and long-term obligations. Revised target at second project restructuring in July 2019: Three selected public administration entities that are in charge of sectors amounting to 10 percent of the state budget expenditures, prepare substantially APSAS compliant financial statements). Reduced time for processing a passport extension request by Achieved. Processing time reduced from 500 to the Consular Department of the Ministry of Foreign Affairs 48 hours. from baseline of 500 to target of 48 hours. Increased access to e-consular services for issuing citizenship, Partially achieved. Number of civil status certificates, and statements on any existing certificates/statements issued from zero to criminal records (data disaggregated by gender). From no 7,000. services in place to 10,000 certificates and statements issued IRIs related to Component 2 in Results Framework Reduced time for police dispatch in emergency situations (in In progress. System tested. Awaiting roll-out. at least two selected pilot regions/marzes) from baseline of 15-20 minutes to target of under 5 minutes. Increased share of citizens providing voluntary feedback on Partially achieved. 60% specific public services from baseline of no feedback mechanism was in place to target of 20. Reduced time to transmit information/messages relevant for In progress. System under procurement the pretrial proceedings between prosecutors’ offices in different regions from baseline of one week by special express mail (send via post office) to target of immediate transmission (electronic transmission). Increased gender disaggregated data is available on the use Partially achieved. Gender disaggregated data of e-services which are provided by selected agencies from on e-services is available for MoFA baseline of no gender disaggregated data to target of gender disaggregated data on e-services available to Ministry of foreign affairs, Prosecutor’s Department, Ethics Commission, Police. Reduced time to issue the certificate of civil status by the Achieved (Reduced from 60 to 10 days) Consular Department of the MoFA from baseline of 50 days to target of 10 days. Results for Component 2 Not in Results Framework Page 14 of 50 The World Bank Third Public Sector Modernization Project (P149913) Indicators in Results Framework Achievement Against end-Target Establishment of an e-Competition system for the E-Competition system facilities analysis of data Competition Protection Commission from relevant markets and monitoring of transactions by legal entities. This 70-80 percent staff time saving in routine data management activities. Reduction in the number of terminated administrative proceedings on violations of the law by 50 percent. IRIs related to Component 3 in Results Framework Number of trainers for the new needs-based training system Partially achieved. 40% achieved in the civil service established under the Project, where ethe baseline is 0 and target is 50. Results for Component 3 Not in Results Framework New ICT Equipment for Government, new Civil Service Achieved. All necessary equipment for efficient Platform, the upgrade and integration of G2G systems for operation of the e-Government systems public administration installed (computers, servers, storage systems, printers, Global Positioning System, IP (Internet Protocol) cameras). Source: World Bank. Restructuring Paper. REPORT NO.: RES43463. (October 2020). Table 4. Results achieved between October 2020 restructuring and project closing. Related Indicators in Results Framework Achievement Against end-Target PDO Indicators Reduced time for processing a passport extension request by Over-Achieved. Processing time reduced further the Consular Department of the Ministry of Foreign Affairs to 24 hours. The male versus female ratio of the from baseline of 500 to target of 48 hours. users was 46/54, with no gender-based difference in service provision. New video module of Consular services was developed after the restructuring to allow video communication in provision of Consular services thus facilitating application for digital at Consulates Increased access to e-consular services for issuing citizenship, Over-Achieved. By end of 2022, 19,691 civil status certificates, and statements on any existing certificates were issued, of which 8,381 civil criminal records (data disaggregated by gender). From no status certificates (3,381 (46%) female, and services in place to 10,000 certificates and statements issued 4,525 (54%) male) and 11,310 certificates for non-criminal records (5,654 (50%) female, and 5,656 (50%) male). This by far exceeds the final target. New video module of Consular services was developed after the restructuring to allow video communication in provision of Consular services, and e-consulted system was integrated with notarial system, thus facilitating remote Page 15 of 50 The World Bank Third Public Sector Modernization Project (P149913) Related Indicators in Results Framework Achievement Against end-Target application for services at the Consulates, including for notarial documents IRIs related to Component 2 in Results Framework Reduced time for police dispatch in emergency situations (in Achieved. Time reduced from 15-20 minutes to 5 at least two selected pilot regions/marzes), from baseline of minutes. In addition, 3 dispatch centers were 15-20 minutes to target of 5 minutes. established. Increased share of citizens providing voluntary feedback on Achieved. 20 percent of all users of specific specific public services from baseline of no feedback public services provide voluntary feedback on mechanism in place to target of 20 percent. selected public services. Reduced time to transmit information/messages relevant for Under phased implementation with expanded the pre-trial proceedings between prosecutors’ offices in scope. Full implementation September 1, 2023, different regions from baseline of one week by special as stipulated by the new Criminal Procedure express mail (send via post office); to target of Immediate Code. transmission (electronic transmission). Increased gender disaggregated data is available on the use Achieved: 46% male, 54% female. of e-services which are provided by selected agencies from baseline of zero gender disaggregated data to target of gender disaggregated data on e-services available to Ministry of foreign affairs, Prosecutor’s Department, Ethics Commission, Police. Reduced time to issue the certificate of civil status by the Over-Achieved. Citizens born before 1983 within Consular Department of the MoFA from baseline of 60 days 5 days. For those born after 1983, this action is to target of 10 days. carried out immediately. Results related to Component 2 Not in Results Framework Unified online platform for around 30–40 digitalized public Exceeded. The unified electronic system for state services including joint electronic platform for notifications and community services and its integration with and permits through an electronic platform providing all the E-Gov Platform developed, 120 public state licenses and permits. services digitized and linked to the platform (e- consular.mfa.am; migration.am, verify.e- gov.am). Unified and integrated system improves access for citizens and businesses. E-platform for document verification to establish a unified Achieved. e-Verify system is operational and is system to issue and validate all state documents (related to currently in use. This has improved access to and education, civil status, notary, property rights, and so on) the delivery of e-government services. The e-Verify system allowed verification of digitally issued documents, facilitating remote service provision and especially, issuance of electronic documents to citizens. This enabled achievement of other results of indicators. Installation of Video Module for E-Consulate System and Achieved. This further improved access and Integration to E-Notary System service provision of the Consular services by allowing video communication and an integration with the E-Notary system that Page 16 of 50 The World Bank Third Public Sector Modernization Project (P149913) Related Indicators in Results Framework Achievement Against end-Target facilitated issuances of notarized documents at the Consulates. Statistical dashboard for decision-makers to collect statistics Achieved. Provides real-time information to from state databases and provide real-time information to decision-makers; thereby facilitating timely decision-makers decision making. Installation of the Electronic Platform for Declarations of Achieved. The Platform improves access of Assets, Incomes, Expenditures and Interests government officials to file their required declarations to comply with requirements of the Corruption Prevention Commission and relevant legislation. Installation of the Criminal Case Pre-Trial Proceedings Achieved. The system improves efficiency of Electronic Management System operation of institutions involved in criminal case pre-trial proceedings replacing a manual system with a highly organized and efficient real time digital and easily traceable workflow. Installation of digital signature and identification systems for Achieved. This application significantly facilitates smartphones access to e-government services using mobile technology. Integrated security system for state e-governance platforms Achieved. This helps ensure the security of the e- governance services. IRIs related to Component 3 in Results Framework Number of trainers for the new needs-based training system Over-Achieved. Trained 68. in the civil service established under the Project. Target 50. Achievement of Objective 1: Improved Quality of Public Financial Reporting Rating: Negligible 28. The project made negligible progress towards improved quality of public financial reporting. Achievement of this objective was dependent on implementation of the GFMIS which was delayed and ultimately cancelled. The Bank invested in analytical work, including business process reviews, in the GFMIS during project preparation. A significant part of the Bank’s supervision focused on the poorly performing GFMIS component. However, MoF commitment to the GFMIS waned and the GFMIS was cancelled. While the Bank’s investment in GFMIS design and engagement with MoF counterparts may inform a future system upgrade, in the absence of tangible results to the ICR rating for this objective is Negligible. Achievement of Objective 2: Improved Access to Selected Enhanced E-governance Services. Rating: High 29. The two PDO outcome indicators used to assess the efficacy of objective 2 exceeded targets. The indicators measured: (a) reduced time for processing a passport extension by the Consular Department of the Ministry of Foreign Affairs; and (b) increased access to e-consular services for issuing citizenship, civil status certificates, and statements on any existing criminal records (data disaggregated by gender). Page 17 of 50 The World Bank Third Public Sector Modernization Project (P149913) Both indicators and their respective targets were unchanged during project implementation. Gender- disaggregated data, presented in table 3, show that there was no gender-based difference in service provision. 20 percent of all users of these public services are currently providing voluntary feedback on service quality and timeliness. 30. All intermediate results indicators related to objective 2 were met or overachieved their targets. The time taken to issue certificates of civil status by the Consular Department of the Ministry of Foreign Affairs (MoFA) of 10 days was achieved, in most cases it was 5 days. Indeed, for citizens born after 1983 certificates of civil status are issued immediately. The Police Operational Management System provides important statistic and real time data on law enforcement activities. The target for police dispatch within five minutes in emergency situations in at least two selected pilot regions/marzes was achieved. Three dispatch centers were established under PSMP 3, in Yerevan (capital city) and in Lori and Shirak marzes, and eight more centers are expected to be established in the near future using the same framework and tools. The e-Criminal Case Management System developed for law enforcement agencies (General Prosecutor’s Office, investigative bodies), introduced digital solutions for criminal proceedings for the first time. This reform digitized manual work processes, allowing data related to pre-trial proceedings to be transmitted between prosecutors’ offices in different regions. In March 2023, after project closing, pilot training on the pre-trial case management system was extended to include the recently created Anti- Corruption Agency. 31. The project has achieved results beyond intermediate results indicators through the development of digital solutions to support to anti-corruption, accountability, and transparency initiatives. A whistleblower platform, launched in May 2019, allows anonymous reporting of corruption crimes to General Prosecutor’s Office. An asset declaration system for the Corruption Prevention Commission, allows faster, more efficient and user-friendly reporting of assets, income, expenses, and interests. An e- Competition system allows the Competition Protection Commission to analyze and compare data from relevant markets and monitor the transactions made by legal entities. This has allowed a 70-80 percent staff time saving in routine data management activities. The availability of improved tools for reporting and detecting corruption has contributed to a digital anti-corruption framework that positively reflected in public perception surveys. While an attribution of results is difficult to establish, Armenia’s ranking on the Transparency International’s Corruption Perception Index sharply improved from 35 points in 2015 (95th in the World) to 49 points in 2021 (58th in the World).3 Armenia’s performance on WGI Control of Corruption and Voice and Accountability has also improved since 2018. 4 32. After the third restructuring, a list of new activities was included into the Component 2 to address the special request received from DPM. 5 The funds were used to cover development of an online platform for public services and digitalization of public services. Although initially 30-40 services were planned to cover, re-allocation allowed to extend that number and fund digitalization of around 120 services (list of services is presented in Annex 7). The e-Catalog provides a complete list of central government administrative services. The project funded development Smart ID, a smartphone IT solution allowing to sign documents electronically, as well as login and access digital documents, facilitate remote 3 https://www.transparency.org/en/cpi/2022 4 https://info.worldbank.org/governance/wgi/Home/Reports 5 The DPM Tigran Avinyan’s (Head of the Steering Committee) letter dated September 11, 2019. Page 18 of 50 The World Bank Third Public Sector Modernization Project (P149913) access. Funds were also used to finance: an e-Notify system allowing digital and automated notifications from state and municipal bodies to citizens and business in a unified online platform in cases provided for by the legislation; an e-Verify which includes a citizen ‘e-Wallet’ system, system for documents verification, and a personal e-Cabinet; a statistics dashboard for decision makers which is run by EKENG and used by the PMO; purchase of hardware for provision of public services; a video module for e- Consular allowing to remotely identify citizens and provide public services. An e-platform for Compulsory Enforcement Service of Judicial Acts that is interoperable with commercial banks, in development at the time of project closing, will modernize the paper based compulsory enforcement business procedures. The government will complete this application with its own resources. 33. Capacity building activities under Component 3 focused on supporting the respective public institutions to efficiently use and deliver new services, thereby enhancing the quality-of-service delivery. Successful installation of the civil service platform and automated HRMIS within the civil service in 2020 strengthened the institutional capacity of the CSC. The HRMIS improves human resource management within public administration agencies while the platform helps the strategic planning of workforce, recruitment, M&E, and most importantly recipient feedback. HRMIS development and roll out was financed under the project. A need-based training system was put in place to address knowledge/skill gaps, the Training Credit System, which includes required/obligatory training for civil service staff. The Civil Service Office is now able to determine gaps in knowledge and competencies among civil servants and carry out periodic training programs (for example, 68 ToTs in 2020). This contributed to the achievement of the PDO with having stronger civil service capacity for public service delivery. Training served as a catalyst to achieve the PDO. Justification of Overall Efficacy Rating With Original PDO: Substantial With Revised PDO: High 34. The ICR uses a split rating in assessing the outcome of the project: efficacy vis-à-vis the original PDO is rated Substantial given the negligible rating for the first outcome area and a high rating for second outcome area; the efficacy rating vis-à-vis the revised PDO is high given the achievements in e- governance services. C. EFFICIENCY Assessment of Efficiency and Rating Rating: Modest 35. The project achieved all of its results targets and delivered a wide range of e-government services within the original budget. Prior to the third restructuring in October 2020, the Project issued contracts for about US$11 million with Bank financing. Almost all of these contracts were for activities under components 2 and 3. The project had disbursed US$8.9 million or 42 percent of the loan amount at that point. The investments under components 2 and 3 had already achieved one PDO outcome target at the time of restructuring and had made significant progress towards the achievement of the remaining outcome and intermediate result indicator targets. The project had spent only US$ 0.4 million of project funds on the preparation of the GFMIS when activities under Component 1 were cancelled. Consequently, cancellation of Component 1 had no direct impact on efficiency (see Table 3). Indeed, the reallocation of US$6 million from Component 1 to Component 2 allowed the project to significantly expand the range of Page 19 of 50 The World Bank Third Public Sector Modernization Project (P149913) e-government services. The Project issued contracts for about US$10 million of the Bank financing between October 2020 and project closing. All but one of these contracts was completed by project closing when the Project had disbursed US$20.28 million, 99.93 percent of the loan amount. The results expected from additional activities were not reflected in the Project results framework. However, all of these activities were completed successfully and were fully operational by the time of project closing (see Table 4). 36. The scope of the e-Government reforms supported by the Project exceed expectations. The quantitative economic analysis undertaken at preparation could not be replicated owing to changes in the scope of the project, given that the original analysis identified efficiency gains from the GFMIS as the primary benefit. However, the e-Government systems have delivered tangible benefits. The implementation of e-Government services improved accessibility and reliability and reduced transaction costs for citizens and the private sector by reducing travel costs and time spent when accessing government administrative services. Government has also benefited from reduced processing time and recordkeeping through automation, both of which have led to lower administrative overhead costs. E- Verify system and e-Consular services which were launched in time to ensure business continuity and uninterrupted access to administrative services for citizens and the private sector during COVID-19 quarantine and lockdown. The public services information platform was widely used after Russia’s invasion of Ukraine, enabling Armenian citizens in Ukraine to receive electronic versions of paperwork (the number of return certificates issues to Armenian citizens in Ukraine was 981). 37. Intangible benefits for the private sector and citizens include improvements in transparency, customer satisfaction, and greater trust in government. Investments in the whistleblowing platform and electronic platform for declarations of assets, incomes, expenditures, and interests has strengthened accountability and may have reduced opportunities for corruption. The Competition Protection Commission’s integrated external website and internal information platform has enhanced transparency and accountability of anti-monopoly business practices. D. JUSTIFICATION OF OVERALL OUTCOME RATING 38. The overall outcome rating for the project is Moderately Satisfactory. This rating is based on the assessments of project relevance, efficacy, and efficiency. Split ratings are applied to differentiate the progress against the original and restructured project. The final rating, which is included in Table 5, is based on the methodology provided in Annex H and Appendix I of the updated World Bank Guidance Note for Implementation Completion and Results Reports for Investment Financing Operations, issued on December 9, 2021. Table 5. Calculation of the Overall Project Outcome Rating Rating Outcome Component Ratings Based on the Ratings Based on 2020 Original PDO Third Project Restructuring Relevance of PDO Substantial Efficacy (achievement of PDO) Substantial High Objective 1: Improved Quality of Public Financial Negligible NA Reporting Objective 2: Improved Access to Selected Enhanced E- High High governance Services Efficiency Modest Page 20 of 50 The World Bank Third Public Sector Modernization Project (P149913) Rating Outcome Component Ratings Based on the Ratings Based on 2020 Original PDO Third Project Restructuring Moderately Outcome Rating Moderately Satisfactory Satisfactory Numerical value of outcome ratinga 4 4 8,904,150 11,381,909 Disbursement (US$) (By October 27, 2020) (After October 27, 2020) Share of Disbursement (%) 43.9 56 Weighted value of the Outcome rating (numerical value 1.76 2.24 × Share of Disbursement) (4 x 0.44) (4 x 0.56) Moderately Satisfactory Final Outcome (Overall Project Outcome Rating) 4 (1.76+2.24 = 4.0) Note: a = Highly Unsatisfactory (1); Unsatisfactory (2); Moderately Unsatisfactory (3); Moderately Satisfactory (4); Satisfactory (5); Highly Satisfactory (6) (based on ICR Guidelines, Appendix 1). E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 39. PSMP 3 promoted gender-inclusive benefits and invested in gender-sensitive consultation and communications. The Project contracted a specialist agency to help manage stakeholder consultations and gather feedback from service users. Gender related risks were identified at the appraisal. The project design used appropriate M&E to measure gender-related inclusion within appropriate project activities gender disaggregated data. The project supported the government’s focus on further promoting gender equality through participatory decision-making, beneficiary feedback mechanisms, and entry points for citizen engagement. The Project established a citizen feedback platform providing gender-disaggregated data that already includes three government bodies and 50 services. Beneficiary feedback included specific grievance redress mechanisms, designed to support implementation of selected e-Governance reforms. There was no discernible gender differentiated impacts. For example, there is no gender-based difference for time processing a passport extension by the MoFA or access to e-consular services. The HRMIS system developed under the Project allows the CSC to collect gender disaggregated data and conduct gender review of women/men in public service employment, positions, and salaries. Institutional Strengthening 40. PSMP3 strengthened institutions responsible for the implementation of the e-Government Strategy. The Project supported the Public Sector Reform Committee, established by Prime Ministerial Decree, in the coordination of public sector reforms and change management activities. The e- Governance Strategy sought to develop an overall enabling environment by strengthening institutional capacity for policy design and coordination and a regulatory framework. While the project did not provide direct financial support to EKENG, the institution responsible for the coordination and management of the e-Governance agenda, it did provide consultancies for specific technical tasks related to project activities including the design of system requirements, preparation of bidding documents, and independent quality assurance. 41. PSMP3 strengthened the institutions delivering services through investments in information systems to that improved efficiency transparency and accountability of their internal operations and delivery of administrative services to citizens and the public. Management systems supported by the Page 21 of 50 The World Bank Third Public Sector Modernization Project (P149913) Project include: the assets declaration system for the Corruption Prevention Commission; the e-Criminal case system for prosecutors and other law enforcement agencies; the statistical dashboard for decision- makers operated by EKENG and used by the PMO. PSMP3 has demonstrated the value of ITC solutions and promoted widespread utilization across public agencies. Improved access to administrative services, reduced transaction costs for citizens and businesses and greater transparency is likely to strengthen trust in public institutions and generate dynamics in favor of successful reforms. Mobilizing Private Sector Financing 42. PSMP 3 did not address the mobilization of private sector financing. The Project focused on the modernization of e-Government services and building implementation capacity through reforms in the civil service. Poverty Reduction and Shared Prosperity 43. The project is expected to have impacted poverty reduction and shared prosperity by providing those below the poverty line with access to administrative services. A World Bank ICT survey (2020) found that 96 percent of Armenian households had access to the internet and 83 percent of the population used the internet daily. 72 percent of the total population (or 78 percent of internet users in the last 3 months) used mobile devices to access the internet. However, according to Armenia Digitalization Strategy of 2019, only 15 percent of population used e-gov tools during the previous 12 months, and according to EKENG, only 5 percent of population used online public services. The team was not able to identify additional information on the use of digital services following project implementation. Other Unintended Outcomes and Impacts 44. There were no unintended outcomes or impacts. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 45. The Project was aligned with Government strategies and benefitted from the support of the key stakeholders. The PDO and project activities were aligned with the Government’s PFM Reform Strategy of 2014 6 and its e-Government Strategy of 2014 7. Staff at the Prime Minister’s Office, particularly the Project Management Group, the Ministry of Foreign Affairs, the Prosecutor General’s Office, Corruption Prevention Commission, EKENG, participated in the design Components 2 and 3 and the selection of the e-Government solutions. MoF leadership recognized the need to the overhaul its information systems to support the introduction of program budgeting, strengthen reporting and internal controls, arguing that the legacy Treasury system lacked critical functionalities and had limited interoperability. MoF was committed to the introduction of a modern integrated GFMIS to support its PFM reform agenda. 46. Bank analytical work informed design of the GFMIS. The PEFA Assessment, published in May 2014, made the case for a GFMIS, arguing that MoF would need to replace the legacy Treasury system with a 6 Government Program was adopted by Government Decision N511-A of May 19, 2014, available here: https://www.gov.am/files/docs/1321.pdf 7 E-Governance Strategy was adopted on April 10, 2014 by government Decision N14-144available here: https://www.e- gov.am/protocols/item/370/ Page 22 of 50 The World Bank Third Public Sector Modernization Project (P149913) modern information system if it was to successfully implement program budgeting and strengthen accounting, reporting and internal control functions. A Bank-financed a Business Process Review (BPR) and analytical work on GFMIS design, funded under ECA PFM trust fund, provided the basis for technical specifications, estimating that the GFMIS would cost up to US$ 15.0 million. The Bank’s analytical work highlighted the technical and political economy risks associated with IFMIS projects. 47. The Bank facilitated the mobilization of parallel grant financing for the GFMIS. During the initial stages of Project preparation, the authorities intended to finance the Project with a World Bank loan and Government counterpart funds. At that time, the Ministry of Finance of the Russian Federation expressed interest in expanding its financing for PFM reforms in ECA beyond the analytical and advisory work supported through its contributions to the ECA PFM trust fund to include co-financing for a World Bank- financed investment project. PSMP 3 was selected as a suitable pilot. The Armenian MoF and the Ministry of Finance of the Russian Federation agreed that the most efficient way to mobilize this contribution would be as parallel financing which would be used to finance the Borrower’s counterpart contribution to the GFMIS. The Russian Federation’s contribution of US$8.2 million in grant funds was channeled directly to the Armenian MoF. Unfortunately, the availability of grant financing adversely impacted Project implementation because MoF preferred to expend the grant before using Bank financing, limiting the Bank’s leverage in discussions on the GFMIS design and contributing to slow progress in the first four years of the project. 48. The e-Government component was designed to focus on low-risk, high reward digitization initiatives. At appraisal PMO and EKENG selected the e-Government solutions to be financed by the Project from a list of agency proposals giving priority to proposals that demonstrated: strong ownership by the beneficiary; commitment to allocate human resources and budgets for maintenance after project closure; enhancement of an existing e-service; existing back-end automation and process reengineering. These criteria ensured that e-services selected for financing demonstrated the institutional and technical conditions for success. 49. Project preparation benefitted from the experience gained by counterparts in the preparation and implementation of PSMP 1 and 2 and other Bank-financed projects. Counterparts in PRSC and FFPMC/PMG were familiar with the task team, Bank procedures, and the project preparation and implementation process. Drawing on this experience, the authorities insisted on the inclusion of provisions for just-in-time technical assistance in the Project design (Component 3.2) that would allow the Project to respond to emerging priorities. Implementation of e-Police and e-Health components under PSMP 2 helped build stakeholder confidence and interest in the Project and the e-Government agenda. This experience also informed the criteria for selection of the most promising e-services, stressing the importance of institutional commitment and experience. The PAD rated overall project risk as moderate in large part due to FFPMC procurement staff’s familiarity with Bank procurement procedures. The PAD highlighted a number of procurement-related risks and proposed appropriate mitigation measures. The selection of appropriate stakeholders at project preparation mitigated unforeseen implementation delays associated with human resource capacity challenges. Project preparation also benefitted from the experience gained in the implementation of change management activities, notably the work of the MoF change management unit in support of the Bank-financed Tax Administration Modernization Project (TAMP, P111942). Page 23 of 50 The World Bank Third Public Sector Modernization Project (P149913) B. KEY FACTORS DURING IMPLEMENTATION Factors subject to government control 50. After a slow start, the Project benefitted from high level commitment to the e-Government agenda, even though commitment was not sustained across all of the agencies participating. In January 2019, two and half years after effectiveness, disbursements amounted to just 14 percent of the loan amount and ISR ratings for PDO and Implementation Progress were downgraded to Moderately Unsatisfactory. At this point, procurement activities were delayed owing to difficulties finalizing specifications for digital solutions (pre-trial case management system, transport information system) and the relaunch of tenders owing to lack of sufficient bidders (police emergency dispatch, electronic document management system). Activities related to the e-Transport systems were dropped owing to changing priorities in the transport agency. MoF had yet to put in place the technical team needed to support the detail design of the GFMIS specifications. Following a change in administration in January 2019, the appointment of a new DPM as Project Director and the MTR in May 2019, project implementation picked up. However, the new MoF team continued to delay work on the GFMIS, largely because the new team believed that the legacy Treasury system had sufficient functionality. MoF and the PGO finally decided to drop GFMIS in the October 2020 third restructuring. Thereafter Government commitment to the project remained strong and disbursements accelerated as the Project launched work on new e-service solutions. 51. FFFMC and PMG ensured strong fiduciary capacity and project management capacity throughout implementation. Experienced FM and procurement staff were able to support implementation of project activities. Once agency management was committed and technical staff had agreed on the specifications for project inputs the procurement staff were well-equipped to prepare and process procurement packages. After initial disruption during the COVID-19 pandemic, the PMG put in place adequate measures to support remote working and ensure continuity of operations. Factors subject to World Bank control. 52. The Bank team’s field presence and continuity in staffing throughout project implementation facilitated dialogue with counterparts and helped the Bank to respond to opportunities. The team’s knowledge of the political and institutional context and their familiarity with the history of the Bank engagement on public sector reform agenda in Armenia helped mitigate the impact of changes in counterpart personnel. The Bank team used the opportunity of the third restructuring to explore opportunities to expand project support for digital government reforms leading to reallocation the uncommitted project funds intended for the GFMIS rather than their cancellation. This expanded the scope of digitalization and modernization efforts, complementing the initial project design and extending the digitization program to more sectors. Successful implementation and strong results in e-Government triggered the government’s interest in a consecutive GovTech-focused project, which became effective before closing of the PSMP 3 through PSMP 4. Factors outside the control of government and/or implementing entities. 53. Political developments impacted the project, distracting attention from project activities and leading to changes in key personnel. The Velvet Revolution in April 2018 led to changes in political landscape and leadership in implementing agencies. There were significant delays as counterparts familiarized themselves with the project and were only ready to resume project activities in mid-2019. From March 2020 through mid-2022, the COVID-19 pandemic led to a change in priorities and initially Page 24 of 50 The World Bank Third Public Sector Modernization Project (P149913) impacted on travel and work patterns. However, the e-services and document management system financed by the Project helped mitigate the impact of lockdowns and remote working and the pandemic ultimately had limited impact on implementation. The conflict between Armenia and Azerbaijan in September to November 2020 and subsequent political unrest distracted attention from the Project. Russia’s invasion of Ukraine in February 2022 brought further instability to the region and necessitated a change in Government priorities to address economic impacts. 54. Although the market for IT services is active and competitive, the interest of IT companies to enter into public tenders is limited. IT companies are more interested in entering contracts with foreign firms. Lengthy and bureaucratic procedures for public bidding, responsibility to be bound by extra level of contractual commitments applicable to public contracts, subject to various levels of audit and inspection by public bodies, absence of strong teams on government side, make public procurement for IT community unattractive. In several cases, the World Bank advised the PMG to conduct market research and organize bidders’ conference to promote broader participation of firms in tendering processes. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 55. M&E design described in the PAD was seriously deficient. The design shortcomings differ for the two elements of the PDO and the three substantive project components. 56. The M&E design provided an adequate basis for monitoring progress against the first element of the PDO – improve quality of public financial reporting – but did not support monitoring of progress in implementation. The PDO indicator focused on the preparation of comprehensive financial statements for two pilot sectors. This constitutes an appropriate outcome, albeit with rather narrow institutional scope. However, the PDO and intermediate results indicators did not breakdown the implementation steps needed to achieve this outcome. The PDO indicator simply referred to the preparation of comprehensive financial statements from year four of the project, lacking the milestones – functional design, business process reengineering, procurement and contracting – needed to guide and monitor GFMIS implementation. These steps were referenced in the text of the PAD but are not reflected in the results framework. Similarly, the intermediate result simply referred to the rollout of newly adopted accounting standards, without indicating when they would be adopted and how implementation would be phased in. The failure to provide implementation milestones indicates was a consequence of lack of clarity and consensus on the GFMIS implementation arrangements and accounting reforms when the project was approved. 57. The M&E design did not provide an adequate basis for assessing improved access to selected enhanced e-government services as stated in the PDO but did provide an adequate framework for monitoring progress in implementation. The results framework for the second part of the PDO and Component 2 was structured around selected e-government services rather than a measure of access to e-government services as a whole. This is a reasonably approach given the limited scope of the service supported by the project as presented in the PAD. Only the PDO indicator that provided information on the number of users (an imperfect proxy for access). The PAD did not present baseline data on the number of users of government e-services in general and did not include measure the number of users of any of the other services supported by the project. Instead, the intermediate results indicators measured Page 25 of 50 The World Bank Third Public Sector Modernization Project (P149913) improvements in efficiency (reduced time for processing passport renewals, certificates of civil status, police dispatch, transmission of information on pretrial proceedings) or aspects of quality (availability of gender disaggregated data, feedback from citizens). The processing time for passport extensions by consular departments was selected as the PDO indicator because this was the only service with a direct impact on citizens and the service was considered high profile because of numerous complaints about the poor performance consular services. The results framework broke most of the indicators down into intermediate targets for each year of the project that could be used to track progress in implementation. 58. The M&E framework did not include indicators that would allow assessment of the impacts of Component 3 on capacity building. The intermediate results indicator used for Component 3 simply measured inputs: the number of trainers for the new needs-based training system in the civil service. The project did not measure the number of staff trained, the skills transferred or provide any measure of enhanced capacity. M&E Implementation 59. The PMG provided the Bank with regular annual M&E reports, submitted on time, providing forward estimates of project activities and disbursement projections. The DPM, as the Head of the Steering Committee of the PSRC, the Project Director and the PMG were responsible for monitoring of the project implementation. The PMG was responsible for data collection for intermediate results indicators, including those related to the progress of multiple e-Government services. Preparation of annual M&E reports was outsourced, as indicated in the PAD, to a firm with expertise in the subject areas of the project. The outsourced M&E contract was financed by the project. 60. While the M&E framework was updated to reflect the cancellation of Component 1 during the third restructuring, it was not updated to reflect the broadened scope of Component 2. There were minor changes to the results framework in the second restructuring to enhance the clarity of IRIs relating to Component 1. The restructuring did not correct the lack of implementation milestones for Component 1,reflecting the continued lack of clarity on GFMIS implementation arrangements. The results framework was adjusted to reflect the cancellation of Component 1 during the third restructuring. However, the results framework was not adjusted to reflect the introduction of new activities and a 50 percent increase in the budget assigned to Component 2. This proved to be a significant shortcoming. Project documentation lacks information on the baseline status of these services, intended implementation milestones, systematic reporting on progress and impacts in implementation in ISRs. M&E Utilization 61. The Bank team discussed M&E with the PMG, counterparts, and the M&E contractor during implementation support missions. Aide Memoires and biannual Implementation Summary and Results Reports (ISRs) on project progress included updates of the results framework. ISRs provided a candid assessment of progress made and challenges. ISR ratings on for PDO and implementation progress were realistic. The Bank team downgraded the project to reflect delays in project implementation. 62. The extent to which the results framework was used to guide project implementation is less clear. The Bank team reported on the difficulties encountered in the launch of preparatory work for the GFMIS in Aide Memoires and ISRs but this information was not included in the results framework. The implementation steps for the activities under Component 2 were included in the initial results framework and were reported in ISRs. However, the results framework provided no information on the new activities Page 26 of 50 The World Bank Third Public Sector Modernization Project (P149913) under Component 2 introduced at the third restructuring. Progress on these activities was discussed in Aide Memoires and ISRs but without reference to previously agreed milestones because there were none. Justification of Overall Rating of Quality of M&E 63. The overall M&E quality is rated Modest. The PMG and Bank team complied with the formal requirements of project reporting. However, there were serious deficiencies in the design of the results framework that prevented its use as a tool for project management. Component 1 lacked information on milestones needed to guide and track progress. This was not corrected at the time of the second restructuring following the Mid-Term Review in July 2019, which had focused on the delays in Component 1, largely because there was still no firm agreement on the implementation arrangements for the GIFMIS. The third restructuring significantly expanded the scope of activities under Component 2 but none of these activities were reflected in the results framework. The lack of information on the baselines and results achieved by e-services added at the third restructuring hindered efforts to assess project impacts in this ICR. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 64. PSMP 3 did not trigger environmental and social safeguards policies. The Project was rated as Category C for environmental purposes at appraisal. The project did not entail activities that could cause social risk, negatively affect the population, required land acquisition, or caused any physical or economic displacement. No adverse social or environmental impacts were identified during implementation. The EA Category remained C throughout. 65. No significant fiduciary compliance issues were observed. The fiduciary risk was rated Moderate throughout implementation. The project complied with the financial covenants of the Legal Agreement including the maintenance of an acceptable FM system and the on-time submission of acceptable audited project financial statements to the Bank. The interim financial reports were received on time and were generally of acceptable quality. The project complied with the Bank’s procurement guidelines. Minor deviations identified by procurement post-reviews typically pertained to procurement process timelines which did not affect project implementation. C. BANK PERFORMANCE Quality at Entry 66. Bank support during project preparation was Moderately Satisfactory. 67. The Bank team ensured that the project was aligned with the relevant Government of Armenia strategic priorities and the Bank’s CPS. All three substantive components supported recently approved government strategies. The Bank team had built a strong relationship with counterparts during the implementation of PSMP 2 and drew on the experience from this project to inform the design of PSMP 3. Project implementation readiness differed across the three substantive components. Component 1 was underdeveloped, reflecting the limited preparatory work on the design of the GFMIS and PFM reforms it was intended to support. The design for Component 2 was well developed, building on prior experience in Armenia in the implementation of e-government reforms it offered targeted support for e-Government services selected on the basis of well-defined criteria considering readiness, commitment and potential impacts. Similarly, Component 3 identified specific actions in support of civil service reforms. The results framework for Component 2 provided a robust framework for monitoring implementation but was not aligned with the PDO. The results framework for Component 3 focused on inputs rather than impacts. Page 27 of 50 The World Bank Third Public Sector Modernization Project (P149913) 68. The Bank decided to proceed with the GFMIS even though preparatory work had not yet started, seizing the opportunity to undertake a reform that posed high risks but offered high rewards. The Bank team had extensive experience with FMIS reforms with a mixed implementation record worldwide 8. The experience was discussed during preparation and lessons learned reflected in project design. The technical team argued for a standalone project to support PFM reforms and GIFMIS implementation recognizing that the Ministry of Finance would need to take the lead in implementation of a GFMIS project, such a project would require prior investment in system design and PFM reforms and would take six to seven years to complete. However, the Government and Bank management preferred to include GFMIS implementation in the PSMP project led from the Deputy Prime Minister’s Office, noting the successful track record of driving reforms in agencies from the center of government. The PAD identified the challenges posed by the lack of adequate preparation for the GFMIS noting that the project would need to develop technical specifications, bidding documents, and a detailed systems design for software development, testing, and operational acceptance during the first year of implementation. The PAD highlighted the risk that “the procurement and implementation of a complex GFMIS is often subject to delay”. The Bank hired an international adviser to assess the compliance of IT solutions with system requirements and support system development. The Bank team acknowledged – but underestimated – the potential for internal resistance to GFMIS reforms, particularly from advocates for the legacy financial management information system. Quality of Supervision 69. Bank performance during supervision is rated Moderately Satisfactory. The Bank team provided timely, high quality operational and technical advice to the authorities. However, Bank supervision was deficient in two respects: first, the Bank failed to agree on a schedule for the preparatory work for the GFMIS and decision points for GFMIS implementation; and second, the team did not strengthen indicators for measurement of access to e-services and did not provide any indicators to monitor progress in the implementation or impact of new activities under Component 2 introduced at the third restructuring (addressed under monitoring . 70. The Bank team provided proactive support throughout implementation. The Bank team maintained continuity despite change of two task team leaders, and one extended as co-leader until the end of the project. The Bank team brought in technical specialists to advise the client when needed, including fielding of a technical mission focused on GFMIS-related issues in June 2019 outside of the standard supervision cycle. Videoconferences and visits were arranged to support e-Government reforms. The engagement of staff from the country staff greatly facilitated communication, building of trust between the Bank technical specialists and counterparts, and allowed a realistic assessment of project risks and dialogue on mitigation measures. ISRs and Aide Memoires were informative and met the required Bank standards. While there were significant deficiencies in the design of the results framework, this did not materially affect Bank support to implementation because the implementation issues were addressed in supervision documents (it has hindered the assessment of project impacts). The MTR mission review was staffed with a strong mix of technical experts and identified the changes needed to expedite project implementation. The Bank team included experienced FM and procurement staff members regularly engaged with the 8See Denner, Cem, Joanna Watkins, and William Dorotinsky. 2011. Financial Management Information Systems: 25 Years of Experience of What Works and What Doesn’t. (https://issuu.com/world.bank.publications/docs/9780821387504) Page 28 of 50 The World Bank Third Public Sector Modernization Project (P149913) PMG to help address challenges in these areas. The Bank team was instrumental in securing parallel financing from the Russian MoF which was concluded shortly after Board approval of the project. All stakeholders expressed their appreciation of the Bank team’s commitment, quick responses, technical advice and assistance when required, and their willingness to streamline processes and procedures to accommodate the client’s requirements. The Bank demonstrated its flexibility when it reallocated funds to Component 2 to expand the scope of project activities during the third restructuring rather than cancelling the unused balance from Component 1. 71. The Bank team established efficient mechanisms for cooperation with development partners supporting public sector modernization reforms. In the absence of government-run formal donor coordination platforms, the Bank team convened biannual donor coordination meetings to discuss ongoing and planned support to the sector, bringing together representatives of the European Union (EU), Council of Europe (CoE), United States Agency for International Development (USAID), United States Department of State, German Agency for International Cooperation (GIZ), and United Nations Development Programme (UNDP). Bank staff held frequent meetings with development partners in the field coordinate support for reforms and project activities. This coordination helped leverage opportunities, such as cooperation with USAID using its TA to inform technical specifications for the asset declaration platform developed by PSMP3 for Corruption Prevention Commission, and ensured effective division of labor, such as agreement whereby GIZ to focused on regional and PSMP3 on national-level public sector reforms. 72. In retrospect one could argue that the Project should have cancelled Component 1 earlier. Component 1 was finally cancelled in October 2020, five years after Project approval, when it was clear that there was no possibility of implementing the GFMIS within the extended project timeframe and the Ministry of Finance confirmed that it did not wish to proceed. Throughout implementation the Bank sought confirmation of the Government’s commitment to the GFMIS and stressed the need to proceed with preparatory work. The authorities consistently reaffirmed their commitment to the GFMIS even through there was no consensus among the key stakeholders in the Ministry of Finance. Decision making was hampered by the lack of an agreed implementation schedule for GFMIS preparatory work identifying decision points for the authorities and the Bank. Justification of Overall Rating of Bank Performance 73. World Bank performance is rated Moderately Satisfactory over the life of the project. The Bank team sought to minimize the impact of implementation delays on the achievement of the PDO and proactively engaged with the client and the relevant counterparts to make requisite changes that were necessary to achieve the PDO. However, monitoring was hampered by deficiencies in the result framework, particularly as regards the lack of adequate milestones to guide and monitor implementation. D. RISK TO DEVELOPMENT OUTCOME 74. The risk to development outcomes is low. The project achievements are linked to the government’s multi-year strategic vision enshrined in the e-Governance strategies. The Government is committed to the digitalization agenda and has followed-up PSMP 3’s successful e-government reforms with PSMP 4 (P176803). The PDO for PSMP 4 is to improve the efficiency of and access to selected public services for businesses and citizens. The project’s four components support: the modernization of public administration; digital service delivery; data management and digital infrastructure; and project management. Page 29 of 50 The World Bank Third Public Sector Modernization Project (P149913) V. LESSONS AND RECOMMENDATIONS 75. The commitment financing for government financial management information systems before the authorities have agreed on implementation arrangements poses a significant risk. Ideally, Bank financing for IFMIS should be committed when the authorities have approved a PFM reform strategy, defined the specifications for the information systems needed to support the strategy, and agreed on the procurement and implementation for the systems. This allows IFMIS implementation to start immediately the project is approved. All too often – as in the case of PSMP 3 – most of this preparatory work is undertaken during project implementation. As a result, discussions about the system design, impact on legacy systems, and implementation modality (in-house v. customized off-the-shelf) take place after project approval. These issues are often contentious and a source institutional resistance. 76. Results frameworks for institutional reforms should include milestones for the implementation steps needed to achieve the intended results. Results frameworks that only provide information on the incremental achievement of outcomes are not particularly useful for monitoring progress in project implementation. Lack of implementation milestones hindered decision making on the IFMIS. The results framework should how to the proposed activities will lead to expected results/outcomes. Dated milestones can help focus attention and instill a sense of urgency to decision making. Information on the implementation steps can facilitate dialogue with stakeholders, ensure that they have the same understanding of the implementation progress, challenges, and impacts on activities and expected outcomes. 77. The PSMP series of projects demonstrates the cumulative impact of Bank support over a decade. In the case of PSMP there was no formal commitment to a series of projects. Instead, each PSMP was proposed by the authorities as the previous operation was approaching its closing date. Future Bank engagements supporting public sector reform might benefit from more systematic planning over a medium- to long-term horizon. Initial assistance can deliver easy wins to establish effective working relationships and structures, build confidence, and increase buy-in thereby creating an enabling environment for more challenging reforms, such as e-Government and civil service, which are difficult to deliver in the short term. 78. The Bank may have to step in to facilitate coordination between development partners support where the government does not provide a formal coordination forum. The PSMP3 team has established a sound cooperation platform with development partners and through biannual donor coordination meetings. Donor coordination allowed the project to avoid duplication of efforts and collaborate on the design and implementation of reforms. TA funded by development partners played an important role in the development of digital solutions under PSMP3. . Page 30 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve access to selected enhanced e-government services. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Average processing time for Hours 500.00 48.00 24.00 public services (hours)(Comment: Reduced 01-Jan-2016 31-Dec-2020 30-Jun-2022 time for processing a passport extension request by the Consular Department of the Ministry of Foreign Affairs. Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 31 of 50 The World Bank Third Public Sector Modernization Project (P149913) Increased access to consular Number 0.00 10,000.00 19,691.00 services for issuing citizenship, civil status 01-Jan-2016 31-Dec-2020 30-Jun-2022 certificates and statements on any existing criminal records (data disaggregated by gender) Comments (achievements against targets): A.2 Intermediate Results Indicators Component: e-Governance Solutions for Improved Service Delivery Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reduced time for police Text 15-20 minutes Under 5 minutes Supply and installation dispatch in emergency of pilot operational situations (in at least two management systems selected pilot for RA Police are regions/marzes). almost accomplished. The full roll-out of the system will be completed in 2022 after installation and testing of the last two modules envisaged by Page 32 of 50 The World Bank Third Public Sector Modernization Project (P149913) the contract. 01-Jan-2016 31-Dec-2020 30-Jun-2022 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increased share of citizens Text No feedback 20.00 The feedback (disaggregated by gender) mechanism in place. mechanism is providing voluntary feedback currently operational. on specific public services The Civil Acts (percentage - data generated Registration Agency, from system) the State Register of the Legal Entities (E- Register) and the Road Police Service are integrated to the system. The mechanism operates for the above- mentioned 3 entities and 74 services. 20% of all users of specific public services provide voluntary feedback on selected public services. Page 33 of 50 The World Bank Third Public Sector Modernization Project (P149913) 01-Jan-2016 31-Dec-2020 30-Jun-2022 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reduced time to transmit Text One week by special Immediate (electronic The supplier has information/messages express mail (send via transmission) already developed and relevant for the pre-trial post office). presented the first proceedings between demo version of the prosecutors’ offices in CCPTEMS to the client different regions. in 2021. The development of the final version of the system is currently in process. 01-Jan-2016 31-Dec-2020 30-Jun-2022 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Increased gender- Text No gender Gender disaggregated 46% male, 54% female disaggregated data is disaggregated data data on eservices is Page 34 of 50 The World Bank Third Public Sector Modernization Project (P149913) available on the use of e- available in line available to ministry services which are provided ministries / agencies of foreign affairs; by selected agencies which will provide e- Prosecutor’s services. Department; Ethics Commission; Police 01-Jan-2016 31-Dec-2020 01-Sep-2021 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Reduced time to issue the Days 60.00 10.00 10.00 certificate of civil status by the Consular Department of 01-Jan-2016 31-Dec-2020 30-Jun-2022 the Ministry of Foreign Affairs Comments (achievements against targets): Component: Capacity Building and Small Capacity Building Interventions Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of trainers for the Number 0.00 50.00 68.00 Page 35 of 50 The World Bank Third Public Sector Modernization Project (P149913) new needs-based training 01-Jan-2016 31-Dec-2020 30-Jun-2022 system in the civil service established under the Project. Comments (achievements against targets): Page 36 of 50 The World Bank Third Public Sector Modernization Project (P149913) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 1. Average processing time for public services (hours)(Comment: Reduced time for Outcome Indicators processing a passport extension request by the Consular Department of the Ministry of Foreign Affairs. (Hours, Custom) 1. Reduced time for police dispatch in emergency situations (in at least two selected pilot regions/marzes) 2. Increased share of citizens (disaggregated by gender) providing voluntary feedback Intermediate Results Indicators on specific public services (percentage - data generated from system) 3. Reduced time to transmit information/messages relevant for the pretrial proceedings between prosecutors’ offices in different regions. 1. Supply and installation of pilot operational management systems for RA (Republic of Armenia) Police are accomplished. Key Outputs by Component 2. The feedback mechanism is introduced and currently operational for 81 services (linked to the achievement of the Objective/Outcome and in 7 entities. 1) 3. e-Criminal case system is available and in a phased implementation process applied in real criminal cases and a real-time data exchange is on between the PGO and its regional offices. Objective/Outcome 2 1. Increased access to consular services for issuing citizenship, civil status certificates, Outcome Indicators and statements on any existing criminal records (data disaggregated by gender) (Number, Custom) 1. Increased gender-disaggregated data are available on the use of e-services which are provided by selected agencies Intermediate Results Indicators 2. Reduced time to issue the certificate of civil status by the Consular Department of the Ministry of Foreign Affairs Page 37 of 50 The World Bank Third Public Sector Modernization Project (P149913) 3. Number of trainers for the new needs-based training system in the civil service established under the Project. 1. The citizen feedback platform was developed, which helps collect gender- Key Outputs by Component disaggregated data on e-services. (linked to the achievement of the Objective/Outcome 2. Electronic system for e-Consular was developed, which allows to apply and receive 2) civil status certificates within limited time. 3. ToTs were organized and 68 people were trained. Page 38 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Migara Oshadi De Silva, Davit Melikyan Task Team Leader(s) Armine Aydinyan Procurement Specialist(s) Garik Sergeyan Financial Management Specialist Joseph Huntington La Cascia Team Member Artak Azizyan Team Member Armine Grigoryan Team Member Olga Sipka Team Member Aleksandre Bibileishvili Team Member Tigran Kostanyan Team Member Sarah G. Michael Social Specialist Tina George Team Member Rocio Mariela Malpica Valera Counsel Sandra Sargent Team Member Gayane Davtyan Team Member Jose C. Janeiro Team Member Andres Eduardo Mac Gaul Team Member Darejan Kapanadze Social Specialist Zaruhi Tokhmakhyan Team Member Page 39 of 50 The World Bank Third Public Sector Modernization Project (P149913) Stepan Anatolievich Titov Team Member Seda Pahlavooni Team Member Dolly Elizabeth Teju Team Member Supervision/ICR Arman Vatyan Task Team Leader(s) Alexander Astvatsatryan, Rahmoune Essalhi, Procurement Specialist(s) Sepehr Fotovat Ahmadi Lusine Asatryan Financial Management Specialist Satik S. Nairian Team Member Sidet Kim Team Member Khuram Farooq Team Member David Jijelava Social Specialist Oksana Rakovych Environmental Specialist B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY15 52.713 225,116.62 FY16 16.428 86,266.83 FY17 -1.350 -1,610.82 Total 67.79 309,772.63 Supervision/ICR FY16 0 2,151.40 FY17 23.750 75,904.06 FY18 21.837 101,133.07 Page 40 of 50 The World Bank Third Public Sector Modernization Project (P149913) FY19 20.213 88,438.43 FY20 10.325 85,691.96 FY21 18.343 150,367.27 FY22 15.720 90,790.49 FY23 7.735 121,948.38 Total 117.92 716,425.06 Page 41 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 3. PROJECT COST BY COMPONENT Amount at Actual at Percentage Components Approval Project Closing of Approval (US$ millions) (US$ millions) 1 Public Financial Management 19.00 0.38 2.0 Information Systems 2 e-Governance Solutions for Improved 14.00 21.50 153.6 Service Delivery 3 Capacity Building and Small Capacity 1.85 1.85 100.0 Building Interventions 4 Project Management 1.85 1.85 100.0 Total 36.70 25.58 69.7 Page 42 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 4. EFFICIENCY ANALYSIS The Annex is intentionally left blank. Page 43 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Page 44 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) 1. Aide Memoires of World Bank Supervision Missions. 2. Aide Memoire of Midterm Review (MTR) Mission, May 20–31, 2019. 3. Final M&E Report on Project Performance and Results Indicators (prepared by the Project Management Group). 4. Implementation Status and Results Reports (ISRs) - Seq No: 1–No:13. 5. Project Appraisal Document (PAD) - World Bank. 6. Project Restructuring Papers 1, 2, and 3 - World Bank. Page 45 of 50 The World Bank Third Public Sector Modernization Project (P149913) ANNEX 7. LIST OF 120 SERVICES DIGITIZED UNDER PSMP3 1 Organization of casinos 2 Trade in explosives, blasting equipment and execution of blasting works 3 Production of civil and service weapons 4 Production of military weapons 5 Arms trade 6 Production of narcotic drugs or psychotropic substances or their precursors established by the Government of the Republic of Armenia 7 Import, export or wholesale trade of narcotic drugs or psychotropic substances or their precursors established by the Government of the Republic of Armenia 8 Use of drugs and psychoactive substances for scientific and educational purposes 9 Examination with the use of narcotics and psychoactive substances 10 Production of industrial hemp 11 Import, export or wholesale of industrial hemp 12 Import and export of military products 13 Transit transportation of military products 14 Intermediary activity of trade in military products (contract negotiations for other persons in the process of trade in military products, as well as support services for acquisition, transfer and sale of these products) 15 Private escrow activity 16 Production of medication 17 Pharmacy activity 18 Medical care and maintenance 19 Wholesale sale of medication 20 Natural gas transportation, distribution, import, export 21 Electricity production (including cogeneration of electricity and heat), transmission, distribution, supply of electricity (including guaranteed), wholesale 22 Thermal energy production, transportation, distribution 23 Electric power and gas supply system operator services 24 Energy market operator services 25 Supply potable water supply and/or drainage (sewage treatment) services 26 Implementation of preschool educational program and (or) preschool service 27 Implementation of elementary general education program 28 Implementation of basic general education program 29 Implementation of secondary education program 30 Implementation of pre-vocational education program 31 Implementation of vocational education program 32 Implementation of bachelor's education program Page 46 of 50 The World Bank Third Public Sector Modernization Project (P149913) 33 Implementation of Master's education program 34 Public electronic communications network 35 Broadcasting of television and radio programs 36 Private multiplexer activity 37 Postal activity 38 Site selection or design or construction or operation or decommissioning of nuclear facilities or radioactive waste storage facilities or radioactive waste landfills 39 Import or export of nuclear materials or radioactive materials or devices containing radioactive materials or radioactive waste 40 Production or use or transportation or storage of nuclear materials 41 Production or use or transportation or storage of radioactive materials 42 Manufacture or use or repair or assembly and adjustment or transportation or storage of devices containing radioactive materials 43 Use or adjustment of generating radiation sources 44 Processing or storage of radioactive waste 45 Implementation of physical protection of nuclear installations and nuclear materials 46 Design or preparation of devices, equipment, systems important from the point of view of safety for facilities using atomic energy 47 Examination of projects and other documents of facilities for the use of atomic energy 48 Activities of persons that have important positions in the field of atomic energy use from the point of view of safety 49 Use of hazardous waste 50 Organization and implementation of lotteries 51 Termination of organization of lotteries 52 Organization of games with winnings 53 Organization and implementation of Internet winning games 54 Termination of organizing Internet winning games 55 Organization of railway activities 56 Implementation of technical inspection of vehicles 57 Suspension of technical inspection of vehicles 58 Realization of passenger transportation with one passenger-taxi car 59 Organization of transport by passenger-taxi cars by individual entrepreneurs and organizations 60 Installation of gas cylinder equipment on motor vehicles for working with compressed natural or liquefied petroleum gas and (or) implementation of periodic certification of gas cylinders 61 Organization of airport taxi service buy individual entrepreneurs and organizations 62 Development of engineering sections of urban planning documents (except for the construction part, as well as works that do not require a construction permit) 63 Examination of urban planning documents (except for works that do not require a construction permit) Page 47 of 50 The World Bank Third Public Sector Modernization Project (P149913) 64 Implementation of construction (except for works that do not require a construction permit, individual residential houses with up to two above-ground and one underground floor, not exceeding 300 square meters, facilities being built according to approved multi-use model projects, auxiliary buildings not exceeding 150 square meters in total area, garages with a total area not exceeding 50 square meters, greenhouses with a total area not exceeding 1000 square meters, non-main buildings of agricultural production purpose not exceeding a total area of 500 square meters and non-main buildings of public importance not exceeding a total area of 100 square meters) 65 Technical supervision of quality of construction (except for works that do not require a construction permit, individual residential houses with up to two above-ground and one underground floor, not exceeding 300 sq.m, facilities being built according to approved multi-use model projects, auxiliary buildings not exceeding 150 square meters in total area, garages with a total area not exceeding 50 square meters, greenhouses with a total area not exceeding 1000 square meters, non-main buildings of agricultural production purpose not exceeding a total area of 500 square meters and non-main buildings of public importance not exceeding a total area of 100 square meters) 66 Engineering research 67 Research of technical state of buildings and facilities 68 Implementation of examinations on activities established by law 69 Trade of transportation means (including agricultural technics) 70 Implementation/termination production of biological preparations used in the field of veterinary medicine 71 Manufacture, import or trade of fireworks products 72 Suspension of manufacture, import or trade in fireworks 73 Implementation of production of biological preparations used in the field of veterinary medicine 74 Suspension of production of biological preparations used in the field of veterinary medicine 75 Suspension of veterinary sanitary examination 76 Suspension of veterinary sanitary examination 77 Production and (or) sale of chemical and biological means of plant protection 78 Suspension of production and/or sale of chemical and biological means of plant protection 79 Public electronic communications services, except for those services for supply of which an implementation of electronic communications network is required 80 Public electronic communications services, except for those services for supply of which an implementation of electronic communications network is required, termination 81 Implementation of drilling activities for the purpose of using underground freshwater 82 Suspension of drilling activities for the use of groundwater 83 Realization and hallmarking of items made of precious metals 84 Suspension of hallmarking and hallmarking of articles made of precious metals Page 48 of 50 The World Bank Third Public Sector Modernization Project (P149913) 85 Distilled alcoholic beverages (with the exception of the production and sale of alcoholic spirits obtained from the distillation of grape wine or grape juice classified under code 220820 of the "External economic activity commodity list" (ATG AA), liqueurs classified under code 220870 and other alcoholic beverages containing up to 9% alcohol classified under code 220890, Manufacture of alcoholic beverages containing 40% or more alcohol distilled from honey, fruits and berries, as well as grape vodkas and grape vodka distillates classified under code 2208) and production of ethyl alcohol from fermentation materials 87 Implementation of beer production 88 Suspension of beer production 89 Production of grape vodka and grape vodka distillates 90 Suspension of production of grape vodka and grape vodka distillates 91 Production of wine, apple and other fruit wines and other alcoholic drinks made from honey 92 Suspension of production of wine, cider and other fruit wines and other alcoholic beverages made from honey 93 Implementation and organization of the place of trade 94 Suspension of organization of place of trade 95 Implementation of genetic engineering activities 96 Suspension of genetic engineering activities 97 Production of alcoholic spirits obtained from the distillation of grape wine or grape juice falling under code 220820 of the "External economic activity commodity list" (ATG AA) classifier and sale of products up to 100 thousand liters (calculated as 100 percent alcohol) 98 After the sale of 100,000 liters (calculated as 100 percent alcohol) by producers of grape wine or grape juice distillate under the code 220820 of the "External economic activity commodity list" (ATG AA) classifier, sale of other batches 99 Production of grape wine or alcoholic spirits obtained from the distillation of grape juice falling under the code 220820 of the "External economic activity commodity list" (ATG AA) classifier, without the right of sale 100 Production of liquors classified under code 220870 of the "Commodity list of foreign economic activity" (FPA) and other alcoholic beverages containing up to 9% alcohol under code 220890 101 Production of alcoholic beverages containing 40% or more alcohol distilled only from honey, fruits and berries (except grapes) classified under code 2208 of the "Commodity List of Foreign Economic Activities" 102 Import of vodka under code 2208 of the "External economic activity commodity list" Page 49 of 50 The World Bank Third Public Sector Modernization Project (P149913) 103 "Releasing for domestic consumption" "Releasing for domestic consumption", "Processing" of up to 100,000 liters (calculated as 100 percent alcohol) of grape wine or grape juice distillate under the code 220820 of the "Commodity list of external economic activity" (ATG AA) classifier "in the customs territory" and "Reprocessing for domestic consumption" customs procedures, as well as imports from EAEU member states, including imports for the purpose of processing 104 After importing 100,000 liters (calculated as 100 percent alcohol) of grape wine or grape juice distilled from the distillation of grape wine under code 220820 of the "External Economic Activity Commodity List" (ATG AA) classifier, importers of other batches from non-EEU member states "Exemption for domestic consumption" ", "Processing in the customs territory" and "Processing for domestic consumption" customs procedures import, as well as import from EAEU member states, including import for the purpose of processing 105 Purchase and sale, import, export, transportation of natural, processed or unprocessed, but unframed or loose diamonds classified under the codes 710210000, 710221000, 710231000 of the commodity lists of foreign economic activity (FET AA)". 106 Certificate on evidence of conviction and wanted status 107 Certificate of marital status 108 Consular registration for more than 180 days 109 Consular record less than 180 days 110 Registration of an individual entrepreneur 111 Termination of citizenship. Waiting for the application to be received 112 Application for copy of birth certificate 113 Application for copy of death certificate 114 Application for copy of marriage certificate 115 Application for a copy of divorce certificate 116 Application for a copy of the paternity certificate 117 Application to change the data of an individual entrepreneur 118 Application for a copy of the state registration certificate of a legal entity 119 Qualification application of construction organization 120 Application for comprehensive examination Page 50 of 50