Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) Report Number: ICRR0022579 1. Project Data Project ID Project Name P111795 NI PFM Modernization TAL Country Practice Area(Lead) Nicaragua Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-48070,IDA-55790,TF-14059 31-Dec-2015 38,353,816.12 Bank Approval Date Closing Date (Actual) 07-Dec-2010 31-Aug-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 3,351,580.26 Revised Commitment 38,346,121.83 3,346,121.83 Actual 38,353,816.12 3,346,121.83 Prepared by Reviewed by ICR Review Coordinator Group Daniel Nogueira-Budny Clay Wescott Jennifer L. Keller IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The original project development objective (PDO) as set out in the Financing Agreement (p. 5) was to enhance the efficiency, performance orientation, and transparency of the Recipient’s public expenditures management. The Project Appraisal Document (PAD, p. 5) had the same formulation. Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) A revised PDO was introduced in 2014 with Additional Financing (Report No. 90599-NI): to enhance the efficiency and transparency of the Recipient’s public expenditure management and strengthen the Recipient’s statistics management capacity (p. 4, see also p 5. of the Financing Agreement for the Additional Finance, dated 6 November 2014). The ICR makes this distinction between the original and revised PDO on p. 7 but erroneously cites the revised PDO as the original PDO on p. 1. For the purposes of the ICRR, the revised PDO can be broken down into three objectives: to enhance the efficiency of the Recipient’s public expenditure management, to enhance the transparency of the Recipient’s public expenditure management, and to strengthen the Recipient’s statistics management capacity. What follows is an assessment of the project’s achievement of these three objectives including a rating for each. (Since the portion of the PDO that was dropped did not witness any significant activities prior to restructuring, the ICRR will follow the lead of the ICR and not undertake a split PDO evaluation). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 07-Nov-2014 c. Will a split evaluation be undertaken? No d. Components Originally, the project had four components: Component 1: Strengthening of institutional capacities of the Ministry of Finance (Originally US$2.1M, revised to US$4.5M) The objective of this component was to strengthen institutional capacities and support the operation of lead entities within the Ministry of Finance and prepare for implementation of a new financial management information System. The component had eight sub-components: Sub-component 1.1: Enhancing the budgeting system: This sub-component was designed to enhance the Ministry of Finance and Public Credit’s (MHCP) budgeting system, through: (i) redefinition of budgeting functions; (ii) strengthening of MHCP’s capacity to gather, integrate and consolidate information from municipal budgets; and (iii) strengthening of MHCP’s capacity to produce, interpret, and publish fiscal statistics. Sub-component 1.2 - Strengthening of the accounting system: This sub-component was designed to strengthen MHCP’s accounting system, through: (i) redefinition of accounting functions and support of its adaptation and implementation; (ii) development of new accounting classifications according to international standards and preparation of necessary manuals and guidelines for their dissemination, and provision of training; (iii) review and improvement of standards and quality of existing Government financial statements; Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) (iv) and preparation of specifications for Government’s asset management system and registration of Government assets. Sub-component 1.3 - Strengthening of the treasury system: This sub-component was designed to support the strengthening of MHCP’s treasury system, through: (i) redefinition of treasury management functions and support of its adaptation and implementation; (ii) review and analysis of specifications for registration and integration of revenue management information into the operations of a single treasury account for Central Government agencies; (iii) promotion of gradual de-concentration of payment functions and implementation of a single treasury account in Central Government agencies; and (iv) provision of training to the MHCP’s treasury office’s personnel in topics related to cash programming, placement, and administration of debt. Sub-component 1.4 - Strengthening of the Public Credit Office: This sub-component was designed to provide technical assistance to strengthen the MHCP’s public credit system to implement and monitor the Government’s national public debt strategy through: (i) acquisition and adaptation of a new software version for Government’s debt management system and development of its interface with existing and new Systems; (ii) provision of technical assistance and training for the implementation of the new software version; (iii) analysis and evaluation of the existing public debt portfolio and development of an updated public debt strategy; and (iv) provision of training to personnel in debt management-related topics. Sub-component 1.5 – Strengthening and integration of the public procurement system: This sub- component was designed to strengthen and integrate the Government’s procurement system, through: (i) redefinition of public procurement functions and development of technical specifications for the integration of the Government’s public procurement system into the new System; (ii) support for initiatives to optimize procurement practices; and (iii) enhancement of MHCP’s procurement office technical capabilities and the provision of specialized training. Sub-component 1.6 – Strengthening of the civil service management unit: This sub-component was designed to provide support to strengthen MHCP’s civil service system, through: (i) redefinition of human resource management functions and the development of technical specifications for the integration of the Government’s civil service information system into the new System, and support for its adaptation and implementation; (ii) development of cross-checking mechanisms to validate personnel information with other related databases; (iii) implementation of the post-classification and personnel performance evaluation modules under Government’s civil service information system; (iv) and training of personnel to operate new functions of the civil service information system and interface them with the new System. Sub-component 1.7 – Strengthening of the National Public Investment System: This sub-component would help strengthen the Government’s National Public Investment System, through: (i) redefinition of the Government’s national public investment system and the development of technical specifications for integration into the new System and support for its adaptation and implementation; (ii) design of instruments and mechanisms to enhance the quality of public investment projects. (iii) design, development, and implementation of a public investment transparency portal to facilitate broad dissemination and monitoring of public investment projects; and (iv) updating and dissemination of a specialized training program to personnel on public investment projects’ preparation, evaluation, and implementation. Sub-component 1.8 - Strengthening of the internal audit system: This sub-component was designed to provide support to strengthen MHCP’s internal audit office, through: (i) design and development of internal auditing manuals and guidelines with specific procedures for the auditing of each System module; (ii) Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) provision of training to MHCP’s internal audit office personnel; and (iii) design and drafting of specifications for the development of a specialized information system to help perform auditing functions on each of the automated modules of the new System. Component 2: Design, acquisition, and development of a renewed Integrated Financial Management System (SIGAF) (Originally US$5.4M, revised to US$13.1M). The objective of this component was to modernize the PFM information system by developing a new functional model, which would consolidate and integrate all the core financial management and administrative processes of the Central Government, with the possibility to be expanded and implemented by decentralized entities and municipalities. This component had three sub-components. Sub-component 2.1 - Acquisition and configuration of the technology architecture and software applications to support a web-based integrated system: This subcomponent was designed to adapt and customize the technology architecture and software applications to support the establishment of the New Software Application, through: (i) preparation of terms of reference and provision of technical support for the bidding process for the acquisition of Application; (ii) acquisition and configuration of the Application; (iii) provision of specialized supervision for implementation of Application; (iv) provision of training to MHCP’s personnel on Application; and (v) preparation of technical documentation for implementation process of Application. Sub-component 2.2 - Design and acquisition of technology infrastructure: This subcomponent was designed to support the design and installation of the technology infrastructure to support the operation of the New Software Application, through: (i) definition of hardware architecture and development of technical specifications for required equipment and supporting goods; (ii) acquisition and installation of computer equipment, telecommunications, security, and other necessary supporting goods and software licenses; (iii) hiring of specialized technical assistance support for data-base administration and maintenance; (iv) provision of technical assistance to certify compliance with information security standards; (v) installation and testing of telecommunication facilities; and (vi) provision of training to MHCP’s personnel on the management and maintenance of installed hardware and supporting goods. Sub-component 2.3 – Strengthening of the MHCP’s Technology office: This subcomponent was designed to strengthen MHCP’s technology office, through: (i) financing of salaries on a declining basis of specialized civil service personnel in MHCP’s technology office; (ii) provision of general information technology training to MHCP’s technology office personnel; and (iii) financing of incremental costs of software licenses and other operating costs for the implementation of existing PFM information system. Component 3: Implementation of SIGAF and PFM capacity building in Central Government agencies (Originally US$1.2M, revised to US$5.9M). The objective of this component was to support the installation and roll-out of the new SIGAF in the Central Government (and eventually in all decentralized agencies and municipalities depending on resource availability), and capacity building efforts to develop users’ capabilities to operate the system. This component had two sub-components. Sub-component 3.1 - Installation and roll-out of the new SIGAF: This subcomponent was designed to support installation and implementation of the new PFM information system, through: (i) provision of technical support for installation and implementation of system at MHCP’s central units and at spending Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) units of all Central Government agencies; (ii) preparation of support material for installation, operation and implementation of system; (iii) review of existing Legal Framework and drafting of amendment proposals when deemed necessary; and (iv) strengthening of MHCP’s technology office to provide support to users. Sub-component 3.2 - Development of a permanent training and knowledge management network on PFM and related public sector administration topics: This subcomponent was designed to develop a permanent training and knowledge management network on PFM and related public sector administration topics through: (i) formulation of a knowledge management model and validation with users; (ii) drafting of necessary guidelines and specific procedures for knowledge management network to be established and operational; (iii) provision of training to MHCP’s central units personnel and other spending units’personnel in the Central Government; (iv) carrying out of surveys and establishment of focus groups to gather information and feedback from users; and (v) implementation of a permanent dissemination program of good practices and information sharing among users. Component 4: Project Management (Originally US$1.2M, revised to US$3.0M). The objective of this component was to provide support to the MHCP to oversee and administer the implementation of the project including the Bank’s funding as well as the co-financing funds. Significant Changes During Implementation A Level 1 restructuring (P150743) was approved on November 7, 2014 (Report No: 90599-NI), bringing in an additional US$25 million in credit to the project, revising the PDO and key indicators, and introducing a fifth component. Several restructurings (nine in total) led to significant reallocations between disbursement categories and changes in loan closing dates. The ICR erroneously notes that this restructuring—the only one that changed components and costs—took place in 2017. The following changes to existing components occurred during the restructuring in 2014: Component 1. A new Subcomponent 1.8 was added—Strengthening capacities of the College of Public Accountants of Nicaragua (the national association of accountants and auditors, CCPN)—as activities in the previous subcomponent to strengthen the internal audit system were completed with the development of internal auditing manuals. This new sub-component was designed to assist the CCPN in designing and implementing a self-sustainable training system for professional accountants to obtain the skills needed to support implementation of international accounting and auditing standards. Component 2. Under this Component, additional support was provided to Subcomponents 2.1, 2.2, and 2.3. The estimated cost of activities under Component 2 was US$10.70 million. Subcomponent 2.1 focused on the configuration of the SIGAF, preparing it to be deployed in 2017. Subcomponent 2.2 focused on strengthening communication security, both concerning information managed internally within the MHCP and that which is published externally through MHCP channels. This activity also supported improving incident monitoring, analysis of vulnerabilities to better prevent security breaches, effective follow-up, and providing prompt solutions. To guarantee the continuity of financial operations in case of a natural disaster Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) or similar events, Subcomponent 2.2 included the construction of a contingency site for the SIGAF Datacenter. Component 3. Under this Component, additional support was provided to support the new SIGAF and promote its use throughout the public administration. TA was provided not only on the application of the system but also on the integrated administrative processes and procedures, support for pilot tests, and the operational acceptance stage. Assistance under this Subcomponent also included support for strengthening subnational PFM capacity through assessments and analytical activities in view of a future expansion of the PFM reform. The estimated amount for these activities under Component 3 was US$2.20 million. Component 4. The implementation of the additional financing required continued support to the MHCP to oversee and administer the Project. The estimated amount for these activities to be executed under Component 4 was US$1.20 million. The following component was added during the restructuring in 2014. Component 5: Enhancing Statistics Management Capacity (Originally US$0.0M, revised to US$8.5M) Recognizing the role that reliable data plays in a results-oriented PFM system, the Government of Nicaragua (GoN) requested support to modernize the capacity of the National Institute of Development Information of Nicaragua (INIDE). The impending 2017 National Census of Population and Housing motivated the GoN to strengthen the national statistics capacity beyond the census. As such, this Component was designed to strengthen INIDE through the implementation of activities critical for the pre- census phase as well as other recurrent statistical operations. Specifically, assistance included support for the design and implementation of a digital Multi-Purpose Cartography. This Component also sought to develop INIDE’s capacity to edit and update the cartography periodically. Support also included the acquisition of the technological infrastructure to carry out the census and other periodic surveys through electronic methods as well as TA to build the capacity to utilize this technology for data collection. Mobile Capture Devices (MCD) were acquired for the fieldwork as well as software and applications necessary for the collection, monitoring, storage and processing of census data. The estimated cost for these activities was US$8.9 million. The following changes were made to the results framework PDO Indicators 3 and 5 were dropped. PDO Indicator 2 was modified (from “Public access to key fiscal information,” to, “Timely disclosure of key fiscal/budget documents compared to PEFA requirements) and PDO Indicator 4 included an additional clause (“Average processing time for the preparation and issuance of financial statements of the Central Government after the closing of the fiscal year”). An additional PDO Indicator was added (“Capacity to carry out statistical data collection is strengthened”). Additionally, eight intermediate indicators were revised and eleven dropped. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) Project cost. At appraisal, the total cost of the project was estimated at US$22.4 million. Additional Financing of US$25.0 million was added in July 2014. The actual project cost at closing was US$50.6 million (ICR, p 2), of which Bank-administrated funds comprised US$35M from IDA and US$3.4M from the European Commission. Financing. The project was initially financed by a US$10.0 million IDA Credit and a recipient-executed trust fund of €3.0 million from the European Union (TF071843), although this was €2 million at the time of appraisal (as per the PAD). The project also included parallel co-financing of US$10 million from the Inter- American Development Bank (IADB) and US$2.7M in Government co-financing. Additional Financing from the World Bank in the form of a US$25.0 million IDA Credit was approved on July 8, 2014. Borrower contribution. The task team reported that, according to Government reports, the borrower contribution was US$2.7 million. Dates. The project was approved on 10 December 2010 and became effective on 6 May 2011. The original Closing Date was 31 December 2015. The project’s closing date was pushed back several times, first to 31 December 2017 (as part of the Additional Financing) then to 31 December 2018, then to 30 November 2019, then to 30 June 2020, then to 31 August 2020. In total, the project was extended for 24 months to accommodate the Additional Financing, and then an additional 32 months to allow for finalization of all activities and full disbursement of funds (much of which was to account for the high degree of political unrest that has gripped the country since 2018, and some of which was due to COVID-19 related delays). On 31 August 2020 the project was closed. 3. Relevance of Objectives Rationale Both the original and revised PDOs were relevant to the country’s development priorities as described in the CPF and Nicaragua’s need for modernized PFM systems and processes. However, they were unrealistically ambitious and complex given country capacity requiring the design and implementation of new IFMIS with 8 systems within a 5-year period. Both the original and revised PDOs were relevant to the Government’s priorities, building on PFM reforms supported jointly by the Bank and the Inter-American Development Bank from 1996-2010 including a modern legal framework (Law 550), fiscal policy capacity, and a medium-term budget framework. The original PDO supported the implementation of the Government’s comprehensive PFM modernization plan 2008-2012, in which upgrading and a full roll-out of the SIGFA were prominent. The revised PDO, which was amended to reflect the Additional Finance support to INIDE and strengthen the capacity of public sector accountants and auditors in partnership with CCPN, was aligned with the Government’s priorities as outlined in the National Human Development Plan (2018-2021): the prudent management of public finances, central government, decentralized entities, and state-owned enterprises, as well as improved effectiveness, efficiency, and quality of public spending. The addition of the objective around strengthening the Recipient’s statistics management capacity was a direct response to the Government’s request for support in preparation for the National Census of Population and Housing. Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) Both the original and revised PDOs were relevant to the Bank’s strategy for Nicaragua. The original PDO was aligned with the World Bank Group’s Country Partnership Strategy for the period FY08-12 (Report No. 39637-Ni), whose fourth pillar aimed at strengthening governance and accountability through modernization and professionalization of state institutions and the promotion of citizen engagement. The revised PDO was aligned with the World Bank’s Country Partnership Framework for FY18-22, whose Objective 7 (Improved Data Availability and Public Sector Management Capacity), focused on improving the availability of public sector data and building the capacity of the Government to facilitate the effective allocation of public resources. Status of PDO Indicators PDO Indicator Baseline Actual End Target Status Fiscal and budgetary projections analyzed under a multi-annual perspective, are linked B B+* B+* Achieved to budget and expenditure policies (PEFA, ID12). Timely disclosure of key fiscal/budget Partially documents compared B B* B* Achieved to PEFA requirements (PEFA, ID10) Average processing time for the preparation and Not issuance of financial 6.0 4.0 3.0 Achieved statements of the Central Government after the fiscal year. Updated cartography Capacity to carry out with national Country's Digital statistical data coverage cartography cartography Achieved collection is used in at is outdated finalized strengthened. least one data collection operation. * Since a second PEFA never took place, the ICR assessed progress using proxy indicators Rating Relevance TBL Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To enhance the efficiency of the Recipient’s public expenditure management Rationale Enhanced efficiency of public expenditure management was to be achieved by modernizing the country’s IFMIS and other PFM systems and processes, which would allow for single data entry, a reduction in transactions, and increased accuracy and quality of financial data. Two of Objective 1’s three PDO indicators relied upon PEFA indicators, where baselines and end targets were only available 6 years after project approval and second ratings (to allow for an assessment of change over time) never took place, so actual ratings at project completion are unknown. The ICR analyzed proxy indicators to assess the project’s achievement of objectives: an intermediate indicator on the number of central government agencies using the multi-year and results-based budgeting methodologies, and the fact that a treasury single account was fully implemented, central government debt is now recorded automatically, and SIGAF functionality across central government and decentralized entities). PDO Indicator 3 was not achieved. Enhanced efficiency was to be achieved by strengthening institutional capacities of the Ministry of Finance, including development and implementation of a new IFMIS (SIGAF) and strengthening of its budgeting, accounting, treasury, public credit, public procurement, civil service management, public investment, and internal audit systems as provided for under Law 550. Outputs  While the time lag to publish monthly budget execution information online decreased, SIGAF is still unable to regularly produce monthly and ad hoc reports without substantive manual corrections according to the assessment conducted for the ICR. In response to IEG’s question, the Director General for Technology, Ministry of Finance, indicated and provided proof to the task team that, as of 1 September 2021, SIGAF is able to produce quarterly and annual reports on budget execution to the National Assembly without manual correction.  All Central Government's payments to vendors, public entities and personnel of Central Government payroll are processed electronically according to Treasury Single Account (TSA) procedures  Six of the planned seven SIGAF systems (Accounting, Budget, Procurement, Public Credit, Public Investment, and Treasury are operational at an over 90 percent rate of satisfaction, although several defects were reported in the ICR and several system modifications are pending.  100% of staff from MHCP units satisfactorily completed training to operate and maintain SIGAF. Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795)  A Contingency Data Center up to international standards was implemented. This facilitates SIGAF’s business continuity in the event of a failure of the main data center.  SIGAF was successfully implemented in 23 central government and 18 decentralized functional entities at the time of the ICR’s writing (18 March 2021). At time of ICRR, the task team verified with the Ministry of Finance that the number of decentralized functional entities using SIGAF has increased to 46, and that seven decentralized entities—i.e., councils, special funds, but also the Mayoralty of Managua—are now using SIGAF as well. Outcomes Determining whether the first PDO indicator was achieved—i.e., whether fiscal and budgetary projections are linked to budget and expenditure policies—is not feasible, due to the lack of comparative scores from the PEFA assessment. The ICR used an intermediate indicator--the number of Central Government agencies using the multi-year and results-based budgeting methodologies--which increased from 45 to 60 (100 percent of all Central Government agencies). This is a reasonable output-level proxy to gauge the targeted outcome. Furthermore, the ICR notes (pp 9-10) that the project supported the operationalization of a business intelligence portal to facilitate budget analysis and produce statistical data for risk management as well as to support budget preparation and execution. Likewise, it is not possible to validate whether the second PDO indicator was achieved i.e., timely disclosure of key fiscal/budget documents compared to PEFA recommendations. That being said, the project did succeed in upgrading SIGAF onto a web-based platform, the consolidation of budget execution reports and financial statements produced in a manner consistent with IPSAS (as well as the IMF Government Finance Statistics Manual, GFSM 2012, and the national legal framework), and full implementation of the TSA, the interfacing of GoN’s Debt Management and Financial Analysis System (SIGADE) with SIGAF to allow for automatic recording of foreign and domestic public debt. To ensure uniformity in the online publication of all Government procurement processes (i.e., not only those entities using SIGAF but for every government entity), SIGAF uses the preexisting Public Procurement Information System and its website (www.nicaraguacompra.gob.ni) to publish its procurement processes. The project was unable to improve processing time for the preparation and issuance of financial statements of the Central Government. FY18 financial statements took 18 months to be published, while FY19 financial statements took 9 months. The ICR considers these delays to be growing pains in using SIGAF to produce the statements; work to address the SIGAF accounting inconsistencies behind these delays have been fixed, according to the task team, meaning that FY20 statements should be published within the recommended timeframe following the close of financial year. Summary: significant achievements were made to improving PFM efficiency, particularly in the launching of the country’s new IFMIS and the upgrading of public sector accounting through mainstreaming international accounting standards, revising budget classifications, developing a unified chart of accounts, reviewing reporting standards, and broadening the TSA. That being said, there is insufficient evidence to conclude that the project achieved most of the targeted outcomes Rating Modest Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) OBJECTIVE 2 Objective To enhance the transparency of the Recipient’s public expenditure management Rationale Enhanced transparency was to be achieved by modernizing the country’s IFMIS and other PFM systems and processes (particularly accounting and budgeting). As was the case for Objective 1, one of Objective 2’s two PDO indicators relied upon PEFA indicators, where baselines and end targets were only available 6 years after project approval and second ratings (to allow for an assessment of change over time) never took place, so actual ratings at project start and completion are unknown. The second PDO indicator (Average processing time for the preparation and issuance of financial statements of the Central Government after the fiscal year) was not achieved. Outputs  Several outputs were achieved. First, Central Government financial statements are presented according to an accounting plan consistent with the 2012 IMF Manual for Government Financial Statistics Standards and applicable legal framework through SIGAF. Second, staff from all MHCP units satisfactorily completed the training to operate and maintain SIGAF. Third, there is improved transparency from SIGAF implementation as outlined above.  The project also supported partial achievement in two areas. First, 90 percent of staff were trained using the online training platform designed for the first training course on International Public Sector Accounting Standards (IPSAS); however, IPSAS are not currently being used and no adjustments have been introduced to adopt them in the near future. Second, six SIGAF systems (The payroll system was dropped in 2020) were operating at an over 90 percent rate of satisfaction. The remaining issues are expected to be resolved by a work plan being developed by the vendor and GoN to correct remaining defects, performance improvement and implementation of system modifications. Outcomes It was impossible to measure improved transparency in GoN’s public expenditure management based on one of two relevant PDO indicators, given the absence of PEFA assessments. As a proxy, the ICR uses the transparency portion of the Open Budget Index to gauge progress. The transparency score for Nicaragua did increase, from 37 in 2010 to 41 in 2019, although this is down from a high of 46 in 2015. The ICR noted progress in publishing key fiscal documents online, including budget documents on the MoF’s website (www.hacienda.gob.ni), procurement documents (www.nicaraguacompra.gob.ni), and a forthcoming website for public consultation. Bringing the average processing time for the preparation and issuance of financial statements of the Central Government after the fiscal year from 6 to 3 months (the second PDO indicator) was not achieved. Instead, average time increased to 8 or 9 months (the ICR mentions both amounts of time). According to the ICR, these delays were caused by “accounting inconsistencies generated by SIGAF that had to be painstakingly Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) and manually corrected by MHCP,” and these are being addressed. The ICR indicates that there is an expectation that end-of-FY20 statements will be published within the three months target. Summary: project measures to improve the transparency of the Recipient’s public expenditure management were not successful overall. While some tangible achievements were recorded, one PDO indicator was not achieved and the second could not be verified due to lack of indicator data. Rating Negligible OBJECTIVE 3 Objective To strengthen the Recipient’s statistics management capacity Rationale Strengthened statistics management capacity was to be achieved by modernizing and strengthening INIDE through the design and implementation of a digital, multi-purpose cartography and electronic data collection. This PDO indicator (Capacity to carry out statistical data collection is strengthened) was achieved. Outputs  All 153 of Nicaragua’s municipalities had an updated cartographic framework for statistical operations. This means that 100 percent of municipalities have the operational technology infrastructure to carry out periodic surveys as well as the software and applications necessary to collect, monitor, store, and process census data.  A system of electronic data collection and processing was successfully developed and piloted. Outcomes The project’s strengthening of GoN’s statistics management capacity was meant to ensure that INDIDE could carry out the National Census of Population and Housing in 2017, as well as future surveys, using electronic methods. The PDO indicator was achieved: “The digital tools developed under this component (notably digital multi-purpose cartography) were successfully tested during pilot population and housing censuses in the municipality of Jalapa, Nueva Segovia in 2017 and the municipality of Somontillo, Chinandega in 2018. The tools were also used to carry out the national survey for children and adolescents covering 20,000 households (ICR, pp 17-18). The ICR notes that due to civil unrest that began in 2018 (a year after the Census was supposed to take place), the Government was forced to postpone the Census to 2022. As such, the Government may very well have to update the cartography again by then. Nevertheless, the digital tools and capacity developed by the project were successfully used to carry out the National Survey for Children and Adolescents. Summary: The measures introduced to modernize and strengthen GoN’s statistical capacity were successfully implemented. Both output targets and the PDO indicator were met. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale While the data indicate improvements in key outputs and outcomes, the project fell short of achieving key indicators. This was exacerbated by the inability of measuring several outcome indicators, due to the project’s reliance upon PEFA indicators and the absence of a repeat PEFA to establish change over time. Regardless, several investments and reforms introduced (and defined as outputs in the evaluation) supported the achievement of progress in the PDOs and generated concrete improvements in PFM, albeit in several instances falling short of project targets and expectations. Overall Efficacy Rating Primary Reason Modest Insufficient evidence 5. Efficiency There was no formal economic cost-benefit analysis prepared either at the time of appraisal or in the context of the 2014 Additional Financing. The PAD simply referred to likely positive fiscal impacts resulting in economic gains but without any attempt to quantify or link these improvements to specific project activities or targets. Annex 4 of the ICR (pp 44-47) presents a summary of the principal achievements of this project and two interrelated World Bank projects for Nicaragua, compared to the situation at appraisal; however, it does not provide sufficient information to allow for an assessment of cost-benefit analysis. Assessments of project benefits were largely qualitative. The majority of the project’s interventions involved activities geared at increasing the efficiency of core government processes, including the use of budget resources (including expenditure, procurement, and debt management). Project interventions may have led to improvements in public sector efficiency in that public servants spend less time manually inputting data and automated functions reduce human error and room for corrupt practices. While one could make the argument that the extension was 32 months—since the original 24-month extension was to accommodate Additional Financing—in actually, the project incurred a cumulative extension of 56 months: the project’s duration was almost doubled. This extension is a sign of inefficiency on the part of the Bank, even if PFM reforms are notoriously time-consuming. Efficiency Rating Modest Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives was rated Substantial, as original and revised objectives were aligned with both government and Bank priorities for the country, although the timeline was overly ambitious for the country context. The overall efficacy was rated Modest (with Negligible for one PDO, modest for one, and Substantial for one). The efficiency was rated Modest. Based on those ratings, the project’s outcome is rated Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The risk to Development Outcome is low. Risks were reviewed throughout the life of the extended project. The ICR rightly notes the lack of stakeholder (MHCP) technical capacity as one of the highest risks to the achievement of project objectives. Nevertheless, achievements are increasingly sustainable as system kinks are worked out and the usage of a fully operational SIGAF is institutionalized. 8. Assessment of Bank Performance a. Quality-at-Entry While the project was designed to support the Government’s PFM reform program, and while it did provide substantial technical assistance that allowed for significant achievements in the area of PFM, its design was unrealistically ambitious and set the Government up to fail given the timelines presented. Additional Financing to accommodate newly important activities allowed for an additional 24 months of implementation time. The project also suffered from deficiencies in the results framework, with PEFA indcators non-existent for both baselines and target dates. There was also failure to explain Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) comprehensively to the Government the benefits and trade-offs of reliance on commercial off-the-shelf (COTS) software. The ICR notes, “the Government’s limited understanding of the functionality of the [COTS IFMIS] system that they were acquiring. A 2011 IT report commissioned by the project identified potential problems with the selection of a COTS solution: there was no functional analysis of the COTS supplied by the market; the decision to use a COTS was not properly analyzed and justified; technical bidding documents did not accurately describe all the Government’s needs; there was no proper assessment of the vendor’s capabilities; and the vendor committed to more than was feasible, under the proposed costs. The project financed study trips to visit the implementation of in-house developments of customized solutions. There were no study trips to look at the experience with COTS” (p 18). Quality-at-Entry Rating Unsatisfactory b. Quality of supervision While supervision remained consistent throughout project implementation and coordination with the IADB was notable, the Bank team did not adequately identify implementation challenges early on, preventing them from designing proactive interventions to keep the project on track, particularly around the SIGAF. Given early delays and challenges, the project’s ratings should have been revised downwards significantly earlier than July 2016, two years after the Additional Financing and restructuring Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Results indicators were drawn from the PEFA. This is problematic for several reasons. First, PEFA indicators used were composite measures, with the PEFA scores used broader than the specific outcomes that project activities were intended to impact. Also, many of the PEFA indicators used did not have baseline data or targets. The PAD noted that baseline and end-target values would be set once the forthcoming PEFA (planned for 2010) took place. That assessment was only completed in August 2015 (using 2013) data, and then only made public in 2019. This means that several PDO indicators had neither baseline nor annual targets until August 2015, which was just four months before the original closing date. Furthermore, as no second PEFA was undertaken during the project, there are no formal values for indicators drawn from the PEFA (the ICR Results Framework lists PEFA baseline scores from a “2013 PEFA” and project-end values from a “2015 PEFA,” Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) whereas, in actuality, those later scores were developed by the ICR team to be able to measure change over time). b. M&E Implementation The Additional Financing and restructuring, following the 2014 Mid-Term Review, led to improved PDO and intermediate indicators. As noted in the ICR (p 18), the revised M&E framework was significantly streamlined. However, during the project Mid-Term Review, in 2014, PEFA indicators were retained, even though baseline data did not exist. c. M&E Utilization M&E data is meant to help identify areas in need of intervention. However, given the complications with the original PDO and intermediate indicators noted above, the need for corrective action may not have been as readily noticeable as it would have been otherwise. It should be noted that the ICR mentions one impact of this project as improving the culture of M&E within the Government. If true, this is a significant outcome that should be documented and replicated elsewhere, even if it cannot be captured by the project’s results framework. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project’s environmental and social assessment categories were rated a “C” technical assistance project providing soft reforms in PFM. No environmental safeguard policies were triggered. b. Fiduciary Compliance Financial management. The ICR (pp. 20-21) notes that the project had satisfactory financial management performance. It also mentions temporary issues related to errors and inconsistencies with the project’s financial system SIGFAPRO: the government’s system for managing development cooperation expenditure. However, those issues did not harm the timeliness or reliability of FM reports production or otherwise FM management and monitoring. Quarterly interim unaudited financial reports and annual audited financial statements were timely and submitted to the World Bank in acceptable form and substance. Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) Procurement was conducted in accordance with World Bank rules and procedures and was overall timely (ICR, p. 21). c. Unintended impacts (Positive or Negative) None were noted. d. Other None were noted. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory IEG notes the unrealistically ambitious design, deficiencies in the results framework, and Moderately Bank Performance Unsatisfactory failure to fully explain software Unsatisfactory options to the client. Supervision during implementation was better than at entry. Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The design called for procurement of a customized off-the-shelf IFMIS system, its deployment, training of staff, and implementation of 8 systems within a 5-year-period. This was unrealistically ambitious given the country capacity and not in line with the Bank's PFM modernization experience elsewhere; in fact, the IFMIS diagnostic report from 2010 by an independent international expert (referenced in the ICR) concluded that “the SIGAF project is the most ambitious IT project ever started in Nicaragua and, as such, is riskiest due to its complexity, size, and management” (Cornelius Hopmann 2010. “Alternativas para la Estrategia de Desarrollo y las Modalidades de Contratación – SIGAF” p 3). IEG supports lessons from the ICR on the need for realistic timelines, reform sequencing, and risk assessments. IEG adapts lessons from the ICR as follows: Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review NI PFM Modernization TAL (P111795) The results framework and supervision of PFM projects should focus on the systems and reforms supported by the project. At the design stage, the two PEFA linked PDO indicators were overly aggregated and did not have baselines or targets. Use of PEFA indicators should be taken only when baseline data exists and when there are adequate assurances that a second assessment will be undertaken at an appropriate time to assess impact. The Bank should focus holistically on PFM reforms needed to achieve intended results, not just systems. ISRs reveal that the Bank team focused primarily on the execution of the main contract (SIGAF) and did not discuss in much detail progress towards achieving other PFM reforms supported by the project. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR developed theories of change for the original and restructured projects and makes a good attempt to gauge the achievement of project objectives using proxy indicators, given the challenges mentioned above with those indicators using PEFA scores. Some minor errors over project iterations were noted in the ICR. IEG complements the task team for their timely and comprehensive responses to its questions and concerns, including the provision of critical additional documents requested for the ICRR. a. Quality of ICR Rating Substantial Page 18 of 18