IEG Report Number: ICRR14863 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 11/09/2015 Country: Ukraine Project ID: P090389 Appraisal Actual Project Name: Public Finance Project Costs (US$M): 65 3.8 Modernization Project L/C Number: Loan/Credit (US$M): 50 3.8 Sector Board: Public Sector Cofinancing (US$M): Governance Cofinanciers: Board Approval Date : 01/08/2008 Closing Date: 06/30/2013 12/31/2014 Sector(s): Sub-national government administration (65%); Central government administration (35%) Theme(s): Public expenditure; financial management and procurement (100%) Prepared by: Reviewed by: ICR Review Group: Coordinator: Katharina Ferl Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Project Appraisal Document (PAD, p. ii) and the Loan Agreement of March 25, 2008 (p. 5), the objective of the project was “to strengthen public financial management in terms of operational efficiency and transparency." During the project restructuring in April 2012, some key outcome indicators and targets were revised. However, since the indicator changes were refinements and the targets were revised upward to reflect the project extension, a split rating is not warranted. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval: 04/24/2012 c. Components: The project included three components: Component 1: Strengthening institutional capacity and operational effectiveness (appraisal estimate US $2.58 million, actual US$2.65 million, 103% of appraisal estimate ): This component was to provide technical assistance to the Ministry of Finance (MoF) in different areas such as selected institutional capacity building activities for Public Financial Management (PFM) reforms, functional review of the MoF, business process re-engineering, and design of ICT solutions. Activities included the improvement of budget planning and mid-term budgeting, methodological support for performance based budgeting capability, development of a monitoring and evaluation (M&E) system for performance based budgeting, support to local budget reforms and for the implementation of accounting reform, creation of a single registry of spending units, development of a unified chart of accounts, improvement of the debt management system, participation in improvement of international financial assistance management, improvement of coordination for internal financial control, and support for change management, including workshops, study tours, and training. Component 2: Development of an integrated public financial management system (appraisal estimate US $44.2 million, actual US$0.18 million, 0.4% of appraisal estimate ): This component was to finance the development of an integrated public financial management system (PFMS) software and installation of central servers, as well as installation of PFMS field hardware and network equipment. Change management training and related technical assistance activities to ensure sustainability of implemented solutions were included as well. Component 3: Project management (appraisal estimate US $1.22 million, actual US$0.82 million, 67% of appraisal estimate): This component was to ensure timely and efficient allocation of resources to carry out project procurement, financial management and audits, M&E, and interaction with all local and international entities involved in project execution. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost: The project was estimated to cost US$65 million. Actual cost was US$3.8 million, 5.9% of the project appraisal estimate. Financing: The project was planned to be financed by a US$50 million IBRD loan, of which US$3.8 million was disbursed (7.6% of appraisal estimate). The remaining amount was cancelled as requested by the borrower. Borrower Contribut ion: The borrower was to contribute US$15 million. The actual contribution was US$0.023 million, 0.2% of the appraisal estimate. Dates: The project was restructured three times: 1. On April 24, 2012 the project was restructured to: i) extend the closing date from June 30, 2013 to June 30, 2015 to support the priorities and reform needs of the MoF; ii) adapt activities under component 2 to enhance PFMS’ functional and technical requirements; iii) include new PFMS modules to support additional needs that were identified during the previous two years; iv) ) revise the procurement plan; v) strengthen the qualification requirements to ensure the participation of only experienced and well qualified suppliers during rebidding of the ICB package; and vi) modify key indicators and targets. 2. On April 1, 2013 the project was restructured to: i) revise the procurement plan; ii) reallocate US$0.85 million from expenditure category 1 to category 2; and iii) add new covenants to the Loan Agreement. 3. On October 16, 2014 the project was restructured to: i) accommodate the government's request of a partial loan cancellation of US$46.06 million. The project first experienced delays in the implementation of the PFMS due to the cancellation of thee procurement processes (see Section 11b). In addition, by mid-2014 t he political and economic situation in the country had created severe fiscal pressure, causing the government to review its priorities and decide not to pursue the option of a customized PFMS. Instead, it was decided to upgrade the hardware and add to the functionality of the already existing Treasury system; and ii) to change the project closing date from June 30, 2015 to December 31, 2014. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial: Ukraine has implemented an agenda of institutional modernization since the beginning of the decade. However, gaps remained, such as the lack of an automated commitment control system, inconsistent integration of pension financial reports into consolidated reports, and a weak governance and oversight framework for state-owned enterprises. The development and implementation of an integrated PFMS was seen as key to overcoming these remaining challenges. At the time of project preparation, the objectives of the project were in line with the government’s commitment to increase transparency, accountability, and efficiency by introducing performance-based budgeting, pubic financial controls and accounting reforms. The objectives were also in line with the Bank’s Country Assistance Strategy (2004-2007), as well as the 2004 Country Economic Memorandum and the 2005 Country Assistance Strategy progress report, which all identified the need to strengthen fiscal discipline, improve public financial management, and integrate fiscal management. The objectives remain in line with the current Bank Country Partnership Strategy (2012-2016), which contains a first pillar focusing on the sustainability and efficiency of public finances and a more transparent and accountable use of public resources. b. Relevance of Design: Modest: The project included activities that were relevant for achieving the project's objective to improve the operational efficiency of public financial management, including providing technical assistance to build capacity within the MoF, developing a PFMS, and strengthening several public financial management functions in areas such as budgeting, accounting, and M&E. However, the project's design did not include activities specifically linked to achievement of the transparency objective. The project's design took relevant risks such as loss of the government commitment to public financial management reform, delays in the establishment of a country-wide network by the MoF, and institutional resistance into account, and mitigation measures were planned. These measures included monitoring government commitment and continuing to maintain dialogue with the MoF on key PFM issues, including network establishment as a covenant implementation, and implementing a change management component to inform all stakeholders about the reform. However, these mitigation efforts were not sufficient. Also, the project's design did not take into account the possibility that the Treasury might hold onto and develop its e-Treasury system rather than support the development and implementation of the new PFMS. 4. Achievement of Objectives (Efficacy): To strengthen public financial management in terms of operational efficiency : Modest Outputs:  The PFMS was not installed and is not operational due to three unsuccessful procurement attempts, and therefore the target was not achieved. The first procurement process was cancelled by the Bank due to an anonymous complaint about the technical compliance of bids. The second procurement process (a single stage bidding process to recover time lost) failed when all of the technical bids were rejected as unresponsive. The third procurement process was cancelled by the government due to changing priorities. Due to fiscal pressure, the government decided to upgrade the hardware and add to the functionality of the existing Treasury system instead of pursuing the implementation of a customized PFMS system.  A methodology for the program-target method of local budgeting was prepared, but its implementation was postponed from 2015 to 2017 due to the lack of the PFMS. Therefore, this target was not achieved.  2,456 staff of the Ministry of Finance, the State Treasury, and the State Financial Inspection Unit were trained on the new methodology and features of the PFMS, achieving the target.  Since the PFMS was not installed, staff was not trained in the operational use of it, not achieving the target. Also, budgetary program passports could not be managed by the new PFMS, not achieving the target of budget preparation of all 692 local budgets being based on the program-target method supported by PFMS.  Timely and comprehensive fiscal management reports were not produced through the PFMS data warehouse, since the PFMS system it was not installed. Therefore, this target was not achieved.  A risk-based planning methodology was developed and adapted by the State Financial Inspection’s Methodology Council in 2012.  19 new public sector accounting standards were developed and adopted based on the International Public Sector Accounting Standards (IPSAS) accrual standards. On January 2, 2015, public sector accounting regulations became effective. However, the ICR (p. 15) states that full accrual accounting will not be completely adopted in 2015 due to the ongoing challenges with the accounting system.  The budget department developed new budget preparation instructions including a mid-term budget framework, linking the budget to the economic and social development program and introducing budget-performance indicators. Outcomes:  Since no baseline study was conducted at the start of the project, it is not possible to determine whether a reduction of 20% in time required to get necessary information from managerial decisions through the use of an integrated management and information platform was achieved. In 2012 and 2013, the Project Implementation Unit (PIU) conducted surveys on four business processes to determine whether there has been a decrease in time required to take managerial decisions. 1. The amount required to provide services to spending units and make other payments by the departments of the State Treasury Service of Ukraine marginally decreased from 3.34 hours in 2012 to 3.25 hours in 2013. 2. The amount of time required to prepare a report on the execution of the state budget was 1.07 hours in 2012 and remained the same in 2013. 3. The amount of time required to prepare consolidated reports on the execution of the state budget was 4.33 hours in 2012 and remained the same in 2013. 4. The amount of time required to prepare reports on the execution of the state budget for specific areas was 8.16 hours in 2012 and remained the same in 2013. 5. The amount of time spent on the formulation of revenue forecasts in preparing the budget by the Ministry of Finance was 190 days in 2012 and remained the same in 2013. 6. The amount of time required for the planning of control and auditing measures by the state financial inspectorate of Ukraine was 2,937 days in 2012 and increased to 2,991 days in 2013. These data suggest that there has not been a reduction in time between 2012 and 2013, largely because the PFMS was not implemented and is not operational. Therefore, this target was not achieved.  The ICR (p. 13) provides alternative measures of results, not measured by the project, of the Public Expenditure and Financial Accountability (PEFA) assessments of 2005 and 2010, including 20 relevant PEFA dimensions. The results show improvements in the linkages between investment budgets and forward expenditure estimates, the extent to which cash flows are being forecasted and monitored, the scope of reports in terms of coverage and compatibility with budget estimates, the quality of information in budget reports, and the completeness of financial statements between 2005 and 2010. Even though these indicators are not related to operational efficiency, measured as reductions in time taken for business processes, they indicate improvements in integration and operational efficiency regarding delivering the desired outcomes of the project. However, along most dimensions the scores did not improve. Also, given the timing of the PEFA assessments, it is questionable to what extent any improvements could be attributed to the project.  The risk-based planning methodology supported the State Financial Inspection Directorate to increase efficiency, and the number of violations detected increased as a percentage of the number of inspections. In 2012, the State Financial Inspection Directorate conducted approximately 8,000 inspections and detected 7,500 financial violations, resulting in a detection rate of 94%. In 2014, the Directorate only conducted 3,500 “high risk” inspections, detecting 3,400 financial violations, resulting in a detection rate of 98%. Being able to decrease the number of inspections through the risk-based planning methodology by 56% had a positive impact on lowering pressure on staff and resources.  The State Financial Inspections Department applied the risk-based planning methodology to improve efficiency of their inspections. To strengthen public financial management in terms of transparency : Negligible Outputs:  The Ministry of Finance published information on state budget execution on its website on a monthly basis. Information included indices of general and special funds on revenues, expenditures, loans and credits; draft laws published after their presentation to the Parliament; and a book on budget execution report for FY 2013. Also, the State Treasury published on its website state and region budget execution reports by functional, economic, and program classifications on a monthly and quarterly basis. However, the ICR (p. 29) states that the quality of all this information only slightly improved across the time period of the project because the PFMS was not implemented.  19 new public sector accounting standards were developed and adopted based on the International Public Sector Accounting Standards (IPSAS) accrual standards. On January 2, 2015, public sector accounting regulations became effective. However, the ICR (p. 15) states that full accrual accounting will not be completely adopted in 2015 due to the ongoing challenges with the accounting system.  Timely and comprehensive fiscal management reports were not produced through the PFMS data warehouse, as the PFMS system it was not installed. This target was therefore not achieved.  The accounting system was made compliant with IPSAS and partially compliant with Government Finance Statistics (GFS). Outcomes:  The Open Budget Index (OBI), which measures the amount, level of detail, and timeliness of budget information published, was made publicly available as an independent assessment of Public Financial Management transparency. Ukraine's score was 55 (out of 100) in 2008, 62 in 2010, indicating that budget transparency did not improve consistently over that period of time.  The 2005 and 2010 PEFA assessments for indicator “public access to key fiscal information” remained the same for both years, suggesting that budget transparency has not improved. 5. Efficiency: Modest: Neither the PAD nor the ICR conducted a traditional economic analysis. The PAD (p. 12) states that the major benefits of the project are difficult to measure and assign a monetary value. The PAD cites several benefits that would result from successful project implementation, including full implementation of performance/program based budgeting, raising efficiency of budget programs audits and using audit results to establish priorities for public expenditure, and improving and increasing transparency, including the mechanism to grant transfers to local budgets to level their financial capacity. The ICR (p. 17) analyzes time and cost overruns and operating costs to determine the efficiency of the project. This review finds that the project disbursed only 5.6% of planned financing over an implementation period that took 2.5 years longer than expected. This was due to prolonged procurement delays that eventually led to the cancellation of the component output that accounted for 95% of the original cost and the cancellation of project funds associated with the PFMS system. On balance, efficiency was Modest. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of objectives is rated Substantial, given the government’s agenda to reform public financial management. Relevance of design is rated Modest, as it is not clear that the project's planned activities addressed the transparency objective. Achievement of the objective to strengthen public financial management in terms of operational efficiency was Modest. Achievement of the objective to strengthen public financial management in terms of transparency was Negligible. Efficiency is rated Modest. These ratings are indicative of major shortcomings in the project's preparation and implementation, and therefore an Outcome rating of Unsatisfactory. a. Outcome Rating: Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Negligible to Low: The Risk to Development Outcome rating is Negligible to Low since the project generated only a few outcomes. The project established a a committed Project Implementation Unit that has the necessary capacity for future public financial management reforms. The ICR (p. 21) states that achievements were made in regards to PFM business processes that are well integrated into regulatory frameworks and internal operating and budget procedures. However, continued reform processes will depend on building capacity within the Ministry of Finance, which currently depends on external technical assistance. Also, discussing next steps of the reform process and bringing the Treasury and the Ministry of Finance on board and in synch will be critical to moving the agenda forward successfully. a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: The project built on the Bank’s Treasury System project that closed in December 2004 and was well aligned with the strategic priorities of the government. The project design was informed by an organizational review of the Ministry of Finance, a Public Expenditure and Financial Accountability Assessment, and a White Paper on Internal Audit and Control of 2005. The Bank identified relevant risks as substantial during project preparation, including lack of government commitment to public financial management reform, delays in the establishment of a country-wide network by the Ministry of Finance, and institutional resistance. Mitigation efforts were not sufficient, particularly that of declining government commitment. The public financial management reform shifted power from the State Treasury to the Ministry of Finance, resulting in resentment and institutional resistance to reform. The Bank’s Quality Assurance Group authored a report in 2009 that questioned the integration of the Treasury into the Ministry of Finance and speculated that this action would put additional burden on Ukraine’s volatile political situation, at a time when attention might have been better solely focused on improving the data system. This point was particularly relevant because resistance by civil servants and changes in government had been identified as potential risks to project implementation. Yet project preparation did not plan adequately for this possibility. The design of the Results Framework had significant shortcomings (see Section 10a). Quality-at-Entry Rating: Moderately Unsatisfactory b. Quality of supervision: The Bank team conducted 13 supervision missions and held 40 video conferences during the project period. A team member in the Kiev country office provided ongoing guidance to the project implementation unit and the Ministry of Finance. The Bank team took appropriate action in referring an anonymous complaint regarding the bidding process for two ICB software and hardware packages to INT. However, the Results Framework was not sufficiently used to monitor progress toward achievement of the project's objectives. During the project restructuring in 2012, the Results Framework was modified. However, the modification had a strong focus on the implementation of the PFMS system, which ultimately did not materialize. The Bank team did not address sufficiently the resistance by the Treasury to the reforms. Even though the Treasury used its own resources to modernize its own e-treasury system and displayed little interest in the successful implementation of the PFMS, the Bank did not address this issue. Instead, the Bank eventually supported neither the e-treasury system, whose development had originally been supported by the Bank in under previous project, nor implementation of the PFMS. The project's performance rating in Implementation Status Reports was downgraded only to Moderately Unsatisfactory for progress towards the development objectives in March 2013, despite only US$3.42 million having been disbursed. This might indicate that the Bank was unrealistic about achieving the project's development objectives, or about reporting on shortcomings, despite significant implementation challenges. Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government had requested the Bank’s assistance to implement a follow on project to the first Treasury project. Several political changes, however, impacted preparation and implementation. In 2005, the Treasury was integrated into the Ministry of Finance, which asked the Bank for support in its reorganization. The Treasury was actively engaged in the design process of the project but that it is not clear if all stakeholders were committed to the institutional integration process of the Treasury and the Ministry of Finance and the decision to implement an integrated PFMS instead of the e-Treasury system. The Treasury continued the development of the e-Treasury system while the Ministry of Finance tried to develop the PFMS. Both entities had a different vision for the e-Treasury and the PFMS. While the Treasury saw the e-Treasury system as an instrument that met most of its needs, the Ministry of Finance believed that it would be unable to deliver functional integration as the PFMS could do. The project also faced several procurement issues that made project implementation challenging (see Section 11b). Due to the severe fiscal pressure Ukraine faced in 2014, a new two-stage procurement process was cancelled, and the government requested the Bank to cancel the undisbursed portion of the loan. The Ministry of Finance did not ensure strong leadership of the project, appointing a deputy minister as project director only in 2013. Also, the project experienced delays at the start of the project period; the Ministry of Finance wanted to complete improvements in PFMS business processes before starting the procurement process for PFMS, which led to an eight-month delay. Government Performance Rating Unsatisfactory b. Implementing Agency Performance: The Project Implementation Unit (PIU) lacked strong leadership and management. The deputy minister, who was appointed as project director in 2013, did not have sufficient time to manage the project. There was therefore a lack of management continuity and understanding among different project component managers, which led to delays in project implementation. The project had eight different managers within a five-year period due to changes in the political environment. Periodic progress reports were prepared and submitted on a regular basis, but there were delays in tracking key performance indicators. Also, baseline data for key performance indicators was never collected, negatively impacting measurement of progress. There is little evidence that M&E results were used to inform decision making. Financial Management was rated Satisfactory throughout project implementation, but procurement faced several challenges (see Section 11b). Implementing Agency Performance Rating : Unsatisfactory Overall Borrower Performance Rating : Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The Results Framework in the PAD included three PDO and 16 intermediate outcome indicators. The objectives were clearly specified; however, the PDO indicators did not reflect those objectives or measure progress towards those objectives well. Public Expenditure and Financial Accountability assessments done in 2005, 2011, and 2015 could have supplied the project with baseline, mid-term, and end of project data by an objective third party. Similarly, a more effective measure of transparency would have been indicators of the Open Budget Index that assesses whether the government makes key budget documents available to the public. b. M&E Implementation: The Results Framework was modified during the April 2012 restructuring. Several PDO and intermediate outcome indicators were revised, and some intermediate indicators were dropped. However, a baseline study for the indicator on operational efficiency did not take place until 2012, close to the original closing date, and a repeat survey was done just a year later, substantially earlier than the revised closing date. It was therefore not possible to measure effectively the project's impact on operational efficiency. c. M&E Utilization: The Results Framework was of limited use due to its design shortcomings. There is little evidence that it was used to inform decision making. M&E Quality Rating: Modest 11. Other Issues a. Safeguards: The project did not trigger any safeguards policy. b. Fiduciary Compliance: Financial Management According to the ICR, Financial Management was consistently rated “Satisfactory” throughout project implementation, and there were no major operational issues. Unqualified reports were issued by auditors, and no internal control or accountability issues were raised. Procurement According to the ICR, the project faced significant procurement challenges that hindered successful project implementation. Procurement was continuously rated “Moderately Unsatisfactory” during project implementation. The two-stage procurement process for two ICB software and hardware packages was cancelled due to allegations of fraud and corruption. An INT investigation found that one of the companies that bid the lowest cost option was found guilty of corruption in another Bank project and therefore could not be granted a Bank contract during its suspension. After the cancellation of this procurement process, a single-stage process was adopted. However, not enough bids were submitted and this procurement process had to be cancelled, resulting in another significant delay for the procurement of the PFMS. A new two-stage procurement process was relaunched in 2013; however, in 2014 the political situation and government priorities changed, and the government decided to cancel the procurement process and requested the cancellation of the undisbursed portion of the loan. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Unsatisfactory Unsatisfactory Risk to Development Negligible to Low Negligible to Low Outcome: Bank Performance: Moderately Unsatisfactory Mitigation efforts for risks were not Unsatisfactory sufficient, there were shortcomings in the Results Framework, and Bank supervision was unrealistically optimistic with regards to progress towards development objectives. Borrower Performance : Moderately Unsatisfactory There was lack of commitment by key Unsatisfactory stakeholders including the Treasury, and lack of strong leadership in the PIU. Quality of ICR: Exemplary NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (p. 24) presents several valuable lessons, including the following:  Change management and political/institutional analysis is critical for the successful implementation of a reform project. The integration of the Treasury with the Ministry of Finance was not sufficiently addressed during project preparation and implementation. A strategy for change management would have been required to mitigate the related risks and create an open dialogue between different stakeholders.  Agreeing on the implementation order of project activities during project preparation can avoid disagreements and delays during project implementation. In this project, the Ministry of Finance postponed the procurement of the PFMS until changes in PFM business processes were completed, leading to an eight-month delay.  Leaving other IT options open during the implementation of a complex integrated financial management system can help to mitigate the risk of an unsuccessful implementation. If this project had also supported the implementation of the e-Treasury system, the country would have been left with a functioning partial IT system until the PFMS was rolled out. This could have also had a positive impact on reducing the Treasury's resistance to the implementation of the PFMS. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR provides a strong and comprehensive overview of project preparation and implementation. Its evidence is robust and of very high quality, drawing from a variety of sources of data. It is appropriately critical and contains all necessary data and analysis. It is results oriented, is internally consistent, and provides lessons that are based on evidence and analysis. Its executive summary provides an extremely useful and concise road map for reading the main body of the ICR, with the project's implementation experience and lessons well summarized. a.Quality of ICR Rating : Exemplary