Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) Report Number: ICRR0022826 1. Project Data Project ID Project Name P145617 PK Sindh Public Sector Management Reform Country Practice Area(Lead) Pakistan Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-55840 31-Aug-2020 47,381,512.18 Bank Approval Date Closing Date (Actual) 28-Jan-2015 28-Feb-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 50,000,000.00 0.00 Revised Commitment 50,000,000.00 0.00 Actual 47,381,512.18 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Judyth L. Twigg Jennifer L. Keller IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) of the Sindh Public Sector Management Reform Project as set out in the Financing Agreement (p. 5) was to strengthen public sector performance in Sindh through improved revenue generation and expenditure management. The PDO in the Project Appraisal Document (PAD) had substantively the same formulation. For purposes of this evaluation, the PDO will be assessed separately for (i) strengthening public sector performance through improved revenue generation and (ii) strengthening public sector performance through Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) improved expenditure management. At a 2018 restructuring, the main outcome indicator and associated target for the second objective was changed from improved budget credibility to improved budget transparency. As this represented a reduction in the scope of that objective, a split rating is performed for this review. The project team confirmed this interpretation of credibility and transparency, defining the latter as a subset of the former, with budget credibility also encompassing clarity, accuracy, and trustworthiness of the budget. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components The project had two components: Component 1: Results-based financing of eligible public sector management reforms (estimated cost at appraisal US$40.0 million; actual cost US$40.0 million). Eligible reforms included improvements in tax administration, public financial management, monitoring of development expenditures, and public procurement. The component was designed to provide incentives for achieving the eligible reforms by means of a results- based financing mechanism consisting of three PDO indicators supported by eight disbursement-linked indicators (DLIs) drawn from among the key performance indicators (PDO indicators and intermediate outcome indicators) included in the reform path of the project. Disbursements from the IDA credit were contingent on satisfactory achievement of DLIs. Component 2: Technical assistance to support capacity-building and institutional strengthening associated with achievement of the eligible public sector management reforms of Component 1, notably to mitigate implementation challenges and promote sustainability of project outcomes (estimated cost at appraisal US$10.0 million; actual cost US$10.0 million). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. Total project cost at appraisal was estimated at US$282 million, including IDA financing of US$50 million and non-IDA (parallel) financing of US$232 million provided by the European Union (EU) and the government. Actual IDA cost at closing was US$47.38 million. No information on actual EU or government financing is included in the ICR. Financing. The project was financed by a US$50.0 million IDA Credit, of which US$47.38 million was disbursed. The difference was due to exchange rate fluctuations during the life of the project. Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) Dates. The IDA Credit was approved on January 28, 2015 and became effective on April 28, 2015. The original closing date August 31, 2020 was extended on May 21, 2020 to February 28, 2021, in large part due to the COVID-19 pandemic that resulted in a lockdown period during which the project made little progress. A Level 2 restructuring was carried out on March 28, 2018. At the time, the project was rated Moderately Unsatisfactory (MU) on both implementation progress and progress towards achieving the PDO. The MU ratings mainly reflected slow progress on strengthening public procurement processes and on several actions related to budget credibility (for instance, capacity building in the eligible reform areas: tax collection, public financial management, procurement, and development monitoring). At the restructuring, one PDO indicator -- improved collection of sales tax on services -- was retained unchanged. Another PDO indicator – improved credibility of the budget – was found to be an uncertain measure due to perennial variations in the budget, and it was replaced by “transparency in budget formulation, allocation, and execution." The third PDO indicator – reduced time taken to process procurement contracts – was dropped due to delays in the implementation of a procurement management information system. 3. Relevance of Objectives Rationale Project objectives aligned with Sindh government priorities in public sector management and with Bank strategy in the sector. Sindh province is the second largest and the most industrialized of Pakistan’s four provinces. It is comparatively wealthy, but at the time the project was developed it was facing uncertain economic growth prospects and declining incomes, as well as weak delivery of public services and static revenue collection. Financial resources were poorly managed. The project objectives addressed state-level revenue and expenditure challenges that were limiting fiscal space and undermining the province’s ability to finance its development needs. In doing so, they aligned with the Sindh government’s own roadmap for fiscal and financial management reform, a Tax Revenue Mobilization Plan introduced in 2014 that drew on the findings of a 2014 Public Expenditure and Financial Accountability (PEFA) assessment. The project was also aligned with the Bank's Pakistan Country Partnership Strategy (CPS) of 2015-2019 that prioritized increasing revenues for delivery of public services, better financial management, and improving the effectiveness of development expenditures. The project objectives also reflected constraints identified in the 2020 Systematic Country Diagnostic, in particular macroeconomic stability and public sector governance, the former emphasizing fiscal management and the latter deepening devolution and strengthening of local government systems. Finally, the project aligned with the technical assistance for public financial management provided to Sindh by the EU and for building up a Sindh Public Procurement Regulatory Authority by the United States Agency for International Development (USAID). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) EFFICACY_TBL OBJECTIVE 1 Objective Strengthen public sector performance in Sindh through improved revenue generation Rationale Improved revenue generation focused on improving the collection of sales tax on services (STS), which was the largest single source of tax revenue in the province. Improved collection was to be driven by the application of a Sindh Tax Revenue Mobilization Reform Plan, including elements of streamlining the tax collection process such as stronger coordination between provincial tax collection agencies (there were three, with Sindh’s Revenue Board, SRB, being the main one and responsible for collecting the STS); institutional development of SRB with enhanced staffing, increased automation, and improved auditing practices; and a broadening of the tax base. Outputs/intermediate outcomes/DLIs  The Sindh Revenue Mobilization Reform Plan was approved in February 2014 and is being implemented (DLI 1), achieving the target  More than 90 percent of SRB staff were hired and trained according to a third-party validated staffing plan (DLI2), achieving the target for enhanced human resource capacity for administration at the SRB.  An automation plan for tax collection and administration was implemented and more than 90 percent of non-compliant registered taxpayers received notices from SRB in FY2018/19 (DLI3), achieving the target for increased automation of systems in SRB.  Risk-based audits of taxpayers were implemented, aiming at specific hit rates or hit rate increases, with an annual review to enhance risk-based criteria (DLI4). However, the newly created tax audit unit had limited capacity (ICR, p. 10).  SRB reporting was regularized, with four quarterly reports and one annual report being released, achieving the target for improved SRB performance transparency.  The tax base was increased from 4,800 to 31,000 taxpayers, exceeding the target of 9,500 by over 300 percent. Outcomes: At the time the operation was initiated, STS represented 42 percent of Singh’s provincial tax revenue, with the remainder derived from 16 other taxes and minor fees. By project closing, its share had risen to 57 percent. While expectations on relevant indicators were substantially met, this achievement should be tempered by the susceptibility of the main indicator to factors external to the project, notably economic fluctuations. Rating Substantial Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) OBJECTIVE 2 Objective Strengthen public sector performance in Sindh through improved expenditure management (original outcome indicators/targets) Rationale Improved expenditure management was initially to be achieved by improving the credibility of budget execution, and by gains from reduced time taken to process procurement contracts. Activities toward these outcomes were to include the adoption of better audit practices, introduction of a debt management system, certification of procurement professionals, establishment of a procurement management information system (MIS), and development of information and communication technology (ICT)-based Annual Development Plans (ADPs). Outputs/intermediate outcomes/DLIs  An internal audit function was established (DLI5). The original aim was that at least seven government departments would conduct internal audits and submit them to the principal accounting office (PAO), and at least five PAOs taking appropriate action in response. At restructuring, the number of audits was reduced to four and PAO interventions to three, considering capacity constraints. The revised output targets were achieved.  A budget strategy paper for FY2019/20 was submitted to the Cabinet (DLI6), and all four budget execution reports for FY2018/19 were submitted to the Provincial Assembly and published on the website of the Finance Department.  696 procurement officials were certified against a target of 500 (DLI7), and a third party review of the certification process was carried out.  17 quarterly departmental development plan monitoring reports were prepared and published (DLI8), achieving the original target of 12 reports published for at least four government departments, all carried out under a third party review, and the revised target of 17 quarterly progress reports covering six departments.  Debt management was originally to include comprehensive loan by loan debt stock and flow details in the budget documents for FY2017/18. The target was revised downward during restructuring, to include only a debt database and a debt manual, and the publication of debt bulletins and a debt sustainability analysis; this was achieved.  Six departments implemented geo-tagging of development schemes for spatial monitoring of implementation of development schemes; as well as publishing that information. The target was met.  Community feedback mechanisms were introduced in five government departments, exceeding the target of four departments. Due to delays in development of the procurement MIS, expected outputs related to contract data being entered into that system and contracts being processed through the system's procurement modules were dropped. The MIS was eventually launched "after considerable delays" (ICR, p. 22). Outcomes Credibility of budget execution could not be adequately measured due to an uncertain definition and baseline. Procurement performance stalled due to delays in the introduction of the planned procurement MIS. Of the two key outcome indicators, the procurement indicator was eliminated altogether at restructuring (although Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) the Bank team continued to provide technical assistance that subsequently resulted in improved processes), credibility in budget execution was replaced by “transparency” while retaining the credibility targets. Achievement of the objective as defined under the original outcome indicators and targets is therefore rated Modest. Rating Modest OBJECTIVE 2 REVISION 1 Revised Objective Strengthen public sector performance in Sindh through improved expenditure management (revised outcome indicators/targets) Revised Rationale After restructuring, improved expenditure management was to be achieved by increasing transparency in budget formulation, allocation, and execution, and strengthening related oversight mechanisms. This outcome was achieved through several of the outputs noted above, including the publication of development plan monitoring reports and budget execution reports. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Under the original outcome indicators/targets: Progress on improved revenue generation exceeded targets that were set for STS collection. However, the performance is tempered by the likelihood of the STS revenue generated being influenced by external, not project-related factors, on economic activity. Achievement of the first objective is therefore rated substantial, with caveats. Achievement of improved expenditure management, as defined under the original outcome indicators/targets, was modest, as budget "credibility" was ill-defined and there was no progress on procurement due to delays in implementing the planned MIS. Consequently, efficacy under the original outcome targets is rated modest. Overall Efficacy Rating Primary Reason Modest Low achievement OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) Under the revised outcome indicators/targets: The rating for the first objective, which was not revised, remains substantial. Efficacy for the second objective under the revised outcome indicators/targets is rated substantial, as there was evidence of improved transparency in budget formulation, allocation, and execution. With a substantial rating for both objectives under the revised project, efficacy is rated substantial. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency The PAD and the ICR both undertook an economic analysis of the project, noting that its net present value, albeit difficult to quantify, was nevertheless likely to be positive. The present value based on actual revenue data was estimated at US$1.036 billion, substantially exceeding the equivalent PAD high estimate of US$56.3 million. The benefits mainly derived from improved capacity to collect revenues and enhanced management of public finances. Implementation was favorably influenced by the commitment demonstrated by the government, and a participatory process that drew together provincial government agencies as well as donors. Still, the project faced implementation issues, including "severe bottlenecks which slowed the implementation of activities" (ICR, p. 26), challenges related to the Sindh government's mandate, implementation capacity, and commitment to adopting the proposed reforms (ICR, p. 26), uncertain demand for reform which, if assessed during project preparation, could "have averted most of the implementation delays" (ICR, p. 27), and high turnover of Bank and project staff (ICR, p. 27). These implementation challenges do not reflect the project extension by six months, due to the impacts of the COVID-19 pandemic, namely the inability of suppliers to move equipment or complete final testing. The shortcomings noted were reflected in progress being rated Moderately Unsatisfactory by several supervision missions. Further extensions were avoided by revising some of the project's outcome and intermediate outcome targets. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the original outcome indicators/targets, relevance was rated high, while efficiency and efficacy were rated modest, for an overall Outcome rating of moderately unsatisfactory. Under the revised outcome indicators, relevance was rated high, efficacy was rated substantial, and efficiency was rated modest, for an overall Outcome rating of moderately satisfactory. According to OPCS/IEG guidelines for restructured projects, the final Outcome rating is determined by the ratings for the original project and the restructured project, weighted by the percentage of the Credit that disbursed before and after restructuring. In this case, US$30.08 million, or 63.5 percent of the total disbursement of US$47.38 million, was disbursed under the original outcome indicators/targets. The weighted value of the outcome under the original outcome indicators/targets is calculated as follows: 3 for moderately unsatisfactory x 0..635 = 1.905. The weighted value of the outcome under the revised outcome indicators/targets is: 4 for moderately satisfactory x 0.365 = 1.46. The weighted average is 1.905 + 1.46 = 3.365, which rounds to 3. Therefore, the combined overall Outcome rating is moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome According to the World Bank’s project team, the operation’s outcomes are rooted in strong sense of political and administrative ownership, as policies underlying the PDO in terms of both revenue generation and expenditure management have been internalized by the provincial government, mitigating the political risk that outcomes achieved through the operation would not be maintained. Still, challenges may arise over time despite the apparent broad support for the reforms, such as the risk of losing key champions, multi- agency coordination, and eventual resistance to evolving reforms. Capacity risks were identified and addressed by several of the institutional reforms introduced under the project, such as staff training, better tax collection, and better information systems for public expenditure management. 8. Assessment of Bank Performance a. Quality-at-Entry Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) The operation was strategically relevant and aligned with government and Bank priorities. It drew on the Sindh government’s tax revenue mobilization plan for introducing medium-term reforms in revenue mobilization, as well as public financial management reform plans using PEFA 2014 findings as a foundation. The introduction of a results-based financing modality drew on previous Bank experiences in applying that modality to reform programs (e.g. the Sindh Education Sector Project, P107300). Implementation arrangements built on existing government mechanisms, with a steering committee overseeing project management. Extensive consultations were carried out between government and Bank teams during preparation. The Bank teams also collaborated with the EU which was supporting PFM reform, focusing on aspects of budget credibility, budget execution and reporting, and accountability. While collaboration between the government and Bank teams may have been close, the Bank’s assessment of institutional capacity in Sindh to implement the operation would fall short. Capacity constraints had been identified by the Bank team during preparation, and precautionary measures taken, notably in recruitment and training of staff, but this would prove to be insufficient as limited capacity continued to make itself felt throughout implementation. Timelines for implementation might have received closer scrutiny, as might other challenges to implementation, such as building institutional mechanisms designed to manage stakeholder risk, and enrolling taxpayers and internal government stakeholders in the reform. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team extended necessary handholding and implementation support, granted by a multidisciplinary team. At mid-term, adjustments were made - with two out of three PDO-level indicators difficult to monitor (one dropped, one revised), and ten out of 15 intermediate results indicators either revised or dropped, the results framework was restructured to allow completion of the operation. This made it possible for the now-truncated project to stay on track, and it was upgraded from Moderately Unsatisfactory to Moderately Satisfactory by the Bank team, at which level it remained until project closing. While the project had Bank staff turnover, this neither caused any negative impact on the project implementation pace nor affected the engagement with the client; in fact, the proactivity and readiness of the Bank staff allow to provide technical advice and extend its support to the PIU on critical implementation matters. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) 9. M&E Design, Implementation, & Utilization a. M&E Design The original results framework did not define outcome indicators to measure achievement of improved expenditure management. Budget credibility was not well defined, which led to the need to restructure the project to clarify the expected outcome of budget transparency and to drop or revise intermediate outcome indicators and DLIs. The project team also explained that there was an implicit division of labor with the EU, with this project focusing on budget transparency, while the EU focused on budget execution and internal checks and controls; that division of labor was not reflected in the broader statement of the PDO. The results framework as presented in the ICR (p. 7) provided a logical sequence of activities that was consistent with key outcomes and the PDO. b. M&E Implementation Monitoring of implementation was conducted by Bank missions and by an M&E cell that was established within the Sindh Planning and Development Department (P&DD). The P&DD, which has overall responsibility for tracking and assessing development schemes, established a dashboard for tracking and reporting of implementation progress and deliverables. Third-party verification was also used to validate performance, including of intermediate outcome indicators and DLIs. The M&E system was strengthened with improved mechanisms to monitor implementation, compliance with procurement and financial management procedures, and achievement of performance indicators. c. M&E Utilization Information drawn from quarterly and annual reports was used by the Project Implementation Unit to improve project implementation and management, and findings were publicly shared. However, the ICR (pp. 22 and 23) notes that there is insufficient evidence to assess how and how effectively available data was used during project implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The Project was assessed as a Category C project. The project involved no construction or physical works, focusing on systemic reforms that had no environmental or social implications. No safeguard policies were triggered. Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) b. Fiduciary Compliance Financial management. Financial management arrangements, which mainly supported the provincial government's expenditures through its recurrent budgeting, remained adequate during the life of the project. Disbursements were made upon the achievement of DLIs, which were verified by third party validation. All DLIs were met within the required timeframe, and related funds were disbursed. The project prepared regular financial reports with support provided by the Bank team. The Bank received the project’s audited Annual Financial Statements within six months of each financial year. The external audit was conducted in accordance with the International Standards of Supreme Audit Institutions, and the Auditor General of Pakistan expressed an unqualified opinion in the audit reports, which were submitted during each financial year. Financial management issues identified in the early phase of implementation were identified and addressed in a timely manner. Procurement. Originally, the procurement needs of the project were to be identified by relevant project agencies and coordinated by the Finance Department and the Sindh Revenue Board; non-SRB procurement was to be coordinated by the Finance Department. The procedures proved slow, and a new procurement plan was prepared under the project's second component to accelerate the allocation of funds and implementation of activities. The new plan better clarified timelines and procedures. Expected results in public procurement were adjusted because of the complexity of the proposed reform, as well as the limited results achieved prior to restructuring. c. Unintended impacts (Positive or Negative) None noted. d. Other None noted. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The majority of project funds were disbursed under the original outcome indicators/targets, under which Moderately achievement was modest. Outcome Satisfactory Unsatisfactory Project efficiency was rated modest due to significant implementation inefficiencies that impacted project performance. Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) Shortcomings in project design impacted the pace of implementation and achievement of results under the Bank Performance Satisfactory Moderately Satisfactory original outcome indicators/targets, requiring project restructuring, which was carried out relatively late. The project's original outcome indicators and targets were not clearly defined, requiring restructuring, and according to Quality of M&E Substantial Modest the ICR, there is insufficient evidence to determine how available data were used during implementation. Quality of ICR --- Modest 12. Lessons The following lessons are drawn from the ICR. Understanding the political context – risks and potential mitigating measures – is as important at sub-national as national levels. Difficulties that arose during implementation of the Sindh project – notably ones related to meeting targets – could not all have been forestalled, but some might have been alleviated with a better understanding of the regional government’s mandate: when it comes to adopting measures to enhance tax revenue mobilization, strengthen the performance of PFM systems, the management of the development portfolio, and the strengthening of the public procurement system, an understanding of the limits of sub-national autonomy is critical. In the case of the Sindh project, the realism of at least some of the targets set for the project might have been better achieved with a more thorough understanding of the constraints in the political economy. Early project restructuring can turn around an initially non-performing project. Bank teams often wait until the mid-term review to comprehensively assess project performance when an earlier restructuring might avoid implementation difficulties and delays. It may not salvage non- or weakly performing components, but it might nevertheless be meaningful in salvaging and allowing a refocus on other better-performing components. 13. Assessment Recommended? No Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PK Sindh Public Sector Management Reform (P145617) 14. Comments on Quality of ICR The ICR concisely summarizes the background factors that underpinned the relevance of project design. However, it could have benefited from a more careful analysis of challenges that seem to have been overlooked in preparing and implementing the project. These challenges are noted in the ICR's lessons and recommendations (pp. 26-27), but they are not reflected in the main text of the document. There is therefore insufficient information on the extent and manner in which capacity constraints, buy-in, etc., affected project performance, and how such issues were addressed to ensure completion of the project. A more thoughtful approach to the discussion around the intended outcomes of budget “credibility” vs “transparency” would also have been helpful. Mostly, while the ICR presents the project's results framework and raises salient points, it does not provide a clear understanding of the logic between activities, outputs, and outcomes. a. Quality of ICR Rating Modest Page 13 of 13