Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Report Number : ICRR0023003 1. Project Data Project ID Program Name P149129 Kenya Devolution Support Project Country Practice Area(Lead) Kenya Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Program Cost (USD) IDA-57650 31-Dec-2020 198,723,844.21 Bank Approval Date Closing Date (Actual) 15-Mar-2016 30-Sep-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 0.00 Revised Commitment 200,000,000.00 0.00 Actual 192,990,399.98 0.00 Prepared by Reviewed by ICR Review Coordinator Group Maria Shkaratan J. W. van Holst Kavita Mathur IEGSD (Unit 4) Pellekaan 2. Program Context and Development Objectives DEVOBJ_TBL a. Objectives The Program-for-Results (PforR) Program’s Development Objective (PDO) was to strengthen the capacity of the Recipient’s core national and county institutions to improve delivery of devolved services at the county level. For the purpose of this ICR review, the PDO will be parsed to assess the extent to which it was achieved as follows: Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) PDO 1: to strengthen the capacity of the Recipient’s core county institutions to improve delivery of devolved services at the county level. PDO 2: to strengthen the capacity of the Recipient’s core national institutions to improve delivery of devolved services at the county level. The PDO remained unchanged. The PforR financing amount also did not change. Several Disbursement- Linked-Indicators (DLIs) went through minor adjustments reflecting that the responsibility for some of the results was clarified or moved from one agency to another and that some of the funds were re-allocated from one DLI to another due to increased demand for county performance grants. Split rating. A split rating of objectives was not justified for this review because neither the scope of the PforR nor its ambition changed during implementation, and the financial commitments remained basically the same. The PDO was not revised; however, the DLIs were adjusted to increase incentives and to better support intended outcomes (but not to change the PforR’s ambition). b. Were the program objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 02-Apr-2020 c. Will a split evaluation be undertaken? No d. Components Description of Activities: (i) Finalization of the operational manual for the Program; (ii) Establishment of the steering and technical committees for the Program; (iii) Establishment of the secretariat for the Program; (iv) Carrying out of the initial sensitization of counties about the Program; (v) Initiate the procurement of the annual capacity and performance assessment (ACPA); (vi) Building the capacity of county governments participating in the Program. The PDO was to be achieved through a PforR operation which supported the implementation of the government’s National Capacity Building Framework (NCBF), aimed at promoting the devolution process to Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) improve service delivery. The PAD stated that the PforR would support implementation of the National Capacity Building Framework Medium-Term Intervention (NCBF-MTI) in five key result areas (KRAs) - (i) Public Financial Management (PFM), (ii) Planning, Monitoring and Evaluation (PME), (iii) Human Resources and Performance Management (HRM), (iv) Devolution and Inter-Governmental Relations, and (v) Civic Education and Public Participation (CE&PP) – and set national and county level results for each KRA. The PforR was designed to support the implementation of the five KRAs of the NCBF-MTI by strengthening institutional capacity for devolved service delivery at national and county levels. The operation had two PDO indicators and eight Disbursement Linked Indicators (DLIs), connected to the KRAs. The DLIs specified the results that had to be reached in order to achieve the PDO, by result area, and provided incentives for the government agencies to implement reforms by releasing funds as soon as the DLI targets were reached. Changes during implementation: Two Level 2 restructurings were undertaken during implementation (see ICR, paras 12 and 13). During the first restructuring, in April 2020, the following changes were made:  Time to produce financial statements under DLI1 was increased from 7 to 9 months, and time to complete the Annual Capacity and Performance Assessment (ACPA) (DLI2) was also extended within the same time frame;  A new DLI2a was added; it articulated that the Ministry of Devolution and Arid and Semi-Arid Lands (MoDA) will coordinate the implementation of the PforR;  County planning mandate was shifted from the MoDA to the National Treasury (NT), reflecting the change in the ministerial responsibilities (DLI3);  Value-for-money (VfM) audit was removed from the ACPA and replaced with a requirement that the NT undertake end-of-year VfM audits (DLI5);  For DLI3-6, a year 6 Disbursement Linked Result (DLR) was added to ensure that NT, MoDA, Kenya School of Government (KSG), and Ministry of Public Service, Youth and Gender Affairs (MoPSYGA) have incentives to implement capacity building plans in FY 2019-20;  Number of counties to sign up for the capacity and performance grant was increased from 35 to 47 (DLI7), and some of the funds were re-allocated from DLI8 to DLI7;  The verification protocol of DLIs 7 and 8 was separated, with DLI8 only covered in the ACPA. During the second restructuring, in December 2020, the PforR’s closing date was changed by nine months, from December 31, 2020, to September 30, 2021, to allow the project team to pay close attention to sustain or even improve the performance on the ACPA indicators. e. Comments on Program Cost, Financing, Borrower Contribution, and Dates PforR Cost. Total PforR cost at appraisal was estimated at US$200 million. This amount was not revised during implementation. Actual disbursement at closing was US$193 million. Financing. The PforR was financed with an IDA credit of US$200 million. Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Borrower Contribution. A Borrower contribution was not expected at appraisal. This was revised during implementation, and a contribution was made in the amount of US$87.3 million. The actual contribution at closing was US$68.9 million. Dates. The PforR was approved on March 15, 2016, and became effective on September 15, 2016. The original Closing Date, December 31, 2020, was extended once, for nine months, to September 30, 2021, at which time the PforR closed. 3. Relevance TBL a. Relevance of Objectives New Rationale Rationale The PforR was designed to directly support the government’s NCBF-MTI by complementing government financing of the five devolution capacity building key results areas (KRAs). It also aimed to support the NCBF-MTI alignment of national and county results. In addition, the PforR focused on enhancing existing county systems to reinforce government’s own PforR and system strengthening initiatives, including through providing results-based financing directly to counties. The PforR identified three broad issues that had a negative impact on the devolved service delivery: (i) lack of county level systems for managing finances, human resources, and planning, monitoring and evaluation; (ii) undeveloped intergovernmental mechanisms; and (iii) unstructured citizen engagement - and selected five KRAs to address them: - KRA 1: Public Financial Management (PFM): county budgeting, revenue management; use of Integrated Financial Management Information Systems (IFMIS); financial accounting, recording, and reporting, procurement, and internal and external audit performance. - KRA 2: Planning, Monitoring and Evaluation (PME): county planning, progress reports, monitoring, and evaluation (M&E), and linkages between county plans and budgets. - KRA 3: Human Resource and Performance Management (HRM): county staffing plans, human resources (HR) competency frameworks, appraisal, and performance contracting systems. - KRA 4: Devolution and Inter-Governmental Relations (D&IGR): introduction of a new performance-based conditional grant. - KRA 5: Civic Education and Public Participation (CE&PP): rollout of civic education and county civic education units; greater number of counties that meet the County Government Act requirements for public participation and transparency. The PDO was aligned with the World Bank’s Country Partnership Strategy (CPS) (FY2014-18), which prioritized actions to help Kenya deliver on the ‘devolution dividend’. Devolution was considered a means to Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) increase equity in development across Kenya, reduce spatial disparities, and promote accountable and responsive local governments. The PDO was also fully aligned with the CPS for Kenya at PforR closure (the CPS FY14–18, extended under the Performance and Learning Review of the CPS to FY20), specifically with the third results area “consistency and equity—delivering a devolution dividend”. Rating Relevance TBL Rating High b. Relevance of DLIs New DLI RELEVANCE TBL DLI 1 DLI Office of the Auditor General (OAG) has conducted county audits on time Rationale The indicator was clearly defined, measurable, and linked to the KRA 1 - Public Financial Management. Audits were conducted for all counties that submitted financial statements, 47 in total. At Program approval, the audits were significantly delayed. The audits were used for assessing overall country fiduciary capacity and governance, as well as to determine how much a county can receive through new grants, and the DLI 1 provided an incentive to perform these functions in a timely fashion. The relevance of DLI 1 is rated High because it supported functions that were critical for the Program and for counties' institutional development, was fully aligned with the results framework (RF), and supported the PDO. Rating High DLI 2 DLI Introduction and timely implementation of ACPA by the Ministry of Devolution and Planning Rationale The indicator was well defined, measurable, and linked to the KRA 1 - Public Financial Management. ACPA would be conducted by MoDP; the indicator would be fulfilled when: (i) ACPA is concluded on time for inclusion of capacity and performance grants' allocations each year and (ii) starting in year 3 of the Program, value for money (VfM) audits are included in the ACPAs. The relevance of DLI 2 is rated High because, together with DLI1, it supported functions that were critical for the Program and for county institutional development, was fully aligned with the (RF), and supported the PDO. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Rating High DLI 3 DLI Ministry of Devolution and Planning implements annual planned activities to strengthen countrywide frameworks and systems and to address county capacity gaps Rationale The indicator had a broader definition but was measurable. It was linked to three KRAs: KRA 2 (Planning, Monitoring and Evaluation), KRA 4 (Devolution and Inter-Governmental Relations), and KRA 5 (Civic Education and Public Participation). Under the defined procedure, in Year 1, the MoDP would prepare and submit a capacity building plan, which would be then verified by the performance contracting unit and by the Program’s Technical Committee and then, if the requirements were met, endorsed by the Program’s Technical Committee. In Years 2-5, the MoDP would be submitting both the annual workplan and the implementation report, and the same verification procedure would follow. The relevance of DLI 3 is rated Substantial because it supported the PDO and was fully aligned with the RF. Rating Substantial DLI 4 DLI Ministry of Public Service, Youth and Gender Affairs (MoPSYGA) implements annual planned activities to strengthen countrywide frameworks and systems and to address county capacity gaps Rationale This indicator had a broader definition but was measurable. It was linked to the KRA 3 - Human Resource and Performance Management. Under the defined procedure, in Year 1, the Directorate of Public Service Management (DPSM) would prepare and submit a capacity building plan, which would be then verified by the performance contracting unit and by the Program’s Technical Committee and then, if the requirements were met, endorsed by the Program’s Technical Committee. In Years 2-5, the DPSM would be submitting both the annual workplan and the implementation report, and the same verification procedure would follow. The relevance of DLI 4 is rated Substantial because it supported the PDO and was fully aligned with the RF. Rating Substantial DLI 5 DLI National Treasury (NT) implements annual planned activities to strengthen countrywide frameworks and systems and to address county capacity gaps Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Rationale This indicator had a broad definition but was also measurable. It was linked to the KRA 1 - Public Financial Management. Under the defined procedure, in Year 1, the National Treasury (NT) would prepare and submit a capacity building plan, which would be then verified by the performance contracting unit and by the Program’s Technical Committee and then, if the requirements were met, endorsed by the Program’s Technical Committee. In Years 2-5, the NT would be submitting both the annual workplan and the implementation report, and the same verification procedure would follow. The relevance of DLI 5 is rated Substantial because it supported the PDO and was fully aligned with the RF. Rating Substantial DLI 6 DLI Kenya School of Government (KSG) implements annual planned activities to address county capacity gaps Rationale The indicator had a broader definition but was also measurable. It was linked to all five KRAs. Under the defined procedure, in Year 1, the Kenya School of Government (KSG) would prepare and submit a capacity building plan, which would be then verified by the performance contracting unit and by the Program’s Technical Committee and then, if the requirements were met, endorsed by the Program’s Technical Committee. In Years 2-5, the KSG would be submitting both the annual workplan and the implementation report, and the same verification procedure would follow. The relevance of DLI 6 is rated Substantial because it supported the PDO and was fully aligned with the RF. Rating Substantial DLI 7 DLI Counties have participated in an annual assessment of performance and met MACs Rationale The indicator was clearly defined, measurable, and linked to the KRA 4 - Devolution and Inter-Governmental Relations. The indicator would be fulfilled when: (i) the ACPA Minimum Access Conditions have been met; (ii) allocation of capacity and performance grant is included in the Division of Revenue Bill and County Allocation of Revenue Bill on the basis of the ACPA results; and (iii) when the funds for the previous tranche have been disbursed to the counties as per Program entitlements, unless such disbursements are withheld due to a violation of law by one or more counties. The relevance of DLI 7 is rated High because it supported functions that were important for the Program and for counties' institutional development, was fully aligned with the results framework (RF), and supported the PDO. Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Rating High DLI 8 DLI Counties have participated in an annual assessment of performance, met minimum access conditions (MACs) and minimum performance conditions (MPCs) for grant funding and implemented projects according to Program requirements Rationale This indicator was clearly defined, measurable, and linked to the KRAs 1-3 and 5 (please see above). The indicator would be fulfilled when: (i) the Minimum Access Conditions (ACPAs) have been met; (ii) allocation of capacity and performance grant is included in the Division of Revenue Bill and County Allocation of Revenue Bill on the basis of the ACPA results; and (iii) when the funds for the previous tranche have been disbursed to the counties as per Program entitlements, unless such disbursements are withheld due to violation of law by one or more counties. The relevance of DLI 8 is rated High because it supported functions that were important for the Program and for counties' institutional development, was fully aligned with the results framework (RF), and supported the PDO. Rating High OVERALL RELEVANCE DLI TBL OVERALL RELEVANCE RATING Rationale Relevance of objectives was rated high, since the objectives were aligned with country priorities, as well as with the WBG strategies at approval and at closure. All DLIs were linked to the RF and consistent with the PDO indicators; they provided strong incentives for institutional performance enhancement. On balance, the relevance of the DLIs is High, providing incentives to cause institutional change in support of the PDO and alliance with the RF. Overall relevance is therefore rated high. Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) OBJECTIVE 1 Objective PDO 1: to strengthen the capacity of the Recipient’s core county institutions to improve delivery of devolved services at the county level. Rationale Theory of Change The theory of change for this objective, developed for the ICR, shows a link between inputs and outputs, and then moves directly to the final result under this objective: improved institutional performance at county level. The immediate outcomes are missing, and the outputs are expressed as a list of main DLIs (completed annual capacity building plans and implemented activities to address capacity gaps and strengthen HRM and PFM frameworks). The DLIs were designed for incentivizing and monitoring progress during Program implementation, they do not fully reflect PforR’s outputs or outcomes; those can be derived from the KRAs. The TOC specifies neither the areas where capacity was improved in county institutions, nor which institutions were involved. It is unclear from the TOC if the project activities were aimed at enabling the county institutions to perform the new tasks of a devolved service provision system or not. Outside of the TOC section, the ICR describes such new tasks as “managing finances, human resources, planning, monitoring, and evaluation” (ICR, page 7) and provides in-depth details on their substance (KPAs 1-5, ICR, page 7), intended outcomes, and achievement of outcomes. The TOC could have included a list of the main intended outcomes and describe outputs in terms of capacity building targeted (rather than input-level indicators such as the number of training sessions provided). Responding to IEG’s question sent to the Bank project team regarding what was done within the targeted county institutions to transform them into agencies capable of sustainable delivery of devolved services and how their capacity to perform new tasks was developed, the team provided the following information. To build capacity, the counties conducted capacity needs assessments and, using the Program grants, developed capacity building plans, focusing on the KRA areas. The grants were then also used for training, development of policy or legislation, purchasing equipment, civic education and engagement, technical assistance (TA), and peer learning. Capacity building was initiated by both the national government and the counties. The counties, based on their needs, could request training services from the Kenya School of Government, or TA from the national government officials, or TA/training from the hired consultants. County expertise was shared using peer meetings and workshops. Program Achievements (based on ICR, paras 23-32 and Annex 1) PDO 1 addressed two of the three issues with the devolved service delivery that the PfoR tackled: (i) the lack of county level systems for managing finances, human resources, and planning, monitoring and evaluation; and (ii) unstructured citizen engagement. The PforR achieved or exceeded all its PDO 1 indicator targets and all but one intermediate indicator targets. The unachieved target was for the intermediate indicator 2.5 “Number of county projects with a satisfactory value-for-money level”; the reason was that at PforR closing, the Value for Money Audit report was yet to be submitted to the Bank. The latter is considered a minor shortcoming. County capacity in the five PforR’s KRA areas improved. The achievement of the PDO indicator 1 (“Counties have strengthened institutional performance as demonstrated in the ACPA”) significantly exceeded the target of average county performance score of 55 percent and reached 71 percent (out of 100 percent). Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) About 83 percent of counties showed constant year on year improvement in their scores, 15 percent had inconsistent up and down performance, while one county, Nairobi, stagnated. This indicator measured performance against the five KRAs adopted by the PforR: PFM, PME, HRM, CE&PP, and investment implementation and environmental and social safeguards performance. County institutional systems covering financial management, planning, procurement, and safeguards were enhanced. At PforR closing, the PDO indicator 2 (“Number of counties which comply with the MPCs”) exceeded the target of 35 counties and reached 38 counties; and the performance on the Intermediate Results Indicator (IRI) 2.2 (“Improved Planning and M&E capacities”) was at 97 percent. The former indicator (PDO indicator 2) measured county capacity on financial management (financial statements and audit opinions), planning (annual planning documents), use of funds in accordance with the investment menu, procurement, core county staffing, and environmental and social safeguards systems. These systems are vital for improving the delivery of devolved services. The latter indicator (IRI 2.2) measured improved performance by counties. The PforR expanded access to key devolved services (transport, water, health, agriculture, and rural trade). As a result of the PforR, counties developed infrastructure and provided modern services, benefitting 2.5 million people. At closure, out of the 121 projects supported in FY16/17 and FY2017/2018, 65 percent were complete, eight percent were above 85 percent completion, and the remainder were below 85 percent completion. The benefits from the new infrastructure included increased access to education services, reduction in post-harvest losses, improved road network in historically marginalized counties, and improved social amenity facilities with the potential to reduce conflict in remote counties. The End of PforR’s Evaluation Report prepared by MoDA demonstrated beneficiaries’ satisfaction with PforR outcomes in the areas of health care (improved access to health care for over 1 million people and upgraded medical equipment); water and sanitation (improved sanitation and reduced time for collecting water); farm-to-market connectivity in remote areas; and increased food security. Planning, monitoring and evaluation improved. The PforR exceeded its targets on PME. Specifically, the number of County Integrated Development Plans (CIDPs) that followed the guidelines increased by 12 percent, exceeding the target of 10 percent; the number of counties producing Annual Progress Reports on time increased by 97 percent, significantly exceeding the target of seven percent; and the number of counties where the M&E Committees meet regularly increased by 17 percent, as compared to the target of five percent. At closing, 95 percent of the counties had set up functional units and established planning and M&E frameworks; 85 percent of counties had functioning County M&E Committees in place; 93 percent of counties produced CIDPs and annual development plans according to quality standards and on time; and 97 percent of counties produced timely Annual Progress Reports on the implementation of their CIDPs. As a result, effectiveness of the county level planning was increased; accuracy of plans improved; and budgeting was synchronized with implementation schedules. Human Resource and Performance Management systems and practices were modernized. The targets were exceeded. At PforR closing, the mean performance score for counties on HRM was 69 percent, up from 35 percent prior to the PforR. County governments adopted model policies, structures, systems, and procedures developed by the national government agencies, including the review of allocation of functions, organization structures, staffing patterns, and HRM practices; staff rationalization and redeployment; planning for staff recruitment and development; and strengthened systems for recruitment, promotions, and appointment. Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Public Financial Management improved. Two targets were exceeded, and one was not achieved (as explained in the first paragraph of this section). Specifically, the number of counties that prepare Annual Environmental and Social Audits/reports increased by 15 percent, as compared with the target of six percent; and county institutional performance score was 71 percent, as compared with the target of 60 percent. Also, the quality of financial statements improved by 18 percent and of financial reporting by 13 percent; budget format and quality increased by 39 percent; automated systems for revenue collection increased by 62 percent, and the OSR increasing by 66 percent; procurement procedures improved by 32 percent; and internal audit improved by 26 percent. However, further improvement of capacity in the areas of cash management, public investment management and revenue automation was still needed at closure. Additionally, county auditing was enhanced. The share of qualified audit opinions was increased from 28 percent to 77 percent, the number of adverse and disclaimed audits was reduced, and two counties with no prior auditing obtained unqualified audits. The time needed to produce audited financial statements shortened, and the OAG produced them within the DLI deadline. However, their certification was delayed due to a late appointment of the Auditor General. Systems for citizen engagement were enhanced. The targets were achieved or exceeded. The mean performance score increased from 50 percent to 82 percent. At PforR closing, 93 percent of counties had set up functional civic education units; 93 percent had established systems for access to information; 93 percent had institutional structures, systems, and processes for public participation; 70 percent had participatory planning and budgeting forums to engage the communities; and 81 percent were sharing core PFM material online and in print. The PforR supported Counties to respond to COVID-19. About 51 percent of the discretionary performance grants to qualifying county governments financed health-care investments. The PforR increased county bed capacity by over 2,700, many of which served as COVID-19 isolation centers; supported the installation of over 10 oxygen plants; and helped the counties to embed measures in capacity building and strategic plans that mitigate the effects of the COVID19 pandemic. Counties repurposed part of the capacity building grants toward COVID-19 awareness raising and training in rural communities, provision of personal protective equipment to over 1,000 front line health workers; and distribution of masks to local communities. The PforR achieved or exceeded all its targets under PDO 1 except one intermediate indicator target. The latter is considered a minor shortcoming. As a result of the PforR, counties’ capacity in all five KRA areas increased, as reflected in the PDO indicator 1, which significantly exceeded the target. Specifically: (i) county capacity on financial management, use of funds as per investment plans, procurement, staffing, and safeguards - systems vital for the delivery of devolved services - was enhanced, exceeding the targets; (ii) access to key devolved services (transport, water, health, agriculture, and rural trade) increased, benefitting 2.5 million people; (iii) PME systems’ improvement efforts lead to increased effectiveness of the county planning and budgeting; (iv) HRM system’s modernization meant more efficient recruitment, promotion, and appointments; (v) PFM reform lead to improved capacity in such areas as financial reporting, budget formulation, revenue collection, procurement, and internal and external audit; and (vi) citizen engagement was enhanced (access to information was provided and systems for public participation were created). This review concludes that the efficacy with which Objective 1 has been achieved is substantial. Rating Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Substantial OBJECTIVE 2 Objective PDO 2: to strengthen the capacity of the Recipient’s core national institutions to improve delivery of devolved services at the county level. Rationale Theory of Change Please see the discussion of the TOC under Objective 1. It also applies to Objective 2. Program Achievements (based on ICR, paras 23-32 and Annex 1) PDO 2 addressed the third issue with the devolved service delivery that the PforR tackled: the undeveloped intergovernmental mechanisms. It also financed capacity building of the national institutions to support the devolved service provision. All related targets were met. Specifically, the following activities (monitored by the RF) were implemented:  Capacity building at MoDA;  Capacity building at the Directorate of Public Service Management (DPSM);  PFM capacity building at NT;  KSG implemented annual planned activities to address county capacity gaps;  Capacity building to strengthen inter-governmental relations. IEG notes that, despite the achievements listed above, the ICR provides inadequate information about the national level capacity building activities, those that were implemented at the MoDA, DPSM, NT, and intended to strengthen inter-governmental relations (in relation to the latter, it is unclear where the activities were implemented). Also, while the RF includes indicators that monitor the implementation of these activities, no quantitative targets were used; instead the RF utilized the “Yes”-“No” unit of measurement. Further, the outcomes of capacity building at the national level are unclear because only the inputs have been mentioned. Nevertheless, the achievement of capacity building activities at the national level was indirectly verified by project monitoring and measurement. Under PDO 2, the PforR invested in capacity building of the national institutions to support the devolved service provision and strengthen inter-governmental relations, meeting all related targets in the following national level institutions: MoDA, DPSM, NT, and KSG. This review therefore concludes that the efficacy with which Objective 2 was achieved was substantial, but only marginally so because of the inadequate information in the ICR on the extent to which national capacity to support the delivery of devolved services to counties in five key results areas was improved. Rating Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Substantial OVERALL EFFICACY Rationale At closing, the average achievements of all DLIs stood at 92 percent. The program exceeded the targets for its two PDO indicators. All intermediate result indicators were achieved or exceeded except one. As a result of the PforR, counties’ capacity in all five KRA areas increased, including on financial management, use of funds, procurement, staffing, and safeguards (systems vital for the delivery of devolved services); access to key devolved services (transport, water, health, agriculture, and rural trade) improved; and citizen engagement was enhanced (access to information was provided and systems for public participation were created). These results, together with the program’s monitoring data, indicated that the extent to which Objective 1 was achieved was substantial. Objective 2 achievement was also substantial, but because ICR provides inadequate information on the increase in national capacity to support devolved services, this achievement is only marginally substantial. Nevertheless, this review rates the program’s overall efficacy as substantial. Rating Substantial 5. Outcome The Program provided structured and relevant support to newly created counties in Kenya, enhancing them with critical systems for the delivery of services to communities. PDO 1 addressed the lack of county level systems for managing finances, human resources, and planning, monitoring and evaluation, and unstructured citizen engagement. PDO 2 addressed the undeveloped intergovernmental mechanisms and financed capacity building of the national institutions to support the devolved service provision. The PforR achieved or exceeded all its PDO indicator targets and all but one intermediate indicator targets. The latter is considered a minor shortcoming. County implementation capacity in all five PforR’s KRA areas was improved. The counties’ financial management, planning, auditing, PFM, PME, and HR systems were improved; access to the devolved services (transport, water, health, agriculture, and rural trade) was expanded; and systems for citizen engagement were enhanced. These outcomes were due to the increased capacity of service providers at the county level and the improved capacity of national institutions to improve the delivery of devolved services at the county level. Since this review rated the Program’s relevance as high and there were only minor shortcomings in its efficacy, the Program’s overall outcome is rated satisfactory. Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Outcome Rating Satisfactory 6. Risk to Development Outcome Technical and Fiduciary risks. Technical risk was mitigated through the assessment of minimum access conditions (MACs) during annual ACPAs. During implementation, capacities of counties improved, evidenced by assessments conducted during implementation. Fiduciary issues (weak procurement, fraud, and corruption) were mitigated through strengthening PFM and procurement, and performance measures. As a result, the PforR enhanced county capacities, and the risk to development outcomes was rated Low. The project supported systems strengthening of, among others, human resources, financial management, citizen participation, PME, procurement, environmental and social safeguards, which allowed counties to perform their basic functions (ICR, para 71). Financing risk. There was a financing risk: counties might not have had sufficient resources to finance operations and maintenance (O&M) of investments supported by the PforR. To mitigate this risk, the counties used the following: (i) expansion of services, which increased the base for service fees/charges, which could be used to maintain and operate the infrastructure; (ii) investment in solar-powered equipment, to reduce O&M; and (iii) increases in budget allocations to support O&M (ICR, para 72). 7. Assessment of Bank Performance a. Quality-at-Entry The PforR objectives were aligned with the CPS and NCBF-MTI. The objectives were realistic and linked to addressing the most critical issues that prevented improvements in service delivery. The RF and the DLIs were well designed and aligned with the PDO and the ToC. The country context supported PforR implementation: in 2013, 47 county governments were set up and assumed service delivery functions, and several government ministries were mandated to promote devolution. The major stakeholders - KSG, Ministry of Public Service, NT, MoDA, and OAG - were included in the PforR activities. However, three important institutions were not included at project appraisal (National Environment Management Authority (NEMA), Commission for Administrative Justice (CAJ), and the Council of Governors) although they were brought in midway during implementation, increasing the focus on environmental and social management (ICR, para 41). The DLI verification protocol was well designed: the ACPA were conducted three times by an independent body, and the integrity of the assessment results was respected by the counties and the Ministries, Departments and Agencies (MDAs). The Annual Capacity and Performance Assessment (ACPA) had a design requirement of including a value for money (VfM) audit, which, as it turned out, was unrealistic to undertake annually due to the audit’s complexity (ICR, paras 42, 53, 67). At restructuring, the requirement was therefore modified, and it was decided that only one VfM audit would be done going forward, by the National Treasury (NT) at PforR closure. As noted in Section 8 on M&E below, while the PDO indicators measured progress on devolved capacity to provide services at the county level and the Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) capacity of national institutions to provide support to the counties, the results framework did not include indicators to measure improvements in service delivery directly. This was a shortcoming at appraisal. On balance, this review rates quality at entry as moderately satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The PforR benefited from the stability in the staffing of the supervision team, which supported strong client relations and helped to manage the PforR implementation despite a high county level staff turnover, a problematic inter-departmental coordination in the counties, and other coordination and management challenges. Another challenge was financial: inadequate budget allocations at the national level throughout PforR implementation. The COVID-19 pandemic exacerbated the problem. Yet another external problem was a below expected level of revenue collection during FY21/22 and the resulting budget cuts. Delays outside of the Bank team’s control were challenging too. The ACPAs were often delayed due to procurement issues, resulting in the ACPA misalignment with the national budget process. In addition, the funds to counties by the National Treasury (NT) were often late and subsequently disbursements of grants to county governments were also late. In addition, during the first two years of implementation, delays occurred with submission of audited financial accounts by the OAG. Lastly, COVID-19 affected the completion of ‘Level 2’ investments. The ICR reports (para 47) that the Bank supervision team consistently addressed these external challenges throughout project implementation. During implementation, the PforR had one TTL (all four years), who was country-based. All KRA leads and most task team members also stayed on the team for the length of implementation and were locally based. This allowed for a close engagement with the client and for provision of effective guidance to the involved agencies. The team also quickly reacted to the results of the MTR and made the needed adjustments. Specifically, at appraisal, a small unit of the MoDA Secretariat was assigned to support the PforR. It turned out, the PforR required more support, and a sub- DLI (DLI2a) was introduced at restructuring to increase the support from MoDA. Also, the adjustment related to removing the annual VfM audit requirements was timely. Overall, the team’s close on-the-ground involvement in managing the PforR supported a satisfactory implementation of the first devolution PforR in Kenya, which was a difficult task, especially considering that it required the team to work closely with 47 county governments and eight national government agencies. This review rates quality of supervision as satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) Moderately Satisfactory 8. M&E Design, Implementation, & Utilization a. M&E Design Indicators for the PforR’s results framework were aligned with the PDO, linked to the Theory of Change (reconstructed for the ICR) and the DLIs, and were operationally sound and easy to monitor. The targets were realistic with two exceptions: the requirements for DLI1 and 2 were overly optimistic and were modified at the MTR. The two PDO indicators measured progress on capacity development and systems to deliver devolved services which were the core issues in the PDO. The DLIs were designed to track progress on national and county level results.(ICR, papa 52). The results framework did not include indicators to measure improvements in service delivery directly, which was a shortcoming because these were intermediate project level outcomes, which had to be measured and monitored for the purpose of reporting on the projects value-added for the beneficiaries (in this case, service recipients). A verification process was designed to monitor results through the ACPA and through annual verification of national level work plan implementation. For DLIs 1, 2, 7 and 8, ACPAs were conducted by independent firms contracted by the MoDA. For DLIs 3, 4, 5 and 6, verification was done through annual workplans and implementation reports. The PforR also used the M&E tools such as county budget implementation reports and financial statements, annual reports, capacity building implementation reports, and the MTR (ICR, para 53). A shortcoming of the M&E design was its insufficient monitoring of the national level capacity building activities in the MoDA, DPSM and the NT, which were aimed at strengthening the national-county inter- governmental relations. The indicators for national results were limited to the input level indicating that capacity building took place, while no outcome level indicators (such as improved performance of the agencies) or output level ones (such as changes in agencies’ practices as a result of training) were used. The targets were non-quantitative: the RF utilized the “Yes”-“No” unit of measurement. IEG notes that a completion report – KPSD End of PforR’s Evaluation Report - with information on stakeholder satisfaction was prepared by MoDA at PforR closure. This report complements the RF data. b. M&E Implementation During implementation, three ACPAs were conducted, collecting data on MACs and MPCs. Disbursements on DLIs 1, 2, 7 and 8 were based entirely on ACPA results. The verification protocol was effective, and the envisaged checks and balances for results verification proved useful. Verification of national level results was done in a consistent manner. Disbursements for DLI 3, 4, 5 and 6 were based fully on verified annual workplans and implementation reports. At program closing, the verification of some of the DLIs was pending, and the VfM audit was still outstanding, which was a minor shortcoming (ICR, para 55). County level improvements on M&E achieved due to the program’s investments, supported program’s M&E performance, including RF monitoring, and contributed to improving the quality of the program’s progress reports. The program also put in place a Geo-Enabling system for Monitoring and Supervision Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) platform capturing status of completion, beneficiaries, and amount received for both grants (ICR, para 56). c. M&E Utilization M&E data were routinely used to inform program management and decision-making. As a result, the program made several adjustments. For example, when it was clear that program coordination was lagging, a sub-DLI was introduced to incentivize program coordination activities (DLI 2a). First program restructuring was also informed by the M&E data. MoDA could benefit if it were to institutionalize the ACPA process as a tool for county performance measurement, going forward. The ACPA could also be a basis for future performance-based grants (ICR, para 57). M&E Quality Rating Substantial 9. Other Issues a. Safeguards The Program invested in addressing environmental and social safeguards capacity gaps identified at the county level. During implementation, after encountering some safeguards issues, the program collaborated with NEMA and CAJ with the following outputs: (i) County Environmental Committees were established and became operational; (ii) NEMA established cooperation with the counties; (iii) compliance with the Environmental Management Coordination Act was achieved; (iv) access to information (ATI) frameworks in 47 counties was achieved; (v) Integrity Assurance Officers in 44 counties were appointed and trained; (vi) CAJ supported the county governments on ATI and GRM; and (vii) the GRM guidelines for counties were developed. The compliance with environment and social safeguards requirements by Counties improved substantially over time. The number of counties that met environment and social safeguards requirements increased from 38 (ACPA1) to 43 (ACPA2) and to 45 (ACPA3) out of the total of 47 counties. Compliance with NEMA guidelines ensured that program’s investments had low negative environmental impacts (ICR, para 59-61). At restructuring, some challenges were identified in the county GRM practices and systems. To address those, the following measures were taken: (i) inclusion of GRM capacity building activities in the program’s plans; (ii) increasing the flexibility in the complaints handling systems vs. the previous strict complaint handling requirements; and (iii) incorporating flexibility to the means of verification of the GRMs based on the practice in the counties (ICR, para 62). b. Fiduciary Compliance The program encountered the following fiduciary challenges: (i) availability of program funds at the County Revenue Fund that were holding the funds; (ii) lack of coordination on training by MDAs; (iii) delayed Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) submission of the implementation status reports to MoDA by one-third of the counties; and (iv) delays with the transfer of the funds to counties. These issues were resolved by: (i) the introduction of a Special Purpose Account per county to ring fence program funds; (ii) collaborative preparation of capacity building plans by the MDAs; and (iii) withholding DLI allocation until the counties submitted their progress reports (ICR, para 64). The program encountered some fiduciary compliance issues. First, audit reports for MDAs were continuously late. Some of the audited financial reports lacked disclosures making it hard to identify program’s eligible expenditures in the financial statements. Some audits had limited expenditure data. Second, fiduciary review of the program activities in the counties by the Internal Auditor Department was only conducted once, although the requirement was to do it annually. To resolve these issues, the program introduced a Statement of Expenditure for expenditure tracking. Some MDAs complied with this new requirement while others did not, as this was not an original program requirement (ICR, para 65). c. Unintended impacts (Positive or Negative) --- d. Other --- 10. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory This review assessed quality at entry as moderately satisfactory Bank Performance Satisfactory Moderately Satisfactory for the reasons mentioned in Section 7a. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 11. Lessons Several lessons are drawn from the ICR: - Ownership of the PforR reforms by national and subnational governments is vital for its success. The PforR was designed based on the government program of devolution and on the identification of the main areas where support was sought by the government. This ensured the PforR's strong collaboration on implementation and expectations for sustainability. - PforR coordination needs to be sufficiently incentivized in its design. The PforR appraisal dedicated no resources to support the lead ministry to coordinate all involved agencies, negatively Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) affecting the PforR implementation. At MTR, a correction was made, and a separate sub-DLI was created for that purpose. This is a lesson for the future: PforR coordination needs to be sufficiently financed. - Annual performance assessment capacity within the government needs to be developed. While the PforR chose to hire a private firm for performance assessments, it is critical to build government capacity for the monitoring of government performance after the PforR closure. Future PforRs could start with an outsourced ACPA arrangement, but then transition to a government-led performance assessments before project closure. - National level agencies mandated with environmental and social safeguards need to be identified and supported to implement capacity building and oversight. The PforR involved extensive infrastructure investments requiring a safeguards capacity at county level. However, at appraisal, two government institutions (NEMA and CAJ), mandated with safeguards, were not included in the PforR capacity building activities. That shortcoming was corrected during implementation. Future PforRs involving infrastructure investments may need to include sustained capacity building of such institutions. 12. Assessment Recommended? No 13. Comments on Quality of ICR The ICR delivers relevant and valid information, substantial evidence, and good analysis. The ratings are supported by the narrative. The quality of evidence is adequate: the sources are credible, referenced, and the presentation of evidence is clear. The annexes are put together as appropriate. The analysis is valid, with the exception of the shortcoming (1) described below. However, results orientation is limited: the ICR was not able to present outcome level results, mainly because of the deficiencies of the RF (shortcoming (2) below). Also, the TOC in the ICR is lacking project outcomes, with the exception of the final outcome of improved institutional performance; it is also unclear in the TOC how institutional improvement was supposed to be achieved. The lessons are useful; they are based on the PforR’s own experience, yet they allow for higher-level conclusions. The ICR is internally consistent, follows the guidelines, and is concise and clear. Two main weaknesses of the ICR are as follows: 1. The ICR failed to evaluate national and county level results separately, under two different objectives. Such evaluation would be logical because the TOC (ICR, page 6) presents three PforR outcomes: (i) counties have strengthened institutional performance as demonstrated in the ACPA; (ii) strengthened institutional capacity at the national level; and (iii) Increased compliance of counties to MPCs. While outcomes (i) and (iii) can be combined for evaluation purposes, outcome (ii) is distinctive, as defined by the PDO. 2. The ICR’s evaluation of the M&E design has limitations: it did not describe the RF deficiency in monitoring national level capacity building activities. Specifically, it did not mention that the indicators were at the input level only, while no outcome or output level indicators (improved performance of the Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Kenya Devolution Support Project (P149129) agencies or changes in agencies’ practices) were used; and it did not state that the targets for these indicators were non-quantitative (the “Yes”-“No” unit of measurement). a. Quality of ICR Rating Substantial Page 20 of 20