Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) Report Number: ICRR0023764 1. Project Data Project ID Project Name P149971 Somalia Capacity Injection Country Practice Area(Lead) Somalia Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A0415,TF-A5472,TF-B0692 30-Jun-2020 34,122,132.72 Bank Approval Date Closing Date (Actual) 11-Aug-2015 31-Dec-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 36,415,000.00 36,415,000.00 Revised Commitment 36,148,605.40 34,122,132.72 Actual 34,122,132.72 34,122,132.72 Prepared by Reviewed by ICR Review Coordinator Group Dimitri Tsarouhas Clay Wescott Donna Kaidou Jeffrey IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) for the Somalia Capacity Injection Project (CIP) was “to strengthen the staffing and institutional capacity of selected line ministries and central agencies to perform core government functions” (PAD, p.11). This project was part of a series of projects (SOP) aimed at boosting Somalia’s public sector capacity development and strengthening core Government functions. The Second Public Finance Management Capacity Strengthening Project aimed at establishing systems for more transparent and Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) accountable management and use of public funds, while the Second Phase of the Recurrent Cost & Reform Financing Project (RCRF) financed recurrent costs of core government functions and support to expert services. This included the payment of civil service salaries in non-security sectors, and advisors and civil servants recruited under the CIP. Objective: For this review, the PDO will be parsed into three objectives that will be referred to as Objectives 1, 2 and 3, as follows:  Objective 1: Support the Federal Government of Somalia (FGS) and Puntland State of Somalia (PSS) in developing civil servants’ capacity to perform key government functions.  Objective 2: Support the Federal Government of Somalia (FGS) and Puntland State of Somalia (PSS) in strengthening civil service management through clear frameworks and procedures.  Objective 3: Support the Federal Government of Somalia (FGS) and Puntland State of Somalia (PSS) in improving policy coordination and monitoring capabilities at the center of Government. Split Rating: a split rating was considered due to the downward revision of one outcome target (the number of staff to be appointed in key positions by the FGS was reduced from 210 to 110). The reduction took place because of an FGS initiative to stay within the available fiscal space. It was decided not to apply a split rating for the following reasons. First, a split rating would have hardly affected project outcome ratings since it reflected only a modest reduction in the ambition of the project. Second, and given the duplication of outcome targets throughout the project cycle for two jurisdictions, the outcome target on numbers of staff was only revised for the FGS, whereas it remained unchanged for PSS. Third, following the revision, neither the theory of change nor the PDO were revised. One of the outcome targets, the number of staff appointed to key positions in the Federal Government of Somalia (FGS) by CIM, was revised downwards in the 2021 restructuring. The project was restructured six times in total, and the main reason behind frequent restructuring was the need to accommodate multiple rounds of donor funding. This was due to the project using a basket-funding mechanism by establishing a multi-donor trust fund (MPF); financing occurred incrementally through several grant amendments, and in line with the depositing of the contributions into the main trustee fund. Hence, and in the interest of clarity, a split rating has not been applied. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 11-Nov-2019 c. Will a split evaluation be undertaken? No d. Components Component 1: Developing Capacity for Key Cross-Cutting Government Functions (US$13 million at appraisal, US$8.6 million actual). The primary objective of this component was to support the recruitment, Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) placement, and management of qualified Somalis in critical functions in targeted institutions at the federal government level (FGS) and Puntland State government level (PSS). The targeted institutions included institutions in both jurisdictions. Regarding the FGS, targeted institutions comprised the Ministry of Finance, Ministry of Planning and International Cooperation, Ministry of Labor and Social Affairs, National Civil Service Commission, Office of the Prime Minister, Office of the President, Central Bank of Somalia, Ministry of Agriculture, Ministry of Energy and Water, and Ministry of Public Works, and Reconstruction. Regarding Puntland State, the institutions in question were the Office of the President, Ministry of Finance, Ministry of Planning Economic Development and International Cooperation, Ministry of Labor, Youth and Sports, Ministry of Livestock and Animal Husbandry, Ministry of Environment, Agriculture and Climate Change, Ministry of Fisheries and Marine Resources, Civil Service Commission, and the State Bank. Sub-Component 1.1: Supporting harmonized and Government-led capacity injection in priority institutions. This sub-component aimed to identify several institutions benefitting from the Capacity Injection Modality (CIM) in the first year of operation, as they were considered critical for project success. CIM was set up for the CIP project and intended to inject capacity at senior advisor and mid-managerial levels in both jurisdictions, setting up a central mechanism for that purpose. At the federal level, priority institutions included the Ministry of Finance, the Central Bank of Somalia, the Prime Minister’s Office, the Ministry of Planning and International Cooperation, the Ministry of Labor and Social Affairs, the National Civil Service Commission, the ministries of Energy and Water, Public Works, and Agriculture. At the Puntland State level, these included the Ministry of Finance, the State Bank of Puntland, the Office of the President, the Ministry of Planning, the Ministry of Labor and Social Affairs, the Puntland Civil Service Commission, the Ministry of Livestock and the Ministry of Fisheries. A set of criteria was identified to designate beneficiary institutions. The criteria used stressed the role played by the institution in strengthening core Government processes, whether the institution delivers flagship programs and/or visible benefits to the Somali people, whether it has a clear mandate and management process, and whether it is committed to reforms and in line with the World Bank/UNDP joint effort to avoid donor effort duplication and waste. Sub-Component 1.2: Providing specialized training and coaching support for recruits and their teams. This sub-component aimed to develop and implement training modules for short-term training of senior advisors and managers of the civil service and specialized short-term training courses for staff performing common functions in FM, procurement, HRM and policy management. Short-term training included a. induction training for all recruits to familiarize them with the basics of working in the Somali public service; b. senior management training targeting senior managers to improve policy development, strategic planning, and change management, and c. specialized technical training courses targeting Government staff to equip cadres with the skills to carry out their tasks in line with the Government’s rules and regulations. Long-term training through curricula development and training modules was envisaged to develop the capacity of Somali training and development providers, including in the tertiary education sector. Processes of certification and evaluation, and training of trainers were also envisaged for both FGS and PSS in conjunction with partner institutions in Africa. Component 2: Strengthening the Policies and Procedures for Civil Service Management (US$12.25 million at appraisal, US$12.4 million actual). The primary objective of this component was to support the Federal Government and the Puntland State of Somalia to manage and implement their civil service and capacity development agenda. It would further support the restructuring of selected ministries and agencies and establish the basis for recruitment and staff placement. Sub-Component 2.1: Supporting Priority Ministries in Re-organization and change management. This sub- component aimed to define mandates, functions, and institutional responsibilities, develop staffing plans for Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) a select set of priority Ministries and support the implementation of a re-organization and change management on the basis of new mandates and strategic functions. A change management team would be established in each selected implementing ministry/agency to guide and oversee the process. At the federal level, these would include the Ministry of Finance, the Central Bank of Somalia, the Prime Minister’s Office, the Ministry of Planning and International Cooperation (statistics and M&E function), the Ministry of Labor and Social Affairs (training and workforce planning), the National Civil Service Commission (personnel management), the ministries of Energy and Water, Public Works, and Agriculture. At the Puntland State level, these would include the Office of the President, the Ministry of Finance, the Ministry of Planning, the Ministry of Labor, the Ministry of Livestock, the Ministry of Agriculture, and the Ministry of Fisheries. Sub-Component 2.2: Strengthening Basic Policies for Civil Service Management. This sub-component aimed to develop, for both FGS and the PSS, basic policy, and management frameworks to provide the enabling environment for civil service performance. It sought to improve the legal, regulatory, policy and administrative environment for fiduciary, human and policy management, and other civil service administration functions. The two basic policies envisaged by this sub-component were pay and grading and pension policy. On the former, the sub-component would support the development of a civil service pay policy and an action plan to include the reclassification of civil service jobs through a pay and grading exercise. On pension policy, the sub-component would fund the development of a pension policy for civil servants, bringing together the Government (employer), civil servants (employees), and representatives of the business community and civil society. Sub-Component 2.3: Civil Service Work Force Management. This sub-component aimed to support the FGS and Puntland in developing a civil service staff rationalization policy to ensure optimal civil service establishment and processes for maintaining and reviewing establishment positions in accordance with demands of ministries and agencies and budget allocations in support of wage bill management. The project would support three major activities under this sub-component, namely (i) fund a human resource audit across the civil service to identify gaps in critical and strategic positions that are vacant or filled temporarily by unqualified and absentee staff, (ii) set up a credible personnel data management system and consequent strengthening of controls on the payroll process and management of civil servants, and (iii) fund a severance policy with participation of key stakeholders including affected retired staff, in support of severance packages and systems and procedures for administration of the packages. Sub-component 2.4: Strengthening the Ethical Foundation of the Civil Service. The sub-component aimed to promote dialogue and discussion on ethical standards for public service; develop a civil service code of ethics; (iii) ensure ethical standards are reflected in the revised civil service legal and regulatory framework; (iv) mainstream the code of conduct in the ministry-level change management processes; (v) support the establishment of adequate mechanisms for enforcement, accountability and complaints, and (vi) develop and implement a communication strategy to disseminate and socialize the code of conduct among civil servants, civil society, business community and international partners. Component 3: Strengthening Policy Management, Coordination and Monitoring Capabilities at the Center of Government (US$6.75 million at appraisal, US$3.5 million actual). The primary objective of this component was to strengthen aid coordination and monitoring and evaluation (M&E) capacity by the ministries of planning at both federal and PSS levels. It would do so by funding the establishment of a policy coordination and monitoring oversight unit in the Office of the Prime Minister (OPM) of the FGS and in the Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) Office of the President (OOP) of Puntland, policy planning units in targeted ministries and agencies, and capacity building and leadership development training for senior officials. Sub-Component 3.1: Strengthening Capacity for Aid Coordination and Monitoring and Evaluation of the Somali Compact. The sub-component aimed to support aid transparency and effectiveness under the Somali Development and Reconstruction Facility (SDRF) and build capacity for monitoring the implementation of the Somali Compact, a framework partnership between Somalia and the international community aiming at peace- and state-building priorities, agreed upon during the 2013 Brussels Conference on Somalia. The Compact recognized capacity development as a key enabler in that process. On aid coordination, an Aid Coordination Unit (ACU) would be set up in the Ministry of Finance at the federal level to collect, curate, analyze and publish aid data; to record aid against the Government’s chart of accounts, to facilitate budget planning; and to track allocations at the sub-national level. On monitoring and evaluation, support would be provided to the Ministries of Planning both at the federal and at the Puntland level and respective units in charge of national M&E and statistics. Sub-Component 3.2: Developing Basic Policy Management Capabilities. The sub-component aimed to improve capabilities to plan future policy development, coordinate key policies across the Government, set and enforce policy development standards, and monitor policy implementation. Specifically, support would be offered to the FGS Prime Minister’s Office (PMO) and Puntland’s President's Office to set and enforce policy standards, support the Capacity Development Units in the FGS Prime Minister’s Office and Puntland’s President’s Office in staffing, training, and office equipment, and help each line Ministry to establish a "focal point" to act as counterpart for policy-related requirements. Sub-Component 3.3: Strengthening leadership for change and delivery. The sub-component aimed to build trust and consensus on priority reforms, strengthen a focus on results, resolve issues between agencies and across Government, and build the capacity of ministry-based reform teams to implement rapid results. Component 4: Project Management and Coordination (US$6 million at appraisal, US$12 million actual). Sub-Component 4.1: Project Management. This sub-component would set up a central Project Coordination Unit (PCU) in the Prime Minister’s Office at the Federal Government level and in the Office of the President at the Puntland State Government level to fund the salaries of key staff and support project-related procurement processes. Sub-Component 4.2: Demonstrating results (Monitoring and Evaluation). This sub-component would fund the hiring of an M&E firm to monitor the project and assist in capacity development at the Ministry's level to collect and analyze data. Restructurings: The project underwent a total of six restructurings for reasons already discussed. The project’s duration was extended twice for a total of 30 months and pushed the completion date from June 2020 to December 2022. This led to the corresponding extension of project indicators and the downward revision of one PDO indicator on the number of staff to be hired (see below). Restructurings were also used for funds’ reallocation under Component 4, with the originally foreseen budget of US$6 million for project management and administration increasing to US$12 million after the last restructuring. The revised components were as follows: Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) Under Component 1, the Young Graduate Scheme that would see young graduates recruited in common functions and selected professional fields with further transition into junior civil service posts was dropped. One of the outcome targets, the number of staff appointed to key positions in the Federal Government of Somalia (FGS) by CIP, was revised from 210 to 110. Under Component 4, the allocated budget for this component was increased. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The Project's original cost at appraisal was US$ 40 million, to be committed through the Somalia Multi-Partner Fund (MPF). A total of US$ 36.4 million was disbursed, and actual project cost at closure was US$ 34.1 million.  A US$ 40 million commitment from Somalia Multi-Partner Trust Funds financed the project. Borrower Contribution: The Federal Government of Somalia and the Puntland State Government did not contribute financially to the project. Dates: The Bank approved the Project on August 11, 2015, and became effective on October 13, 2015. The original closing date was June 30, 2020; however, this was revised twice, and the project closing date was extended by 30 months to December 30, 2022. The Project Mid-term Review was completed on December 10, 2017. The first extension was due to the need for a wider time window to complete project activities in priority areas such as the human resources information management system (HRMIS), the pay and grading and pension system for civil servants as well as the function and implementation of training institutes. The second extension was due to implementation delays associated with the outbreak of COVID- 19. 3. Relevance of Objectives Rationale The ability to hire qualified personnel in the civil service and offer them training and budgetary support would boost public trust in the government’s capacity to deliver essential services. The lack of clarity on the mandates and functions across federal government institutions and between the federal and subnational levels of government in Somalia hindered service delivery capacity. The necessary legal and regulatory frameworks to guide ministries’ work were frequently not implemented and, in some cases, were non-existent. Somalia’s political and economic instability has had consequences in terms of material infrastructure since equipment and records of most government agencies and ministries were destroyed or heavily damaged during the civil war. The civil service was heavily understaffed due to a mass exodus of qualified personnel during the long years of civil conflict, diminishing the pool of qualified labor. Senior civil service levels remained understaffed, and some of them were occupied by staff without the requisite qualifications and skills. There was also a shortage of technical and policy expertise, and almost all FGS institutions recruited advisors to support senior management in highly technical functions. Informed by the Somalia Risk and Resilience Assessment and Systematic Country Diagnostic (2018), the theory of change emphasized that a clear definition of ministerial roles and responsibilities, alongside the Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) development of solid training curricula and a needs-based skills development process for senior staff personnel, would boost civil service management and allow for the effective execution of key core Government functions, in turn enhancing popular trust towards state institutions. The project would also support the improvement of policy coordination between and across the civil service, linking the federal government of Somalia to sub-national levels of government to boost service delivery. The theory of change was clear and outcome-oriented, premised on a realistic set of critical assumptions. Most intermediate results indicators were relevant, specific, measurable, and achievable. Still, the IRIs did not incorporate foreseen spending on hardware and office equipment that would be budgeted through CIP, and such spending was not part of the results framework. The IRIs used to assess staff satisfaction with training provided were limited to training without seeking to measure its content and quality. There is no evidence of a possible attempt to include certain measurement standards to gauge the effectiveness of the training provided. The Project Development Objectives were highly relevant and well-aligned with WB-supported strategies. The FY2014-2016 Interim Strategy Note (ISN) focused on two areas: (i) strengthening core economic institutions and (ii) expanding economic opportunity. CIP’s focus on increasing staff numbers, revamping the pay & grading as well as pension systems in the civil service and offering training that will allow for long-term retaining of highly qualified staff in core institutions aligned fully with the ISN’s call to boost institutions such as the Ministry of Finance, the Central Bank of Somalia, and the Ministry of Planning and International Cooperation. The project contributed to the Somalia Multi-Partner Fund's objective of fostering socio-economic recovery and stabilization by supporting the payment of civil service salaries and the running costs of government institutions and financing technical assistance. The project also aligned with the Somalia Compact’s emphasis on the need to build core public sector capacities as a precondition for rebuilding trust in Somalia’s institutions. The Country Partnership Framework (CPF) FY2019-23, launched when the project was active, included a 'Strengthening Institutions to Delivery Services' focus area, reflecting the increased focus on service delivery reflected in the revised PDO indicators. CIP contributes to and is mentioned under Objective 1.1. Improve public finance management and institutional effectiveness regarding its contribution to training and staffing the civil service as well as boosting the pay and grading, pensions, and severance country framework. The focus on institution-building and financing services was maintained in the CPF after the mid-term PLR review. Additionally, the strengthening of the Somali state apparatus and functioning is a crucial contribution to addressing the drivers of conflict and Somalia's main development constraints, as identified by the SCD. The project was consistent with the Somalia Flagship Program on building resilient and efficient institutions. The federal government set out the 2014 Somalia Flagship Program in partnership with the WB and the United Nations. It was premised on three components: developing capacity for core Government functions, strengthening the frameworks and procedures for civil service management, and strengthening capacity at the center of Government to coordinate policy and lead reform effectively. Further, the Federal Government’s Financial Governance Program identified civil service reform and institutional capacity building as an invaluable reform layer on the path toward the sustainable economic recovery of Somalia. Rating Relevance TBL Rating High Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Support the Federal Government of Somalia (FGS) and the Puntland State of Somalia (PSS) in developing civil servants’ capacity to perform key government functions. Rationale Theory of Change: The overall theory of change (TOC) was that government-led capacity injection in priority institutions and training provision would translate into a reinforced capacity by the civil service to perform key government functions. The TOC was premised on a realistic assessment of shortages in qualified personnel and staff by the FGS and PSS and the difference that training provisions would make in the recruitment and upkeep of qualified personnel. However, some of the outputs expected to result from planned activities were far-fetched for Somalia’s FCV context: implementing as well as developing a CIM pay scale implied the institutional capacity to harmonize pay scales in the civil service, which was lacking in Somalia. Intermediate results and outputs  The percentage of injected staff with salaries compliant with the harmonized pay scale rose from 90 to 100 percent in both the FGS and PS, achieving the set target.  The number of staff who state that they have used the training they received on the job rose to 90 percent from 80 percent in the FGS, exceeding the 80 percent target and reaching the 80 percent target in PSS.  The percentage of staff who reported adequate training was 75 percent in FGs against a 70 percent target; the equivalent percentage in PSS was 80 percent, but staff’s self-reporting in PS concerned “improvement” rather than training adequacy. Outcomes Against Targets The share of CIM staff appointed to key positions in FGS by project closure did not hit the initial or reduced target after restructuring, as it reached 86 percent of the revised and reduced project target (95 out of 110). The proportion of staff that remained in the service after 12 months exceeded the target of 70 percent for both FGS and PSS. Although the total number of staff recruited by FGS through the CIM mechanism over the project life was 151, that number was subsequently reduced through attrition. Some CIM staff have departed from their posts, while some CIM positions were canceled and replaced with consultants or civil servants. CIM staff retention indicators were exceeded: while the target retention rate was 70 percent for both FGS and PSS, the actual retention rate reached 94 percent and 97 percent in FGS and PSS, respectively. Staff training was facilitated through the setup of civil service training institutions and the development of training programs, namely the School of Management and Public Administration (SOMPA) in the Somalia National University, Mogadishu, and the Institute of Public Administration and Management (IPAM) in the Puntland State University, Garowe. SOMPA delivered 24 courses and trained 960 middle and senior-level management staff, while IPAM trained 110 CIM recruits. Twenty-four courses were delivered, and 688 middle Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) and senior-level management staff were trained. The e-learning course materials prepared by SOMPA and IPAM were neither digitalized nor tested by project closure. The project contributed to strengthening the staffing of selected line ministries and central agencies. One IRI was not achieved at closure. Rating Substantial OBJECTIVE 2 Objective Support the Federal Government of Somalia (FGS) and the Puntland State of Somalia (PSS) in strengthening civil service management through clear frameworks and procedures. Rationale Theory of Change: The overall theory of change (TOC) was that support offered to key ministries in managing the civil service workforce and revamping their organizational mandate would enable the formation of a transparent, efficient, and accountable civil service. The theory of change of the second objective was less articulate than the one for the first objective, primarily because it relied on an overtly ambitious set of outputs to accomplish a new civil service management culture. The establishment and enforcement of a civil service code necessitated not only governmental commitment (a critical assumption incorporated in the TOC) but also the institutional capacity to hold policy-makers accountable and use transparent evaluation criteria, which were both lacking at the time of operation. Intermediate results and outputs  A new pay policy for the civil service has been prepared and submitted for Cabinet approval at the FGS, although not in PS.  The percentage of senior staff in supported institutions with job descriptions reached 100 percent in both FGS and PSS.  The percentage of civil servants with complete electronic personal records in participating ministries reached 100 percent in FGS against a 95 percent target and reached the target in PSS.  Guidelines for the recruitment and performance appraisal of civil servants have been developed and approved by a Steering Committee in both the FGS and PSS.  A new pension policy and phased plan for implementation has been developed and submitted for Cabinet approval in both FGS and PS. Outcomes Against Targets The number of agencies and ministries with operational Human Resources (HR) and Financial Management (FM) functions that are fully staffed and operational was met (in FGS) and exceeded (in PSS) at the project closure. The proportion of injected staff with salaries compliant with a harmonized pay scale exceeded 90 percent in both FGS and PSS, reaching 100 percent in both. In contrast, the target of seven change Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) management plans being formulated and implemented by project closure was exceeded in both FGS (9) and PSS (10). A pension policy has been prepared in both FGS and PSS, but a new civil service pay policy in PSS was completed after project closure. The project contributed to strengthening the institutional capacity of selected line ministries and central agencies. It allowed for the formulation of new pension and civil servants’ pay and grading policies while preparing a new HRMIS system. However, neither the pay nor pension policy has been implemented by project closure, raising concerns about the fiscal sustainability of the civil servants’ wage bill. HRMIS implementation has been delayed, potentially leading to backtracking in establishing new HR rules and procedures. Rating Modest OBJECTIVE 3 Objective Support the Federal Government of Somalia (FGS) and the Puntland State of Somalia (PSS) in improving policy coordination and monitoring capabilities at the center of Government. Rationale Theory of Change: The overall theory of change (TOC) was that results-based project management strengthened M&E, and better aid coordination would allow for better policy coordination and monitoring at the center of Government. The TOC was premised on a realistic assessment of the policy coordination needs of the FGS and PSS and the added value that WB intervention could make. It did not adequately consider the sustainability dimension of expected outcomes, given the overt dependence of Somali authorities on external support to maintain civil service operations. Intermediate results and outputs  In FGS, 35 Ministries, agencies, and institutions produce bi-annual progress reports against a target of eight, whereas the equivalent number of agencies in PS is 8, equal to the target.  In FGS, the ACU publishes aid data annually.  Office of the Prime Minister (OPM) and Office of the President (OOP) guidelines for policy submissions to the Cabinet have been developed and approved in both FGS and PS.  The percentage of participants in the Executive Leadership Program in FGS exceeded the set target of 50 to reach 141, with 20 percent being female trainees against a target of 10 percent. The target was not reached, however, in PSS. Outcomes Against Targets Operational plans and clusters have been set up for all key Ministries, and the relevant targets have been exceeded by closure. The Puntland Integrated Monitoring and Evaluation System (PIMES) has been set up to Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) track the performance of ministries and agencies. Aid data was annually published until 2021, and it is unclear why this has ceased since. Guidelines for the submission of policy proposals to the Offices of President and Prime Minister in the two jurisdictions have become operational as well. The project strengthened the policy monitoring, evaluation and coordination capacity of key ministries and agencies in the two jurisdictions. The use of statistical data for monitoring and performance evaluation has increased as well, although some disruptions in service have occurred, such as the post-2021 aid data flow. Once intermediate indicator in PSS on Executive Program Leadership trainees (IRI-10) was not achieved at closure. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The PDOs were substantially relevant to the project's overarching objective. All but one of the set PDO targets were met and/or surpassed. Ten out of twelve intermediate indicators were met, with the remaining two met or exceeded in FGS but not met in PSS, namely those on establishing a civil service pay policy and training at least 50 participants in the Executive Leadership Program. Overall Efficacy Rating Substantial 5. Efficiency Traditional measures of efficiency, such as financial rate of return and cost per unit of input or output, are hardly applicable to the project given the lack of economic data, the ongoing conflict and fragility in southern Somalia, and the uncertainty in determining costs and benefits. What is possible is to determine the extent to which available resources were put to as good a use as possible and whether cost-effectiveness was a guiding criterion in Somalia’s FCV context and the project’s funding modalities. The project was implemented within budget, and most of the project funding was used at closure— US$34.1 million out of US$36.4 million, or 93.7 percent. The unallocated amount of US$2 million was to be used for contingencies that did not prove necessary and has been reallocated to the SERP Project instead to fund non-achieved CIM training activities. Several reallocations allowed for better resource use, and costs for three out of the four core project components ended up using fewer resources than originally allocated. Component 1 on Developing Capacity for Key Cross-Cutting Government Functions absorbed 66 percent of allocated resources since salaries for CIM staff injected in the civil service in both FGS and PSS were funded by a different project. Reduced expenditure is also linked to the FGS decision to reduce the end target of the number of staff appointed to key positions from 210 to 110 by project closure. Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) Under Component 2, the headcount of the civil service staff conducted as part of the audit enabled the elimination of ghost staff from the payroll, potentially allowing for significant fiscal savings that would have otherwise not materialized. The project also funded the acquisition of hardware – office furniture, computers, other ICT equipment, biometric attendance registration systems, and vehicles – that facilitate the performance of functions in the FGS and PSS civil service in line with the PDO. Implementation and procurement delays, as well as task duplications, reduced overall efficiency. The 30- month extension of the project (38 percent) raised the administrative costs for the World Bank. Although the outbreak of the pandemic essentially caused the second 18-month project extension, the original project extension in 2019 was down to a lengthy and often inconclusive procurement process that negatively affected project efficiency as it ran up spending. A crucial element was the allocated budget to PCU expenditure, the costs of which proved much larger than originally anticipated and eventually led to the doubling of cost overruns for project management and coordination (from US$6 million to US$12 million). The ICRR concurs with the ICR assessment that implementation delays suggest the project’s economic rate return was lower than otherwise possible relative to the timely project completion scenario. At appraisal, it was decided that project implementation would occur through two PCUs, one for FGS and another for PSS. The result was that technical assistance activities were often duplicated, leading to a waste of resources and overspending. Work was effectively supplied by twin consultants’ teams working in the two jurisdictions, often implementing the same set of TA activities, and ending up with the same deliverables. The sustainability of Somalia’s civil service wage bill would have benefited from the operationalization of HRMIS, which was not complete by project closure and, therefore, an ineffective use of committed resources. Based on the above, the project’s overall efficiency is rated as Modest. The project was implemented within budget, but implementation required a 30-month extension, and the target of staff hired via CIM was reduced for FGS in the last restructuring. The estimates at appraisal had to be revisited, especially those for project administrative costs. Sustained success of initiated reforms will require further donor support. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) CIP achieved significant gains in a difficult environment. The project injected new staff into the civil service, achieved significant retention rates beyond the first year of employment, and enabled the staffing and full operationalization of units on human resources and financial management in both jurisdictions. CIP also offered valuable technical assistance to develop merit-based staff recruitment and placement modalities, advertisement, selection, and onboarding. It facilitated significant progress in defining the mandates, vision, mission, and functions of beneficiary ministries, and project support was extended beyond the originally envisaged Ministries and Agencies, enhancing the number of beneficiaries. The project also allowed for the setup of civil service codes of conduct in FGS and PSS. Most importantly, CIP showed that it was possible to work with government systems in Somalia rather than the off-budget approach taken by most international donors. This helped to strengthen the Somali state and increase its legitimacy, thus tackling some of Somalia's drivers of conflict. Key project deliverables were not implemented by project closure, and the scale of the project’s ambition had to be retroactively lowered. The goal set for the number of competent staff hired through a merit-based recruitment process was not reached in both jurisdictions, even after restructuring and the downsizing of the relevant target. The approval of new staff ceilings was pending on project closure, and such approval corresponds to the fulfillment of core aspects of the project’s developmental objectives pertaining to limiting ad hoc hiring in the civil service, managing the wage bill, and improving civil service management. Progress on formulating concrete policies for pay and grading as well as pensions was made, yet had not been implemented by project closure, and challenges pertaining to off-budget financing for personnel costs and the distinction of donor-funded staff between those who belong to the civil service and those who do not continues to exist. The Relevance of the PDO is rated High, the overall efficacy is Substantial, the efficiency is rated Modest, and the overall outcome is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The risk to the project's development outcomes is significant. It spans most risk categories, primarily due to Somalia's status as a fragile state marked by weak institutions, limited state capacity, and ongoing conflict. These challenges pose significant obstacles to continuing capacity-strengthening implementation and long-term sustainability. An escalation of the security risk could erode the government's effectiveness, potentially draining resources required for civil servant salaries. A potential breakdown in relations between the PSS and the FGS would undermine the progress made in hiring and training competent staff. Competing interests and power struggles characterize the political landscape in Somalia, further jeopardizing the sustainability of development outcomes. One of the most important risks to development is the modest financial capacity of the government of Somalia and its continued dependence on donor support to foot the wage bill. This includes not only the hiring of competent staff and the regular wage remuneration of civil servants but also the provision of technical capacity and training needed to sustain and enlarge the sphere of civil service competence in both jurisdictions. The Somali central government, as well as state governments such as the PSS, remain largely in an embryotic state, and clan- Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) based rivalries can undermine political leadership in support of project reforms and associated policy. Policy implementation remains a key challenge in the country's context. 8. Assessment of Bank Performance a. Quality-at-Entry Project design focused on core capacity-building activities. The project team recognized that building trust in Somali institutions and enhancing state legitimacy went through the setup of a capable, merit-based and well-resourced civil service, as argued by the Somalia systematic country diagnostic. Given the lessons derived from the Somalia SCD, more emphasis on merit-based recruitment could have been attempted. The project rationale departed from the sound argument that institutional capacity- building in FCV contexts is a long-term exercise, in line with the WDR of 2011, and CIP was the first in SOP aiming at that goal. CIP was implemented through government systems rather than creating a parallel system, which was essential for building institutional capacity. Project design reflected lessons from other FCV country contexts in terms of capacity injection, civil service reform and project implementation. Drawing from lessons learned in Afghanistan, Timor- Leste, Liberia and South Sudan, the project team adopted a “domestic ownership” approach through which externally funded staff would be accountable to the Government to avoid the formation of a parallel civil service and a proliferation of ad hoc and asymmetric arrangements. To keep costs under control, the project team also concentrated operations and outputs in a core number of ministries and agencies to selectively build capacity first and expand beneficiaries (which indeed happened) only once technical and institutional standards had risen sufficiently in core institutions. The unallocated US$ 2 million to be used as contingency reflected sound budgetary planning in that it allowed for potential flexible use to cover unexpected costs expected in FCV contexts. Project sequencing did not take adequate account of contextual constraints, resulting in inefficient implementation arrangements. Existing data at preparation pointed to the fragility of Somalia, the political tensions that existed in the country and the need to make efficient use of resources to maximize output towards the PDO. The project did not avoid the effective duplication of tasks in FGS and PSS since it established two separate PCUs that essentially doubled Component 4 costs, resulting in efficiency losses pertaining to two different consultants and experts’ teams. Lessons derived from previous FCV operations were not always reflected in implementation. The project aimed not only at boosting the existing capacity of the civil service to perform its tasks adequately but also to ensure sustainability through(re)training and the adoption of pay and pension policy that will reduce dependence on external funding and boost state legitimacy. The adoption and implementation of such policies is a long and often arduous process that depends, among other things, on government buy- in and adequate technical and administrative capacity. Policy development on pensions and pay was declared a priority area in the PAD. By project closure and despite the 30-month extension, none of the two flagship sustainability policies were implemented, and delays were linked to anticipated political factors associated with ongoing election cycles in 2021 and 2022. This then had knock-on effects, reflected in the non-retirement of civil servants and the inability to recruit new staff. Component 1 was originally about “the recruitment and placement of a small number of highly qualified advisors and civil servants to core Government agencies” (PAD, p. 46). The focus on a small number of advisors and civil Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) servants was replaced by the hiring, management, and training of “qualified Somalis” in both the FGS and PSS. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Political support at the highest level enabled the implementation of difficult reforms. The close engagement with high-level government officials allowed the World Bank to have a strategic dialogue with the government on reforms and continue supporting the reform process despite frequent changes in government and associated priorities. Securing availability of the WB-executed part of the Trust Fund used to fund high-frequency and as-needed implementation support. The Bank’s ongoing dialogue with the government frequently facilitated effective monitoring despite security challenges. Security constraints increased the difficulties for implementation and supervision. It was difficult to organize in-country missions due to security threats, including implementation support missions, and the onslaught of the pandemic made on-site visits for supervision and control very difficult. The Bank offered close support to implementing agencies throughout project implementation, including during the preparation of the procurement documents, review of deliverables, discussions with consultants, and technical advice and support in implementation related to the pay and grading, pensions, and HRMIS reforms. Supervision required significant engagement from the Bank, and the Bank ensured continuity and in-house knowledge by proceeding with only one TTL change through the project’s life cycle. The Bank also maintained an active hands-on dialogue supplemented with frequent face-to-face meetings and consultations with the client government in a frequently changing and volatile environment. Capacity constraints inhibited project implementation. Institutional capacity is weak due to the lack of defined processes and responsibilities, which the project sought to address, as well as the lack of familiarity with World Bank procedures, especially regarding local vendors and their ability to bid for World Bank contracts. This led to multiple requests for clarifications to the PCUs, in addition to complaints. Many suppliers also faced challenges with bidding documents published in English and not in Somali. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) a. M&E Design The project’s theory of change was clear and outcome-oriented, although it entailed some logical gaps. The core PDO on strengthening staffing and institutional capacity of selected line ministries and central agencies was measured by quantitative indicators on the number of staff appointed by the Capacity Injection Modality (CIM), the share of staff remaining in service, and the number of line ministries with operational HR and FM functions. One of the ToC’s critical assumptions on the salience of international commitment was at odds with the criticism of a donor-dependence culture in Somalia. As discussed under components, the 2019 restructuring reduced the reform agenda on FGS staffing at core ministries and agencies with no corresponding reduction in financial commitments. Most intermediate results indicators were relevant, specific, measurable, achievable, relevant, and time-bound, although IRI-10 on participation in the Executive Leadership program had to be dropped for PSS. The IRIs did not incorporate foreseen spending on hardware and office equipment that would be budgeted through CIP, and such spending was not part of the results framework. The IRIs used to assess staff satisfaction with training provided did not attempt to measure its content and quality, and the lack of a pre-training baseline to gauge results made the developmental impact of the training provided uncertain. There is no evidence of a possible attempt to include internationally accepted evaluation standards to gauge the effectiveness of the training provided (e.g., Evaluating Training Programs: The Four Levels (3rd edition) by Donald L. Kirkpatrick, 2006). This is an important logical gap since it remains hard to confirm whether the training offered to civil servants has indeed led to institutional strengthening. b. M&E Implementation The M&E was well carried out through the two PCUs who produced regular reports based on WB templates. The Results Framework was adjusted in 2021 to align one of the PDO targets with the revised project’s scope. Reporting on IRI-3 on measuring results of on-the-job training and IRI-4 on achieving improvements in skills and knowledge compared to the pre-training baseline (IRI-4) was focused on absolute numbers of staff satisfied with training in the first years of the project’s duration, including in relevant ISRs, and that shifted to percentage towards the final years. That constituted an improvement in methodology over time. Exit interviews suggest high satisfaction levels with training offered by beneficiaries. The CIM methodology used in the project became a template used by international donors in Somalia. c. M&E Utilization The M&E framework was well utilized. Capacity constraints in an FCV context notwithstanding, data collected by the project were used in the preparation of the mission reports and ISRs. M&E results pertaining to gender indicators informed the development of the FY2019-22 CPF (extended to FY23) regarding the desire by the government to increase female recruitment into middle and upper management positions under a new National Public Sector Reform Strategy to 30 percent of the total intake, up from 20 percent under the project. In addition, the PCU used the data collected by the project to increase project visibility through the project website and social media. Frequent restructurings did not lead to more ambitious indicators, which could have enhanced the reform drive associated with the PDO-level indicators. Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Category C, and no environmental or social compliance issues occurred during implementation. No specific environmental or social safeguard instruments were required, and no safeguard policies were triggered. b. Fiduciary Compliance The project maintained adequate financial management arrangements, with performance rated Moderately Satisfactory through its lifetime cycle. Both PIUs, in FGS and PSS, were headed by experienced accountants and were supported by the team from the External Assistance Fiduciary Section within the Accountant General’s office. The project submitted all interim financial reports within the stipulated timelines. The project maintained one designated account (DA) at the Central Bank of Somalia for disbursements related to the FGS-level activities and another DA at the First Community Bank for disbursements to Puntland. The overall level of disbursements at the date of closure was noted to be highly satisfactory. Internal control gaps in the system of internal control were noted, and the project continued to address these issues progressively. The overall project procurement performance is rated as Moderately Satisfactory. Project procurement was conducted in accordance with the WB’s procurement guidelines under IBRD loans and credit and in accordance with the approved Procurement Plan. Since the national procurement law was not fully operational, customized Bank standard procurement documents were used to approach the national market. PCUs implemented the CIP in FGS-OPM and PSS-OOP. Procurement delays were mostly due to the challenging operating environment and weak capacity of procurement staff as well as unfamiliarity with the WB’s procurement requirements. Additional procurement support was offered as implementation progressed, including hands-on support by the WB procurement and technical teams. Recommendations made during implementation support missions, combined with the training, resulted in improved processes and a reduction in procurement processing turn-around time. In 2020/2021, procurement delays were experienced mainly due to the COVID-19 pandemic. Mitigation measures taken included extensions of contracts and bids, allowing bidders, suppliers, and consultants to submit their bids or proposals electronically with secured passwords, and online opening of bids and proposals. c. Unintended impacts (Positive or Negative) An unintended benefit of the CIP was that the CIM methodology became the foundation for comparability of pay structures across donors operating in Somalia. The CIM methodology is currently being used by many United Nations partners to set salaries for their PIUs. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Task duplication through two separate PCUs and delays in the development and Bank Performance Satisfactory Moderately Satisfactory implementation of flagship P&G and pension policy reform reduced the project’s developmental impact. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Front-load tough reform areas to build momentum. At appraisal, the project envisaged the capacity to design and then implement both P&G and pension policy within the lifetime of the project, which proved unrealistic. As one of the ICR lessons rightly underlines, such long-term, and inherently difficult reforms face multiple veto players, opposed coalitions, and entrenched interests set against change. In the presence of government commitment to reform, they should be introduced at the very early stages of any reform drive. This will convince reform beneficiaries (who, in the case of pension reform, are many but dispersed and hardly speak as a pressure or interest group) that the developmental impact of the reform will be positive and thus enlarge the pro-change constituency. The same lesson applies to staffing core Government: the RF envisaged gradual hiring of staff through the CIM, and this proved unrealistic. IEG concurs with the ICR that frontloading such hiring under conducive budgetary conditions would have allowed for other reform priorities to follow. By the same token, and given the country context, it would have been important to consider a focus on merit-based recruitment in the civil service and focus on a system that goes beyond dividing civil service jobs according to clan and minority group status, instead of emphasizing pay and pension policy aspects. Consider the administrative implications of duplicating tasks in a federal system. The ICR rightly draws attention to PCU cost. The setup of two PCUs in FGS and PSS reduced efficiency and run-up costs to a disproportionate extent, not allowing for a cost reduction after the fourth restructuring reduced the project scope. In federal systems, it is imperative to rely on one PCU for Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) both fiscal and administrative reasons, and it is best to incorporate this into project design from the outset. The ICR rightly emphasizes the need for additional resource commitment to FCV states during implementation. The project led to substantial gains in development pertaining to institutional capacity building, and one of the main reasons was the devotion of substantial WB resources to supervision and oversight. This is a precondition for success in FCV contexts where instability is endemic and a light-touch approach unlikely to work. Future FCV operations could well draw inspiration from the hiring and utilization of country experts to assist implementation. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR offers an in-depth summary of the project, substantiating its ratings with available data and evidence. It maintains a largely consistent alignment with the project's developmental objectives. It adheres to most of the established guidelines and maintains a results-oriented focus. Lessons drawn from the project are articulated clearly, are practical, and are mostly substantiated by the evidence presented in the ICR. Although the quality of the evidence and analysis is generally robust, there are gaps in data completeness and information. The report does not include a discussion of the project's theory of change, since none was formulated at approval, instead outlining a retroactive ToC through a diagram. It would have been very beneficial to outline further how the ToC looks ipso facto, aiding the reader in comprehending the basis for the given ratings. The ICR’s risk section could have more strongly acknowledged the existential risks to the sustainability of the project results. Also, the ICR doesn't give much sense of the difficult context for carrying out the operation. For example, due to security risks, Bank project staff had to be based in a neighboring country (Kenya), and visiting missions mainly resided and worked at the airport, considered the most secure location. Visits to ministries were accompanied by armed convoys. a. Quality of ICR Rating Substantial Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Somalia Capacity Injection (P149971) Page 20 of 20