Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) Report Number : ICRR0021279 1. Project Data Project ID Project Name P113421 AF: Pension Admin and Safety Net Country Practice Area(Lead) Additional Financing Afghanistan Social Protection & Labor P122642,P122642 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-H5230,IDA-H8480 31-Dec-2013 18,814,334.25 Bank Approval Date Closing Date (Actual) 15-Oct-2009 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 7,500,000.00 0.00 Revised Commitment 19,995,992.22 0.00 Actual 19,228,703.57 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives According to the Financing Agreement (p. 4), the project's objectives were: "(i) to improve the administration of the Recipient's public pension schemes; and (ii) to pilot a modest social safety net program as a first step to developing a sustainable approach to safety nets in the country." At an Additional Financing (AF) of US$ 12.5 million in 2013, the objectives were revised: "(i) improve the administration of the public pension schemes; and (ii) develop administrative systems for safety nets Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) interventions, with focus on targeting and benefit payment delivery, while delivering cash benefits to the poorest families in targeted pilot districts." IEG concurs with the ICR's assessment that the two sets of objectives were substantively identical. Part (ii) of the revised objectives added some specificity (including the "focus on targeting and benefit payment delivery") but broadened the project's scope, adding activities related to the development of a Management Information System (MIS) and rehabilitation of buildings for the Martyrs and Disabled Pension Program (MDPP), expansion of the safety net pilot to four districts, and piloting of conditional cash transfers. The AF was commensurate with the expanded activities. A split rating is therefore not warranted. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 13-Jun-2013 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project contained three components: 1. Pensions (appraisal, US$ 3.3 million; US$ 4.0 million added at AF; actual, US$ 7.86 million) was to (i) design new administrative arrangements for the public pension system, and (ii) modernize the existing pension system. Activities under (i) were to include evaluation of existing systems and processes; elaboration of operational procedures, manuals, and guidelines; design and implementation of a new MIS for the Public Sector Pension Program (PSPP) Pension Department; securing basic information technology (IT) system maintenance, office infrastructure, and electricity supply for the Pension Department in Kabul; and development of a strategy for administrative integration of the payment systems of pensions for families of Martyrs and Disabled and public sector retirement pensions. Activities under (ii) were to include design and implementation of a pensioners census; digitization of records; and surveying pensioners under existing schemes to monitor reform outcomes. These activities were to cover both the public sector retirement schemes covering the civil service, military, and police, and the pension scheme for families of Martyrs and Disabled. 2. Safety nets (appraisal, US$ 3.2 million; US$ 5.4 million added at AF; actual, US$ 9.12 million) was to help design a pilot safety net program, including selection of target groups and pilot areas, and implement the pilot over a period of two years. Toward the end of the project period, an operational evaluation was to Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) be undertaken to determine whether and how to scale up the scheme. This component also included activities to strengthen the institutional capacity of the Ministry of Labor, Social Affairs, Martyrs, and Disabled (MoLSAMD) to plan and administer its own programs, develop and maintain simple databases on ongoing social protection activities, and use those databases to advance the national social protection agenda. 3. Project management (appraisal, US$ 1.0 million; US$ 3.1 million added at AF; actual, US$ 2.63 million) was to facilitate project implementation through provision of consultancy services in project management, financial management, and procurement; design and implementation of training programs and study tours for management and staff; and overall implementation support. At the 2013 AF, all three components were revised. Under the pension component, the AF covered cost overruns for office infrastructure and IT equipment and added activities for design and implementation of administrative modernization of the MDPP. Under the safety net component, the AF scaled up the pilot to four districts, including associated costs for service providers and third-party facilitating partners. It also covered the cost of evaluations of process, targeting accuracy, and project impact. Under the project management component, the AF covered activities to strengthen the institutional capacity of MoLSAMD (which had originally been under the second component), and continued to finance salaries for project staff and all project administrative costs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost, financing, and Borrower contribution: The operation was initially financed by an International Development Association (IDA) Grant of US$ 7.5 million. AF in the form of an IDA Grant of US$ 12.5 million was added at a June 2013 restructuring. At the AF, funds were shifted from the pension and project management components to the safety net component, in order to provide support for more poor and vulnerable households. Actual total disbursement was US$ 19.61 million. At project closure, US$ 414,000 was unused and remained in the special account. No government contribution was planned or made. Dates: The project was approved on October 15, 2009 and became effective on February 9, 2010. The mid-term review was held in July 2013. The AF and revision of objectives, as well as an extension of the original closing date of December 31, 2013, took place on June 13, 2013. The closing date was extended two additional times, in April 2016 (by 12 months) and February 2017 (by six months). The project closed on December 31, 2017. 3. Relevance of Objectives Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) Rationale At appraisal, Afghanistan was one of the poorest countries in the world. Per capita income was around US$ 300. 42% of the population lived below the poverty line, and another 20% clustered just above the poverty line. High levels of chronic poverty were driven by an array of factors: natural disasters, extreme seasonal variations in food production, security challenges, and the 2008 global economic crisis. Existing government benefits were small, weakly administered, and inadequately targeted and coordinated. A well-designed social protection system, suitably adapted to the country context, was intended to help households mitigate and cope with the risks of falling into poverty. The objectives were intended to support specific elements of the Afghanistan National Development Strategy (ANDS) at appraisal toward implementing a modern and financially sound pension system and developing a targeted safety net scheme. They were relevant to the government's 2007 pension reform program that aimed to introduce a sustainable, affordable, equitable, and well-administered pension system. No analogous government reform program or strategy for safety net programming existed at appraisal; the Emergency Project Paper (pp. 3-5) noted that the ANDS provided only a loose strategic framework within which various existing projects and programs fit, and that coordination functions and administrative capacity were extremely weak. The objectives were highly aligned with the Bank's Interim Strategy Note at appraisal (FY 2009-2011), supporting government efforts to reduce poverty, strengthen governance, and improve service delivery. The objectives remained highly relevant to the Bank's Country Partnership Framework at closing (FY 2017- 2020), which contained an objective under the social inclusion pillar to enhance social protection for the poor and vulnerable, and specific support for pension management under its objective to improve public financial management. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Improve the administration of the public pension schemes. Rationale The project envisioned improving the administration of public pension schemes by developing a new mechanism of claims processing and benefit delivery, as well as a new MIS to convert paper-based pensioner records to digital records. These interventions would, in principle, minimize leakage and human error. Administrative improvements were also to accrue through the development and elaboration of Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) operational procedures, manuals, and guidelines, as well as staff training, to strengthen MoLSAMD's capacity for record-keeping and service delivery. Outputs The project developed an Operational Manual, including operational procedures, rules, and terms of reference, for the PSPP Pension Department. This manual now guides all pension administration operations. A PSPP-MIS and biometric systems were developed and installed, and software was developed to link that system with bank payments. 88 PSPP Pension Department staff were trained in implementation of new rules and procedures, exceeding the target of 50. An analogous manual for the MDPP was developed, and an MDPP-MIS and biometric system were developed there, but these did not go into use until MoLSAMD started registering beneficiaries after the project closed. 40 MDPP staff were trained on new rules and procedures, exceeding the target of 35. PSPP office buildings were renovated and furnished in Kabul and four provinces. Similar renovations were provided for the Kabul Martyrs and Disabled office. Individual bank accounts were opened for all pensioners. A PSPP pensions census was conducted, as planned. 123,000 old records were scanned and digitized. A new reporting and monitoring system for PSPP was installed and functional in four provinces, not meeting the target of six provinces. PSPP pension eligibility verification and certification procedures were defined and completed in Kabul and in four provinces, not meeting the target of Kabul plus six provinces. 74 agencies and ministries were connected to the PSPP-MIS and were actively submitting online applications for new pensioners, exceeding the target of 35 agencies and ministries. Outcomes 114,256 PSPP beneficiaries (all eligible pensioners) were registered in the new PSPP-MIS with complete electronic records, an improvement over a baseline of 220 in 2012, and exceeding the target of 85,000. Of these, about 73,000 were transferred from the old paper-based system, and about 41,000 were persons who newly entered into retirement after the new system was introduced. Another 50,000 persons had been on the old paper-based registries, but their records were found to be ineligible because of duplication, multiple registrations, or multiple payments. Removing these duplications saved an estimated US$ 19 million in pension expenditures from 2013-2017. Rather than visiting pension departments in person, pensioners now receive payments directly to their bank accounts. Registration, certification, and benefit calculation is now done only at entry into retirement and once again after two years, through a one-stop window, rather than through the previous burdensome requirement of in-person annual visits to multiple offices. Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) 3,000 MDPP beneficiaries were registered in the MDPP-MIS, not achieving the target of 40,000, largely because of limited ownership at the MoLSAMD that delayed procurement of IT equipment and hiring of necessary staff. After the project closed, the government, with support from the Bank, began to make progress in implementing the MDPP-MIS. The ICR (pp. 14, 26) noted that the MoLSAMD continues to register beneficiaries after project closure, and that it is likely that the target of 40,000 will be achieved by the end of 2018. The project team later stated that, as of November 2018, only 4,665 beneficiaries had been registered, but that progress was expected to accelerate in 2019 due to the successful procurement of IT and hardware and delivery of relevant training in 24 provinces. The project team also noted that the MoLSAMD was recently restructured, with the Martyrs and Disabled Deputy Ministry that had been responsible for the MDPP becoming the National Authority for Supporting People with Disability and Families of Martyrs. With this change, it is anticipated that the newly established Authority will take the reform more seriously, and roll out of the MDPP-MIS will be expedited. The percentage of beneficiaries reporting satisfaction with the customer service facilities offered by the Pension Department (waiting area, office building, etc.) decreased from 51% in 2012 to 45% in 2017, not reaching the target of 75%. The ICR (p. 26) postulated that, despite improvements in the waiting area (installation of air conditioning, new furniture, and male and female toilets), customers were frustrated with challenges encountered during the transition from a paper-based to an automated system. The Pension Department did not anticipate the large number of pensioners who turned up to register with the new system, and according to the ICR's explanation, the satisfaction survey was taken at a time when the unpleasant initial experience was still fresh in respondents' minds. However, the ICR's discussion of safeguards compliance (see Section 10a) raises questions about ongoing maintenance and safety at the renovated MDPP customer service facility. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Pilot a modest social safety net program as a first step to developing a sustainable approach to safety nets in the country / develop administrative systems for safety nets interventions, with focus on targeting and benefit payment delivery, while delivering cash benefits to the poorest families in targeted pilot districts. Rationale The project envisioned that sustainable progress toward improved targeting and delivery of social benefit payments would be achieved through design and implementation, in selected areas, of a safety net administration system that would deliver a targeted cash transfer. This system was to develop and test Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) implementation arrangements, targeting systems, and payment systems to identify the poor and vulnerable and to deliver cash safely and on time. This pilot was then to serve as the foundation for a nation-wide program. Outputs The project developed an MIS, used to pilot both a social registry and a beneficiary registry. An appeals and grievances mechanism was in place and functional, as planned, though the ICR (p. 32) reported that its effectiveness was limited due to the limited capacity of the community committees responsible for its management. The project developed, tested, and evaluated targeting mechanisms for beneficiary selection. Community- based targeting (CBT) was used for the first pilot during 2010-2013: an unconditional cash payment of US$ 40 - US$ 200 (based on the number of dependents in the household) was transferred manually to the poorest 10% of households living in rural areas of three districts to smooth consumption during the winter. Review of this pilot found high inclusion and exclusion errors, risk of elite capture, susceptibility to leakage, and ineffectiveness due to the transfer's one-off nature. Under the AF, these lessons were applied to four additional districts. The expanded pilot introduced proxy means-testing (PMT) targeting to reach the poorest 20% of families with children under five; it reduced the transfer amount but increased frequency, and introduced e-payment. A follow-on evaluation was later conducted to measure the accuracy of the PMT targeting mechanism. Process and impact evaluations were also conducted. Those reports confirmed that, under the expanded pilot, about two-thirds of beneficiaries were from the poorest 40% of the population, a performance that, according to the ICR (pp. 14-15), compares well with other international experience. Inclusion and exclusion errors were relatively small (18% and 28% respectively), a significant improvement over the CBT pilot where the errors were 85% and 80%. The process evaluation showed that 76% of respondents in program communities were aware of the benefit, and more than 90% perceived the targeting as fair and transparent. The project contracted with 11 non-governmental organizations as implementing partners of safety net activities, to mobilize communities, register potential beneficiaries, and select eligible beneficiaries based on established targeting mechanisms. Four staff members were employed in the safety nets-dedicated unit within MoLSAMD, which is no change from the baseline in 2012, and did not achieve the target of six. The project contracted a commercial bank and two mobile money operators to transfer cash to safety net beneficiaries. Payment modalities were electronic (with funds withdrawal on cards) and M-Paisa (using mobile devices); for some remote communities still inaccessible to banks and mobile agents, transfers were manual and in cash. Annual spot checks were carried out through third-party verification mechanisms and the MoLSAMD's internal systems, to verify compliance with safety net benefits delivery procedures. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) A planned conditional cash transfer (CCT) program was not implemented due to lack of supply-side (health and nutrition) services, as well as lack of capacity to monitor and enforce conditions. As a result, no beneficiaries participated in nutrition and hygiene awareness sessions in the CCT communities, not achieving the target of 70% of families. Outcomes 31,456 poor families with children under five years of age received assistance under the safety net intervention, exceeding the target of 28,000. Preliminary impact evaluation findings indicated that the program contributed to increased food security, consumption of durable goods, and school attendance, and reduced child labor among male children (ICR, Annex 4). The percentage of safety net beneficiaries receiving payments on time increased from 40% in 2012 to 72% in 2017, exceeding the target of 60%. The percentage of safety net beneficiaries in project villages who were from the poorest 30% of the population increased from 32% in 2012 to 55% in 2017, not reaching the target of 60%. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale The project significantly improved administration and performance of the pension system serving public sector retirees, and it made progress (on which the government is currently building) in strengthening the pension system serving Martyrs and Disabled. Two pilot safety net programs, with associated administrative systems, were implemented and assessed. Achievement of both objectives is therefore rated Substantial, as is overall Efficacy. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency The Emergency Project Paper (EPP, Annex 9) envisioned that the project could reverse the balances of the public sector pension scheme, moving it from a program requiring subsidies covering 80% of expenditures to Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) one generating temporary surplus of a similar magnitude (that could then be used to clear accumulated liabilities). It was envisioned that the safety net component would "set the basis for a mechanism that would allow more efficient channeling of scarce public resources to the neediest among the population" (EPP, p. 12). No formal economic analysis was undertaken at appraisal. The ICR stated that the pensions-related activities, at a cost of US$ 4.5 million (based on the PSPP only), created fiscal savings of US$ 19 million during 2013-2017 through elimination of ineligible payments, ghost beneficiaries, and other irregularities. The ICR (p. 47) was careful to note, however, that this does not necessarily translate into a 300% rate of return, as some proportion of the reduction in the number of registered beneficiaries was likely due to deaths. The ICR's demographic analysis, based on reasonable and conservative assumptions, found that about 31% of the reduction in beneficiaries was likely due to improvement in record-keeping rather than natural death, accounting for US$ 5.8 million of the savings, and translating into about US$ 1.3 saved for each invested dollar and a return on investment of about 30% (ICR, p. 48). The ICR (p. 16) estimated that the project's investment in MDPP, with benefits not fully realized during the project period, will nonetheless eventually produce fiscal savings between US$ 4.9 million and US$ 6.5 million if the government's commitment to completing the reform holds (ICR, p. 16). This estimate was based on less valid assumptions than that for the PSPP, and the ICR (p. 48) was careful to acknowledge that "this fiscal saving cannot be directly attributed to the project." The ICR also observed that project management costs accounted for about 15% of total costs. The ICR interpreted this positively, given the complexity of the project's implementation environment and the fact that much of the project entailed a complex set of pilots. It was an important shortcoming, however, that implementation delays prevented all planned outputs from being realized during the project's lifetime. Extensions of the closing date totaled 18 months. Uncertainty around the period before, during, and after 2014 elections produced lack of focus by relevant government entities. Although the AF was approved in April of 2013, enrollment of beneficiaries in the safety net component was not completed until the end of 2015, and the first round of transfer took place only in 2016. As a result, the total number of transfers was smaller than planned. Similarly, activities under the MDPP were delayed due to resistance of Pension Department staff to transitioning from the old legacy system as well as ambiguities in the roles and responsibilities of various stakeholders. As a result, key technology procurements and hiring of staff were significantly delayed. Under new leadership in the relevant units at MoLSAMD in August 2017, the situation improved, and implementation of MDPP activities was ongoing after project closure. High staff turnover was a continual challenge, with three project managers in one two-year period, extended staff vacancies at several key positions, and a relationship between MoLSAMD management and the Project Coordination Unit (PCU) staff characterized in the ICR as "difficult" (ICR, p. 18). Procurement-related challenges created major bottlenecks during most of the project's implementation period (see Section 10b). Due to these shortcomings, efficiency is rated Modest. Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated High due to specific alignment with country conditions, government strategy, and Bank strategy. Achievement of both objectives is rated Substantial. The project significantly improved administration and performance of the pension system serving public sector retirees, and it made progress (on which the government is currently building) in strengthening the pension system serving Martyrs and Disabled. Two pilot safety net programs, with associated administrative systems, were implemented and assessed. Efficiency, however, is rated Modest due to delays that impacted performance. These ratings indicate moderate shortcomings in the project's preparation and implementation, and Outcome is therefore rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The ICR (p. 17) noted the project's impact on training of staff and strengthening of community institutions, particularly hands-on skills in coordinating community mobilization processes, providing feedback to communities, accepting and addressing complaints from marginalized community members, and implementation monitoring. Similar skills development and business opportunities were reported as provided to commercial banks and mobile money transfer firms. However, the ICR (p. 11) also highighted important limitations in institutional strengthening, exemplified by the MoLSAMD's inability to assume project coordination functions from the PCU. Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) Government commitment to reform is mixed. PCU staff are continuing to support the MoLSAMD, with salaries financed by the budget, to continue the pension reforms and build ministry capacity. In April 2018, the Cabinet approved further steps to enhance the pension system, including parametric reform measures for the civil service and security forces, the establishment of a public pension fund for contributory schemes, and the introduction of a voluntary pension scheme for formal private sector workers. The project team indicated that recent restructuring of the MoLSAMD, involving the creation of a National Authority for Supporting People with Disability and Families of Martyrs, may expedite rollout of the MDPP-MIS. However, according to the ICR (pp. 12, 23), the project's safety net component will not be scaled up because of a lack of political support and shifting government priorities. The Bank continues to engage relevant government entities in policy dialogue on how best to use the systems built and lessons learned from the pilots. 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared, as was the case for all projects in Afghanistan at the time, under OP/BP 8.0, Rapid Response to Crises and Emergencies. Risk assessment acknowledged security, political, and operational risks, highlighting risk due to low MoLSAMD capacity, leadership changes at the 2009 elections, possible waning of political commitment to the safety net program as the impact of the global food crisis recedes, and especially weak procurement capacity. Mitigation measures were well conceived and thorough. The project drew from prior Bank analytical work on pensions and safety nets, specifically a June 2008 report "Public Sector Pension Scheme: From Crisis Management to Comprehensive Reform Strategy," as well as experiences and lessons learned in developing social insurance and assistance schemes in other parts of the world. One of those key lessons was the importance of contracting with third-party implementing partner organizations to compensate for capacity gaps at government ministries; in this case, such arrangements facilitated implementation in remote, inaccessible, and insecure areas. The safety net pilot was carefully designed to offer practical lessons and recommendations prior to scale-up. There were several moderate shortcomings. Project costs were underestimated, later requiring AF. The plan to shift day-to-day project management from the PCU to the MoLSAMD overestimated the latter's capacity; ultimately, this shift was not made, and the PCU retained responsibility for project implementation and management. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project was implemented under highly challenging conditions, including unpredictable and risky security, poor accessibility, weak capacity, and high staff turnover in government entities. Within this challenging environment, the Bank team provided close support to the PCU and relevant government Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) departments. Supervision missions, with teams containing staff and consultants with relevant expertise and experience, were regular, and aides-memoire candidly documented progress, bottlenecks, and agreed actions. When procurement bottlenecks slowed implementation, the Bank exerted a focused engagement with the MoLSAMD procurement department head and senior staff, producing integration of procurement support functions into the ministry's procurement department and allowing for more timely action moving forward. The Bank team led three evaluations whose recommendations shaped the course of implementation. The team continues to work with government counterparts to continue pension reforms initiated under the project. It was a minor shortcoming that, according to the ICR, follow- up on post procurement reviews and recommendations was not complete. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The EPP outlined sets of process and outcome indicators under both objectives. Institutional responsibility for data monitoring and collection was well specified, and most indicators had baselines and year-to-year targets. For pensions, data were to be drawn from the new MIS, envisioned to produce timely reports and track achievement of performance indicators, as well as a beneficiary survey. For safety nets, process and impact information were to be drawn from qualitative and quantitative surveys undertaken by the PCU and the MoLSAMD units involved with implementation of the pilot. Overall, M&E was to be carried out by the PCU, with assistance from the M&E Unit in the MoLSAMD's Grant Management Unit. b. M&E Implementation The PCU was staffed with two M&E specialists who managed day-to-day data collection and reporting. MIS systems developed under the project tracked progress under both the pensions and safety net components. For the safety net activities, implementing partner organizations collected data from targeted communities, compiled that data, and entered information into the MIS. Cash transfer data were collected and compiled by the banks and mobile money transfer agencies contracted by the project. The project also contracted an independent firm to conduct spot-checks to monitor the accuracy and timelines of cash Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) transfers. Three evaluations -- of the safety net implementation process, targeting accuracy, and impact -- were conducted (ICR, p. 20). c. M&E Utilization Evaluations financed by the project were used to generate evidence on the effectiveness of project interventions. The findings were used to adjust the targeting approach for the second phase of the safety nets pilot (see Section 4), and for overall performance assessment. However, regular progress reports submitted by the PCU were often late and of poor quality, rendering their utility minimal (ICR, p. 20). M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was originally classified as Environmental Assessment category C and did not trigger any safeguard policies. No resettlement or land acquisition was planned, and the project was anticipated to have no impact on the environment. The ICR (p. 21) stated that the project triggered OP/BP 4.01 Environmental Assessment, and OP/BP 4.12 Involuntary Resettlement, but that was not reflected in the EPP. At the 2013 AF, the environment category was changed from C to B, as it became clear that the rehabilitation of offices and facilities for the Pension Department was going to be larger in scale than originally expected, and as civil works were expanded for the Martyrs and Disabled Pension Department (MDPD). The project developed an Environmental and Social Management Framework to lay out the specific requirements, processes, and responsibilities for ensuring that activities would not have negative impacts on the environment or people. The Bank's environmental safeguards team conducted meetings, field visits, and assessments of overall safeguards compliance at project closing and found several health and safety issues at the rehabilitated building for the MDPD: a gas cooking stove in the customer waiting area, poor air circulation due to malfunctioning air conditioners, and poorly maintained toilets posing a potential health hazard. The government agreed to address these issues (ICR, p. 21). The last Implementation Status Report (June 2017) provided an overall safeguards rating of Moderately Unsatisfactory. The project team later confirmed that all safety and hygiene issues had been addressed. b. Fiduciary Compliance Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) Financial management (FM) was handled by the PCU under supervision of the MoLSAMD finance directorate. The project maintained agreed FM arrangements over most of the project implementation period, except for brief periods where staff turnover created challenges. The project's FM was rated Moderately Satisfactory in Implementation Status Reports throughout the project's lifetime, with shortcomings arising from low budget credibility, high turnover of FM staff, and weak fixed asset management. There were no ineligible expenditures or overdue audit reports over the project's lifetime (ICR, p. 21). Procurement was conducted by the PCU under supervision and guidance from the MoLSAMD. The PCU was staffed with only one procurement officer, an arrangement that was "never adequate both in numbers and capability" (ICR, p. 21). Overall, according to the ICR, the project experienced "poor procurement performance," creating major bottlenecks during most of the project implementation period. Post procurement reviews identified weaknesses including faulty processes, lack of proper documentation, and weak contract management. These shortcomings contributed to implementation delays for critical activities and reduced project efficiency. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Moderately Bank Performance Moderately Satisfactory --- Satisfactory Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR (pp. 23-24) offered several lessons, including: Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review AF: Pension Admin and Safety Net (P113421) Government ownership and commitment is key to implementation of reforms in challenging political and security environments. In this case, implementation of the pension component covering survivors of martyrs and disabled persons suffered because of the limited involvement and leadership of relevant government entities. Third-party arrangements can bridge capacity gaps and facilitate effective implementation. In this case, the delivery of cash through mobile banking services in provinces where conventional banking services were not available was central to implementation of safety net activities. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, candid, and evidence based. It followed established guidelines. It provided a coherent implementation narrative aligned with a straightforward assessment of achievements, shortcomings, and the factors that produced those shortcomings, and the quality of evidence provided was consistently adequate. The constraints of the political and security environments were well explained. The lessons, while insightful, were few in number; additional useful observations and learning could have been gleaned from this project's experience. The ICR incorrectly recorded the project's safeguards at appraisal. a. Quality of ICR Rating Substantial Page 15 of 15