Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) Report Number: ICRR0024092 1. Project Data Project ID Project Name P127328 ML Emergency Safety Nets project Country Practice Area(Lead) Mali Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-D3480,IDA-H8350,TF-A2384,TF- 30-Jun-2018 106,427,341.11 A7655 Bank Approval Date Closing Date (Actual) 30-Apr-2013 30-Jun-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 70,000,000.00 12,400,000.00 Revised Commitment 114,061,555.47 8,847,117.64 Actual 106,427,341.11 8,847,117.64 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Salim J. Habayeb Susan Ann Caceres IEGHC (Unit 2) P157892_TBL Project ID Project Name P157892 ML-Safety Nets Project (Jigisemejiri) ( P157892 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 0 Bank Approval Date Closing Date (Actual) Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) 01-Sep-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 0.00 Revised Commitment 0.00 0.00 Actual 0.00 0.00 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (p. 5) and the Emergency Project Paper (EPP, p. 6), the project's objectives were "to provide targeted cash transfers to the poor and food insecure households and to establish building blocks for a national safety net system in Mali." At an Additional Financing (AF) approved in September 2016, the objectives were revised to: "to increase access to targeted cash transfers for poor and vulnerable households and build an adaptive national safety net system in the Recipient's territory" (Project Paper, p. 11). Some outcome targets were increased and one reduced at the 2016 AF and at another AF approved in July 2018. As the ICR (p. 14) argued, these revisions of the objectives and outcome targets took place in line with expansion of the project's resources and scope. "Food insecure" was replaced with "vulnerable" because the latter was considered to be more measurable. "Adaptive" was added to reflect the objective of building a social safety net system that is adaptive to shocks. With increased ambition and widening of the project's scope, a split rating is not performed, and the project is assessed here on the basis of its revised objectives and outcome targets. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 01-Sep-2016 c. Will a split evaluation be undertaken? No d. Components Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) The project was approved with three components: 1. Cash transfer program and accompanying measures (appraisal: US$56.5 million; actual; US$86.2 million) were to mitigate current poverty and vulnerability in selected areas by smoothing and increasing consumption, mainly food consumption of households, through targeted and regular cash transfers, and in the medium and long term, to improve the human capital of children by creating incentives for poor households to invest in their children's health and education through the combination of cash transfers, accompanying measures, and delivery of a piloted preventive nutrition package. Quarterly cash transfers were to reach about a quarter of the chronically food-poor population in Mali for three years, delivered to the head of the household (based on recent evidence showing that the head of household was responsible for provision of food for all household members). The transfers were to be made by qualified payment agencies, including the post office, private or public banks, microfinance institutions, or mobile telephone companies. Accompanying measures related to health and education (information sessions and campaigns to promote good practices in health, education, nutrition, family planning, and investment/savings planning) were to generate incentives for households to invest in human capital accumulation that can reduce the intergenerational transmission of poverty. Cash transfer beneficiaries were expected to attend these activities, but failure to attend was not to result in penalty or suspension. In about ten percent of villages selected for the cash transfer program, an additional pilot preventive care nutritional package was expected to reach all children ages 0-59 months as well as all pregnant women. Household eligibility for the cash transfer program was to be based on a combination of geographic and community-based targeting mechanisms, focusing initially on the southern regions of Sikasso, Segou, Mopti, Koulikoro, and Kayes and the district of Bamako, where 90 percent of the population was concentrated and where a number of households were sheltering internally displaced persons, with displaced people from the North with children at risk also eligible. The 2016 AF added activities on labor-intensive public works (LIPW), to provide 60 days of work over two years to populations affected by recurrent shocks on community works, including the rehabilitation of basic services infrastructure and land management; and an income-generating activity (IGA) program, targeted at cash transfer beneficiaries with a particular focus on woman-headed households, consisting of an initial community awareness-raising workshop followed by the creation of savings groups, the development of business plans, and the provision of subsidies and coaching. 2. Establishment of a basic safety net system (appraisal: US$6.8 million; actual; US$5.82 million) was to develop a system to support long-term safety net interventions in the country. Activities were to include the development of a management information system (MIS), including modules to track payments and compliance with accompanying measures and the nutritional package, prepare reporting information, and set up a grievance reporting system; setting up of a unified registry of potential beneficiaries; setting up of an information, education, and communications campaign; establishment of monitoring and evaluation (M&E) procedures; delivery of training programs at the central and regional levels; technical assistance to design a cash-for-work program; and additional studies on the targeting mechanism, payment mechanism, and accompanying measures. 3. Project management (appraisal: US$7.9 million; actual; US$10.51 million) was to ensure that the project implementation unit (Safety Nets Technical Management Unit, UTGFS, under the Ministry of Economy and Finance and Budget, MEFB) had the capacity and resources to carry out the project in conformity with the financing agreement, Project Implementation Manual, and other operational documents. It was to finance UTGFS staff (non-civil servant) salaries, including operational costs, at the regional level; equipment and Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) operating costs for UTGFS directly linked to project management; regular internal audits and annual external audits; and training of UTGFS personnel at both the central and regional levels. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was initially financed by a US$70 million grant from the International Development Association (IDA), processed as an Emergency Recovery Loan; AF of US$10 million from the Sahel Adaptive Social Protection Program Multi-Donor Trust Fund (SASPP-MDTF), approved in September 2016; and a second AF of US$52 million in the form of an IDA grant and US$2.4 million from the SASPP-MDTF, approved in July 2018. Total planned financing, inclusive of both AFs, was therefore US$134.4 million. At a February 2023 restructuring, US$13.5 million was cancelled. The project actually disbursed US$106.4 million, with differences due to exchange rate fluctuations. The government pre-financed preparatory activities in the amount of approximately US$320,000 to support staff salary, duties, and operational costs of about US$1.2 million. The project was approved on April 30, 2013, became effective on August 17, 2013, underwent a mid-term review in June of 2016, and closed on June 30, 2023, five years later than its original closing date of June 30, 2018. It was restructured and/or received AF five times:  September 1, 2016: Change in statement of project objectives, first AF, introduction of new subcomponents on LIPW and IGA (triggering a change in the project's safeguards and environmental assessment category), expansion of number of outcome indicators and ambition of some outcome targets, and extension of the closing date by 18 months, to December 31, 2019.  July 5, 2018: Second AF, expansion of the project's geographic scope and number of beneficiaries, shift in emphasis of the accompanying measures to emphasize women's participation, removal of some outcome indicators and addition of others, increase in ambition of some outcome targets, introduction of a Contingency Emergency Response Component that was ultimately not triggered, and extension of the closing date of the original grant to December 30, 2022.  September 26, 2019: Extension of the closing dates of the SASPP-MTDF grants.  June 23, 2020: Extension of the closing date of one of the SASPP-MTDF grants.  June 28, 2023: Cancellation of US$13.5 million, and extension of the closing date of the original grant to June 30, 2023. 3. Relevance of Objectives Rationale The project's objectives were highly relevant to country conditions. At the time of project appraisal, the political and security situation in Mali had been deteriorating for over a year. In early 2012, the northern regions of the country were occupied by armed separatist and jihad groups, and a military coup in March overthrew the elected government and prevented the organization of elections that had been scheduled for Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) the following month. Order was gradually restored through the formation of a Transition Government of National Unity, the launch of United Nations Security Council-based military operations to support the Malian army in the North, and the adoption by Parliament of a political road map for moving the country forward and restoring elected governance. The Bank resumed activities in the country in October of 2012. These political events exacerbated the population's existing vulnerability to shocks and highlighted the desperate living conditions of a majority of the country's population. Mali is landlocked, rural, has a narrow natural resource base, is vulnerable to adverse climate conditions (especially drought), and suffers from high levels of poverty (43.6 percent in 2010) and chronic food insecurity (more than 25 percent of the population). Following the 2012 political crisis, hundreds of thousands of Malians fled to southern parts of the country and to neighboring countries, heightening challenges with food insecurity. The instability put tremendous pressure on basic service delivery and increased already strong demand for social safety net programs that could both protect the population in the short term and promote investments to help families to be better protected from future shocks. Existing safety net programs, managed in an uncoordinated manner by various ministries, were poorly targeted and provided ad hoc emergency assistance rather than systematic support. There was no comprehensive social protection strategy, appropriate institutional setup, or inter-ministerial coordination. The objectives were also highly relevant to government strategy. The Growth and Poverty Reduction Strategy Framework III (2012-2017) prioritized reduction of chronic poverty and food insecurity and improvement of the living conditions of the poorest populations. It featured three strategic areas: (i) promotion of accelerated, diversified, and sustainable growth, and creation of jobs and activities that generate income; (ii) strengthening of the foundations for long-term development and equitable access to quality social services; and (iii) support for institutional development and governance. Under the second strategic area, the government explicitly aimed to develop integrated programs that would share good and efficient tools and procedures under an effective safety net system, with a particular focus on cash transfer initiatives. MEFB set up a multi-sector technical team in 2011 to begin preparing this project, which was initially conceived as investment lending but, following the events of early 2012, was transformed into this Emergency Recovery Loan. The government named the project "Jigiséméjiri," which means "Tree of Hope" in the national Bambara language. The objectives remained aligned with the government's 2019-2023 Strategic Framework for Economic Recovery and Sustainable Development, which promoted social protection to prevent vulnerability and social exclusion, and which called for the development of methods to improve the effective and efficient implementation of social protection initiatives benefiting the most vulnerable groups. The objectives were also highly aligned with Bank strategy. The Bank had prepared a Social Safety Nets Review in 2011 showing that the scope and coverage of the existing system was limited, and that existing interventions (mostly food transfers) were too small in scale and poorly targeted. An OP 7.30 assessment mission conducted in June of 2012 confirmed that the project fit emergency criteria and was eligible for processing under OP/BP 8.0: it established and/or preserved human, institutional, and/or social capital, including economic reintegration of vulnerable groups; it assisted with the crucial initial stages of building capacity for longer-term reconstruction, disaster management, and risk reduction; and it supported measures to mitigate or avert the potential effects of an imminent emergency/crisis in a country at high risk. The objectives were relevant to the Country Assistance Strategy (CAS) in effect at appraisal (FY08-11), which contained an objective to strengthen public sector performance through improved delivery of basic services and protection of the vulnerable. It responded to the April 2011 CAS Progress Report that recommended a two-stage approach toward social protection, assisting the poorest and most vulnerable to cope with shocks in the short term while moving toward development of an efficient safety net system in the Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) medium and longer term. The objectives remained relevant to the next Country Partnership Framework (CPF, FY16-20), which contained a Focus Area on building resilience and an objective to strengthen safety nets and improve risk management mechanisms for the poor and vulnerable, and to the findings of the 2022 Systematic Country Diagnostic, which stressed the importance of social safety nets to help households avoid negative coping mechanisms in response to shocks. A new CPF covering the period 2024-2026 is currently under development. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase access to targeted cash transfers for poor and vulnerable households Rationale The theory of change for this objective held that provision of direct cash transfers to households, along with accompanying measures in health, nutrition, and education made available to both beneficiary and non- beneficiary households, and augmented by preventive nutrition interventions targeted at pregnant women and children under the age of five years old, would deliver cash transfers to poor and food-insecure households. Targeting would combine geographical and community-based approaches to ensure focus on the poor and those who were food-insecure. This logic was predicated on a series of assumptions: that there would be sustained security and political stability; that qualified staff could be recruited; that project implementation would occur gradually, in order to manage complexity; and that the project's technical management unit and its procedures would be put in place. Outputs The cash transfer and accompanying measures activities were implemented as planned. Quarterly direct cash transfers were made to 62,000 poor households in the five regions of the south, plus the district of Bumako, for a period of 36 months in 2014-2018. Additional financing enabled the implementation of a second phase of cash transfers during 2020-2023, covering 25,000 poor households in the southern and Gao regions. The transfer amount was increased in this second phase to compensate for inflation. The project team confirmed that targeting was implemented as planned. Geographic targeting focused on the commune, with the poorest communes selected, and all villages within a poor commune automatically included. Household targeting was implemented by "commune committees" and "village committees," trained by the project in targeting and provided with criteria by which to identify beneficiaries. Village committees established initial lists of potential beneficiaries. Members of commune committees then traveled to each Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) village to validate the lists. The Project Implementation Unit also undertook socio-surveys to ensure that poor households were selected. The accompanying measures consisted of a twelve-session package of program information and community campaigns, delivered by non-governmental organizations (NGOs) to raise awareness of the importance of nutrition, hygiene, and education, to both beneficiary and non-beneficiary households, for a period of 36 months. 89.6 percent of beneficiaries complied with the requirement of participating in accompanying measures, exceeding the target of 75 percent. LIPW activities were introduced in 2017, offering 60 workdays to participating beneficiaries. IGA activities were piloted in 2017-2018 and scaled up under the project's second AF. 70.12 percent of program beneficiaries reported that they were aware of IGA objectives and entitlements, not reaching the target of 80 percent awareness. The ICR (Annex 1) explained that the 70.12 percent achievement was recorded in December of 2019, before the bulk of the project's IGA support was implemented, accounting for the shortfall. No survey was conducted to reflect any increase in awareness during the project's latter years. 73.52 percent of program beneficiaries reported that they were aware of LIPW program objectives and entitlements, not reaching the target of 85 percent awareness. The ICR (Annex 1) explained the shortfall as due to difficulty finding payment agencies that could meet strict operational security requirements in the North. 556,780 workdays were created through the LIPW program, not reaching the target of 1.62 million workdays. Only 105 of 540 planned microprojects were finalized, mainly in the southern regions. The ICR (Annex 1) explained that LIPW activities were interrupted in the central and northern regions due to the security situation, accounting for the shortfall. The percentage of beneficiaries receiving regular payments within the agreed time frame for LIPW activities, as specified in the Operations Manual, was 36 percent, not reaching the target of 80 percent of beneficiaries, again because of the lack of payment agencies in conflict- affected areas. Only 18.1 percent of beneficiary households reported that their livelihoods had benefited from LIPW-created assets, not reaching the target of 75 percent of households. Nutritional packages were provided to 105,000 beneficiaries through a contract agreement with the World Food Program. 70,963 children benefited from nutritional packages in households receiving cash transfers, not reaching the target of 102,000 households. The partial achievement of this target was due to the inclusion of pregnant women as recipients of nutritional packages (because of community demand); 34,037 pregnant women received the packages. Had those women been included in the indicator, the target of 102,000 beneficiaries would have been exceeded (70,963 children + 34,037 pregnant women = 105,000 beneficiaries). 32,000 households received one-time cash transfers to cope with the impact of the COVID-19 crisis. 1.3 million potential beneficiary households were registered and identified, exceeding the original target of 122,000 and revised target of 110,000 households. Outcomes Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) 105,491 households (576,907 individuals) had access to cash transfers established by the project, exceeding the original target of 62,000 and revised target of 100,000 households (550,000 individuals). Of those, 103,541 households received direct cash transfers paid over a period of 36 months, exceeding the original target of 62,000 and revised target of 87,000 households. 14,668 households benefited from LIPW programs, not reaching the target of 27,000 households. 27,997 households benefited from IGA programs, not reaching the target of 32,000 households. The shortfall in reaching the targeted number of intended beneficiary households with LIPW and IGA programs was due to the 2020-2021 political crisis, the insecurity situation in the center and North of the country, and issues with contractors and payment agencies. 100,000 children ages 0-5 benefited from cash transfers, not reaching the original target of 180,000 children, but meeting the revised target of 100,000 children. 7,270 households with children under three years old benefited from counseling services provided by a Case Management System (CMS), not reaching the target of 10,000 households. Of those 7,270 households, 81 percent benefited from at least three sessions of counseling services provided by CMS, exceeding the target of 75 percent of households with children under three years old. There were 576,907 direct project beneficiaries, of whom 47.03 percent were female, exceeding both the original target of 400,000 beneficiaries (40 percent female) and the revised target of 550,000 beneficiaries (40 percent female). The cash transfers had a positive impact on poverty. The percentage of beneficiary households that were below the poverty line decreased from 88 percent at the beginning of the cash transfer program to 66 percent at project closing, exceeding the target of a reduction to 75 percent of households. The project's impact evaluation, whose methodology was well described in the ICR, found a slight decrease in relative poverty (4.6 percent) and the poverty gap (8 percent) in the treatment group compared to the control (late entry) group at project midline (ICR, p. 15, and see Section 9a). The ICR (p. 15) also reported that the project's impact evaluation also found improvements in households' food security, savings, and asset base, but specific data were not provided. The ICR (p. 36) indicated that the project's designation of heads of households, mostly men, as cash transfer recipients limited the project's impact on women's empowerment and on health and child nutrition knowledge. Achievement of this objective is rated Substantial, but barely so. Although targets were exceeded for increasing access to cash transfers, there were shortfalls in reaching the targeted number of intended beneficiary households with LIPW and IGA programs. Rating Substantial OBJECTIVE 2 Objective Build an adaptive national safety net system Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) Rationale The theory of change for this objective held that development of a management information system and social registry; the delivery of information, education, and communication activities about the registry; and the establishment of monitoring and evaluation procedures, all supported by relevant technical assistance and studies, would lead to the establishment of the basic building blocks for a national safety net system. Outputs A Unified Social Registry (USR) was designed and established. It is housed in the Ministry of Health and Social Development and run by the UTGFS, which controls funds and coordinates activities of development partners. Key instruments include a questionnaire for collecting household data, a host server, a web portal, a web application, and a mobile application. A cash transfer delivery mechanism was developed. Information, education, and communication activities were carried out related to the USR; 733 communal committees had awareness-raising sessions and were trained in targeting. Civil servants in 146 cercles (administrative subdivisions below the regional level) also had awareness-raising sessions. Training was conducted in 300 communal committees and 15,000 village committees on monitoring project activities. In 2021, mobile money cash transfers were piloted in the region of Mopti in response to periods of movement restrictions due to COVID-19 and insecurity. These payments reached 3,000 households. A qualitative evaluation of the pilot found that mobile payments gave recipients flexibility and reduced waiting and travel time, but that network issues in remote sites presented challenges. Outcomes 1.3 million households were registered in the USR, exceeding the target of 80,000 households. The USR became the platform for the government and development partners to deliver other safety net programs at national scale. The registry includes households registered under the project, socioeconomic data for the government's emergency cash transfer program in response to COVID-19 and other government safety net initiatives, data supporting interventions by several NGOs, and over 220,000 households potentially eligible for the National Medical Assistance Agency scheme. The system demonstrated adaptiveness by providing emergency transfers to over 31,000 households during the 2018 lean season, and through the 2021-2022 COVID-19 emergency social program targeted at poor households. Outcome targets were exceeded. However, reported outcomes do not completely reflect the stated objective, as they focus only on the social registry and not also on other key building blocks of an adaptive national safety net system. Achievement of the objective is therefore rated Substantial. Rating Substantial Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Achievement of the objective to increase access to cash transfers for poor and vulnerable households is rated Substantial, but barely so. Although targets were exceeded for increasing access to cash transfers, there were shortfalls in reaching the targeted number of intended beneficiary households with LIPW and IGA programs. Achievement of the objective to build an adaptive national safety net system is rated Substantial. The Unified Social Registry is now the platform for the government and development partners to deliver other safety net programs at national scale, and it has demonstrated flexibility and adaptability in response to crisis situations. Overall efficacy is therefore rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency This project's approach differed from that of other agencies, such as the World Food Program and some NGOs, that also initiated cash transfer programs in 2018; their emphasis was on higher-value payments for shorter durations, often limited to the three-month lean period. This project, by contrast, intended to ensure consumption smoothing and build resilience over a longer, 36-month time frame. The economic analysis at appraisal (EPP, Annex 10) showed that the anticipated level of cash transfer (on average, the equivalent of US$20/month) was commensurate with international experience, likely to have a significant impact on the level and quality of household food consumption, and likely to reduce the food poverty headcount by 4.6 percent, the food poverty gap by 5.8 percent, and the severity of poverty in the targeted regions by over 7 percent. These estimates were high, relative to other experiments with cash transfers in developing countries, and the EPP (p. 17) acknowledged that ex post assessments might differ. Transfers at the anticipated level to all poor households (43.5 percent of the population) over a three-year period were equivalent to one percent of 2013 gross domestic product (GDP), sustainable given the prevailing level of government expenditure on safety net programs. The project was expected to cover about one-third of chronically food-poor households for a total cost of about 0.14 percent of 2013 GDP. The ICR's analysis of economic efficiency (pp. 20-21) focused on the tradeoffs between giving a higher transfer value to fewer beneficiaries or a lower transfer value to a larger number of beneficiaries. An ex post simulation found that giving less money to more households would have lifted a slightly larger number of people above the food poverty line, but the ICR argued that the project's approach -- more money to fewer beneficiaries -- was more effective in allowing the targeted poor to meet their basic needs, accumulate assets, invest in businesses, and therefore enhance resilience in the longer term. The ICR also demonstrated that the administrative costs of the USR were in line with comparators, which tend to have cost-transfer ratios under 2 percent of benefits. The cost-transfer ratio of targeting through the project-constructed social registry was 2.5 percent including costs associated with the initial setup of the system, and 1.15 percent excluding those costs. Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) The project's implementation efficiency experienced significant shortcomings. The UTGFS was set up and functioned smoothly, benefiting from specialized and experienced staff. Cash transfers began only six months after project effectiveness. The project's gradual implementation approach allowed an initial focus on districts less affected by security challenges, with a scale-up to other areas once delivery systems had been fully tested. Vertical coordination between central and decentralized structures was strong. However, implementation of the activities supported by the project's AF was "seriously impacted by compound effects deriving from the political and security crisis" and the COVID-19 pandemic in 2020-2022 (ICR, p. 21). The sociopolitical environment led the Bank to repeatedly suspend disbursements to the entire country portfolio, severely disrupting project implementation. The suspensions in August 2020, May 2021, and January 2022 cumulatively totaled 18 months. From 2020 onwards, escalation of security challenges, particularly in the northern part of the country, limited access to some project areas. Some field and supervision activities, including LIPW, IGA, accompanying measures associated with the direct cash transfers, training events, and data collection, were suspended during the COVID-19 pandemic. In addition, there were procurement shortcomings arising from inadequate staffing to handle the increased scope of work once the LIPW and IGA activities were added (see Section 10b), issues that were "compounded by lack of internal coordination" (ICR, p. 26). Despite multiple, focused efforts by the Bank to compensate for these disruptions, including the adoption of a gradual implementation approach (ICR, p. 25), strengthened security measures (ICR, p. 27), and the introduction of framework contracts (ICR, p. 36), it remained the case that “insecurity upsurge, COVID-19 restrictions, and the recurring political crisis seriously affected the volume of project activities” (ICR, p. 26). On balance, given the disruptions caused by security challenges and the COVID-19 pandemic, the repeated suspensions of disbursements negatively affecting project delivery, and shortcomings with procurement, project efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives were highly relevant to country context, government strategy, and Bank strategy. Efficacy was overall Substantial. Achievement of the objective to increase access to cash transfers for poor and vulnerable households is rated Substantial, but barely so. Although targets were exceeded for increasing Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) access to cash transfers, there were shortfalls in reaching the targeted number of intended beneficiary households with LIPW and IGA programs. Achievement of the objective to build an adaptive national safety net system is rated Substantial. The Unified Social Registry is now the platform for the government and development partners to deliver other safety net programs at national scale, and it has demonstrated flexibility and adaptability in response to crisis situations. Project efficiency is rated Modest, given the disruptions caused by security challenges and the COVID-19 pandemic, repeated suspensions of disbursements, and significant shortcomings with procurement that caused delays and shortfalls in implementing planned activities. Taken together, these ratings indicate that there were moderate shortcomings with the project's preparation, implementation, and achievement, resulting in an Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome The project built and institutionalized a social registry, enabling the government sustainably to scale up safety net programs both under routine circumstances and in response to shocks. The ICR (p. 34) noted, however, that further steps are required, as the main management structures supporting the USR are entirely dependent on World Bank financing and technical support. The government has limited financial and technical capacity to take over the system. Even more significant risks stem from persistent conflict and insecurity. Jihadist attacks and ongoing violence continue to force people to flee affected areas, exacerbating poverty and vulnerability. 8. Assessment of Bank Performance a. Quality-at-Entry The project was well aligned with government and Bank strategies. Its design was based on extensive analytical work and policy dialogue. In the 2-3 years preceding the project, the government, the Bank, and other development partners had been engaged in an extensive dialogue on safety nets. These interactions, as well as experience with similar projects in other African countries, had yielded the following lessons that were incorporated into project design: the need for a strong, multisectoral institutional framework for social protection policy; the importance of improved targeting to vulnerable populations (through a combination of geographic and individual targeting) and strengthening of government capacity to manage and monitor programs; the need to implement regular cash transfers over a relatively long period of time (24 to 36 months) to sustainably strengthen household food security; the need for a rigorous MIS to inform strategic decision making; the potential for accompanying measures to improve the health status of participants and, in particular, the nutritional status of children, with spillover effects on non-participating households in the same communities; the effectiveness of civil society-based delivery mechanisms in implementing a project and fostering capacity building, particularly at the local level; and the need for coordination mechanisms among donors. Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) Institutional arrangements were well specified at appraisal. In addition to specifying UTGFS as the implementing agency, the Prime Minister's office designated a National Steering Committee to provide policy orientation and supervise implementation. UTGFS was mandated as the lead agency for operationalizing social safety nets in the country, with existing government entities at the regional, district, and commune levels providing program staffing under UTGFS direction. A Project Implementation Manual defined implementation mechanisms. The project's overall risk rating was High, primarily due to very high security and political risks that were compounded by risks related to implementation capacity, technical complexity, budget, and governance. An Operational Risk Assessment Framework (EPP, Annex 4) identified a full range of mitigation measures, including reliance on a strong technical unit (UTGFS) to ensure strict application of Bank rules and standards, recruitment of technically qualified staff, a clear operational manual, a robust MIS, a plan to start with a relatively small number of beneficiaries and ramp up the project's coverage after procedures and capacity had been tested, and training and technical assistance. The EPP (p. 26) acknowledged that there was little the project could do to mitigate political and security risks. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank responded effectively to most challenges during project implementation, including the government's request for additional financing. Missions were organized twice per year and systematically supported the UTGFS. The Bank team actively collaborated with partners, particularly in the USR technical working group, facilitating smooth institutionalization of the USR. Key risks and mitigation measures identified at the design stage had to be revised toward the end of the project (2020-2022). The increased number of security incidents, some of which involved financial institutions, led the project to intensify security directives and introduce bank guarantees for payment agencies. The Bank team showed flexibility in response to the changing environment, for example, negotiating higher fees for payment agencies to work in insecure areas to support the additional security measures required to reach those areas. Overall, meetings with the UTGFS became more frequent after the disbursement suspensions, political and security crises, and COVID-19 restrictions in 2020-2021. Process evaluations were commissioned to assess the status of suspended activities and determine strategies for resumption. The team made efforts to address procurement issues through capacity building, tight supervision, and revision of terms of reference and documentation. Procurement capacity nevertheless remained low. There were some shortcomings in supervision. Bank support was insufficient to address capacity gaps that arose during the project's latter stages, particularly in procurement. The ICR (p. 33) noted that UTGFS staff were affected by high task team leader turnover and "consequent discontinuity in the management approach and supporting style." The ICR (p. 33) further noted that the Bank team "could have discussed bolder project changes with the government to face the extremely challenging context" that affected implementation of activities later in the project's lifetime. Finally, limited steps were taken toward ensuring sustainability of project outcomes, with the management structures of the safety net program and social Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) registry still financially and technically dependent on World Bank resources (ICR, p. 34). Within the extremely challenging country and implementation context, these shortcomings are considered minor. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project's theory of change, objectives, and results framework were clear and straightforward. The MIS developed under the project was to support data collection, processing, and information management across all levels (UTGFS and regional), to facilitate tracking of progress and measurement of results. Data capture was paper-based at project launch, but a digital system was to be established and implemented early in the project period. The M&E specialist of UTGFS was responsible for overall M&E. M&E support was to be provided under the project's second component, which contained a dedicated M&E subcomponent covering four annual process evaluations, annual spot checks at the village level, one full impact evaluation with two rounds of data collection (carried out by a recognized international firm), and annual independent audits of the M&E system. The results of these assessments were to inform adjustment of program settings and targeting and of the information, education, and communication strategy. The results framework contained only one outcome indicator (number of households registered in the USR) to measure achievement of the second objective. The ICR (pp. 27-28) points out that an additional indicator measuring usage of the USR by other programs would have strengthened the measurement of outcomes. b. M&E Implementation M&E activities were mostly implemented as planned. Quarterly monitoring reports were submitted in a timely manner. Both the mid-term review and process evaluations identified programmatic issues and recommended improvements. The results framework and indicators were appropriately adjusted at the project's additional financing. Monitoring engaged local authorities, local liaison agents, NGOs, decentralized government structures, and village committees. In 20 of the country's 49 districts, the head of social development services carried out monitoring activities and submitted reports. Local monitoring agents at the district level ensured coordination between field partners and the UTGFS at the central level. The ICR (p. 38) noted that remote monitoring was impossible in the field, as electronic devices could not be carried without endangering local staff and the population, and the use of GPS coordinates was forbidden. The impact evaluation was conducted as planned, on the basis of three surveys: a baseline in 2014-2015, a midline in 2016, and an endline in 2018; however, no impact evaluation was Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) added after the project's additional financings, limiting the assessment of the added IGA and LIPW activities. There were additional moderate M&E implementation shortcomings. The ICR (p. 28) noted that the project's M&E specialist experienced challenges in measuring cumulative indicators, most likely due to the phased approach of project-financed activities. The Bank team noted that there were some discrepancies in the way the M&E specialist consolidated data provided by sector specialists. In addition, while the project MIS was fed regularly with data on direct cash transfers, the modules covering LIPW, IGA, and accompanying measures were sent to the M&E team and relevant project specialists but not entered directly into the MIS. According to the ICR (pp. 28-29), "this lack of integration was reflected in the monitoring and annual reports, which contained exhaustive information on cash transfers, but not on other activities." c. M&E Utilization According to the ICR (p. 29), impact evaluations helped to assess what was working and what was not, and to recommend improvements in the design of project-financed interventions. However, no evidence was produced on the impact and cost-effectiveness of the IGA and LIPW interventions compared to other livelihood support alternatives, limiting the information base from which the project might have adjusted its activities and allocation of resources. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was initially assessed as environmental assessment category "C" and triggered none of the Bank's safeguard policies. With the introduction of AF in 2016, the project received environmental assessment classification "B" and triggered four safeguard policies: Environmental Assessment (OP/BP 4.01), Pest Management (OP 4.09), Involuntary Resettlement (OP/BP 4.12), and Physical Cultural Resources (OP/BP 4.11). Environmental and social safeguards were handled by decentralized technical services and NGOs at the local levels. A collaboration agreement was signed between the project and the National Directorate of Social Protection and Solidarity Economy to ensure coordination. Two safeguards specialists were hired in 2020 and 2021 to train key stakeholders and staff on the environmental and social management plan and to conduct quarterly supervision missions. Safeguards activities were classified as moderately satisfactory from 2019-2022, primarily because of delays in implementation of action plans; it was taking up to six months for the safeguards specialists to screen LIPW and IGA microprojects and for local authorities to approve them. Quarterly safeguards monitoring and progress reports were submitted to the Bank team, with some occasional delays. The project put an incident management plan in place to identify and adjust mitigation, follow-up, and settlement procedures. From 2019 forward, 13 incidents were registered: four related to robbery, five to the resurgence of jihadism, and four stemming from road accidents. All cases were settled by project closure. Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) An attack on three project beneficiaries in Niafunké in April 2019 led to the drafting and implementation of improved security risk management directives. In September 2021, there was a theft of FCFA 21.9 million of project funds from a branch of a participating NGO, which reimbursed the funds. As mitigation, payment agencies were requested to present bank guarantees on funds advanced by the project. One payment agency was not able to satisfy this requirement and, as a result, was unable to continue its activities. The ICR (p. 31) stated that the project complied with all triggered safeguard policies, with expected frameworks and plans put in place, microprojects screened, and incident action plans used to improve mitigation measures. b. Fiduciary Compliance A financial management assessment was conducted at the MEFB at appraisal. This assessment determined that adequate arrangements were in place to ensure that funds were used for intended purposes. However, the project's overall financial management risk was assessed as High due to the complexity of the project, the weak institutional infrastructure for good governance, and the country's limited capacity and experience in managing similar activities (EPP, p. 22). Mitigation measures were well specified (EPP, Annex 5) in a financial management action plan. UTGFS had overall financial management responsibility for the project. Specific measures were incorporated to mitigate risks of fraud and corruption, which were acknowledged as issues in Mali: a thorough information campaign to promote transparency and accountability to local communities; the recruitment of payment agencies on the basis of their qualifications; a rigorous MIS for reporting; a complaint mechanism to address cases of improper cash transfer or beneficiary identification; and conducting of physical verification and independent assessments to ensure the appropriate function of transfer mechanisms. The ICR (p. 31) reported that administrative, accounting, and financial procedures were mostly sound and were regularly updated to reflect the additional financings. Financial reports and statements were submitted in a timely manner and were of acceptable quality. Procurement issues delayed planned external audits in 2019 and 2020; those audits were completed in 2021. Financial statements were certified with qualified opinions due to missing supporting documents on advances for field missions and payment agencies. The financial management team made efforts to improve the recovery of documentation, but delays persisted (ICR, p. 31). The project's financial management rating was Moderately Satisfactory from 2020 through project closing. Procurement risk was assessed as High at appraisal. Mitigation measures (EPP, Annex 6) included preparation of a procurement plan for the project's first year, implementation support from the Bank team, and the hiring of a procurement specialist by UTGFS, which was responsible for project procurement. The ICR (p. 31) reported that the project faced "important delays" in the procurement process due to capacity and coordination issues, a high number of procurements, slow local administrative processes, and the limited capacity of the procurement specialist to adapt to new procedures. Contracts with service providers were initially prepared per district, which led to a high number of contracts with identical specifications. Beginning in 2022, following recommendations of the Bank team, new contracts were encouraged to cover entire regions (framework contracts) in cases where implementing agencies had the necessary capacity. Overall, procurement challenges caused significant delays in the implementation of LIPW and IGA activities. The procurement of tools and materials for LIPW activities was hindered by a lack of proper specifications by implementing partners, requiring the procurement specialist to return lists with requests for clarification, and resulting in situations where contracts with payment agencies and NGOs expired before tools and materials could be delivered. In addition, the initial procurement for the pilot nutrition Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) program produced financial bids that were too high and products that did not meet specifications; eventually, the project partnered with the World Food Program for implementation of the nutrition program. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR (pp. 35-38) produced a series of insightful lessons and recommendations, with highlights including: In low-capacity and conflict-affected settings, safety net project design can benefit from simplicity and gradual scale-up. In this case, the initial direct cash transfers were effectively piloted and expanded, but challenges affected the addition of more complexity with the income- generating and labor-intensive public works elements. The one-year period set to implement each labor-intensive public works cycle led to procurement issues that produced significant delays. Ultimately, the project did not produce evidence of the potential benefits of labor-intensive public works compared to administratively simpler alternatives. Fragility and conflict can challenge the implementation even of well-planned monitoring and accountability mechanisms. In this case, initial M&E activities—including a careful impact evaluation—were carried out as planned, but monitoring activities following additional financing were affected by the insecure context, leaving the project with no assessment of the impact and cost- effectiveness of the income-generating and labor-intensive public works activities. IEG adds the following lesson: Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML Emergency Safety Nets project (P127328) In cases where conflict and crisis are severely impacting key elements of project implementation, significant project restructuring can focus implementation on what is working. In this case, the direct cash transfers were well-targeted to poor and vulnerable households, and the social registry was developed and used effectively even in a highly challenging context. However, persisting with the struggling labor-intensive public works activities represented a "missed opportunity" (ICR, p. 34) to reallocate funds toward other well-performing sub-components that could have better contributed to achievement of project objectives. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was clear, well written, and candid. It followed established guidelines. It was analytically sound and careful to cite sources of data and underlying analysis. However, there were moderate shortcomings. The ICR was unnecessarily repetitive and long. Its lessons were robust and insightful, but in some cases the discussion of lessons contained information and insights that did not emerge directly from the main part of the ICR. The Outcome rating in the Data Sheet did not match the rating in the document's main text. a. Quality of ICR Rating Substantial Page 18 of 18