Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) Report Number: ICRR0023931 1. Project Data Project ID Project Name P150754 Comoros Social Safety Net Project Country Practice Area(Lead) Comoros Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-68380,IDA-D0320,IDA-D5520 30-Jun-2019 29,758,609.59 Bank Approval Date Closing Date (Actual) 19-Mar-2015 30-Jun-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 6,000,000.00 0.00 Revised Commitment 29,983,903.85 0.00 Actual 29,758,609.59 0.00 Prepared by Reviewed by ICR Review Coordinator Group Rima Al-Azar Salim J. Habayeb Susan Ann Caceres IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (Schedule 1, p. 4), and the Project Appraisal Document (PAD, p. 7), the Project Development Objective (PDO) was to "increase poor communities’ access to safety net and nutrition services." This PDO was slightly revised in December 2019 following the first Additional Financing (AF) and became to “increase poor communities’ access to the safety net, nutrition, and community services” (ICR, p. 11). For the purpose of this ICRR, the project’s PDO can be unpacked as follows: Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) Objective 1. Increase poor communities’ access to safety net programs; Objective 2. Increase poor communities’ access to nutrition services; and Objective 3. Increase poor communities’ access to community services. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The project included three essential elements of a social safety net system: Component 1: Establishment of a productive and disaster responsive safety net (Original International Development Association (IDA) financing US$4 million; Total Cost, US$24.93 million). The aim of this component was to provide income support to poor households through two subcomponents:  Subcomponent 1.1: Establishing a productive safety net (Original IDA financing US$3.2 million; Total Cost, US$3.43 million). The productive safety net program was designed to provide targeted poor families with income support in exchange for their participation in productive cash-for-work activities in the poorest villages of Comoros over a three-year period. The subcomponent had a gender sensitive approach to targeting, and aimed to support works in reforestation, water management, terracing, etc.  Subcomponent 1.2: Ensuring an early recovery response to natural disasters (Original IDA financing US$0.8 million; Total Cost, US$15 million). This subcomponent aimed to finance post-disaster recovery response, cash transfers and the rehabilitation and reconstruction of small village infrastructure damaged by natural disasters.  Subcomponent 1.3: (Total Cost, US$6.04 million): This subcomponent was introduced following a restructuring in response to the COVID-19 pandemic (in March 2021). It was designed to provide unconditional cash transfers (UCT) to increase poor communities’ access to safety net, nutrition, and community services. This adjustment also expanded the scope of the project to urban and peri-urban areas affected by the economic impact of the pandemic. Component 2: Improving the nutrition of young children and mothers in poor communities (Original IDA financing US$1 million; Total Cost, US$1.17 million). This component was intended to complement the safety net activities and provide preventive nutrition services in the same villages benefitting from the productive safety net activities. It was designed to deliver a minimum package of nutrition-specific interventions for children under five years of age, with a focus on the ‘first thousand days’ window of Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) opportunity (from conception to two years of age) and promoting better nutrition and reproductive health for mothers. Component 3: Strengthening Safety Net Management, Coordination, and M&E (Original IDA financing US$1 million; Total Cost, US$4.36 million). This component focused on building the institutional capacity for coordination, monitoring and evaluation of the social protection system in Comoros. By establishing a long- term social safety net delivery system, the component aimed to support the development of foundational tools such as a management information system (MIS). Two components were significantly revised during implementation. Component 1 was adjusted and scaled up to support the response to two shocks: Kenneth cyclone and the COVID-19 pandemic. In addition, this component expanded the scope of the project from rural to urban and peri- urban areas affected by the economic impact of the pandemic. Component 3 was also scaled up to ensure proper management of the expanded activities under the project, and to support the foundational elements of a safety net system. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates The SSNP had six restructurings; four of which were ‘Level Two’ restructurings: 1. In February 2019, the first restructuring took place based on the request by the Government of Comoros (GoC) to reinforce the nutrition activities. The restructuring: (i) adjusted targets of selected indicators; (ii) reallocated funding between categories; (iii) documented the institutional changes; and (iv) added an expenditure category to strengthen the capacity building, monitoring and communication of the community-based nutrition services. 2. In June 2019, the second restructuring extended the project closing date. 3. In December 2019, Additional Financing (AF) of US$ 18 million was approved to respond to the effects of the cyclone Kenneth as requested by the GoC (AF1, P171633, Report No: PAD3536). 4. In June 2020, the third restructuring added new activities related to COVID-19 response and therefore reallocated funding between categories and updated the Results Framework. 5. In March 2021, the following changes were introduced: (i) re-allocation of costs to subcomponent 1.2; (ii) extension of the Project’s closing date; and (iii) revision of the Results Framework to ensure full alignment of activities to monitored results (Second Additional Financing for the Social Safety Net Comoros, p. 16). The subsequent AF (AF2, P174866, Report No: PAD4149) was approved in March 2021, in the amount of US$6 million. 6. In November 2022, the fourth restructuring took place and reallocated funding between categories without change in the Results Framework. As a result of these restructurings and to allow for the implementation of activities introduced to respond to the two external shocks, the project was extended twice: (1) from June 30, 2019, until December 31, 2022; and (2) from December 31, 2022, until June 30, 2023. These extensions were introduced respectively to add activities in response to the Kenneth Cyclone and to allow for the preparation and rollout of the activities under the COVID-19 response (ICR, p. 12). Finally, there were important institutional changes during the during the life of the project. The implementation agency changed twice. First, when the Fond d’Appui pour le Développement Communautaire (FADC) was dissolved and the National Agency for Design and Implementation of Projects Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) (Agence Nationale de Conception et d’Exécution des Projets, ANACEP) was established in January 2019. The second time was following AF1 when the project was moved to the Ministry of Health, Solidarity, Social Protection and Gender Promotion (MoH&S) under the Directorate of Solidarity and Social Protection (DoSSP) (ICR, p. 12). 3. Relevance of Objectives Rationale The project was highly relevant since it addressed the development challenges faced by the country, namely: (1) lack of access to essential services, (2) widespread malnutrition among children under 5 years old, (3) narrow scope of social protection and nutrition services due to limited resources, and (4) vulnerability of human development efforts due to external shocks (cyclone, COVID- 19 pandemic, etc.) (PAD, pp. 3-6). The project was fully aligned with the GoC and the Bank’s policies and strategies. Furthermore, it was highly relevant as it addressed the needs of the population and was adaptable since it was able to respond to two emergencies: cyclone Kenneth, in April 2019, and the COVID-19 pandemic, in 2020. The project was aligned with the GoC strategies and sectoral policies. Specifically, it was in line with the government’s Poverty Reduction and Growth Strategy Paper (PRGSP) and its National Social Protection Policy which was adopted in 2016. The parent project was also fully aligned with the World Bank Group’s FY14-FY17 Country Partnership Framework (CPF). Specifically, the project supported the achievement of the CPF’s Pillar I: Increased Public Sector Capacity - which sought to strengthen service delivery and reduce poverty – by providing increased access of the poor population to productive safety net and nutrition services to achieve longer term human development. Furthermore, the closing project remained fully aligned with the World Bank Group’s CPF objectives for FY20–24, in particular, Focus Area 1 which centered on crisis response and building resilience and investment in human capital. Finally, the project contributed to the World Bank’s global strategy of reducing absolute poverty and promoting shared prosperity. At closing, objectives remained fully aligned with CPF’s objectives for FY20–24 (Report No. 145699-KM), in particular with the CPF Focus Area 1 which focuses on crisis response and building resilience and investment in human capital (ICR, p. 13). Finally, and in addition to being fully aligned with national and Bank strategies, the project maintained its relevance through its adaptability to respond to external shocks and to address the emerging needs of the vulnerable population both in rural as well as urban areas. Rating Relevance TBL Rating High Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase poor communities’ access to safety net programs Rationale The Theory of Change (ToC) held that by (i) providing cash transfers (conditional or unconditional), and (ii) training, households would increase their resilience and coping abilities. Furthermore, by implementing cash-for-work interventions for rehabilitating community infrastructure, resilience at the community level would be enhanced. Given that four out of five rural households were classified as poor; whereas in urban areas, only one in four was poor (PAD, p. 1), this objective initially aimed at addressing the widespread poverty, especially rural one, by establishing a productive safety net to provide selected poor households with cash-for-work opportunities (PAD, p. 9). It also included a sub-component which could be triggered in case of natural disasters since the Comoros are exposed to both geophysical disasters (such as volcanic eruptions, earthquakes, and landslides) as well as hydro meteorological hazards (such as tropical storms, torrential rains, floods, rising sea levels, erosion, and tidal waves) (PAD, p. 2). Through this subcomponent, funding was available for the rehabilitation and reconstruction of small village infrastructure damaged by natural disasters and emergency cash-for-work activities for clean-up and repair (PAD, p. 12). Finally, and following the COVID-19 pandemic which caused the economy to shrink by -1.4 percent at the end of 2020 (Second Additional Financing, p. 9), a third sub-component was introduced to provide unconditional cash transfers (UCT) to urban and peri-urban households negatively impacted by the pandemic. The project used Proxy Mean Testing (PMT) combined with vulnerability scoring to target the rural poor and vulnerable urban beneficiaries which was complemented with community validation to move to a more objective targeting (ICR, p. 18). According to the project team, this approach was more successful in rural areas than in urban ones since community validation was easier to implement in rural communities. Outputs and Intermediate Results The project provided a variety of grants: (1) productive cash-for-work which consisted of 60 days a year, paid at $2.3 a day for 5 hours' work for 3-years interventions; (2) a one-time public works activity with 50 days per individual and a daily wage of $5.6 for 5 hours of work; and (3) a recovery grant totaling $700 per household paid in three instalments of 10 percent (unconditional), 70 percent (conditional) and 20 percent (conditional). Furthermore, the grants were provided to rural and urban beneficiaries to respond to different shocks. The ICR does not provide an analysis on any differences in the different types of interventions in terms of targeting or operational challenges faced. The project exceeded all the targets it had established: Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) The number of beneficiaries who benefitted from safety nets programs were185,439 (original target was 100,000). The number of households receiving recovery grants reached 10,290 (original target was 10,000). Finally, the number of households benefitting from UCTs was 21,556 (original target was 10,000) (ICR, p. 15). In addition, the project created 891,204 person/days of employment and assisted 10,200 beneficiaries with preparing recovery plans for implementing their income-generating activities. Outcomes According to a beneficiary assessment and impact evaluation carried out at the end of the project (ICR, p. 15), the following outcomes were noted. But the robustness of the methodology underlying these two main sources of evidence is not known from the ICR -- some information about the methodology was provided subsequently in the Region's response.  Increase in assets: More than 26 percent of the beneficiaries were able to acquire at least one additional household equipment during the project.  Success of income-generating activities (IGAs): More than 87 percent of IGAs created by the beneficiaries were on good track without any particular problem.  Expansion of economic activities: More than 3,170 new economic activities were launched with the profits from the Income Generating Activities (IGAs) funded initially.  Increase in resilience: More than 90 percent of beneficiary households’ incomes have returned to or even exceeded pre-COVID levels. In addition, beneficiary households have returned to their pre- pandemic sources of income: 87.4 percent of households for wage income, 88.5 percent for income from non-agricultural businesses and 80.5 percent for income from agriculture.  Potential decrease in negative coping mechanisms. Benefitting households were able to recover consumption levels close to those seen pre-COVID-19 pandemic, suggesting that the project may have contributed to reducing negative coping mechanisms such as selling assets to cope with shocks in the short term (ICR, p. 21).  There were some issues regarding linking technical training to IGAs. Rating Substantial OBJECTIVE 2 Objective Increase poor communities’ access to nutrition services Rationale The ToC held that by building the capacity of community health workers and providing nutrition- sensitive training to communities and reproductive health sensitization to women, health and nutrition outcomes at the household level would be improved. In the Comoros, malnutrition among children under the age of five is widespread. In 2008, about one-third of all children under 5 years of age suffered from chronic malnutrition, which negatively affected their physical Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) and cognitive development. Malnutrition remained mostly concentrated in rural areas and among children with mothers who lacked education. In addition, and according to the 2012 Demographic and Health Survey- Multiple Indicator Cluster Survey (EDS-MICS II), 16 percent of children had low birth weight Contraception use (14 percent of women of childbearing age) and early breastfeeding (12 percent of the same age group) are very low. Furthermore, critical services such as nutrition counselling and family planning are largely non-existent (PAD, p. 3). With support from UNICEF, the government had established therapeutic feeding centers (known as Ambulatory Nutrition Recovery Centers or CRENA) at health facilities across the country for the treatment of severely malnourished children. Nevertheless, access to these centers remained limited as there is no organized referral system from doctors to the CRENA nor has there been any attempt to provide communities with preventive information and education on nutrition (PAD, p. 4). To address these issues, the project planned to promote three types of behavior: (i) exclusive breast feeding; (ii) food diversification (in both quantity and quality); and (iii) hand washing before meals (PAD, p. 12). Outputs and Interim Results The project exceeded all the targets it had established under this Outcome, mainly: It succeeded in reaching 33,4000 pregnant/lactating women, adolescent girls and/or children under age five- with basic nutrition services, more than four times the original target which was 9,600. In particular, the number of children under 24 months reached was 6,654 which constituted more than twice what was initially targeted (2,800). Furthermore, the number of children under five years enrolled in the growth monitoring program were 14,518 which exceeded the initial target of 9,200 by more than 150 percent. Finally, the project succeeded in training 105 Community Health Workers trained to provide Infant and Young Child Feeding (IYCF) services over-reaching the initial target of 105 and 294 mother leaders, almost double what was initially planned. Outcomes According to a beneficiary assessment and impact evaluation carried out at the end of the project (ICR, p. 15), the following outcomes were achieved:  Increased food consumption: Around 30 percent of beneficiary households were able to increase their number of meals per day.  Improved interaction between parents and children. Awareness-raising among beneficiary households seems to have increased parents’ concern regarding their children's education and health compared to households that were not targeted by awareness raising activities – though the ICR does not describe the methodology the methodology used for carrying out the beneficiary assessment nor on the sample size. The ICR also suggests that the difference between the treatment and control group may not be statistically significant (ICR, p. 16).  Greater food security resilience. Though almost 70 percent of UCT beneficiary households reported food difficulties, in terms of the Survival Strategy Index, the vast majority of targeted households (85.9 percent) were classified in the low survival index category. The project succeeded in preventing all Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) beneficiary households from further falling into the severe Survival Strategy Index category (ICR, p. 16). Rating Substantial OBJECTIVE 3 Objective Increase poor communities’ access to community services Rationale The ToC held that by rehabilitating damaged community infrastructure, such as schools, water tanks and health facilities, it would increase communities’ access to basic social services. Furthermore, by implementing cash-for-work interventions to rehabilitate the infrastructure, the economic recovery of households affected by the cyclone Kenneth. Outputs and Interim Results The project succeeded in rehabilitating 157 classrooms and 93 other infrastructure (such as water tanks, tracks or gutters), exceeding the initial target of 80 classrooms and 20 other infrastructure. The number of rehabilitated and equipped classroom constituted 35 percent of those affected by Cyclone Kenneth (ICR, p. 17). Furthermore, the infrastructure was rehabilitated though cash-for-work activities which created 891,204 person/days compared to the target of 600,000 person/days. Finally, though not reported in the ICR, according to the project team, the project also supported reforestation and land rehabilitation cash-for-work activities. Outcomes The project improved access to safety net and nutrition services in 187 poor communities. This achievement captured the multi-dimensional contributions of the project to service delivery through livelihood support, training, and awareness campaigns, as well as community-led infrastructure rehabilitation. Community-level recovery was achieved through the multiplier effect of cash-for-work interventions that prioritized infrastructure that is key for service delivery. As a result, the rehabilitated Wellness Spaces provided a venue to implement awareness raising activities regarding family planning and nutrition for children aged 6 to 23 months. These spaces focused on two main areas: (i) family planning and (ii) nutrition for children aged 6 to 23 months. The impact assessment found that there was a significant difference between the female beneficiaries who attended family planning sessions in the Wellness Spaces and those who did not: female beneficiaries were more likely to practice family planning when they had been made aware of the subject through community gatherings. The ICR notes “There was a significant difference of 34 percentage points between the two groups, 100 percent versus 66 percent” (ICR, p. 17). This result warrants greater explanation. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) Water conservation (through the construction of water tanks), reforestation and land management have contributed to increasing communities’ resilience to climate change. Finally, the ICR reported that beneficiaries had a good perception of the activities implemented by the project, with over 82 percent of beneficiaries giving the project an above-average rating (ICR, p. 17). However, beyond the use of rehabilitated Wellness Spaces - which total number is not provided in the ICR, there are no specific data on utilization of community services which would demonstrate an increase of poor communities access to various services. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project's overall efficacy is rated Substantial based on the aggregated achievements of project objectives, as discussed above. Overall Efficacy Rating Substantial 5. Efficiency Efficiency as analyzed based on coverage (including inclusion/exclusion errors), timeliness, cost, capacity, and implementation arrangements. High coverage of the poor population. The ICR notes that the project succeeded in reaching 51 percent of all the population living in poverty (ICR, p. 45) and this coverage further increased in rural areas where the SSNP reached more than 66 percent of the rural poor. This high coverage and the substantially higher number of beneficiaries reached than originally targeted was due to a favorable exchange rate (ICR, p. 47). Inclusion/exclusion errors are not mentioned in the ICR. The ICR does not discuss whether any errors of inclusion/exclusion took place though this was mentioned as a potential issue in the PAD and that specific tools would be designed to reduce such errors (PAD, p. 15). This information was subsequently provided by the Region which indicated that the inclusion error stood at 19.2 percent. There were errors, in particular in urban settings, and the project received complaints which were resolved. The project became effective in less than four months (ICR, p. 2) however it later experienced delays. While mostly due to external factors (the implementation agency changed twice, cyclone Kenneth occurred, and Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) COVID 19 struck), the project closed 4 years later than originally planned. During the initial stages of the project, the implementation agency was changed three times which led to initial delays (ICR, p. 21). The COVID-19 also created implementation challenges due to lockdowns, office closures and social distancing (ICR, p. 21). The ICR mentions that there were disruptions related to COVID-19 (ICR, p. 14); however, does not explain how despite the delays and difficulties in implementation, the project was able to achieve and exceed target results (ICR, p. 14). The project's operational cost decreased over time. The SSNP had a set a target of 20 percent for operational cost in the parent project and this had decreased to less than 9 percent at closing and compares the project favorably with other similar projects in the region (e.g., in Madagascar, the overall operational cost was at 10.8 percent, while in Burundi these costs reached 14 percent) (ICR, p. 18). However, the ICR alludes that this increase in efficiency is also due “lower cost of activities such as unconditional cash transfers that were introduced later in the life of the project” which were implemented in urban and peri-urban areas (ICR, p. 18). It would have been interesting to analyze the efficiency of the parent project (prior to the additional finance to address COVID-19) and to compare the efficiency of implementing in rural versus urban areas. The Internal Rate of Return (IRR) at the end of the project is higher than originally estimated. The return on investments analysis carried out at the end of the project for recovery grants activities shows a benefit to cost ratio of 2.3 and an estimated IRR of 11.3 percent, which compares favorably to the 8.2 percent IRR estimated at appraisal (ICR, Annex 4, p. 46). The Efficiency Analysis notes that the profitability of livestock activities financed by the project was quite high, in particular, with regard to sheep and goat breeding (ICR, p. 47). Increased project staff capacity contributed to the project’s efficiency. The project's operating costs improved over time due to capacity building of the staff. The project acquired experience as activities progressed and was able to limit the expansion of its staff numbers despite the expansion of activities (ICR, p. 36). Decentralized implementation arrangements contributed to enhancing the SSNP’s efficiency. The principle of subsidiarity was gradually applied with technical and administrative decisions being devolved at the regional directorate level while respecting all project implementation procedures (ICR, p. 18). However, there were varying capacities on the different islands which created implementation challenges. The ICR points out that the project contributed to the overall efficiency and resilience of the SP system through enhanced targeting and the introduction of mobile money (ICR, p. 18). The introduction of Proxy Means Testing and vulnerability scoring improved the sector’s effectiveness and developed its capacity. In addition, the SSNP introduced mobile money payments during the COVID-19 response, and all payments were delivered through this approach. The ICR notes that this improved the project’s efficiency (ICR, p. 18) without discussing whether the project experienced any challenges due to the country’s frequently unreliable internet connectivity as mentioned in the PAD (p. 13). The ICRR notes that these efficiency gains accruing to the program and/or sector should not be confused with the efficiency of the project itself. The ICR suggests some issues with digital payments and logistical challenges. Efficiency Rating Substantial Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 20.00 Appraisal  8.20  Not Applicable 20.00 ICR Estimate  11.30  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s overall rating is Satisfactory. The project was highly relevant and aligned to both the Government’s as well as partners’ - including the World Bank – priorities. The robustness of the methodology used for the Beneficiary Assessment and Impact Evaluation was not discussed in the ICR – some information about the methodology was provided subsequently in the Region’s response. Regarding targeting effectiveness, the inclusion error stood at 19.2 percent. There also appear to have been issues with digital payments and logistical challenges and linking of technical training to IGAs. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The risk to development outcome is analyzed across the following dimensions: policy, financial, organizational, technical capacity regarding the strengthening of a social safety net system. At the policy level, the project provided input and coordination to the development of the national Social Protection Strategy. In addition, it contributed to the framework for establishing a social registry (ICR, p. 20). In terms of financial sustainability, financing of safety nets remains overwhelmingly supported through external support. However, there has been continuous dialogue between the World Bank and the IMF to ensure that more local resources are channeled towards SP and to position SP financing as a key investment for human capital accumulation and to enable reforms (ICR, p. 25). Organizational strengthening - through the development of operational tools – is the aspect that is most likely to achieve and sustain the development outcomes. The operational tools that were developed (e.g., establishment of the beneficiary registry, the PMT for targeting, MIS, the M&E system) will be key for further developing and improving the efficiency of the SP sector in the Comoros (ICR, p. 18). Furthermore, the project’s database includes more than 55,000 potentially vulnerable households Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) representing 39 percent of all households in the Comoros which would allow the further expansion of SP in the country (ICR, p. 46). In terms of human/technical capacity, the project developed the skills and technical capacity at three different levels: (1) implementing agency, (2) community level; and (3) financial service delivery. The SSNP organized several exchange missions for government counterparts to learn from other countries’ experience in implementing and scaling up Productive Safety Nets and in delivering Accompanying Measure packages. In addition, the project organized virtual workshops and provided technical support on Financial Management, Payment Systems, Digitalization, and the Grievance Mechanism (ICR, p.25). At the community level, both mother leaders as well as community health workers were trained. This has allowed other projects to leverage them for the delivery of subsequent support for health and nutrition beyond the SSNP and has further contributed to the development of Comoros’s new community health platform (ICR, p. 16). Finally, the SSNP, through the introduction of payments through mobile money and microcredit institutions, has contributed to: (i) increasing financial inclusion; and (ii) developing new skills among these institutions in terms widescale transfers and offering savings and credit operations (ICR, p. 46). At the beneficiary/household level and as mentioned under Objective 2 above, the income-generating activities and cash transfers have contributed to their economic sustainability through the creation of IGAs, increasing household assets and expanding economic activity at the community level. Overall, the project’s sustainability is evidenced by its extension and scaling up over time, becoming the main vehicle for shock response in the country (ICR, p. 14) and being used by subsequent projects (e.g., the Shock Responsive and Resilient Social Safety Net Project (P179291, Report No: PAD5129)) to deliver services (ICR, p. 18). 8. Assessment of Bank Performance a. Quality-at-Entry The project’s design had some strong elements, in particular the use of analytical assessments and lessons learnt from other national and international social protection programs. The project’s design benefitted from several analytical assessments, including the 2012 Demographic and Health Survey-Multiple Indicator Cluster Survey (EDS-MICS II) and a Social Protection Assessment carried out by the World Bank (PAD, p. 3). In addition, the design was partially modeled on other successful and cost-effective social protection programs, such as the National Community Nutrition program in Madagascar and Bangladesh’s Rural Advancement Committee (BRAC) approach which combined nutrition counseling with cash transfers to promote behavior change (PAD, p. 12). In addition, the project incorporated the lessons learnt from two previous Bank operations, namely the Comoros Emergency Crisis Response Project (P120631, Report No: 54445 – KM) (ICR, p. 24) and the Community Development Support Fund (Fonds d’Appui au Developpement Comunautaire or FADC) (PAD, p. 5). Finally, the project incorporated lessons from a Bank Social Protection Assessment which identified significant shortcomings with nutrition programs that were geographically limited and did not offer free-of-charge access to basic health care (ICR, p. 8). Community level engagement on nutrition Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) and best practices was also complemented through targeting with income support, and results suggest a positive impact (ICR, p. 16). The design included (i) developing and applying targeting criteria (Proxy Means Testing/community validation/post intervention surveys); and (2) conducting household surveys with baseline and endline data, while also applying the double difference method with Propensity Score Matching (PSM). The project’s implementation arrangements relied on a mix of national systems (FADC) and partnership with UNICEF to ensure sufficient capacity. However, the ICR does not mention any collaboration with technical institutions to provide the capacity building necessary to ensure the success of IGAs. This was addressed under the Lessons Learnt section as a weakness in project design which led to some beneficiaries to give up on their IGA (ICR, p. 26). The project’s design was quite complex given the low-capacity context. The introduction of the nutrition aspects, additional measures and infrastructure rehabilitation required significant capacity building which was provided through the Rapid Social Response Trust Fund (ICR, p. 24). Finally, since as per World Bank guidelines, including the Theory of Change in Project Appraisal Documents was mandatory as of May 2018, the project design did not include a ToC which presented the pathways from inputs, outputs to outcomes and objectives. Nevertheless, the PAD included a detailed Results Framework which allowed the ICR to re-construct a ToC. The Results Framework is discussed below under M&E Design. Quality-at-Entry Rating Satisfactory b. Quality of supervision The following elements were considered to have contributed to the quality of the Bank’s supervision: i. Conducting timely and regular implementation support missions, involving a multi-sectoral team to cover the different aspects of the project; ii. Organizing additional ad-hoc meetings as needed; iii. Drafting detailed Aide Memoires and Implementation Status and Results Reports (ISR), providing in-depth assessments of the challenges faced; iv. Preparing quarterly financial reporting and annual financial and operational audits; v. Organizing exchange visits and knowledge transfer with other countries implementing Social Protection programs; and vi. Organizing virtual workshops, seminars for additional Bank staff support on Financial Management, Payment Systems, Digitalization, and the Grievance Mechanism. Quality of Supervision Rating Highly Satisfactory Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Though the PAD did not include a Theory of Change, the Results Framework and results indicators were closely aligned to the PDO and the underlying ToC. However, whereas the ICR provides detailed information on outcome-level results, the Results Framework Baseline monitored results only at the output level. The ICR mentions that data collection included monitoring of several outcomes including changes in the behavior of safety net beneficiaries (e.g., changes in eating habits, children’s school enrolment and attendance, utilization of health and nutrition services, and changes in productive activities) (ICR, p. 22). Sex-disaggregated monitoring was included unevenly in the Results Framework. The project monitored the number of women supported through IGA activities; however, it did not disaggregate between “pregnant/lactating women, adolescent girls and/or children under age five-reached by basic nutrition services” (ICR, p. 29). Furthermore, it is unclear why though the total number of beneficiaries of safety net programs increased from the original target of 5,890 to 100,000, the number of female targeted beneficiaries increased very slightly from a target of 2,000 women to 2,507 (ICR, p. 28). b. M&E Implementation The ICR provides brief information on M&E implementation and does not include any information regarding the Implementation Status and Results Reports nor the Aide Memoires (quantity, frequency). A few indicators were fine-tuned during implementation. For example, following the AF2, the indicator for monitoring infrastructure rehabilitation was split to distinguish between classroom and other infrastructure (ICR, p. 34). A new indicator was introduced to monitor the efficiency of the grievance mechanism (Percentage of grievances received and resolved within 45 days) (AF2, p. 17). Interestingly, the Results Framework monitored the project’s efficiency through several indicators, including (1) the project’s operational costs; (2) the number of Operational Audits implemented; (3) the percentage of beneficiary households of the Productive Safety Net activities registered in the PIU's MIS; and (4) the periodic report on the Grievance Redress Mechanism; and (5) the percentage of grievances received and resolved within 45 days (ICR, pp. 36-38). The ICR does not mention whether a baseline was conducted.The robustness of the methodology used for the Beneficiary Assessment and the Impact Evaluation was not discussed in the ICR but the Region provided some of this information subsequently. Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) Finally, the project collaborated with UNICEF to monitor the nutrition component which ensured harmonization and complementarity with other nutrition initiatives (ICR, p. 22). The robustness of the methodology used for the Beneficiary Assessment and the Impact Evaluation is not known. c. M&E Utilization The ICR does not discuss M&E utilization. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project's overall safeguards rating was recorded as satisfactory in the Operations Portal. Two safeguards were triggered: Involuntary Resettlement Policy (OP/BP 4.12) and the Environmental Assessment Policy (OP/BP 4.01). As a result, the project developed an Environmental and Social Management Framework (ESMF) that included an Environmental and Social Management Plan (ESMP) to ensure that activities were environmentally and socially sustainable (ICR, p. 23). Though the project envisaged implementing activities to prevent/diminish the effects of climate change on communities (e.g., reforestation and other climate-relevant activities), the ICR does not mention if such activities were implemented, nor does it discuss whether any voluntary/involuntary resettlement took place. It also does not mention whether the ISRs provided low safeguards' ratings during the project's lifetime. In terms of social safeguards, and in order to address the COVID-19 pandemic, the project developed an Emergency Response Manual to prevent and limit the spread of COVID-19 (AF, February 5, 2021, p. 28). Furthermore, occupational Health and Safety procedures were also designed for the project, and no incidents were reported formally (ICR, p. 23). Finally, the project tracked grievances and the time taken for their resolution through a Grievance Mechanism, which it monitored under the M&E indicators. Four periodic reports on the Grievance Redress Mechanism were produced (ICR, p. 38) and 100 percent of all complaints received by the project have been addressed, in most of the case within a few days from reception (ICR, p. 38). However, the ICR does not mention the number nor the type of complaints received. The ICR does not also explicitly state if the project complied with the Bank's safeguards policies. Of the 17 ISRs, overall safeguard compliance was rated MU in one ISR and was not rated in another, with the rest of the ISR ratings being MS or S (the last one had an S rating). Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) b. Fiduciary Compliance In terms of financial management, the ICR mentions that the project followed Bank rules and procedures regarding financial and audit aspects (ICR, p. 23). It also states that financial management improved over time without mentioning what were the initial weaknesses (ICR, p. 23). Interim Financial Reports (IFRs) and audit reports were submitted on a timely basis and were of acceptable standards. Furthermore, to strengthen fiduciary compliance, the project recruited an accountant in each regional office. The regional accountants were also supported by an accountant and financial managers at the national level who provided cap capacity building training and FM-related support (ICR, p. 23). Of the 17 ISRs, Financial Management was rated MU in three, and S or MS in the rest. One ineligible expense was identified during the transition from ANACEP to the new implementing entity. This was corrected and the expense was reimbursed (ICR, p. 24). Similarly, the ICR noted that procurement capacity had improved during the project’s implementation. Nevertheless, some challenges remained and were associated with (i) staff turnover, (ii) record keeping, and (iii) timely submission of procurement plans due to challenges with connectivity (ICR, p. 24). Of the 17 ISRs, Financial Management was rated MU in three and MS or S in the rest (the last one had an S rating). Procurement was rated S in all 17 ISRs. c. Unintended impacts (Positive or Negative) The ICR identified two positive unintended impact. 1. The support measures provided in the wellness spaces achieved a multiplier effect on human development. A significant difference was observed between households in which one member had discussed family planning in the Wellness Spaces, and those in which the member had not: Significantly, more female beneficiaries were likely to practice family planning when they had been made aware of the subject through community gatherings than those who had not gone to the Wellness Spaces. The difference was substantial (34 percentage points), with 100 percent of women who discussed family planning versus 66 percent who did not. (ICR, p. 17) 2. Awareness-raising among beneficiary households seems to contribute to more interaction between parents and children within the household. Indeed, benefiting parents are much more concerned about their children's education and health than parents in households who did not attend or did not benefit from the project, however the difference was two percentage points (5 percent versus 2 percent) which may not be statistically significant (ICR, p 16). d. Other The project supported women through different activities:  Improving women's economic status by providing IGA grants (around 77 percent of the recipients were women); Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754)  Implementing gender-oriented safety net activities reaching a high number of women (67 percent of beneficiaries);  Providing basic nutrition services to pregnant and lactating women and adolescent girls; and  Empowering women through training around 300 women leaders at the community level. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment There were some issues with targeting and timeliness. There also appear to have been issues with digital payments and Outcome Highly Satisfactory Satisfactory logistical challenges and linking of technical training to IGAs. Finally, compliance with the Bank's safeguards and fiduciary policies is not known There were some issues relating to quality-at-entry including the Bank Performance Highly Satisfactory Satisfactory complexity of the project's design in relation to the low capacity environment. Quality of M&E Substantial Substantial Quality of ICR --- Modest 12. Lessons The ICRR confirms the series of lessons and recommendations emerging from the project’s experience (ICR, pp. 26-27). The ICRR summarizes the most relevant lessons below: Financial as well as technical support can increase the effectiveness and sustainability of Income-Generating Activities. In particular in rural areas, where IGAs are related to the agriculture sector, partnering with government (e.g., Ministry of Production) and/or Non-Governmental Organizations who have technical expertise can increase the chances of IGAs being successful and sustainable. Rehabilitating community infrastructure is an opportunity to improve not only the economic and social resilience at the community level, but also to improve resilience to future climate shocks. In particular, in countries that are vulnerable to climate change, cash-for-work programs can be used to build more resilient infrastructure that can be managed and maintained by the community. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) Balancing the introduction of digitalization with capacity building and technical support improves the efficiency of implementation while ensures the greater inclusion of vulnerable population. Training on the use of digital means and ensuring internet connectivity allowed the project to reach a larger number of beneficiaries, especially in urban areas where internet connectivity was available. The complexity in implementing at the decentralized level with varying levels of capacities among the different islands requires a varied approach to technical support and capacity building. In addition to different capacities, remote and difficult to access locations have additional logistical challenges, in particular if the project is supporting the rehabilitation of infrastructure. Effective communication between the central and regional authorities ensures both smooth implementation and ownership at all levels, in particular when approaches are being implemented for the first time in a low-capacity context. Additional Note: The ICR missed an opportunity to draw lessons regarding some important aspects of the project such as (1) implementation challenges in rural versus urban areas and how they were addressed; (2) the different outcomes reached through unconditional cash transfers versus IGA activities versus cash transfers with accompanying measures; and (3) the complexity of adding an infrastructure rehabilitation component in a social protection program. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR had some positive features. However, the reconstructed Theory of Change (ToC) based on the project’s Results Framework could have been strengthened with the inclusion of the key assumptions underlying the ToC (e.g., existence of sufficient capacities at the decentralized level; access to remote locations, internet connectivity). Given the inclusion of urban beneficiaries to respond to the COVID-19 pandemic, the ICR could have provided an analysis between the rural/urban differences in terms of targeting, implementation challenges faced and differences in project efficiency. Community validation of poor households would seem to be easier to implement in rural areas so the rural-urban differences in poverty targeting merited more discussion. The ICR could have been more candid and addressed the inclusion/exclusion errors and how they were addressed. The ICR should also have discussed the robustness of the methodology underlying the two main sources of evidence for project results, i.e., the beneficiary assessment and the impact evaluation. The Lessons Learned section provides hints regarding several challenges that have impacted both the project’s efficiency (e.g., digital payments and logistical challenges) as well as effectiveness (e.g., importance of linking technical training to IGAs) which were not raised in the Efficacy/Efficiency sections. Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Comoros Social Safety Net Project (P150754) The ICR does not assess M&E Utilization. Finally, in Annex 1. Results Framework and Key Outputs, not all the IRIs are rated and there are no comments to discuss the achievement or lack thereof. The ICR does not also explicitly state if the project complied with the Bank's safeguards and fiduciary policies. a. Quality of ICR Rating Modest Page 19 of 19