Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) Report Number: ICRR0022723 1. Project Data Project ID Project Name P158769 Sindh-MSAN Country Practice Area(Lead) Pakistan Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A5202 31-Jan-2020 2,961,895.41 Bank Approval Date Closing Date (Actual) 24-Oct-2017 31-Aug-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 5,000,000.00 5,000,000.00 Revised Commitment 5,000,000.00 2,961,895.41 Actual 2,961,895.41 2,961,895.41 Prepared by Reviewed by ICR Review Coordinator Group Kavita Mathur Fernando Manibog Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO) was "to increase the dietary diversity and improve sanitation and hygiene practices in targeted project districts areas" (PAD, p. 9, Trust Fund Agreement is not available in Operations Portal). b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) No c. Will a split evaluation be undertaken? No d. Components The project included three components: Component 1 - Support to Saaf Suthro Sindh (appraisal estimate US$18.7 million, actual US$2.64 million). This component would focus on creating outdoor defecation free (ODF) jurisdiction, promoting handwashing in communities, and building the capacity of Government and NGO staff, as well as collecting and analyzing field data on open defecation, hygiene and sanitation practices in the communities. The component includes rehabilitation and new construction of selected and prioritized water supply and sewerage facilities according to village requirements, to ensure that people have sufficient water for toilets and to avoid inundating streets with sewage water. This component had three subcomponents: Subcomponent 1a – Building Capacity for ODF Certification; Subcomponent 1b – Behavior Change for Improved Hygiene; and Subcomponent 1c – Incentivizing Good Hygiene Behavior. Component 2 - Agriculture for Nutrition (appraisal estimate US$4.22 million, actual US$0.15 million). This component would support integrated homestead production for dietary diversity. Specifically, it intended to finance assets and provide technical assistance to communities and households to grow vegetables, pulses, and fruit, and to raise livestock and fish, primarily for household consumption to promote diversified diets. This component had three subcomponents: Subcomponent 2a – Food Production and Management; Subcomponent 2b – Awareness Raising on Nutrition and Healthy Diet; Subcomponent 2c – Capacity Development for Government Staff. Component 3 - Project management, monitoring, and coordination (appraisal estimate US$4.09 million, actual US$0.17 million). This component would will finance the costs of project management, social mobilization, monitoring and evaluation, learning, impact evaluation, media communications, and intersectoral coordination. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The estimated cost at appraisal was US$27.01 million. The actual cost was US$2.96 million Financing. At appraisal, US$17.56 million was committed by the Pakistan Partnership for Improved Nutrition Trust Fund (PPIN-TF), which was funded by DfID and the Australian Department of Foreign Affairs and Trade (DFAT). The Grant agreement actually reflected the "initial tranche" of US$5.0 million (ICR, p. 6). The actual financing was US$2.96 million. Recipient Contribution. The planned contribution was US$9.45 million. The actual contribution was US$75,000 (ICR para 8) (the datasheet reports zero). Page 2 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) Dates and Restructurings. The project closed on August 31, 2020, four months earlier than the original closing date of December 31, 2020 (PAD Basic Information datasheet). The project underwent two Level 2 restructurings to change the closing date. The first restructuring was on March 4, 2018, and reduced the project duration by six months by changing the grant agreement closing date to June 30, 2020. This was necessary to allow for the standard six-month financial closure period of the parent Multi-Donor Trust Fund financing the project, the PPIN, which had an end-disbursement date of December 31, 2020 and required a four month grace period for withdrawal applications as well as an additional two months for closing of account (ICR para 15). The second restructuring on June 28, 2020 extended the closing date of the Project (June 30, 2020) by two months to August 31, 2020. This required a waiver to reduce the standard six-months financial closure period. 3. Relevance of Objectives Rationale Context. At appraisal, Sindh's nutrition indicators were among the worst in Pakistan. The 2015 Multiple Indicator Cluster Survey (MICS) undertaken by the United Nations Children's Fund (UNICEF) showed that Sindh’s malnutrition and micronutrient deficiency levels were a serious public health problem: 15.4 percent of children under the age of five exhibited wasting, and 48 percent stunting. This is not only above national figures but approximately twice the global average (ICR para 2). Consistency with Government Strategy. In 2001, the government of Pakistan conducted a National Nutrition Survey. Following the survey, the government established a Nutrition Wing and developed a National Nutrition Program. However, this program met with limited success as the Provincial Nutrition Cells were unable to develop and deliver integrated nutrition programs. In 2013, Pakistan joined the "Scaling Up Nutrition", a global push for action and investment to improve maternal and child nutrition. In 2014, the government had formulated Provincial Policy Guidance Notes and Inter-Sectoral Nutrition Strategies (PAD para 5). The Project was built on existing nutrition-related interventions in the province of Sindh, by scaling-up the Saaf Suthro Sindh (SSS) program, a nutrition-sanitation approach to foster the creation of Open Defecation Free (ODF) villages with the help of non-governmental organizations (NGOs). The Project also aimed to boost the Chief Minister’s Accelerated Action Program (AAP) for Reduction of Stunting and Malnutrition in selected areas of four districts of Sindh through its Agriculture for Nutrition component. Consistency with the Bank Strategy. The project was aligned with the World Bank's Country Partnership Strategy for Pakistan (CPS FY15–FY21) which supported “improved access to maternal and child health services” under Results Area 4. Specifically, the CPS mentioned “plans for scaling up nutrition interventions” and “5,000 villages certified as ‘open defecation free’ reducing the incidence of diseases”. . Page 3 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) Previous Bank Experience. The Bank has a long history of partnership and collaboration with Pakistan. This project was coordinated with other World Bank investments aimed at reducing malnutrition in Sindh, notably the (i) Enhanced Nutrition for Mothers and Children Project (ENMCP; P115889) and (ii) the Sindh Enhancing Response to Reduce Stunting (SERRS; P161624). This project aimed to harmonize several cross-cutting areas with these: behavior change communication, a planned information technology (IT) platform for monitoring and evaluation (M&E), social mobilization efforts, institutional arrangements, and a standardized impact evaluation (ICR para 3). The statement the objectives was clear, and the PDO was in line with the current Bank’s Country Partnership Strategy (CPS) for Pakistan and relevant to the priority of Sindh Province. However, the PDO appears to be over-ambitious and not properly scaled to the existing serious weaknesses in institutional capacity. The relevance of objectives is rated substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase the dietary diversity in targeted project districts areas. Rationale The project aimed to address the underlying issues of chronic malnutrition through agricultural intervention to introduce production of nutritious food for consumption through homestead gardening practices and increasing household knowledge of food and nutrition. The theory of change was that the project inputs -- such as (a) demonstration plots and ponds (by the Department of Agriculture) and grants using farmer field school (FFS) approach and technical assistance to communities and households to grow vegetables, pulses, and fruit, and to raise livestock and fish primarily for household consumption; (b) grants to households and/or community groups to purchase assets needed for integrated homestead production (e.g., kitchen gardens, livestock production, fishponds, and food storage and preservation); and (c) awareness raising on nutrition and healthy diet -- would yield outputs including (i) pilot demonstration projects for agriculture, livestock and fisheries using farmer business schools to help families/community groups market surplus food for additional income; (ii) increased availability of diversified good groups; and (iii) behavior change around diets and child feeding practices. The expected outcome would be increased dietary diversity. Outputs/Outcomes The ICR reports (para 25) that the activities aimed at achieving this objective were not implemented. There was no progress after the initial planning stages, funds were not disbursed and therefore no progress was made towards the associated PDO indicators. The ICR explains (para 38) component 2 “ Agriculture for Nutrition” was originally designed as a pilot to inform the Chief Minister of Sindh’s Accelerated Action Page 4 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) Program (AAP) for Reduction of Stunting and Malnutrition was completed. However, the AAP was under implementation by the time this project reached implementation readiness. Therefore, there was no need for the “pilot”. Moreover, there were other issues (a) the project director was located three hours (by road) from project sites; (b) staff assigned to support the implementation were split across multiple departments, contributing to weak coordination and slow processes. The three PDO indicators were not achieved: (i) “percentage of targeted households that increase minimum dietary diversity for children” – actual 7.4, original target 20; baseline 7.4; (ii) “percentage of targeted households that increase dietary diversity for females of childbearing age” – actual 0, original target 20; baseline 0; and (iii) “increase in the number of food groups consumed by target households (dietary diversity)” – actual 2.9, original target 5.0; baseline 2.9. Rating Negligible OBJECTIVE 2 Objective To improve sanitation and hygiene practices in targeted project districts areas. Rationale The project was expected to contribute to improving the underlying conditions leading to chronic malnutrition through agricultural intervention to introduce production of nutritious food for consumption through homestead gardening practices and increasing household knowledge of food and nutrition (objective 1) and sanitation interventions focusing on behavioral change to achieve open defecation-free villages and hygiene practices (objective 2). The inputs such as provision of rehabilitation and new construction of selected and prioritized water supply and sewerage facilities, creating Ambassadors of Change (AOCs) to promote awareness-raising about ODF, incentivizing good hygiene behavior - would result in outputs such as toilets with sufficient water, ODF jurisdictions and handwashing stations. The expected outcome would be improved sanitation and hygiene practices in targeted areas. Outputs No latrines were constructed on school premises. The target was 2,600. The project hired 13 NGOs as Implementing Partners, one for each targeted districts. The IPs prepared village profiles of all 5,200 villages, established and operationalized Village Organizations (VOs) in 2,400 villages. District Open Defecation Free Plans were prepared the 13 districts and were approved by the respective District Coordination Committee for Nutrition (DCCN) (ICR para 22). By closing, 2,400 out of 5,200 Page 5 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) targeted villages were certified ODF. It is not clear in the ICR how ODF status was achieved since the project did not construct any latrines. The IPs conducted 6,750 hygiene promotion sessions in 2,400 villages of the 13 districts. 14,000 people compared to the target of 11,000 were trained in hygiene behavior/sanitation practices. Outcomes At project close, 23 percent of the total rural population in targeted districts were certified ODF, substantially lower than the target of 50 percent (baseline was 0). Direct project beneficiaries were 1 million compared to the target of 2 million. Female beneficiaries were reported as 50 percent. The project improved critical hygiene behavior to prevent transmission of contaminants that lead to diarrheal and other diseases. This involved washing hands with soap and water after defecating. At project closure 60 percent of people washed their hands with soap after defecation, lower than the 70 percent target (baseline was 36 percent). Rating Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project did not achieve the objective of increasing the dietary diversity in targeted project districts areas. The objective of improving sanitation and hygiene practices in targeted project districts areas was modestly achieved. Overall efficacy is modest. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency Economic Analysis. In the PAD, there was inconsistency in the benefit cost ratio in the main text (ranged from 2.7 to 7.6) and the Annex 2 (ranged from 14.4 to 28.5). The present value of the benefits of the project were estimated in the range of US$55.68 to US$156.48 million (PAD para 41). The benefits stemmed from reduced infant mortality, saved Disability Adjusted Life Years (DALYs) and benefits associated with reduced coping costs due to lower diarrheal disease incidence as a result of open-defecation-free (ODF) villages to be achieved under Component 1, as well as a valuation of reduced infant mortality associated with greater dietary diversity under Component 2. Consuming healthy food to boost nutrition would decrease malnutrition, which Page 6 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) greatly compromises children’s immune system, making them more susceptible to infectious diseases and limits bone growth. Malnutrition negatively affects brain development, causing delays in cognitive development. At closing, the benefit-cost ratio included only benefits from Component 1, as Component 2, was not successfully implemented. The benefit-cost ratio ranged from 5 to 13.6. The ICR (para 31) explains that the higher than expected benefit-cost ratio is primarily the result of cost savings. These were partly due to more efficient than expected implementation partners (i.e., lower than planned contract costs) under Component 1. Operational Efficiency: The procurement delays resulted in slow implementation of the project. Overall, the project efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the project objectives to the country context and their alignment with the World Bank strategy were rated substantial. The project did not achieve the objective of increasing the dietary diversity in targeted project districts areas. The objective of improving sanitation and hygiene practices in targeted project districts areas was modestly achieved. Overall efficacy is modest. The project’s efficiency in achieving the project objective was rated modest. Based on these, overall outcome is moderately unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome Stakeholder Risk. The stakeholder risk significant. The ICR (para 46) points to international experience which indicates that some members of communities that have achieved open-defecation-free status Page 7 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) gradually reverse to open defecation unless continued follow-up is provided to entrench the use of toilets.. Moreover, due to the abrupt closure of project, activities relating to construction of toilets were not completed. Therefore, behavior change (to use toilets instead of open -defections) was not achieved. 8. Assessment of Bank Performance a. Quality-at-Entry The project was highly relevant to Bank strategy and country priorities. Safeguards assignment and M&E arrangements were appropriate. However, the World Bank overestimated implementation readiness and set an ambitious implementation timeline. Moreover, the implementation arrangement for component 2 were inadequate as it involved coordinating the two PMUs across three government departments (ICR para 45). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The ICR reports (para 45) that the project team was proactive in resolving issues related to the resignation of project director or the slow opening of project accounts. The main shortcoming of supervision was the failure to cancel component 2, once it became clear that client focus had shifted, and nutrition related activities would not be implemented. The Bank did plan to carry out restructuring, but this was prevented by the late decision to close the project instead of extending it. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Under the Project Management, Monitoring, Evaluation, and Coordination component of the project, baseline data collection and analysis were to be carried out by a third party consultancy firm. The project also aimed to equip villagers designated as “Ambassadors of Change” with smartphones for Page 8 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) continuous, local monitoring. The outcome indicators were adequate to measure the achievement of objectives. b. M&E Implementation The third party consultancy firm provided only the baseline data and the endline survey was not carried out because of the abrupt closure of the project. The smartphones were never procured and distributed due to initial delays. c. M&E Utilization The ICR does not report on M&E utilization. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Environmental Category "B" and three safeguards policies were triggered: Environmental Assessment OP/BP 4.01; Pest Management OP 4.09; and Involuntary Resettlement OP/BP 4.12. At appraisal, no adverse or negative environmental impacts were expected to result from the project (PAD para 64). OP 4.12, Involuntary Resettlement was triggered in case the project needed to acquire small pieces of land for certain interventions (e.g., storage facilities) (PAD para 61). An Environmental and Social Management Framework (ESMF) and Resettlement Policy Framework (RPF) were prepared at appraisal. During implementation, no resettlement took place and the two specific environmental hazards highlighted at appraisal – construction of latrines and use of pesticides – never materialized as these activities of the project were not implemented. The overall safeguards rating remained Satisfactory throughout the project duration. b. Fiduciary Compliance Financial Management. The ICR reports (para 41) that the financial management of the project was Satisfactory. The Interim Financial Reports (IFRs) were submitted on time. Legally all contracts under the project were closed without issue on August 31, 2020. Page 9 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) Procurement. The ICR reports (para 42) that the project complied with the World Bank procurement procedures. Procurement of several contracts was delayed due to COVID-19 and award of other contracts had to be halted as a result of the decision to close the TF in August 2020. The Procurement Management rating in the final ISR was Moderately Satisfactory. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The following lesson is adapted from the ICR: Small Grants incur high transaction costs and are more challenging to implement. The project experience shows that small grants attract similar procedural requirements as larger financing, and thereby have high transaction costs, requiring a high level of attention, staff time and political support to navigate efficiently. The limited funding itself made it more difficult to obtain political and administrative attention and support, thus aggravating delays. Moreover, procedural delays are a particular risk for projects of short duration as a slow start-up period consumes a relatively larger part of the total time available for implementation. Therefore, in evaluating future small-grants, the Bank needs to carefully consider the relatively high transaction costs, insist on realistic project timelines and proceed only if there is a particularly high level of client support and expected impacts relative to the small project size. 13. Assessment Recommended? Page 10 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh-MSAN (P158769) No 14. Comments on Quality of ICR The ICR provides a detailed overview of the project. The narrative supports the ratings and available evidence. It is candid and generally aligned to the project development objective. The ICR’s lessons are clear and based on evidence outlined in the ICR. However, a major shortcoming is the absence of a section on the project's theory of change. Overall, the quality of the ICR is rated substantial. a. Quality of ICR Rating Substantial Page 11 of 11