Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) Report Number: ICRR0024045 1. Project Data Project ID Project Name P150520 Punjab Rural Water Sector Imp Project Country Practice Area(Lead) India Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-84870 31-Mar-2021 189,417,613.16 Bank Approval Date Closing Date (Actual) 24-Mar-2015 30-Mar-2023 IBRD/IDA (USD) Grants (USD) Original Commitment 248,000,000.00 0.00 Revised Commitment 189,417,613.16 0.00 Actual 189,417,613.16 0.00 Prepared by Reviewed by ICR Review Coordinator Group Craig Phillip Kullmann Fernando Manibog Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO) as stated in the PAD (p. 3) and Financing Agreement (Schedule 1, p. 6) was “improve water and sanitation service levels, reduce open defecation, and strengthen service delivery arrangements in targeted villages in Punjab”. The PDO did not change during the life of the project, and this review has assessed the PDO achievement in terms of the following three objectives: Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) I. Improve water and sanitation service levels in targeted villages in Punjab. II. Reduce open defecation in targeted villages in Punjab. III. Strengthen service delivery arrangements in targeted villages in Punjab. The PDO outcome targets related to improving water service levels as well as sanitation service levels were materially reduced in scope during one restructuring of project; therefore, the ICR Review opted to conduct a split evaluation. The sanitation services aspect of PDO 1 consists of increasing access and use of sewerage connections and household toilets not connected to a centralized system. For this ICRR, the household toilets are captured under PDO 2 given the direct linkage with reducing open defecation. Access to sewerage is retained under PDO 1 because they are centralized systems managed by Gram Panchayat Water and Sanitation Committees (GPWSC). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components Component 1: Transformation - Improved Livability of Villages. (US$160 million at appraisal, and US$77.3 million actual). This component financed the following activities: Subcomponent 1(a). Strengthening water supply schemes for improved service delivery. Improvements of water sources, treatment, storage, disinfection, and distribution systems. This included the rehabilitation of damaged canal-based schemes for sourcing water in approximately 110 single villages or multi-villages. Approximately 570 villages with poor water services were estimated to benefit. Subcomponent 1(b). Sewerage schemes with treatment systems. Construction of new conveyance systems, household connections, pavement restoration, and sewerage treatment plants. About 315 villages were estimated to benefit under this subcomponent. Component 2: Inclusive Household Water and Sanitation Services. (US$85 million at appraisal, and US$96.86 million actual). This component financed: Subcomponent 2(a). Improved access to water through household connections. Construction of private water connections inside households as well as extension of distribution systems was estimated to benefit 650,000 households. Subcomponent 2(b). Operational improvement for improved water supply service levels. Capacity building and technical support to Gram Panchayat Water Supply and Sanitation Committees (GPWSC) and also information, education, and communication (IEC) activities for targeted villages. Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) Subcomponent 2(c). Household sanitation (Grants for construction of household toilets). Construction of toilets by providing a household subsidy of INR 6,000 to an estimated 625,000 poor households, social mobilization and hygiene behavior change activities, along with monitoring activities. Component 3: Improved Water Quality. (US$59.38 million at appraisal, and US$85.76 million actual). This component financed: (I) water quality monitoring activities, and the development of a water quality monitoring and mitigation strategy; and (ii) water treatment solutions for contaminants such as arsenic, fluoride, and iron for an estimated 150 villages; (iii) and the construction of surface water supply schemes in affected districts such as Moga and Barnala with an estimated 121 villages. Component 4: Strengthening Institutions and Project Management. (US$49 million at appraisal, and US$29.28 million actual). This component financed: (i) the design and implementation of management information system (MIS) for District Water Supply and Sanitation (DWSS) units; (ii) capacity building and technical assistance to GPWSCs; and (iii) the provision of support for project management, including audits, M&E, technical assistance, training, and operating costs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The actual project cost is US$ 309.9 million, lower than the appraisal estimates of US$354 million. This is largely explained by the government requesting the cancelation of US$50 million from the loan. Project financing. At appraisal, the IBRD financing was estimated at US$248 million. This amount was reduced to US$198 million as requested by the government. The amount used was US$189.4 million, which was 95.7 percent of the reduced loan amount. The ICR (p. 14) indicated that the government requested the reductions because of slow implementation. Borrower contribution. At appraisal, the borrower’s contribution was estimated at US106 million. The actual amount contributed by the borrower was US$120.5 million at project closing. Dates. The Project became effective on June 19, 2015, and was completed by March 30, 2023, following three extensions that added 24 months to the project period. A Mid Term Review (MTR) was conducted on November 13, 2017. The project was restructured eight times:  On October 12, 2018, the first level 2 restructuring modified the results framework as a result in a change in components and their costs. This restructuring also reallocated funds between disbursement categories. At the time of restructuring, US$114.47 million was disbursed (60.4 percent) of the US$189.42 million.  On September 28, 2020, a second level 2 restructuring reduced the loan amount by US$20 million at the request of the government because of slow implementation. These cancellations required a reduction in funds allocated against components and disbursement categories, as well as PDO targets in the results framework. At the time of restructuring, US$135.65 million was disbursed (71.6 percent) of the US$189.42 million.  On January 21, 2021, a third level 2 restructuring reduced the loan amount by an additional US$30 million at the request of the government because of slow implementation. This restructuring resulted Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) in a slight reduction of the PDO target related to water quality. At the time of restructuring, US$135.65 million was disbursed (71.6 percent) of the US$189.42 million.  On March 31, 2021, a fourth level 2 restructuring modified the loan closing date by three months from March 31, 2021, to June 30, 2021. At the time of restructuring, US$135.65 million was disbursed (71.6 percent) of the US$189.42 million.  On June 9, 2021, a fifth level 2 restructuring was used re-allocate project funds to support the acquisition of emergency medical equipment and supplies for immediate COVID-19 treatment, and modifications to Component 4 with additional activities. At the time of restructuring, US$145.78 million was disbursed (76.9 percent) of the US$189.42 million.  On June 22, 2021, a sixth level 2 restructuring modified the loan closing date by an additional 15 months from June 30, 2021, to September 31, 2022. At the time of restructuring, US$148.31 million was disbursed (78.2 percent) of the US$189.42 million.  On December 10, 2021, a seventh level 2 restructuring modified allocations across components and modified some non-PDO targets in the results framework. At the time of restructuring, US$154.82 million was disbursed (81.7 percent) of the US$189.42 million.  On September 14, 2022, an eighth level 2 restructuring modified the loan closing date by six months from September 30, 2022, to March 31, 2023. 3. Relevance of Objectives Rationale The PDOs remained relevant to the World Bank’s Country Partnership Framework (CPF) FY18-25. The CPF’s Focus Area 3: Investing in Human Capital has five objectives of which one is dedicated to improving access to rural water supply and sanitation services contributing to health and economic welfare of the rural population. (CPF, p. 20) In addition, the PDO is aligned with the CPF’s approach to strengthen public sector institutions by improving capabilities within the state government and rural governing bodies, enabling development of accountability mechanisms, including support for digital platforms, as well as enabling citizen engagement channels, and strengthened planning, budgeting, and M&E. (CPF, p.32) The PDOs are aligned with and directly contributed to two of the government’s national programs: i) the ‘Jal Jeevan Mission’ (JJM) that aimed to provide 55 liters per capita per day (lpcd) to every rural household on a continuous and long-term basis by 2024 and ii) the Swachh Bharat Mission-Grameen (SBM-G) that aimed to eliminate open defecation in rural areas. The PDOs are also aligned with the State of Punjab’s policy on improved rural WSS service levels. The Project builds on lessons learned and scaled up activities supported by the preceding Punjab Rural Water and Sanitation Project (PRWSP; P090592; closed December 2014). This predecessor project was Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) developed in line with the State of Punjab’s decentralization policy to devolve functions and responsibilities from the central to the Gram Panchayat (GP) level. The PRWSP adopted a decentralized service delivery and community-driven development (CDD) approach with greater involvement of the GPs. The Project increased its ambition and scale and accounted for lessons learned from the predecessor project. (ICR, p. 11) Considering the needs in rural areas of Punjab, the alignment with national and state level programs and policies, and the increased ambition over the predecessor project, the relevance of the PDOs is rated as high. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve water and sanitation service levels in targeted villages in Punjab. Rationale The theory of change (ToC) in the Implementation Completion Report (ICR, p. 12) identified that the outcome of interest was to improve water and sanitation service levels in targeted villages of Punjab. The inputs identified to reach the water related outcomes included: (i) contracts to rehabilitate or newly construct water works (e.g. canals, tube wells, treatment plants/solutions, storage, distribution networks, and household connections); (ii) water quality laboratories and related water quality monitoring equipment, and studies; (iii) information, education, and communication (IEC) materials to improve hygiene behaviors; (iv) contracts to provide technical assistance for operation and maintenance staff for water supply. For sanitation related outcomes the inputs included: (i) contracts for sewerage treatments, networks, and household connections; and (ii) technical assistance for operation and maintenance of sewerage systems. As a result of these inputs, expected outputs would have been: (i) water works and sewerage works constructed; (ii) water quality laboratories constructed, procedures developed, geographic areas mapped; (iii) IEC materials developed and community activities executed for improved hygiene; and (iv) training and technical assistance provided to GPWSCs. As result of these outputs, expected intermediate outcomes would have been: (i) functioning water systems (production, transport, treatment, storage, distribution), and sewerage (collection, transport, and treatment) systems with household connections; (ii) water quality monitoring systems functioning and informing decision making; (iv) GPWSCs operating and maintaining systems, and (v) households practicing improved hygienic behaviors. The result of these intermediate outcomes would be the PDO outcome of increased access to improved water and sanitation service levels contributing to improved livability of villages and health outcomes. Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) The causal links between project activities, outputs, and intermediate outcomes were clear, and convincing and did not have any logical gaps. The indicators identified for the outputs and outcomes are specific, realistic and measurable. The key assumptions identified in the ICR were that there would be sufficient groundwater availability and willingness of rural households to pay for services. Outputs: The output level indicators for this objective in the results framework included:  579,071 new household water connections were established out of the original target of 724,100 connections. (not achieved)  529,183 piped household water connections that are benefiting from rehabilitation works undertaken by the project out of an original target of 619,330. (not achieved)  8,150 new household sewer connections were constructed under the project out of an original target of 84,969 connections. (ICR, p. 24) (not achieved) The ICR reported the following additional outputs that did not have baselines and targets and were not included in the results framework, but are related to inputs in the ToC:  72 damaged canal-based water schemes were rehabilitated.  493 new water supply schemes were built.  263 handpump schemes were upgraded to piped water supplies.  5 regional water quality testing labs were established in SAS Nagar, Sangrur, Amritsar, Moga, Hoshiarpur and promoted community-led surveillance at the scheme/village level. Outcomes: The outcome level indicators for this objective in the results framework included:  3,747 villages with higher water service levels managed by GPWSCs were achieved out of an original target of 6,556. These villages met the following four criteria simultaneously: i) scheme actively managed by GPWSC, ii) water supplied at least 8 hours per day iii) 70 liters per capita per day (lpcd) or more supplied, and iv) 95 percent of all households connected identified above. (not achieved)  4,465 GPWSCs that are managing O&M of water supply schemes through full cost recovery out of an original target of 5,841. (not achieved)  42 GPWSCs that are managing O&M of sewerage schemes through full cost recovery out of an original target of 412. (not achieved)  9,598 villages received improved water quality out of an original target of 9,304. (achieved)  8,279,434 direct project beneficiaries were reached out of an original target of 8,473,604. This target was not achieved. The project was able to achieve the reported total number of direct beneficiaries despite the other PDO indicators not being achieved because this indicator counted households that benefited from any of the following: (i) new piped water connection, (ii) an existing connection that benefited from rehabilitation, (iii) a new connection to sewerage and (iv) a new individual household latrine (IHHL). The household figures for IHHLs are reported under PDO 2 but are also captured under the total direct beneficiary indicator. Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520)  3,774,405 direct project female beneficiaries reached out of an original target of 4,028,305. This target was not achieved.  2,830,876 Social Caste (SC) beneficiaries reached out of an original target of 2,446,881. This target was exceeded. Low-income households significantly benefited from the project as it covered those households who lacked access to any type of water or sanitation service by expanding coverage within villages of water schemes or construction of household latrines. The ICR (pg. 16) reported the following additional outcomes that did not have targets and were not included in the results framework. These outcomes represent progress in targeted villages to improve the quality of water supply services in at least one of the four criteria considered for a higher level of service. As a point of reference, the original target for the number of villages with a higher level of service was 6,556:  5,965 villages that had water systems managed by GPWSCs (criteria i)  5,869 villages had achieved over 8 hours of regular supply (tubewell schemes) (criteria ii)  9,903 villages were supplying >70 lpcd. (criteria iii)  7,077 villages had achieved 95 percent water supply coverage. (criteria iv)  4,465 villages had 100 percent cost recovery for O&M of water supply systems  1,111 villages had canal-based water schemes operating over 6 hours per day The Project made progress but did not achieve either of the output targets related to household connections, nor did it achieve 5 out of the 7 original PDO or supporting intermediate outcome indicator targets. The only PDO targets that were achieved were the number of villages with improved water quality and social caste beneficiaries reached. The ICR reported a series of additional outputs not in the results framework that supported the results achieved for the PDO indicator. In addition, the ICR reported a series of outcomes that did not have specific indicators in the results framework but were directly related to some of the original PDO indicators. The objective to increase access to improved water supply and sewerage was modestly achieved against the original targets. The efficacy rating is thus Modest. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective There was no change in the objective. Revised Rationale At the second project restructuring, the original PDO remained unchanged but some of the PDO targets decreased in scope. Theory of change Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) The TOC that was provided under Objective 1 above applies to the revised Objective 1 as well. Changes made during the restructuring were to reflect modified targets that were aligned with reductions in the loan amount. Outputs:  579,071 new household water connections were established out of the revised target of 540,070 connections. (exceeded) (ICR, p. 24)  529,183 piped household water connections that are benefiting from rehabilitation works undertaken by the project out of the revised target of 492,150. (ICR, p. 24) (exceeded)  8,150 new household sewer connections constructed under the project were reported out of a revised target of 11,150 connections. (ICR, p. 24) (not achieved) See original Objective 1 for the additional outputs that did not have targets nor were included in the results framework. Outcomes:  3,747 villages with higher services levels managed by GPWSCs were achieved out of the revised target of 5,406. These villages met the four criteria identified above. (not achieved)  4,465 GPWSCs that are managing O&M of water supply schemes through full cost recovery out of the revised target of 5,301. (not achieved)  42 GPWSCs that are managing O&M of sewerage schemes through full cost recovery out of a revised target of 110. (not achieved)  9,598 villages receiving improved water quality out of the revised target of 9,757 at the second structuring. (not achieved)  8,279,434 direct project beneficiaries reached out of the revised target at the second restructuring of 7,377,984. This target was exceeded. This result is disaggregated by 3,184,891 people benefiting from new water connections, 2,706,825 people benefiting from rehabilitated water systems, 670,603 people benefiting from improved water quality interventions, and 1,717,117 million people benefiting from individual household toilets. The total number of beneficiaries was discounted by the number of people who gained access to sewerage and a percentage who gained access to individual household toilets to avoid double counting households that also benefited from water supply interventions. This disaggregation was not monitored through the results framework but was tracked by the client (ICRR interview).  3,774,405 direct project female beneficiaries reached out of the revised target of 3,870,539 at the second restructuring. This target was not achieved.  2,830,876 Social Caste (SC) beneficiaries reached out of the revised target of 2,360,481 at the second restructuring. This target was exceeded. See original Objective 1 for the additional outcomes that did not have targets nor were included in the results framework. The second restructuring decreased the PDO targets for the number of villages with higher levels of service managed by GPWSCs; number of GPWSCs that are managing O&M of water schemes through full cost recovery along with the intermediate outcome indicators for number of new piped household connections and household connections benefiting from rehabilitation works. The project exceeded the revised water output Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) targets and did not achieve the revised sewerage targets. The project made progress toward the revised outcome targets. The direct beneficiary target was exceeded because of improvements in specific characteristics of water supply (e.g., > 8 hours per day or liters per person per day or 70 lpcd). While the project did not achieve all the revised outcome targets, five out of the seven indicators were substantially achieved or exceeded resulting in an efficacy rating of Substantial with shortcomings in achieving revised sewerage targets. Revised Rating Substantial OBJECTIVE 2 Objective To reduce open defecation in targeted villages in the Punjab. Rationale The ToC in the Implementation Completion Report (ICR, p. 12) identified that the outcome of interest was reduce open defecation in targeted villages of Punjab. The inputs identified to reach the outcome included: (i) household grants to construct toilets, (ii) information, education, and communication (IEC) materials to improve sanitation behaviors; and (iv) technical assistance to train a cadre of social mobilization staff in the District Sanitation Committees and at the Gram Panchayat level. As a result of these inputs, expected outputs would have been: (i) household grants disbursed; and (iii) IEC materials developed, and community activities executed for improved hygiene along with social mobilization efforts at household and school level. As result of these outputs, expected intermediate outcomes would have been: (i) household toilets constructed and used, and (ii) hygiene behaviors improved. The result of these intermediate outcomes would be the PDO outcome of reduced open defecation contributing to improved livability of villages and health outcomes. The causal links between project activities, outputs and intermediate outcomes were clear and convincing. The indicators identified for the outputs and outcomes are measurable and logical. There were no key assumptions identified in the ICR for this PDO’s ToC, which was a shortcoming. Such assumptions would have been that household subsidies were distributed in a timely manner, behavior change communications (BCC) would be intensive at the household and school level, sufficient number of people would have been trained to deliver BCC interventions, and that there was an adequate supply chain of materials and technical support to help households design and construct toilets. Outputs: The output level indicators for this objective in the results framework included:  2,569,627 people were trained to improve hygiene behavior/sanitation practices out of an original target of 2,015,000 people. (exceeded) Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520)  1,233,421 females trained to improve hygiene behavior/sanitation practices out of an original target of 1,007,500 females. (exceeded).  456,533 latrines constructed out of an original target of 480,410 (not achieved). Outcomes: The outcome level indicator for this objective in the results framework included:  13,726 villages declared open defecation free out of the original target of 6,900. (exceeded). During the October 2018 restructuring the target was revised upwards to 13,726; therefore, achieving the revised upward target. The sanitation component of the project received priority early on in the Project because of the political attention given to the subject as a result of the SBM national program, which closed in 2019. Hence most of the results for ODF villages were achieved by 2018 and the end target was revised upward to reflect the achievements. This PDO is not considered under the split evaluation, because the PDO target was increased over time despite the output target decreasing. To achieve ODF status, households may have used their resources to construct toilets if they did not have access to the government subsidy. The household subsidy was used to track the number of toilets constructed through the project. The PDO was verified through a third-party entity required under the Swachh Bharat Mission guidelines (ICR, p 12). The ICRR interview also revealed that the PDO was evaluated using the gold standard of evaluations - a multiple-arm, cluster randomized control design to assess the short-run effects of the program in rural Punjab, India on the construction of toilets, the practice of open defecation, and beneficiaries’ hygiene-related awareness and practices. The mid-term survey of the evaluation found, “that the program was successful in increasing the coverage of safely managed toilets and led to a decline in open defecation among rural households in project intervention areas. A significant reduction in open defecation was observed among households that already had toilets. The effect was less pronounced among households without toilets, which were also the primary targets of the program.” An endline planned for October 2020 was not carried out because the control areas had been contaminated with project interventions. The open defecation free status of the state was evaluated through four different verification mechanisms. (ICRR interview) PDO 2 was a significant part of the project and was a high political priority for the federal and state government. The project was the main financing vehicle for PDO 2 to help the state achieve the targets set under the Swachh Bharat Mission. The central government did contribute to the subsidy that was given to the households for the toilets; however, without the project financing the state of Punjab would not have been able to achieve the results across the state. The evidence presented and the results achieved result in a High rating. Rating High OBJECTIVE 2 REVISION 1 Revised Objective No Change Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) Revised Rationale As above. Revised Rating High OBJECTIVE 3 Objective To strengthen service delivery arrangements in targeted villages in the Punjab. Rationale The ToC in the Implementation Completion Report (ICR, p. 12) identified that the outcome of interest was to strengthen service delivery arrangements. The inputs identified to reach the outcome included: (i) an institutional strengthening plan; (ii) development of a management information system (MIS) to monitor water supply; (iii) provision of office space and equipment; (iv) surveys and audits; (v) technical assistance to government staff on social mobilization, hygiene promotion for water and sanitation, and operation and maintenance; and (vi) financing for water quality monitoring and mitigation strategy. As a result of these inputs, expected outputs would have been: (i) new functions created in the District Water Supply and Sanitation (DWSS) units and terms of references for engineering staff revised to incorporate operation and maintenance of systems; (ii) new GPWSCs created at the Gram Panchayat level and trained on their management and operation and maintenance functions; (iii) MIS developed; and (iv) training and capacity building conducted; (v) water quality monitoring and mitigation strategy developed and laboratories built. As result of these outputs, expected intermediate outcomes would have been: (i) DWSS units have shifted focus from construction to supporting GPWSCs with operation and maintenance, (ii) improved monitoring of water quality and operation of water systems, (iii) GPWSCs are operating and maintaining systems and collecting tariffs, (iv) communities and villages are being mobilized to improve hygiene behaviors; (v) geographic areas with poor water quality mapped and water quality testing being implemented. The result of these intermediate outcomes would be the PDO outcome of strengthened capability by the government and communities to manage and deliver quality water and sanitation services leading to improved livability of villages. The causal links between project activities, outputs and intermediate outcomes were relatively clear. One gap was that operation and maintenance of sewerage systems did not appear to be part of the functions as the focus was on water supply schemes. The indicators identified for the outputs and outcomes are measurable and logical. The ToC did not have any critical assumptions, which was a shortcoming such the DWSS reforms to shift from a focus on construction to supporting GPWSCs with operation and maintenance would be accepted by the government agencies. Outputs: Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) The output level indicators for this objective in the results framework included:  Annual report published on state-wide rural drinking water quality status (yes). This was a binary target and was achieved.  DWSS organization transition plan for improved service delivery approved and implemented (yes). This was a binary target and was achieved. In addition to the achievements listed in the RF, the ICR also reported on the following achievements not identified in the RF and hence without target values:  Female members of each GPSWC were trained to use field test kits and conduct monitoring. (ICR, p. 11)  Institutional upgrades contributed to measurement of water quality at the village, district, and HQ levels with five accredited laboratories with infrastructure, systems, processes, and staff. Water quality data were published annually facilitating targeted interventions as needed. (ICR, p. 11)  Jal Seva mobile app developed by the project allowing water system users to log grievances, system managers to report key issues and water quality information, and for department staff to log scheme development information in real-time. (ICR, p 12)  400 field-level coordinators and social / IEC staff were hired to scale up capacity building efforts of GPWSC and community members. (ICR, p.13)  Establishment of the Punjab Rural Water Supply Utility as a State-owned Company to manage large multi-village schemes. At project closure this utility was in its infancy; however, it presents a shift in the operating modalities of rural water supply governance. Outcomes: The outcome level indicators for this objective in the results framework included:  22 districts with MIS operationalized and are carrying out monthly reporting of water system data out of a target of 22 districts. (achieved) The PDO 3 outcome indicator partly covers strengthened service delivery arrangements with respect to monitoring water supply services. However, a shortcoming in measuring this PDO is that the institutional arrangements for service delivery are not captured. Nevertheless, these aspects are captured under PDO 1 with the number of villages managed by GPWSCs that are operating with full cost recovery, which was substantially achieved. The output level targets captured elements of strengthened institutional arrangements. A better outcome indicator could have been community satisfaction with the way GPWSCs manage water supply systems. While the ToC and results framework have minor shortcomings the rating is a Substantial with moderate shortcoming. Rating Substantial OBJECTIVE 3 REVISION 1 Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) Revised Objective No change Revised Rationale As above. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The ICR documented the achievement of the output, intermediate, and outcome indicators. Due to the shortcomings in achievement of the project's PDO 1 it was rated as Modest, PDO 2 was rated as High, and PDO 3 was rated as Substantial with moderate shortcomings; therefore, the overall efficacy rating is a Substantial with moderate shortcomings. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale Under Revision 1, the overall efficacy is assessed as Substantial with moderate shortcomings based on underlying efficacy ratings of Substantial with moderate shortcomings for PDO 1, High for PDO 2, and Substantial with moderate shortcomings for PDO 3. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency Ex Ante Economic Efficiency: At appraisal, the economic rate of return (EIRR) was estimated at 24 percent with a Net Present Value of US$399.4 million for the investments based on various benefit streams for water that included: (i) benefits from time saved in water collection / improved access to household toilets, (ii) benefits from incremental demand for water, (iii) health benefits due to the reduction in incidence of diseases (diarrhea, hepatitis, enteric fever, dengue and malaria as well as cancer, and (iv) benefits from time saved due to lost Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) productive days on account of occurrence of diseases. (PAD p. 14) For reference, this NPV calculation used a discount rate of 10 percent. Ex Post Economic Efficiency: The economic analysis at project completion in the ICR included the same benefits streams identified at appraisal (ICR, p. 14). The EIRR at closing was estimated at 48.5 percent with a Net Present Value of US$506.4 million. The ex-post analysis also used a discount rate of 10 percent. The ex- ante analysis assumed 80 percent of project completion at close. Operational and Administrative Efficiency: The project was designed to be implemented over 6 years. Delays in implementation were a result of several factors outside of the control of the government that included: (i) the COVID-19 pandemic that put restrictions on contractors to conduct activities, (iii) flooding in 2023, and (iii) statewide strikes by farmers. Delays under the control of government included delays in the flow of funds from the state treasury to the project account, which impacted payments to contractors and contributed to delays in the physical progress of works. (ICR, p. 16) The large number of the interventions, high transaction costs, and the time intensive nature of reaching thousands of rural villages with interventions justified a 6-year implementation period. The delays because of COVID along with other factors were reasonable to extend the project by 24 months given the scale and scope. As a result, disbursements reached US$ 82 million (43 percent) by the time of the MTR in November 2017. These challenges led to a number of actions including: (i) canceling US$ 50 million of the loan proceeds, (ii) reductions in scope of water supply works, and (iii) an extension of the project closing date by 24 months. Conclusion. The project's estimated EIRR at 48.5 percent at project close was a significant result compared to a range of 10-12 percent discount rate suggested for South Asian countries in the World Bank economic analysis guidelines. While the project required a 24-month extension, the project doubled the targets for open defection free villages and substantially achieved the revised desired results for household toilets. The cost of building household toilets was cheaper per unit costs than originally anticipated. The high EIRR is driven by time savings for accessing water (approximately 51 percent of the benefits) and time saving accessing a toilet (approximately 44 percent of the benefits), which carries a lower cost than water supply interventions. The EIRR is high and could be an overestimate. Inquiry with the task team revealed that model assumes 100 open defecation free (ODF) status as per the verification; however, sensitivity analysis of the EIRR suggests that if that if ODF was 50 percent then the EIRR would be reduced to 30 percent. While the EIRR could be overestimated, it is still likely to be significantly higher than the 10-12 percent discount rate for the country. Given the EIRR and the extended time to implement the project, the efficiency rating is Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 80.00 Appraisal  24.00  Not Applicable Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) 97.00 ICR Estimate  48.50  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The PDO was highly relevant to the government and Bank strategy for the state of Punjab and the national government programs. The overall efficacy of the three PDOs was rated as Substantial with moderate shortcomings given the supporting evidence that the targets were almost fully achieved, and several outputs were achieved or exceeded the targets. Efficiency was rated as Substantial. The economic analysis showed that the project had a high EIRR. The project benefited from higher targets for PDO 2 to reduce open defecation. The overall outcome, accounting for the split evaluation, was rated Moderately Satisfactory, reflecting moderate shortcomings in the Project's achievement of its objectives and in its efficiency and extended period of implementation. The table below shows a comparison between the without restructuring and with restructuring in September 2020, which upgraded Split 1 for PDO 1 from Modest to Substantial accounting for progress made against the PDO targets and the achievement of the outcomes. With Without Restructuring Restructuring (Sept 2020) Relevance High High PDO Objective 1 – Improved Water Supply and Modest Substantial Sanitation Supply Service Levels PDO Objective 2 – Reduced Open Defecation High High PDO Objective 3 – Strengthened institutional capacity Substantial Substantial Efficacy Substantial Substantial Efficiency Substantial Substantial Moderately Outcome Satisfactory Satisfactory 4 Numerical value of outcome rating 5 Amount disbursed (out of US$189.4 million total US$135.65 US$53.73 disbursed) Share of disbursements 72% 28% Weighted value of outcome ratings 2.88 1.4 2.88+1.4 = 4.28 rounding to 4; Final Outcome rating Moderately Satisfactory a. Outcome Rating Moderately Satisfactory Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) 7. Risk to Development Outcome There are three sector key related risks that are beyond project influence and may impact continued operational, financial, and social sustainability of outcomes if not addressed: i) absence of policy on metering and tariffs, and conflicting tariff policies for rural water supply in the State as outlined in the ICR. In addition, DWSS will need to: ii) secure sufficient and well-trained staff at DWSS and the newly created rural utility for carrying out social, IEC and engineering activities to support a larger number of schemes; and iii) in the mid- to long-term, water resources availability constraints due to Punjab’s increasing groundwater depletion concerns particularly in agricultural areas that may lead to competing water demands and impact the transition to 24/7 supply schemes. (ICR, p. 19) 8. Assessment of Bank Performance a. Quality-at-Entry This project built on lessons learned from the predecessor project - Punjab Rural Water and Sanitation Project and incorporated lessons into the design. The Bank had a high level of buy-in from the state over the project design because of its experience with the predecessor project, and alignment with the national and state programming. The project provided resources and technical assistance and served as a vehicle for the state to implement the national water and sanitation programs. The adequately assessed associated risks, despite minor shortcomings such as challenges eventually experienced with transfers from the state treasury to the project. The quality at entry included sound project concept, objectives, and approach; adequate technical, financial, and economic analysis; adequate consideration of environmental and social safeguards, fiduciary assessments – with shortcomings, and high relevance to the policy and institutional context, and sound implementation arrangements. Quality-at-Entry Rating Satisfactory b. Quality of supervision The Bank conducted a total of 18 supervision missions, which is an average of three per year. In addition, the team conducted several technical missions, which helped in early identification and resolving emerging issues. Supervision missions consistently included specialists in procurement, financial management, social and environmental safeguards, and other subject matter specialists. The number of restructurings undertaken reflected the team’s responsiveness to the client, and their ability to adapt during implementation. Under the first restructuring several outcome targets were increased, but as project implementation continued and slowed down the targets were revised downward to reflect the reality on the ground. The Bank team correctly identified lagging performance necessitating cancellations and associated changes in the indicator targets and adopted a phased approach while processing extensions to ensure the Project’s overall success. The aide memoires were explicit and identified weaknesses in implementation, and the Bank team was proactive in addressing bottlenecks with action plans and were responsive to Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) federal and state government requests for restructurings. The Bank team was diligent in monitoring results of the project and proactively identified weaknesses resulting in improvements by the government and improved quality of information over time. A shortcoming in supervision was that the ICR team observed disparities in figures for the original targets and the restructured targets as cited in the restructuring paper and the 14th and 15th supervision reports. (ICR, p. 19) Also, there were some adjustments that were made in tracking the PDO-1 indicator “Villages with higher service delivery levels managed by the GPSWCs” that were not reflected in results framework. The ICR accounted for this discrepancy and only reflected the results that adhered to the results framework definitions of higher service standards. The Bank team provided dedicated support to the client and was proactive in identifying and resolving challenges as demonstrated through the number of restructurings demonstrating adaptability by the Bank team. The team conducted regular missions and provided technical support at all levels, which helped overcome moderate shortcomings in M&E design and gaps in project reporting and documentation. The overall rating of World Bank performance is Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The results framework adequately captured the outputs, intermediate outcomes, and outcomes. PDO 1 indicators for water supply were ambitious in nature requiring a higher level of service compared to the predecessor project. The PDO 1 indicators did not capture the sanitation aspects, but these were captured at the intermediation outcome level. The project depended on robust monitoring and evaluation systems during implementation. This included a baseline survey to help set the initial targets, and a gold standard impact evaluation to measure PDO 2. While the endline of the impact evaluation was not carried, the mid- line evaluation demonstrated the project was having a positive impact on reducing open defecation. The management information system supported by the project to collect digital information on an array of parameters for water supply schemes was instrumental. PDO 2 was also monitored through the government’s SBM program including verification protocols for achieving open defecation free status. The indicator for PDO 3 on institutional strengthening was measured through output type indicators rather than outcome indicators because it focused on whether the MIS system was operational. Overall, the M&E design was adequate. b. M&E Implementation The M&E implementation was done in an adequate manner with some moderate shortcomings. The Bank team provided close support during implementation and strengthened the capacity of M&E staff in the project implementation unit. The Bank staff conducted an average of 3 supervision missions per year Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) (total of 18 over six years) along with a series of technical meetings. The close supervision by the Bank team helped identify weaknesses in the data that was being captured in the MIS prior to the mid-term review. As a result, the scrutiny by the Bank staff helped improve the quality of the information over time that was being reported in the MIS and field level data collection efforts. The development and application of the Jal Seva app that was implemented at the district and Gram Panchayat levels helped improve the quality and quantity of data being reported and used for decision making. (ICR, p. 17). The Bank team was proactive in working with the government to adjust targets to reflect adjustments to the project financing. The project relied on third party verifications required under the Swachh Bharat Mission for PDO 2. In addition, the Bank team used trust funds to finance a randomized control impact evaluation for PDO 2. (ICRR interview). The baseline and mid-term evaluations were completed and showed that the project was having an impact. The endline for the impact evaluation was not carried out because the control areas were contaminated with project interventions. The project also conducted an endline survey using a qualitative methodology. A quantitative survey for the endline to capture PDO 1 elements such as time saved would have better supported the measurement of the benefits for PDO 1. Overall, the M&E implementation was modest. c. M&E Utilization The M&E system was proactively used by the project implementing unit and the Bank team during implementation to track project indicators. The Bank team closely monitored procurement and tracked physical progress of infrastructure installation, which informed the decision to undertake restructuring and adjust indicator targets in the results framework to reflect changes in the project. The overall quality of M&E is rated substantial. The M&E system adequately captured the achievement of the development objective of the project. Various methods were used during the life of the Project to monitor outputs and outcomes. The overall M&E utilization rating is substantial. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The Project was designated as Category B. Environmental and social risks were deemed high because of the large-scale tunneling activity. The safeguard policies triggered were: (i) Environmental Assessment (OP/BP 4.01), (ii) Forests (OP/BP 4.36) and (iii) International Waterways (OP/BP 7.50). For the latter, an exception under paragraph 7(a) of OP 7.50 was obtained following an assessment by the project team. During project implementation, the Involuntary Resettlement (OP/BP 4.12) safeguard policy was triggered to ensure compliance of construction works for five multi-village schemes. An Environmental Management Framework (EMF) and Safeguards Management Action Plan (SMAP) were developed, and environmental management provisions were included in bid documents for key activities. A shortcoming was that the Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) project experienced under-staffing in social and environment teams throughout the life of the project. One factor that mitigated the shortage of staff in the social and environment teams was that the social staff dedicated to behavior change communications activities conducted contributed to understanding of environmental management issues in WSS at the field level. At closing, the overall Safeguards rating was Moderately Satisfactory (MS). b. Fiduciary Compliance Financial Management: The project complied with the Bank's financial management requirements. A salient feature of the project was that a significant portion of the expenditure was incurred at the community level through GPWSCs with capacity support and guidance from the DWSS. This arrangement was supported by a concurrent audit mechanism that worked efficiently. The project experienced challenges that included: (i) high and long-standing state auditor general disallowances, (ii) pending audit objections and (iii) backlog in payment of contractor bills which resulted from shortages of funds and state fiscal pressures. Finance staffing vacancies at a few divisions were a concern throughout the project, but the work was handled by finance officers from other divisions in DWSS. The financial management rating was downgraded to moderately unsatisfactory during the life of the project in 2021 and was subsequently upgraded when the issue of disallowances was addressed and closed with a moderately satisfactory rating. (ICR, p. 18) Procurement: The Project introduced a common procurement framework to streamline community-based procurement (CBP) for small single village schemes. Procurement was rated “Moderately Satisfactory” until a downgrade to “Moderately Unsatisfactory” in January 2022. The downgrading was due to high staff turnover leading to delays and deficiencies in complying with procurement procedures regardless of the contract type despite several Bank-led training courses. At project closing, this was upgraded back to Moderately Satisfactory upon resolution of delays in closing financial accounts and in paying contractors. (ICR, p. 18) c. Unintended impacts (Positive or Negative) None d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) Bank Performance Satisfactory Satisfactory Quality of M&E Modest Substantial Quality of ICR --- Substantial 12. Lessons  Strong citizen engagement and behavior change communication activities are critical to changing behaviors in water and sanitation projects. The premise of the project was to change behaviors around sanitation and hygiene habits, particularly reducing open defecation. This entailed massive social mobilization efforts at the village level, coordinated with other behavior change communication efforts through different media platforms. The Project demonstrated through its randomized control evaluation that the behavior change communication efforts in program had an effect on reducing open defection.  CDD interventions can be impactful when institutionalized in government’s approach and programming. The CDD approach in the state of Punjab appears to have strengthened service delivery arrangements at the community level due to their level of engagement and ownership over the installed infrastructure. This project demonstrated the link between strengthened capacity of GPWSCs and improved access, reliability, and efficiency of water services. The Project’s investments in social outreach through well-structured IEC activities and dedicated staff were critical in building the GPWSCs capacity and thus in achieving project outcomes as evidenced by the endline survey results. (ICR, p. 20)  Measuring service delivery outcomes at the Department and village levels was critical to shift from a focus on construction towards service delivery. Similar to the paradigm shift in sanitation, there was a shift for water supply as well. The project was measured in terms of access and quality of water services delivered, and not on the number of physical assets created (connections or length of pipes installed). This shift in focus from asset creation to performance and reliability is important for long-term sustainability. Strengthening the district functions to support community management through GPWSCs empowered communities to operate and manage this infrastructure with technical backstopping from government. The focus on monitoring and evaluation increased and accountability and was central in shifting the focus of the DWSS from a civil works agency to that of a service delivery agency, and beneficiaries had the ability to drive local decision making on water availability. (ICR, p. 20)  Beneficiary led construction of IHHL assets contributed to greater ownership, use and improved maintenance of assets created. Historically, the government of India had taken a supply driven approach to toilet construction, which went unused by households. As a result, a change in paradigm shifted from supply driven to demand driven, which was embodied by the SBM national program and implemented through this project. A demand driven approach with BCC interventions and coupled with financial support and household led construction were key factors in high achievement of sanitation outcomes as beneficiaries were able to choose location, design, and construct the IHHLs in a manner that was suitable to each household. Their contribution either in the form of cash or labor in Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Punjab Rural Water Sector Imp Project (P150520) construction resulted in a strong sense of ownership and establishment of satisfied services in terms of utilization, good quality, and continued infrastructure asset maintenance over a long period. (ICR, p. 20) 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was well-written, concise, provided a good analysis, and did a good job of documenting the results and data sources. The narrative was clear and supported the ratings. The ToC was adequate as it captured the overall logic of the project. The documentation of data sources that were provided in the main body of the ICR or in the footnotes increased the robustness of the document. The ICR captured the proactive nature of the Bank team in supporting the client with course corrections, appropriately updating the results framework through restructurings – while remaining within the framework of the ToC. a. Quality of ICR Rating Substantial Page 21 of 21