Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) Report Number : ICRR0022074 1. Project Data Project ID Project Name P127435 VN-Rural Water Supply & Sanitation (P4R) Country Practice Area(Lead) Vietnam Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51760,TF-13061 31-Jul-2018 178,624,019.27 Bank Approval Date Closing Date (Actual) 01-Nov-2012 31-Jul-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 8,319,208.01 Revised Commitment 204,787,900.53 7,065,108.54 Actual 175,684,102.71 4,945,000.89 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero Fernando Manibog Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives According to the Financing Agreement (FA, p.5) and the Program Appraisal Document (PAD, paragraph 20), the Program Development Objective (PDO) of the Third Phase of the National Target Program for Rural Water Supply and Sanitation was "to increase sustained access to water supply and sanitation services and improve sector planning, monitoring and evaluation in the Participating Provinces." This review will assess the following objectives: • To increase sustained access to water supply services in the Participating Provinces • To increase sustained access to sanitation services in the Participating Provinces Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) • To improve sector planning and monitoring and evaluation (M&E) in the Participating Provinces b. Were the project objectives/key associated outcome targets revised during implementation? No c. Components 1. Increase access to water supply services: (US$56.8 million at appraisal, US$56.2 million actual) This component would finance the construction of commune-level water supply schemes such as treatment plants, distribution networks, and household connections. 2. Increase access to sanitation services: (US$21.4 million at appraisal, US$20.6 million actual) This component would finance the construction of hygienic latrines in commune households, schools, and health centers. 3. Information, Education, and Communication (IEC) and Behavior Change Campaigns (BCC) (US$9.9 million at appraisal, US$8.7 million actual) This component would finance campaigns to disseminate information about the program, raise awareness to program benefits, and facilitate the modification of rural residents' attitudes and behavior toward connecting, using, and maintaining their water and sanitation systems. 4. Capacity Building: (US$6.0 million at appraisal, revised to US$4.8 million, US$4.9 million actual). This component financed activities outlined in the the Program Action Plan (PAP), provided implementation support, and developed a results-based verification and disbursement process. The PAP identified the following actions: (i) transparency and citizen engagement, (ii) procurement, (iii) financial management and auditing, (iv) land acquisition, and (v) ethnic minorities. Implementation support financed Bank supervision, capacity building, technical expertise, and communication support to implementing entities at both the national and provincial levels. Results-based verification and disbursement process financed the baseline survey and verification protocol. After implementing agencies reported annual achievements against the six Disbursement Linked Indicators (DLIs), these were verified by an independent verification agent through document and physical audits. Annual disbursements were then made against the verified results. In addition to Bank financing, a parallel grant financing from the Australian Agency for International Development (AusAID) also supported technical assistance activities to address capacity gaps in the following areas: (i) strengthening the planning and monitoring capacity of implementing agencies, including transparency and social accountability; (ii) improving financial management, operations and maintenance (O&M), and water quality management practices of implementing agencies that contribute to the sustainability of water supply systems and services; (iii) improving the capacity of implementing agencies for managing sustainable hygienic sanitation, developing sanitation actions plans, and a community rewards mechanism; (iv) improving governance such as fiduciary, environmental, and social practices; (v) strengthening the capacity of the State Audit of Vietnam (SAV) to conduct annual verification of program results reported by the provinces and external audits; and (vii) support the Rural Water Supply and Sanitation Partnership by financing workshops and sessions to share lessons learned and emerging best practices from the program. Annex 5 of the ICR reported on the outcomes of this parallel financing. Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Program Cost: The original program cost reached US$206.04 million, consisting of US$200.0 million (equivalent to Special Drawing Rights credit of 131.5 million) and a grant of US$6.04 million. The project disbursed US$175.68 at closing. The difference of US$30.36 million was reported as savings because the private sector injected resources during implementation. Financing: The program was financed by a credit from the International Development Association (IDA) through its Program for Results (P4R) financing mechanism. The Government of Australia's Agency for International Development (AusAID) provided additional parallel grant financing. Borrower Contribution: The Government committed to contribute US$60.0 million and disbursed US$27.0 million at closing. Dates: The program was approved on November 1, 2012 and became effective on April 24, 2013. The Mid Term Review (MTR) was conducted on September 15, 2015. The program was originally scheduled to close on July 31, 2018 but was extended by 12 more months to close on July 31, 2019. There was one restructuring on June 16, 2016 to introduce changes to the Results Framework and extend the project closing date. The Disbursement Linked Indicators (DLI) matrix and Bank disbursement arrangements were modified as a result. The main changes to the Results Framework were (see Section 4 Efficacy below): • Adding a Key Outcome Indicator - number of direct project beneficiaries, of which percent were female • Adding an Intermediate Results Indicator - (iv) number of communes where schools and health clinics have maintained hygienic status • Separating the (DLIs) to better match sector specific targets with the achievements for water and for sanitation • Adding a new sanitation sustainability DLI - number of communes where schools and health clinics have maintained hygienic status 3. Relevance of Objectives Rationale This program's objectives were highly relevant to the government's Rural Water and Sanitation System (RWSS) sector strategy. In 2000, the government set the overall vision and goals for its 2020 National Rural Clean Water Supply and Sanitation Strategy and established the National Target Program (NTP) for Rural Water Supply and Sanitation across all 63 provinces of Vietnam. By 2011, the government designed the Third Phase (2010-2020) of the NTP for the New Rural Development (NRD) consisting of: (i) an increase in demand driven investments; (ii) making operations and maintenance (O&M) more sustainable; and (iii) building the capacity of provincial agencies and service providers. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) In 2014, in collaboration with the World Bank, Australia, and the United Kingdom's Department for International Development (DfID), the government initiated Vietnam 2035 as a supplemental road map for economic, political, and social reform called Vietnam 2035. The objectives were also relevant to four of the six areas in that road map: (i) enable economic modernization where the private sector is in the lead; (ii) build technological and innovative capacity; (iii) chart an environmentally sustainable development path and enhance climate resilience; and (iv) promote equality and inclusion of marginalized groups, and develop policies to promote a harmonious middle-class society. The objectives were also highly relevant to the World Bank's Country Partnership Framework (CPF) for Vietnam for the period 2018-2022. This PforR program contributed to meeting the following three CPF objectives -- CPF Objective 5 -- Broaden economic participation of ethnic minorities, women, and vulnerable groups; CPF Objective 10 -- Increase climate resilience and strengthen disaster risk management; and CPF Objective 11 - Strengthen natural resource management and improve water security. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective •To increase sustained access to water supply services Rationale The Theory of Change: The program supported investments in rural water supply services using best practices to generate sustainable water supply by constructing commune level water supply schemes. These schemes included treatment plants, distribution networks, and households connections. Achieving targets triggered disbursements against the allocations for this objective. Disbursements then led to achieving the increase in access to water services and supported the sustainability of water supply schemes. The key outcome indicators were: • Number of people with water supply connections from sustainable water systems • Number of new functioning water supply connections OUTPUTS: • 399,964 new functioning water supply connections were made (original target 340,000, target exceeded) • 1,559 new water supply and sanitation facilities in schools, and health centers were constructed and in use (original target, 1,440, target exceeded) Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) •179 communes (original target 150, target exceeded) where schools and health centers have maintained hygienic status OUTCOMES: • 1,054,575 beneficiaries have water supply connections from sustainable water system (original target 850,000 target exceeded). The DLI for new functioning water supply connections outcome was US$55.2 million against an allocation US$56.8 million. The DLI for the number of people with water supply connections from sustainable water system was US$9.9 million against an allocation of US$12.8 million. The evidence for the increased access to water supply system was provided by the number of new beneficiaries and new connections built (399,964 against a target of 340,000, target exceeded) • 1,759,842 beneficiaries from the rural areas have access to improved water sources (original target 1,496,000, target exceeded) • 2,149,765 direct beneficiaries, 51 percent were women (original target 1,861,000 target exceeded, of which 51 percent were female, target exceeded). SAV independently verified the sustainability of 80 water supply schemes by confirming that: • the schemes complied with the Ministry of Health (MOH) clean water standards; • at least 70 percent of the planned water connections were working 24/7, with a monthly average of 10 percent failure rate, using bills issued and paid as evidence; • tariff revenues covered the cost of operations and maintenance (O&M); • non-revenue water (NRW) was less than 25 percent; and • the water scheme used was an approved management model (public utility, private enterprise, or cooperative). • DLI1 was achieved with 144 functioning water service connection (FWSC) schemes and 359,581 new connections . • DLI3 (people with connections to sustainable systems) was achieved with 105 water schemes operating for at least 2 years. A sampling of 80 schemes representing 318,275 new connections were sustainable. With the above outcomes, achieving this objective was rated high. Rating High PHREVDELTBL PHEFFICACYTBL Objective 2 Objective •To increase sustained access to sanitation services Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) Rationale The Theory of Change: The program supported investments to support increased access to rural water sanitation such as construction of hygienic latrines in commune households, schools, and health centers and behavioral change campaigns. These investments used best practices to ensure sustainability. Achieving targets triggered disbursements against the allocations for this objective. This action achieved the objective to increase access to sanitation services and support the sustainability of community wide sanitation schemes. The key outcome indicators were: • Number of people with access to commune-wide sanitation services, disaggregated by gender • Number of newly constructed improved household sanitary latrines • New of new latrines constructed and in use in schools and health centers • Number of communes where schools and clinics maintained hygienic status OUTPUTS: • 142,280 of newly constructed improved household sanitary latrines (original target, 130,00, target exceeded). The DLI for this indicator was US$20.6 million against an allocation of US$21.4 million. • 1,492 new latrines were constructed and in use in schools and health centers (original target, 1,440, target exceeded) • 179 communes (original target 150, target exceeded) where schools and health centers have maintained hygienic status. The DLI for this output indicator was US$3.1 million against an allocation of US$4.7 million. • 600 training events were conducted in 238 communes, with over 2 million participants. Training covered latrine maintenance and positive WASH messages. In schools, students participated in start of the year lessons on WASH benefits and practices, training on how to use latrine facilities in schools and at home. The Vietnam Health Environment Management Agency (VIHEMA) coordinated with the commune, district level health agencies, social organizations like the Women's Union, to educate commune residents, assign maintenance responsibilities, conduct the campaigns, monitor the communes for compliance, and coordinate verification of achieving CWS status. • Over 300,000 leaflets, 500 posters, and regular loudspeaker announcements were produced annually to promote clean water, sanitation, and improved behavior in "Water Access, Sanitation, and Hygiene" (WASH). OUTCOMES: • 1,407,669 beneficiaries from the rural areas had access to community wide sanitation (original target 1,275,000, target exceeded). DLI4 (the number of people with access to commune-wide sanitation) disbursed US$21.4 million against an allocation of US$26.0 million. • 203 communes participated with 191 meeting Commune Wide Sanitation (CWS) status. A CWS status met the following conditions: (i) 100 percent of public kindergarten, primary, and secondary schools and Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) 100 percent of commune health centers met MOH standards for clean water and hygienic sanitation facilities (clean water and soap for hand washing); (ii) at least 70 percent of households met MOH standards for hygienic sanitation: (iii) 100 percent of households use latrines; and (iv) the commune was free of open defecation • 89 percent of households in the communes had hygienic sanitary latrines • 100 percent of schools had hygienic sanitation facilities, achieving target • 100 percent of health centers had hygienic sanitation facilities, achieving target CWS covered households, schools, and health centers and was used to achieve higher rates of sanitation (DLI4) over a wider area. DLI5 targets required the use and maintenance of sanitation facilities in schools and health centers to contribute to long-term sustainability. Long-term sustainability factors differed between households and schools and health centers. Communes were given financial incentives to assign responsibility and accountability for maintaining latrines. There were no sustainability concerns for latrines of commune households because a Bank-funded 2015 study (Sanitation and Hygiene in the Northern Mountains and Central Highlands of Vietnam by the London School of Hygiene and Tropical Medicine) indicated that household latrines in rural Vietnam were properly used and maintained. In addition, design of household latrines were sized to be emptied after more than 5 years. Households contracted private sludge operators to empty their septic tanks. Families used savings or accessed short-term subsidized loans to construct household sanitary latrines. Informal interviews with households from five communes in five provinces confirmed their familiarity with septage management and cleaning to support positive trends in maintenance of sanitation services (ICR, paragraph 34). As a result, achieving this objective was rated high. Rating High PHREVDELTBL PHEFFICACYTBL Objective 3 Objective •To improve water supply and sanitation services sector planning, and monitoring and evaluation (M&E) Rationale The Theory of Change: The program introduced a previously non-existent planning process for water supply and sanitation services in the participating provinces. The activities strengthened the planning skills of provincial implementing entities, and improved their monitoring and evaluation (M&E) systems to produce higher quality planning documents. The jointly developed Program Action Plan (PAP) identified these activities based on best practices to ensure that the country systems complied with fiduciary and social development policies: (i) transparency and citizen engagement, (ii) procurement, (iii) financial management and auditing, (iv) land acquisition, and (v) ethnic minorities. The Bank provided Implementation support, supervision, capacity building, technical expertise, and communication support to the national and provincial implementing agencies. In addition, the program supported raising awareness of the benefits from the program through information and education campaigns and modified the behavior of beneficiaries through Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) behavioral change campaigns. These activities were designed to influence the behavior of the rural beneficiaries to connect, use, and maintain the delivered rural water supply and sanitation services. The key outcome indicators were: • Number of provincial annual plans and program reports from participating provinces • Number of disclosed Ministry of Agriculture and Rural Development (MARD) program reports OUTPUTS: • 102 provincial annual plans and program reports from participating provinces (original target 85, revised target 102, target achieved). • 6 MARD program reports were disclosed to the public (original target 5, revised target 6, target achieved) OUTCOMES associated with the Program Action Plan: • The DLI for the number of provincial annual plans approved by participating provinces and the number of program reports disclosed to the public was US$8.7 million against an allocation of US$9.9 million. • Achieved the installation of the regularly updated transparency platform, except for Hung Yen province. Call centers were fully operational and served as the beneficiary feedback system. Participating provinces enhanced transparency with public consultation, corruption, fiduciary, procurement, program databases on implementation complaints and responses. • MARD and provinces prepared procurement reports where • there were no bid or proposal that was rejected due to offered price exceeding the cost estimate. • no state owned enterprises (SOEs) participated in the works that generated results for the Program, meeting target. • no debarred firms on the local/national/Bank debarment list participated in the works that generated results for the Program, meeting the target. • at least 50 percent of consultancy contracts and 50 percent of works contracts for water supply sub- projects were competitively bid, increasing to 80 percent by project closing. • Audit reports were delivered on time and of satisfactory quality, achieving the target. • Land acquisition, compensation, and resettlement were successfully completed in all project provinces with no issues reported. • The Ethnic Minorities Development Plan was prepared and implemented by PCERWASS in the two provinces that had ethnic minority people. All participating provinces prepared annual investment plans. MARD and all provinces prepared an annual report and disclosed these to public. Reporting served as an incentive for provinces to plan for budgeting, procurement, and implementation. Strengthened capacity at the provincial level was evident from the reduced works implementation time from 4 to 2 years. Independent verification offered incentives to comply Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) with higher technical standards and improve construction quality. This objective achieved substantial outcomes. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale Overall Efficacy Rating --- 5. Efficiency --- Efficiency Rating --- a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objective was rated high. The efficacy of all the first two objectives was rated high while the efficacy of the third objective was rated substantial. The overall efficacy was rated high. There is no rating for efficiency in a PforR financed program. Program efficiency is embedded in the Program Operations Manual and the implemented Program Action Plan that provides evidence in achieving targets to meet Disbursement Linked Indicators. According to the guidelines, the project outcome is rated highly satisfactory. Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) a. Outcome Rating Highly Satisfactory 7. Risk to Development Outcome The following pose risks to development outcome: • Technical Risks: Delivering rural water supply at the commune level was assessed to be sustainable after operating for a period of only two years. Sustainability was reflected in meeting the operations and maintenance (O&M) needs of the rural water supply schemes. With over 60 percent of the water schemes now operated by private enterprises who would presumably focus on cost recovery, there is a risk that O&M costs may not remain stable to be fully covered by revenues in the long term. At closing, program supported water schemes had an average cost recovery rate of 1.18 (ICR, paragraph 72). This risk was mitigated by the M&E system that had adopted sustainability factors in the rural water and sanitation sector and the strong foundation provided by the experience in the use of disbursement linked indicators to trigger financial flows. • Financial Risk: Keeping rural water tariff rates low may not not keep up with the costs of producing piped water supply. This tendency posed risks to the overall financial sustainability of water supply schemes. Provincial People's Committees establish tariffs at the provincial level and may not be sensitive to market forces. To mitigate this risk, schemes supported under the program promoted higher connection rates, increased consumption, and resulting increase in revenues. In addition, behavioral campaigns and outreach on benefits are expected to raise awareness to paying for services.T here may be need to explore a framework of subsidies for poorer residents. • Government Ownership and Commitment Risk: There is risk that government commitment to sustaining the program outcomes may not be retained over the long term. This risk would be mitigated by continuing dialogue between regulators at the provincial level, local rural water agencies, and water system operators. • Environmental and Legal and Regulatory Risks: Remaining gaps in the access to sustainable rural water supply and sanitation pose a risk to the outcomes of this program. In addition, the acknowledged weak regulatory framework that covers fecal sludge management needs to be addressed. These risks were mitigated by the plan of the government to expand the applicability of the sustainability factors in the nationwide M&E system to the rest of the country and pursuing other similar results-based programs in the future. • Macroeconomic Risk. The COVID-19 pandemic in 2020 indicate bleak health, economic, and social consequences worldwide, and poses high risks to sustaining the outcome achieved by the program. Severe impact on the economy is expected. There would be pressure on prioritizing the spending for health rather than other sectors. Disruptions to employment and income earnings in rural areas may pose risks to the willingness to pay water and sanitation tariff rates. Uncertainty with maintenance needs would negatively affect long term sustainability in improvements introduced under the program. The difficult and uncertain Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) global and national macroeconomic environment in the near- to probably medium-term pose high risks to sustaining outcomes achieved in this program as well as in the wider public investment needs. 8. Assessment of Bank Performance a. Quality-at-Entry The Bank used the PforR financing mechanism for the Rural Water Supply and Sanitation Project as a strategic approach to support the implementation of the Third Phase of the National Targeted Program (NTP3) for Rural Water Supply and Sanitation in Vietnam. The approach effectively tackled the weaknesses of the NTP3. The technical, financial, economic, and safeguards aspects of the PforR approach assessed the use of the country's own systems and adequately identified capacity and institutional gaps. The Bank team obtained grant financing to strengthen the program's TA. The M&E arrangements were well structured, using the national M&E system and identified indicators for improvement.The disbursement linked indicators were well defined. An independent body verified targets achieved. This was the first PforR financing instrument used by Bank in the water sector and the first time to be implemented in Vietnam. All program elements were reflected in the Results Framework. The program was aligned with key government priorities.The Bank team used the PforR approach anchored on the NTP3/NRD Bank performance was highly satisfactory at entry. Using country systems, the capacity gaps were well identified and adequately supported by the parallel TA program. Program preparation included clear and concise PAD and PAP documents that adequately addressed the sector and institutional challenges and risks. Quality-at-Entry Rating Highly Satisfactory b. Quality of supervision The Bank conducted regular supervision missions and communicated program progress, identified emerging technical issues issues, and recommended corrective measures in a timely manner. Supervision focused on building the implementing agencies' capacities and meeting all DLI targets. The Bank proactively monitored environmental and social safeguards compliance issues. In the first years of implementation, when the TA was delayed because of slower than expected procurement, the Bank team provided capacity building to help provincial agencies to prepare the annual investment plans and other activities. In particular, the Bank provided timely and relevant analyses during the MTR and restructuring. Implementation bottlenecks were identified with recommended corrective measures. When progress in the sanitation DLIs were lagging in some provinces because latrine costs were too high, the Bank promoted new low-cost technologies for rural household latrines, as an evidence of the results focus. The performance of the Bank was highly satisfactory. The Bank team provided high quality support and proactive, strategic advice to technical and management issues. Following the government's satisfaction Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) with the flexibility and fast disbursement features of this program, the government pursued four additional PforR operations. Quality of Supervision Rating Highly Satisfactory Overall Bank Performance Rating Highly Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The program provided clear objectives built on the existing National Target Program (NTP) for Rural Water and Sanitation Program. The theory of change documented how the existing NTP M&E system of 14 indicators for rural water supply and sanitation system would also be used for the program M&E. The Program introduced six new DLIs, four PDO Indicators, and a number of intermediate outcome indicators to be tracked and reported annually closely following the program's Results Framework. All of the RF indicators and DLIs were well- defined, linked to the PDO, and offered implementing agencies meaningful metrics to monitor outputs and impacts. The indicators and DLIs had clear baselines and realistic targets; clearly defined in the POM with detailed sustainability and community wide sanitation requirements. Data was collected at the local level, processed at the provincial level for use in their annual investment planning and tracking purposes. Summary investments and progress reports of the indicators were assembled by the National Center for Rural Water Supply and Sanitation (NCERWASS), in cooperation with the NTP Standing Office for Rural Water and Sanitation Services. The system used a geographical information system (GIS)-based M&E platform, which mapped data on investments, coverage, service delivery to the commune level for the entire country. The parallel grant financed TA provided training on improving data collection, and using the existing M&E system as foundation of provincial plans (PAD, paragraphs 48-50). The M&E system was well embedded in the provincial and national institutions. b. M&E Implementation The program M&E was implemented by MARD, through NCERWASS. They consolidated DLI results, prepared annual Comprehensive Results Reports (CRR). Weaknesses in design were addressed during implementation. Specific indicators were added. For example, MARD introduced new indicators for water supply service provision. The Vietnam Health Environment Agency (VIHEMA) also approved a new set of sanitation performance indicators for all 63 province. The Ministry of Education and Training (MOET) joined VIHEMA’s initiative and developed a new set of school sanitation indicators. MOET reported on sanitation in schools, The Provincial Centers for Rural Water Supply and Sanitation (PCERWASS) implemented the program M&E at the provincial level. Data was collected at the local level (commune and district), compiled and processed by PCERWASS at the provincial level (with support from the Department of Health (DoH), and Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) summarized at the national level by NCERWASS. According to the ICR, program M&E was generally complete, timely, and efficient. There were no compliance or quality issues. Data reported were independently verified and used for disbursement. SAV conducted the DLI verification process through document review and physical inspections for a sample of reporting communes. Verified achievements were documented in Annual Verification Reports. The verification protocol used rigorous statistical methods. SAV and the implementing agencies received regular training on verification and M&E c. M&E Utilization The M&E findings were communicated to various stakeholders, from the communes to the provincial entities to the national agencies. The impact of the reports made following the M&E system were reflected in the DLI verification proces of independent verification and resulting disbursements. M&E data reported on program outcomes. The National Target Program's own M&E system for water supply and sanitation system integrated DLI3 (conditions defining sustainability) and DLI4 (community wide sanitation requirements) into its M&E system. MARD, NCERWASS, MOH and VIHEMA adopted and scaling up sector specific monitoring systems used under this program for the rest of the country. MARD approved including key sustainability elements in water supply service indicators while VIHEMA established and piloted a new data platform in all 8 provinces to monitor sanitation in households, health stations and schools, scaled up to 21 provinces, with plans to cover all 63 provinces. The overall quality of the Program M&E was rated substantial. Annual data collection cycle by provinces improved the link to the national planning and budgeting processes. Sustainability features of program indicators were integrated into the revised national M&E system for rural water supply and sanitation. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The program was assigned OP/BP 9.00, which established the policy and planning elements necessary to achieve outcomes consistent with PforR objectives. The policy, OP 9.00, Program-for-Results Financing, was first issued in February 2012, revised in April 2013, and further updated in April 2015. The elements of the policy guided Bank implementation support and assessed the government's systems and their capacity to plan and implement effective measures for environmental and social risk management. Program investments supported safer and more reliable water supply and improved sanitation and were expected to have positive environmental and public health impacts (PAD, paragraph 78). The Environmental and Social Systems Assessment required for PforRs concluded that no significant, adverse environmental or social impacts were anticipated. Any environmental and social issues (e.g., land acquisition, ethnic minority issues) were satisfactorily addressed in the PAP and POM (ICR, paragraph 65). Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) The MARD Inspectorate created a hotline for beneficiary feedback where the project affected people can report their complaints as part of the Grievance Redress System. The parallel AusAID grant financed a capacity building program to address compliance shortcomings in institutional capacity and financial resources. The grant financed oversight of the environmental assessment system, technical design of water supply systems, screening of environmental effects, and public consultation, among others. Planning and monitoring of environmental and social aspects were satisfactorily implemented with some initial delays. Institutional arrangements for environmental management were mandated and established at all levels of government. The Environmental Protection Plan and applicable regulations for community based water schemes and sanitation facilities were submitted as planned. All required environmental and social reports were submitted and approved, indicating satisfactory compliance. The Grievance Redress System worked. Land-acquisition-affected households were fully compensated following government regulations. Among Program provinces, only Quang Ninh had beneficiaries who were both ethnic minorities (EM) and land-acquisition-affected people. Guidelines in the PAP and POM for consultation and participation of EM beneficiaries were followed. The ICR indicated that the environmental risk regarding the proper disposal of fecal sludge in participating communes was outside the scope of the program. b. Fiduciary Compliance Financial Management : At preparation, the Bank and the government jointly conducted a Fiduciary Systems Assessment. This review identified improvements in auditing, competitive procurement, transparency, and anti- corruption as part of the PAP and POM. Technical assistance activities addressed capacity building needs. At implementation, the PAP was successfully carried out with minor delays. The Bank was satisfied with Program fiduciary management. Procurement: Procurement was part of the Fiduciary Systems Assessment. Technical Assistance addressed capacity building gaps. The Bank reported no procurement problems encountered. MARD/PCERWASS informed contractors and operators of sustainability factors to be met by infrastructure investments.The PAP actions improved procurement methods, strengthened competition, and encouraged private sector participation. Procurement was reported satisfactory. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) Reason for Ratings ICR IEG Disagreements/Comment Outcome Highly Satisfactory Highly Satisfactory --- Bank Performance Highly Satisfactory Highly Satisfactory --- Quality of M&E Substantial Substantial --- Quality of ICR High --- 12. Lessons This was the first PforR program in Vietnam, in the East Asia region of the World Bank, and in the Water & Sanitation sector. Four additional PforR operations in Vietnam followed this operation. The ICR offered six lessons and recommendations, four of these are below: • PforRs as a financing mechanism allow for adopting procedures that optimize the flexibility offered by results-based financing. In this Program, processes were adjusted to address the challenges of the systems in place. These included timely releases of budgets and transfers from the central to the provincial governments, coordinating annual investment planning at the local level, allocating achievable targets, and improving results verification to justify disbursements. Another example was separating the DLIs for the water sector from the sanitation sector. Making the DLIs sector specific allowed one sector to disburse accordingly while the other sector experienced delays. The lesson learned from this operation call for carefully analyzing government systems and procedures to be used under the program, simulating recommended improvements to ensure that effective funding flows or pre-financing arrangements would be seamless inserted into existing processes. Confirmed effective improvements could be included into the Program financing agreement, the PAP, or POM. • Strong implementation support is critical for success. This program was founded on an existing NTP/NRD program. This PforR operations provided budget support to provincial government investment programs based on achieved periodically verified results. Institutional capacity limitations at the Provincial level were acknowledged, hence, the program included capacity building TA. This component was addressed by an additional parallel grant financing. Components of that TA program responded to the capacity gaps in implementing the program. By program closing, with effective Bank supervision, technical assistance offered, a jointly established strong verification protocol, and a PAP that was informed by best practices, all of the DLI’s were achieved. The PforR instrument helped address long-term institutional barriers, and reshaped sector incentives with the help of the capacity building TA that targeted the provincial implementation agencies in conjunction with the program. The lesson learned from this operation for future PforR operations was that sufficient scope and funding should be included for institutional and capacity building tapping into resources such as other Bank instruments (e.g., an Investment Project Financing), or grants from other donor partners. • The Program could be used to encourage the private sector to participate in delivering rural water supply and sanitation services. In this program, private sector providers financed and operated sustainable water schemes in the 8 participating provinces. Private sector entities were convinced by clear, time bound Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Rural Water Supply & Sanitation (P4R) (P127435) investment plans and predictable availability of financing based on transparent results-based indicators. The lesson from this operation was that the private sector entities were likely to invest in a sector governed by a legal and transparent regulatory framework. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided a concise and detailed overview of the program. The theory of change showed how the program supported the Third Phase of the National Target Program for Rural Water Supply and Sanitation Services in 8 of the 63 provinces in Vietnam. The evidence supporting the results of the operations was synchronized with the independently verified indicators that triggered the disbursements. The narrative supported the ratings using the results framework. The justifications for the ratings were summarized under each section of the report. The analysis based on the evidence reported under the disbursement linked indicators triangulated the conclusions reached. The lessons for consideration in future PforR programs were informed by this operation. The annexes further clarified the results of the operations. The results of the parallel technical assistance found in Annex 5 strengthened the evidence on the outcome of this operation. The summary of the Borrower's comments on the ICR in Annex 6 provided an understanding of the program requirements, and a candid assessment of early staffing challenges, the poor quality of provincial reports to verify initial results, and confusion in coordinating financial management early on that negatively affected implementation. While there was no explicit statement of compliance regarding OP/BP 9.00 for PforR-financed programs, the ICR provided evidence of compliance and implementation. a. Quality of ICR Rating High Page 16 of 16