VIETNAM Results-Based Rural Water Supply and Sanitation under the National Target Program Report No. 171263 MAY 11, 2022 © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org Attribution—Please cite the work as follows: World Bank. 2022. Vietnam— Results-Based Rural Water Supply and Sanitation under the National Target Program. Independent Evaluation Group, Project Performance Assessment Report 171263 Washington, DC: World Bank. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Report No.: 171263 PROJECT PERFORMANCE ASSESSMENT REPORT Vietnam Results-Based Rural Water Supply and Sanitation under the National Target Program (IDA-51760, TF-13061) May 11, 2022 Finance, Private Sector, Infrastructure and Sustainable Development Independent Evaluation Group Abbreviations DLI disbursement-linked indicator NTP National Target Program PAP Program Action Plan PDO project development objective PforR program-for-results PI project development objective indicator RWSS rural water supply and sanitation All dollar amounts are US dollars unless otherwise indicated. IEG Management and PPAR Team Director-General, Independent Evaluation Ms. Alison Evans Director, Finance, Private Sector, Infrastructure and Sustainable Development Ms. Carmen Nonay Manager, Infrastructure and Sustainable Development Mr. Christopher Nelson Task Manager Ms. Chikako Miwa This report was prepared by Chikako Miwa, who assessed the project in August 2021. Le Tuyen Hong Hai provided technical assistance as a local consultant. The report was peer reviewed by Avjeet Singh and panel reviewed by Fernando Manibog. Viktoriya Yevsyeyeva provided administrative support. Note: IEG = Independent Evaluation Group; PPAR = Project Performance Assessment Report. ii Contents Abbreviations ..................................................................................................................................................... ii Data........................................................................................................................................................................ v Summary ............................................................................................................................................................vii 1. Background, Context, and Design .........................................................................................................1 Background and Context........................................................................................................................... 1 Objective, Design, and Financing ...........................................................................................................3 2. What Worked, What Didn’t Work, and Why? ...................................................................................8 Results..............................................................................................................................................................8 What Worked, and Why ...........................................................................................................................11 Design and Preparation .......................................................................................................................................... 11 Implementation and Supervision........................................................................................................................ 13 What Didn’t Work, and Why .................................................................................................................. 15 Design and Preparation ......................................................................................................................................... 15 Implementation and Supervision........................................................................................................................ 19 3. Lessons.......................................................................................................................................................... 19 Bibliography..................................................................................................................................................... 23 Figures Figure 1.1. Theory of Change.......................................................................................................................... 7 Figure 2.1. Poverty Rate in All 100 Communes in Bac Ninh Province and Groups in Selected 30 Communes .................................................................................................................................................. 16 Tables Table 2.1. Selection and Grouping of Target Communes .................................................................. 15 Table 2.2. Ethnic Minorities in the Eight Participating Provinces of the Results-Based RWSS under the National Target Program Program-for-Results................................................................. 17 Appendixes Appendix A. Ratings ..................................................................................................................................... 26 Appendix B. Fiduciary, Environmental, and Social Aspects............................................................ 39 iii Appendix C. Methods and Evidence ...................................................................................................... 41 Appendix D. World Bank Rural Water Supply and Sanitation Projects in Vietnam .............. 43 Appendix E. Poverty Rates by Commune in Bac Ninh Province .................................................. 45 Appendix F. Borrower Comments ........................................................................................................... 48 iv Data This is a Project Performance Assessment Report by the Independent Evaluation Group on the Vietnam Results-Based Rural Water Supply and Sanitation under the National Target Program (P127435). The lending instrument for this Program-for-Results operation and the methodology for this evaluation are discussed in appendix C. This Project Performance Assessment Report presents its findings and conclusions based on a review of the World Bank’s project documentation, combined with a virtual mission to Vietnam conducted in August 2021. Independent Evaluation Group conducted interviews with a range of different stakeholders linked to the project, including government officials, the implementing agency, and World Bank staff. Following standard Independent Evaluation Group procedure, copies of the draft Project Performance Assessment Report were shared with relevant government officials for their review and comment. Comments are included in appendix F. Basic Data World Bank IDA financing Country Vietnam commitment $200,000,000 Global Practice Water Planned program cost $266,042,000 Program name Results-Based Rural Water Revised program total cost $204,787,901 Supply and Sanitation under the National Target Program Project ID P127435 Actual amount disbursed $202,684,103 Financing instrument Program-for-Results Environmental assessment n.a. financing category Financing source IDA-51760, TF-13061 Dates Event Original Date Actual Date Approval 01/11/2012 01/11/2012 Effectiveness 24/04/2013 24/04/2013 Restructuring 16/06/2016 16/06/2016 Mid-Term Review 15/09/2015 15/09/2015 Closing 31/07/2018 31/07/2019 v Key Staff Responsible Management Appraisal Completion Program team leaders Parameswaran Iyer Lilian Peña Pereira Weiss Hoa Thi Hoang Hung Sy Pham Practice manager Jennifer J. Sara Sudipto Sarkar Regional director John A. Roome Benoit Bosquet Country director Victoria Kwakwa Ousmane Dione vi Summary Background and Description Vietnam has had an impressive record of inclusive economic growth over the two decades before appraisal of this program, but poverty has remained concentrated in rural areas and among ethnic minorities. Although rural access to water and sanitation improved along with economic growth, challenges remained regarding the sustainability of the initiatives that provide water services and hygiene. To address those sectoral challenges, the government of Vietnam updated the National Rural Clean Water Supply and Sanitation Strategy in 2011 by strengthening community participation, demand- responsive approaches, and cost recovery. To achieve the objectives in the national strategy, the government started to implement the third phase of the National Target Programs (NTPs) for Rural Water Supply and Sanitation (2011–15). The World Bank has been supporting Vietnam’s rural water supply and sanitation (RWSS) sector for more than 20 years, with a focus on promoting sustainable approaches for water supply service provision through higher levels of cost recovery and more accountable enterprise management arrangements. The World Bank program, Results- Based Rural Water Supply and Sanitation under the National Target Program (Results- Based RWSS under the NTP PforR or RWSS PforR) was the first operation in Vietnam and in the Water Global Practice to adopt the PforR financial instrument, which introduced a results-based approach in fund allocations and disbursements. The RWSS sector in Vietnam demonstrated favorable conditions for piloting the PforR, given the existence of a large NTP, the World Bank’s strong presence in the sector, and government interest in shifting toward a focus on results and sustainability. The program’s aim of improving the financial and operational sustainability of the RWSS sector was also relevant to the national road map for economic, political, and social reform called Vietnam 2035. The program’s objectives were well aligned with the Country Partnership Strategy for fiscal years (FY)12–16 and the Country Partnership Framework for FY18–22. The Results-Based RWSS under the NTP PforR’s project development objective, as stated in the financing agreement, was “to increase sustained access to water supply and sanitation services and improve sector planning, monitoring and evaluation in the Participating Provinces of the National Target Program for Rural Water Supply and Sanitation.” The credit ($200 million) and a grant ($6.04 million) were appraised in September 2012 and approved by the Board of Executive Directors on November 1, 2012. The operation became effective on April 24, 2013. The Results-Based RWSS under the NTP PforR had a restructuring on June 16, 2016 to revise the disbursement-linked indicators (DLIs) and extend the loan closing date. It was closed on July 31, 2019, a year vii after the original closing date of July 31, 2018. (Refer to the Implementation Completion and Results Report Review for additional information [World Bank 2020b].) Results The results presented in this section track the two core elements of the results-based financing model used by the RWSS PforR: infrastructure investments in water supply and sanitation facilities, and technical assistance and implementation support to change the way that the government agencies worked on such investments. The infrastructure investment results were satisfactory. The RWSS PforR was fully successful in increasing access to clean water and to water delivery services that were operated and financed in a sustainable manner. The PforR exceeded the targets for the indicators that embedded sustainability elements: the number of people with a water supply connection from a sustainable water system (project development objective indicator 1, DLI III), which required results verifications to be conducted two years after the schemes were operationalized in a financially sustainable manner. (Refer to the Implementation Completion and Results Report Review for the actual results of the other indicators [World Bank 2020b].) The process of infrastructure investments, particularly the efficiency in design and execution of works, was also improved. The average construction time of water schemes in the PforR areas was 20 months, which was substantially shorter than the 34-months construction time in the non-PforR areas. The water schemes built under the RWSS PforR, which served an average of 2.8 communes and 3,104 households per scheme, were larger and more efficient compared with previous water schemes and those outside the PforR areas, which typically served a single commune and an average of 685 households per scheme. In addition, the RWSS PforR successfully increased rural sanitation investments. Among households, 142,280 improved household sanitary latrines were newly constructed, exceeding the target of 130,000 latrines (109 percent of the target). Among public buildings such as schools and health centers, 1,559 clean water supply and hygienic sanitation facilities were constructed and in use, exceeding the target of 1,440 facilities (108 percent of the target). The sustainability of sanitation facilities in public buildings was measured by the number of communes where schools and health centers maintained hygienic status for two years after the sanitation facilities were put into use (intermediate results indicator 4, DLI V), which achieved the actual results of 179 communes, exceeding the target of 150 communes (119 percent of the target). The RWSS PforR changed the way the eight participating provincial governments and the national government worked on the RWSS infrastructure investments during the implementation. Provincial investment plans and program reports from eight participating provinces were developed (project development objective indicator 3, DLI viii VI), and one program report from the Ministry of Agriculture and Rural Development was consolidated and published (intermediate results indicator 5, DLI VI) annually throughout the six-year duration of the PforR (a total of 102 plans and reports, meeting 100 percent of the revised target). What Worked, and Why Design and Preparation The DLIs chosen for the PforR were consistent with and relevant to the development plan for the water supply and sanitation sector in Vietnam. In addition, the PforR was a good instrument to address issues of collusion and the dominance of dependent state- owned enterprises in contract procurement and management. The RWSS PforR promoted competitive bidding to enhance transparency and accountability in public agencies’ procurement processes. The results-based financing approach addressed the frequent delays in water scheme constructions. The PforR enhanced results-based financing and independent results verification, which resulted in (i) transferring risk associated with construction delays to service providers and contractors to motivate them to deliver results quickly and cost-effectively; (ii) incentivizing implementing agencies to select target areas based on criteria including the households’ willingness to pay for water connections; and (iii) making the contractors accountable for ensuring quality, transparency, and accountability throughout the construction process. The World Bank used the PforR to try to encourage the provincial governments to acknowledge that financing from, and relationships with, the private sector were fundamental to provincial investments. The RWSS PforR conducted beneficiary needs assessments to design DLIs that incentivize the participating provinces to develop demand-driven schemes and reward those where tariff revenues cover operational costs. The PforR also contributed to diversifying financing for rural water service delivery by promoting private sector–led service delivery models. The beneficiary feedback system contributed to enhancing transparency and social accountability by allowing anonymous feedback for the first time. At the program closing, the public freely used the phone-based beneficiary feedback system to ask for clarifications and register complaints on the procedure, quality, and sufficiency in quantity of constructions. Preparatory analytical work and good government planning made the PforR’s preparation fast and efficient. The PforR took less than a year to prepare because of the government’s solid RWSS program that evolved over 10 years with technical and financial support from the World Bank. (Refer to appendix D for the World Bank’s projects in the RWSS sector before and after the RWSS PforR.) This enabled the World ix Bank team to initiate the design and preparation and complete three assessments on technical, fiduciary, and environmental and social aspects that informed the development of the Program Action Plan. Implementation and Supervision The joint and harmonized implementation of the DLIs and the verifications of results by independent agencies created incentives to achieve higher levels of accountability and transparency. Regarding the monitoring and evaluation (M&E) system, the provinces were incentivized to ensure timely achievements of quality results to receive disbursements after results were verified. Although the progress measured by the M&E indicators was not disclosed earlier, the PforR developed a web-based information platform that allowed the provincial stakeholders to observe the progress of DLI implementation in other participating provinces. The involvement of private enterprises contributed to increased revenues and higher staff productivity, which led to improved cost recovery ratios for water supply schemes. Before the Results-Based RWSS under the NTP PforR, the rural water market was seen as an unattractive loss maker, and private investment in most of the eight participating provinces was limited. Rural systems were seen as scattered and difficult to manage, and potential for profit was low. The PforR ensured the provision of the states’ fiscal budget support to the participating provinces by strengthening sector planning. The privately managed schemes in rural areas of Vietnam had higher profits that averaged 35 percent of overall revenues, compared with 20 percent or less for other types of service providers. This was in line with the overall trend of an increase in private sector investments in the RWSS sector. Public-private partnerships in water and sanitation increased from $88.9 million in 2016 to $159.14 million in 2019. Moreover, privately managed water schemes had higher staff productivity. On average, one worker in a private enterprise distributed 3,850 cubic meters of water supply monthly, compared with 970 cubic meters for staff in community- or public-led service provision. What Didn’t Work, and Why Design and Preparation In results-based operations, the drive to achieve the DLI targets could result in prioritizing communities that are relatively more advanced and already have lower poverty rates at program start-up. To ensure receipt of fund disbursements, the operations may focus on producing tangible results as early as possible, thus incentivizing the assignment of a lower priority to more complex activities. However, the latter may be critical to the broader development of borrower systems and capacities and to longer-term goals of inclusion of minorities and poverty reduction. For example, x for Bac Ninh province, the selection of the targeted 31 communes (of 100 total) showed a trend in which the communes with the 50 lowest poverty rates (61 percent of the selected communes) were more likely to be selected rather than the communes with the 50 highest poverty rates (39 percent). The same trend is evident in the prioritization and grouping of selected communes for implementation sequencing. Provincial governments prioritized communes already on the way to meeting targets, rather than those with the greatest need. The high requirements of the DLIs motivated the provincial-level agencies to prioritize communes that had high achievability of DLIs. The provinces conducted surveys to assess levels of commitment among the potential water users, resulting in selecting communes with better economic conditions such as the presence of industrial parks. Moreover, if a commune had any school or a health center that was unlikely to achieve either the water or the sanitation standards set by the commune-wide sanitation DLI, then that commune would not be prioritized. The shared perception among provincial agencies was that the selection and prioritization of participating communes were keyed to high achievability of results, which involved a risk that poor communes would be less likely to be selected and prioritized at the planning stage. The selection criteria of areas for water scheme constructions resulted in a low level of inclusion of ethnic minorities at the provincial level. Although ethnic minorities accounted for approximately 14 percent of the Vietnamese population, they accounted for 50 percent of the poorest population. During implementation, only two provinces—Quang Ninh and Phu Tho—had ethnic minorities groups living in the RWSS PforR area for water program constructions, for which the Ethnic Minorities Development Plan was prepared and implemented. This indicated that Thanh Hoa, which had the largest demographic representation of ethnic minorities among the eight provinces, did not have ethnic minorities in the PforR areas for water scheme constructions. According to the Ministry of Agriculture and Rural Development, the provinces’ criteria for selecting locations of water schemes were based on minimizing land acquisition and resettlement of ethnic minority households to comply with action 4 on resettlement and action 5 on ethnic minorities in the Program Action Plan. Implementation and Supervision Both the online M&E database and the beneficiary feedback system were discontinued when the Results-Based RWSS under the NTP PforR ended because of a lack of budget and the technical challenge. Given technical limitations on data compatibility, the data stored in the PforR’s M&E database were not transferred to the present website disclosing the data of the M&E system on rural water supply established by the xi Directorate of Water Resource Management in the Ministry of Agriculture and Rural Development. In addition, the mobile phone–based M&E system was introduced to the participating provinces to collect and analyze M&E data on access to water and sanitation facilities, but the use was not universal and was limited for the pilot purpose. Nevertheless, the M&E rating remains substantial; although the M&E online platform was discontinued, some M&E indicators and methodologies to monitor the results for the M&E indicators were sustained in the follow-on PforR being implemented in the different provinces. The misalignment of the disbursement schedules between the government system and the RWSS PforR caused frictions in financial management. The contractor of the water supply schemes bore the cash deficiencies in the annual budgets of the PforR in the participating provinces, per the contract provision. The daily recording of the expenditures related to the PforR posed another challenge. During the Independent Evaluation Group’s interviews, respondents pointed out that the PforR could have benefited from stronger political buy-in of the central ministries with responsibilities for budget planning and allocation, that is, the Ministry of Finance and the Ministry of Planning and Investment. The Project Performance Assessment Report rates the outcome and the World Bank’s performance highly satisfactory, and the quality of monitoring and evaluation substantial (table S.1) These ratings are the same as those in the Implementation Completion and Results Report (ICR) and the ICR Review. Table S.1. Project Ratings for the Vietnam Results-Based Rural Water Supply and Sanitation under the National Target Program Indicator ICR ICR Review PPAR Outcome Highly satisfactory Highly satisfactory Highly satisfactory Bank performance Highly satisfactory Highly satisfactory Highly satisfactory Quality of monitoring and evaluation Substantial Substantial Substantial Source: World Bank 2020a, 2020b. Note: The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible Global Practice. The ICR Review is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. ICR = Implementation Completion and Results Report; PPAR = Project Performance Assessment Report. The evaluation methodology and evidence sources are described in appendix C. Lessons The Results-Based RWSS under the NTP PforR’s experience suggests the following lessons: • PforR design needs to be closely aligned with national policies and regulations, particularly regarding financial management. To avoid friction and imbalances xii in financial management, the design of PforRs needs to ensure that cash deficiencies do not occur in the provincial and subnational annual budgets because of differences among the disbursement schedules of the government system and the PforR operation. An exacerbating factor is that the Vietnamese budgeting and transfer system did not have a mechanism for adaptable performance-based transfers from the central government to provinces to enable results-based budget allocations. In 2020, the government of Vietnam also issued a decree (No. 56/2020/ND-CP) that specified that official development assistance and concessional loans be used to finance only capital expenditures and not recurrent expenditures, which prevented the development of a new PforR operation. • PforR design and implementation need to exercise equity and inclusivity in targeting beneficiaries to avoid selection bias against hard-to-reach ethnic and the poorest minorities and to reduce their vulnerability in the long term. Results- based operations designed with DLIs have a built-in incentive to achieve results efficiently, which could motivate the provincial agencies to de-prioritize serving the vulnerable populations and areas. When selecting and prioritizing communes, the RWSS PforR reflected an incentive to focus more on achievability of DLIs rather than reaching out to those with the greatest need. It resulted in inadequate representation of poor and ethnic minorities as target beneficiaries in relation to their share of the total population. Concrete measures need to be adopted to ensure that results-based operations provide implementing agencies with incentives to work in challenging areas, for example, the integration of poverty alleviation as a DLI target. • The existence of an enabling environment for private participation could enhance the effectiveness of PforRs. The results-based approach fosters the mobilization of private actors. The national policies and regulations promoted public-private partnerships, which were reflected in the design of the RWSS PforR. PforRs need to assess and harness opportunities to allow private enterprises to lead improvements in cost recovery ratios of water supply schemes. The experience with the RWSS PforR shows that Vietnam’s favorable enabling environment for private sector participation has enhanced the operation’s implementation effectiveness by improving the financial and operational sustainability of the country’s RWSS sector. Carmen Nonay, Director Finance, Private Sector, Infrastructure, and Sustainable Development Independent Evaluation Group xiii 1. Background, Context, and Design Background and Context 1.1 Sector background. Vietnam has had an impressive record of inclusive economic growth. The annual gross domestic product growth averaged approximately 6.4 percent over the past two decades (World Bank 2019c). Vietnam has also achieved impressive poverty reduction. The general poverty rate decreased from 37.4 percent in 1998 to 5.8 percent in 2016 (Vietnam GSO 2019). Notwithstanding these achievements, development challenges persist. Poverty gains are fragile, with remaining poverty concentrated in rural areas and among ethnic minorities. Most poor people—9 out of 10—live in rural areas where access to basic services is very low (World Bank 2017). Although ethnic minorities account for approximately 14 percent of the Vietnamese population, they account for 50 percent of the poorest population (Vietnam GSO 2010). 1.2 Challenges relating to water quality are notable in rural areas in Vietnam despite the increased access to water. Although, officially about 75 percent of the rural population of Vietnam had access to hygienic water (that is, water protected from contamination),1 access in rural areas to clean water (that is, water meeting quality standards) was estimated to be approximately 35 percent.2 Only 34.8 percent of rural households use tap water as their main water source. Water pollution and contamination in the groundwater caused by pesticides, fertilizers, feces, iron, and arsenic were specific problems. Groundwater, which generations of local people have used for cooking and drinking, was becoming increasingly contaminated by toxic pollutants. Surface water sources with large distribution networks presented significant management challenges. Piped water networks were either broken or failing to reach households. Weaknesses have been noted relating to the sustainability of water supply systems, with inadequate attention paid to arrangements for operation and maintenance and cost recovery. The rural population also faced challenges relating to hygiene when using latrines. Only 56 percent of rural households had hygienic latrines, and less than 20 percent of people washed their hands with soap at key moments (World Bank 2019a). Maintenance of sanitation facilities was also weak. Sanitation facilities in schools and health centers were not properly maintained. Access to properly maintained hygienic rural sanitation was low at 51 percent (World Bank 2012c). 1.3 Context for the Program-for-Results (PforR) operation. The government of Vietnam has been proactive in responding to the challenge of increasing access to, and ensuring the sustainability of, rural water supply and sanitation services. The National Rural Clean Water Supply and Sanitation Strategy (2000–20) introduced the concepts of community participation, demand-responsive approaches, and cost recovery while 1 setting the overall vision and goals for the sector. To achieve the objectives in the national strategy, the government of Vietnam has increased its investments in the sector and implemented three phases of National Target Programs (NTPs) for rural water supply and sanitation (RWSS; 2000–05, 2006–10, and 2011–15). Since 2016, the sector- specific NTPs focused on RWSS evolved into the broader National Target Program for New Rural Development (2016–20), which encompassed other rural development sectors in addition to RWSS. 1.4 The World Bank has been supporting Vietnam’s RWSS sector for more than 20 years, beginning with assisting the government of Vietnam in developing the National Rural Clean Water Supply and Sanitation Strategy. (Refer to appendix D for the World Bank Rural Water Supply and Sanitation Projects in Vietnam.) The World Bank supported the Red River Delta Rural Water Supply and Sanitation Project (P077287, fiscal years [FY]06–13), which was successful in promoting sustainable approaches for water supply service provision because it resulted in higher levels of cost recovery and more accountable enterprise management arrangements. The World Bank has also helped the government of Vietnam to successfully test the output-based approach to rural water supply under a project supported by the Global Partnership for Output- Based Aid (now the Global Partnership for Results-Based Approaches). The newly introduced results-based approach aimed to shift the incentive structure of the project from one based on inputs and activities to one based on outputs (World Bank 2016c). The RWSS PforR’s objectives were well aligned with the Country Partnership Strategy FY12–16 (World Bank 2011) and the Country Partnership Framework FY18–22 (World Bank 2017). 1.5 The Results-Based Rural Water Supply and Sanitation under the National Target Program ( Results-Based RWSS under the NTP PforR or simply RWSS PforR) was the first operation in Vietnam and in the World Bank’s Water Global Practice to adopt the PforR financial instrument, which introduced a results-based approach in fund allocations and disbursements. The PforR financial instrument’s main features were (i) financing and supporting borrowers’ programs; (ii) disbursing on achievement of program results; (iii) focusing on strengthening the institutional capacity and the processes and procedures needed for programs to achieve their desired results; and (iv) providing assurance on the fiduciary, environmental, and social aspects of the programs. The RWSS sector in Vietnam demonstrated favorable conditions for piloting the PforR, given the existence of a large national program, the World Bank’s strong presence in the sector, and government interest in shifting toward a focus on results and sustainability. 2 Objective, Design, and Financing 1.6 This PforR’s project development objective (PDO) was “to increase sustained access to water supply and sanitation services and improve sector planning, monitoring and evaluation in the Participating Provinces of the National Target Program for Rural Water Supply and Sanitation” (World Bank 2013). The formulation of the PDO was identical between the financing agreement and the Program Appraisal Document (World Bank 2012c). The original PDO remained unchanged throughout project implementation. 1.7 Design. The PforR was a results-based approach to support the third phase of NTP RWSS in eight geographically clustered provinces in the three subregions: (i) Hanoi, Hung Yen, Bac Ninh, Ha Nam, and Vinh Phuc in the Red River Delta subregion; (ii) Phu Tho and Quang Ninh in the Northeast subregion; and (iii) Thanh Hoa in the Central Highlands subregion. Those provinces were regarded as having stronger economic conditions. The five criteria for selecting the eight provinces are (i) dense population (the eight provinces are home to one-sixth of the national population); (ii) significant numbers of poor people, ensuring a poverty reduction impact of investments; (iii) relatively strong capacity in the public sector at the local level, which is required to test the new results-based planning and management systems; (iv) the high presence of pollutants in groundwater, such as arsenic, which requires expansion of safe piped water through community water systems; and (v) geographic proximity to each other and ability to build on the existing World Bank project in the Red River Delta (for example, Red River Delta Rural Water Supply and Sanitation Project; World Bank 2012c). Weaknesses identified with the prior phases of the NTP RWSS—which are described in a subsequent paragraph—became the main drivers of the government’s decision to pursue a targeted, results-based approach to RWSS under the third phase of the NTP RWSS. The World Bank chose the PforR instrument to assist the government in delivering priority results more efficiently by strengthening its own country system and leveraging World Bank financing with that of other partner development organizations (World Bank 2016b). 1.8 There were four main components in the PforR (see appendix A for details). Component 1 (increase access to water supply services) and Component 2 (increase access to sanitation services) financed the construction of commune-level water supply programs and hygienic latrines in households, schools, and health centers. Component 3 (information, education, and communication, and behavior change campaigns) facilitated the modification of rural residents’ attitudes and behavior toward using and maintaining their water supply and sanitation systems. Component 4 (capacity building) provided implementation support. 3 1.9 In parallel, the Australian Agency for International Development (now Australian Aid) provided a technical assistance grant to address gaps in the institutional capacity of the implementing agencies and the State Audit of Vietnam. 1.10 Financing and key dates. The credit ($200 million) and a grant ($6.04 million) to Vietnam was appraised in September 2012, and was approved by the Board of Executive Directors on November 1, 2012. It became effective on April 24, 2013. The Results-Based RWSS under the NTP PforR had a restructuring on June 16, 2016 to revise the disbursement-linked indicators (DLIs) and extend the program closing date. The PforR disbursed $175.68 million at closing. The difference of $30.36 million was reported as savings because the private sector injected resources during implementation. The PforR was closed on July 31, 2019, a year after the original closing date of July 31, 2018. 1.11 The intention of the PforR was to address weaknesses that were identified during the prior phases of the NTP RWSS. These include the following: i. Insufficient attention to household and institutional sanitation. Sector investments tended to focus on water supply at the expense of sanitation. ii. Lack of financial sustainability of water supply systems because of low household connection ratios and low household willingness to pay the connection fees and tariffs. iii. Weaknesses in the expenditure framework. The process of reconciling actual budget allocations to overambitious provincial plans resulted in water supply contracts being funded ahead of other activities such as sanitation, awareness- raising, and behavioral change activities. iv. Investment resources spread thin. Funding shortages prevented timely contract payment and contractors slowed their work to match payments. v. System inefficiencies, where little incentives were provided to promote cost- efficiency in design and construction. vi. Governance weaknesses in fiduciary, social, and environmental management, for example, civil society’s inadequate access to information on community selection, procurement, and contract management, and the lack of a grievance resolution mechanism (World Bank 2012c). 1.12 The measures to address these weaknesses were integrated in the results-based disbursement approach and technical assistance. 4 1.13 The PforR also intended to overcome challenges identified in the fiduciary and environmental and social systems assessments, such as (i) the limited transparency and accountability in planning and implementing RWSS investments; (ii) the reported complaints of fraud, corruption, and collusion in the public procurement process in the water sector (that is, prevalence of direct contracting and dominance of the state-owned enterprises); (iii) the limited capacity for financial and results auditing; (iv) the potential negative effects caused by land acquisitions and resettlements; and (v) the lack of operational procedures for adequate implementation of the legal framework for ethnic development investments and activities. The measures to address those challenges were integrated in the Program Action Plan (PAP). 1.14 Theory of change. The RWSS PforR’s theory of change envisioned that inputs (such as designing the DLI mechanism that connects water and sanitation disbursements; providing the technical assistance on cost-efficient design and contracting process of water supply systems; and supporting implementation of awareness-raising and behavioral change activities) would increase attention and budget allocation to sanitation, resulting in an output of more balanced investments in water supply and sanitation. It also envisioned that an input of designing the DLI on sustainability of water schemes—that required operation and maintenance cost recovery and benchmarks for nonrevenue water—would increase financial sustainability of water supply systems, and also result in the output of an improvement in their financial performance indicators. Additionally, the theory of change envisioned that inputs (such as designing the results-based disbursement arrangement with a fixed disbursement amount for each DLI and providing the technical assistance on an independent results verification mechanism) would incentivize contractors to complete constructions swiftly and with quality to pass the verifications and receive disbursements, resulting in outputs of improved efficiency in construction in time and costs while maintaining construction quality. 1.15 The theory of change also envisioned that inputs such as supporting the implementation of PAP and activities to produce results that tallied toward DLIs, and also providing technical assistance would result in outputs such as (i) a monitoring and evaluation (M&E) information system built and disclosed; (ii) five-year and annual investment plans created by each participating province; (iii) annual program reports consolidated and disclosed by the Ministry of Agriculture and Rural Development; (iv) staff trained in national and provincial agencies on planning, M&E, fiduciary management, and independent results verification; (v) proportion of competitively bid contracts for construction of water supply schemes under public institutions increased; (vi) internal and external audits completed and the recommendations addressed; (vii) land acquisition minimized and compensation for acquired lands provided; and (viii) 5 guidelines on working with ethnic minorities developed and implemented. Those outputs were expected to result in the outcomes of improved efficiency, transparency, and accountability in the planning and implementation system of the NTP RWSS to deliver sustainable investments. The outcome was expected to contribute to long-term outcomes such as improved health and living standards in participating provinces. A critical assumption was that participating provinces selected communes with higher vulnerabilities (poor households, ethnic minorities) as the PforR’s target communes. Figure 1.1 presents the theory of change diagram. 6 Figure 1.1. Theory of Change Source: Independent Evaluation Group. Note: DLI = disbursement-linked indicators; M&E = monitoring and evaluation; NTP RWSS = National Target Program for Rural Water Supply and Sanitation; PforR = Program-for- Results; RWSS = rural water supply and sanitation. 7 2. What Worked, What Didn’t Work, and Why? Results 2.1 The results presented in this section track the core elements of the results-based financing model used by the Results-Based RWSS under the NTP PforR Project Performance Assessment Report: infrastructure investments in water supply and sanitation facilities, and technical assistance and implementation support to change the way that the government agencies worked on such investments. This assessment draws on the findings of Independent Evaluation Group (IEG) interviews and a review of program records. 2.2 The results regarding supporting infrastructure investments were fully successful. Regarding water access, the PforR not only increased access to clean water but also to the sustainable water system.3Referring to the theory of change, the design of the sustainability DLI for water supply addressed the development challenges of the lack of financial sustainability of water supply systems. Securing sustained access to water supply was based on overcoming the constraints posed by the low-cost recovery rate and the high nonrevenue water rate. The RWSS PforR exceeded the targets for the indicators that embedded sustainability elements: the number of people with water supply connection from a sustainable water system (PDO indicator [PI] 1, DLI III), which required results verifications to be conducted two years after the schemes were operationalized. At program closing, 1,054,575 people had water supply connections from a sustainable water system, exceeding the target of 850,000 people (124 percent of the target; World Bank 2020a). It means that of the 1,759,842 people who gained access to improved water sources (PI 2), approximately 60 percent gained access that were supplied by service providers with financial sustainability (World Bank 2020a). The sustainability performance for a subset of water supply schemes in the eight provinces that was measured for DLI III was remarkable, with the average cost recovery ratio of the water supply schemes of 1.28 and the average nonrevenue water ratio of 19 percent (decreasing from the more than 25 percent estimated at appraisal;4 World Bank 2020a). When compared with the counterfactual, the cost recovery ratio and the nonrevenue water ratio in the area covered under the PforR (that is, 1.18 and 22 percent, respectively) showed better results than those of the area not covered under the PforR (that is, 1.14 and 34 percent, respectively; World Bank 2020a). Moreover, the process of infrastructure investments was improved. 2.3 Referring again to the theory of change, the design featured the results-based disbursement arrangement with a fixed disbursement amount for each DLI and provided technical assistance on an independent results verification mechanism; these 8 features addressed the development challenge of the system inefficiencies and incentivized contractors to complete constructions swiftly and with quality to pass the verifications and receive disbursements, resulting in the improved efficiency in design and execution of works. The average construction time of water schemes in the PforR areas was 20 months, which was substantially shorter than 34 months in the non-PforR areas (World Bank 2020a). The water schemes built under the PforR, which served an average of 2.8 communes and 3,104 households per scheme, were larger and more efficient compared with the previous water schemes and those outside the PforR areas, which typically served a single commune and an average of 685 households per scheme (World Bank 2020a). 2.4 In addition, infrastructure investment success was also evident regarding sanitation. The PforR successfully increased rural sanitation investments. The links among the disbursements for the water supply results and the sanitation results, the cost-efficient design and implementation of water supply constructions, and the awareness-raising activities for sanitation all served to address the development challenges of less attention to household and institutional sanitation, in keeping with the theory of change. Among households, 142,280 improved household sanitary latrines were newly constructed,5 exceeding the target of 130,000 latrines (109 percent of the target; World Bank 2020a). Among public buildings such as schools and health centers, 1,559 water supply and sanitation facilities were constructed and in use, exceeding the target of 1,440 facilities (108 percent of the target; World Bank 2020a). In the early periods of the implementation, the achievements of the sanitation DLIs outpaced the water supply DLIs, resulting in diminishing ties between water supply and sanitation disbursements after the Mid-Term Review. However, the sustainability of household sanitation facilities was not comprehensively assessed and measured by any indicators. The respondents to IEG’s interviews confirmed that the Vietnam Health Environment Management Agency introduced a model of a local store network for sales of toilet parts for installation and rehabilitation. The sustainability of sanitation facilities in public buildings was assessed by counting the number of communes where schools and health clinics have maintained hygienic status (intermediate results indicator 4, DLI V), which also required results verifications to be conducted two years after the sanitation facilities were put in use. The actual results of both intermediate results indicator 4 and DLI V were 179 communes, exceeding the target of 150 communes (119 percent of the target; World Bank 2020a). Provinces financially incentivized local communes to assign maintenance responsibilities for public sanitation facilities; however, to what extent the assigned responsibilities were fulfilled beyond the program period was unknown. Moreover, the time period (that is, two years after the start of use) set by the sustainability DLIs for the public sanitation facilities was questionable. During IEG’s interviews, respondents commented that the septic tank was the popular type of latrine 9 in the participating provinces, and though its cost would be relatively higher, it would last 10 to 15 years with proper maintenance.6 2.5 The PforR sought to change the way the national and the eight participating provincial governments worked on the RWSS infrastructure investments and achieved results. The results-based disbursement model under the PforR encouraged government agencies to improve efficiency in implementation by strategically planning investments based on the M&E data. In addition, compliance with the national legal framework and the international standards at the provincial level was improved by establishing operational procedures and guidelines, which contributed to the improvement in transparency and accountability of the RWSS investments. Referring to the theory of change, the implementation support from the World Bank team and the technical assistance funded by the Australian Agency for International Development led to the establishment of the web-based PforR information platform; the completion of internal and external audits and independent results verification; the mitigation of land acquisition and adequate compensation provision; the development of the guideline to work with ethnic minorities; the development of provincial investment plans and program reports from eight participating provinces (PI 3, DLI VI); and consolidation and publication of one program report from Ministry of Agriculture and Rural Development (intermediate results indicator 5, DLI VI) annually through the PforR’s six-year duration (the total of 102 plans and reports, meeting 100 percent of the revised target; World Bank 2020a). In line with the PAP, the following results were achieved: i. Eight provinces established and maintained a database of complaints on corruption in procurement and program implementation, and responded to the received complaints in a timely manner. ii. Eight provinces complied with regulations on procurement, such as ensuring the ratio of competitive bidding among consultancy and works contracts according to the agreed rate on a yearly basis. iii. Eight provinces prepared annual financial statements and followed an internal audit process, which supported completion of annual internal audit reports by Ministry of Agriculture and Rural Development. iv. Households that were affected by the program’s land acquisition fully received all compensation and support in accordance with the state’s policies. v. Development plans for ethnic minorities were developed and implemented in Quang Ninh and Phu Tho, both of which had areas where ethnic minorities resided (World Bank 2020a). 10 What Worked, and Why Design and Preparation 2.6 The DLIs chosen for the PforR were consistent and relevant to the development plan for the water supply and sanitation sector in Vietnam. Each DLI was designed to correspond with either a PDO indicator or an intermediate results indicator. Thus, achieving a DLI would be equivalent to achieving a key milestone toward achieving the PDO. The strategic allocation of funds to each DLI matched the sequencing of disbursements. Moreover, the DLIs embedded the quality requirements set by government standards. The definitions of the terms such as functioning water supply connections, hygienic sanitation facilities, and improved household sanitary latrines reflected the national technical standards and regulations to ensure quality services.7 2.7 The World Bank’s PforR platform was an effective instrument to address issues of collusion and the dominance of dependent state-owned enterprises in contract procurement and management. The World Bank was first to introduce in Vietnam the performance-based works contract involving the participation of the private sector in the operation and maintenance of a part of the public water supply system. The World Bank also introduced bidding using design-build contracts for the construction of water supply schemes.8 Building on experience, the Results-Based RWSS under the NTP PforR expanded its support for strengthening institutional capacity on contract procurement by establishing the actions in section 2 of the PAP, which prohibited single sourcing of a contractor and promoted competitive bidding to enhance transparency and accountability. Another major challenge was the multiple delays in water scheme constructions that resulted in the extension of the contractual period for more than twice the initial period (World Bank 2012b). The delays were mainly caused by (i) delays in payments to contractors because of unsystematic flow of funds; (ii) delays in construction of distribution systems to households because of social and administrative conditions such as lack of willingness to connect and local coordination; and (iii) inactive oversight supervision of the contractor’s performance by inadequately paid consultants. The World Bank addressed those causes of delays by introducing the results-based financing approach. At first, the World Bank supported the East Meets West Foundation to pilot the output-based aid approach in the rural water supply sector under the Global Partnership for Results-Based Approaches.9 The project supported by this Global Partnership for Results-Based Approaches showed that transferring risk to the service provider motivated them to deliver results swiftly and cost-effectively, an important lesson for the follow-on projects (Suardi 2011). Then, the RWSS PforR was launched with specific designs to address the challenge of construction delays, as described in the next paragraph. Transferring risk to service providers and contractors motivated them to deliver results as quickly and cost-effectively as possible to receive disbursements. 11 The results-based payment incentivized implementing agencies to select target areas based on criteria including the households’ willingness to connect pipes to their households through the contribution of cash or labor. The mandatory independent verification motivated the contractors to ensure that quality, transparency, and accountability were maintained at desired levels throughout the process. 2.8 The World Bank used the PforR to try to persuade the provincial governments that financing from and relationships with the private sector were fundamental to their investments. The risks to sustainability of the water supply investments included the following: i. Local planning processes that did not include capacity building or meet the communities’ needs and demands. ii. An over-reliance on public funds, especially from the central government. iii. The inadequate financial situation of many water schemes because of low revenues from tariffs. iv. The limited technical and management capacity of service providers (World Bank 2016a). 2.9 Before the RWSS PforR, many water supply schemes were overdesigned under the National Target Program mainly because of the limited involvement of the beneficiaries. The PforR conducted beneficiary needs assessments to design DLIs that incentivize the participating provinces to develop demand-driven schemes and reward those where tariff revenues cover operational costs. The PforR also contributed diversifying financing for rural water service delivery by promoting private sector–led service delivery models (that is, joint stock companies, build-own-operate-transfer, build-own-operate, and concession-type contracts) that showed the best prospects for improving financial efficiencies for rural water service delivery (World Bank 2016a). 2.10 The beneficiary feedback system contributed to enhancing transparency and social accountability by allowing the beneficiaries to provide anonymous feedback for the first time. The National Center for Rural Water Supply and Sanitation set up a toll- free number and software to automatically synthesize and classify the feedback content and transfer it to the relevant agencies. Between 2017 and 2018, the beneficiary feedback system received 283 calls and provided responses to 123 calls (that is, handled 43 percent of the total calls; NCERWASS 2019). Among those, the top three most frequently reported content were on (i) household toilets, including requests to clarify loan policy, definition of hygienic latrines, and proper use of septic tanks (71 calls, 25.1 percent of total calls); (ii) procedures for installing water meters, valves, and faucets, including 12 complaints on complexity of processes, long wait times, and forced sales of company equipment with prices higher than the market (70 calls, 24.7 percent); and (iii) school toilets, including complaints on low quality of tools and equipment, irresponsibility of owners or contractors within the warranty period, and insufficiency in the number of toilets per students (30 calls, 10.6 percent; NCERWASS 2019). Hanoi was the province that most frequently made calls and received responses (117 calls made, 80 calls handled), followed by Thanh Hoa (56 calls made, 24 calls handled; NCERWASS 2019). Before the RWSS PforR, as a baseline, anonymous complaints were not admitted. At program closing, the public could provide their feedback more freely to ask for clarifications and complain about the procedure, quality, and adequacy in the volume of constructions. The satisfaction level of people who used the beneficiary feedback system requires further investigation. 2.11 Preparatory analytical work and good government planning made the preparation for the PforR engagement fast and efficient in Vietnam. Although the length of time the World Bank took to prepare its early PforR operations varied across countries (World Bank 2015), the PforR took far less than a year for preparation. This was possible because of the existence of the solid government program on RWSS that had already evolved for 10 years before the launch of the PforR with technical and financial support from the World Bank. Those enabled the World Bank team to take the initiative of designing and preparing the PforR and accelerate the completion of three assessments on technical, fiduciary, and environmental and social aspects to inform development of the PAP. When the World Bank identified capacity gaps in the implementing agencies, proposed actions to address those gaps were included in the PAP. Implementation and Supervision 2.12 The joint and harmonized implementation of the DLIs and the verifications of independent results created incentives to achieve higher levels of accountability and transparency. Regarding the M&E system, the provinces were incentivized to ensure timely achievement of quality results to receive disbursements after results were verified by independent verification agencies. Before the Results-Based RWSS under the NTP PforR, the progress measured by the M&E indicators was not disclosed in a website. During implementation, the web-based information platform for the PforR (http://pl.pforr.vn) was developed under the Ministry of Agriculture and Rural Development per PAP action 1, which allowed the provincial stakeholders to observe progress of DLI implementations in other participating provinces. The website was visited 275,124 times.10 Regarding the provincial data on DLIs, on February 8, 2017, the website published Vinh Phuc province’s five-year provincial plans and annual provincial plans from 2013 to 2017, and annual provincial reports from 2014 to 2016.11 13 Among those, the top three most viewed documents by July 22, 2017, were the annual provincial plan for 2017 (viewed 472 times), the annual provincial report for 2016 (viewed 449 times), and the annual provincial report for 2014 (viewed 449 times)12. Whether the online access to the disclosed documents led to the flow of information to communes requires further investigation. 2.13 The involvement of private enterprises contributed to increased revenues and higher staff productivity, which led to improved cost recovery ratios for water supply schemes. Private enterprises operated 87 of 144 functioning water supply connections supported under the Results-Based RWSS under the NTP PforR (World Bank 2020a). Before the PforR, the rural water market was seen as an unattractive loss maker, and there was limited private investment in most of the eight participating provinces. Rural systems were seen as scattered, difficult to manage, and with low potential for profit. In four of the eight PforR provinces, the Ministry of Agriculture and Rural Development estimated that in 2012, the private sector fully or partially financed only 16 water supply schemes, with $2.15 million in private investment. The RWSS PforR ensured the provision of the states’ fiscal budget support to the participating provinces by strengthening sector planning. In Ha Nam province, all the projects were supported with the state’s budget at the maximum level regulated by the Decision 131/2009/TTg (from 60 to 75 percent of the total investment), thereby attracting private sector investments to account for approximately 30 percent of the total water supply capacity (Institute for Sustainable Futures [ISF] 2014). By 2018, the private sector provided full or partial financing of 63 water supply investments totaling $53.65 million (World Bank 2020a). The privately managed schemes in rural areas of Vietnam had higher profits that averaged 35 percent of overall revenues, compared with 20 percent or less for other types of service providers (ISF 2014). This was in line with the overall trend of an increase in private sector investments in the sector. Public-private partnerships in water and sanitation increased from $88.9 million in 2016 to $159.14 million in 2019.13 Moreover, there was higher staff productivity in privately managed water schemes. On average, one worker in a private enterprise distributed 3,850 cubic meters of water supply monthly, compared with 970 cubic meters for staff in community- or public-led service provision (ISF 2014). 2.14 The World Bank’s close implementation support helped the relevant government counterparts reach a better position on to how to manage their investments. At the two- year review, the government commented on the importance of the country-based World Bank staff’s support to the government’s PforR teams to overcome their steep learning curves (World Bank 2015). During IEG’s interview, the World Bank team provided an example of their proactive support: securing a trust fund for recruiting a local consultant to provide technical support to each school and health center, which helped the 14 government of Vietnam catch up with delays in the implementation of the sanitation sustainability DLI in the past two years. The World Bank’s timely support continued throughout the RWSS PforR, as confirmed by both internal World Bank staff and external respondents during IEG’s interviews. What Didn’t Work, and Why Design and Preparation 2.15 In results-based operations, the drive to achieve the DLI targets could result in prioritizing communities that are relatively more advanced and already have lower poverty rates at program start-up. To ensure receipt of fund disbursements, the operations may focus on producing tangible results as early as possible, thus incentivizing the assigning of a lower priority status to more complex activities. However, the latter may be critical to the broader development of borrower systems and capacities and to longer-term goals of inclusion of minority populations and reduction of poverty. For example, for Bac Ninh province, the selection of the targeted 31 communes (of 100 total) showed a trend in which the communes with the 50 lowest poverty rates (61 percent of the selected communes) were more likely to be selected than the communes with the 50 highest poverty rates (39 percent; see table 2.1). The same trend is evident in the prioritization and grouping of selected communes for implementation sequencing. Table 2.1. Selection and Grouping of Target Communes Ratio over All the Communes Selected Communes Selected Mean of Group Communes (no.) (%) Number With 50 highest poverty rates 12 39 2 With 50 lowest poverty rates 19 61 1.18 Total 31 100 Source: Monitoring and evaluation indicator report by the Provincial Center for Rural Water Supply and Sanitation and Department of Labor, Invalids, and Social Affairs Bac Ninh, cited in Bac Ninh Province 2012. Note: Poor households are those with average income less than or equal to 400.000 Vietnamese dong per person per month. Group number implied the prioritization of implementation: group 1 was implemented first, group 2 was implemented second, and group 3 was implemented last during the Results-Based Rural Water Supply and Sanitation under the National Target Program Program-for-Results. 2.16 The tendency to favor communes that were less poor is presented visually in figure 2.1, where for each commune, the dots show whether it was selected and group numbers, and the line shows each commune’s poverty rate. Although the data on poverty rates by communes were available for this Project Performance Assessment Report only for Bac Ninh province, respondents confirmed during IEG’s interviews that such a tendency existed, as discussed in the next paragraph. 15 Figure 2.1. Poverty Rate in All 100 Communes in Bac Ninh Province and Groups in Selected 30 Communes 20 18 16 14 12 Percent 10 8 6 4 2 0 0 10 20 30 40 50 60 70 80 90 100 Commune 1 to Commune 100 in Bac Ninh Province Poverty Rate (%) Group (1, 2, or 3), indicating a sequence of implementation Source: Monitoring and Evaluation Indicator report by the Provincial Center for Rural Water Supply and Sanitation and Department of Labor, Invalids, and Social Affairs Bac Ninh, cited in Bac Ninh Province 2012. Note: Poor households are those with average income less than or equal to 400,000 Vietnamese dong per person per month. 2.17 Provincial governments prioritized communes already on the way to meeting targets, rather than those in greatest need. The DLIs’ high requirements motivated the provincial-level agencies to prioritize communes that had high achievability of DLIs (Bank Information Center 2014). The provinces conducted surveys to assess levels of commitments among the potential water users, resulting in the selection of communes with better economic conditions shown by features such as the presence of industrial parks. Moreover, if a commune had any school or a health center that was unlikely to achieve either the water or sanitation standards set by the commune-wide sanitation DLI, then that commune would not be prioritized. The shared perception among provincial agencies was that the selection and prioritization of participating communes were keyed to high achievability of results, which involved a risk that impoverished communes were less likely to be selected and prioritized at the planning stage. 2.18 Subsidies intended to enable impoverished households to access water and sanitation services might not have reached the most impoverished. During IEG’s interviews, respondents confirmed that poor and near-poor households in the target communes received subsidies for water use, connection, and latrine construction to achieve the DLI targets and receive disbursements. In early periods of the RWSS PforR’s implementation, nongovernmental organizations and the provincial government 16 suggested that very poor households had difficulty paying for water connections and latrines, even where subsidized credits were available (Bank Information Center 2014). At program closing, based on data by the Ministry of Agriculture and Rural Development, approximately 26 percent of poor households in the eight provinces (26,911 of 104,100 poor households) received subsidies from the program to build hygienic latrines (World Bank 2020a). Whether the subsidies reached households among the poorest of poor people requires further investigation. 2.19 The selection criteria of areas for water scheme constructions resulted in a low level of inclusion of ethnic minorities at the provincial level. Although ethnic minorities accounted for approximately 14 percent of the Vietnamese population, they accounted for 50 percent of the poorest population (Vietnam GSO 2010). The top three ethnic minority groups that had the largest populations in the Red River Delta were Muong group (75,222 people), Dao group (63,819 people), and San Diu group (57,832 people; Vietnam GSO 2009). As shown in table 2.2, two of eight participating provinces had higher representations of ethnic minorities than the national representation of approximately 14 percent: Thanh Hoa (17.6 percent) and Phu Tho (15.8 percent; GSO 2009). In those provinces, Muong group was the largest ethnic minority group, with 341,359 people (57 percent of ethnic minorities) in Thanh Hoa province and 184,181 people (89 percent of ethnic minorities) in Phu Tho province (Vietnam GSO 2009). Table 2.2. Ethnic Minorities in the Eight Participating Provinces of the Results-Based RWSS under the National Target Program Program-for-Results Province Total Population Ethnic Minority People Percentage Hanoi 6,451,909 81,665 1.3 Vinh Phuc 999,786 42,859 4.3 Pho Tho 1,316,389 207,398 15.8 Bac Ninh 1,024,472 3,411 0.3 Hung Yen 1,127,903 1,436 0.1 Quang Ninh 1,144,988 133,194 11.6 Ha Nam 784,045 1,640 0.2 Thanh Hoa 3,400,595 599,274 17.6 Source: Vietnam 2009. 2.20 The ethnic minority groups that participated in the Results-Based RWSS under the NTP PforR differed from the demographic representation in the national census described in the preceding paragraph of this Project Performance Assessment Report. According to the PforR’s Environmental and Social Systems Assessment, the ethnic minority beneficiary groups in the PforR included the San Diu group in Quang Ninh province; and the Muong, Thai, and Dao groups in Phu Tho and Vinh Phuc provinces (World Bank 2012a). During implementation, only two provinces—Quang Ninh and 17 Phu Tho—had ethnic minority groups living in the PforR area for water scheme constructions, for which the Ethnic Minorities Development Plan was prepared and implemented (World Bank 2019b). This indicated that Thanh Hoa, which had the largest demographic representation of ethnic minorities among the eight provinces, did not have ethnic minorities in the PforR areas for water scheme constructions. According to Ministry of Agriculture and Rural Development, the provinces’ criteria for selecting locations of water schemes were based on minimizing land acquisition and resettlement of ethnic minority households, to comply with action 4 on resettlement and action 5 on ethnic minorities in the PAP (MARD 2019). 2.21 Whether there might have been a more appropriate DLI to address key development challenges on sanitation in the eight provinces is unclear and needs further study. The indicator on the number of people with access to commune-wide sanitation (PI 2, DLI IV) did not add much value to the results framework. Under the RWSS PforR, commune-wide sanitation was considered achieved when all the following conditions were met: (i) 100 percent of public kindergarten, primary, and secondary schools had clean water and hygienic sanitation facilities as defined by the Ministry of Health standards; (ii) 100 percent of commune health centers had clean water and hygienic sanitation facilities according to the Ministry of Health standards; (iii) at least 70 percent of households had hygienic sanitation according to the Ministry of Health standards; and (iv) 100 percent of households used latrines of some kind (World Bank 2012c). All the eight provinces had almost achieved conditions (iii) and (iv) on household latrines before the PforR because the coverage of improved latrines was 68.5 percent in the eight provinces in 2010, which was higher than 50.6 percent in comparison provinces in 2010, before the RWSS PforR (World Bank 2016c).14 Two of the eight provinces almost fully achieved all the four conditions before the PforR. For example, Bac Ninh province’s baseline in 2012 indicated that 90 percent of schools, health centers, and public buildings in target communes had clean water and hygienic sanitation facilities against the target of 100 percent (Bac Ninh Province 2012). The Bac Ninh province also had already met a condition before the RWSS PforR started with a baseline of 77 percent of households reported to have acceptable, hygienic, functional latrines within their household confines, against the target of at least 70 percent (Bac Ninh Province 2012). Likewise, in Hanoi, 89.7 percent (1,816 of 2,024) of schools had hygienic water and latrines before the PforR, and 79.5 percent of rural households had a hygienic latrine before the PforR (Hanoi 2012). The DLI on commune-wide sanitation was expected to serve as a proxy for greater health benefits (World Bank 2012c); however, the DLI did not serve the expected role because waterborne diseases were not a critical issue in the Red River Delta. In rural areas of the country, 9.7 percent of under-five children were reported to have had diarrhea in the past two weeks—a higher ratio compared with 6.0 percent in urban areas (GSO and UNICEF 2015). However, diarrhea prevalence rates were lower in Hung Yen 18 province (1.80 percent) and in Thanh Hoa province (1.30 percent), indicating that waterborne disease was not an acute issue in some of the PforR’s target provinces even before the start of the operation (Chase and Do 2010). During IEG’s interviews, respondents indicated that different divisions in agencies in the health sector collected and managed the data on community sanitation and people’s health conditions, and thus these data were not accessible all the time. Implementation and Supervision 2.22 Both the online M&E database and the beneficiary feedback system were discontinued when the PforR ended because of the lack of budget. Given technical limitations on data compatibility, the data stored in the PforR’s M&E database were not transferred to the current website for disclosing these data on rural water supply established by the Directorate of Water Resource Management in Ministry of Agriculture and Rural Development.15Error! Hyperlink reference not valid. In addition, the mobile phone–based M&E system was introduced to the participating provinces to collect and analyze M&E data on access to water and sanitation facilities, but the use was not universal and was limited for the pilot purpose (World Bank 2020a).16 2.23 The misalignment of the disbursement schedules between the government system and the Results-Based RWSS PforR caused frictions in financial management. There were some cash deficiencies in the PforR’s annual budgets in the participating provinces. The contractor of the water supply programs bore the shortage of capital that was not covered by the 20 percent advance, per the contract provision. There were other challenges in the day-to-day operations such as recording the expenditures related to the PforR. During IEG’s interviews, respondents pointed out that the PforR could have benefited from stronger political buy-in by the central ministries with responsibilities for budget planning and allocation, that is, the Ministry of Finance and the Ministry of Planning and Investment. 3. Lessons 3.1 The experience from this PforR suggests the following lessons: 3.2 PforR design needs to be closely aligned with national policies and regulations, particularly regarding financial management. To avoid frictions and imbalances in financial management, the design of PforRs needs to ensure that cash deficiencies do not occur in the provincial and subnational annual budgets because of differences between the disbursement schedules of the government system and the PforR operation. An exacerbating factor is that the Vietnamese budgeting and transfer system did not have a mechanism for adaptable performance-based transfers from the central government to 19 provinces to enable results-based budget allocations (World Bank 2020a). In 2020, the government of Vietnam also issued a decree (No. 56/2020/ND-CP) that official development assistance and concessional loans be used to finance only capital expenditures and not recurrent expenditures, which prevented the development of a new PforR operation. 3.3 Both the design and implementation of the PforR need to exercise equity and inclusivity in targeting beneficiaries to avoid selection bias against hard-to-reach ethnic and the poorest minorities and to reduce their vulnerability in the long term. Results- based operations designed with DLIs have a built-in incentive to focus on achieving results efficiently, which could cause the provincial agencies to stay away from serving the vulnerable populations and areas. When selecting and prioritizing communes to be intervened under the RWSS PforR, there was an incentive to focus more on achievability of DLIs than reaching out to those with the greatest need. It resulted in inadequate representation of poor and ethnic minorities as target beneficiaries in relation to their share of the total population. Concrete measures need to be adopted to ensure that results-based operations provide implementing agencies with incentives to work in challenging areas, for example, the integration of poverty alleviation as a DLI target. 3.4 The existence of an enabling environment for private participation could enhance the effectiveness of PforRs. The results-based approach fosters the mobilization of private actors. The national policies and regulations promoted public-private partnerships, which were reflected in the design of the RWSS PforR. PforRs need to assess and harness opportunities to allow private enterprises to lead improvements in cost recovery ratios of water supply schemes. The experience with the Results-Based RWSS under the NTP PforR shows that Vietnam’s favorable enabling environment for private sector participation has enhanced the operation’s implementation effectiveness by improving the financial and operational sustainability of the country’s RWSS sector. 1Access to hygienic water in rural areas as reported in the National Target Program on Rural Water and Sanitation 2012–15 document, approved by Prime Minister’s Decision 366 (cited in World Bank 2012c). “Hygienic” is equivalent to the World Health Organization–United Nations Children’s Fund Joint Monitoring Program for Water Supply and Sanitation definition of “improved water source” or a source that “by nature of its construction or through active intervention, is protected from outside contamination, in particular from contamination with fecal matter.” 2Access to clean water in rural areas as of 2010, as reported in the National Target Program on Rural Water and Sanitation 2012–15 document, approved by Prime Minister’s Decision 366 (cited 20 in World Bank 2012c). “Clean” water, according to the Ministry of Health definition, refers to specific water quality parameters defined in the standards QCVN 02: 2009/BY. 3Clean water, according to Ministry of Health definition, refers to specific water quality parameters defined in the standards QCVN 02: 2009/BY. Sustainable water systems are those which, two years after the start of operation (defined as the month in which the first water is supplied, billed, and paid for by customer households), satisfy the following characteristics: (i) provide clean water, as defined by the Ministry of Health standards (QCVN 02: 2009/BYT) including, if necessary, specific control and measures for arsenic; (ii) the system is operating under a recognized management model; (iii) at least 85 percent of the number of planned system connections are working (providing water 24 hours a day, seven days a week and allowing for a 10 percent failure rate on a monthly average) and have had bills issued and paid; (iv) net revenue from tariffs is sufficient to cover operation and maintenance costs; and (v) nonrevenue water is less than 25 percent the month before verification is conducted (World Bank 2013). 4Cost recovery was defined in the Operational Manual (Vietnam 2016, page 40): “Operation and maintenance costs are covered by operating revenue from tariffs for the last 3 months of the reporting period.” 5 As defined by Standard QCVN 01:2011/BYT (World Bank 2013). 6Because the economic conditions of the rural populations in the participating provinces were relatively good, they showed a preference for building latrines with long periods of durability. Most of the sanitation facilities built under the PforR were septic tanks, which cost more than organic composting latrines that were mainly used in more remote areas. 7The national standards and regulations applied to the Results-Based RWSS under the NTP PforR included QCVN 01:2011/BYT, which was the National Technical Regulation on Hygienic Conditions for Latrines; and QCVN 02:2009/BYT, which was the National Technical Regulations on Domestic Water Quality (applied to small-size water plants with capacity less than 1,000 cubic meters per day night; Vietnam 2016). 8The procurement measures were first introduced to Vietnam under the Urban Water Supply Development Project (P073763) and were replicated in other projects. 9Vietnam East Meets West Foundation Global Partnership on Output-Based Aid (GPOBA) Rural Water Supply Development Project (P104528, fiscal years 2008–12). The project development objective was to increase sustainable access to piped water services to low-income households in rural central provinces of Vietnam through an innovative community-based approach, involving public-private partnership. An independent verification agent hired by GPOBA verified outputs in the water programs and reported quarterly to the World Bank and GPOBA with recommendations regarding eligible subsidy payments. GPOBA changed its name to the Global Partnership for Results-Based Approaches in February 2019. 10The number of visits to the website is per the Wayback Machine internet archive (https://web.archive.org/) as of February 26, 2020. Accessed the internet archive on October 29, 2021. 21 11The number of visits to the website is per the Wayback Machine internet archive (https://web.archive.org/) as of July 22, 2017. Accessed the internet archive on October 29, 2021. 12 The number of visits to the website is per the Wayback Machine internet archive (https://web.archive.org/) as of July 22, 2017. Accessed the internet archive on October 29, 2021. 13The amounts of private sector investments in the water and sanitation sector are per the World Bank’s Private Participation in Infrastructure Project database (https://ppi.worldbank.org/en/ppi). Accessed on October 29, 2021. 14Comparison provinces were Hoa Binh, Thai Nguyen, Hai Duong, Nam Dinh, and Nghe An. Those were the provinces in the Red River Delta and a number of northern provinces that share similarity of geographic conditions with Thanh Hoa and Phu Tho. 15 See http://nuocsachnongthon.thuyloivietnam.vn/ 16Six provinces used the mobile phone–based M&E system for water connection information, five provinces used it for household sanitation information, and three provinces used it for information on sanitation facilities in public buildings. 22 Bibliography Bac Ninh Province. 2012. “Five-Year Provincial Plan for Results-Based Rural Water Supply and Sanitation 2013–17.” Bac Ninh Province, Vietnam. Bank Information Center. 2014. “World Bank PforR in Practice: Results -Based Rural Water Supply and Sanitation under the National Target Program, Vietnam.” Bank Information Center, Washington, DC. Carrard, Naomi, Ben Madden, Joanne Chong, Melita Grant, Tuyến Phương Nghiêm, Ly Hà Bùi, Huế Thị Thu Hà, and Juliet Willetts. 2019. “Are Piped Water Services Reaching Poor Households? Empirical Evidence from Rural Vietnam.” Water Research 153 (April): 239– 50. Chase, Claire, and Quy-Toan Do. 2010. “Scaling Up Handwashing Behavior: Findings from the Impact Evaluation Baseline Survey in Vietnam.” Water and Sanitation Program technical paper. World Bank, Washington, DC. Hanoi. 2012. “Five-Year Provincial Plan for Results-Based Rural Water Supply and Sanitation 2013–17.” Hanoi, Vietnam. ISF (Institute for Sustainable Futures). 2014. “Rural Water Supply Systems and the Role of the Private Sector.” Policy Brief 3, ISF, University of Technology Sydney, Sydney, Australia. NCERWASS (National Center for Rural Water Supply and Sanitation). 2019. “Vietnam—Results- Based Rural Water Supply and Sanitation Under the National Target Program: Beneficiary Feedback System Report.” NCERWASS, Hanoi. Suardi, Mario. 2011. “Output-Based Aid in Vietnam: Access to Piped Water Services for Rural Households.” OBApproaches Note 41, Global Partnership on Output-Based Aid, Washington, DC. Vietnam, Government of. 2016. Results-Based Rural Water Supply and Sanitation under National Target Program (2013–18) Final Operational Manual (attached with Decision No. 157/QD- BNN-TCTL dated 23 January 2013; revised on November 29, 2016), Hanoi, Vietnam. Vietnam, GSO (General Statistics Office of Vietnam). 2009. The 2009 Vietnam Population and Housing Census: Completed Results, Hanoi, Vietnam. Vietnam, GSO (General Statistics Office of Vietnam). 2010. The 2010 Vietnam Household Living Standard Survey, Hanoi, Vietnam. Vietnam, GSO (General Statistics Office of Vietnam). 2019. Completed Results of the 2019 Vietnam Population and Housing Census, Hanoi, Vietnam. Vietnam, GSO (General Statistics Office of Vietnam) and UNICEF (United Nations Children’s Fund). 2015. Vietnam Multiple Indicator Cluster Survey 2014, Final Report, Hanoi, Vietnam. 23 Vietnam, Ministry of Agriculture and Rural Development. 2019. Evaluation on Completion of Results-Based Rural Water Supply and Sanitation Under the National Target Program in 8 Red River Delta Provinces, Financed by the World Bank (Credit Number: 5176-VN). Hanoi, Vietnam. World Bank. 2011. Vietnam—Country Partnership Strategy, FY12–16. Washington, DC: World Bank. World Bank. 2012a. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program.” Environmental and Social Systems Assessment E3058, World Bank, Washington, DC. World Bank. 2012b. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program.” Fiduciary Systems Assessment, World Bank, Washington, DC. World Bank. 2012c. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program.” Program Appraisal Document 70529 -VN, World Bank, Washington, DC. World Bank. 2013. Vietnam—Credit Number 5176-VN Financing Agreement. Washington, DC: World Bank. World Bank. 2015. “Program-for-Results: Two-Year Review.” Operations Policy and Country Services 95123, World Bank, Washington, DC. World Bank. 2016a. “Global Study on Sustainable Service Delivery Models for Rural Water: Synthesis of Emerging Findings.” World Bank, Washington, DC. World Bank. 2016b. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/program-for- results-full.pdf. World Bank. 2016c. Socialist Republic of Vietnam Results-Based Rural Water Supply and Sanitation under the National Target Program: Impact Evaluation. Washington, DC: World Bank. World Bank. 2017. Vietnam—Country Partnership Framework, FY18–22. Washington, DC: World Bank. World Bank. 2019a. “Built to Last—Upgrading Rural Water Supply and Sanitation Through Results-Based Financing: The Vietnam Experience.” Results Brief (April 29). https://www.worldbank.org/en/results/2020/04/29/built-to-last---upgrading-rural-water- supply-and-sanitation-through-results-based-financing-the-vietnam-experience. World Bank. 2019b. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program (P127435).” Implementation Status and Results Report ISR33898, Independent Evaluation Group, World Bank, Washington, DC. 24 World Bank. 2019c. “Vietnam: Toward a Safe, Clean, and Resilient Water System.” World Bank, Washington, DC. World Bank. 2020a. “Vietnam—Results-Based Rural Water Supply and Sanitation under the National Target Program.” Implementation Completion and Results Report ICR4947, Independent Evaluation Group, World Bank, Washington, DC. World Bank. 2020b. “Vietnam—Results-Based Rural Water Supply and Sanitation under the National Target Program.” Implementation Completion and Results Report Review ICRR22074, Independent Evaluation Group, World Bank, Washington, DC. 25 Appendix A. Ratings Vietnam Results-Based Rural Water Supply and Sanitation under the National Target Program The Project Performance Assessment Report rates the outcome and the World Bank’s performance highly satisfactory, and the quality of monitoring and evaluation (M&E) substantial (table A.1.) These ratings are the same as those in the Implementation Completion and Results Report and the Implementation Completion and Results Report Review. Table A.1. ICR, ICR Review, and PPAR Ratings Indicator ICR ICR Review PPAR Outcome Highly satisfactory Highly satisfactory Highly satisfactory Bank performance Highly satisfactory Highly satisfactory Highly satisfactory Quality of monitoring and evaluation Substantial Substantial Substantial Source: World Bank 2020a, 2020b. Note: The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible Global Practice. The ICR Review is an intermediate Independent Evaluation Group (IEG) product that seeks to independently validate the findings of the ICR. PPAR = Project Performance Assessment Report. There were four main components in the Results-Based Rural Water Supply and Sanitation (RWSS) under the National Target Program (NTP) Program-for-Results (PforR). Component 1: Increase access to water supply services (estimate: $56.8 million; actual: $56.2 million). This component financed the construction of commune-level water supply programs such as treatment plants, distribution networks, and household connections. Component 2: Increase access to sanitation services (estimate: $21.4 million; actual: $20.6 million). This component financed the construction of hygienic latrines in commune households, schools, and health centers. Component 3: Information, education, and communication; and behavior change campaigns (estimate: $9.9 million; actual: $8.7 million). This component financed campaigns to disseminate information about the program; raise awareness of program benefits; and facilitate the modification of rural residents’ attitudes and behavior toward connecting, using, and maintaining their water and sanitation systems. Component 4: Capacity building (estimate: $6.0 million; revised: $4.8 million; actual: $4.9 million). This component financed activities outlined in the Program Action Plan, provided implementation support, and developed a results-based verification and 26 disbursement process. The Program Action Plan identified the following actions: (i) transparency and citizen engagement, (ii) procurement, (iii) financial management and auditing, (iv) land acquisition, and (v) focus on ethnic minorities. Implementation support financed World Bank supervision as well as capacity building, technical expertise, and communication support to implementing entities at both the national and provincial levels. After implementing agencies reported annual achievements against the six disbursement-linked indicators (DLIs), an independent verification agent verified these through documentation and physical audits. Annual disbursements were then made against the verified results. In addition to World Bank financing, a parallel grant financing from the Australian Agency for International Development (now Australian Aid) also supported technical assistance activities to address capacity gaps by helping with the following actions: (i) strengthening the planning and monitoring capacity of implementing agencies, including transparency and social accountability; (ii) improving financial management, operation and maintenance, and water quality management practices of implementing agencies that contribute to the sustainability of water supply systems and services; (iii) improving the capacity of implementing agencies for managing sustainable hygienic sanitation, developing sanitation action plans, and a community rewards mechanism; (iv) improving governance such as fiduciary and environmental and social practices; (v) strengthening the capacity of the State Audit of Vietnam to conduct annual verification of program results reported by the provinces and external audits; and (vii) supporting the Rural Water Supply and Sanitation Partnership by financing workshops and sessions to share lessons learned and emerging best practices from the program. 1. Relevance of the Objectives Objectives The Results-Based RWSS under the NTP PforR’s project development objective (PDO) was “to increase sustained access to water supply and sanitation services and improve sector planning, monitoring and evaluation in the Participating Provinces of the National Target Program for Rural Water Supply and Sanitation” (World Bank 2013, 5). Relevance of the Objectives Alignment with the government strategy. Throughout the implementation, the Results- Based RWSS under the NTP PforR’s objectives were in line with National Rural Clean Water Supply and Sanitation Strategy to 2020 (Vietnam Ministry of Construction and MARD 2000). The national strategy had development objectives on improved health, improved living conditions, and reduced environmental pollution from human and livestock excreta. It also introduced the concepts of demand-responsive approaches and 27 cost recovery to the rural water supply and sanitation (RWSS) sector. The RWSS PforR was designed to support the implementation of the National Target Program for Rural Water Supply and Sanitation’s third phase (2011–15), which was the government of Vietnam’s primary instrument to achieve the strategy’s objectives. The strategy’s development objectives were the overarching objectives of the RWSS PforR. At program closing, the RWSS PforR’s objectives were also relevant to the road map for economic, political, and social reform called Vietnam 2035 (World Bank and MPI 2016)— specifically, to four of the six areas in that road map: (i) enable economic modernization where the private sector is in the lead, (ii) build technological and innovative capacity, (iii) chart an environmentally sustainable development path and enhance climate resilience, and (iv) promote equality and inclusion of marginalized groups and develop policies to promote a harmonious middle-class society. Alignment with the World Bank’s assistance strategy. At appraisal, the RWSS PforR’s objectives were well aligned with the Country Partnership Strategy for fiscal years (FY)12–16, which aimed for an improved efficiency in the water sector and an “improved basic infrastructure and public service delivery” (World Bank 2011, 26). At program closing, the objectives were in line with the Country Partnership Framework for FY18–22, which had objectives aiming to “broaden economic participation of ethnic minorities, women, and vulnerable groups;” “improve access to quality public and private health services and reduce malnutrition;” “increase climate resilience and strengthen disaster risk management;” and “strengthen natural resource management and improve water security” (World Bank 2017, 23). The RWSS PforR’s objectives were also in line with the World Bank Group seeking to “strengthen private sector participation in the water sector, both as a provider of water services and as an investor” (World Bank 2017). The objectives of the RWSS PforR were more closely aligned with the Country Partnership Strategy than with the Country Partnership Framework, which had a stronger focus on outcomes. Previous sector experience. The World Bank has been supporting the rural water supply and sanitation sector in Vietnam for more than 20 years, beginning with its assistance to the government in developing the National Rural Clean Water Supply and Sanitation Strategy through 2020. The adaptable program loan project that preceded the RWSS PforR, the Red River Delta Rural Water Supply and Sanitation Project (P077287, FY06–13), successfully promoted enterprise management arrangements among water supply schemes in four provinces in the Red River Delta region and resulted in higher levels of cost recovery and accountability. The World Bank also supported the government as it successfully tested the output-based approach in water supply services in rural areas of Hanoi under a program supported by the Global Program for Output- Based Aid. 28 Country capacity and adequacy of the Program-for-Results (PforR) instrument. The RWSS sector in Vietnam was ideal for piloting the PforR, given the existence of a large National Target Program, the World Bank’s strong presence in the sector, and government interest in shifting toward a focus on results and sustainability. The PforR financial instrument was chosen as an alternative to the investment project financing financial instrument in this case, with an aim to improve sustainability in RWSS by providing support for water supply management arrangements and cost recovery and investment planning on water supply and sanitation at the provincial and national levels. The eight participating provinces in the Red River Delta regions had reasonable financial and operational capacities to implement the results-based disbursement model, based on their relatively good socioeconomic conditions. The model in the PforR financial instrument was expected to support clients in enhancing the effectiveness and efficiency of their development programs to achieve tangible and sustainable results. It aligned well with the RWSS sector context, the government interest, and the participating provinces’ capacities at that time. Disbursement-Linked Indicators Original DLIs. There were three DLIs with two subindicators under each DLI, as summarized in table A.2. DLI 1, which consisted of the delivery of infrastructure, had the largest fund allocation of 64 percent of total International Development Association credit. The subindicators were interlinked to incentivize the achievement of both targets for water supply and sanitation because the coverage of sanitation facilities was smaller than the coverage of water supply. DLI 2, which consisted of the sustainability of water supply systems and the access to commune-wide sanitation, aimed for a higher level of results than DLI 1. DLI 3, which consisted of the number of provincial annual plans and program reports, was the only nonscalable DLI. Table A.2. Original DLIs and DLI Allocation as Share of Total International Development Association Credit (percent) DLI Allocation as Share of Total IDA Description of DLIs Credit DLI 1 64 DLI 1.1: Number of new functioning water supply connections DLI 1.2: Number of newly constructed improved household sanitary latrines DLI 2 30 DLI 2.1: Number of people with water supply connections from sustainable water systems DLI 2.2: Number of people with access to commune-wide sanitation 29 DLI Allocation as Share of Total IDA Description of DLIs Credit DLI 3 6 DLI 3.1: Number of provincial annual plans approved by participating provinces DLI 3.2: Number of program reports disclosed to the public Total IDA credit 100 Source: World Bank 2012. Note: DLI = disbursement-linked indicator; IDA = International Development Association. Revised DLIs. The DLIs were revised at the restructuring in 2016 (table A.3). The interlinks in the subindicators were removed based on steady progress in achieving targets for the sanitation-related indicators in the early periods of implementation. DLI V was added, related to the sustainability of sanitation facilities in schools and health clinics. The DLIs related to sector planning, monitoring, and evaluation were combined to one DLI (DLI IV). The funds allocated to the new DLI on the sustainability of public sanitation facilities (that is, DLI IV) were transferred from the DLIs on infrastructure delivery (that is, DLI I and II), shifting the focus of the results framework more toward the sustainability of public sanitation facilities. Table A.3. Revised DLIs and DLI Allocation as Share of Total International Development Association Credit (percent) DLI Allocation as Share of Description of DLIs Total IDA Credit DLI I: Number of new functioning water supply connections 43 DLI II: Number of newly constructed improved household sanitary latrines 16 DLI III: Number of people with water supply connections from sustainable water 10 systems DLI IV: Number of people with access to commune-wide sanitation 20 DLI V: Number of communes where schools and health clinics have maintained 4 hygienic status DLI VI: Number of provincial annual plans approved by participating provinces 7 and program reports publicly disclosed Total IDA credit 100 Source: World Bank 2020a. Note: DLI = disbursement-linked indicator; IDA = International Development Association. 30 Table A.4. Revised DLIs and Share of Target Achieved Share of Target Indicator and Achieved Results Indicators DLI Number) Baseline Target Actual (%) Objective (i): Increase sustained access to water supply services Number of people with PI 1 DLI III 0 850,000 1,054,575 124 water supply connection from sustainable water systema Number of new II-1 DLI I 0 340,000 399,964 118 functioning water supply connections Objective (ii): Increase sustained access to sanitation services Number of people with PI 2 DLI IV 0 1,275,000 1,407,669 110 access to commune-wide sanitationb Number of newly II-2 DLI II 0 130,000 142,280 109 constructed improved household sanitary latrines New water and sanitation II-3 0 1,440 1,559 108 facilities in schools and health centers constructed and in use Number of communes II-4 DLI V 0 150 179 119 where schools and health clinics have maintained hygienic status Objective (iii): Improve sector planning, monitoring, and evaluation Number of provincial PI 3 DLI VI Existing plans Original: 85 102 100 annual plans and not Revised: 102 program reports from comprehensive, participating provinces partially implemented Program reports from II-5 DLI VI Reporting of Original: 5 6 100 MARD disclosed to the results not Revised: 6 public comprehensive 31 Share of Target Indicator and Achieved Results Indicators DLI Number) Baseline Target Actual (%) Corporate outcome indicator Direct project PI 4 0 1,861,000 2,149,765 116 beneficiaries (number) Of which female 51 (percentage) Source: World Bank 2020a. Note: DLI = disbursement-linked indicator; II = intermediate results indicator; MARD = Ministry of Agriculture and Rural Development; PI = project development objective indicator. a. Sustainable water systems are those which, two years after the start of operation (defined as the month in which the first water is supplied, billed, and paid for by customer households), satisfy the characteristics described in the Program Appraisal Document (World Bank 2012). b. Commune-wide sanitation relates to communes in which the characteristics described in the Program Appraisal Document have been achieved (World Bank 2012). Relevance of the Disbursement-Linked Indicators The DLIs were relevant to the PDO and the results framework, as summarized in table A.4. All six DLIs were linked with project development objective indicators (that is, DLI III, IV, and VI), intermediate results indicators (that is, DLI I and II), or both (that is, DLI V). The DLIs focused mainly on outputs. The World Bank’s previous sector experience in enhancing the enterprise model of water supply schemes and testing an output-based aid approach in the Red River Delta region, and the eight provinces’ relatively high financial and operational capacities ensured adequacy of the PforR instrument. The Results-Based RWSS under the NTP PforR’s objectives were in line with the government strategy and the World Bank’s assistance strategy for the sector at program closing. Although still aligned with the overarching sector development objectives in these strategies and previous World Bank support, the PforR’s objectives, design, and DLIs have a strong focus on outputs and are not directly related to those long-term, overarching objectives. The mechanics of the PforR instrument embed incentives that encouraged the implementing agencies to efficiently achieve results linked to disbursements, with the result that better-off communes with stronger institutional capacity receive greater priority compared with poorer and ethnic minority areas that are harder to reach and have lower capacity to deliver results, as discussed in section 2 of this Project Performance Assessment Report. Overall, the relevance of the objectives is rated high. 2. Efficacy The Project Performance Assessment Report’s assessment of the Results-Based RWSS under the NTP PforR’s results analyzes the PDO’s three objectives separately: (i) to increase sustained access to water supply, (ii) to increase sustained access to sanitation 32 services, and (iii) to improve sector planning, monitoring, and evaluation. This assessment draws on the findings of Independent Evaluation Group (IEG) interviews and a review of this PforR’s records. Objective 1: To increase sustained access to water supply IEG’s interviews and the review of program records found that the RWSS PforR was fully successful in increasing access to sustainable water supply services. The objective was critical to addressing the fundamental challenges to sustainable rural water supply services as articulated in the theory of change. Securing sustained access to water supply was based on overcoming the constraints posed by a low-cost recovery rate, high nonrevenue water rate, and low water quality. The definitions of the terms used in the DLIs (such as functioning water supply connections) were embedded in the quality requirements set by the Ministry of Health standard (World Bank 2013), which addressed the development challenge of low water quality. The number of people with a water supply connection from a sustainable water system (PDO indicator 1, DLI III), which required results verifications to be conducted two years after the schemes were operationalized, was 1,054,575 people, exceeding the target of 850,000 people (124 percent of the target; World Bank 2020a). At the closing of the RWSS PforR, the sustainability performance was remarkable for a subset of water supply schemes in the eight provinces that was measured for DLI III: the average cost recovery ratio of the water supply programs was 1.28, and the average nonrevenue water ratio was 19 percent (decreasing from more than 25 percent estimated at appraisal; World Bank 2020a).1 Of the 144 water supply schemes supported under the PforR, private enterprises operated 87 of them, accounting for 60 percent (World Bank 2020a).2 When compared with the counterfactual, the cost recovery ratio and the nonrevenue water ratio in the area covered under the RWSS PforR (that is, 1.18 and 22 percent, respectively) showed better results than those of the area not covered under the RWSS PforR (that is, 1.14 and 34 percent, respectively; World Bank 2020a). Objective 2: To increase sustained access to sanitation services The number of communes where schools and health clinics have maintained hygienic status (intermediate results indicator 4, DLI V)—which also required results verifications to be conducted two years after the sanitation facilities were put in use—was 179 communes, exceeding the target of 150 communes (119 percent of the target; World Bank 2020a). Regarding public sanitation facilities, provinces financially incentivized local communes to assign maintenance responsibilities (World Bank 2020a); however, to what extent the assigned responsibilities were fulfilled beyond the program period was unknown. Moreover, the time period (that is, two years after the start of use) set by the sustainability DLIs for the public sanitation facilities was questionable. During IEG’s 33 interviews, respondents commented that the septic tank was the popular type of latrine in the participating provinces, and though its cost would be relatively higher, it would last 10 to 15 years with proper maintenance. Regarding the sustainability of household sanitation facilities, the respondents to IEG’s interviews confirmed that the Vietnam Health Environment Management Agency introduced a model of a local store network for sales of toilet parts for installation and rehabilitation. Objective 3: To improve sector planning, monitoring, and evaluation The Results-Based RWSS under the NTP PforR also achieved documented success in improving sector planning, monitoring, and evaluation. Referring to the theory of change, the operation addressed the development challenge of inefficiency and nontransparency in planning and processing RWSS investments. The results-based disbursement model under the PforR encouraged government agencies to improve efficiency in implementation by strategically planning investments based on demands. The RWSS PforR fully met the targets of provincial annual investment plans and annual program reports from eight participating provinces (project development objective indicator 3, DLI VI) and one consolidated annual program report from the Ministry of Agriculture and Rural Development disclosed to the public (intermediate results indicator 5, DLI VI), totaling 102 reports in six years (World Bank 2020a). Overall Efficacy All the PDO indicators and the intermediate results indicators for the three objectives of the PDO were achieved at program closing, as shown in table A.4. Two years after the operationalization, both the water supply schemes and the public sanitation facilities were functional. The incentive system established by the results-based disbursement model improved sector planning and M&E of the government agencies. Overall, the efficacy is rated high. 3. Outcome The relevance of objectives was substantial, recognizing that the formulation of the PDO was not fully outcome-oriented and was without any reference to the overarching objectives envisioned in the strategies of the government, World Bank assistance, and the preceding World Bank operations. Therefore, the DLIs and the program design were focused too much on outputs. The overall efficacy was highly satisfactory because all three objectives in the PDO were achieved. Overall, the outcome is rated satisfactory. 34 4. Risk to Development Outcome At the time of the Project Performance Assessment Report, potential risks to the development outcomes were identified. Financial sustainability risk on the water supply schemes. Two years after the closure of the Results-Based RWSS under the NTP PforR, the IEG mission found that there were only low to moderate risks to the water supply programs’ financial sustainability. This resulted from the solid foundation for effective, long-term system maintenance and financial sustainability established by the activities that led to meeting the PforR’s sustainability DLI. This assessment was supported by interviews with provincial water service providers and the relevant government agencies responsible for rural water service provision. There is a continuing trend of private sector participation in operations of provincial water supply schemes. Compared with community- or public- led service provision, the privately managed water schemes in rural areas had higher profits and higher staff productivity (ISF 2014). Risk related to institutional support. In 2020, the government of Vietnam published a decree on management and use of official development assistance and concessional loans granted by foreign donors, which stated that “ODA [official development assistance] and concessional loans shall be used to finance capital expenditures only, and are not available for recurrent expenditures” (Vietnam 2020). This decree eliminated future possibilities for implementing a new PforR in the country. In the long term, this may erode the accumulated knowledge and experience on the results-based disbursement approach through the RWSS PforR and other PforRs that were being implemented in the country. 5. Bank Performance Quality at Entry The strategic relevance and approach of the PforR were aligned with the sectoral challenges identified in the series of NTPs in the RWSS sector. The World Bank’s assessments of technical, fiduciary, and economic and social aspects of the program informed and assisted the government in developing the Program Action Plan and the operations manual. Regarding institutional aspects, the grant financing for the parallel technical assistance was secured to strengthen planning, M&E, and verification capacities of the implementing agencies. The M&E arrangements were well structured, used the national M&E system, and identified indicators for improvement. The results verification system was designed to ensure independence and accurate representation of the results. Because the Results-Based RWSS under the NTP PforR was the first PforR financing instrument used by the World Bank in the water sector and the first to be 35 implemented in Vietnam, the World Bank team in the country worked closely with the Vietnamese government to enhance their understanding on the new instrument. The choice of the PforR instrument instead of the second phase adaptable program loan of the Red River Delta Rural Water Supply and Sanitation Project was appropriate, considering the government’s commitments and the institutional capacities of the sectoral ministries and the eight provinces. Overall, the quality at entry is rated highly satisfactory. Quality of Supervision The World Bank conducted regular supervision missions and effectively communicated the Results-Based RWSS under the NTP PforR’s implementation progress; identified emerging technical issues; and recommended corrective measures in a timely manner. Supervision focused on building the implementing agencies’ capacities and meeting all DLI targets. The World Bank proactively monitored environmental and social safeguards compliance issues. In the first years of implementation, when the technical assistance was delayed because of slower-than-expected procurement, the World Bank team provided capacity building to help provincial agencies prepare the annual investment plans and other activities. The World Bank provided timely and relevant analyses during the Mid-Term Review and restructuring. From the initiation of the RWSS PforR, the World Bank team monitored performance with the intention of adequately informing the Mid-Term Review to distill lessons and conduct any necessary restructurings. Implementation bottlenecks were identified, and corrective measures were recommended. When progress in the sanitation DLIs was lagging in some provinces because latrine costs were too high, the World Bank promoted new low-cost technologies for rural household latrines as evidence of the results focus. Overall, the quality of supervision is rated highly satisfactory. With quality at entry as highly satisfactory and quality of supervision also highly satisfactory, overall Bank performance is rated highly satisfactory. 6. Quality of Monitoring and Evaluation Design The theory of change toward the achievement of the PDO was clear and reflected in the results framework. The PDO clearly specified three objectives: sustained access to water supply, sustained access to sanitation, and improved planning and M&E. The indicators encompassed all results specified in the PDO statement, except for the missing DLI at appraisal to measure the sustainability of public sanitation facilities. The results framework focused mainly on outputs that were observable within the program duration. The terms used in the DLIs—sustainable water system and commune-wide 36 sanitation—were clearly defined. In general, the protocol and methodology for results monitoring and verification were technically sound and credible, though there were some ambiguities in measurability such as the number of samples originally required. The baselines and targets were available for all indicators. The M&E design and arrangements were well embedded institutionally because the existing NTP M&E system of 14 indicators for RWSS systems was adjusted and used for the RWSS PforR. Implementation The M&E system was well implemented in the provincial and national institutions. M&E data were collected at the local level and processed at the provincial level. The National Center for Rural Water Supply and Sanitation, in cooperation with the National Target Program Standing Office for Rural Water and Sanitation Services, assembled summary investments and progress reports of the indicators. The system used a geographical information system–based M&E platform, which mapped data on investments, coverage, and service delivery to the commune level for the entire country. The parallel grant financed technical assistance and provided training on improving data collection and using the existing M&E system as the foundation for provincial plans. Data submission from provinces to the central agencies was completed in a reasonable time frame. Some ambiguities in results verification methodologies of the DLIs were resolved in early periods of implementation. At restructuring, the DLIs were strengthened as follows: The interlinks among the water supply indicators and the sanitation indicators were diminished because there were some cases where the sanitation indicators outperformed the water supply indicators; and the sustainability DLI IV for the public sanitation facilities was added. There were no compliance or quality issues. Data reported were independently verified and used for disbursement. Use The M&E data were used in the provinces to make their annual investment plans, track the progress of implementation, and develop provincial annual reports. The provincial plans and reports were consolidated at the central level for public disclosure. The M&E data informed modification of the results framework at restructuring. Regarding the independent results verification, the State Audit of Vietnam became a regional reference for other countries and shared their experience in conferences in other countries. Overall, the quality of M&E is rated substantial. Bibliography ISF (Institute for Sustainable Futures). 2014. “Rural Water Supply Systems and the Role of the Private Sector.” Policy Brief 3, ISF, University of Technology Sydney, Sydney, Australia. 37 Vietnam, Government of. 2013. Results-Based Rural Water Supply and Sanitation under the National Target Program (2013–18) Final Operational Manual (attached with Decision No. 157/QD- BNN-TCTL, January 23), Hanoi, Vietnam. Vietnam, Government of. 2020. “Decree on Management and Use of Official Development Assistance (ODA) and Concessional Loans Granted by Foreign Donors.” Decree 56/2020/ND-CP, Hanoi, Vietnam. Vietnam, Ministry of Construction and MARD (Ministry of Agriculture and Rural Development). 2000. “National Rural Clean Water Supply and Sanitation Strategy Up to Year 2020.” Hanoi, Vietnam. World Bank. 2011. Vietnam—Country Partnership Strategy, FY12–16. Washington, DC: World Bank. World Bank. 2012. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program.” Program Appraisal Document 70529 -VN, World Bank, Washington, DC. World Bank. 2013. Vietnam—Credit Number 5176-VN Financing Agreement. Washington, DC: World Bank. World Bank. 2016. “Socialist Republic of Vietnam, Vietnam Results -Based Rural Water Supply and Sanitation under the National Target Program: Impact Evaluation.” Report ACS18130, World Bank, Washington, DC. World Bank. 2017. Vietnam—Country Partnership Framework, FY18–22. Washington, DC: World Bank. World Bank. 2020a. “Vietnam—Results-Based Rural Water Supply and Sanitation under the National Target Program.” Implementation Completion and Results Report ICR4947, Independent Evaluation Group, World Bank, Washington, DC. 1Cost recovery was defined in the Operational Manual (Vietnam 2013) as “Operation and maintenance costs are covered by operating revenue from tariffs for the last 3 months of the reporting period.” 2By the deadline for the government of Vietnam to request disbursements, only 93.8 percent of the commitment was disbursed, mainly because of increases in the private sector investment in rural water supply programs and the concessional credit from the Social and Policy Bank of Vietnam, which offset the total investment required from the government. 38 Appendix B. Fiduciary, Environmental, and Social Aspects Program-for-Results projects use government systems for their implementation, unlike investment project financing operations. During program preparation, the World Bank assessed the program in relation to technical, fiduciary, and environmental and social impacts to suggest recurring actions to strengthen the government system. The assessments reflected actions in the Program Action Plan (PAP). Financial Management To comply with the PAP’s Action 3 on financial management and internal audit capacity, each province took two actions: (i) preparing program financial statements and (ii) ensuring that the program internal audit function was working. The provinces prepared annual financial statements and followed an internal audit process. The Ministry of Agriculture and Rural Development submitted annual internal audit results reports to the World Bank. Procurement To comply with the PAP’s Action 2 on procurement practices, each province took four actions: (i) evaluating all proposals for detailed designs, construction supervision, and bids for civil works; (ii) ensuring that at least 50 percent of the water subprojects’ consultancy contracts and 50 percent of works contracts were competitively bid, increasing to 80 percent by the end of the program period; (iii) excluding dependent state-owned enterprises from participation; and (iv) excluding debarred firms on local, national, or bank debarment lists from participation. According to the procurement reports prepared by the Ministry of Agriculture and Rural Development, the provinces complied with all the actions. Progress on strengthening provincial government agencies’ capacities on competitive bidding was notable, considering the baseline at the Fiduciary Systems Assessment requiring almost all consultant contracts and about 60 percent of civil works contracts to be awarded on a single-source-selection or direct contracting basis, respectively (World Bank 2012).1 Environmental and Social Aspects To comply with the PAP’s Action 4 on land acquisition, each province took three actions: (i) avoiding and minimizing adverse land acquisition impacts; (ii) compensating people for loss of land and assets; and (iii) using a valuation mechanism for land, assets, and income. According to the detailed verification results, 1,116 households were 39 affected by the area of land acquired—136,616 square meters of public land and 260,384 square meters of agricultural land; four households had more than 30 percent of cultivated land acquired; no households lost their residential land; and no houses had to relocate (World Bank 2020). Land acquisition, compensation, and resettlement were completed in the participating provinces. No issue was reported regarding influx of labor, which was monitored during the past two periods of program implementation. To comply with the PAP’s Action 5 on ethnic minorities, each province followed guidelines for working with ethnic minorities. The Provincial Center for Rural Water Supply and Sanitation in Quang Ninh and Phu Tho provinces prepared and implemented the Ethnic Minorities Development Plan; both provinces had ethnic minority people in the program area. Ethnic minority people in the target communes were provided with information on the program, including activities, benefits, implementation schedule, and compensation. The program received support from the ethnic minority households in Quang Ninh and Phu Tho provinces that started to use clean water from water schemes built under the program (World Bank 2020). Environmental aspects related to the construction of the water supply schemes were integrated in the PAP’s Action 2. In accordance with the environmental protection law, all the water supply schemes under the Results-Based Rural Water Supply and Sanitation under the National Target Program Program-for-Results made commitments toward environmental protection, which were integrated into program bidding documents and contractors’ construction contracts. The PforR supervised contractors’ compliance with environmental management plans. Moreover, the project owners of water supply schemes reviewed and screened the potential environmental impacts of the schemes to ensure that all the water from the program was free of arsenic. References World Bank. 2012. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program.” Fiduciary Systems Assessment, World Bank, Washington, DC. World Bank. 2020. “Vietnam—Results-Based Rural Water Supply and Sanitation Under the National Target Program.” Implementation Completion and Results Report ICR4947, Independent Evaluation Group, World Bank, Washington, DC. 1Water supply programs financed by private entities, for which procurement was not governed by national procurement laws and regulations, were not monitored in this clause. 40 Appendix C. Methods and Evidence This report is a Project Performance Assessment Report. This instrument and its methodology are described at https://ieg.worldbankgroup.org/methodology/PPAR. This Project Performance Assessment Report gathered the evidence reported in section 2 to support its findings and conclusions using the following methodology. Review of Results-Based Rural Water Supply and Sanitation under the National Target Program Program-for-Results documents. The Independent Evaluation Group (IEG) evaluator began with a review of the documentation produced by the World Bank and the government agencies during project identification, preparation, and implementation. This included the World Bank’s Country Partnership Strategy (2011), Technical Assessment (2012), Fiduciary Systems Assessment (2012), Environmental and Social Systems Assessment (2012), Program Appraisal Document (2012), financing agreement (2013), process evaluation (2016), Operations Policy and Country Services Assessment of PforR instruments (2016), IEG early assessment of PforR instruments (2016), Country Partnership Framework (2017), relevant aide-mémoire, provincial annual and five-year plans and reports, Implementation Completion and Results Report (2020), and IEG Implementation Completion and Results Report Review (2020). The national and provincial governments’ documents included annual provincial investment plans and reports, annual program reports, Gender Integration Activities Assessment Report (2015), Operational Manual (2016), the summary of the Implementation Completion and Results Report for the Recipient-Executed AusAID Trust Fund (TF 13061) financed by the Australian government (2016), the evaluation on completion of the Results-Based Rural Water Supply and Sanitation Program in eight Red River Delta provinces financed by the World Bank under the National Target Program prepared by the Ministry of Agriculture and Rural Development (2019), and Audit Report (2020). Review of scholarly articles and reports by published by authors external to the World Bank and the government. The IEG evaluator reviewed scholarly articles and reports from academia, nongovernmental organizations, the United Nations’ funds and agencies, and multilateral development banks on relevant topics including sustainability of rural water service provisions by types of management models, the private sector’s role in rural water supply services, and progress and challenges on rural hygiene and sanitation. Virtual mission in Vietnam. The IEG evaluator led a three-week field mission to Vietnam from August 16 to 31, 2021, to meet with program stakeholders at the national and provincial levels. The mission covered a series of meetings conducted online with Vietnamese government officials in the national agencies involved in program 41 implementation. These included the Ministry of Agriculture and Rural Development, the National Center for Rural Water Supply and Sanitation, Vietnam Health Environment Management Agency, and the State Audit of Vietnam. The IEG team also met virtually with officials in the provincial agencies involved in program implementation. These included Provincial Centers for Rural Water Supply and Sanitation in the provinces of Bac Ninh, Hung Yen, Thanh Hoa, and Vinh Phuc; and the Center for Disease Control and Prevention in Bac Ninh province. The IEG team met with government officials who played key roles in designing and implementing the Results- Based Rural Water Supply and Sanitation under the National Target Program Program- for-Results (see appendix D) to collect their perspective on the overall success of the project; challenges and positive outcomes; coordination among implementing agencies; lessons learned; and changes that occurred at the institutional level through the duration, and as a result, of the program. Selection of provincial agencies for virtual meetings. The selection of provincial agencies for the virtual meetings was made based on criteria proposed by the IEG evaluator to ensure collection of substantial responses from the stakeholders who were involved in the actual implementation. The criteria considered (i) diversity in geographic location (provinces next to Hanoi, then farther from Hanoi); (ii) diversity in the ministerial instruction system responsible for the objectives on water supply and sanitation (agencies under the Ministry of Agriculture and Rural Development and the Ministry of Health); and (iii) water supply scheme implementation status (sustainable and ongoing operations). The subproject sites were selected based on these criteria. Given the limited number of subprojects that the virtual mission could cover, it was essential to adopt a purposive sampling procedure rather than a random sampling procedure to enable the mission to understand what worked and what did not work in these specific cases. Limitation. The IEG mission team was unable to conduct interviews with the private water supply providers and the beneficiaries of the rural water supply and sanitation services. To complement this, the IEG mission team interviewed personnel at the National Center for Rural Water Supply and Sanitation and the Provincial Center for Rural Water Supply and Sanitation, members of which had closely worked with the team on the Results-Based Rural Water Supply and Sanitation under the National Target Program Program-for-Results. 42 Appendix D. World Bank Rural Water Supply and Sanitation Projects in Vietnam Table D.1. World Bank Projects in the Rural Water Supply and Sanitation Sector over Time Outcome Financing Means of Rating in Geographic Project or Program Instrument Expected Results Verification ICRR Targeting Urban Water Supply Development Specific Health and Beneficiaries’ Moderately Urban: 27 Project (P073763) investment loan economic inception unsatisfactory provinces potential of the survey across the resident country households enhanced Red River Delta Rural Water Adaptable Poverty alleviation End Satisfactory Suburban: 4 Supply and Sanitation Project program loan in participating evaluation provinces in (P077287) communes on the Red through enhanced beneficiaries’ River Delta health and health and region household household economy economy Vietnam EMWF GPOBA Rural Recipient- Access to clean Independent Not Urban: 5 Water Supply Development executed grant water services, verification applicable provinces in Project (P104528) through the which would agency the central GPOBA deliver satisfactory part service for at least six months Results-Based Rural Water Supply PforR Sustained access Independent Highly Suburban: 8 and Sanitation under the National (at least two years verification satisfactory provinces in Target Program (P127435) after agency the Red operationalization) River Delta to water supply region and sanitation services and improved sector planning, M&E Results-Based Scaling Up Rural PforR Hygiene behavior Independent Not available Rural: 21 Sanitation and Water Supply increased, and verification yet provinces in Program (P152693) access to RWSS agency the northern increased and mountains sustained and Central Highlands Source: World Bank 2007, 2013a, 2013b, 2015, 2020. Note: EMWF = East Meets West Foundation; GPOBA = Global Partnership on Output-Based Aid (now the Global Partnership for Results-Based Approaches); M&E = monitoring and evaluation; PforR = Program-for-Results; RWSS = rural water supply and sanitation. 43 References World Bank. 2007. “Vietnam—EMWF GPOBA Rural Water Supply Development Project.” Commitment Paper 49294, World Bank, Washington, DC. World Bank. 2013a. “Vietnam—Red River Delta Rural Water Supply and Sanitation Project.” Implementation Completion and Results Report ICR2752, Independent Evaluation Group, World Bank, Washington, DC. World Bank. 2013b. “Vietnam—Water Supply Development Project.” Implementation Completion and Results Report ICR2828, Independent Evaluation Group, World Bank, Washington, DC. World Bank. 2015. “Vietnam—Results-based Scaling up Rural Sanitation and Water Supply Program.” Program Appraisal Document 100485-VN, World Bank, Washington, DC. World Bank. 2020. “Vietnam—Results-Based Rural Water Supply and Sanitation under the National Target Program.” Implementation Completion and Results Report ICR4947, Independent Evaluation Group, World Bank, Washington, DC. 44 Appendix E. Poverty Rates by Commune in Bac Ninh Province Table E.1. Poverty Rates in Bac Ninh Province by Commune Number of Poverty District Commune Population Households (%) Group Huyện Gia bình Xã Song giang 6,625 1,780 18.04 Huyện Gia bình Xã Giang sơn 6,455 1,682 15.92 Huyện Gia bình Xã Vạn ninh 6,724 1,964 15.2 2 Huyện Gia bình Xã Quỳnh phú 5,889 1,739 12.75 Huyện: Lương Tài Xã: lai hạ 4,163 1,339 12.5 3 Huyện Thuận Thành Xã ngũ thái 7,282 1,930 12.38 Huyện: Lương Tài Xã: mỹ hương 5,523 1,785 12.21 Huyện: Lương Tài Xã: lâm thao 6,356 1,797 12.19 Huyện: Lương Tài Xã: phú lương 3,736 1,054 12.01 Huyện: Lương Tài Xã: bình định 8,550 2,456 11.66 2 Huyện Quế võ Xã Chi lăng 8,290 2,020 11.42 Huyện Quế võ Xã Phù lương 5,309 1,416 11.11 Huyện Yên phong Xã Tam đa 11,475 3,116 10.93 Huyện Gia bình Xã Đông cứu 5,395 1,693 10.89 Huyện: Lương Tài Xã: an thịnh 9,833 2,964 10.45 Huyện Quế võ Xã Yên giả 4,571 1,200 10.02 Huyện Gia bình Xã Thái bảo 5,814 1,879 9.94 2 Huyện: Lương Tài Xã: phú hòa 10,040 2,894 9.83 1 Huyện Quế võ Xã Đào viên 10,385 2,901 9.77 Huyện Quế võ Xã Đức long 6,298 1,555 9.68 Huyện: Lương Tài Xã: quảng phú 10,990 2,774 9.67 Huyện Quế võ Xã Ngọc xá 8,777 2,269 9.66 Huyện Quế võ Xã Việt thống 5,596 1,177 9.59 3 Huyện Quế võ Xã Hán quảng 4,165 938 9.52 Huyện Gia bình Xã Cao đức 5,288 1,734 9.33 1 Huyện: Lương Tài Xã: trung kênh 9,382 2,915 9.18 Huyện Quế võ Xã Châu phong 5,720 1,674 8.63 Huyện: Lương Tài Xã: tân lãng 5,475 1,648 8.59 Huyện Gia bình Xã Đại bái 9,332 2,192 8.49 Huyện: Tiên Du Xã: minh đạo 6,556 1,549 8.35 2 Huyện Quế võ Xã Bồng lai 8,286 2,199 8.34 Huyện Gia bình Xã Xuân lai 8,419 2,403 8.03 Huyện Quế võ Xã Quế tân 6,279 1,629 8.01 Huyện Quế võ Xã Nhân hoà 7,787 1,851 7.97 3 45 Number of Poverty District Commune Population Households (%) Group Huyện Gia bình Xã Lãng ngâm 7,056 1,875 7.89 Huyện Quế võ Xã Phù lãng 7,191 1,778 7.84 1 Huyện Quế võ Xã Cách bi 5,884 1,494 7.71 Huyện: Lương Tài Xã: trung chính 8,293 2,574 7.39 Huyện Thuận Thành Xã trạm lộ 7,930 2,279 7.37 2 Huyện Thuận Thành Xã hà mãn 5,619 1,326 7.36 Huyện: Lương Tài Xã: minh tân 4,720 1,407 7.32 Huyện Gia bình Xã Đại lai 7,711 2,246 7.19 Huyện Thuận Thành Xã đình tổ 11,167 2,912 7.15 Huyện: Lương Tài Xã: trừng xá 4,554 1,634 7.12 Huyện Quế võ Xã Mộ đạo 4,872 1,157 7.09 Huyện Thuận Thành Xã ninh xá 8,700 2,298 6.97 2 Huyện Thuận Thành Xã song liễu 4,205 1,128 6.95 TP Bắc ninh Xã: khúc xuyên 3,542 1,034 6.87 Huyện Yên phong Xã Yên trung 11,857 2,750 6.87 Huyện Yên phong Xã Trung nghĩa 10,067 2,429 6.87 Huyện Quế võ Xã Bằng an 4,254 1,007 6.83 Huyện Thuận Thành Xã đại đồng thành 11,282 2,728 6.74 Huyện Yên phong Xã Dũng liệt 8,225 1,858 6.66 Huyện Yên phong Xã Long châu 8,231 2,100 6.61 Huyện Yên phong Xã Tam giang 10,315 2,555 6.49 1 Huyện Yên phong Xã Đông phong 7,935 2,015 6.46 Huyện Thuận Thành Xã trí quả 8,230 2,113 6.36 TP Bắc ninh Xã: nam sơn 9,093 2,574 6.32 Huyện Yên phong Xã Thuỵ hoà 7,463 1,740 6.2 Huyện Quế võ Xã Đại xuân 9,207 2,241 6.15 3 Huyện Yên phong Xã Đông tiến 7,080 1,821 6.11 2 Huyện Thuận Thành Xã thanh khương 6,500 1,534 6.1 Huyện: Tiên Du Xã: phật tích 6,397 1,844 6.03 2 Huyện Thuận Thành Xã nguyệt đức 8,229 2,142 5.8 Huyện Gia bình Xã Bình Dương 5,918 1,778 5.47 1 TP Bắc ninh Xã: kim chân 4,572 1,301 5.42 Huyện Yên phong Xã Văn môn 10,350 2,285 5.33 Huyện Gia bình Xã Nhân thắng 8,193 2,657 5.3 2 TP Bắc ninh Xã: hoà long 10,324 2,569 5.28 Huyện Quế võ Xã Việt hùng 9,361 2,418 5.27 Huyện Thuận Thành Xã hoài thượng 8,850 2,324 5.04 2 Huyện: Tiên Du Xã: liên bão 9,317 2,637 5 2 Huyện: Tiên Du Xã: hiên vân 6,127 1,706 4.91 Huyện: Tiên Du Xã: lạc vệ 12,181 3,328 4.87 46 Number of Poverty District Commune Population Households (%) Group Huyện: Tiên Du Xã: phú lâm 15,652 3,948 4.81 3 Huyện Yên phong Xã Hoà tiến 8,397 2,109 4.79 2 Huyện Yên phong Xã Yên phụ 11,621 2,506 4.47 1 Huyện: Tiên Du Xã: Tri phương 7,716 2,250 4.46 1 Huyện Thuận Thành Xã gia đông 9,185 2,425 4.18 Huyện Thuận Thành Xã nghĩa đạo 8,243 2,231 3.98 2 TP Bắc ninh Xã: Khắc niệm 9,317 2,253 3.97 1 Huyện Quế võ Xã Phượng mao 5,133 1,297 3.86 Huyện: Tiên Du Xã: việt đoàn 10,040 2,706 3.77 Huyện Thuận Thành Xã Mão Điền 12,081 3,467 3.74 2 Huyện: Tiên Du Xã: đại đồng 10,837 2,775 3.64 Huyện: Tiên Du Xã: cảnh hưng 5,406 1,442 3.41 Huyện Thuận Thành Xã Xuân lâm 6,562 1,695 3.41 Huyện Thuận Thành Xã song hồ 5,821 1,473 2.94 Huyện: Tiên Du Xã: tân chi 7,513 2,121 2.88 TX Từ sơn Xã Tam sơn 11,468 3,288 2.83 Huyện Thuận Thành Xã an bình 7,880 2,158 2.77 Huyện: Tiên Du Xã: hoàn sơn 12,307 3,357 2.72 1 TX Từ sơn Xã Phù chẩn 9,150 2,078 2.43 TX Từ sơn Xã Tương giang 11,312 3,575 2.38 Huyện Yên phong Xã Đông thọ 7,268 1,701 2.34 1 Huyện: Tiên Du Xã: nội duệ 8,017 2,108 2.22 TX Từ sơn Xã Hương mạc 14,630 3,258 2.16 3 TX Từ sơn Xã Phù khê 9,716 2,621 1.99 3 TP Bắc ninh Xã: phong khê 9,424 2,256 1.49 Huyện Quế võ Xã Phương liễu 10,185 3,899 0.75 Source: Monitoring and evaluation indicator report by the Provincial Center for Rural Water Supply and Sanitation and Department of Labor, Invalids, and Social Affairs Bac Ninh, cited in Bac Ninh Province 2012. Note: Poor households are those with average income less than or equal to 400.000 Vietnamese dong per person per month. 47 Appendix F. Borrower Comments 48