Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Report Number : ICRR0023483 1. Project Data Project ID Program Name P143596 Nat'l Urban Devt Program - NM Country Practice Area(Lead) Vietnam Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Program Cost (USD) IDA-54760 30-Jun-2021 199,299,493.79 Bank Approval Date Closing Date (Actual) 05-Jun-2014 30-Jun-2022 IBRD/IDA (USD) Grants (USD) Original Commitment 250,000,000.00 0.00 Revised Commitment 250,000,000.00 0.00 Actual 199,299,493.79 0.00 Prepared by Reviewed by ICR Review Coordinator Group Cynthia Nunez-Ollero Vibecke Dixon Avjeet Singh IEGSD (Unit 4) 2. Program Context and Development Objectives DEVOBJ_TBL a. Objectives According to the Financing Agreement (FA, p.5) and the Program Appraisal Document (PAD, paragraph 24) the Program Development Objective of this Results Based National Urban Development Program in the Northern Mountains Region of Vietnam was "to strengthen the capacity of participating Northern Mountains cities to plan, implement, and sustain urban infrastructure." The program targeted seven Northern Mountains cities—Bac Kan, Cao Bang, Dien Bien Phu, Hoa Binh, Thai Nguyen, Tuyen Quang and ,Yen Bai. Page 1 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) This PforR program supported the following activities to implement the government's National Urban Development Program (NUDP): (i) performance-based transfers for local infrastructure in participating cities; (ii) capacity support to participating cities; and (iii) support to national development policy and oversight. This review will assess the program performance against the following objectives: to strengthen the capacity of participating Northern Mountains cities  to plan urban infrastructure.  to implement urban infrastructure.  to sustain urban infrastructure. b. Were the program objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The program has the following four components under four results areas (RAs) supported by disbursement- linked indicators (DLIs): 1. Institutional Strengthening at City Level (US$55.0 million at appraisal, US$35.4 million actual, as reported in the ICR, Annex 1B) This component was to finance activities to strengthen the capacity of the target cities to plan infrastructure investments. This component was to form part of the Results Area 1 of the program: Institutional Strengthening at City Level. Two Disbursement Linked Indicators (DLIs) were to support this component:  DLI 1.1: Enhanced Annual City Plans Approved and Disclosed to the public (PDO Outcome Indicator 1)  DLI 1.2 Professionally staffed management unit in place in each participating City People's Committee. 2. Delivery of Infrastructure Services in Participating Cities (US$155 million at appraisal, US$89.2 million actual) This component was to finance activities that would strengthen the capacity of City People’s Committees (CPCs) to implement urban infrastructure sub-projects. This component was to form part of the Results Area 2 of the program: Delivery of Infrastructure Services in Participating Cities. One DLI was to support this component:  DLI 2: Local urban Infrastructure investments delivered under the Participating City's approved enhanced annual city plan. 3. Sustainable Infrastructure Investments (US$20 million at appraisal, US$7.5 million actual) This component was to finance the following activities: (i) develop improved mechanisms for operations and maintenance (O&M) of infrastructure; (ii) enhance fiscal capacity to fund O&M; and (iii) sustain ongoing Page 2 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) levels of investment in infrastructure and replacement. This component was to form part of the Results Area 3 of the program Sustainable Infrastructure Investments. Two DLIs were to support this component:  DLI 3.1 Asset management plan adopted and completed local urban infrastructure sub-projects in full service (PDO Outcome indicator 3)  DLI 3.2 Increase in own resource revenue 4. National Policy, Support and Oversight. (US$20 million at appraisal, US$10.6 million actual) This component was to finance activities to enhance the enabling policy environment, technical capacity, and oversight systems of the national government in support of cities’ urban development efforts. This component was to form the Results Area 4 of the program: National Policy Support and Oversight. Three DLIs were to support this component:  DLI 4.1: Implementation Strategy for National Urban Development Program adopted with annual milestones (PDO Outcome indicator 4)  DLI 4.2: Professionally staffed unit in place in the Ministry of Construction (MOC), preparation of annual capacity development plans and capacity building support provided to cities in accordance with such plans.  DLI 4.3: Completed program report. e. Comments on Program Cost, Financing, Borrower Contribution, and Dates Program Cost: The original program cost was US$250 million. The credit disbursed US$199.3 million (from the Data Sheet. The World Bank task team clarified that the disbursements in ICR Annex 1B based on the DLI disbursements was SDR 142.8 million). Two factors led to the cancellation of the undisbursed amount: (i) the target for DLI 3.2 (increase in own-sources revenue) was not achieved. The allocation was not disbursed, and (ii) the government implemented a policy change in 2019 that prohibited Overseas Development Assistance (ODA) funds from financing tax and capacity development activities (ICR, paragraph 59). This affected the program's capacity building activities. Financing: The International Development Association (IDA) financed this P4R program. Borrower Contribution: The government committed to contribute US$50 million at appraisal. The government disbursed US$53.5 million. The ICR noted that an Operations Portal error reflected US$20 million as government contribution at appraisal (ICR, Annex 3). Dates: The program was approved on June 5, 2014, and was made effective on October 30, 2014. The Mid Term Review (MTR) was conducted on May 7, 2018. The program was to close initially on June 30, 2021 but was extended by another 12 months to close on June 30, 2022, according to a level 2 restructuring approved on June 30, 2021 (Restructuring Paper, Report No. RES47072). The extension allowed (i) the completion of 19 remaining infrastructure works approved under the Enhanced Annual City Plans (EACPs) and (ii) the completion of the resettlement sites in five cities. Split Rating: No split rating of the outcome will be undertaken. The program PDO remained unchanged throughout the operation. The scope of the PforR activities also remained unchanged. Restructuring Page 3 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) realigned resources to match changes in target values. The cancelled funds did not impact the Theory of Change (TOC). 3. Relevance TBL a. Relevance of Objectives New Rationale Alignment with Sector Strategy: The PDOs of this PforR operation were aligned with the government's National Urban Development Program (NUDP), which addressed growing inequality due to the poor urban management of lagging regions. The NUDP set national targets for efficient, sustainable, and equitable urban services. The central government sought to enhance the capacity of these regions in urban development management by offering participating cities and provinces a comprehensive package of performance-based fiscal and capacity support combined with enhanced oversight. The Ministry of Construction (MOC) prepared a National Urban Development Strategy (NUDS) to implement the NUDP and outlined the regional rollout of the government’s priorities for urban development. This PforR piloted the rollout of the NUDS by targeting the poorer cities of the Northern Mountains Region. The program supported strengthening urban management systems for investment planning, works management, operational and financial sustainability, and public consultation. The PDOs were also relevant to the government's development priorities laid out in Vietnam 2035. This plan was a joint government-World Bank report that outlined the country’s development directions. This PforR helped the government implement three of six key programs to help Vietnam reach the following 2035 goals: (i) reshape urban policies and investments for more dynamic cities and urban centers by preparing high-quality city plans and strengthening cities’ capacity to plan; (ii) pursue an environmentally sustainable development path by increasing adaptation and resilience to changing climate patterns, by supporting climate-resilient urban infrastructure and strengthening cities’ capacity to implement; and (iii) promote equality and inclusion among marginalized groups for a harmonious middle-class, by prioritizing infrastructure investments in low-income areas, adopting inclusive implementation processes for ethnic minorities (EMs) and women, and monitoring program results. The PforR PDOs were relevant to the three other recent government policies. First, Resolution No. 06-NQ/TW, effective January 24, 2022 on planning, construction, management, and sustainable development of urban areas in Vietnam till 2030. Second, Decision 438 No. 438/QD-TTg, effective March 25, 2021, on approved schemes for developing climate-smart cities in 2021 – 2030. Third, Decision 950 No. 950/QD-TTg, effective August 1, 2018, on approved schemes for developing smart, sustainable cities in Vietnam in the 2018 – 2025 period. This last policy supported effective investment planning in response to the cities’ growth needs, economic impacts, and poverty reduction. Sustainable urban infrastructure would be met by enhancing institutional capacity and integrating climate risks into infrastructure investments. This PforR was to equip the cities in these lagging regions to meet the demands of a burgeoning population by increasing their institutional capacity to plan, implement, and sustain urban infrastructure. The government developed an urban classification system with six classes of cities - the first was the special cities of Ho Chi Minh City and Hanoi. The other five were classified by levels of economic activities, physical development, population, population density, and infrastructure provision. By 2020, in addition to the two special cities (Ho Chi Minh City and Hanoi), Vietnam had 20 class I cities, 29 class II cities, 45 class III towns, 85 class IV provincial towns, and 652 class V small townships. Ten of 19 cities in the Northern Mountains Region were Page 4 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Class IV cities with less than 100,000 residents. The region had poor connectivity within cities, between urban centers, and surrounding peri-urban and rural areas. Many urban areas lacked basic urban and social infrastructure such as roads, sanitation, drainage, flood protection facilities, street lighting, public green space, and schools. The mountainous setting of the growing urban settlements raised geotechnical and land availability challenges. Low-income settlements were often located in flood- or landslide-prone areas and were exposed to climate disasters. The provision of serviced land in safe areas for high-density housing and economic activities was limited. City infrastructure was built without accounting for intensifying climate change risks. World Bank Previous Sector and Country Experience: The PDOs remained relevant to the current World Bank Group Country Partnership Framework (CPF) for Vietnam for FY18-22. A new CPF to cover the period FY23-27 was under preparation at closing. The PDOs were particularly relevant to Objective 4, “Improve planning, management, and delivery of infrastructure and land in cities,” with program financing focused on improving the institutional capacity of the target cities of the Northern Region to plan, deliver, and sustain infrastructure service delivery. The PDOs also supported achieving Objective 5, “Improve the welfare and development of ethnic minorities, women, and vulnerable groups,” under Focus Area 1, “Enable inclusive growth and private sector participation.” The program was designed to encourage the participation of these groups in prioritizing infrastructure investments. The PDOs were also relevant to Objective 10, “Increase climate resilience and strengthen disaster risk management,” under Focus Area 3, “Ensure environmental sustainability and resilience.” The program financed the construction of drainage systems, embankments, channel rehabilitation, and retaining walls to enhance communities’ resilience to the impacts of climate change and natural disasters in low-income areas most affected by disasters, which contributed to achieving Objective 10. The program activities also complemented the World Bank’s ongoing policy engagement in Vietnam on decentralization and strategy for second-tier cities focused on access to climate-resilient basic urban infrastructure services. Country Capacity and Adequacy of PforR Instrument: The government piloted NUDP implementation in the Northern Mountains Region to address extreme poverty and promote shared prosperity. The growing urban settlements in the small cities of the Northern Mountains Region were to serve as economic and social hubs of the regional economy. MOC was mandated to coordinate the program’s activities. MOC provided capacity support to cities coordinated, appraised, and supervised cities’ urban development activities. This PforR was the instrument of choice to pilot the rollout of the first NUDP regional sub-program in the Northern Mountains Region. The PforR targeted the poorest region to highlight the development disparities among cities. The program defined what the NUDP lacked - the implementation modalities for effective planning, financing, implementation, and monitoring systems of the cities' urban strategies. This PforR piloted a scalable regional approach to achieve the NUDP objectives using disbursement-linked incentives under four Results Areas. The PforR was to implement the NUDP by (i) supporting urban infrastructure investment (Results Area 2 - Delivery of Infrastructure Services and Results Area 3 - Sustainable Infrastructure Investments), (ii) strengthening urban management by improving mechanisms for investment planning and works management, increasing public engagement, and building capacity of local officials (Results Area 1 - Institutional Strengthening at the City Level), and (iii) piloting a results-based financing approach to NUDP implementation and developing an implementation framework for the next phase. These incentives were to lead to adopting measures that would strengthen national government oversight (Results Area 4). Overall, the PDOs were highly relevant to the country's urban development strategy at closing. The PDOs were also relevant and contributed to the achievement of the World Bank's objectives for the country under the CPF. The relevance of the PDOs to the country's strategy and the World Bank's objectives is rated High. Page 5 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) The PforR program sought to address the development problem associated with the lagging capacity of the poorer cities in the country's Northern region. The PDOs were appropriately pitched to the urban development capacity of the pilot cities. Rating Relevance TBL Rating High b. Relevance of DLIs New DLI RELEVANCE TBL DLI 1 DLI Enhanced Annual City Plans approved and disclosed to the public Rationale DLI 1.1 - Enhanced Annual City Plans (EACPs) approved and disclosed to the public (target: 7). This was a PDO outcome indicator. DLI 1.2 - Professionally-staffed management units in place within each Participating City People’s Committee (target: 7). This was an Intermediate Results Indicator (IRI). DLIs 1.1 and 1.2 were strategic and provided incentives for institutional change. All cities in Vietnam used to prepare two annual planning documents: (i) first, the annual update to the Five Year Social-Economic Development Plan (SEDP) with targets and budget allocation across sectors; and (ii) second, the annual construction investment plan specifying the infrastructure investments for the year. These plans were submitted for approval as data tables without justification for the budget allocated. These DLIs incentivized the cities to consolidate the previous two annual plans as an Enhanced Annual City Plan (EACP) and to ensure adequate staff capacity. The enhanced plan contained a description and rationale for the proposed investments, targeting sub-project service delivery to poor households, monitoring implementation progress, and strengthening the investment prioritization process for the next five-year SEDP. The plans were to include all local government expenditures, including infrastructure investments, under this program. DLI.1 was a PDO level outcome indicator for objective 1. and provided a measure and incentive to achieve the program results. The DLIs were adequately defined, specific, time-bound, highly relevant, and achievable based on the actions included in the Program Action Plan (PAP). The State Auditor of Vietnam (SAV) was designated as the independent third party that verified compliance with the actions to trigger the release of funds. with clear protocols and methodologies for internal monitoring and verification contained in the Program Operations Manual (POM). The amounts allocated to each city were proportional to their population size, subject to a basic minimum allocation to ensure that all the cities could finance at least a basic level of infrastructure investments. The disbursement amounts were weighted to reflect mutually reinforcing relationships, providing adequate incentives for the participating cities to achieve institutional performance targets and infrastructure delivery. Performance-based transfers were allocated to the participating cities subject to the maximum Page 6 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) allocations over the Program's life. These allocations provided an average annual allocation of US$5.5 million per city, or US$47 per capita (ICR, footnote 28). The DLIs were aligned with the PDO and well-integrated into the results framework. The DLIs were consistent with the Theory of Change (TOC); these DLIs facilitated the achievement of the targets of the first Results Area by strengthening planning capacity for infrastructure delivery. DLI 1.1 was a PDO level outcome indicator. DLI 1.2 was an intermediate outcome indicator of the results framework. Actions associated with achieving the DLIs addressed constraints to achieve the PDOs. Overall, the relevance of these combined DLIs to the objectives is rated High. These DLIs facilitated institutional change by combining the previous reports into one plan, with better information to inform constituents; and, second, publishing these plans reflecting investment levels agreed with constituents following a consultative process. Rating High DLI 2 DLI DLI 2 - Local urban infrastructure investments delivered according to each Participating City’s approved Enhanced Annual City Plan (target: Yes) Rationale This DLI was allocated fixed amounts and provided cities with incentives to aim for cost-efficient allocation of resources. The amounts allocated to each city were proportional to their population size, subject to a basic minimum allocation to ensure that the cities could finance basic infrastructure investments. Cost savings could finance additional urban or social infrastructure priorities. Eligible investments under DLI 2 were prioritized using screening criteria, appropriate fiduciary governance standards, and environmental and social safeguards embedded in its definition and verification protocols. These incentivized cities to strengthen their procurement and public expenditure management, public consultation, land acquisition and resettlement practices, and grievance redress mechanisms. The DLI was broadly stated, but the PAP and the POM provided specific, measurable, achievable design standards. However, the first-year investments were to be pre-financed because the DLI would be released after verifying completed works. The cities had the following options - (i) advance of up to 25 percent of their DLI 2 allocation; (ii) counterpart funds of up to 10 - 20 percent of construction costs for a total of about 35 - 45 percent of the construction costs; and (iii) contractors could borrow from commercial sources, although some required guarantees by the cities. Funds would be disbursed after the cities completed the previous years’ works. They are independently verified by SAV using clear protocols and methodologies to meet the infrastructure design, construction, and supervision standards. Achieving this DLI's target would improve the infrastructure's quality (see Section 4 Efficacy below). Completed subproject investments, a grievance redress mechanism, and trained staff in place to support DLI 2. All infrastructure investments were to be selected according to an eligible investment menu. The investments were to be aligned with each city’s approved plan. Projects were to be screened and prioritized with residents' participation as agreed in the POM. The cities had guidelines for a complaints Page 7 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) redress mechanism and capacity support for procurement, contract management, quality control, and social and environmental risk management. DLI 2 served as an intermediate outcome indicator of the results framework and provided the measure and incentive to achieve the program outcomes. These DLIs facilitated achieving the second Results Area targets by delivering the infrastructure services as designed. Overall, this DLI was aligned with the PDO. Incentives in this DLI contributed to the program's performance and results. The relevance of this DLI to the PDO is rated Substantial because of the adequacy of resources for the pre-financing mechanism (see Section 4 Efficacy below). Rating Substantial DLI 3 DLI Asset Management Plan adopted and local urban infrastructure sub-projects in full service after completion Rationale  DLI 3.1 - Asset management plan adopted and local urban infrastructure sub-projects in full service after completion (target: Yes). This was a PDO outcome indicator.  DLI 3.2 – Increase in own-sources revenue. The original target for this DLI was at least a 12 percent increase the previous year but was not reflected in the Operations Portal at appraisal (ICR, footnote.17). The ICR reported a target of 7 instead. These DLIs were designed to support the maintenance needs of the completed infrastructure investments to ensure their sustainability after closing. The amounts allocated to each city were proportional to their population size, subject to a basic minimum allocation to ensure that all the cities could finance at least a basic level of infrastructure investments. The DLIs were specific, measurable, achievable, and consistent with the incentives to achieve outcomes. SAV served as the independent third party that verified results using protocols and methodologies in the POM. The DLIs were aligned with the PDO and the results framework. DLI 3.1 was an outcome indicator under Results Area 3, Sustainable Infrastructure Investments. DLI 3.2 was an intermediate outcome indicator with a 12 percent annual increase target set at appraisal when the inflation rates were 18.7 - 4.1 percent (2011 - 2014). In addition, that target complemented the prevailing government’s policy direction of retaining revenues from corporate income taxes on medium and large enterprises in cities’ budgets. Both contributed to providing incentives to achieve program outcomes. The DLI 3.2 of at least a 12 percent increase in own source revenue was not captured at appraisal in the Operations Portal. The ICR reported this as a target of 7 or all target cities reporting an increase in their own source revenue. This target did not correspond to the definition of the DLI. Overall, these DLIs were aligned with the PDO, one being an outcome indicator. Incentives in this DLI contributed to the program's performance and results. Relevance of this DLI to the PDO is rated Substantial. Rating Substantial Page 8 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) DLI 4 DLI Implementation Strategy for National Urban Development Program adopted with annual milestones Rationale  DLI 4.1 - Implementation Strategy for National Urban Development Program adopted with annual milestones (target: Yes. The PAD noted that this indicator was to be measured by a “Yes/No,” but the end target in the results framework was “NUDP implementation in phase 2 cities commenced,” not just that there is one). The ICR reported on the Yes/No measure. This was a PDO outcome indicator.  DLI 4.2 - Professionally-staffed unit in place in MOC, preparation of Annual Capacity Development Plans, and capacity building support provided to cities in accordance with such plans (target: 100 percent)  DLI 4.3 - Completed Program Report (target: Yes) This Program report was to be prepared by MOC and consolidated the CPCs’ financial statements for Program-related expenditures, incentivized central government’s close monitoring of the Program’s activities, results, expenditures, and performance-based disbursements. These DLIs were designed to enhance the capacity of the national level urban development agencies to achieve the program outcomes. The DLIs were a mix of intermediate and outcome-level results. These DLIs were clearly defined, specific, achievable, measurable, and time-bound, with clear protocols and methodologies for monitoring and third-party verification following the POM. MOC prepared the target for DLI 4.3 (Program report) and consolidated the CPCs’ financial statements for program-related expenditures. This target incentivized the central government to monitor program activities, results, expenditures, and performance-based disbursements. These DLIs were aligned with the PDOs. One DLI (4.1) was assigned as an outcome indicator under Results Area 4, National Policy, Support, and Oversight. This results area recognized the role of the central government (MOC) in improving the enabling environment for urban development and management at the city level by enhancing their capacity with qualified staff to coordinate, monitor, evaluate, and provide capacity support to cities to manage the first NUDP performance-based regional program. Overall, these DLIs were aligned with the PDO. One DLI served as an outcome indicator. Incentives in these DLIs contributed to program performance and results. Relevance of this DLI to the PDO is rated Substantial. Rating Substantial OVERALL RELEVANCE DLI TBL OVERALL RELEVANCE RATING Rationale The PforR PDOs and DLIs remained highly relevant to and aligned with the government and World Bank strategic objectives for the urban sector development in Vietnam, and the PforR approach served to incentivize the achievement of the PDO and Program implementation efficiency. Page 9 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) All eight DLIs were clearly defined and measurable. All had clear protocols and methodologies for internal monitoring. A third party, SAV, independently verified all DLI results, as prescribed in the POM. The DLIs were well structured to support the delivery of the Program’s intended outcomes with generally achievable targets. DLIs complemented each other. For example, DLIs measuring institutional strengthening were critical requirements for infrastructure delivery and sustainability as they supported an enabling policy environment; dedicated local management capacity was essential for effective local planning and sub-project execution; and national oversight was essential to strengthening results monitoring and evaluation (M&E). Overall, the PDOs were aligned with the country's sector strategy for the Northern Province's urban areas. Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To strengthen the capacity of participating Northern Mountains cities to plan urban infrastructure. Rationale Theory of Change (TOC): No TOC was prepared at appraisal, but the results framework provided the causal logic behind the inputs, outputs, and expected outcomes of this objective. The target seven cities lacked the capacity to identify investments that would increase their constituents' economic opportunities and predict their future needs. The cities also lacked the capacity to integrate climate risks into capital investment planning. A TOC was prepared for the ICR. The program was to support the achievement of the objectives of the NUDP by strengthening the capacities of the target cities to plan urban infrastructure. This objective was to be supported by activities under Results Area 1 - Institutional Strengthening at City Level. Inputs: were to consist of professional training of city-level management staff in infrastructure development planning processes and prioritization following the city’s socio-economic development strategy. Inputs were also to include training the professional city staff in capital investment planning, economic and financial feasibility analysis, master planning techniques, climate risk mitigation and adaptation, and analyzing the poverty reduction impact of the investments. Outputs were to include the published Enhanced Annual City Plans (EACPs) and professionally trained staff in city management units. Outcome was to strengthen the capacity of participating Northern Mountains cities to plan urban infrastructure, as evident in these published EACPs. The TOC included the following assumption, which, if actualized, would increase the likelihood that the outcomes would be achieved: (i) an independent verifying agency conducted annual performance assessments on time. This assumption occurred and boosted the outcome of this objective. The TOC was clear and convincing. The outputs provided a logical result to the project activities or inputs. The strengthened planning capacity led to the publishing of an Enhanced Annual City Plan ready for Page 10 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) implementation. No logical gaps were noted in the inputs that led to outputs that led to outcomes. The outcome was attributable to the project. OUTPUTS: Under Results Area 1, Institutional Strengthening at City Level, the following output targets were achieved:  IR 1 and DLI 1.2 - All seven cities had professionally staffed management units in each participating City People’s Committee (CPC) as targeted. All seven cities maintained professionally staffed management units, including a director and two vice directors, a chief accountant, three technical engineers, a procurement expert, an environmental safeguards expert, a social safeguards expert, an M&E expert, and two administrative staff, as prescribed in the POM. MOC provided annual planning support and technical assistance to cities’ management unit staff to build their capacity in capital investment planning, feasibility analysis, asset management, and investment M&E. All seven cities retained the majority of their management units’ staff after Program closing given their strong planning, implementation, and monitoring capacities built during Program implementation.  All output targets under his results area were achieved. OUTCOMES: Under Results Area 1, the following outcome targets were achieved:  DLI 1.1 - The strengthened capacity of participating Northern Mountain cities to plan urban infrastructure was evident in the publication of the EACPs, achieving the target. All seven cities developed their plans, including local government expenditures and infrastructure investments under this program. The plans emphasized service delivery to poor households, described implementation status and strengthened the investment selection and prioritization process underlying the next five-year SEDP. The plans consolidated the previous planning documents, reflecting improved investment selection and prioritization processes. The plans included the following: o A statement of anticipated revenues and expenditures of the CPC for the following financial year. o A detailed list of all infrastructure sub-projects that would be implemented in the following year, with a rationale for each sub-project. o The status of ongoing and completed infrastructure investments and budget allocations for their O&M allowed for closer monitoring of expenditures and progress. o All infrastructure sub-projects proposed for implementation in the following year were reviewed, screened, and prioritized using the following weighted criteria-based system:  positive economic impacts, including a significant impact on poverty reduction and local economic development  access to markets and economic development opportunities for poor households in peri-urban areas  positive environmental impacts and increased community resilience to climate change and natural disasters  benefits to a high proportion of poor and low-income households, addressing women and ethnic minority community members, and promoting gender equality  minimal negative social impacts on communities. Page 11 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Overall, the project's efficacy in achieving this objective is rated Substantial. All outcome targets were achieved and provided evidence for improved capacity to plan infrastructure investments. Rating Substantial OBJECTIVE 2 Objective To strengthen the capacity of participating Northern Mountains cities to implement urban infrastructure. Rationale Theory of Change: The Program was to support the NUDP in strengthening the capacities of selected cities in the Northern Mountains Region. to implement urban infrastructure. Inputs: were to include financing local urban infrastructure investments according to the EACPs. Inputs were also to include the training of staff in fiduciary (procurement) and safeguards requirements, establishing local grievance mechanisms, resettlement, compensation practices, and guidelines to ensure the quality of designs; ensuring the participation of ethnic minorities, women, and vulnerable groups in planning and prioritization of investment needs; and capacity to supervise subproject implementation. Outputs were to consist of four Intermediate Results (IRs), including people benefiting from primary/secondary and tertiary/social infrastructure services, new and upgraded roads and bridges completed, resettlement areas completed, and low-income areas upgraded. The outputs were also to include the staff trained. The results of these infrastructure investments were to meet the conditions of DLI 2 under Results Area 2. All infrastructure investments were to be selected according to an eligible investment menu, screening, and prioritization criteria in the POM and aligned with the approved EACP. The POM included constituents' participation guidelines, a complaints redress mechanism, and capacity support for procurement, contract management, quality control, and social and environmental risk management. The TOC was clear and convincing. The inputs or activities were logically linked to the outputs and the outcomes. There were no observed logical gaps other than the assumptions that would have pointed to the risks of implementation delays and justifying the target values or lack thereof for the infrastructure investments to be completed. All expected outputs and outcomes were attributable to the project. In addition to the assumption under Objective 1, the TOC included the following assumptions, which, if actualized, would increase the likelihood that the outcomes would be achieved: (i) program funds and government counterpart funds are transferred in a timely manner, (ii) local contractors and supervision consultants work with sufficient capacity and meet the infrastructure quality standards, and (iv) capacity support activities are implemented. However, since this objective was focused on implementing the investments, assumptions surrounding the financing needs to pre-finance the investments were omitted. DLI 2 was associated with this objective. The POM required cities to pre-finance the construction of the priority infrastructure using up to 25 percent of their DLI 2 allocation plus 10-20 percent of their counterpart funds. Construction companies would borrow from other sources. The government adopted new policies at implementation that limited the resources available to cities to use in pre-financing infrastructure investments. Those new policies caused implementation delays and affected the achievement of the targets for DLI 3.2 (see below). Page 12 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) OUTPUTS The urban infrastructure investments prioritized in the city plans were completed at closing. The targets were identified during the planning process. A total of 128 infrastructure sub-projects (total cost of US$284 million (VND6,597 billion), of which construction cost was US$205 million (VND4,765 billion) The following primary/secondary infrastructure services were reported:  IR 2 - 739,125 people benefited from primary/secondary infrastructure services, exceeding the target of 651,500.  IR 5 - 128.67 km of urban roads were built, exceeding the target of 101.9 km (8.2 km were new, and 120.47 kilometers were upgraded). In most of the 63 sub-projects, roads were upgraded in all seven cities by resurfacing, repairing, or providing adequate drainage, streetlighting, or pedestrian paths. Their locations and types of upgrading were screened and justified according to the criteria in the plans. The new roads were linked to existing roads or provided access to peri-urban areas. 10 bridges were constructed, and two bridges, seven embankment works, three channels, and four reservoirs were rehabilitated. All targets for these specific investments were established during implementation as each EACP was completed. The following tertiary/social infrastructure services were supported:  IR 3 - 151,481 people benefited from improved tertiary/social infrastructure services (disaggregated by gender), exceeding the target of 107,800. There were 62,822 female beneficiaries, exceeding the target of 53,275.  IR 4 - 16 percent of the beneficiaries were Ethnic Minority beneficiaries, exceeding the target of 14 percent. This represented the proportion of ethnic minorities, male and female, benefitting from the program out of the total ethnic minority population, male and female, in the seven provinces.  IR 6 - Developed 291.3 hectares of resettlement area, exceeding the target of 236 ha. Five sub- projects developed resettlement areas in five cities (Bac Kan, Dien Bien Phu, Thai Nguyen, Tuyen Quang, and Yen Bai) with roads, drainage, and utility service lines for households relocated under the program or other government-financed projects (in the case of Dien Bien Phu). Two high schools, ten kindergartens, and three community centers were upgraded or newly built. All targets for these specific investments were established during implementation as each EACP was completed.  IR 7 - Upgraded 592.9 hectares of low-income areas, exceeding the target of 333 ha. Seventeen sub- projects upgraded low-income areas in four cities (Cao Bang, Dien Bien Phu, Tuyen Quang, and Yen Bai) with various interventions, such as road resurfacing or repair, road-side drains, water supply pipes, or street lighting. All targets for these specific investments were established during implementation as each EACP was completed.  IR 8 and DLI 3.2 (Increase in own-sources revenue). 3 of the 7 target cities achieved an increase in their source revenue. At appraisal, the original target was for cities to increase their own-source revenues by 12 percent over the previous year. This target was set because of high inflation rates (see Section 3 above, Relevance of DLI 3.2). Between 2015 and 2021, inflation rates declined to 0.6 - 3.5 percent. At the same time, the government adopted new budget policies that required local governments to transfer collected taxes on companies with registered capital of more than VND10 billion to provincial revenues. This limited the city's ability to increase its source revenues and led to the unmet original target of a 12 percent increase from the previous year. However, since this target remained unreported in the Operations Portal at appraisal, a target of 7 cities was reported at closing (see also Objective 3 below for IR 8 and DLI 3.2) Page 13 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) OUTCOMES:  The capacity of participating Northern Mountains cities to implement urban infrastructure was strengthened. The program benefited 912,508 people, exceeding the target of 759,300, with 434,340 females exceeding the target of 374,869 females.  The ICR report claims results at the impact level from informal interviews with selected beneficiaries, such as health benefits, the value of reduced incidence of accidents (vehicular or pedestrian), diseases or hospitalization, increase in value of traded goods, or satisfaction rates: o In Informal interviews, beneficiaries reported that resurfaced roads reduced travel time, dust, and traffic accidents. Road-side drains made roads passable. Students attended classes during the rainy season. New roads increased access to markets and public services. o The relocated households were satisfied with their well-serviced land, clean environment, and accessibility to markets and public services. o The resurfaced roads helped residents save time and physical effort when bringing their agricultural produce or livestock to markets. Road-side drains helped their hygiene condition by protecting them from flooding and resultant exposure to infiltrated pollutants of septage or agricultural chemicals. New streetlights helped extend productive or social activities. Upgraded areas increased the value of their properties. The upgraded areas offered new business opportunities to sell food or agricultural produce to more visitors to their neighborhoods. o None of these claims were triangulated with other evidence or data and can thus not be validated.  City representatives interviewed at closing, and MOC officials confirmed improved overall quality of infrastructure because of the rigorous application by SAV of standards for infrastructure design, construction, and supervision before releasing DLI 2 resources.  Cities improved the cost-effectiveness of their investments by using cost savings to finance more investments than initially planned. Bac Kan built another school in addition to the planned 12 infrastructure sub-projects; Thai Nguyen upgraded another road intersection in addition to the planned 12 infrastructure sub-projects; and Tuyen Quang upgraded two link roads in addition to the planned 16 infrastructure sub-projects (ICR, footnote 29). Overall, the efficacy of the project to achieve this objective is rated Substantial, with moderate shortcomings. All output targets were achieved as evidence of improved capacity to implement the infrastructure investments. In some cases, cities encountered implementation delays because of national government expenditure policies that affected the cities’ ability to pre-finance investments. The moderate shortcoming comes from a lack of indicators to signify a strengthened capacity to implement beyond the number of beneficiaries. There was no data on how these infrastructure investments implemented by the cities improved the lives and livelihoods of its residents other than cited anecdotes. Rating Substantial OBJECTIVE 3 Objective To strengthen the capacity of participating Northern Mountains cities to sustain urban infrastructure. Page 14 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Rationale Theory of Change: The Program was to support the NUDP in strengthening the capacities of selected cities in the Northern Mountains Region to sustain the built urban infrastructure. This objective was to be met under two Results Areas: (3) Sustainable Infrastructure Investments and (4) National Policy, Support, and Oversight. The strengthened urban development capacity at the national level was an implied outcome. Inputs were to include asset management training and enhancing cities’ own-source revenue generation. Inputs were also to include training of the central government in urban development to provide capacity support to cities, such as in policy formulation, procurement, contract management, internal and external audit, monitoring and evaluation (M&E), and verification of results. Inputs also included preparing an implementation strategy and expenditure framework for the NUDP rollout. Outputs included asset management guidelines, plans, revenue enhancement strategies, a professionally staffed management unit in the Ministry of Construction (MOC), and an implementation strategy for the NUDP. The completed program report was to support DLI 4.3. As noted, a change in government policy on tax revenue systems led to a decrease in cities’ own-sources revenue, affecting their ability to achieve one of the sustainability targets fully. There was also a legal constraint prohibiting cities from determining the tax rates and utility service charges, further affecting cities’ ability to allocate resources for O&M needs. Outcomes were expected to be enhanced fiscal capacities of cities to fund, operate, and maintain their assets (improved mechanism for infrastructure O&M) and sustained capital investments evident in adopting the asset management plans and increase in its own-source revenues. Additionally, the enhanced capacity of the MOC to coordinate and guide participating cities in implementing results-based programs and financial mechanisms of the NUDP was an implied outcome. DLI 3.1 was supported by the cities adopting its asset management plans with fully functioning infrastructure investments. DLI 3.2 was supported by an increase in own source revenues for O&M needs. DLI 4.1 was supported by the implementation of the NUDP strategy with milestones. DLI 4.2 was supported by having professional staff at MOC and training delivered to city officials according to the Annual Capacity Development Plans. The TOC was clear and convincing. The activities and inputs provided the causal logic that led to outputs and outcomes. There was a gap in the lack of mitigating measures to address risks of possible national policies that may run counter to the intent of the inputs under this TOC. There were also no indicators to strengthen the causal link between inputs, outputs, and outcomes of national level MOC capacity improvements in urban development that would enhance city-level urban management. The expected outputs and outcomes were attributable to the project. The assumption under the first objective also applied to this TOC. However, since this objective focused on the sustainability of the completed investments, the TOC could have added an assumption regarding complementary national government policies that supported cities' efforts to raise their resources for O&M. See Outputs below. OUTPUTS: The following expected outputs were under Results Area 3:  IRI 8 or DLI 3.2 - 3 of the target 7 cities increased their source revenues for O&M. In the PAD, the seven cities were to target a minimum 12 percent year-on-year increase in own-sources revenue, excluding revenues from land transactions and rentals. This target was unmet because the government adopted a new tax revenue policy requiring cities to transfer taxes collected from companies with registered capital of more than VND10 billion to provinces. Cities could also not determine the tax rates and utility service charges for local infrastructure services. The State Auditor of Vietnam reported varying results for increasing own-source revenues: in 2015, all cities except Yen Page 15 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Bai reached a minimum 12 percent increase in own-sources revenue compared to the previous year; in 2016, only two cities (Hoa Binh and Tuyen Quang) met the target; in 2017, only Thai Nguyen met the target; in 2018, four cities (Bac Kan, Cao Bang, Thai Nguyen, and Tuyen Quang) achieved the target; in 2019, when COVID-19 broke out, none of the seven cities achieved the target; and in 2020, three cities (Bac Kan, Cao Bang, and Hoa Binh) achieved the target.  IR 9 and DLI 4.2 - 100 percent of professionally trained staff were in place at MOC, including a director, two vice directors, a finance management and internal auditor, an institutional development and capacity building expert, two technical engineers, a procurement expert, an environmental safeguards expert, a social safeguards expert, an M&E expert, and two administrative staff, as prescribed in the POM, achieving the target. MOC provided capacity support to cities to complete their Annual Capacity Development Plans, achieving the target.  IR 10 and DLI 4.3 - MOC completed annual Program reports, summarizing key implementation aspects, results, and status of PAP actions, as targeted.  All cities adopted asset management plans as targeted. These plans featured the quantity of identified assets per asset class (roads, bridges, drainage, embankments, retaining walls, water supply system, streetlights, buildings, etc.), their use-life, age, condition, estimated annual maintenance cost, responsibilities, and procedures for planning and conducting maintenance tasks; and the operation of completed sub-projects as designed. OUTCOMES:  DLI 3.1 - The strengthened capacity of participating Northern Mountains cities to sustain urban infrastructure was evident in the adoption of their respective Asset Management Plans and fully functioning local urban infrastructure sub-projects benefiting 912,508 people exceeding the target of 759,300, with 434,340 females exceeding the target of 374,869 female beneficiaries. The PAD noted a Yes/No unit of measure target and an end target of “a minimum of 80 percent of local urban infrastructure sub-projects completed free from physical damage and fully function as designed.  DLI 4.1 - At the national level, the outcome of a strengthened national institutional capacity for urban development was evident in adopting the Phase 2 NUDP Implementation Strategy for cities in the Mekong Delta Region in 2019 with annual milestones. The implementation of this strategy commenced in 2020. Overall, the efficacy of the project to achieve this objective is rated as Substantial with moderate shortcomings. While the target for the adoption of Asset Management Plans and fully functioning local urban infrastructure sub-projects was achieved (and exceeded in terms of the beneficiaries), the outcome regarding own source revenue generation to sustain O&M of completed assets was partially achieved. Rating Substantial OVERALL EFFICACY Rationale Page 16 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Overall efficacy is rated Substantial. The efficacy of the project to achieve the first and second objective is rated Substantial, while the third objective is rated Modest because of partial achievement of the target regarding own source revenue generation to fund the O&M needs of the completed assets. Seven of the eight DLIs were substantially achieved. Rating Substantial 5. Outcome The overall outcome is rated Satisfactory. The relevance of the program objectives and associated DLIs were aligned with the government priorities and the World Bank's strategic objectives for developing the urban sector in Vietnam. Relevance of objectives is rated High. The overall efficacy of the program to achieve all its objectives is rated Substantial with minor shortcomings. The overall outcome is rated Satisfactory. Outcome Rating Satisfactory 6. Risk to Development Outcome The following pose risks to development outcomes (ICR, paragraph 82)  Technical Risk: In this program, the central government piloted the use of the EACPs by the participating cities. These plans combined two previously prepared annual reports. These EACPs provided the justification and prioritization of the city's investments with provisions for monitoring by residents since the information is made public through websites. Keeping this information updated is a moderate risk. CPCs could implement a program for periodic rollout of updates.  Financial Risk In this program, all seven cities allocated O&M budgets for the completed infrastructure according to their Asset Management Plan and to meet the requirements of DLI 2. All cities have committed to continuing to allocate O&M budgets and staff in the future. However, urban infrastructure is not a revenue-generating service, and rate-setting for the services is not determined by local authorities and is not likely to change soon. MOC and the cities would need to be creative in raising source revenues, setting rates that meet O&M budgets, and implementing cities' asset management plans.  External shocks: In this program, implementation was challenged by the COVID-19 pandemic. Such external shocks could reduce the ability of cities to raise revenues or disrupt the planning process to meet urban infrastructure needs. The NUDP and the modalities implemented in this program set out the processes cities need to implement - screening and prioritization - of infrastructure needs. This program introduced the EACPs that cities would continue to use under the NUDP to prioritize infrastructure investments to reduce poverty and create potential regional hubs. Page 17 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) 7. Assessment of Bank Performance a. Quality-at-Entry The World Bank's performance in ensuring the quality at entry is rated Satisfactory. The World Bank team designed this PforR with the government’s urban development priorities embodied in the NUDP. This operation was the first PforR regional pilot of the NUDP. The team used the PforR instrument to target seven Northern Mountains cities selected because of their potential as regional economic centers—design prescribed prioritization of investments with high socio-economic results to optimize development impact. Adequate technical, fiduciary, and environmental and social systems assessments informed the design of the incentive schemes. Mitigation measures to address risks were adequate. These were included in the Program Action Plan (PAP) and the Program Operations Manual (POM). However, the team acknowledged a couple of shortcomings. One, measures in the PAP to address land acquisition risks (e.g., site clearance and compensation) could have benefited by adding measures such as escrow accounts for land acquisition, sharing best practices in market valuation, and how to manage grievances. Second, some risks were missed at the appraisal that occurred at implementation. These included changes in tax policy, inflation, adopting the national Medium-Term Investment Plan (MTIP, a five-year fixed investment and budget plan of the national government) with insufficient project resources, and cities’ ability to pre-finance infrastructure investments to raise their revenues. Mitigation measures could have included a more modest target for annual own- sources revenue increase, larger DLI advances, or closer engagement with the government for more sufficient budgets. Lessons from World Bank urbanization reviews in China, Colombia, and India informed the design of DLIs. These included the importance of sustainable urbanization finance, connecting smaller cities to a system of cities, and capacity building for improving urban service delivery. Findings of urban analytical work in Vietnam on evaluating investment performance on service delivery and assessing the constraints and opportunities for enhancing local government capability to access financing for infrastructure development also informed design (ICR, footnotes 8 and 9). Overall, the Bank's performance at entry is rated Satisfactory. The readiness to implement was reflected in the absence of conditions when the project was to become effective. All design elements, including the POM, PAP, DLIs, and Project Management Units (PMUs) at city and MOC levels, were in place. Initial sets of infrastructure investments were identified. Feasibility studies were completed. Minor shortcomings at entry were the lack of an outcome indicator relative to the impact of both the training and technical assistance on the improved national urban development capacity, an overestimated target in own- sources revenue, missing the risks associated with the national policies limiting local tariff setting or own source revenue targeting. Quality-at-Entry Rating Satisfactory b. Quality of supervision Page 18 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) The World Bank team conducted 16 supervision missions with technical and operational experts over the 8-year implementation period. Key staff were based in Hanoi. This facilitated close dialogue with MOC and the cities, continuous implementation support, and regular field missions. The World Bank team conducted additional technical visits between missions to clarify emerging issues or assist the cities as needed. Monthly coordination meetings were held among MOC and the participating cities’ Project Management Units (PMUs). The World Bank team trained MOC and the cities on the PforR instrument, environmental and social safeguards, procurement, and financial management. The World Bank team acknowledged (ICR, paragraph 80) that a PforR performance-based financing expert in the team could have helped to address delayed contractor payments. Risk assessment and mitigation efforts did not anticipate the legal impediments for cities to determine their local tax rates and utility service charges, affecting their capacity to generate their source revenues. A national tax policy change also affected this outcome, which was associated with DLI 2 (ICR, paragraph 58 and footnote 33). The World Bank team conducted the Mid-term Review (MTR) in May 2018 and identified the following issues: (i) insufficient budget allocation in the MTIP and subsequent delays in the annual budget allocation to the cities; (ii) delays in land clearance and resettlement; and (iii) cities unlikely to meet DLI 3.2 targets on the increase in own-sources revenue. The following corrective measures were adopted: (i) securing additional MTIP allocations by closely coordinating with MOC, Provincial People's Committees (PPCs), the Ministry of Finance, and the Ministry of Planning and Investment; (ii) closer monitoring of land clearance and resettlement actions, and (III) detailed assessments of the cities’ own-sources revenue. Annual budgets began to increase by early 2020. Land clearance and resettlement actions improved. The 2021 restructuring extended the program closing date but left the target for DLI 3.2 unchanged. The COVID-19 pandemic, beginning in 2020, led to implementation delays affecting consultations with communities on infrastructure investment planning, mobilization of contractors to construction sites, and steering and monitoring meetings and field visits. The World Bank team extended the closing date to complete delayed activities. Overall, the quality of Bank performance at supervision is rated Satisfactory with moderate shortcomings regarding the mitigating measures to address policy changes to facilitate cities meeting conditions for disbursement. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 8. M&E Design, Implementation, & Utilization a. M&E Design The M&E system was designed to coordinate data flow between city and national governments, assisted by an independent third-party evaluation agency, the State Auditor of Vietnam (SAV). The TOC, constructed at closing, showed the causal link from key activities to outputs, intermediate results in the results framework that achieved targets that triggered disbursements to achieve the expected Page 19 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) outcomes. The objectives were clearly stated and supported the overall NUDP objectives. The indicators encompassed all relevant outcomes. The DLIs were clearly defined, measurable, and aligned with the PDOs. The intermediate results indicators and the outputs captured the contribution of the activities to achieving outcomes. The indicators were specific, measurable, achievable, relevant, and time- bound. All indicators had baselines and targets, data sources, and methodologies. However, the M&E design did not include an outcome indicator for the strengthened national urban development capacity. The targets for cities to generate their own-source revenues proved to be unrealistic. b. M&E Implementation The MOC implemented the M&E system as designed. The SAV implemented the verification protocols and completed their annual assessments. Each city formed a community monitoring group and supplemented the reporting of implementation progress. MOC and the cities prepared mid-term reports and completion reports. The noted weakness in design regarding the lack of outcome indicators for strengthened urban development capacity at the national level remained unaddressed at implementation. At closing, informal surveys were conducted to assess impacts on job creation, increase in beneficiaries’ income, and increased business opportunities. c. M&E Utilization M&E data was used to inform program progress and prepare corrective measures in response to implementation challenges. The results verification reports determined disbursements. M&E data- informed MOC on areas where cities needed to improve. The World Bank supervision team and MOC used M&E reports to assess progress toward achieving the outcomes. M&E data informed the restructuring and extension of the closing date. Overall, the M&E system is rated Substantial. The M&E was well designed, except for the lack of an outcome indicator that would measure strengthened urban development capacity at the national level. Cities and MOC measured all outcomes and acknowledged the overly optimistic target of own-sources revenue increase by cities. MOC, SAV, and the cities implemented M&E as designed, and data was used to inform the operation. M&E Quality Rating Substantial 9. Other Issues a. Safeguards Environmental and Social Safeguards. The project was not assigned an Environmental Category and did not trigger safeguards policies. Instead, as a PforR, the project triggered the Bank’s Operational Policy/Bank Procedures (OP/BP) 9.00 (Program-for Results Financing), establishing the policy and planning elements necessary to achieve outcomes consistent with the PforR objectives. The Bank policy OP/BP 9.00 Program for Results (PforR) Financing was first issued in February 2012 and revised in April 2013 and April Page 20 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) 2015. An Environmental and Social Systems Assessment (ESSA) examined the program's scope, context, and potential impacts from an environmental and social perspective. The environmental and social management systems and implementing capacities of the respective government agencies participating in the Program were evaluated as to their consistency with the core principles in the). The assessment concluded that the applicable systems were consistent with OP/BP 9.00 (PAD, paragraph 84). The consolidated social and environmental risk assessment was substantial, but typical investments were small to medium-scale civil works with moderate environmental impacts. The program DLIs and actions in the PAPs addressed environmental risk mitigation measures (see Section 3 Relevance above). The POM guided the cities on procedures, such as environmental codes of practice (ECOPs) and worker’s code of conduct, to avoid, minimize, and mitigate environmental and social impacts. The program supported the MOC in building environmental risk management capacity in each city. Environmental Protection Plans and Environment Impact Assessments were prepared for infrastructure sub-projects and publicly disclosed in the CPC offices, relevant community offices, and sub-project sites. MOC and each city assigned staff to be responsible for the program's environmental, health, and safety (EHS) aspects. Construction supervisors were assigned to monitor contractors’ compliance with EHS aspects. Contractors undertook timely corrective measures to address any concerns in meeting EHS requirements (e.g., disposal of topsoil, use of personal protection equipment, and signage at construction sites). Social Safeguards: In social risk management, each sub-project prepared detailed plans for compensation, support, and resettlement involving land acquisition or physical displacement of people or properties and publicly disclosed in the CPC offices, relevant community offices, and sub-project sites to comply with national requirements and the POM. MOC and each city assigned staff to coordinate land acquisition and resettlement activities. Long-standing resettlement grievance cases were resolved. Implementation delays in site clearance, land acquisition, and resettlement were marked by (i) a lack of sufficient funds to undertake site clearance and land acquisition, (ii) CPCs lengthy appraisal and approval of design and land acquisition plans, and (iii) delays in obtaining consent to resettle from affected households (ICR, paragraph 60). Some construction works were delayed and contributed to the justification of extending the program closing date because resettlement site development lacked funding (e.g., in Dien Bien Phu). No pending cases were reported at closing. All cities conducted community participation and consultations as planned. The program complied with environmental, health, and safety (EHS) requirements and implemented PAP actions that indicated compliance with environmental and social risks (ICR, paragraphs 70-71). b. Fiduciary Compliance Financial management. Financial management complied with the World Bank policy for PforR financing (ICR, paragraph 76). The POM included guidance to MOC and the cities on fiduciary requirements and procedures, with specific guidance on the performance-based transfer system and funds flow arrangements. All required financial audit reports were submitted on time with unqualified opinions. Financial issues raised in management letters, such as recategorizing certain expenditures and arithmetic errors, were corrected. MOC recruited an independent consulting firm to conduct internal audits in all participating provinces. The provinces and cities implemented their recommendations. The delayed annual budget allocations by the PPCs affected the program’s disbursement rate in 2016-2019 (ICR, paragraph 75). Page 21 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Procurement. Procurement complied with the World Bank policy for PforR financing (ICR, paragraph 76). The PAP included a set of procurement-related actions: (i) Action 2 - procurement practices through competitive bidding, excluding dependent state-owned enterprises, ensuring valid bids, and excluding debarred firms; and (ii) Action 3 - effective internal audit function. The results verification reports confirmed that the cities satisfactorily complied with the national procurement law, regulations, and procurement- related PAP actions. Nonmaterial shortcomings and deviations in cost estimation, contract amendment, price adjustment, and contract documentation were noted and addressed. Most selected contractors were qualified (ICR, paragraph 74). c. Unintended impacts (Positive or Negative) --- d. Other --- 10. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- High 11. Lessons The ICR presented six lessons from the operation, slightly paraphrased below (ICR, paragraphs 83- 88):  Well-defined results assessment protocols and a credible and comprehensive verification system may help achieve the planned outcomes. This program used a transparent M&E system at national and city levels, linking it with the MIS and independent verification. The verification-based disbursement process was an important learning-by-doing tool that incentivized results. The verification protocols included the engagement of a credible independent agent with clear responsibilities and capacity. World Bank quality assurance improved the credibility of the performance assessment SAV conducted. Cities claimed that SAV verification improved the quality of their infrastructure investments.  DLIs and PAPs may target inclusive urban management, strengthen management capacity, and contribute to poverty reduction in the target area. In this program, DLIs were used to strengthen capacities at the city and national levels. DLIs focused on improving the cities’ capacity and systems to plan and implement urban infrastructure investments. National capacity to support cities in urban management processes also improved. PAP Page 22 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) actions ensured the delivery of quality infrastructure that benefited the poorer, vulnerable groups such as women and ethnic minorities in these target areas.  Adding expenditure and payment strategies may boost a city's ability to pre- finance infrastructure works. In this program, payment delays to contractors resulted from the city's inability to pre-finance infrastructure works. DLI advances, and available counterpart funds also proved insufficient. MTIP also had inadequate allocations for both IDA and counterpart funds. Future design may find adding expenditure and contractor payment strategies helpful to help cities address their lack of capacity to pre-finance investments.  If national policies do not reflect support for cities to generate their own resources and set local taxes and user fees for service delivery to residents, this may affect the cities’ capacity to sustain the assets they build and the service delivery provided to their residents. The cities' fiscal condition would be enhanced if they could increase their revenues from their sources. In this program, cities increasingly assumed more responsibilities to deliver urban infrastructure and services to residents. These cities often lacked the management capacity to plan, implement, manage, and sustain the financing needs of the investments and provide for O&M from their resources. The 12 percent annual increase target for revenues set at appraisal in response to high inflation rates proved unachievable. This was compounded by a change in tax and public financial management policies set at the national level that negated local resource mobilization efforts.  PforR financing is a useful instrument to improve the performance of both national and sub-national governments. In this program, the result indicators focused on improving the performance of both the national and sub-national governments. DLI targets enhanced the national government’s capacity and improved systems for urban policy and implementation support to cities. The program demonstrated the effectiveness of creating synergies in assisting two aspects in delivering infrastructure services. One focused on strengthening the national agencies' policy environment and oversight system, while the other increased technical support to cities. 12. Assessment Recommended? No 13. Comments on Quality of ICR The ICR followed the guidelines and provided a clear picture of the operation. The ICR was results-oriented, highlighting the impact of the reforms introduced under the program. The ICR was internally consistent. The salient points of the operation were adequately summarized. References throughout the report established links from the PAP and the POM to the SAV use of clear protocols and methodologies to verify achievements. Evidence was clearly linked to the outcomes. The analysis of the evidence justified the ratings. The annexes provided visual evidence of the achievements reported under the program. Lessons were practical and based on the operation. The ICR was candid, acknowledging the risks that were missed, the lack of a relevant outcome indicator, and the overambitious target for one DLI. Page 23 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Nat'l Urban Devt Program - NM (P143596) Overall, the quality of the ICR is rated High. a. Quality of ICR Rating High Page 24 of 24